{"id":"dcedd95d-746e-43f1-9183-84df8e420f7d","arxiv_id":"2501.03989","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An expository review of information retrieval models and the technical challenges of implementing the right to be forgotten through de-indexing.","lead":"This paper is a tutorial explaining how search engines index and retrieve content, then discusses how the right to be forgotten relies on de-indexing and may affect semantic relationships in embedding spaces. A generalist might read it to understand the technical basics behind privacy-related content removal online.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3's central claim conflates de-indexing with retraining: removing a URL from a search index changes retrieval results, not embedding-space coordinates, so the alleged 'weakening' of associations is at best a metaphor.","rationale":"The reader's verdict was UNVERDICTED, with the weakest assumption being that de-indexing affects the embedding space itself. I agree with this assessment and with the specific identification of Section 3 as the locus of the problem. The paper is an introductory review, and most of Section 2 is a competent summary of classical IR models and LLM basics. The only substantive argument connecting de-indexing to semantic dynamics is the Smith/X/Y example, and that example is technically inaccurate if read literally. However, because the paper does not present this as a formal research claim or offer empirical evidence, the appropriate disposition remains UNVERDICTED rather than ACCEPT or REJECT. The concern does not change the reader's verdict, but it strengthens the case for treating the paper as a tutorial whose Section 3 should be revised. The concrete test would clarify whether the authors intended a literal claim about embeddings or a metaphorical statement about retrieval visibility; if the latter, the text should be rewritten to avoid misleading non-experts, which is the paper's stated audience.","tokens_in":10845,"tokens_out":3423,"duration_ms":34622,"concrete_test":"Use a fixed sentence embedding model (e.g., all-MiniLM-L6-v2) on a small corpus containing three documents: D1 linking Smith and X, D2 linking Smith and Y, and a control document. Compute cosine similarities between embeddings of Smith-related text and X-related text (and Y-related text) before and after D1 is removed from the search index. Because the model is frozen, all pairwise similarities remain bit-identical; only retrieval results change. Then, to test the stronger reading, apply a machine-unlearning method that removes D1's influence and recompute similarities. The decisive check is whether the paper's 'embedding space' claim is about frozen vectors or updated weights: if only retrieval results change, the Section 3 mechanism is metaphorical, not literal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central illustrative mechanism (Section 3, paragraph 4) is that de-indexing 'disrupts their contextual relationships in the embedding space,' and specifically that if documents linking 'Smith' and 'X' are de-indexed, 'their semantic association weakens' while associations with 'Y' may 'grow stronger.' This is presented as a dynamic adjustment of the embedding space with profound implications for retrieval and collective memory. The claim conflates two distinct objects: the embedding model, whose parameters and output vectors are fixed once trained, and the retrieval index, which determines which embedded documents are returned for a query. Removing a URL from a search index changes the candidate set and therefore changes which associations are surfaced to users, but it does not move the remaining document vectors or alter their pairwise cosine similarities. The 'weakening' of Smith-X and 'strengthening' of Smith-Y is at most a statement about co-occurrence in ranked results, not about embedding geometry. Because the paper explicitly frames this as 'dynamic adjustment of relationships in the embedding space' and draws conclusions about machine forgetting and collective memory, the technical inaccuracy is load-bearing: the claimed unintended consequences rest on a mechanism that, as stated, does not occur. A corrected version would either restrict the claim to retrieval-visible associations or move to a machine-unlearning setting where model weights are actually updated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a tutorial-style preprint aimed at non-experts, introducing the foundations of information retrieval (boolean, vector space, probabilistic, embedding-based models, and LLMs) and connecting them to the legal and technical practice of de-indexing in the context of the right to be forgotten (RTBF). The first half of the paper is a concise review of classic IR models and modern neural retrieval, with formal definitions of the term-document matrix, inverted index, cosine similarity, BM25, word/document embeddings, and transformer attention. The second half argues that de-indexing, while leaving the original web content in place, reduces its visibility and 'disrupts the contextual relationships in the embedding space,' using a hypothetical 'Mr. John Smith' example to suggest that removing documents linking a person to a sensitive topic weakens that association and may strengthen others. The paper concludes with a discussion of machine unlearning, retrieval-augmented generation (RAG), and the broader societal implications for collective memory and privacy.","tokens_in":70,"tokens_out":3790,"duration_ms":95014,"significance":"If its central mechanism were accurate, the paper would highlight a subtle and practically important consequence of RTBF: that de-indexing does not merely hide a URL but reshapes the semantic associations a user sees. The tutorial sections are largely sound and well-suited to a non-specialist audience; the derivations of TDM, inverted index, VSM, BM25, and attention are correct and clearly presented. The paper also usefully introduces machine unlearning and RAG as complementary or alternative mechanisms for forgetting. However, the paper's distinctive claim is currently stated in a way that conflates the retrieval index with the embedding model, and it is not supported by data or a formal argument. Once that claim is corrected or explicitly reframed as speculation about retrieval-visible associations, the paper could serve as a readable primer for legal and policy audiences, but in its current form the central technical assertion is not reliable.","major_comments":[{"comment":"The claim that de-indexing 'disrupts their contextual relationships in the embedding space' is technically inaccurate as stated. Removing a URL from a search engine's index changes the candidate set of documents that can be retrieved for a query, but it does not alter the learned embedding vectors of the remaining documents or their pairwise cosine similarities. The subsequent example about 'Smith' and 'X' weakening and 'Smith' and 'Y' growing stronger is therefore at most a statement about which associations appear in ranked results, not about geometry of the embedding space. Because this mechanism is used to argue for 'profound implications' and 'machine forgetting,' the paper should either restrict the claim to retrieval-visible associations or explicitly move the discussion to a machine unlearning setting where model weights are actually updated.","section":"Section 3, paragraph 4"},{"comment":"The paper itself concedes that 'it is not easy to predict the consequences that such an operation can have on the embedding space at a larger scale,' which is an acknowledgment that no evidence or formal argument is provided for the central illustrative claim. For a claim that is load-bearing for the paper's privacy implications, the manuscript should either provide a concrete demonstration (e.g., a small toy corpus where de-indexing measurably changes retrieval-visible associations) or clearly label this as a speculative hypothesis rather than a technical consequence of de-indexing.","section":"Section 3, same paragraph"},{"comment":"The introduction of machine unlearning and RAG is useful but underdeveloped in relation to the paper's main argument. The text presents RAG as a way to 'simulate forgetting' by updating the external knowledge base, but does not reconcile this with the earlier claim about embedding-space disruption. If the authors intend RAG as a corrective to the conflated claim, that connection should be made explicit; otherwise the reader is left with two incompatible accounts of how forgetting operates.","section":"Section 3, final subsection"}],"minor_comments":[{"comment":"The phrase 'challenge of forgetfulness' is unusual; 'challenge of forgetting' would be more natural in this context.","section":"Abstract"},{"comment":"The loss function is written as 'L(θ) = − NX i=1 log Pθ(...)'; the 'NX' appears to be a rendering error and should be the summation notation '\\sum_{i=1}^{N}'.","section":"Section 2.5.1, Eq. (7)"},{"comment":"The attention formula is correct, but the denominator is written as '√dk' without a subscript definition for dk in the immediately preceding text; clarify that dk is the dimension of the key vectors.","section":"Section 2.5.1, Eq. (8)"},{"comment":"Several references (e.g., [8], [10], [14], [17], [29]) include 'Accessed: 2025-01-03' even though they are arXiv preprints or conference papers, not web pages; this field should be removed or restricted to actual URLs.","section":"References"},{"comment":"The inequality 'CosineSimilarity(vking, vqueen) > CosineSimilarity(vking, vcar)' is illustrative but the subscripts are not formatted consistently; use 'v_king', 'v_queen', 'v_car' for clarity.","section":"Section 2.4.1"}],"recommendation":"major_revision","confidential_remarks":"This is more of a tutorial/position paper than a research contribution, which may be appropriate for the target venue if it accepts such material. The main technical concern is the embedding-space claim in Section 3; it is a categorical error that, if not fixed, would mislead the intended non-expert readership. The paper would also benefit from a more cautious framing that explicitly separates de-indexing (which changes retrieval-visible associations) from model unlearning (which changes the learned representation). I recommend major revision rather than rejection because the error is localized and can be corrected within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a teaching paper, not a research contribution. The survey of IR models is accurate and well-paced, so if you teach or write for non-experts it has real utility. But the one place it makes an original-sounding claim—Section 3, where de-indexing is said to \"disrupt contextual relationships in the embedding space\" and \"weaken\" the Smith–X association—is technically off. Removing a URL from a search index changes which documents are candidates for retrieval; it does not move the remaining vectors or change their pairwise similarities. The authors seem to be describing a change in what users are exposed to, not a change in the embedding geometry. That isn't fatal for a tutorial, and I don't think it's deceptive—they explicitly say consequences are unpredictable—but as written the claim will mislead readers who know how embeddings work.\n\nWhat the paper does well: the explanations of Boolean retrieval, the vector space model, BM25, word and document embeddings, and the LLM training pipeline are accurate and pitched at the right level. The later discussion of machine unlearning and RAG is a useful bridge from legal notions of forgetting to technical mechanisms. The reference list is standard but fine, and the John Smith example does illustrate the privacy tension clearly.\n\nThe soft spots, in order: (1) the embedding-space disruption claim, as above; (2) no evidence or data is offered for that claim, though the paper frames it as a consequence of de-indexing rather than a metaphor; (3) the novelty is essentially nil—everything in the tutorial is available in textbooks and surveys. The paper is honest about being a review, so this is not a deception, just a low research ceiling.\n\nWho this is for: readers with no IR background who want to understand how RTBF interacts with search engines. I wouldn't bring it to a research reading group, but I would hand it to a new student or a collaborator outside CS.\n\nRecommendation: if the venue publishes tutorials or educational pieces, this deserves a regular peer review—but only after Section 3 is corrected to distinguish index-level visibility from model-level representations. Without that fix, the central example makes a claim the field will not accept.","headline":"A readable IR tutorial whose one original-sounding claim about de-indexing and embedding-space associations is technically muddled.","tokens_in":605,"tokens_out":971,"would_cite":false,"duration_ms":37311,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"De-indexing a page also rewires the semantic relationships in a search engine's embedding space, the paper argues.","keywords":["right to be forgotten","de-indexing","information retrieval","document embeddings","machine unlearning","retrieval-augmented generation","collective memory","search engines"],"falsifier":"Measure the cosine similarity between the embedding of a person's name and the embedding of a sensitive topic before and after de-indexing all documents that link the two; if the similarity does not change, the claimed disruption of the embedding space does not occur in the model itself, only in what users are shown.","tokens_in":10673,"feed_emoji":"🔍","tokens_out":3862,"duration_ms":39273,"temperature":0.7,"pith_summary":"The paper argues that when search engines de-index documents to honor the right to be forgotten, they do not simply hide a page: they also disrupt the contextual relationships that embeddings encode, weakening some associations and strengthening others. It surveys boolean, vector-space, probabilistic, and embedding-based information retrieval models to make this mechanism understandable to non-experts. The intended upshot is that de-indexing is a delicate intervention in a semantic landscape, with outcomes that are hard to predict and that may affect both the requester and collective memory. This matters because privacy decisions made one URL at a time could quietly change what everyone sees.","feed_headline":"De-indexing rewires the semantic map behind search","feed_subtitle":"Removing one URL can weaken old associations and strengthen new ones, complicating the right to be forgotten.","key_machinery":"The two coupled mechanisms are the inverted index, which lets search engines selectively remove specific URLs from results, and document embeddings, dense vector representations that place semantically similar texts near one another in a continuous space. De-indexing operates on the index, but the paper's claim is that its effects propagate into the embedding space, where the semantic neighborhood of a person's name is rearranged. That rearrangement is the mechanism that produces both the intended weakening of sensitive associations and the unintended strengthening of other associations.","core_discovery":"The central claim is that removing documents from a search index changes the geometry of the embedding space in which documents and queries are represented. De-indexing pages that link a person to a sensitive topic weakens that semantic association, while other associations, such as the person's link to an unrelated topic, may grow stronger. The paper illustrates this with a hypothetical individual, John Smith, whose court-case documents are de-indexed, and argues that the resulting shifts in the embedding space are impossible to predict at scale. Because the paper is a review and conceptual contribution, this claim is derived from the properties of embeddings rather than demonstrated with experiments.","pith_inferences":["The paper assumes that de-indexing changes the embedding coordinates themselves, but a more conservative reading is that it changes which documents users encounter; these two interpretations are testably different.","A direct empirical test would measure whether queries about a person surface new sensitive topics after de-indexing, and whether the shift is larger when nearby documents are also removed.","The same semantic-recalibration effect could apply to collective memory, meaning that the public record is distorted by de-indexing in ways that are harder to detect than a missing URL.","If the paper's reasoning is correct, the right to be forgotten cannot be reduced to a per-URL switch; compliance would need to include an assessment of semantic side effects."],"forward_implications":["Right-to-be-forgotten decisions should be evaluated not only by which results disappear, but by which new associations become more prominent afterward.","Search providers could audit de-indexing operations by measuring shifts in cosine similarity between person names and sensitive topics before and after removal.","Regulators and courts should treat de-indexing as a transformation of the information landscape, not as a binary deletion of content.","Retrieval-augmented generation offers a more controlled route to forgetting, because it allows knowledge to be edited in an external base rather than in the model's parameters.","Machine unlearning research suggests that complete forgetting in large language models is difficult, because models tend to retain knowledge even after unlearning attempts."],"supporting_citations":[{"why":"Provides the word2vec embedding method that grounds the semantic-space argument.","marker":"[20]"},{"why":"Extends embeddings to whole documents, giving the paper its notion of document-level semantic relationships.","marker":"[15]"},{"why":"Supplies the contextual embedding architecture that modern retrieval systems use, making the Smith–X scenario plausible.","marker":"[12]"},{"why":"Frames the impact on collective memory with an empirical model of attention decay.","marker":"[7]"},{"why":"Defines machine unlearning and surveys its approaches, the alternative the paper contrasts with de-indexing.","marker":"[17]"},{"why":"Cited as an approach that enables simulated forgetting in retrieval-augmented generation contexts.","marker":"[14]"},{"why":"Discusses combining machine unlearning with retrieval-augmented generation, supporting the paper's RAG-based forgetting proposal.","marker":"[29]"},{"why":"Shows that large language models retain knowledge after unlearning attempts, motivating the difficulty of complete forgetting.","marker":"[8]"}],"fun_headline_variants":["De-indexing rewires search's semantic map","Right to be forgotten shrinks search memory","When pages vanish, embeddings shift","Search forgetting alters information geometry","De-indexing makes search associations unpredictable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that removing a page from a search index changes the semantic relationships among concepts in the embedding space, even though embedding coordinates are normally fixed once a model is trained.","fun_headline_variants_meta":{"raw":{"variants":["De-indexing rewires search's semantic map","Right to be forgotten shrinks search memory","When pages vanish, embeddings shift","Search forgetting alters information geometry","De-indexing makes search associations unpredictable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1144,"prompt_tokens":807,"completion_tokens":337,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":423,"tokens_out":337,"duration_ms":3535,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:41:21.202910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the cosine similarity between the embedding of a person's name and the embedding of a sensitive topic before and after de-indexing all documents that link the two; if the similarity does not change, the claimed disruption of the embedding space does not occur in the model itself, only in what users are shown.","supporting_citations":[{"cited_title":"Distributed representations of sentences and docu- ments","cited_arxiv_id":null,"evidence_quote":"Extends embeddings to whole documents, giving the paper its notion of document-level semantic relationships."},{"cited_title":"The universal decay of collective memory and attention.Nature human behaviour, 3(1):82–91, 2019","cited_arxiv_id":null,"evidence_quote":"Frames the impact on collective memory with an empirical model of attention decay."},{"cited_title":"Unlearning in ai and machine learning, 2024","cited_arxiv_id":null,"evidence_quote":"Defines machine unlearning and surveys its approaches, the alternative the paper contrasts with de-indexing."},{"cited_title":"Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection","cited_arxiv_id":null,"evidence_quote":"Cited as an approach that enables simulated forgetting in retrieval-augmented generation contexts."},{"cited_title":"Deep forgetting & unlearning for safely-scoped llms, 2023","cited_arxiv_id":null,"evidence_quote":"Shows that large language models retain knowledge after unlearning attempts, motivating the difficulty of complete forgetting."}],"review_version":1}