Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

(De)-Indexing and the Right to be Forgotten

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read De-indexing a page also rewires the semantic relationships in a search engine's embedding space, the paper argues.

desk verdict A readable IR tutorial whose one original-sounding claim about de-indexing and embedding-space associations is technically muddled. read the letter →

arxiv 2501.03989 v1 pith:QIP2GLLK submitted 2025-01-07 cs.CY cs.IR

Salvatore Vilella , Giancarlo Ruffo This is my paper · ORCID
classification cs.CYcs.IR
keywords righttobeforgottende-indexinginformationretrievaldocumentembeddingsmachineunlearningretrieval-augmentedgenerationcollectivememorysearchengines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when search engines de-index documents to honor the right to be forgotten, they do not simply hide a page: they also disrupt the contextual relationships that embeddings encode, weakening some associations and strengthening others. It surveys boolean, vector-space, probabilistic, and embedding-based information retrieval models to make this mechanism understandable to non-experts. The intended upshot is that de-indexing is a delicate intervention in a semantic landscape, with outcomes that are hard to predict and that may affect both the requester and collective memory. This matters because privacy decisions made one URL at a time could quietly change what everyone sees.

What carries the argument

The two coupled mechanisms are the inverted index, which lets search engines selectively remove specific URLs from results, and document embeddings, dense vector representations that place semantically similar texts near one another in a continuous space. De-indexing operates on the index, but the paper's claim is that its effects propagate into the embedding space, where the semantic neighborhood of a person's name is rearranged. That rearrangement is the mechanism that produces both the intended weakening of sensitive associations and the unintended strengthening of other associations.

What would settle it

Measure the cosine similarity between the embedding of a person's name and the embedding of a sensitive topic before and after de-indexing all documents that link the two; if the similarity does not change, the claimed disruption of the embedding space does not occur in the model itself, only in what users are shown.

Watch

Extended reading notes

Core claim

The central claim is that removing documents from a search index changes the geometry of the embedding space in which documents and queries are represented. De-indexing pages that link a person to a sensitive topic weakens that semantic association, while other associations, such as the person's link to an unrelated topic, may grow stronger. The paper illustrates this with a hypothetical individual, John Smith, whose court-case documents are de-indexed, and argues that the resulting shifts in the embedding space are impossible to predict at scale. Because the paper is a review and conceptual contribution, this claim is derived from the properties of embeddings rather than demonstrated with experiments.

Load-bearing premise

The load-bearing premise is that removing a page from a search index changes the semantic relationships among concepts in the embedding space, even though embedding coordinates are normally fixed once a model is trained.

Editorial extensions

If this is right

  • Right-to-be-forgotten decisions should be evaluated not only by which results disappear, but by which new associations become more prominent afterward.
  • Search providers could audit de-indexing operations by measuring shifts in cosine similarity between person names and sensitive topics before and after removal.
  • Regulators and courts should treat de-indexing as a transformation of the information landscape, not as a binary deletion of content.
  • Retrieval-augmented generation offers a more controlled route to forgetting, because it allows knowledge to be edited in an external base rather than in the model's parameters.
  • Machine unlearning research suggests that complete forgetting in large language models is difficult, because models tend to retain knowledge even after unlearning attempts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper assumes that de-indexing changes the embedding coordinates themselves, but a more conservative reading is that it changes which documents users encounter; these two interpretations are testably different.
  • A direct empirical test would measure whether queries about a person surface new sensitive topics after de-indexing, and whether the shift is larger when nearby documents are also removed.
  • The same semantic-recalibration effect could apply to collective memory, meaning that the public record is distorted by de-indexing in ways that are harder to detect than a missing URL.
  • If the paper's reasoning is correct, the right to be forgotten cannot be reduced to a per-URL switch; compliance would need to include an assessment of semantic side effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper is a tutorial-style preprint aimed at non-experts, introducing the foundations of information retrieval (boolean, vector space, probabilistic, embedding-based models, and LLMs) and connecting them to the legal and technical practice of de-indexing in the context of the right to be forgotten (RTBF). The first half of the paper is a concise review of classic IR models and modern neural retrieval, with formal definitions of the term-document matrix, inverted index, cosine similarity, BM25, word/document embeddings, and transformer attention. The second half argues that de-indexing, while leaving the original web content in place, reduces its visibility and 'disrupts the contextual relationships in the embedding space,' using a hypothetical 'Mr. John Smith' example to suggest that removing documents linking a person to a sensitive topic weakens that association and may strengthen others. The paper concludes with a discussion of machine unlearning, retrieval-augmented generation (RAG), and the broader societal implications for collective memory and privacy.

Significance. If its central mechanism were accurate, the paper would highlight a subtle and practically important consequence of RTBF: that de-indexing does not merely hide a URL but reshapes the semantic associations a user sees. The tutorial sections are largely sound and well-suited to a non-specialist audience; the derivations of TDM, inverted index, VSM, BM25, and attention are correct and clearly presented. The paper also usefully introduces machine unlearning and RAG as complementary or alternative mechanisms for forgetting. However, the paper's distinctive claim is currently stated in a way that conflates the retrieval index with the embedding model, and it is not supported by data or a formal argument. Once that claim is corrected or explicitly reframed as speculation about retrieval-visible associations, the paper could serve as a readable primer for legal and policy audiences, but in its current form the central technical assertion is not reliable.

major comments (3)
  1. [Section 3, paragraph 4] The claim that de-indexing 'disrupts their contextual relationships in the embedding space' is technically inaccurate as stated. Removing a URL from a search engine's index changes the candidate set of documents that can be retrieved for a query, but it does not alter the learned embedding vectors of the remaining documents or their pairwise cosine similarities. The subsequent example about 'Smith' and 'X' weakening and 'Smith' and 'Y' growing stronger is therefore at most a statement about which associations appear in ranked results, not about geometry of the embedding space. Because this mechanism is used to argue for 'profound implications' and 'machine forgetting,' the paper should either restrict the claim to retrieval-visible associations or explicitly move the discussion to a machine unlearning setting where model weights are actually updated.
  2. [Section 3, same paragraph] The paper itself concedes that 'it is not easy to predict the consequences that such an operation can have on the embedding space at a larger scale,' which is an acknowledgment that no evidence or formal argument is provided for the central illustrative claim. For a claim that is load-bearing for the paper's privacy implications, the manuscript should either provide a concrete demonstration (e.g., a small toy corpus where de-indexing measurably changes retrieval-visible associations) or clearly label this as a speculative hypothesis rather than a technical consequence of de-indexing.
  3. [Section 3, final subsection] The introduction of machine unlearning and RAG is useful but underdeveloped in relation to the paper's main argument. The text presents RAG as a way to 'simulate forgetting' by updating the external knowledge base, but does not reconcile this with the earlier claim about embedding-space disruption. If the authors intend RAG as a corrective to the conflated claim, that connection should be made explicit; otherwise the reader is left with two incompatible accounts of how forgetting operates.
minor comments (5)
  1. [Abstract] The phrase 'challenge of forgetfulness' is unusual; 'challenge of forgetting' would be more natural in this context.
  2. [Section 2.5.1, Eq. (7)] The loss function is written as 'L(θ) = − NX i=1 log Pθ(...)'; the 'NX' appears to be a rendering error and should be the summation notation '\sum_{i=1}^{N}'.
  3. [Section 2.5.1, Eq. (8)] The attention formula is correct, but the denominator is written as '√dk' without a subscript definition for dk in the immediately preceding text; clarify that dk is the dimension of the key vectors.
  4. [References] Several references (e.g., [8], [10], [14], [17], [29]) include 'Accessed: 2025-01-03' even though they are arXiv preprints or conference papers, not web pages; this field should be removed or restricted to actual URLs.
  5. [Section 2.4.1] The inequality 'CosineSimilarity(vking, vqueen) > CosineSimilarity(vking, vcar)' is illustrative but the subscripts are not formatted consistently; use 'v_king', 'v_queen', 'v_car' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is a self-contained tutorial that makes no fitted predictions and does not rely on a load-bearing self-citation chain.

full rationale

This paper is an expository overview of information retrieval models, embeddings, LLMs, and the relation of de-indexing to the right to be forgotten. It derives no new equations, fits no parameters, and makes no empirical prediction that could reduce to its inputs by construction. The closest thing to a central claim is the Section 3 observation that de-indexing documents 'disrupts their contextual relationships in the embedding space' and that associations such as Smith-X may weaken while Smith-Y may strengthen. That statement is presented as a conceptual illustration, not as a result derived from the paper's own definitions or from the cited models; it is therefore not a circular derivation. The paper's references include standard textbooks and prior work by other authors, and no load-bearing argument is justified solely by a self-citation. A possible technical concern about whether de-indexing changes embedding coordinates as opposed to retrieval-visible associations is a correctness or precision issue, not a circularity issue, and the paper explicitly frames these dynamics as implications rather than proven outcomes. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on an assumed connection between de-indexing and embedding-space relationships, which is not proven. No free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption Removing documents from a search index disrupts their contextual relationships in the embedding space.
    Section 3 states this without evidence; actual embedding models are static, so de-indexing may only change retrieval results, not learned representations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of (De)-Indexing and the Right to be Forgotten." pith.science (2026). https://pith.science/paper/QIP2GLLK

@misc{pith2026250103989,
  author       = {Pith},
  title        = {Pith review of: (De)-Indexing and the Right to be Forgotten},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QIP2GLLK}},
  note         = {Machine review of arXiv:2501.03989}
}
read the original abstract

In the digital age, the challenge of forgetfulness has emerged as a significant concern, particularly regarding the management of personal data and its accessibility online. The right to be forgotten (RTBF) allows individuals to request the removal of outdated or harmful information from public access, yet implementing this right poses substantial technical difficulties for search engines. This paper aims to introduce non-experts to the foundational concepts of information retrieval (IR) and de-indexing, which are critical for understanding how search engines can effectively "forget" certain content. We will explore various IR models, including boolean, probabilistic, vector space, and embedding-based approaches, as well as the role of Large Language Models (LLMs) in enhancing data processing capabilities. By providing this overview, we seek to highlight the complexities involved in balancing individual privacy rights with the operational challenges faced by search engines in managing information visibility.

Figures

Figures reproduced from arXiv: 2501.03989 by the authors.

Figure 1
Figure 1. Left: interest in time of the queries machine learning and llm on Google. We can see how LLMs gain momentum and approach the level of the machine learning query over time. Right: the evolutionary tree of LLMs [31]. LLMs, just as any other supervised deep learning model, undergo an initial train￾ing phase, and a second phase of fine tuning. As Google puts it in its Introduction to LLMs [11], these phases can be intui… view at source ↗
Figure 2
Figure 2. An intuitive comparison between the steps required to train a special-service [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  2. SoK: Machine Unlearning for Large Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  2. [2]

    What is retrieval-augmented generation (rag)?, 2024

    Amazon Web Services (A WS). What is retrieval-augmented generation (rag)?, 2024. Accessed: 2024-12-12

    work page 2024

  3. [3]

    Modern Information Retrieval: The Concepts and Technology Behind Search

    Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval: The Concepts and Technology Behind Search. Addison-Wesley, Reading, MA, 1999

    work page 1999

  4. [4]

    Enriching word vectors with subword information

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the association for compu- tational linguistics, 5:135–146, 2017

    2017

  5. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  6. [6]

    Information Retrieval: Im- plementing and Evaluating Search Engines

    Stefan Buttcher, Charles Clarke, and Gordon Cormack. Information Retrieval: Im- plementing and Evaluating Search Engines. MIT Press, Cambridge, MA, 2nd edition, 2016

    work page 2016

  7. [7]

    The universal decay of collective memory and attention.Nature human behaviour, 3(1):82–91, 2019

    Cristian Candia, C Jara-Figueroa, Carlos Rodriguez-Sickert, Albert-L´ aszl´ o Barab´ asi, and C´ esar A Hidalgo. The universal decay of collective memory and attention.Nature human behaviour, 3(1):82–91, 2019. 15

    work page 2019

  8. [8]

    Deep forgetting & unlearning for safely-scoped llms, 2023

    Stephen Casper. Deep forgetting & unlearning for safely-scoped llms, 2023. Accessed: 2025-01-03

    work page 2023

Show all 32 references
  1. [9]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  2. [10]

    Continual pre-training mitigates forgetting in lan- guage and vision

    Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting in lan- guage and vision. arXiv preprint arXiv:2205.09357, 2022. Accessed: 2025-01-03

    2022 arXiv

  3. [11]

    Introduction to large language models, 2024

    Google Developers. Introduction to large language models, 2024. Accessed: 2024- 12-10

    2024

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    Beyond [cls] through ranking by generation

    Cicero dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. Beyond [cls] through ranking by generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1722–1727, 2020

    2020

  6. [14]

    Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection

    Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. Accessed: 2025-01-03

    2024

  7. [15]

    Distributed representations of sentences and docu- ments

    Quoc Le and Tomas Mikolov. Distributed representations of sentences and docu- ments. In International conference on machine learning, pages 1188–1196. PMLR, 2014

    2014

  8. [16]

    Pretrained transformers for text ranking: BERT and beyond

    Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: BERT and beyond. arXiv preprint, arXiv:2010.06467, 2020

    2010 arXiv

  9. [17]

    Unlearning in ai and machine learning, 2024

    Ken Ziyu Liu. Unlearning in ai and machine learning, 2024. Accessed: 2025-01-03

    2024

  10. [18]

    Rethinking machine unlearning for large language models

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024

    2024 arXiv

  11. [19]

    Manning, Prabhakar Raghavan, and Hinrich Sch¨ utze

    Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch¨ utze. Introduction to Information Retrieval. MIT Press, Cambridge, MA, 1st edition, 2008

    2008

  12. [20]

    Efficient estimation of word representations in vector space

    Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 3781, 2013

    2013 arXiv

  13. [21]

    Passage re-ranking with BERT

    Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv preprint, arXiv:1901.04085, 2019

    1901 arXiv

  14. [22]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. 16

    2014

  15. [23]

    RAG-Fusion: a New Take on Retrieval-Augmented Generation

    Zackary Rackauckas. RAG-Fusion: a New Take on Retrieval-Augmented Generation. arXiv preprint arXiv:2402.03367, 2024. Accessed: 2025-01-03

    2024 arXiv

  16. [24]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  17. [25]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  18. [26]

    Exploring the limits of transfer learn- ing with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  19. [27]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333– 389, 2009

    2009

  20. [28]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  21. [29]

    When machine unlearning meets retrieval-augmented generation (rag): Keep secret or forget knowledge? arXiv preprint arXiv:2410.15267, 2024

    Shang Wang, Tianqing Zhu, Dayong Ye, and Wanlei Zhou. When machine unlearning meets retrieval-augmented generation (rag): Keep secret or forget knowledge? arXiv preprint arXiv:2410.15267, 2024. Accessed: 2025-01-03

    2024 arXiv

  22. [30]

    Witten, Alistair Moffat, and Timothy C

    Ian H. Witten, Alistair Moffat, and Timothy C. Bell. Managing Gigabytes: Com- pressing and Indexing Documents and Images. Morgan Kaufmann, San Francisco, CA, 1999

    1999

  23. [31]

    Harnessing the power of llms in practice: A survey on chatgpt and beyond

    Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond. 2023

    2023

  24. [32]

    Deep query likelihood model for in- formation retrieval

    Shengyao Zhuang, Hang Li, and Guido Zuccon. Deep query likelihood model for in- formation retrieval. In Proceedings of the 43rd European Conference on Information Retrieval (ECIR), 2021. 17

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.