The reviewed record of science sign in
Pith

arxiv: 2606.03001 · v2 · pith:L4FF2GRS · submitted 2026-06-02 · cs.DC

FOLD: Fuzzy Online Deduplication for Very Large Evolving Datasets via Approximate Nearest Neighbor Search

Nelson Bore , Pritish Mishra , Constantin Adam , Eyal de Lara , Oana Balmau This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 08:45 UTCgrok-4.3pith:L4FF2GRSrecord.jsonopen to challenge →

classification cs.DC
keywords fuzzy deduplicationonline deduplicationHNSW indexapproximate nearest neighbor searchLLM training corporaJaccard similaritybitmap representationevolving datasets
0
0 comments X

The pith

RAD uses an HNSW index with bitmap representation to deliver high-recall fuzzy deduplication on growing datasets at up to 8x throughput.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents RAD as an online fuzzy deduplication system for very large, continuously growing corpora such as those used to train large language models. Classic LSH approaches require each new batch to be checked against expanding buckets, causing throughput to degrade as the admitted set increases. RAD maintains an incrementally built HNSW graph over admitted documents and retrieves small candidate neighborhoods for incoming items. The key adaptation is a bitmap encoding that supplies a more discriminative signal aligned with Jaccard similarity, avoiding the distance crowding that otherwise hurts recall in standard HNSW traversal. On four LLM-scale collections the system keeps 0.94-0.97 recall relative to strong LSH baselines at 30 million documents while raising throughput by as much as 8 times and preserving stable scaling.

Core claim

RAD is the first online fuzzy deduplication system to employ an HNSW index rather than repeated LSH bucket scans. Because direct Jaccard similarity produces unreliable distances inside the graph, RAD substitutes a bitmap representation that yields a more separable, Jaccard-aligned signal. This change lets bounded graph traversal recover high-recall candidate sets, resulting in 0.94-0.97 recall versus state-of-the-art LSH at 30 million documents together with up to an 8 times throughput gain on LM1B, C4, RealNews, and Common Crawl while throughput remains stable as the corpus grows.

What carries the argument

The bitmap representation that supplies a more discriminative, Jaccard-aligned signal during HNSW search.

If this is right

  • Throughput stays constant rather than declining as the admitted corpus passes 30 million documents.
  • Each incoming document is checked against a small neighborhood instead of rescanning the entire corpus.
  • Continuous ingestion of new web-scale data becomes practical without periodic full re-deduplication.
  • The same index supports both deduplication and later nearest-neighbor retrieval over the cleaned collection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The bitmap trick may transfer to other graph indexes when the native distance metric crowds scores.
  • Combining RAD with periodic offline LSH passes could further raise recall at modest extra cost.
  • The stable scaling property would let the same pipeline handle streaming updates from multiple sources simultaneously.

Load-bearing premise

The bitmap representation supplies a sufficiently discriminative, Jaccard-aligned signal inside the HNSW graph so that bounded-step traversal still recovers high-recall candidate sets.

What would settle it

A measurement at 100 million documents showing recall below 0.9 or throughput that no longer increases with added hardware would falsify the claim of stable high-recall online deduplication.

Figures

Figures reproduced from arXiv: 2606.03001 by Constantin Adam, Eyal de Lara, Nelson Bore, Oana Balmau, Pritish Mishra.

Figure 1
Figure 1. Figure 1: Steps involved in state-of-the-art Fuzzy Dedu [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Throughput (left) and recall (right) for Com￾mon Crawl, as the corpus grows to 5M documents. None of the baselines manage to maintain both high re￾call and high throughput as the dataset grows. To validate IBM DPK as a good-enough ground truth, we perform a brute-force search over 3M-document sub￾sets of our evaluated datasets [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FOLD workflow. For each incoming document batch, the documents’ bitmap signatures are generated (➊). Then, FOLD removes near-duplicates inside the batch(➋). Next, for each input document, the closest neighbors are retrieved from the corpus indexed via an HNSW graph (➌) and the duplicates are filtered out using a fixed threshold (➍). Finally, the remaining documents in the input batch are inserted into the … view at source ↗
Figure 4
Figure 4. Figure 4: Example HNSW level-0 navigation with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical confirmation on a 1,500-document CC-MAIN sample with 𝐽 ≥ 0.7 ground-truth clusters. (a) During HNSW construction, FOLD-Jaccard selects ≈ 95% of each node’s true top-𝑀 neighbors, compared with ≈ 51% for MinHash–Jaccard. (b) At query time, FOLD-Jaccard reaches recall ≥ 0.95 within 3 opened nodes, while MinHash–Jaccard needs 5 and reaches only 0.70 by depth 5. cause many low-overlap local candidates… view at source ↗
Figure 7
Figure 7. Figure 7: FOLD preserves the high-throughput, high-recall operating point as the corpus grows. At 30M documents, FOLD maintains 0.94–0.97 recall and 220–551 docs/sec across the four workloads. DPK and Milvus lose throughput as the corpus grows, while FAISS (Jaccard) keeps bounded graph-search throughput but its recall remains lower and dataset-dependent [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Common Crawl breakdown. Top: document outcomes per 100K-document streaming slice. Bottom: latency by stage. FOLD keeps search time stable and turns duplicate drops into lower insertion time; Milvus and DPK lose those savings to growing search cost, while FAISS (Jaccard) stays low-latency but low-recall. All methods are shown to 4M-20M documents. remain nearly flat, moving from 0.81→0.80 min and 0.72→0.73 m… view at source ↗
Figure 10
Figure 10. Figure 10: shows that FOLD remains stable across the full 1M–50M range. In this extended run, throughput starts at 467 docs/sec at 1M, peaks at 648 docs/sec at 10M, and then stays in a narrow 544–599 docs/sec band from 11M to 50M, ending at 574 docs/sec. The key result is the steady state: FOLD does not show a late-scale through￾put collapse under continuous insertion. Its candidate retrieval, bitmap–Jaccard scoring… view at source ↗
read the original abstract

Fuzzy deduplication is key to constructing large language model training corpora. However, classic Locality-Sensitive Hashing (LSH) pipelines scale poorly as corpora grow and are ill-suited to continuous ingestion. The main issue is that each new document batch must be checked against the admitted corpus before insertion. As the corpus grows, the LSH buckets grow: each query can hit several large buckets and must scan the returned candidates. To solve this problem, we present RAD (Retrieval-Augmented Deduplication), an online fuzzy deduplication system that delivers both high recall and throughput for evolving datasets. RAD maintains an incrementally updated HNSW index over admitted documents, retrieving a small, high-quality candidate neighborhood for each incoming document instead of repeatedly re-scanning the accumulated corpus. RAD is the first online fuzzy deduplication system to use HNSW, leading to stable throughput as datasets grow. However, it is not easy to maintain high recall when using HNSW-style indexes. The core issue is the distance metric between graph nodes. Jaccard similarity, the metric used for fuzzy deduplication, yields low recall when applied out-of-the-box with an HNSW index. It leads to distance score crowding, making graph traversal unreliable within a bounded number of steps. RAD addresses this with a bitmap representation that provides a more discriminative, Jaccard-aligned signal during HNSW search. Across four LLM-scale datasets (LM1B, C4, RealNews, and Common Crawl), RAD preserves the scaling trajectory needed for online fuzzy deduplication: at 30M documents, it maintains 0.94-0.97 recall relative to state-of-the-art LSH solutions, and delivers up to an 8x throughput increase.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper presents RAD (Retrieval-Augmented Deduplication), an online fuzzy deduplication system for very large evolving datasets. It maintains an incrementally updated HNSW index over admitted documents and introduces a bitmap representation to provide a more discriminative Jaccard-aligned signal during search, addressing distance score crowding that arises when applying Jaccard similarity directly in HNSW. The central claims are that RAD is the first online fuzzy deduplication system to use HNSW, achieves 0.94-0.97 recall relative to state-of-the-art LSH solutions at 30M documents across four LLM-scale datasets (LM1B, C4, RealNews, Common Crawl), and delivers up to 8x throughput increase while preserving stable scaling.

Significance. If the performance numbers and the bitmap-HNSW alignment hold under rigorous measurement, the work would be significant for LLM corpus construction pipelines that require continuous ingestion without periodic full rescans. It would demonstrate that approximate nearest-neighbor graphs can replace bucket-scanning LSH for fuzzy deduplication while maintaining high recall and improving throughput scaling.

major comments (3)
  1. [Abstract] Abstract: the headline recall range 0.94-0.97 is stated without any description of how recall was computed against the LSH baseline (exact matching criterion, candidate ranking, ground-truth construction), without error bars, and without the number of runs or statistical tests. These omissions make the quantitative comparison impossible to evaluate.
  2. [Abstract] Abstract: the claim that the bitmap supplies a 'more discriminative, Jaccard-aligned signal' that resolves distance score crowding and enables bounded-step HNSW traversal to recover high-recall candidates is load-bearing for all reported recall numbers, yet the manuscript supplies no construction details, no parameter settings for the bitmap, and no ablation showing that the mapping preserves the original similarity ordering sufficiently for HNSW's approximate search.
  3. [Abstract] Abstract: HNSW parameters (M, efConstruction, efSearch) and any incremental-maintenance parameters are not reported, so it is impossible to determine whether the stated 8x throughput gain is attributable to the index structure, the bitmap, or particular tuning choices.
minor comments (1)
  1. Title refers to FOLD while the body describes RAD; the relationship between the two names should be clarified.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below and will revise the manuscript to improve clarity, reproducibility, and completeness of the reported results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline recall range 0.94-0.97 is stated without any description of how recall was computed against the LSH baseline (exact matching criterion, candidate ranking, ground-truth construction), without error bars, and without the number of runs or statistical tests. These omissions make the quantitative comparison impossible to evaluate.

    Authors: We agree that the abstract requires additional context on the recall metric to allow proper evaluation. In the revision we will expand the abstract with a concise description of the recall computation (exact Jaccard-threshold matching to the LSH baseline, candidate ranking procedure, and ground-truth construction). Full experimental details, including the number of runs, error bars, and any statistical tests, will be added to Section 4 (Experimental Setup) and the results tables. revision: yes

  2. Referee: [Abstract] Abstract: the claim that the bitmap supplies a 'more discriminative, Jaccard-aligned signal' that resolves distance score crowding and enables bounded-step HNSW traversal to recover high-recall candidates is load-bearing for all reported recall numbers, yet the manuscript supplies no construction details, no parameter settings for the bitmap, and no ablation showing that the mapping preserves the original similarity ordering sufficiently for HNSW's approximate search.

    Authors: The full manuscript describes bitmap construction and parameter settings in Section 3.2. We acknowledge, however, that these details are not summarized in the abstract and that an explicit ablation on ordering preservation is not highlighted. We will add a brief description of the bitmap representation and its Jaccard alignment to the abstract, and we will ensure the ablation study (already present in the experiments) is clearly referenced and expanded in the revised text. revision: yes

  3. Referee: [Abstract] Abstract: HNSW parameters (M, efConstruction, efSearch) and any incremental-maintenance parameters are not reported, so it is impossible to determine whether the stated 8x throughput gain is attributable to the index structure, the bitmap, or particular tuning choices.

    Authors: We agree that the abstract should report the key HNSW parameters. The values of M, efConstruction, and efSearch, together with incremental-maintenance settings, are given in Section 4. We will include the principal parameter values in the abstract (or a footnote) and will add a short discussion clarifying the contribution of the index structure versus the bitmap to the observed throughput scaling. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical system evaluation is self-contained

full rationale

The paper presents RAD as an engineering system that replaces LSH bucket scanning with an incrementally maintained HNSW index plus a bitmap representation to mitigate Jaccard distance crowding. No equations, fitted parameters, or derived quantities appear. The headline metrics (0.94-0.97 recall vs. LSH at 30 M documents, up to 8x throughput) are reported as direct experimental outcomes on LM1B, C4, RealNews, and Common Crawl; they are not obtained by fitting any model component on the evaluation data and then re-using that fit as a prediction. The bitmap choice is motivated by an observed failure mode of raw Jaccard in HNSW, but this is an ansatz justified by the problem statement rather than a self-referential definition or self-citation chain. No load-bearing step reduces to its own inputs by construction, satisfying the default expectation of no circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The contribution is an engineering system rather than a mathematical derivation; the abstract introduces no free parameters, axioms, or new postulated entities.

pith-pipeline@v0.9.1-grok · 5862 in / 1226 out tokens · 25935 ms · 2026-06-28T08:45:09.850049+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    URLhttps:// commoncrawl.org/

    Common Crawl Corpus, 2013–. URLhttps:// commoncrawl.org/

    2013

  2. [2]

    URLhttps://towardsdatascience.com/ understanding-the-hypergeometric- distribution-e6540c7fec3c/

    The hypergeometric distribution, 2022. URLhttps://towardsdatascience.com/ understanding-the-hypergeometric- distribution-e6540c7fec3c/

    2022

  3. [3]

    URLhttps://github.com/modelscope/ data-juicer/blob/main/data_juicer/ ops/deduplicator/document_minhash_ deduplicator.py

    Data Juicer MinHash Deduplication, 2023. URLhttps://github.com/modelscope/ data-juicer/blob/main/data_juicer/ ops/deduplicator/document_minhash_ deduplicator.py

    2023

  4. [4]

    URLhttps://github.com/huggingface/ datatrove/blob/main/examples/minhash_ deduplication.py

    DataTrove MinHash Deduplication, 2023. URLhttps://github.com/huggingface/ datatrove/blob/main/examples/minhash_ deduplication.py

    2023

  5. [5]

    URL https://github.com/togethercomputer/ RedPajama-Data/blob/main/app/src/run_ lsh.py

    RedPajama MinHash Deduplication, 2023. URL https://github.com/togethercomputer/ RedPajama-Data/blob/main/app/src/run_ lsh.py

    2023

  6. [6]

    URL http://commoncrawl.org/

    Common Crawl: 2024-30 Snapshot, 2024. URL http://commoncrawl.org/

    2024

  7. [7]

    URLhttps://github.com/ facebookresearch/faiss

    FAISS, 2025. URLhttps://github.com/ facebookresearch/faiss

    2025

  8. [8]

    Hierarchical Navigable Small Worlds (HNSW),

  9. [9]

    URLhttp://pinecone.io/learn/ series/faiss/hnsw/

  10. [10]

    URL https://milvus.io/blog/minhash-lsh-in- milvus-the-secret-weapon-for-fighting- duplicates-in-llm-training-data.md

    MinHash LSH in Milvus, 2025. URL https://milvus.io/blog/minhash-lsh-in- milvus-the-secret-weapon-for-fighting- duplicates-in-llm-training-data.md

    2025

  11. [11]

    URLhttps:// milvus.io/docs/index-explained.md/

    Index Explained, 2025. URLhttps:// milvus.io/docs/index-explained.md/

    2025

  12. [12]

    URLhttps: //www.modelscope.cn/home

    ModelScope, 2025. URLhttps: //www.modelscope.cn/home

    2025

  13. [13]

    URL https://docs.nvidia.com/nemo-framework/ user-guide/latest/datacuration/ gpudeduplication.html

    NVIDIA Datacuration, 2025. URL https://docs.nvidia.com/nemo-framework/ user-guide/latest/datacuration/ gpudeduplication.html

    2025

  14. [14]

    Amro Abbas, Kushal Tirumala, D ´aniel Simig, Surya Ganguli, and Ari S. Morcos. SemDeDup: Data-efficient learning at web-scale through seman- tic deduplication. arXiv preprint arXiv:2303.09540, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Costa-juss `a

    Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-juss `a. Continual lifelong learn- ing in natural language processing: A survey. In The International Conference on Computational Linguistics, 2020

    2020

  16. [16]

    Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7), 1970

    1970

  17. [17]

    A. Z. Broder. On the resemblance and containment of documents. In The Compression and Complexity of SEQUENCES, 1997

    1997

  18. [18]

    Andrei Z. Broder. On the resemblance and con- tainment of documents. In Proceedings of the Compression and Complexity of SEQUENCES, 1997

    1997

  19. [19]

    Andrei Z. Broder. Identifying and filtering near- duplicate documents. In The Annual Symposium on Combinatorial Pattern Matching, 2000

    2000

  20. [20]

    Broder, Steven C

    Andrei Z. Broder, Steven C. Glassman, Mark S. Manasse, and Geoffrey Zweig. Syntactic clustering of the web. InProceedings of the Sixth International World Wide Web Conference, 1997

    1997

  21. [21]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagiel- ski, Katherine Lee, Florian Tram `er, and Chiyuan Zhang. Quantifying memorization across neural language models. In The International Conference on Learning Representations (ICLR), 2023

    2023

  22. [22]

    Charikar

    Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In The Annual ACM Symposium on Theory of Computing, 2002. ISBN 1581134959

    2002

  23. [23]

    One billion word benchmark for measur- ing progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measur- ing progress in statistical language modeling. 2013

    2013

  24. [24]

    Hashing tech- niques: A survey and taxonomy

    Lianhua Chi and Xingquan Zhu. Hashing tech- niques: A survey and taxonomy. ACM Computing Surveys (CSUR), 50(1), 2017

    2017

  25. [25]

    Documenting the english colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. Documenting the english colossal clean crawled corpus. 2021

    2021

  26. [26]

    Mate: Multi-attribute table extraction

    Mahdi Esmailoghli, Jorge-Arnulfo Quian ´e-Ruiz, and Ziawasch Abedjan. Mate: Multi-attribute table extraction. Proceedings of the VLDB Endowment, 15(8), 2022. 15

    2022

  27. [27]

    Qahtan, Ahmed Elmagarmid, Ihab F

    Raul Castro Fernandez, Essam Mansour, Abdul- hakim A. Qahtan, Ahmed Elmagarmid, Ihab F. Ilyas, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. Lazo: A cardinality- based method for coupled estimation of jaccard similarity and containment. In 2019 IEEE 35th International Conference on Data Engineering, 2019

    2019

  28. [28]

    Unleashing graph par- titioning for large-scale nearest neighbor search

    Lars Gottesb¨ uren, Laxman Dhulipala, Rajesh Ja- yaram, and Jakub Lacki. Unleashing graph par- titioning for large-scale nearest neighbor search. 2024

    2024

  29. [29]

    Wiki-40b: Multilingual language model dataset

    Mandy Guo, Zihang Dai, Denny Vrande ˇci`c, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In The Language Resources and Evaluation Conference, 2020

    2020

  30. [30]

    Scaling laws and interpretability of learning from repeated data

    Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tris- tan Hume, et al. Scaling laws and interpretability of learning from repeated data. 2022

    2022

  31. [31]

    Christopher J. Hughes. Single-Instruction Multiple-Data Execution. Synthesis Lectures on Computer Architecture. 2015. ISBN 978-3-031- 00618-0

    2015

  32. [32]

    Approximate nearest neighbors: Towards removing the curse of dimensionality

    Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In The Annual ACM Symposium on Theory of Computing (STOC), 1998

    1998

  33. [33]

    Hanson, Weiyun Huang, Michal Nowakiewicz, and Vassilis Papadimos

    Per- ˚Ake Larson, Adrian Birka, Eric N. Hanson, Weiyun Huang, Michal Nowakiewicz, and Vassilis Papadimos. Real-time analytical processing with SQL server. The VLDB Journal, 8(12), 2015

    2015

  34. [34]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nys- trom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In The Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

    2022

  35. [35]

    Tic-lm: A web-scale benchmark for time- continual LLM pretraining

    Jeffrey Li, Mohammadreza Armandpour, Iman Mirzadeh, Sachin Mehta, Vaishaal Shankar, Raviteja Vemulapalli, Samy Bengio, Oncel Tuzel, Mehrdad Farajtabar, Hadi Pouransari, and Fartash Faghri. Tic-lm: A web-scale benchmark for time- continual LLM pretraining. In The Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

    2025

  36. [36]

    Sparse indexing: Large scale, in- line deduplication using sampling and locality

    Mark Lillibridge, Kave Eshghi, Deepavali Bhag- wat, Vinay Deolalikar, Greg Trezise, and Pe- ter Camble. Sparse indexing: Large scale, in- line deduplication using sampling and locality. In 7th USENIX Conference on File and Storage Technologies (FAST 09), 2009

    2009

  37. [37]

    Malkov and Dmitry A

    Yury A. Malkov and Dmitry A. Yashunin. Efficient and robust approximate nearest neighbor search us- ing hierarchical navigable small world graphs. 2016

    2016

  38. [38]

    Suffix arrays: A new method for on-line string searches

    Udi Manber and Gene Myers. Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing, 22(5), 1993

    1993

  39. [39]

    Sur- vey of vector database management systems

    James Jie Pan, Jianguo Wang, and Guoliang Li. Sur- vey of vector database management systems. The VLDB Journal, 33(5), 2024

    2024

  40. [40]

    Don’t maintain twice, it’s alright: Merged metadata management in deduplication file system with GogetaFS

    Yanqi Pan, Wen Xia, Erci Xu, Hao Huang, Xi- angyu Zou, and Shiyi Li. Don’t maintain twice, it’s alright: Merged metadata management in deduplication file system with GogetaFS. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), 2025

    2025

  41. [41]

    A survey and clas- sification of storage deduplication systems

    Jo ˜ao Paulo and Jos ´e Pereira. A survey and clas- sification of storage deduplication systems. ACM Computing Surveys, 47(1), 2014

    2014

  42. [42]

    Orestis Polychroniou, Arun Raghavan, and Ken- neth A. Ross. Rethinking simd vectorization for in-memory databases. In The ACM SIGMOD International Conference on Management of Data,

  43. [43]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21(1), 2020. ISSN 1532-4435

    2020

  44. [44]

    iDedup: Latency-aware, in- line data deduplication for primary storage

    Kiran Srinivasan, Tim Bisson, Garth Goodson, and Kaladhar Voruganti. iDedup: Latency-aware, in- line data deduplication for primary storage. In 10th USENIX Conference on File and Storage Technologies (FAST 12), 2012

    2012

  45. [45]

    Kushal Tirumala, Daniel Simig, Armen Agha- janyan, and Ari S. Morcos. D4: Improving LLM pretraining via document de-duplication and di- versification. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  46. [46]

    Carey, and Chen Li

    Rares Vernica, Michael J. Carey, and Chen Li. Efficient parallel set-similarity joins using mapre- duce. In Proceedings of the 2010 ACM SIGMOD 16 International Conference on Management of Data, 2010

    2010

  47. [47]

    Data-prep-kit: Getting your data ready for llm application development

    David Wood, Boris Lublinsky, Alexy Roytman, Shivdeep Singh, Constantin Adam, Abdulhamid Adebayo, Sungeun An, Yuan Chi Chang, Xuan- Hong Dang, Nirmit Desai, Michele Dolfi, Ha- jar Emami-Gohari, Revital Eres, Takuya Goto, Dhiraj Joshi, Yan Koyfman, Mohammad Nassar, Hima Patel, Paramesvaran Selvam, Yousaf Shah, Saptha Surendran, Daiki Tsuzuku, Petros Zerfos,...

    2024

  48. [48]

    Efficient similarity joins for near duplicate de- tection

    Chuan Xiao, Wei Wang, Xuemin Lin, and Jeffrey Xu Yu. Efficient similarity joins for near duplicate de- tection. In Proceedings of the 17th International Conference on World Wide Web, 2008

    2008

  49. [49]

    Top-k set similarity joins

    Chuan Xiao, Wei Wang, Xuemin Lin, and Haichuan Shang. Top-k set similarity joins. In Proceedings of the 25th IEEE International Conference on Data Engineering, 2009

    2009

  50. [50]

    Efficient similarity joins for near-duplicate detection

    Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu, and Guoren Wang. Efficient similarity joins for near-duplicate detection. ACM Transactions on Database Systems, 36, 2011

    2011

  51. [51]

    Harmony: A scalable distributed vector database for high- throughput approximate nearest neighbor search

    Qian Xu, Feng Zhang, Chengxi Li, Lei Cao, Zheng Chen, Jidong Zhai, and Xiaoyong Du. Harmony: A scalable distributed vector database for high- throughput approximate nearest neighbor search. ACM on Management of Data, 3(4), 2025

    2025

  52. [52]

    Defending against neural fake news

    Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  53. [53]

    Retsim: Resilient and efficient text similarity

    Marina Zhang, Owen Vallis, Aysegul Bumin, Tanay Vakharia, and Elie Bursztein. Retsim: Resilient and efficient text similarity. 2023

    2023

  54. [54]

    Avoid- ing the disk bottleneck in the data domain dedupli- cation file system

    Benjamin Zhu, Kai Li, and Hugo Patterson. Avoid- ing the disk bottleneck in the data domain dedupli- cation file system. In 6th USENIX Conference on File and Storage Technologies (FAST 08), 2008

    2008

  55. [55]

    Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Ren ´ee J. Miller. Lsh ensemble: Internet- scale domain search. Proceedings of the VLDB Endowment, 9(12), 2016

    2016

  56. [56]

    Erkang Zhu, Dong Deng, Fatemeh Nargesian, Wen- tao Zhu, and Ren´ee J. Miller. Josie: Overlap set sim- ilarity search for finding joinable tables in data lakes. In Proceedings of the 2019 International Conference on Management of Data, 2019

    2019

  57. [57]

    Building a high- performance fine-grained deduplication framework for backup storage with high deduplication ratio

    Xiangyu Zou, Wen Xia, Philip Shilane, Hai- jun Zhang, and Xuan Wang. Building a high- performance fine-grained deduplication framework for backup storage with high deduplication ratio. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022. A Appendix: Bitmap Collision Analysis Overview.We provide the full derivation for the colli- sion analysis...

    2022