Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Certified Mitigation of Worst-Case LLM Copyright Infringement

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BloomScrub certifies that large language model outputs can be kept free of long verbatim quotes from an indexed copyrighted corpus.

desk verdict BloomScrub's certified guarantee is real but only covers verbatim quotes from the indexed corpus; the paper is a solid, useful contribution that overstates its scope a bit and has a small algorithmic bug to fix. read the letter →

arxiv 2504.16046 v2 pith:VYZQZ7TG submitted 2025-04-22 cs.CL

Jingyu Zhang , Jiacan Yu , Marc Marone , Benjamin Van Durme , Daniel Khashabi This is my paper · ORCID
classification cs.CL
keywords copyrighttakedownBloomfilterworst-caseinfringementverbatimquotedetectioninference-timemitigationcertifiedriskreductionLLMmemorizationabstention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper contends that the copyright risk that matters most—a model occasionally emitting a long verbatim passage from protected text—can be eliminated at inference time, without retraining and without logit-level access. It introduces BloomScrub, which builds a Bloom filter over character $n$-grams of a copyrighted corpus, scans each generated response for verbatim quotes, and repeatedly rewrites detected quotes with an instruction-tuned model. If, after a fixed number of rewriting rounds, a quote longer than a user-chosen threshold $\tau$ remains, the system refuses to answer. The certified statement is therefore that every served response either contains no normalized verbatim substring longer than $\tau$ from the indexed corpus, or is a refusal; on two news corpora the method drives the percentage of responses containing a quote longer than 100 characters from roughly 20% down to near zero, reaching 0.0% at $\tau=50$ with abstention, while keeping fact-answering utility essentially intact. The reason a sympathetic reader should care is that it turns a fuzzy legal worry into an auditable, corpus-relative guarantee at low deployment cost.

What carries the argument

The central object is a fixed-width Bloom filter over character $n$-grams ($n=25$) of the copyrighted corpus, used as the quote extractor $E_C$. Its defining property is zero false negatives: every $n$-gram inserted is always reported, which makes detection of every quote of length $\geq n$ a certainty rather than a heuristic. The extractor aggregates $k$ consecutive $n$-gram hits at one-character offsets into a single quote of length $n+k-1$, so near-verbatim stitched quotes are caught. The second component is iterative quote-guided rewriting: the extracted quotes are inserted into a template telling a rewrite model to paraphrase the text and completely rewrite those spans; detection and rewriting alternate until no quote exceeds $\tau$ or the iteration budget is exhausted. The optional abstention step is what converts soft reduction into a hard, certified statement: when the loop cannot clean the response, the system returns a refusal, so the guarantee holds unconditionally for the indexed corpus.

What would settle it

Run BloomScrub on a large test set and then re-check every emitted response with an exact, false-positive-free index (for example, a suffix array built from the same normalized corpus $C$); if any non-refusal response contains a substring longer than $\tau$ that appears in $C$, the certified claim is false. A cheaper partial check is to count how often the Bloom filter's false positives trigger rewrites: a high false-positive rate would reveal the cost of the sketch, though it would not by itself break the certificate.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a lightweight inference-time loop can provide certified protection against worst-case verbatim copying. For a protected corpus $C$ and a length threshold $\tau$, the guarantee is: any final response is either a refusal, or no normalized substring of it longer than $\tau$ appears in $C$. The certificate is carried by the Bloom filter's zero-false-negative property: every character $n$-gram of $C$ is inserted, so any quote of length at least $n=25$ is detected, and the extractor merges consecutive overlapping hits into one long span, catching near-verbatim stitched quotes. The paper's experiments compare BloomScrub with decoding-time baselines and show it produces the lowest maximal character- and word-level LCS, nearly eliminates quotes above 100 characters, and preserves QA F1 and information-quality scores; removing the quote-guidance from rewriting makes risk metrics worse, supporting the claim that explicit quote extraction is load-bearing. With abstention, the method reaches $0.0\%$ on its corpus-level metric $\%R_{>Q}(\tau)$ at $\tau=50$.

Load-bearing premise

The guarantee covers only verbatim substrings of the specific normalized corpus $C$ loaded into the Bloom filter; any paraphrase, synonym substitution, light edit, or quote from a document outside that index is invisible to the detector and outside the certificate.

Editorial extensions

If this is right

  • A provider can publish the indexed corpus sketch and the threshold $\tau$, and any auditor can verify compliance by exact matching, turning a claim about copyright risk into a checkable property of the served output stream.
  • Updating the protected corpus—for instance, after a new licensing agreement—requires only rebuilding the Bloom filter; no model retraining and no access to model logits is needed.
  • Because each rewrite iteration tightens the effective threshold, a single deployment can tune enforcement stringency at inference time to match the legal appetite of the use case.
  • The corpus-level metric $\%R_{>Q}(\tau)$ gives future mitigation methods a worst-case benchmark that does not depend on which ground-truth continuation happens to be paired with a prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If certified verbatim removal becomes a standard deployment feature, the practical legal question shifts to what is loaded into the index; the guarantee is only as broad as the corpus $C$, so a provider's indexing choices become a policy decision.
  • The same detect-rewrite-abstain loop transfers to any exact-match constraint—personally identifying information, toxic phrases, confidential identifiers—because the certificate depends only on the zero-false-negative filter and the rewrite model, not on copyright semantics.
  • A natural next experiment, which the paper does not run, is to replace the Bloom filter with an exact suffix-array index: that would keep the certificate intact and remove the small false-positive cost, with memory as the trade-off.
  • The paper's own limitation that paraphrases are out of scope suggests room for a layered guard: a verbatim certificate from this method plus a similarity-based detector for non-literal copying; the two attack different failure modes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes BLOOMSCRUB, an inference-time mitigation pipeline that interleaves Bloom-filter-based detection of verbatim spans from a copyrighted corpus C with LLM-based rewriting, and optionally replaces the response with a refusal if long spans persist. The formal claim is that, because a Bloom filter has no false negatives, any normalized verbatim substring of length at least n from the indexed corpus is detected, and the iterative loop either rewrites it below a threshold tau or abstains, so emitted responses contain no quote from C longer than tau. The authors evaluate on NewsSpan and NewsQA with a fine-tuned Llama-3.1-8B generator, comparing to SysPrompt, MemFree decoding, and R-CAD on a new corpus-level metric %R>Q(tau) plus reference-based LCS/ACS, information quality, and QA utility. Results show near-zero long-quote rates with small utility loss.

Significance. If the guarantee is stated at the right level of scope, this is a useful and clean result: it gives a verifiable hard constraint on verbatim regurgitation from a fixed, indexed corpus, at inference time, with black-box model access, and it scales with a Bloom filter. The paper's decision to make the evaluation corpus-level rather than reference-bound is a genuine improvement, and the iterative quote-guided rewriting is a simple and practical mechanism. The Limitations section is honest about the method's narrow scope for non-literal copying. The main gap is a mismatch between the formal, corpus-relative certificate and the broad 'certified copyright takedown' language in the title and abstract; tightening this framing is essential before publication.

major comments (3)
  1. [Abstract; §3.1; §3.2(A)] The certificate is only for normalized verbatim matches against the specific, fully indexed corpus C. Since only n-grams inserted into the Bloom filter can be detected, any quote from a source outside C, and any paraphrase or lightly edited passage whose normalized n-gram sequence is not in C, is outside the guarantee. The abstract's 'worst-case LLM copyright infringement' and 'copyrighted sources' should be qualified throughout to 'the indexed corpus C'; the Limitations section already concedes non-literal copying, but the title and abstract should not imply a broader certificate. I suggest stating the guarantee as a formal theorem with explicit quantifiers: for a fixed normalized corpus C, with probability 1 (no false negatives), every emitted response is either a refusal or contains no substring s of C with |s| > tau.
  2. [Introduction; Table 2; Figure 2] The Introduction says BLOOMSCRUB provides 'certified copyright takedown—completely eliminating long verbatim quotes from copyrighted sources.' Without abstention, Table 2 and Figure 2 report %R>Q(50) values of 10.9% (NewsSpan) and 5.6% (NewsQA) for BLOOMSCRUB, and %R>Q(100) of 0.0% and 0.1%. Thus 'complete elimination' is only true when the abstention mechanism is enabled, as reported in Table 3. Please revise the claim to explicitly tie elimination to the abstention option, and report both modes consistently in the abstract and introduction.
  3. [§3.2(B); Algorithm 1] Algorithm 1 checks the abstention condition at line 9 using quotes extracted at line 4, i.e., from the response before the final rewrite. As written, whenever the loop exits because the iteration limit is reached after a rewrite, the system abstains even if that final rewrite removed all long quotes; conversely, the check does not inspect the final y that would actually be returned. This is conservative for the safety guarantee but is inconsistent with the prose in §3.2(B) ('if ... the rewrite model still fails to remove all long verbatim quotes') and inflates the measured abstention cost. Please change the pseudocode to re-run the quote extractor on the final y before the abstention decision, and adjust the loop condition so that imax denotes the maximum number of rewrites (e.g., while i < imax).
minor comments (6)
  1. [§4.3] The citation for MemFree decoding is to Ippolito et al. (2019), but the described method is from Ippolito et al. (2022); the bibliography contains both, so the citation should be corrected.
  2. [Figure 6] Figure 6 lacks axis labels and a clear caption for the y-axis, which makes the Levenshtein-distance comparison hard to read.
  3. [Algorithm 1] Algorithm 1's line 8 contains the fragment 'i + +'; use 'i ← i+1' or 'i = i+1' for clarity.
  4. [Appendix A] In Appendix A, 'under the pertaining mode' appears to be a typo for 'pretraining mode'; please clarify the fine-tuning setup.
  5. [Tables 2–4] The empirical comparison would be stronger with standard errors or multiple fine-tuning seeds, since some utility comparisons are essentially null results (e.g., QA F1 47.8 vs 47.9 on NewsSpan) and a single run cannot establish equivalence.
  6. [§3.2(A)] The normalization used before n-gram insertion (whitespace, punctuation, case) should be defined precisely, since the exact coverage of the certificate depends on this normalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the certified guarantee is an algorithmic invariant of the Bloom filter's no-false-negative property plus the abstention rule, not a prediction fitted to its own inputs.

full rationale

I walked the derivation chain. The central certificate in §3.2 is an algorithmic invariant, not an empirical prediction: Algorithm 1 loops quote extraction EC(y) via a fixed-width Bloom filter and rewriting until maxlen(q1...qn)<τ, and optionally abstains; because a Bloom filter has zero false negatives, every character n-gram of C present in y is a hit, so every verbatim span of length ≥n is extracted, and any span longer than τ is caught before the loop exits. Thus the claimed 'no quote from C longer than τ' is true by construction whenever the response is not a refusal; this is the definition of the certificate, not a reduction of a prediction to its fitted input. The empirical metric %R>Q(τ) is computed with a separate Bloom filter that likewise has no false negatives, so it cannot manufacture clean outputs; false positives would only over-report risk, making the evaluation conservative. Self-citations to Data Portraits and the NewsSpan dataset are implementation and benchmark citations, and the load-bearing Bloom filter property is cited to Bloom (1970), an external mathematical result. The paper's own Limitations section concedes that non-literal copying and unindexed sources are outside the certificate, which narrows the practical scope of the claim but does not make the derivation circular. Because no fitted parameter is renamed as a prediction, no uniqueness claim is imported from the authors' prior work, and no result is forced by self-citation, there is no circular step.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No constants are fit to the evaluation results in the manner of a prediction masquerading as derivation; the certified statement follows from Bloom filter semantics. The empirical design nevertheless depends on several hand-set hyperparameters and on domain assumptions about legal risk, normalization equivalence, and judge validity, which are listed above.

free parameters (5)
  • risk threshold tau = 50 or 100 characters
    Defines the boundary between acceptable and risky quote length; the paper varies it in Fig. 3, and the Table 3 comparison changes it together with abstention.
  • Bloom filter gram width n = 25 characters
    Quote extractor indexes character n-grams of length 25; chosen to cover all quotes above the 50- and 100-character thresholds.
  • max rewrite iterations imax = 5
    Limits rewrite cost; if the output is not clean after this many rewrites, the algorithm abstains.
  • rewrite prompt template = template in Appendix B.2
    The instruction given to the rewrite model; the ablation shows quote guidance matters, so the template is a load-bearing design choice.
  • Bloom filter sizing and hash count = not reported
    The paper states a false positive rate below 0.001 but does not give array size or number of hash functions, so memory and over-scrubbing behavior are not exactly reproducible.
assumptions (6)
  • standard math A standard Bloom filter has zero false negatives.
    Invoked in §3.2(A) to argue all quotes of length at least n are extracted; the certification rests on this.
  • domain assumption Normalization of whitespace, punctuation, and cases captures verbatim copying.
    §3.2(A) normalizes both corpus and response before n-gram matching; quotes that differ after normalization are invisible to the detector.
  • domain assumption Fine-tuning Llama-3.1-8B on the target corpus creates a realistic memorization-risk testbed.
    §4.1 evaluates only a generator fine-tuned on NewsSpan or NewsQA; generalization to models with different memorization profiles is not established.
  • domain assumption LLM-as-judge scores for relevance, faithfulness, and hallucination are valid quality measures.
    Appendix D uses Llama-3.3-70B-Instruct with rubric prompts; no human agreement is reported.
  • domain assumption Long verbatim quotes are the worst-case copyright risk worth certifying.
    §1 and §3.1 frame the task around fair use and transformativeness; the Limitations section acknowledges non-literal copying is not addressed.
  • domain assumption The off-the-shelf rewrite model can remove long quotes without destroying information.
    §3.2(B) depends on Prewrite for successful scrubbing; when it fails, abstention bounds risk but may reduce utility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Certified Mitigation of Worst-Case LLM Copyright Infringement." pith.science (2026). https://pith.science/paper/VYZQZ7TG

@misc{pith2026250416046,
  author       = {Pith},
  title        = {Pith review of: Certified Mitigation of Worst-Case LLM Copyright Infringement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VYZQZ7TG}},
  note         = {Machine review of arXiv:2504.16046}
}
read the original abstract

The exposure of large language models (LLMs) to copyrighted material during pre-training raises concerns about unintentional copyright infringement post deployment. This has driven the development of "copyright takedown" methods, post-training approaches aimed at preventing models from generating content substantially similar to copyrighted ones. While current mitigation approaches are somewhat effective for average-case risks, we demonstrate that they overlook worst-case copyright risks exhibits by the existence of long, verbatim quotes from copyrighted sources. We propose BloomScrub, a remarkably simple yet highly effective inference-time approach that provides certified copyright takedown. Our method repeatedly interleaves quote detection with rewriting techniques to transform potentially infringing segments. By leveraging efficient data sketches (Bloom filters), our approach enables scalable copyright screening even for large-scale real-world corpora. When quotes beyond a length threshold cannot be removed, the system can abstain from responding, offering certified risk reduction. Experimental results show that BloomScrub reduces infringement risk, preserves utility, and accommodates different levels of enforcement stringency with adaptive abstention. Our results suggest that lightweight, inference-time methods can be surprisingly effective for copyright prevention.

Figures

Figures reproduced from arXiv: 2504.16046 by the authors.

Figure 1
Figure 1. BLOOMSCRUB works by interleaving two key steps: (1) using a Bloom filter to extract high-risk quotes from model responses, and (2) apply guided rewriting to “scrub” these quotes from the text. This iterative process ensures removal of high-risk quotes while preserving utility. alternating steps: (1) Quoted span detection, which employs a Bloom filter (Bloom, 1970) to efficiently identify verbatim segments at scale, … view at source ↗
Figure 2
Figure 2. BLOOMSCRUB drastically outperforms other methods on long quote reduction. gram prefix unchanged, risking utility while failing to remove the (n − 1)-gram quote (§6.2). Certifying risk reduction through abstention If the max iteration for rewrite is achieved and rewrite model still fails to remove all long verbatim quotes, the BLOOMSCRUB system has the option of abstaining from producing a continuation. In this case,… view at source ↗
Figure 3
Figure 3. Inference-time adaptability of BLOOMSCRUB to different risk threshold τ . As the risk threshold decreases, the mean number of rewrite iterations increases, and BLOOMSCRUB continues to reduce max character LCS and percentage of examples with quotes longer than 100 characters. sponse and corresponding ground truth {yi , gi} N i=1, Maxf = max i f(yi , gi), where f = LCSchar, LCSword, or ACS. We focus on the maximum LCS… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of number of rewrites under dif [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Levenshtein distance between ground truth [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anthropic. 2023. System prompts. https://docs.anthropic.com/en/docs/system-prompts

    work page 2023

  4. [4]

    documentation debt

    Jack Bandy and Nicholas Vincent. 2021. https://arxiv.org/abs/2105.05241 Addressing "documentation debt" in machine learning research: A retrospective datasheet for bookcorpus . Preprint, arXiv:2105.05241

    arXiv 2021

  5. [5]

    Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023. https://proceedings.neurips.cc/paper\_files/paper/2023/file/59404fb89d6194641c69ae99ecdf8f6d-Paper-Conference.pdf Emergent and predictable memorization in large language models . In Advances in Neural Information Processing Sys...

    work page 2023

  6. [6]

    Burton H. Bloom. 1970. https://doi.org/10.1145/362686.362692 Space/time trade-offs in hash coding with allowable errors . Communications of the ACM, 13(7):422–426

    arXiv 1970

  7. [7]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. https://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models . In International Conference on Learning Representations ICLR

    arXiv 2023

  8. [8]

    Brown, Dawn Xiaodong Song, \'U lfar Erlingsson, Alina Oprea, and Colin Raffel

    Nicholas Carlini, Florian Tram \`e r, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, \'U lfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. https://arxiv.org/abs/2012.07805 Extracting training data from large language models . In USENIX Security Symposium USENIX

    arXiv 2020

Show all 41 references
  1. [9]

    Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. https://arxiv.org/abs/2305.00118 Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4 . In Conference on Empirical Methods in Natural Language Processing EMNLP

    2023 arXiv

  2. [10]

    Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, and Pang Wei Koh. 2024. https://arxiv.org/abs/2407.07087 Copybench: Measuring literal and non-literal reproduction of copyright-protected text in lan...

    2024 arXiv

  3. [11]

    Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2024. https://arxiv.org/abs/2403.12958 Dated data: Tracing knowledge cutoffs in large language models . In Conference on Language Modeling COLM

    2024 arXiv

  4. [12]

    Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. 2024. https://arxiv.org/abs/2406.10952 Avoiding copyright infringement via large language model unlearning . Preprint, arXiv:2406.10952

    2024 arXiv

  5. [13]

    Dow Jones & Company

    Inc. Dow Jones & Company. 2024. https://www.courtlistener.com/docket/69280523/dow-jones-company-inc-v-perplexity-ai-inc/ Dow jones & company, inc. v. perplexity ai, inc. No. 1:24-cv-07984 (S.D.N.Y. filed Oct. 21, 2024)

    2024

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  7. [15]

    Ronen Eldan and Mark Russinovich. 2023. https://arxiv.org/abs/2310.02238 Who's harry potter? approximate unlearning in llms . Preprint, arXiv:2310.02238

    2023 arXiv

  8. [16]

    Feist Publications, Inc. v. Rural Tel. Serv. Co. 1991. 499 U.S. 340

    1991

  9. [17]

    Gino Fontana

    Avv. Gino Fontana. 2024. https://doi.org/10.1111/jwip.12331 Web scraping: Jurisprudence and legal doctrines . The Journal of World Intellectual Property, n/a(n/a)

    2024 doi

  10. [18]

    Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein. 2024. https://arxiv.org/abs/2406.10209 Be like a goldfish, don't memorize! mitigating memorization ...

    2024 arXiv

  11. [19]

    Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. https://api.semanticscholar.org/CorpusID:264590727 Sok: Memorization in general-purpose large language models . ArXiv, abs/2310.18362

    2023 arXiv

  12. [20]

    Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. 2023. https://arxiv.org/abs/2303.15715 Foundation models and fair use . Journal of Machine Learning Research JMLR , 24(400):1--79

    2023 arXiv

  13. [21]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  14. [22]

    Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1--37

    2022

  15. [23]

    Daphne Ippolito, Reno Kriz, Jo \ a o Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752--3762

    2019

  16. [24]

    Daphne Ippolito, Florian Tram \`e r, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022. https://arxiv.org/abs/2210.17546 Preventing verbatim memorization in language models gives a false sense of privacy . arX...

    2022 arXiv

  17. [25]

    Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. https://arxiv.org/abs/2310.13771 Copyright violations and large language models . Preprint, arXiv:2310.13771

    2023 arXiv

  18. [26]

    Katherine Lee, A Feder Cooper, and James Grimmelmann. 2023. Talkin''bout ai generation: Copyright and the generative-ai supply chain. arXiv preprint arXiv:2309.08133

    2023 arXiv

  19. [27]

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. https://arxiv.org/abs/2107.06499 Deduplicating training data makes language models better . In Annual Meeting of the Association for Computational Ling...

    2022 arXiv

  20. [28]

    Mark A Lemley and Bryan Casey. 2020. Fair learning. Tex. L. Rev., 99:743

    2020

  21. [29]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE: A Package for Automatic Evaluation of Summaries . In ACL Workshop on Text Summarization Branches Out

    2004

  22. [30]

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024 a . https://openreview.net/forum?id=u2vAyMeLMm Infini-gram: Scaling unbounded n-gram language models to a trillion tokens . In First Conference on Language Modeling

    2024

  23. [31]

    Xiaoze Liu, Ting Sun, Tianyang Xu, Feijie Wu, Cunxiang Wang, Xiaoqian Wang, and Jing Gao. 2024 b . https://arxiv.org/abs/2406.12975 Shield: Evaluation and defense strategies for copyright compliance in llm text generation . Preprint, arXiv:2406.12975

    2024 arXiv

  24. [32]

    Lipton, and J

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. https://arxiv.org/abs/2401.06121 Tofu: A task of fictitious unlearning for llms . Preprint, arXiv:2401.06121

    2024 arXiv

  25. [33]

    Marc Marone and Benjamin Van Durme . 2023. https://arxiv.org/abs/2303.03919 Data portraits: Recording foundation model training data . arXiv preprint arXiv:2303.03919

    2023 arXiv

  26. [34]

    Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. 2024. https://arxiv.org/abs/2402.09363 Copyright traps for large language models . Preprint, arXiv:2402.09363

    2024 arXiv

  27. [35]

    Mosaic Research . 2024. Introducing dbrx: A new state-of-the-art open llm. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

    2024

  28. [36]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT-4 technical report

    2023 arXiv

  29. [37]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. 2023. https://arxiv.org/abs/2305.14739 Trusting your evidence: Hallucinate less with context-aware decoding . Preprint, arXiv:2305.14739

    2023 arXiv

  30. [38]

    Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. https://arxiv.org/abs/1611.09830 Newsqa: A machine comprehension dataset . CoRR, abs/1611.09830

    2016 arXiv

  31. [39]

    Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson

    Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A. Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. 2024. https://arxiv.org/abs/2406.18664 Evaluating copyright takedown methods for language models . Preprint, arXiv:2406.18664

    2024 arXiv

  32. [40]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . In Advances in Neural Information Processing...

    2023 arXiv

  33. [41]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. https://doi.org/10.18653/v1/2024.acl-demos.38 L lama F actory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.