Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Anchored Decoding provably bounds generation near a permissive safe model, cutting measurable verbatim copying by up to 75 percent with fluency and factuality nearly intact.

desk verdict Solid method and clean local KL-budget proofs, but the formal K-NAF guarantee as stated doesn't cover the implementation's logit-processed safe distribution—worth fixing before the 'provable' claim ships. read the letter →

arxiv 2602.07120 v2 pith:LMVVVY6F submitted 2026-02-06 cs.CL

Jacqueline He , Jonathan Hayase , Wen-tau Yih , Sewoong Oh , Luke Zettlemoyer , Pang Wei Koh This is my paper · ORCID
classification cs.CL
keywords AnchoredDecodingK-NearAccess-Freenesscopyrightmitigationdecoding-timeinterventionKLdivergencebudgetpermissivelylicensedlanguagemodelsbyte-levelverbatimmemorization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a plug-in, training-free change to how a language model samples next tokens can suppress verbatim reproduction of copyrighted text while keeping most of the model's utility. The method, Anchored Decoding, fuses the next-token distribution of any risky model with that of a safe model trained only on permissively licensed data, under a user-chosen information budget. It proves that if each per-step divergence from the safe model stays within a budget and the per-step budgets sum to at most the global budget, the whole generated sequence provably satisfies the K-NAF criterion—a formal bound on how far the output distribution can drift from the safe model. Two mechanisms make the budget practical: a prefix debt that spends part of the budget when the prompt already looks memorized, and an adaptive banking rule that rolls unused budget forward for later risky steps. A byte-level version removes the shared-tokenizer requirement, and experiments across six model pairs report closing up to 75 percent of the measurable copying gap at near-original fluency and factuality.

What carries the argument

The central object is the K-Near Access-Freeness (K-NAF) criterion: a bound on the KL divergence between the generated distribution and a safe reference distribution. The carrying mechanism is a per-step closed-form fusion p*_t proportional to p_s^{λ/(1+λ)} p_r^{1/(1+λ)}, a weighted geometric mean of the two next-token distributions, with λ chosen so the per-step KL budget holds. Theorem 3.1 and Proposition 3.4 show that per-step budgets summing to at most K produce a sequence-level guarantee, and an adaptive banking rule k_t = max(0, (t+1)k − accumulated spend − prefix debt) lets unspent budget roll forward. The prefix debt δ_init(x), a mean of the top-n positive log-likelihood ratios over

What would settle it

Run Anchored Decoding with a tight budget on a prompt whose protected continuation is also assigned high probability by the safe anchor (for example, a famous quote that appears inside the safe corpus). If the model still emits the protected span verbatim, then K-NAF proximity alone does not track copyright risk, refuting the practical safety claim even though the formal divergence bound may hold.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that copyright-risk control can live entirely in the decoding stage: by solving a small constrained optimization at every token, one can interpolate between a risky and a safe model so that the output distribution provably stays within a user-chosen divergence budget of the safe model's distribution, satisfying the K-NAF criterion. The paper shows that local per-step constraints compose into a global sequence-level guarantee via the chain rule for KL divergence, and that two heuristics—prefix debt and adaptive banking—make budget allocation effective rather than wasteful. Empirically, the method closes up to 75 percent of the copying gap between the

Load-bearing premise

The whole guarantee is relative to a safe model assumed to be trained exclusively on permissively licensed text; if that model contains latent protected text, the bound is relative to an anchor that can itself reproduce copyrighted material.

Editorial extensions

If this is right

  • Any off-the-shelf autoregressive model with exposed logits can be made K-NAF-compliant without retraining or access to its pre-training data.
  • The user-chosen budget K provides a tunable risk-utility trade-off with a formal sequence-level guarantee, not just a heuristic reduction in copying metrics.
  • The byte-level extension enables fusion with safe models that use bespoke tokenizers, which is the common case for permissively trained models.
  • Empirically, at the high-protection operating point (NCR ≥ 75%), the method preserves near-original fluency and factuality, outperforming prior two-model and single-model baselines.
  • The inference overhead is modest (about 1.1x at token level), making the method practical for real-time deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper is right, the same reference-anchored machinery could bound generation against any trusted distribution—for privacy redaction, policy compliance, or domain restrictions—by swapping the safe model for an appropriate reference; this is a direct extension the paper gestures at but does not test.
  • The guarantee is only as strong as the safe model's own training data: latent protected fragments in permissively labeled corpora would make the bound vacuous as a legal safeguard, so deployment should audit the safe anchor's provenance, not just the divergence.
  • The reliance on per-step KL between risky and safe models as a memorization signal may disproportionately suppress legitimate long-tail facts when the safe model is much smaller; a useful follow-up would be to measure and compensate for this suppression per knowledge-frequency bucket.
  • A concrete testable extension: evaluate whether the top-n LLR prefix debt transfers to other regurgitation triggers such as code, lyrics, or non-English text, where the LLR tail may behave differently.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Anchored Decoding, an inference-time method that fuses a risky language model with a permissively trained safe language model by solving a per-step KL-constrained projection. The authors prove a global K-NAF guarantee (Theorem 3.1 and Proposition 3.4) showing that if per-step KL budgets sum to at most K, the autoregressive distribution is within KL divergence K of the safe model. They add a prefix-debt mechanism and an adaptive banking rule, and introduce a byte-level variant built on ByteSampler to handle mismatched tokenizers. A new safe model, TinyComma 1.8B, is released. Experiments across six model pairs report a new Pareto frontier for risk-utility trade-offs, closing up to 75% of the measured copying gap between risky and safe baselines at modest inference overhead.

Significance. If the stated guarantee holds for the actual implementation, this is a valuable contribution: a tunable, plug-and-play decoding method with a formal divergence bound, extending K-NAF from a theoretical criterion to practical autoregressive decoding. The paper has genuine strengths: the KL projection has a clean closed form, the adaptive-banking proof is self-contained and correct for the mathematical abstraction, the ablation study is thorough, and the authors release code and a new permissively trained model. However, the central ``provable'' claim depends on an identified gap between theorem and implementation, and the headline empirical metric is partly circular. These issues do not invalidate the core idea but require correction before the claims as stated can be accepted.

major comments (3)
  1. [Appendix B.4, Theorem 3.1, Proposition 3.4] The formal guarantee is stated relative to the raw safe model p_s, but the implementation applies logit processors and warpers (temperature, repetition penalty) to both p_r and p_s before running the KL projection. The enforced constraint is D_KL(p*_t || p_s^proc) ≤ k_t, not D_KL(p*_t || p_s^raw) ≤ k_t. Consequently, Theorem 3.1 and Proposition 3.4 do not imply D_KL(p* || p_s^raw) ≤ K. The paper never bounds D_KL(p_s^proc || p_s^raw) or adjusts the guarantee. Since the title and abstract claim a provable bound relative to a permissively trained safe model, this is a load-bearing mismatch. The authors should either prove the guarantee for the actual decoded distribution, remove logit processing from the safe side, or quantify the additional divergence introduced by processing.
  2. [Section 4, Eq. (NCR)] The normalized copying reduction is defined as (m_r - m)/(m_r - m_s). Since Anchored Decoding constructs p* as a KL ball around p_s, pushing the output toward p_s mechanically drives NCR upward; a method that simply emitted p_s would score 100% by definition. The headline ``closing up to 75% of the measurable copying gap'' is therefore not an independent measure of risk reduction. The appendix reports absolute metric values, but the main text's high-protection operating point (NCR≥75%) is built on this circular normalization. The authors should re-present the main risk-utility results with absolute metrics or with a less construction-dependent normalization.
  3. [Abstract, Section 1, Appendix A.3] The claim of ``provably reducing copyright risk'' goes beyond what K-NAF provides. K-NAF bounds divergence to p_s, not legal non-infringement. As the paper acknowledges in A.3, p_s may contain latent protected fragments or be trained on data whose permissive provenance is incorrectly verified. In that case, the guarantee only bounds divergence to a model that may still reproduce protected text. The authors should scope the abstract and title claims to ``bounded divergence from a permissively trained safe model,'' or provide a guarantee relative to a verified permissive distribution.
minor comments (5)
  1. [Table 1] The row label ``PROXIMAL/PROXIMALByte'' does not match the method name ``Anchored Decoding / Anchored Byte Decoding'' used elsewhere. Please unify the notation.
  2. [Section 6.3] Typo: ``Copyright prefixes display heavier right tails in prefix LLRs than either CopyrightorFactual prefixes'' should read ``than either Creative or Factual prefixes.'
  3. [Abstract vs. Section 5.1] The abstract says ``eliminating up to 75% of the measurable copying gap,'' while Section 5.1 says ``closing up to 75%.'' The former overstates the result; please use consistent wording.
  4. [Appendix C.3] Typo: ``as smallernmay overreact to a a few spurious outliers'' contains a duplicated article.
  5. [Section 4.3] The prefix-debt window n=5 is selected after sweeping n on the same evaluation domains (Appendix C.3). The paper argues insensitivity, which mitigates the concern, but the procedure should be stated as model selection on the evaluation set.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the K-NAF bound follows from a genuine chain-rule argument; the main caveats are conditional guarantees, not definitional reductions.

full rationale

The central claim 'provably satisfies K-NAF' is a theorem (Thm 3.1, Prop 3.4) derived from the per-step KL constraints via the chain rule; the per-step constraints are the method's construction, and the theorem shows how they compose into a sequence-level bound. This is a mathematical derivation, not a prediction fitted from data. The empirical NCR is defined relative to p_s, so high NCR partly reflects the method's intentional movement toward p_s; however, the paper is transparent that p_s is the gold standard, and the underlying copying metrics (ROUGE, LCS, MinHash) are computed against external copyrighted text (BOOKS), so the utility/factuality tradeoff is an independent empirical result. The ByteSampler citation (Hayase et al., 2025) is a co-author's prior work, but it is a concrete software framework with code, and the token-level guarantee does not rely on it. The paper's Appendix A.3 explicitly acknowledges that the guarantee is conditional on p_s actually being permissively trained and free of latent leakage; that is a limitation of the interpretation, not circularity. The implementation-level gap (logit processors applied before the KL projection, App. B.4) means the theorem's guarantee is formally relative to the processed p_s, not the raw p_s; this is a correctness risk, not a circular reduction. No load-bearing step reduces to its own input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central theorem relies on standard KL chain-rule mathematics plus the availability of a verifiably safe anchor. The empirical performance depends on user choices k and n, and the byte-level path depends on ByteSampler's exactness. No new physical or theoretical entity is postulated.

free parameters (2)
  • k (per-step nominal KL budget) = swept over {0.1, 0.5, 1, 1.5, 2, 3, 4, 5, 10, 15, 20}
    User-chosen risk-utility knob; all Pareto results depend on sweeping it and on the later selection of the NCR>=75 operating point.
  • n (prefix-debt window) = 5
    Set after a sensitivity sweep over n = 1, 3, 5, 7 on the same BOOKS/BIOS evaluation (Appendix C.3). Results are described as largely insensitive, but this is still a tuning decision on the benchmark.
assumptions (4)
  • domain assumption A safe model p_s trained exclusively on permissively licensed text exists, and its data provenance can be verified.
    Invoked in Sec. 2 and discussed in Appendix A.3. If p_s contains latent protected fragments, K-NAF bounds divergence only to a model that may itself reproduce protected text.
  • domain assumption The risky and safe models induce distributions over a common support, or their BPE tokenizers admit exact byte marginalization via ByteSampler.
    Token-level Anchored Decoding requires shared vocabulary (Sec. 2); byte-level decoding relies on ByteSampler's exact next-byte distributions (Sec. 3.6).
  • domain assumption K-NAF divergence to the safe model is an appropriate formalization of copyright risk.
    Definition 2.1 is adopted from Vyas et al. (2023). The paper itself notes in the Impact Statement that no legal conclusions should be inferred, so the formal guarantee is not a legal guarantee.
  • domain assumption ByteSampler's Valid Covering Tree traversal exactly marginalizes token probabilities into next-byte probabilities.
    Borrowed from Hayase et al. (2025); it is a prior algorithmic framework and is not re-proven in this paper.
invented entities (1)
  • TinyComma 1.8B independent evidence
    purpose: A permissively trained safe anchor model using the Llama 3 tokenizer, enabling token-level fusion with Llama-family risky models.
    Released as a public model checkpoint and benchmarked on OLMES tasks (Fig. 7); it is an artifact rather than a postulated theoretical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model." pith.science (2026). https://pith.science/paper/LMVVVY6F

@misc{pith2026260207120,
  author       = {Pith},
  title        = {Pith review of: Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMVVVY6F}},
  note         = {Machine review of arXiv:2602.07120}
}
abstract

Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding adaptively allocates a user-chosen information budget over the generation trajectory and enforces per-step constraints that yield a sequence-level guarantee, enabling a tunable risk-utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as Anchored$_{\mathrm{Byte}}$ Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler framework (Hayase et al., 2025). Across six model pairs on long-form metrics for copying risk and utility, Anchored and Anchored$_{\mathrm{Byte}}$ Decoding define a new Pareto frontier, preserving near-original fluency and factuality while closing up to 75% of the measurable copying gap between the risky baseline and a safe reference, at a modest inference overhead.

Figures

Figures reproduced from arXiv: 2602.07120 by the authors.

Figure 1
Figure 1. (a). Given the opening line of J.R.R. Tolkien’s The Fellowship of the Ring (1954), the risky LM outputs its verbatim continua￾tion, while the safe LM produces a less fluent, repetitive alternative. ANCHORED DECODING generates in bounded proximity to the safe LM within a budget K, while leveraging utility from the risky LM, and produces a plausible, non-infringing continuation. (b). With the safe-risky LM pair {TinyC… view at source ↗
Figure 2
Figure 2. ANCHOREDByte DECODING (in purple) achieves the best risk-utility tradeoff at the byte level across five model pairs. We report the average of three seeds; error bars show standard deviation. The shaded threshold denotes the high-protection operating point, where the Normalized Copyright Reduction (NCR)≥ 75%. NCR and fluency are evaluated on BOOKS, and factuality on BIOS [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Risk-utility tradeoffs for ANCHORED DECODING ablations. We ablate three axes: (i) optimization objective, (ii) prefix debt, and (iii) budgeting strategy. For brevity, our methods are labeled as ANC. DEC. Ablation results [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: High-copying regions are front-loaded under both byte-level and token-level decoding. We plot histograms (bin width of 5) of the start position of copying metrics (LCS and ACS) on COPYRIGHT generations. Copying tends to cluster at early positions. first token (TTFT), t…
Figure 6
Figure 6. Figure 6: Left: KDEs of per-step prefix log-likelihood ratios (LLR) at the token and byte levels. A positive LLR means that pr assigns higher probability than the ps to the realized next step; large positive LLR events occur most often in Copyright. Right: KDEs of δinit(x), the …
Figure 7
Figure 7. Figure 7: Benchmarking LMs on natural language tasks using the OLMES evaluation suite (Gu et al., 2025). Our TinyComma 1.8B (marked with a gold star ⋆) achieves the best downstream task performance among open models of its size range, trailing only behind the Comma 7B and risky …
Figure 8
Figure 8. Figure 8: Debt window trade-off curves for ANCHORED DECODING prefix debt calculation, using n = 1, 3, 5, 7. We use the token￾level model pair {TinyComma 1.8B, Llama 3.1 70B}. The optimal trade-off region is the upper-right corner [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 9
Figure 9. Figure 9: Prefix debt correlates with copying metrics. ANCHORED DECODING prefix debt deciles with token-level model pair {TinyComma 1.8B, Llama 3.1 70B}. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_9.png]
Figure 10
Figure 10. Figure 10: Adaptive budgets track per-step KL spikes. Randomly sampled Copyright trajectories at token (top) and byte (bottom) granularities. We plot per-step divergences KL(p ∗ ∥ps) (blue) and KL(p ∗ ∥pr) (orange), the diagnostic KL(pr∥ps) (green, where shown), and the logged b…
Figure 11
Figure 11. Figure 11: Factual precision on BIOS stratified by entity rarity as a function of the budget parameter k (log scale). As k increases, precision rises across buckets, with slower improvement for long-tail entities than for head and torso entities. 0.12 0.10 0.08 0.06 0.04 0.02 0.…
Figure 12
Figure 12. Figure 12: Risk-utility tradeoff plots on non-literal copying. ANCHORED DECODING remains Pareto-optimal. We evaluate the fluency and extent of character overlap>3 in open-ended story generation. 48 [PITH_FULL_IMAGE:figures/full_fig_p048_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 linked inside Pith

  1. [3]

    Give me the first chapter of Harry Potterin Spanish

    We also report thenon-verbatim fluencyof the model generation, which is scored following App. D.4. We use the token-level model pair{TinyComma 1.8B, Llama 3.1 70B}. We sweep ANCHOREDDECODINGacross various k, and disable the prefix debt, since it is primarily designed to curb literal copying and we find it can hurt fluency in this non-literal setting. We a...

    1997

  2. [5]

    Meeus, M., Jain, S., Rei, M., and de Montjoye, Y .-A

    URLhttps://openreview.net/forum? id=B41hNBoWLo. Meeus, M., Jain, S., Rei, M., and de Montjoye, Y .-A. Did the neurons read your book? document-level member- ship inference for large language models. InProceedings of the 33rd USENIX Conference on Security Symposium, SEC ’24, USA, 2024. USENIX Association. ISBN 978- 1-939133-44-1. Meta AI. Introducing llama...

    2024

  3. [9]

    Qwen Team

    URLhttps://openreview.net/forum? id=gNiT81iag0. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm.github. io/blog/qwen2.5/. R´enyi, A. On measures of entropy and information. InProceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability. Uni- versity of California Press, 1961. URLhttps: //api.s...

    arXiv 2024

  4. [10]

    Song, Y ., Kim, Y ., and Iyyer, M

    URLhttps://openreview.net/forum? id=zWqr3MQuNs. Song, Y ., Kim, Y ., and Iyyer, M. VeriScore: Evalu- ating the factuality of verifiable claims in long-form text generation. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pp. 9447–9474, Miami, Florida, USA, Novem- ber 2024. As...

    arXiv 2024

  5. [11]

    Yao, Y ., Xu, X., and YangLiu

    URLhttps://openreview.net/forum? id=ar8aRMrmod. Yao, Y ., Xu, X., and YangLiu. Large language model un- learning. In Globerson, A., Mackey, L., Belgrave, D., 20 Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp....

  6. [12]

    doi: 10.52202/079017-3346. Yen, H. Retrieval tools, 2025. URLhttps://github. com/howard-yen/RetrievalTools. Ypma, T. J. Historical development of the newton–raphson method.SIAM Review, 37(4):531–551, 1995. doi: 10.1137/1037125. URLhttps://doi.org/10. 1137/1037125. Zhang, J., Sun, J., Yeats, E., Ouyang, Y ., Kuo, M., Zhang, J., Yang, H. F., and Li, H. Min-...

    doi:10.52202/079017-3346 2025

  7. [14]

    expert” model by downweighting undesirable charac- teristics from an “amateur

    URLhttps://openreview.net/forum? id=MXLBXjQkmb. Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023...

    2023

  8. [16]

    the”, “ to

    and adopt an efficient retrieval-based procedure in order to reflect realistic deployment where auxiliary text sources (e.g., from retrieval) may contain protected content that should not be copied. Namely, we construct task-specific blocklists by retrieving from quasi-oracle datastores: the Books3 portion of the Pile (Gao et al., 2020) for copyright eval...

    2020

Show all 19 references
  1. [282]

    findings-naacl.282/

    URLhttps://aclanthology.org/2025. findings-naacl.282/. Hans, A., Kirchenbauer, J., Wen, Y ., Jain, N., Kazemi, H., Singhania, P., Singh, S., Somepalli, G., Geip- ing, J., Bhatele, A., and Goldstein, T. Be like a goldfish, don’t memorize! mitigating memorization in generative L...

    2025

  2. [458]

    emnlp-main.458/

    URLhttps://aclanthology.org/2023. emnlp-main.458/. Karpukhin, V ., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. InPro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural La...

    2023 arXiv

  3. [741]

    emnlp-main.741/

    URLhttps://aclanthology.org/2023. emnlp-main.741/. Min, S., Gururangan, S., Wallace, E., Shi, W., Hajishirzi, H., Smith, N. A., and Zettlemoyer, L. SILO lan- guage models: Isolating legal risk in a nonparamet- ric datastore. InThe Twelfth International Conference on Learning R...

    2023 doi

  4. [800]

    Which way do you reckon Zaphod?

    We highlight verbatim spans from the reference in red. Input:Complete the prefix: away, four figures pounded up a corridor looking for a way out. They emerged into a wide open-plan computer bay. They glanced about wildly. “Which way do you reckon Zaphod?” said Ford. “At a wild...

    1985

  5. [1784]

    Zhang, R., Lin, L., Bai, Y ., and Mei, S

    URLhttps://aclanthology.org/ 2025.emnlp-main.1784/. Zhang, R., Lin, L., Bai, Y ., and Mei, S. Negative preference optimization: From catastrophic collapse to effective un- learning. InFirst Conference on Language Modeling,

    2025

  6. [1997]

    Carlini, N., Tram`er, F., Wallace, E., Jagielski, M., Herbert- V oss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, ´U., Oprea, A., and Raffel, C

    doi: 10.1109/SEQUEN.1997.666900. Carlini, N., Tram`er, F., Wallace, E., Jagielski, M., Herbert- V oss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, ´U., Oprea, A., and Raffel, C. Extract- ing training data from large language models. In30th USENIX Security Sympos...

    1997

  7. [2015]

    Accessed: 2026-01-14. OpenAI. tiktoken.https://github.com/openai/ tiktoken, 2022. Accessed: 2026-01-03. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, ...

    2026 arXiv

  8. [2022]

    Pretraining consists of two stages: (1) a 156B-token general training stage over the entire Common Pile, following domain weights specified by Kandpal et al

    for 169.5B tokens on the Common Pile. Pretraining consists of two stages: (1) a 156B-token general training stage over the entire Common Pile, following domain weights specified by Kandpal et al. (2025), and (2) a 13.5B-token cooldown stage on a weighted mixture of three high-...

    2025

  9. [2023]

    Write a factual biography about{entity}. Include only factual information that you are confident in.\n Biography:

    that leverages recent improvements from Song et al. (2024). FActScore consists of (1) claim extraction: using an LLM to decompose generations into atomic,verifiableclaims, (2) evidence retrieval: fetching relevant evidence for each individual claim from a reliable knowledge ba...

    2024

  10. [2024]

    Hayase, J., Liu, A., Smith, N

    URLhttps://openreview.net/forum? id=DylSyAfmWs. Hayase, J., Liu, A., Smith, N. A., and Oh, S. Sampling from your language model one byte at a time, 2025. URL https://arxiv.org/abs/2506.14123. He, L., Huang, Y ., Shi, W., Xie, T., Liu, H., Wang, Y ., Zettlemoyer, L., Zhang, C.,...

    2025 arXiv

  11. [2025]

    Open Knowledge Foundation

    URLhttps://openreview.net/forum? id=0OTVNEm9N4. Open Knowledge Foundation. Open definition 2.1. https://opendefinition.org/od/2.1/en/,

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.