Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Quantization removes memorized text faster than capability, but still leaves most of it extractable at scale—compression is not a privacy defense.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:21 UTC pith:HAC36RBN

load-bearing objection First direct measurement of verbatim extraction across quantization with a capability control; main effect convincing, but the selectivity metric is threshold-sensitive and the scale-trend claim is not supported by the data. the 3 major comments →

arxiv 2607.25451 v1 pith:HAC36RBN submitted 2026-07-28 cs.LG cs.CR

Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

Akshay Sasi This is my paper
classification cs.LG cs.CR
keywords verbatim extractionquantizationmemorizationprivacylarge language modelsmembership inferenceselectivity ratiopost-training quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Using open language models whose training corpus and memorized sequences are both public, this paper measures how much training data a model can still reproduce word for word after it is quantized to lower precision. It finds that verbatim extraction decays faster than general capability (perplexity) at every precision and model size tested—a 'selective forgetting' effect that holds under two independent quantization methods. But the selectivity is insufficient: at the largest model studied, four-bit quantization keeps roughly 95% of capability while still reproducing about 72% of memorized sequences, and the surviving fraction grows with model size. The paper argues that practitioners should not rely on compression to scrub memorized data, and should track extraction rather than membership inference to understand real privacy exposure.

Core claim

The central discovery is that quantizing a language model removes its ability to reproduce memorized training passages faster than it removes its general ability to model text, yet the absolute amount of memorized content that survives compression remains high. The authors introduce a selectivity ratio—the logarithm of the retained extraction fraction divided by the logarithm of the retained capability fraction—and show it exceeds one in every configuration, across two different quantizers and two evaluation corpora. At the largest model in the study, four-bit compression retains about 72% of memorized sequences while losing only about 4% of perplexity-based capability, and the retained frac

What carries the argument

The key machinery is a paired measurement protocol and the selectivity ratio. Each known memorized 64-token sequence is split into a 32-token prompt and a 32-token target; the model generates a continuation greedily and an exact token-for-token match counts as successful extraction. Capability is measured at the same precision as perplexity on held-out text. The selectivity ratio s = log(m)/log(c) compares the retained fraction of extraction (m) to the retained fraction of capability (c), with s > 1 meaning memories decay faster than skills. Measuring both at every precision is what makes the selective-forgetting interpretation possible.

Load-bearing premise

The whole comparison rests on the assumption that the public list of memorized sequences, used with this paper's specific 32-token-prompt/32-token-target greedy protocol, is the correct ground truth for what the full-precision model memorized—yet the full-precision reference extracts only 75–83% of those sequences, so all absolute rates and retained fractions are conditional on this protocol.

What would settle it

If a different ground-truth protocol (for example, longer prompts or sampling-based extraction) brings full-precision extraction rates close to 100%, the retained-fraction ratios would shift materially; should the four-bit retained memorization fraction then fall at or below the retained capability fraction (selectivity ≤ 1) for any model, the claim that quantization selectively forgets memorized content would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Quantization cannot be relied on as a privacy mechanism: at the largest scale tested, four-bit compression still leaves the majority of memorized sequences extractable.
  • Privacy claims about compressed models should be evaluated with verbatim extraction rates, not membership-inference attack success, because the two metrics respond differently to quantization.
  • Because the surviving memorized fraction grows with model size, larger deployed models are likely to leak more of their memorized content after compression.
  • The effect persists across two algorithmically unrelated quantizers, indicating it stems from low-precision rounding itself rather than any one implementation.
  • Quantization shortens how far a model can faithfully reproduce a passage before drifting, rather than just flipping a few borderline sequences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extrapolating the scale trend suggests that at multi-billion-parameter sizes, four-bit quantization may preserve nearly all memorized content while sacrificing only a few percent of capability, making extraction attacks nearly as easy as on the uncompressed model—an extrapolation the paper flags as a limitation.
  • The selectivity gap hints that memorized content occupies a more precision-sensitive subspace of the weights than general knowledge; a testable prediction is that weight perturbation will shift outputs on memorized prompts far more than on ordinary prompts.
  • Using sampling-based or probabilistic extraction instead of greedy exact match would likely change absolute extraction rates, but the relative ordering across precisions could persist; this is an experiment the released code could support.
  • Quantization-aware training might affect memorized content differently than post-training quantization; comparing verbatim extraction from the two training regimes at the same bit width would isolate whether selective forgetting is a consequence of rounding noise or of the training procedure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper measures verbatim extraction from Pythia models (160M, 410M, 1B) across five precision levels (FP32/FP16, INT8, NF4, FP4) plus an independent RTN quantizer, using the public Biderman et al. (2023b) memorized-sequence set. It reports that exact-match extraction falls faster than perplexity degrades (selectivity s = log m / log c > 1) at every configuration, but that at 1B, NF4 retains 71.8% of memories while keeping 95.9% of capability. The paper concludes that quantization is not a privacy defense and that extraction, not membership inference, should be the metric practitioners watch. All code, sampled evaluation data, and per-configuration results are released.

Significance. This is a timely, well-scoped study that directly targets verbatim extraction rather than membership inference, uses an external ground-truth memorized set, includes a capability control at every precision level, and cross-checks with two independent quantization algorithms and two evaluation corpora. The released artifacts are a genuine strength and make the work easy to build on. The qualitative pattern—memorization appears more precision-sensitive than capability—is visible in every row of Table 1. However, the headline selectivity metric and the scale-trend claim need strengthening before the broader conclusions are fully supported.

major comments (3)
  1. [§3, Selectivity (s = log m / log c)] The selectivity ratio is mechanically inflated by the 32-token exact-match threshold. If quantization multiplies each target-token match probability by a roughly constant factor ρ, the probability of matching all 32 tokens scales as ρ^32, while perplexity scales as ρ; the ratio s would then be ~32 even when memorized and ordinary tokens degrade identically. The observed s values (2–8) are smaller but still biased upward by this threshold effect. The paper records per-token accuracy and mean correct-prefix length but never reports them or uses them in s. The reported collapse of mean correct-prefix length (~28 to ~12 at NF4) suggests a real per-token effect, but the current central metric cannot distinguish it from the threshold artifact. Please tabulate per-token accuracy under quantization and compute a continuous per-token selectivity, or otherwise calibrate the all-or-nothing threshol
  2. [Table 1, NF4 rows; §4.2] The headline claim that the surviving fraction 'grows with model size' is contradicted by the paper's own numbers: NF4 retained fraction m is 0.389 at 160M, 0.273 at 410M, and 0.718 at 1B. This is not monotonic. The text acknowledges the 410M dip but still extrapolates to larger models. Since no confidence intervals, sample sizes, or variance estimates are reported, the difference between 0.389 and 0.273 may be within sampling noise. Please provide uncertainty quantification and a trend test, or soften the claim to 'the largest model retains the most' rather than 'grows with model size.'
  3. [§3 Models and ground truth; Table 1] The full-precision reference extracts only 0.756–0.834 of the official memorized set, yet the official set is described as sequences 'the full-precision model reproduces greedily.' Under the stated 32-token-prompt/32-token-target protocol, one would expect the reference to be near 1.0. The shortfall suggests a protocol mismatch—e.g., a different prompt format, generation scheme, or subset selection in the original memorization list. This makes the absolute extraction rates protocol-conditional. In particular, the phrase in §4.2 that NF4 'still reproduces 72% of the memorized sequences it knew at FP16' refers to a relative fraction; the absolute NF4 extraction rate is 0.596, not 0.72. Please clarify the mapping between the official memorized set and the paper's protocol, and report absolute extraction rates relative to the full memorized set as well as the relative retained fraction.
minor comments (5)
  1. [Abstract and §4.2] The phrase 'the surviving fraction grows with model size' overstates the non-monotonic data; see major comment. Similarly, 'still reproducing over seventy percent of memorized sequences' in the introduction is ambiguous and potentially misleading without the reference denominator.
  2. [Figures 2 and 3] No confidence intervals or error bars are shown, even though the extraction rates are based on a sampled subset. Please add at least bootstrap intervals or state the sample size in the caption.
  3. [§3 Quantization / Table 1] The RTN quantizer is not evaluated at 160M, so the claim that the effect appears 'under two unrelated quantization algorithms' is not checked at the smallest scale. Please state why, or add the 160M RTN row.
  4. [Throughout] Minor typos: 'survives quantizationgrowswith' in the abstract; 'V oss' in the Carlini et al. reference. Also consider marking preprints with future dates (e.g., Li and Gao 2026) as such.
  5. [§3 Selectivity] The selectivity formula is scale-free, but its value at 1B NF4 (s=8.0) is extremely sensitive to small changes in c because c is close to 1 (log c ≈ -0.04). Reporting s without uncertainty intervals is risky, especially for the largest model.

Circularity Check

0 steps flagged

No significant circularity: extraction and perplexity are directly measured against external public ground truth, and the selectivity ratio is a definition over those measurements, not a fitted or self-referential construct.

full rationale

The paper's derivation chain is self-contained in the relevant sense. Ground-truth memorized sequences come from the external public Pythia memorized-evals set (Biderman et al., 2023b), not from this paper. For each precision level the authors directly measure verbatim extraction by greedy continuation with an exact-match criterion and directly measure capability by perplexity on held-out corpora. The quantities m(q), c(q), and s(q) are definitions over these two measured families, and the main claims follow by comparing measured numbers across a table. There is no fitted parameter that is later called a prediction, no equation that reduces to another by construction, and no load-bearing self-citation: the author is not citing their own prior results to justify the central premise. Even the strongest methodological concern—that a 32-token exact-match threshold may make memorization decay look faster than a continuous per-token perplexity measure—is a validity or interpretation issue about metric sensitivity, not a circularity: the paper does not assume the conclusion in defining s, and the reported correct-prefix-length and per-token accuracy statistics are additional empirical observations rather than components that enforce the result. The full-precision extraction rates being below 1 also raise a protocol-validity question about the external memorization list, but that concerns the quality of an outside ground truth, not a self-derived input. Accordingly, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claim depends on external ground truth (Pythia memorized-evals set) and on modeling choices: perplexity as capability proxy, the selectivity-ratio definition, and the extrapolation to larger models. No numbers are fitted to data; no new entities are introduced.

axioms (5)
  • domain assumption The Pythia memorized-evals set (Biderman et al., 2023b) accurately enumerates the sequences each model memorized and is directly usable under the paper's 32-token-prompt/32-token-target greedy protocol.
    Used as ground truth in Section 3; reference extraction is only 0.75–0.83, suggesting the list was not constructed under exactly this protocol, yet the paper treats it as the gold standard.
  • domain assumption Perplexity on Pile and WikiText is a valid proxy for general model capability at a given precision.
    Capability control in Section 3 and Figures 2–3; if perplexity under-quantifies capability loss, the selectivity ratio overstates how selectively memorization decays.
  • ad hoc to paper The selectivity ratio s = log m / log c, with m and c defined as retained fractions, is the correct scale-free measure for comparing decay rates.
    Introduced in Section 3; it is a definition, not a derived result, and its interpretation depends on log-loss being the right comparison scale for both quantities.
  • domain assumption The behavior of Pythia models up to 1B parameters extrapolates, at least directionally, to larger deployed models.
    Used in Sections 4.2 and 5 to argue that deployment-scale models retain even more memorized data; explicitly an extrapolation, not a measurement.
  • domain assumption bitsandbytes NF4/FP4 and the hand-written RTN quantizer are representative of post-training quantization methods.
    Used to claim robustness across two unrelated algorithms in Section 4.3.

pith-pipeline@v1.3.0-alltime-deepseek · 7415 in / 11794 out tokens · 113479 ms · 2026-08-01T02:21:45.081326+00:00 · methodology

0 comments
read the original abstract

Language models are almost always quantized before they are deployed, and a growing line of work asks whether quantization also lowers their privacy risk. That work measures privacy almost entirely with membership inference. We think this is the wrong thing to measure for the risk that most people actually worry about, namely a model reproducing its training data word for word, and we measure that directly. Using the Pythia models and the public set of sequences each of them is known to have memorized, we track verbatim extraction across five precision levels, from full precision down to four bits, and across three model sizes, while measuring general capability (perplexity) at every point. We find two things. Quantization is a selective forgetter: verbatim memorization falls off faster than capability at every precision and every model size we tried, and this holds under two unrelated quantization algorithms and two evaluation corpora. But the selectivity is not enough to make quantization a privacy defense, which cuts against the optimistic reading of earlier membership-inference results. At the largest model we study, four-bit quantization still reproduces most of the memorized sequences while giving up only a few percent of capability, and the fraction of memorized data that survives quantization grows with model size. We conclude that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching. All code, sampled evaluation data, and per-configuration results are released.

Figures

Figures reproduced from arXiv: 2607.25451 by Akshay Sasi.

Figure 1
Figure 1. Figure 1: The measurement protocol. Each 64-token ground-truth memorized sequence is split into a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Verbatim extraction against precision, for the three model sizes. Filled squares are the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The capability–memorization plane. Each point is one model at one precision, relative to [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 17 linked inside Pith

  1. [1]

    Zhang, T

    C. Zhang, T. Qu, Z. Li, T. Zhang, J. Pang, and S. Mauw. Bits for privacy: Evaluating post-training quantization via membership inference.arXiv:2512.15335,

    arXiv

  2. [4]

    arXiv:2304.11158. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In30th USENIX Security Symposium,

    Pith/arXiv arXiv

  3. [6]

    arXiv:2202.07646. T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  4. [8]

    arXiv:2305.14314. T. Dettmers and L. Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. InInternational Conference on Machine Learning (ICML),

    Pith/arXiv arXiv

  5. [9]

    arXiv:2212.09720. E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. GPTQ: Accurate post-training quantization for gen- erative pre-trained transformers. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  6. [10]

    arXiv:2210.17323. 7 L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800GB dataset of diverse text for language modeling.arXiv:2101.00027,

    Pith/arXiv arXiv

  7. [15]

    arXiv:1610.05820. V . Smith, A. S. Shamsabadi, C. Ashurst, and A. Weller. Identifying and mitigating privacy risks stemming from language models: A survey.arXiv:2310.01424,

    Pith/arXiv arXiv

  8. [16]

    arXiv:2211.10438. Z. Zhang, F. Wang, X. Li, Z. Wu, X. Tang, H. Liu, Q. He, W. Yin, and S. Wang. Catastrophic failure of LLM unlearning via quantization. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  9. [17]

    arXiv:2410.16454. M. Gupta, N. Waghela, S. Gupta, S. Goel, and S. Shanmugavelu. Pruning as a defense: Reducing memorization in large language models.arXiv:2502.15796,

    Pith/arXiv arXiv

  10. [18]

    Z. Zhou, J. Xiang, C. Chen, and S. Su. Quantifying and analyzing entity-level memorization in large language models.arXiv:2308.15727,

    Pith/arXiv arXiv

  11. [2017]

    arXiv:1609.07843. M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wal- lace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv:2311.17035,

    Pith/arXiv arXiv

  12. [2020]

    Hayes, M

    J. Hayes, M. Jagielski, I. Shumailov, and M. Nasr. Measuring memorization through probabilistic discoverable extraction.arXiv:2410.19482,

    Pith/arXiv arXiv

  13. [2021]

    arXiv:2012.07805. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang. Quantifying memorization across neural language models. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv 2012

  14. [2022]

    arXiv:2208.07339. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  15. [2023]

    arXiv:2304.01373. S. Biderman, U. S. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raff. Emergent and predictable memorization in large language models. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  16. [2024]

    M. N. Haque, H. Yang, Z. Yang, and B. Xu. How quantization impacts privacy risk on LLMs for code? arXiv:2508.00128,

    Pith/arXiv arXiv

  17. [2025]

    Aubinais, P

    E. Aubinais, P. Formont, P. Piantanida, and E. Gassiat. Membership inference risks in quantized models: A theoretical and empirical study.arXiv:2502.06567,

    Pith/arXiv arXiv

  18. [2026]

    arXiv:2507.16872. J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration.arXiv:2306.00978,

    Pith/arXiv arXiv