Pith. sign in

REVIEW 4 major objections 6 minor 17 references

Counterfactual Influence as a Distributional Quantity

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Memorization is a distributional quantity: self-influence alone misses the risk of near-duplicates.

desk verdict Solid empirical study of full counterfactual influence shows near-duplicates lower self-influence but remain extractable, with a new margin statistic, but the headline generalization rests on synthetic duplicates and a qualitative CIFAR-10 demo. read the letter →

arxiv 2506.20481 v1 pith:KJKL5OGQ submitted 2025-06-25 cs.LG cs.AIcs.CLcs.CR

Matthieu Meeus , Igor Shilov , Georgios Kaissis , Yves-Alexandre de Montjoye This is my paper
classification cs.LGcs.AIcs.CLcs.CR
keywords counterfactualinfluenceself-influencememorizationnear-duplicatesextractabilitydistributionprivacyrisklargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that how much a model memorizes a training sample cannot be judged by looking at that sample's influence on its own prediction alone. Instead, memorization should be read as a distributional quantity: every training sample exerts some influence on every other, and the shape of this distribution reveals whether a record is unique or has near-duplicates. The central finding is that records with near-duplicates have much lower self-influence than unique records, yet are far more extractable from the model by prompting. That means self-influence can severely underestimate tangible privacy and memorization risks whenever training data contains near-duplicates. The authors demonstrate this for a GPT-Neo 1.3B model on Natural Questions and for ResNet models on CIFAR-10, and propose the ratio of the largest to second-largest influence as a marker of near-duplicates.

What carries the argument

The central object is the counterfactual influence matrix $I$ with entries $I(x_i \Rightarrow x_t)$, the expected change in model loss on target record $x_t$ when training record $x_i$ is included versus excluded. The diagonal $I(x_t \Rightarrow x_t)$ is the usual self-influence. The paper reads each row or column as a distribution and condenses it with the Top-1 Influence Margin $IM(x_t) = \frac{\max_i I(x_i \Rightarrow x_t)}{\max_{i \neq i^\star} I(x_i \Rightarrow x_t)}$, where $i^\star$ is the most influential record; this ratio measures how dominant a single training point is. The machinery is empirical: $M = 1000$ models are trained on random subsets of the target dataset, and the expectations are approximated by averaging losses across the models that include or exclude each record.

What would settle it

Compute the full influence matrix on a corpus with a ground-truth map of natural near-duplicates, and measure the precision and recall of Top-1 Influence Margin against that map. If low margins occur as often among unique records as among near-duplicates, or if near-duplicate records show high self-influence, the paper's central claim would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that counterfactual self-influence is not a sufficient summary of how a model memorizes a training record. Treating counterfactual influence as a distribution—the vector of influences from all training samples on each target—shows that near-duplicates share influence among themselves: their presence lowers self-influence while raising extractability. In the language-model experiments, records with artificial near-duplicates have mean self-influence $0.495$ versus $1.410$ for unique records, and mean BLEU extraction $0.363$ versus $0.070$. The Top-1 Influence Margin, the ratio of the largest to second-largest influence, separates the two groups ($9.1$ versus $1.3$). On CIFAR-10, the records with the smallest margins are visually near-duplicates, suggesting the distributional signal extends to real data.

Load-bearing premise

The conclusion rests on the assumption that artificially constructed, slightly altered copies of a record behave like the natural near-duplicates found in real training data, and that a small visual spot-check is enough to confirm the influence signal identifies them.

Editorial extensions

If this is right

  • Self-influence alone is an unreliable risk metric when near-duplicates are present; records with near-duplicates can score low on self-influence while being highly extractable.
  • The Top-1 Influence Margin can flag near-duplicate records in real datasets without a separate duplicate-detection pass.
  • Privacy and copyright audits that rely on self-influence rankings will systematically underestimate the memorization of redundant records.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension not tested in the paper: if near-duplicate clusters rather than individual records are the unit of memorization, then deduplication policies should target clusters rather than exact duplicates, because removing one copy from a cluster may leave the other copies' influence intact.
  • Extension not tested in the paper: the Top-1 Influence Margin could be repurposed as a probe for benchmark contamination or copyrighted near-duplicates in training corpora, but its precision and recall on naturally occurring duplicates are unmeasured.
  • Extension not tested in the paper: the tradeoff between self-influence and extractability may shift with model scale and data distribution, so the distributional analysis should be re-run on larger models and non-question-answering data before treating the numbers as universal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that memorization should be studied through the full counterfactual influence distribution over training samples rather than through self-influence alone. Using 1,000 finetuned GPT-NEO 1.3B models on a 1,500-record Natural Questions subset, it computes the full influence matrix for 1,000 unique records and 100 records each accompanied by five artificially crafted near-duplicates (single-token replacements via the Areplace algorithm). The authors report that records with near-duplicates have substantially lower self-influence (0.495 vs. 1.410) but substantially higher BLEU-based extractability (0.363 vs. 0.070), and that the Top-1 Influence Margin separates the two groups (1.29 vs. 9.11). A CIFAR-10 experiment with 1,000 ResNet models suggests that low Top-1 Influence Margin identifies natural near-duplicates, illustrated on six visually inspected targets.

Significance. If the central claim holds, the paper would provide concrete evidence that self-influence underestimates memorization-related risk in the presence of near-duplicates, with implications for privacy auditing and data attribution. The strengths of the work include a retraining-based influence estimator with a large number of models (M=1,000), a stability analysis in Appendix B using Spearman correlation and partition-based standard deviations, and a clearly specified experimental protocol. The main weakness is that the language-model result is established entirely on synthetic, single-token-replacement near-duplicates, and the CIFAR-10 validation is qualitative rather than quantitative. The general claim that 'self-influence alone underestimates tangible risks whenever near-duplicates are present' is therefore conditional on an untested representativeness assumption.

major comments (4)
  1. [Section 4, 'Near-duplicates'] The central language-model finding is built exclusively on near-duplicates generated by the Areplace algorithm, which replaces a single randomly chosen token in the answer while keeping the question identical. This construction guarantees that the model sees the same prompt multiple times, which may inflate BLEU scores and make the influence pattern less representative of natural near-duplicates such as paraphrased answers, multi-token edits, or duplicated documents. To support the abstract's and Section 5's general claim that self-influence underestimates risk 'in the presence of (near-)duplicates,' the authors should test at least one additional, more natural near-duplicate construction or explicitly restrict the claim to single-token-replacement duplicates.
  2. [Section 5, 'Identifying near-duplicates in CIFAR-10' and Appendix D] The CIFAR-10 validation of Top-1 Influence Margin as a near-duplicate indicator is qualitative: the analysis inspects only the six targets with the smallest margin among those with above-median self-influence and judges by eye that the most influential other sample is a near-duplicate. No precision, recall, ROC, or comparison against a random baseline is provided, so the reader cannot assess how reliably low margin indicates near-duplication, nor how often it arises for other reasons (e.g., easy, highly redundant samples). A quantitative evaluation over labeled near-duplicate pairs in CIFAR-10, or at least a comparison of margin distributions for known duplicate versus non-duplicate pairs, is needed to support the claim that 'simply looking at the influence distributions reveals the presence of near-duplicates.'
  3. [Section 5, Table 1 and Appendix C] The extractability comparison rests on mean BLEU scores of 0.363 +/- 0.313 for near-duplicate records versus 0.070 +/- 0.114 for unique records. The large standard deviations and the absence of exact-extraction rates or a statistical test make it difficult to judge how strongly the two groups differ at the record level. Moreover, since the near-duplicates themselves are in the training set, a high BLEU score may reflect generation of one of the near-duplicate variants rather than the target answer itself; the example in Appendix C illustrates exactly this (the generated answer differs from the reference answer and is closer to a near-duplicate). The paper should report the distributions, exact-match rates, and a clearly stated near-extractability threshold, and should qualify what 'extractable' means when the model reproduces a near-duplicate rather than the target record.
  4. [Section 2, Eq. (1), and Section 5] The qualitative direction of the self-influence reduction in the presence of near-duplicates follows directly from the definition in Eq. (1): if near-duplicates appear both when the target is included and when it is excluded, the expected loss in the exclusion condition is lower, mechanically reducing I(x_t => x_t). The authors acknowledge this mechanism in Section 2, but the Results section presents the reduction as an empirical finding. The paper should more explicitly separate the definitional component from the empirical contributions (the magnitude of the effect and the extractability asymmetry), so that the novelty claim is not overstated.
minor comments (6)
  1. [Abstract] There is a missing space in 'howall training samples' in the abstract; please correct the typo.
  2. [Figure 1 caption] The caption contains 'T op-left 50×50' with an erroneous space in 'Top'; please fix.
  3. [Section 5 and Appendix D] The phrase 'was larger that the median' should read 'was larger than the median' in both the main text and the appendix figure captions.
  4. [Appendix B, Figure 4] The y-axis label 'Spearman s R correlation' is missing the apostrophe; it should be 'Spearman's R correlation.'
  5. [Section 5] The sentence 'memorization is multi-faceted phenomenon' should read 'a multi-faceted phenomenon' or 'multi-faceted.'
  6. [Appendix D] The appendix states that the sensitivity analysis 'replicates the analysis from Appendix B,' but Figure 8 shows only the self-influence stability and not the influence-margin stability; please clarify or add the margin stability analysis.

Circularity Check

1 steps flagged · score 6.0 of 10

Lower self-influence for near-duplicate records is a definitional consequence of Eq. 1, though the extraction (BLEU) result is independent; partial circularity.

  1. self definitional [Section 2 (Eq. 1) and Section 5 (Table 1)]
    "Examining Equation 1, when the data distribution D contains many near-duplicates x′t of xt, these duplicates are likely to be included in the sets where xt ∈Dj and where xt /∈Dj. If these near-duplicates strongly influence model behavior on xt, the marginal effect of including xt itself might get diminished. ... We first observe that self-influence is substantially smaller for samples with near-duplicates (0.495) than for unique records (1.410)."

    Equation 1 defines self-influence I(xt⇒xt) as the difference between expected loss with xt excluded and expected loss with xt included. The Section 2 passage states that near-duplicates are present in both expectations, so the marginal effect of xt is reduced. The paper then presents the reduced self-influence of near-duplicate records (Table 1) as an empirical finding. That reduction is the same mechanism spelled out in Section 2, i.e., it follows from the definition together with the experimental construction of adding near-duplicates; it is not a separate discovery. The independent part of the claim is the BLEU-based extractability result, which is not definitional.

full rationale

The paper computes influence by retraining 1,000 models (Eq. 1) and extractability by BLEU, so the quantitative results are not fitted to the conclusions. The central circular step is confined to the claim that 'the presence of (near-)duplicates seriously reduces self-influence': this is essentially the definitional marginal-effect statement from Section 2, and the paper itself says so. The finding that such samples are near-extractable is empirical and gives the paper independent content, as does the Top-1 Influence Margin analysis and the CIFAR-10 illustration. The CIFAR-10 validation is qualitative (visual inspection of six targets with no precision/recall), which is a validity limitation rather than a circularity. Self-citations (Shilov et al. 2024 for the Areplace construction; Tournier & De Montjoye 2022 for the ratio idea) are present but not load-bearing: the Areplace citation is to externally falsifiable prior experiments and the ratio citation is incidental. Overall, the derivation is not forced by a self-citation chain, but one headline component reduces by construction; hence score 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's quantitative results are not fitted to data: influence is estimated by retraining (Eq 1) and extractability by BLEU, an external metric. The main axioms are representativeness of the model/dataset/near-duplicate construction, and the reliability of the Monte Carlo estimate for the full influence matrix, which is checked only for self-influence in Appendix B.

assumptions (4)
  • standard math The empirical expectation in Eq 1 can be approximated by averaging over M=1000 independently sampled training subsets (Monte Carlo approximation).
    Section 4 'Measuring influence' and Appendix B justify via Spearman correlation and standard deviation, but the check is performed for self-influence only, not for the full influence matrix.
  • domain assumption GPT-NEO 1.3B finetuned on Natural Questions is a representative setting for LLM memorization.
    Section 4; the paper uses one small model family, one dataset, and one finetuning recipe, so generalization to larger models and other data distributions is assumed.
  • domain assumption Single-token-replacement near-duplicates (Areplace) capture the relevant properties of real-world near-duplicates.
    Section 4; near-duplicates are crafted by replacing one random token. The generalization to natural duplicates is assumed and only qualitatively supported by the CIFAR-10 visual inspection.
  • domain assumption BLEU score computed with nltk defaults is a valid measure of near-extractable memorization risk.
    Section 4 'Measuring extraction'; the approximate-memorization framing follows Ippolito et al. 2023, but no calibration or threshold analysis for BLEU is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Counterfactual Influence as a Distributional Quantity." pith.science (2026). https://pith.science/paper/KJKL5OGQ

@misc{pith2026250620481,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Influence as a Distributional Quantity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJKL5OGQ}},
  note         = {Machine review of arXiv:2506.20481}
}
read the original abstract

Machine learning models are known to memorize samples from their training data, raising concerns around privacy and generalization. Counterfactual self-influence is a popular metric to study memorization, quantifying how the model's prediction for a sample changes depending on the sample's inclusion in the training dataset. However, recent work has shown memorization to be affected by factors beyond self-influence, with other training samples, in particular (near-)duplicates, having a large impact. We here study memorization treating counterfactual influence as a distributional quantity, taking into account how all training samples influence how a sample is memorized. For a small language model, we compute the full influence distribution of training samples on each other and analyze its properties. We find that solely looking at self-influence can severely underestimate tangible risks associated with memorization: the presence of (near-)duplicates seriously reduces self-influence, while we find these samples to be (near-)extractable. We observe similar patterns for image classification, where simply looking at the influence distributions reveals the presence of near-duplicates in CIFAR-10. Our findings highlight that memorization stems from complex interactions across training data and is better captured by the full influence distribution than by self-influence alone.

Figures

Figures reproduced from arXiv: 2506.20481 by the authors.

Figure 1
Figure 1. Influence of training on xi on the model’s prediction for xt (I(xi ⇒ xt)). Results for all 1, 500 records in Dt (subset from Natural Questions, with artificial near-duplicates), and M = 1, 000 GPT-NEO 1.3B models: (a) Influence matrix I; (b–c) Influence distributions over all xi for selected target records xt, either (b) unique or (c) with near-duplicates. Record type (xt) Self-influence I(xt ⇒ xt) BLEU Top-1 Influe… view at source ↗
Figure 2
Figure 2. Identifying near-duplicates in CIFAR-10 through the influence distribution: (a) the target sample [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Loss curves (a) and BLEU scores (b) for GPT-NEO 1.3B trained on the entire target dataset [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Measuring the reliability of computed influence values for increasing number [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Identifying near-duplicates in CIFAR-10 through the distribution of counterfactual influence. For each row, from [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Identifying near-duplicates in CIFAR-10 through the distribution of counterfactual influence. For each row, from [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Loss curves (a) and accuracy (b) for CIFAR-10 trained on the entire target dataset [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Measuring the reliability of computed influence values for increasing number [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 9 canonical work pages

  1. [3]

    Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    arXiv

  2. [4]

    B. Number of modelsMrequired to reliably estimate influence To understand the number of models M needed to reliably estimate the influence values I(x i ⇒x t) (expectations from Equation 1), we replicate the analysis from (Zhang et al., 2023). Specifically, we take the entire set of results obtained by using Mmax = 1,000 models in our main experiment, and ...

    work page 2023

  3. [8]

    Did the neurons read your book? document-level membership inference for large language models

    Meeus, M., Jain, S., Rei, M., and de Montjoye, Y .-A. Did the neurons read your book? document-level membership inference for large language models. InProceedings of the 33rd USENIX Conference on Security Symposium, pp. 2369–2385, 2024a. Meeus, M., Shilov, I., Faysse, M., and de Montjoye, Y .-A. Copyright traps for large language models. InForty-first Int...

    work page 2022

  4. [9]

    F., Ippolito, D., Choquette-Choo, C

    Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tram`er, F., and Lee, K. Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035,

    arXiv

  5. [11]

    Mosaic mem- ory: Fuzzy duplication in copyright traps for large lan- guage models.arXiv preprint arXiv:2405.15523,

    Shilov, I., Meeus, M., and de Montjoye, Y .-A. Mosaic mem- ory: Fuzzy duplication in copyright traps for large lan- guage models.arXiv preprint arXiv:2405.15523,

    arXiv

  6. [12]

    Wang, X., Antoniades, A., Elazar, Y ., Amayuelas, A., Albalak, A., Zhang, K., and Wang, W. Y . Gener- alization vs memorization: Tracing language models’ capabilities back to pretraining data.arXiv preprint arXiv:2407.14985,

    arXiv

  7. [14]

    We generate an answer from the model using greedy decoding prompted on ‘Q: {qi} A:’ to get a′ i and report the corresponding BLEU score between the reference answer ai and the generated one a′ i. We report one sample for each kind of record xt: a regular member only included once in Dt, a member for which the record was included in Dt alongside its ndup −...

    work page 2021

  8. [39]

    Favre’s retire- ment announcement was made dur- ing a press conference at the 2008 NFL Honors ceremony in the East Room of the White House... 0.0864 Member with near- duplicates where did the name chilean sea bass come from Patagonian toothfish The name ”Chilean Seabass” was invented by a fish wholesaler named Lee Lantz in

    work page 2008

Show all 17 references
  1. [1974]

    A., Lee, K., and Cooper, A

    Hayes, J., Swanberg, M., Chaudhari, H., Yona, I., Shu- mailov, I., Nasr, M., Choquette-Choo, C. A., Lee, K., and Cooper, A. F. Measuring memorization in language models via probabilistic extraction. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of...

    2025

  2. [1977]

    He considered ”Pacific sea bass” and ”South Amer- ican sea bass” before settling on ”Chilean sea bass”

    He was looking for a name that would make it attractive to the American market. He considered ”Pacific sea bass” and ”South Amer- ican sea bass” before settling on ”Chilean sea bass”... (seabass) The name ”Chilean Seabass” was invented by a fish wholesaler named John L. L. Smi...

    2004

  3. [2016]

    models on a subset from CIFAR-10 (Krizhevsky et al., 2009). Each model is trained for 30 epochs using SGD (momentum 0.9, weight decay 5e−4 ), with a batch size of 128 and a linear learning rate schedule that warms up over 5 epochs to 0.5 before decaying to zero by the end of t...

    2009

  4. [2019]

    Z., Choquette-Choo, C

    Liu, K. Z., Choquette-Choo, C. A., Jagielski, M., Kairouz, P., Koyejo, S., Liang, P., and Papernot, N. Language mod- els may verbatim complete text they were not explicitly trained on.arXiv preprint arXiv:2503.17514,

    arXiv

  5. [2020]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

    arXiv

  6. [2022]

    K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y ., Pratapa, A., Neiswanger, W., Strubell, E., Mitamura, T., et al

    Choe, S. K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y ., Pratapa, A., Neiswanger, W., Strubell, E., Mitamura, T., et al. What is your data worth to gpt? llm-scale data valuation with influence functions.arXiv preprint arXiv:2405.13954,

    arXiv

  7. [2023]

    Copy- right violations and large language models

    Karamolegkou, A., Li, J., Zhou, L., and Søgaard, A. Copy- right violations and large language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  8. [2024]

    Petroni, F., Rockt¨aschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y ., and Miller, A. Language models as knowl- edge bases? InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu...

    2019

  9. [2025]

    Membership infer- ence attacks against language models via neighbourhood comparison.arXiv preprint arXiv:2305.18462,

    Mattern, J., Mireshghallah, F., Jin, Z., Sch ¨olkopf, B., Sachan, M., and Berg-Kirkpatrick, T. Membership infer- ence attacks against language models via neighbourhood comparison.arXiv preprint arXiv:2305.18462,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.