Pith. sign in

REVIEW 3 major objections 5 minor 41 references

What Really is a Member? Discrediting Membership Inference via Poisoning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Membership inference tests remain unreliable under neighborhood-based definitions of membership, because a data-poisoning adversary can flip their predictions.

desk verdict PoisonM is a strong empirical attack, but Theorem 5.1's proof relies on an unjustified distributional equality and the theorem is false as stated; major revision needed. read the letter →

arxiv 2506.06003 v1 pith:SVKVLO6G submitted 2025-06-06 cs.LG cs.CR

Neal Mangaokar , Ashish Hooda , Zhuohang Li , Bradley A. Malin , Kassem Fawaz , Somesh Jha , Atul Prakash , Amrita Roy Chowdhury This is my paper
classification cs.LGcs.CR
keywords membershiptestsinferencepoisoningdefinitionpointtesttraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that membership inference (MI) tests, which try to determine whether a specific text was in a language model's training data, stay unreliable even after the definition of 'member' is relaxed to include semantic neighbors. It proves a fundamental trade-off: the better a test separates members from non-members on clean data, the more easily a poisoning adversary can flip its predictions by substituting a few carefully chosen training points. The attack, called PoisonM, maps a neighbor of the target point to a non-neighbor that produces the same membership score, so the test is fooled while the true membership label under the relaxed definition stays unchanged. Empirically, PoisonM drives the AUC of five standard tests below random across four neighborhood definitions, including values as low as 0.043 for exact-match LOSS and 0.089 for the 7-gram reference test. If the paper is right, no existing MI test of the studied kinds can be considered reliable in any setting where an adversary might have planted data.

What carries the argument

The machinery is Theorem 5.1, a trade-off bound that ties a test's clean-data advantage (Youden's J statistic integrated over all thresholds) to its robust advantage under worst-case membership-invariant substitution: the sum of the two is at most the expected mapping error $\delta^*$. The companion mechanism is PoisonM, which samples a neighbor (or non-neighbor) of the target point and then greedily replaces tokens to minimize the difference between the membership score produced by the poison and the score produced by the sampled point, while keeping the poison just outside (or inside) the neighborhood ball. The attack succeeds because the superlevel sets of MI tests—the regions that trigger high membership scores—are misaligned with the neighborhood boundaries defined by n-gram overlap, embedding similarity, edit distance, or exact match.

What would settle it

A concrete falsifying test would be to construct a membership test whose superlevel sets align with the neighborhood boundary, for instance a test that thresholds on exact n-gram overlap with the training distribution, and then run PoisonM on that test: if the AUC stays at or above random, the attack fails. Alternatively, directly measure the expected mapping error $\delta^*$ for a given test and neighborhood pair; if it is large, the theorem's bound is vacuous and the attack should not be able to push the test below random.

Watch

Extended reading notes

Core claim

The central claim is that membership inference remains unreliable even under the relaxed, neighborhood-based definition of membership, because an adversary can poison the training dataset to cause incorrect predictions for a target point while preserving that point's membership label. The formal result, Theorem 5.1, states that the test's advantage under worst-case membership-invariant poisoning (robust sensitivity plus robust specificity minus one, integrated over thresholds) plus its advantage without poisoning is bounded above by the expected mapping error $\delta^*$: $$\int_0^\infty (\mathrm{RSens}_{b_1}(T_\gamma, x_t) + \mathrm{RSpec}_{b_2}(T_\gamma, x_t) - 1)\,d\gamma + \int_0^\infty (\mathrm{Sens}(T_\gamma, x_t) + \mathrm{Spec}(T_\gamma, x_t) - 1)\,d\gamma \leq \delta^*.$$ When the mapping error is small, a test that performs well on clean data turns its own strength against itself: the better it separates members from non-members, the more vulnerable it is to poisoning. The empirical counterpart is PoisonM, a concrete attack that consistently reduces the AUC of existing MI tests to below random, and in many cases close to zero, across n-gram, embedding, edit-distance, and exact-match neighborhood definitions.

Load-bearing premise

The central claim rests on the assumption that low-error poisons actually exist for realistic tests and neighborhoods: that token substitutions can map a neighbor of the target to a non-neighbor that produces nearly the same membership score, an assumption the paper demonstrates empirically in fine-tuning settings but does not prove in general.

Editorial extensions

If this is right

  • If the paper is correct, MI tests cannot be used as reliable evidence in copyright disputes, fairness audits, or regulatory compliance checks whenever an adversary might have planted data in the training pipeline.
  • The theorem implies a direct trade-off: there is no MI test that is both highly accurate on clean data and robust to this form of poisoning, so test designers must choose between the two.
  • Dataset inference, the ensemble-based extension of MI testing, also collapses under poisoning, because driving individual tests below random drives their ensemble below random.
  • The attack works across four common neighborhood definitions (n-gram, embedding, edit distance, exact match) and across model sizes from 2.7B to 12B, suggesting the fragility is not an artifact of one test or one neighborhood.
  • Because the attack requires only small budgets for false negatives (a single substituted point, $b_1=1$) it is practical against fine-tuning pipelines where poisoned data can be injected through public web sources.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editor's inference: the trade-off in Theorem 5.1 likely extends beyond membership inference to any score-based auditing tool that thresholds a model-derived statistic, since the argument only relies on the misalignment between the test's superlevel sets and the semantic notion it claims to measure.
  • Editor's inference: the paper's heuristic justification for the existence of low-error poisons suggests that a principled characterization of when superlevel sets align with neighborhood boundaries could lead to a new class of tests that are both accurate and robust, if such alignment can be enforced.
  • Editor's inference: the attack's white-box requirement (access to the target model's activations) could potentially be relaxed using transferable poisons, which would make the threat more realistic for API-only settings; this is not claimed by the paper.
  • Editor's inference: the same fragility is likely to appear in image domains under approximate membership definitions based on rotations, crops, or filters, as the paper itself suggests, and a direct test of that prediction would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that membership inference (MI) tests remain unreliable even under relaxed, neighborhood-based definitions of membership. The authors formalize a pointwise framework for neighborhood membership, propose a poisoning attack (PoisonM) that substitutes training points with 'worst-case' neighbors or non-neighbors, and state a general trade-off theorem (Theorem 5.1) bounding the sum of a test's clean advantage and its robust advantage by an expected mapping error δ*. They instantiate PoisonM for four neighborhood definitions (n-gram, embedding, edit distance, exact match) and report experiments on fine-tuned Pythia models showing that five popular MI tests drop to AUC below random, often close to zero.

Significance. If Theorem 5.1 were correct, it would be a strong conceptual result: a fundamental accuracy-robustness trade-off for membership inference under relaxed definitions, with implications for copyright, auditing, and privacy regulation. The empirical attack is also a useful contribution: it is MI-test agnostic, works across several neighborhood definitions, and the reported AUC reductions are large and consistent. The paper is clearly organized and the experimental setup is reasonably detailed. However, the central theorem is false as stated, and the paper never measures the key quantity δ* that its own bound depends on; the empirical results, while interesting, do not validate the theoretical claim as presented.

major comments (3)
  1. [Section 5, Theorem 5.1; Appendix A.7] The quantity δ* is never measured or bounded in the experiments. The paragraph after Theorem 5.1 states 'This is also empirically validated in Section 6,' but Section 6 reports only AUC and TPR@1% FPR of the tests before and after poisoning. A reduction in AUC is consistent with a small δ* but does not demonstrate it; without an estimate of δ*, the theorem's bound is vacuous and the claimed trade-off is not empirically confirmed.
  2. [Section 6.1, Section 6.2] The evaluation is limited to a white-box fine-tuning setting: Algorithm 1 requires access to the target model's activations fθ and the training algorithm, and all experiments use Pythia models fine-tuned on poisoned data. The threat model of Section 4 is a web-poisoning adversary who does not necessarily know the target model or training procedure, and the paper does not demonstrate that PoisonM transfers to a realistic black-box or pre-training scenario. Furthermore, the paper does not quantitatively verify that the generated poisons preserve the target's neighborhood membership; the experimental protocol relies on algorithmic stopping criteria (e.g., 'stop once the point qualifies as a neighbor') without reporting how often the resulting point is a true neighbor/non-neighbor or whether the substitutions change the membership status of other target points.
  3. [Theorem 5.1, Appendix A.4, Appendix A.5] The theorem statement and proofs contain notation errors that hinder verification. In the definition of δ*, the second term writes 'b2=|v∩Nr(xt)|, v∼D' with v undefined (presumably b2=|D∩\bar{N}_r(xt)|). In Lemma A.4 the PoisonMap argument is written as \bar{N}_r(x) where the surrounding text and Eq. (4) indicate S=N_r(x) should appear for neighbor poisons; Lemma A.5 has the analogous issue. These ambiguities should be corrected before the theorem can be checked.
minor comments (5)
  1. [Definition 4.2] The complement of N_r(x) is typeset as 'N r(x)' in several places, making it easy to confuse with N_r(x); please use \bar{N}_r(x) consistently.
  2. [Section 5, paragraph after Theorem 5.1] The word 'Specfically' should be 'Specifically'.
  3. [Algorithm 1, line 8] The notation 'Substitute i(xpoison,min(Loss_S(i,xpoison,θ,xt)))' is unclear; please write 'Replace the i-th token with the token minimizing the loss'.
  4. [References] References [15] and [33] are the same paper (Persistent Pre-Training Poisoning of LLMs) and should be deduplicated.
  5. [Table 2] The 'Natural' AUC values are all below 0.65, which is quite low for membership inference tests; the paper should comment on whether this reflects the canary/background protocol, the fine-tuning setting, or the inherent difficulty of MI on LLMs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trade-off bound is a direct derivation from definitions; the unproved distributional identity in the proof is a validity concern, not a circular one.

full rationale

The paper's central formal claim (Theorem 5.1) is derived as follows: Lemma 3.5 identifies the clean advantage integral with an expected score difference; Lemmas A.4 and A.5 bound the worst-case robust expectations by targeted-expansion expectations plus the mapping error δ*, using only the definition of PoisonM's mapping error and Jensen's inequality; Theorem A.7 then subtracts the clean advantage via two asserted distributional identities. This chain does not fit any parameter to data that it later 'predicts,' and it does not rename or cite into existence the trade-off. The mapping error δ* is an independently defined, empirically measured quantity (the minimal achievable score discrepancy), and the theorem's force is conditional on δ* being small, which the paper supports by direct experimental attack evaluation rather than by construction. The one self-citation ([21], on certificate-based refutation of MI tests) is background context, not load-bearing. The main caveats are validity concerns rather than circularity: the distributional equalities in Appendix A.7 are asserted without proof and appear to require an invariance of the dataset distribution that is not stated among Definitions 3.1-4.2; also, the practical existence of low-error poisons is demonstrated empirically, not proven. Neither issue makes the derivation circular, so no circular step is flagged.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central theorem relies only on the neighborhood model, the perturbation sets, and standard probability manipulations. The empirical attack adds hand-chosen hyperparameters (λ, radii, budgets) and assumes a heuristic property of token substitutions that is validated only experimentally. No new physical or modeled entities are introduced.

free parameters (4)
  • lambda_embedding = -1.5
    Hand-chosen weight balancing activation similarity against embedding distance in Table 1; no sensitivity analysis is given. It affects which poisons are produced.
  • lambda_edit = 1.5
    Hand-chosen weight in the edit-distance poison loss (Table 1); no sensitivity analysis is given.
  • neighborhood radii (n-gram k, embedding c, edit l) = k=7, c=0.9, l=0.48
    Fixed evaluation hyperparameters that define membership; changing radius changes attack difficulty (see Figure 3), so results are tied to these choices.
  • poison budget = b1=1 (members), b2=10 (non-members)
    Chosen for the experiments; the authors flag in Limitations that b2=10 is needed, which is a practical constraint.
assumptions (4)
  • domain assumption Membership is defined by a distance ball N_r(x), and a thresholded MI test score is a valid signal for membership (Definitions 3.1, 3.2).
    The entire framework and the theorem's metrics are built on this definition; it is a modeling choice, not derived.
  • standard math The training algorithm L and dataset distribution D admit well-defined expectations, and the score Tγ is nonnegative and integrable.
    Used in Lemma A.1 survival-function representation and in Jensen's inequality in Lemmas A.4-A.5.
  • domain assumption An adversary can plant arbitrary points into the (fine-tuning) training dataset while the model owner is honest (Section 4 threat model).
    The threat model assumes internet-scale poisoning is feasible, citing Carlini et al. [14].
  • ad hoc to paper Token-level substitutions preserve the membership score behavior of the model closely enough for the constructed poison to transfer (Algorithm 1, Table 1 losses).
    PoisonM's success is not derived; it is a heuristic property of LLM activations that the experiments test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Really is a Member? Discrediting Membership Inference via Poisoning." pith.science (2026). https://pith.science/paper/SVKVLO6G

@misc{pith2026250606003,
  author       = {Pith},
  title        = {Pith review of: What Really is a Member? Discrediting Membership Inference via Poisoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVKVLO6G}},
  note         = {Machine review of arXiv:2506.06003}
}
read the original abstract

Membership inference tests aim to determine whether a particular data point was included in a language model's training set. However, recent works have shown that such tests often fail under the strict definition of membership based on exact matching, and have suggested relaxing this definition to include semantic neighbors as members as well. In this work, we show that membership inference tests are still unreliable under this relaxation - it is possible to poison the training dataset in a way that causes the test to produce incorrect predictions for a target point. We theoretically reveal a trade-off between a test's accuracy and its robustness to poisoning. We also present a concrete instantiation of this poisoning attack and empirically validate its effectiveness. Our results show that it can degrade the performance of existing tests to well below random.

Figures

Figures reproduced from arXiv: 2506.06003 by the authors.

Figure 1
Figure 1. MI tests are not robust to membership invariant perturbations (e.g., substitution). By leveraging the mis￾alignment between the membership neighborhood and the test’s superlevel set, one can alter a dataset—without changing the ground truth label of a target xt—by substituting non-neighbors with worst-case non-neighbors to cause the test Tγ to mispredict xt as a member. Conversely, replacing a neighbor with a worst-… view at source ↗
Figure 2
Figure 2. ROC curves for MI tests using the n-gram (k=7) neighborhood on AI4Privacy. Dataset Inference. Dataset inference extends MI testing to whole datasets by (1) ensembling existing tests via a linear model and (2) using a T-test to compare scores from a suspect set to a reference set of known non-members [20]. We test whether poisoning affects this method by evaluating it on [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. TPR and FPR of the LOSS test after poisoning using n-gram neighborhood definitions ∈ [5, 7, 9, 11] on AI4Privacy. Impact of Model Size. We also study how model size affects poisoning success by repeating our n-gram (k = 7) experiments on AI4Privacy using Pythia 2.7B and 12B. PoisonM consistently reduces test performance across all sizes (see [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 36 canonical work pages

  1. [1]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), pages 3–18. 9 IEEE, 2017

    work page 2017

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. InProc. of ICML, 2023

    work page 2023

  3. [3]

    The times sues openai and microsoft over a.i

    The New York Times. The times sues openai and microsoft over a.i. use of copyrighted work. https: //www.nytimes.com/2023/12/27/business/media/new- york- times- open- ai- microsoft- lawsuit.html, 2023. Accessed: 05/15/2025

    work page 2023

  4. [4]

    Us authors’ copyright lawsuits against openai and microsoft combined in new york with newspaper actions

    The Guardian. Us authors’ copyright lawsuits against openai and microsoft combined in new york with newspaper actions. https : / / www . theguardian . com / books / 2025 / apr / 04 / us - authors- copyright- lawsuits- against- openai- and- microsoft- combined- in- new- york- with-newspaper-actions, 2025. Accessed: 05/15/2025

    work page 2025

  5. [5]

    Publisher ziff davis sues openai for copyright infringement

    Reuters. Publisher ziff davis sues openai for copyright infringement. https://www.reuters.com/ business/publisher- ziff- davis- sues- openai- copyright- infringement- 2025- 04- 24/ ,

    work page 2025

  6. [6]

    Understanding the eu ai act: Requirements and next steps

    isaca.org. Understanding the eu ai act: Requirements and next steps. https://www.isaca.org/ resources/white-papers/2024/understanding-the-eu-ai-act, 2024. Accessed: 05/15/2025

    work page 2024

  7. [7]

    Transparency and accountability in ai systems: safeguarding wellbeing in the age of algorithmic decision-making.Frontiers in Human Dynamics, 6:1421273, 2024

    Ben Chester Cheong. Transparency and accountability in ai systems: safeguarding wellbeing in the age of algorithmic decision-making.Frontiers in Human Dynamics, 6:1421273, 2024

    work page 2024

  8. [8]

    Artificial intelligence bias auditing–current approaches, challenges and lessons from practice.Review of Accounting and Finance, (ahead-of-print), 2025

    Sabina Lacmanovic and Marinko Skare. Artificial intelligence bias auditing–current approaches, challenges and lessons from practice.Review of Accounting and Finance, (ahead-of-print), 2025

    work page 2025

Show all 41 references
  1. [9]

    Do Membership Inference Attacks Work on Large Language Models? InProc

    Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do Membership Inference Attacks Work on Large Language Models? InProc. of COLM, 2024

    2024

  2. [10]

    Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data.arXiv preprint arXiv:2409.19798, 2024

    Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Position: Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data.arXiv preprint arXiv:2409.19798, 2024

    2024 arXiv

  3. [11]

    Blind Baselines Beat Membership Inference Attacks for Foundation Models.arXiv preprint arXiv:2406.16201, 2024

    Debeshee Das, Jie Zhang, and Florian Tramèr. Blind Baselines Beat Membership Inference Attacks for Foundation Models.arXiv preprint arXiv:2406.16201, 2024

    2024 arXiv

  4. [12]

    What Does it Mean for a Language Model to Preserve Privacy? InProc

    Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What Does it Mean for a Language Model to Preserve Privacy? InProc. of FAccT, 2022

    2022

  5. [13]

    Language Models May Verbatim Complete Text They Were Not Explicitly Trained On

    Ken Ziyu Liu, Christopher A Choquette-Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, and Nicolas Papernot. Language Models May Verbatim Complete Text They Were Not Explicitly Trained On. InProc. of ICLR, 2025

    2025

  6. [14]

    Poisoning Web-scale Training Datasets is Practical

    Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-scale Training Datasets is Practical. InProc. of IEEE S&P, pages 407–425. IEEE, 2024

    2024

  7. [15]

    Persistent Pre-Training Poisoning of LLMs

    Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent Pre-Training Poisoning of LLMs. InProc. of ICLR, 2025

    2025

  8. [16]

    Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. InProc. of CSF, 2018

    2018

  9. [17]

    Detecting Pretraining Data from Large Language Models

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. InProc. of ICLR, 2024

    2024

  10. [18]

    Extracting Training Data from Large Language Models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting Training Data from Large Language Models. InProc. of USENIX Security, 2021

    2021

  11. [19]

    Membership Inference Attacks against Language Models via Neighbourhood Comparison

    Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership Inference Attacks against Language Models via Neighbourhood Comparison. InProc. of ACL, 2023

    2023

  12. [20]

    LLM Dataset Inference: Did you train on my dataset? InProc

    Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. LLM Dataset Inference: Did you train on my dataset? InProc. of NeurIPS, 2024. 10

    2024

  13. [21]

    On the Reliability of Membership Inference Attacks

    Amrita Roy Chowdhury, Zhifeng Kong, and Kamalika Chaudhuri. On the Reliability of Membership Inference Attacks. InProc. of SaTML, 2025

    2025

  14. [22]

    Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models

    Yuxin Wen, Leo Marchyok, Sanghyun Hong, Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models. InProc. of NeurIPS, 2024

    2024

  15. [23]

    Index for rating diagnostic tests.Cancer, 3(1):32–35, 1950

    William J Youden. Index for rating diagnostic tests.Cancer, 3(1):32–35, 1950

    1950

  16. [24]

    Towards Deep Learning Models Resistant to Adversarial Attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. InProc. of ICLR, 2018

    2018

  17. [25]

    Intriguing Properties of Neural Networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. InProc. of ICLR, 2014

    2014

  18. [26]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  19. [27]

    Enhanced Membership Inference Attacks against Machine Learning Models

    Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. InProc. of CCS, CCS ’22, page 3093–3106, New York, NY , USA, 2022. Association for Computing Machinery

    2022

  20. [28]

    Noisy Neighbors: Efficient membership inference attacks against LLMs

    Filippo Galli, Luca Melis, and Tommaso Cucinotta. Noisy Neighbors: Efficient membership inference attacks against LLMs. InProc. of the Fifth Workshop on Privacy in Natural Language Processing, 2024

    2024

  21. [29]

    Multilingual E5 Text Embeddings: A Technical Report.arXiv preprint arXiv:2402.05672, 2024

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 Text Embeddings: A Technical Report.arXiv preprint arXiv:2402.05672, 2024

    2024 arXiv

  22. [30]

    MTEB: Massive Text Embedding Benchmark

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Embedding Benchmark. InProc. of EACL, 2023

    2023

  23. [31]

    PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models

    Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, and Furong Huang. PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models. In Luis Chiruzzo, Alan R...

    2025

  24. [32]

    Adversarial Example Defense: Ensembles of Weak Defenses are not Strong

    Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial Example Defense: Ensembles of Weak Defenses are not Strong. InProc. of USENIX WOOT, 2017

    2017

  25. [33]

    Persistent pre-training poisoning of LLMs

    Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent pre-training poisoning of LLMs. InProc. of ICLR, 2025. 11 A Proofs Lemma A.1.(Expectation using survival function). LetXbe a random var...

    2025

  26. [35]

    semantics

    = 1, then we have E[X] = Z ∞ s=0 P(X > s)ds. Proof. E[X] = Z ∞ x=0 xP(X=x)dx = Z ∞ x=0 P(X=x) Z x s=0 ds dx = Z ∞ s=0 Z ∞ x=s P(X=x)dx ds = Z ∞ s=0 P(X > s)ds Lemma A.2.(Restatement of Lemma 3.5). The advantage of a membership inference test is given by E D∼D θ=L(D) (T(x, θ)|x...

  27. [36]

    Here we can setλ= 0since replacing tokens automatically breaks upn-grams for free

    N-gram(n=k): Iteratively (a) select a token uniformly at random, (b) replace it with a token from the vocabulary such that last-layer activations of resulting sequence have maximum cosine similarity with activations of xt, where activations are computed using model that will b...

  28. [37]

    Embedding(cosine_sim=c): Iteratively (a) select a token uniformly at random, (b) pick the token that both maximizes activation cosine similarity and also minimizes (weighted by factor of λ=−1.5) cosine similarity of the embedding (fromE) of the resulting sequence with that ofx t

  29. [38]

    4.ExactMatch:Same procedure as that forn-grams, but only a single iteration

    EditDistance(distance=l): Same procedure as that for embeddings, except we now maximize the edit distance (λ= 1.5) instead of minimizing embedding cosine similarities. 4.ExactMatch:Same procedure as that forn-grams, but only a single iteration. Finding Poison Neighbors.We samp...

  30. [39]

    (c) repeat until cosine similarity exceedsc

    Embedding(cosine_sim=c): Iteratively (a) select a token uniformly at random, and (b) replace it with a token from vocabulary that maximizes cosine similarity of the embedding (under E) of the resulting sequence withx t’s embedding. (c) repeat until cosine similarity exceedsc. 16

  31. [40]

    (c) repeat until edit distance drops belowl

    EditDistance(distance=l): Iteratively (a) randomly insert, delete, or substitute charac- ters (b) greedily keep the mutation only if it decreases edit distance. (c) repeat until edit distance drops belowl

  32. [41]

    ExactMatch: Worst-case neighbors do not exist under exact matching, since the neighbor- hood ball holds a radius of0. D Societal Impacts This work presents a novel poisoning vulnerability that could be used by a real-world adversary to manipulate the outcome of a high-stakes m...

  33. [2025]

    Accessed: 05/15/2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.