REVIEW 4 major objections 4 minor 34 references
Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The number of adversarial attack iterations reveals whether a sample was in the model's training set.
desk verdict Genuinely new feature, but the paper's own motivation contradicts the mechanism and no sensitivity analysis backs up the claim of reliability; worth a serious referee but not acceptance as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the iteration counter attached to three existing adversarial attacks: SimBA for score-based black-box access, HopSkipJumpAttack for label-only black-box access, and PGD for white-box access. IMIA runs the chosen attack, records the number of iterations until it produces an adversarial example, and applies a threshold to that count. The single feature carries the entire argument: it must separate members from non-members across attack algorithms, model architectures, and datasets without needing training data or shadow models.
What would settle it
A control experiment that matches member and non-member samples on prediction confidence and distance to the decision boundary, or repeats the attacks across many random seeds, would show whether iteration counts carry independent membership signal or merely re-encode sample difficulty and attack randomness.
Extended reading notes
Core claim
The central discovery is the iteration gap: when generating adversarial examples with SimBA, HopSkipJumpAttack, or PGD, member samples consistently require more iterations than non-member samples, even though the paper describes members as lying closer to the decision boundary. IMIA formalizes this into a metric-based attack: run the appropriate adversarial strategy, count the iterations until a successful adversarial example is found, and classify the sample as a member when the count exceeds a threshold. The paper presents this as a universal, lightweight alternative to soft-output metrics, because the same rule works when only hard labels are available, and it argues that the iteration count exposes a privacy leak that confidence and distance features only partially capture.
Load-bearing premise
The attack assumes that the extra iterations members require are caused by membership itself, not by sample difficulty, attack randomness, or the particular hyperparameters chosen for the adversarial algorithm.
Editorial extensions
If this is right
- Membership inference can be conducted with a single scalar (attack iteration count) without training data or shadow models.
- The same attack rule transfers across access levels by swapping the adversarial algorithm: score-based black-box, label-only black-box, and white-box.
- Models that expose only hard labels remain vulnerable to a metric-based membership inference attack, which soft-output baselines cannot cover.
- An adversary can run the attack on a single target sample, making it usable when only one record's membership is at stake.
- The reported white-box gains imply that gradient-based attack trajectories carry membership signal beyond the loss baseline.
Reading between the lines
- If iteration counts separate members only because they proxy confidence or boundary distance, IMIA would be a roundabout re-expression of existing metrics; a direct test would compare members and non-members matched on those quantities.
- The same iteration-count feature may transfer to other domains such as text or tabular data, where gradient-based and decision-based adversarial attacks already exist.
- One testable extension is that defenses which shrink the iteration gap, such as adversarial training or output smoothing, would lower IMIA's AUROC and could serve as privacy defenses.
- The choice of query budget and threshold are themselves attack parameters; future work could treat them as tunable variables that trade attack cost against inference accuracy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IMIA, a membership inference attack that uses the number of iterations required by an adversarial attack (SimBA, HopSkipJumpAttack, or PGD) to generate an adversarial example from a target sample as a membership signal. Experiments on CIFAR10, CIFAR100, and STL10 with four architectures compare IMIA against score-based baselines (softmax response, prediction entropy, modified entropy), a label-only boundary-distance baseline, and the white-box loss baseline. The authors report AUROC and accuracy with standard deviations, claiming that the iteration count is a reliable and universal membership feature across black-box, label-only, and white-box settings. No code is provided.
Significance. If established, the idea of using adversarial-iteration counts as a membership signal is interesting and potentially lightweight, since IMIA avoids shadow models and training data. The experimental coverage is broad: three datasets, four architectures, and three attack settings, with standard deviations reported. However, the central claim of reliability is not yet supported: the motivation contains an internal contradiction, the threshold-selection protocol is missing, the iteration counter is defined by hand-tuned and uncensored attack hyperparameters, and the method underperforms baselines in a nontrivial number of configurations. The contribution is incremental over existing boundary-distance and confidence-based attacks unless these issues are resolved.
major comments (4)
- [Section 3.1] The motivating statement says that 'member samples, being closer to the decision boundary, generally require more iterations to generate adversarial examples than non-member samples.' This is internally contradictory: samples closer to the boundary should require fewer iterations to cross it, and the boundary-distance baseline of Choquette-Choo et al. (reference [12]) treats a larger boundary distance as the member signal. If members are actually farther from the boundary, then IMIA's iteration count is essentially a proxy for boundary distance or confidence; if members are closer, the expected sign of the iteration effect is reversed. The paper must reconcile this statement with the empirical iteration histogram and with the boundary-distance baseline.
- [Tables 1-3] The inference-accuracy results in Tables 1, 2, and 3 depend on the threshold τ in Algorithm 1, but the manuscript never specifies how τ is selected for the reported accuracy numbers. Without a protocol (for example, choosing τ on a separate calibration set or reporting the maximum attainable accuracy on the evaluation set), the accuracy comparisons are not reproducible and may be optimistically biased. Please specify the exact threshold-selection rule used for every table.
- [Appendix A and Section 3.2] The iteration count is defined by attack hyperparameters that are not varied, so the observed signal may be an artifact of the configuration. For SimBA (Table 4), max_iters=300 with an unbounded L∞ bound and random basis order means that samples never flipping within 300 iterations are censored, mixing genuine convergence with budget exhaustion. For HSJA (Table 5), the paper sets num_iterations=100 but does not state which counter is actually measured as the 'number of iterations.' For PGD (Sec. 3.2), 50 steps with α=0.001 and a random start mean the first successful iteration is heavily influenced by initialization and step size. Without a sensitivity analysis over epsilon, max_iters, step size, and random seeds, the reported AUROC gains may not be stable. Provide such an analysis or justify why the ranking is invariant.
- [Tables 1-2] The claim that the iteration count is a 'reliable feature' is weakened by the results themselves. In Table 1, IMIA is below Modified Entropy in AUROC for CIFAR100-VGG (67.36 vs 69.61), STL10-ResNeXt (69.25 vs 70.84), STL10-VGG (61.32 vs 62.68), and CIFAR10-ResNeXt (73.34 vs 73.54), among others; in Table 2, IMIA underperforms the boundary baseline in several comparisons, e.g., CIFAR100-ResNet AUROC 85.95 vs 86.12 and CIFAR10-ResNeXt AUROC 69.77 vs 70.43. The paper should either temper the universality claim or analyze the conditions under which the iteration feature helps versus hurts.
minor comments (4)
- [Section 3.2] In the SimBA optimization formulation, the notation 'd' and 'M' is used without definition, and 'queries ≤ M' is vague; please define these quantities precisely.
- [Figure 1] Figure 1(a) is described as a scatter diagram of boundary distance versus iteration count, but the text does not interpret what the reader should conclude from it; Figure 1(b) uses 2k samples per class, which differs from the 3k samples used in the evaluation sets, so the figure should clarify this discrepancy.
- [Section 4.3] The sentence 'in the white-box setting, We choose' has an erroneous capital 'W' after a comma; please correct this and similar grammatical issues throughout the manuscript.
- [General] No code or public release is mentioned; given the demonstrated hyperparameter sensitivity of the iteration counter, providing code or detailed per-seed results would materially aid reproducibility and verification.
Circularity Check
No circular derivation: IMIA's iteration-count feature is measured from standard adversarial attacks and evaluated with swept thresholds; the central claim does not reduce to its own inputs.
full rationale
The derivation chain is self-contained. In Algorithm 1 the feature is defined as the output of an external adversarial attack: 'τt ← S(fθ, (xt, yt))', and the attack strategies (SimBA, HSJA, PGD) are pre-existing methods whose iteration counts are not defined in terms of membership. The decision rule in Algorithm 1 is a threshold comparison, 'return 1(τt ≥ τ)', and the paper reports AUROC and accuracy over thresholds on balanced member/non-member sets; no parameter is fitted to membership labels in order to construct the iteration-count feature. The Section 3.1 claim that 'member samples, being closer to the decision boundary, generally require more iterations to generate adversarial examples than non-member samples' is an empirical histogram observation, not a definitional consequence of the paper's equations, and the later evaluation sets are separate from the motivating figure. There are no load-bearing self-citations: the cited attack papers and MIA baselines are external works. The closest concern is Appendix A's statement that the SimBA perturbation size was 'set a small value for observing the number of iterations during the adversarial samples’ generation process'; this is hyperparameter selection rather than an equation-level reduction of the predicted quantity to an input. The internally strained motivation (members 'closer to the boundary' yet needing 'more iterations') and the absence of sensitivity analysis are correctness and robustness limitations, not circularity, because no predicted result is identical by construction to a fitted input or to a self-citation chain.
Assumptions & free parameters
free parameters (3)
- SimBA perturbation size epsilon =
0.05
- HSJA number of iterations =
100
- PGD step budget and step size =
50 steps, alpha=0.001, epsilon=3/255
assumptions (2)
- domain assumption The number of adversarial iterations is a stable, membership-correlated quantity under the chosen attack configurations.
- domain assumption Members and non-members are comparable in input difficulty, so iteration differences reflect membership rather than sample hardness or attack randomness.
Cite this review
Pith. "Pith review of Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack." pith.science (2026). https://pith.science/paper/Q7ESIA24
@misc{pith2026250602711,
author = {Pith},
title = {Pith review of: Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q7ESIA24}},
note = {Machine review of arXiv:2506.02711}
}
read the original abstract
Membership inference attack (MIA) has become one of the most widely used and effective methods for evaluating the privacy risks of machine learning models. These attacks aim to determine whether a specific sample is part of the model's training set by analyzing the model's output. While traditional membership inference attacks focus on leveraging the model's posterior output, such as confidence on the target sample, we propose IMIA, a novel attack strategy that utilizes the process of generating adversarial samples to infer membership. We propose to infer the member properties of the target sample using the number of iterations required to generate its adversarial sample. We conduct experiments across multiple models and datasets, and our results demonstrate that the number of iterations for generating an adversarial sample is a reliable feature for membership inference, achieving strong performance both in black-box and white-box attack scenarios. This work provides a new perspective for evaluating model privacy and highlights the potential of adversarial example-based features for privacy leakage assessment.
Figures
Reference graph
Works this paper leans on
-
[12]
Label- only membership inference attacks
Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label- only membership inference attacks. In International conference on machine learning , pages 1964–1974. PMLR, 2021
1964
-
[1]
Superdriverai: Towards design and implementation for end-to-end learning-based autonomous driving
Shunsuke Aoki, Issei Yamamoto, Daiki Shiotsuka, Yuichi Inoue, Kento Tokuhiro, and Keita Miwa. Superdriverai: Towards design and implementation for end-to-end learning-based autonomous driving. In 2023 IEEE V ehicular Networking Conference (VNC), pages 195–198. IEEE, 2023
work page 2023
-
[2]
Scalable membership inference attacks via quantile regression
Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z Wu. Scalable membership inference attacks via quantile regression. Advances in Neural Information Processing Systems, 36, 2024
work page 2024
-
[3]
Towards evaluating the robustness of neural networks
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39–57. Ieee, 2017
2017
-
[4]
The secret sharer: Evaluating and testing unintended memorization in neural networks
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19) , pages 267–284, 2019
work page 2019
-
[5]
Extracting training data from large language models
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633–2650, 2021
2021
-
[6]
Membership inference attacks from first principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022
2022
-
[7]
Quantifying memorization across neural language models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022
arXiv 2022
Show all 34 references
-
[8]
Forecasting stock market crisis events using deep and statistical machine learning techniques
Sotirios P Chatzis, Vassilis Siakoulis, Anastasios Petropoulos, Evangelos Stavroulakis, and Nikos Vlachogiannakis. Forecasting stock market crisis events using deep and statistical machine learning techniques. Expert systems with applications , 112:353–371, 2018
2018
-
[9]
Chameleon: Increasing label-only membership leakage with adaptive poisoning
Harsh Chaudhari, Giorgio Severi, Alina Oprea, and Jonathan Ullman. Chameleon: Increasing label-only membership leakage with adaptive poisoning. arXiv preprint arXiv:2310.03838 , 2023
2023 arXiv
-
[10]
Hopskipjumpattack: A query-efficient decision-based attack
Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattack: A query-efficient decision-based attack. In 2020 ieee symposium on security and privacy (sp) , pages 1277–1294. IEEE, 2020
2020
-
[11]
Amplifying membership exposure via data poisoning
Yufei Chen, Chao Shen, Yun Shen, Cong Wang, and Yang Zhang. Amplifying membership exposure via data poisoning. Advances in Neural Information Processing Systems , 35:29830– 29844, 2022
2022
-
[13]
Privacy side channels in machine learning systems
Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramèr. Privacy side channels in machine learning systems. In 33rd USENIX Security Symposium (USENIX Security 24) , pages 6861–6848, 2024. 10
2024
-
[14]
Leveraging adversarial examples to quantify membership information leakage
Ganesh Del Grosso, Hamid Jalalzai, Georg Pichler, Catuscia Palamidessi, and Pablo Piantanida. Leveraging adversarial examples to quantify membership information leakage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10399–10409, 2022
2022
-
[15]
Risk assessment for hospital readmissions: Insights from machine learning algorithms
Rohit R Dixit. Risk assessment for hospital readmissions: Insights from machine learning algorithms. Sage Science Review of Applied Machine Learning , 4(2):1–15, 2021
2021
-
[16]
Explaining and harnessing adversar- ial examples
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572, 2014
2014 arXiv
-
[17]
Simple black-box adversarial attacks
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International conference on machine learning , pages 2484–2493. PMLR, 2019
2019
-
[18]
Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017
Aleksander Madry. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017
2017 arXiv
-
[19]
Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP) , pages 739–753. IEEE, 2019
2019
-
[20]
Scalable extraction of training data from (production) language models
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ip- polito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311...
2023 arXiv
-
[21]
Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon
USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir SV , Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, et al. Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon. arXiv preprint a...
2024 arXiv
-
[22]
Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018
2018 arXiv
-
[23]
Machine learning as an early warning system to predict financial crisis
Aristeidis Samitas, Elias Kampouris, and Dimitris Kenourgios. Machine learning as an early warning system to predict financial crisis. International Review of Financial Analysis , 71: 101507, 2020
2020
-
[24]
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017
2017
-
[25]
Systematic evaluation of privacy risks of machine learning models
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2615–2632, 2021
2021
-
[26]
Privacy risks of securing machine learning models against adversarial examples
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 241–257, 2019
2019
-
[27]
Understanding practical membership privacy of deep learning
Marlon Tobaben, Gauri Pradhan, Yuan He, Joonas Jälkö, and Antti Honkela. Understanding practical membership privacy of deep learning. arXiv preprint arXiv:2402.06674, 2024
2024
-
[28]
The space of transferable adversarial examples
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03453, 2017
2017 arXiv
-
[29]
Truth serum: Poisoning machine learning models to reveal their se- crets
Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their se- crets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security...
2022
-
[30]
On the importance of difficulty calibration in membership inference attacks
Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440, 2021. 11
2021 arXiv
-
[31]
You only query once: An efficient label-only membership inference attack
Yutong Wu, Han Qiu, Shangwei Guo, Jiwei Li, and Tianwei Zhang. You only query once: An efficient label-only membership inference attack. In The Twelfth International Conference on Learning Representations
-
[32]
Privacy risk in machine learning: Analyzing the connection to overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE, 2018
2018
-
[33]
Machine learning-based vehicle intention trajectory recognition and prediction for autonomous driving
Hanyi Yu, Shuning Huo, Mengran Zhu, Yulu Gong, and Yafei Xiang. Machine learning-based vehicle intention trajectory recognition and prediction for autonomous driving. In 2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE) , pages 771–775. ...
2024
-
[34]
Low-cost high-power membership inference attacks
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03262, 2023. 12 A Implementation details for each adversarial strategy In Table 4, we show the implementation details for the Simple Black-box Attack (...
2023 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.