REVIEW 2 major objections 4 minor 39 references
Evasion risk in deployed AI systems is set by the information the interface returns, not by the black-box or white-box label.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 21:00 UTC pith:DSRFSHCD
load-bearing objection Useful procurement-oriented taxonomy of interface signals; risk ordering holds up, Metadata is the weak spot. the 2 major comments →
Signal-based Model Access Risk Analysis for AI System Operations Security
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the security posture of a face recognition system is fundamentally shaped by the level of access an adversary can obtain to the system's outputs, and this access is determined by the deployment interface rather than by the model's internal complexity or the vendor's reputation. It organizes years of adversarial machine learning research into a six-level signal-based taxonomy — None, Metadata, Decision-Only, Score/Rank, Embedding, White-Box — and shows that each level enables qualitatively different attack efficiencies. The sharpest inflection point is between Decision-Only and Score/Rank: continuous score feedback converts the attack problem from discrete bo
What carries the argument
The Signal-based Model Access Risk Taxonomy (SMART): a six-level classification of adversary access — None, Metadata, Decision-Only, Score/Rank, Embedding, White-Box — defined by the information signal a deployed system returns to queries. It serves as the paper's organizing device: for each level it catalogs documented attack methods (transfer attacks, boundary wandering, finite-difference gradient estimation, embedding-space optimization, gradient descent), query budgets, success rates, and procurement scenarios. The taxonomy does the causal work by attributing risk to the interface, not the model or the vendor.
Load-bearing premise
The framework assumes that the information signal a system's interface returns is the dominant and relatively stable determinant of attack feasibility, treating operational defenses such as rate limits, monitoring, liveness checks, and human oversight as add-ons rather than as factors that can change the effective access level; the cited query-budget numbers come from studies that generally assume unthrottled, undefended endpoints.
What would settle it
Run the same face-recognition model behind three interfaces — binary accept/reject, continuous similarity score, and raw embedding — with identical trained weights and realistic rate limits and anomaly monitoring, then measure query budgets to successful evasion. If decision-only and score-based attacks require similar query budgets (same order of magnitude) rather than the predicted 10–100x gap, the taxonomy's risk ordering would collapse.
If this is right
- A commercial API that returns confidence scores is one to two orders of magnitude easier to attack than one that returns only accept/reject, so score-based access should be treated as a security-relevant design choice, not a convenience.
- Procurement decisions implicitly define the maximum plausible access level for external and insider adversaries; open-source base models guarantee white-box access, while commercial cloud APIs typically grant score/rank access to any internet-connected attacker.
- Organizations should prefer binary decision outputs for 1:1 verification unless continuous scores are operationally necessary, and should consider quantizing scores to coarse categories to obstruct gradient estimation.
- Embedding access, common in interoperable SDKs, enables targeted impersonation optimization directly in feature space and allows face reconstruction from templates, turning an embedding data breach into a biometric breach.
- Systems should be evaluated under white-box assumptions even when deployed with restricted access, because edge reverse engineering, insider access, and model leakage routinely elevate access levels after deployment.
Where Pith is reading between the lines
- If the interface signal is the dominant risk factor, the familiar black-box/white-box distinction systematically mis-buckets deployed systems: two 'black-box' services can differ by an order of magnitude in attack cost. A natural follow-up is for regulators and insurers to require vendors to disclose the exact output schema (binary, score, embedding) as a standard security metric.
- The same logic should transfer to other API-exposed models with continuous outputs, such as large language models that return token probabilities or recommendation systems returning relevance scores; each such interface likely has an analogous risk ladder.
- A testable extension: quantify the marginal security gain from suppressing score outputs in a real deployment by measuring the query-budget ratio between decision-only and score interfaces with identical model weights; this would turn the taxonomy's qualitative ordering into a procurement-usable number.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Signal-based Model Access Risk Taxonomy (SMART), a six-level classification of adversarial access to deployed AI systems: None, Metadata, Decision-Only, Score/Rank, Embedding, and White-Box, organized by the information signal returned by the deployment interface rather than classical white-box/gray-box/black-box knowledge categories. It surveys evasion attacks, primarily in face recognition, at each access level, reporting query budgets and success rates from the literature, and maps procurement options (open-source, commercial APIs, on-premise, edge, custom development) to plausible maximum access levels for external and insider threats. The central claim is that the nature and richness of the interface signal—not model accuracy or vendor reputation—fundamentally shapes evasion risk, with the Decision-Only to Score/Rank transition reducing query budgets by one to two orders of magnitude.
Significance. If correct, SMART offers a deployment-oriented framework that could inform procurement decisions, API interface design, and adversarial risk assessment. The paper's main strength is its synthetic organization of a large body of adversarial machine-learning literature into a coherent six-level framework, with a concrete face-recognition application and a useful procurement mapping table (Table 2). The shift from knowledge-based to signal-based threat modeling is a meaningful conceptual contribution, and the framework yields practically important, falsifiable predictions (e.g., query-budget ratios, success-rate escalations). The paper also provides actionable evaluation criteria and sensible baseline recommendations, such as treating white-box resilience as a baseline procurement requirement. However, the central risk ordering is not empirically validated, and, as argued below, it is not robust to the operational mitigations the paper itself recommends, which tempers the strength of the conclusions.
major comments (2)
- [§2.4, §3.1, §3.4, Table 2] The central claim that interface signal fundamentally shapes evasion risk assumes the information content of each access category is fixed. The paper itself contradicts this: §2.4 states that quantized scores ('low/medium/high') would make gradient estimation much harder, and §3.4 recommends rate limiting, lockouts, output control, and monitoring for Score/Rank systems. Yet the quantitative risk ordering in §3.1 (e.g., 20,000+ decision queries vs. 200–500 score queries; >90% success rates) is drawn from studies of generally undefended, unthrottled endpoints. Under the paper's own recommended mitigations, a Score/Rank API with quantization and strict rate limits can degrade toward Decision-Only, while a Decision-Only API with timing side channels may approach Score/Rank. The risk ordering in Fig. 1/Table 1 is therefore not stable, and the procurement guidance 'prefer Decision-Only when po
- [§2.4, §3.1–3.2] Quantitative statements are presented as general facts without sufficient qualification. For instance, §2.4 reports that score-based attacks achieve 'success rates exceeding 90% against commercial services' with 'a few hundred requests,' citing general-purpose black-box attack papers [15,16] rather than face-recognition-specific studies, and provides no variance, dataset dependence, or endpoint conditions. The '20,000+ vs. 200–500' query ratio is cited to [8,31], but no meta-analysis or error bars are given. Since the taxonomy's practical value depends on these numbers to justify risk ordering and procurement advice, the authors should either provide a more systematic meta-analysis with explicit ranges and conditions, or frame the numbers clearly as illustrative order-of-magnitude findings from specific studies, not universal guarantees.
minor comments (4)
- [Throughout] The extracted text contains numerous missing spaces and formatting artifacts (e.g., 'WeaddressthisgapbypresentingSMART', 'growsN2'). The final PDF should be carefully typeset to avoid these issues.
- [§2.5] The notation 'growsN2' should be rendered as 'grows as N²' for clarity.
- [§1] The paper states that previous surveys organize attacks into white-box, gray-box, and black-box categories but does not cite specific surveys for this claim. Adding one or two representative references would strengthen the framing.
- [§2.2] The Metadata category is described as providing 'coarse system knowledge' but the paper correctly notes that the literature on evasion attacks has not directly studied this category. This is a reasonable inference, but the authors should flag it as extrapolation rather than evidence, which they partly do.
Circularity Check
No significant circularity: the taxonomy synthesizes external literature; no fitted parameters, equations, or self-citations.
full rationale
The paper presents a taxonomy of adversarial access levels and maps existing evasion-attack literature onto it. No parameters are fitted, no equations are derived, and no prediction is computed from data within the paper. The central claim that richer output signals lower query budgets is supported by citations to independent empirical studies (e.g., refs [8] and [31] for score-vs-decision query counts). The taxonomy is definitional in the sense that categories are defined by information content, but the risk ordering is not derived from the definitions alone; it is grounded in external attack results. There are no author self-citations (none of the listed references share authors with this manuscript), so neither self-citation nor imported uniqueness arguments appear. The concern that operational mitigations may alter effective access is a robustness/validity caveat, not a circular derivation: the paper does not define access categories in terms of the risk levels it then predicts. Therefore no circular step is identifiable.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Adversarial examples crafted for one face recognition model transfer to other models trained with similar metric-learning objectives, even without queries.
- domain assumption Repeated binary accept/reject queries allow approximation of a model's decision boundary with sufficient queries.
- domain assumption Finite-difference estimation from continuous scores enables query-efficient adversarial optimization.
- domain assumption Face embeddings retain sufficient biometric information that template inversion or face reconstruction is practical.
- domain assumption Query access enables model extraction that improves subsequent evasion attacks.
read the original abstract
Artificial intelligence (AI) systems are now ubiquitous across domains such as security, finance, healthcare, consumer technology, and large-scale cloud services, where they process massive volumes of data and make consequential decisions daily. This widespread adoption has created a broad attack surface through which adversaries can manipulate, evade, extract information from, or otherwise subvert deployed models. Depending on system design and exposure, attackers may have very different forms of access: some observe only final decisions, while others receive confidence scores, intermediate representations, or even full model parameters. While previous surveys typically organize evasion attacks into white-box, gray-box, and black-box categories based on the attacker's knowledge of model internals (architecture, parameters, gradients), this taxonomy often conflates different deployment scenarios that provide vastly different output signals, all labeled as ``black-box'' despite enabling fundamentally different attack strategies. Understanding how evasion attack strategies adapt to the specific information signals returned by deployed systems is critical for organizations making procurement and deployment decisions. To address this gap, we introduce the Signal-based Model Access Risk Taxonomy (SMART), a deployment-oriented framework that classifies attacker access according to the nature and richness of the information signals available from deployed AI systems. Using this taxonomy, we provide a structured overview of evasion attacks across progressively richer levels of information exposure, highlighting how deployment interfaces influence attack capabilities and informing more secure AI deployment and procurement decisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Ieee Access 6, 14410–14430 (2018)
Akhtar, N., Mian, A.: Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access 6, 14410–14430 (2018)
2018
-
[2]
In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security
Biggio, B., Roli, F.: Wild patterns: Ten years after the rise of adversarial machine learning. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. pp. 2154–2156 (2018)
2018
-
[3]
arXiv preprint arXiv:1712.04248 (2017)
Brendel, W., Rauber, J., Bethge, M.: Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv:1712.04248 (2017)
Pith/arXiv arXiv 2017
-
[4]
arXiv preprint arXiv:1712.09665 (2017)
Brown, T.B., Mané, D., Roy, A., Abadi, M., Gilmer, J.: Adversarial patch. arXiv preprint arXiv:1712.09665 (2017)
Pith/arXiv arXiv 2017
-
[5]
arXiv preprint arXiv:1810.00069 (2018)
Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., Mukhopadhyay, D.: Adversarial attacks and de- fences: A survey. arXiv preprint arXiv:1810.00069 (2018)
Pith/arXiv arXiv 2018
-
[6]
In: 2020 ieee symposium on security and privacy (sp)
Chen, J., Jordan, M.I., Wainwright, M.J.: Hopskipjumpattack: A query-efficient decision-based attack. In: 2020 ieee symposium on security and privacy (sp). pp. 1277–1294. IEEE (2020)
2020
-
[7]
In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Chen, J., Gu, Q.: Rays: A ray searching method for hard-label adversarial attack. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1739–1747 (2020)
2020
-
[8]
In: Proceedings of the 10th ACM workshop on artificial intelligence and security
Chen,P.Y.,Zhang,H.,Sharma,Y.,Yi,J.,Hsieh,C.J.:Zoo:Zerothorderoptimizationbasedblack-boxattacks to deep neural networks without training substitute models. In: Proceedings of the 10th ACM workshop on artificial intelligence and security. pp. 15–26 (2017)
2017
-
[9]
In: interna- tional conference on machine learning
Cohen, J., Rosenfeld, E., Kolter, Z.: Certified adversarial robustness via randomized smoothing. In: interna- tional conference on machine learning. pp. 1310–1320. PMLR (2019)
2019
-
[10]
In: 2018 International joint conference on neural networks (IJCNN)
Correia-Silva, J.R., Berriel, R.F., Badue, C., De Souza, A.F., Oliveira-Santos, T.: Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data. In: 2018 International joint conference on neural networks (IJCNN). pp. 1–8. IEEE (2018)
2018
-
[11]
In: 2020 IEEE International Joint Conference on Biometrics (IJCB)
Deb, D., Zhang, J., Jain, A.K.: Advfaces: Adversarial face synthesis. In: 2020 IEEE International Joint Conference on Biometrics (IJCB). pp. 1–10. IEEE (2020)
2020
-
[12]
In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Dong, Y., Su, H., Wu, B., Li, Z., Liu, W., Zhang, T., Zhu, J.: Efficient decision-based black-box adversarial attacks on face recognition. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7714–7722 (2019) Signal-based Model Access Risk Analysis for AI System Operations Security 11
2019
-
[13]
In: Proceedings of the 22nd ACM SIGSAC conference on computer and communications security
Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: Proceedings of the 22nd ACM SIGSAC conference on computer and communications security. pp. 1322–1333 (2015)
2015
-
[14]
arXiv preprint arXiv:1412.6572 (2014)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
Pith/arXiv arXiv 2014
-
[15]
In: International conference on machine learning
Ilyas, A., Engstrom, L., Athalye, A., Lin, J.: Black-box adversarial attacks with limited queries and informa- tion. In: International conference on machine learning. pp. 2137–2146. PMLR (2018)
2018
-
[16]
arXiv preprint arXiv:1807.07978 (2018)
Ilyas, A., Engstrom, L., Madry, A.: Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint arXiv:1807.07978 (2018)
Pith/arXiv arXiv 2018
-
[17]
In: 2020 25th international conference on pattern recognition (ICPR)
Komkov, S., Petiushko, A.: Advhat: Real-world adversarial attack on arcface face id system. In: 2020 25th international conference on pattern recognition (ICPR). pp. 819–826. IEEE (2021)
2020
-
[18]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Li, H., Xu, X., Zhang, X., Yang, S., Li, B.: Qeba: Query-efficient boundary-based blackbox attack. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1221–1230 (2020)
2020
-
[19]
arXiv preprint arXiv:1706.06083 (2017)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
Pith/arXiv arXiv 2017
-
[20]
IEEE transactions on pattern analysis and machine intelligence41(5), 1188–1202 (2018)
Mai, G., Cao, K., Yuen, P.C., Jain, A.K.: On the reconstruction of face images from deep face templates. IEEE transactions on pattern analysis and machine intelligence41(5), 1188–1202 (2018)
2018
-
[21]
In: Explainable AI: interpreting, explaining and visualizing deep learning, pp
Oh, S.J., Schiele, B., Fritz, M.: Towards reverse-engineering black-box neural networks. In: Explainable AI: interpreting, explaining and visualizing deep learning, pp. 121–144. Springer (2019)
2019
-
[22]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Orekondy, T., Schiele, B., Fritz, M.: Knockoff nets: Stealing functionality of black-box models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4954–4963 (2019)
2019
-
[23]
In: Proceedings of the 2017 ACM on Asia conference on computer and communications security
Papernot,N.,McDaniel,P.,Goodfellow,I.,Jha,S.,Celik,Z.B.,Swami,A.:Practicalblack-boxattacksagainst machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security. pp. 506–519 (2017)
2017
-
[24]
In: 2016 IEEE European symposium on security and privacy (EuroS&P)
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European symposium on security and privacy (EuroS&P). pp. 372–387. IEEE (2016)
2016
-
[25]
Engineering6(3), 346–360 (2020)
Ren, K., Zheng, T., Qin, Z., Liu, X.: Adversarial attacks and defenses in deep learning. Engineering6(3), 346–360 (2020)
2020
-
[26]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Shahreza, H.O., George, A., Marcel, S.: Face reconstruction from face embeddings using adapter to a face foundation model. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 5584– 5593 (2025)
2025
-
[27]
IEEE Transactions on Pattern Analysis and Machine Intelligence 45(12), 14248–14265 (2023)
Shahreza, H.O., Marcel, S.: Comprehensive vulnerability evaluation of face recognition systems to template inversion attacks via 3d face reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(12), 14248–14265 (2023)
2023
-
[28]
In: Proceedings of the 2016 acm sigsac conference on computer and communications security
Sharif, M., Bhagavatula, S., Bauer, L., Reiter, M.K.: Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In: Proceedings of the 2016 acm sigsac conference on computer and communications security. pp. 1528–1540 (2016)
2016
-
[29]
In: 25th USENIX security symposium (USENIX Security 16)
Tramèr,F.,Zhang,F.,Juels,A.,Reiter,M.K.,Ristenpart,T.:Stealingmachinelearningmodelsviaprediction {APIs}. In: 25th USENIX security symposium (USENIX Security 16). pp. 601–618 (2016)
2016
-
[30]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Truong, J.B., Maini, P., Walls, R.J., Papernot, N.: Data-free model extraction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4771–4780 (2021)
2021
-
[31]
In: Proceedings of the AAAI conference on artificial intelligence
Tu, C.C., Ting, P., Chen, P.Y., Liu, S., Zhang, H., Yi, J., Hsieh, C.J., Cheng, S.M.: Autozoom: Autoencoder- based zeroth order optimization method for attacking black-box neural networks. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 742–749 (2019)
2019
-
[32]
IEEE Access9, 92735–92756 (2021)
Vakhshiteh, F., Nickabadi, A., Ramachandra, R.: Adversarial attacks against face recognition: A comprehen- sive study. IEEE Access9, 92735–92756 (2021)
2021
-
[33]
In: 2021 16th IEEE international conference on automatic face and gesture recognition (FG 2021)
Wang, H., Wang, S., Jin, Z., Wang, Y., Chen, C., Tistarelli, M.: Similarity-based gray-box adversarial attack against deep face recognition. In: 2021 16th IEEE international conference on automatic face and gesture recognition (FG 2021). pp. 1–8. IEEE (2021)
2021
-
[34]
arXiv preprint arXiv:2504.18015 (2025)
Wang, H., Wang, S., Lu, C.S., Echizen, I.: Diffumi: Training-free universal model inversion via unconditional diffusion for face recognition. arXiv preprint arXiv:2504.18015 (2025)
Pith/arXiv arXiv 2025
-
[35]
arXiv e-prints pp
Wang, H., Wang, S., Lu, C.S., Echizen, I.: Diffusion-driven universal model inversion attack for face recogni- tion. arXiv e-prints pp. arXiv–2504 (2025)
2025
-
[36]
Neurocomputing p
Wang, M., Zhou, J., Li, T., Meng, G., Chen, K.: A survey on physical adversarial attacks against face recognition systems. Neurocomputing p. 132485 (2025)
2025
-
[37]
In: NDSS
Yu, H., Yang, K., Zhang, T., Tsai, Y.Y., Ho, T.Y., Jin, Y.: Cloudleak: Large-scale deep learning models stealing through adversarial examples. In: NDSS. vol. 38, p. 102 (2020)
2020
-
[38]
IEEE Transac- tions on Information Forensics and Security16, 1452–1466 (2020)
Zhong, Y., Deng, W.: Towards transferable adversarial attack against deep face recognition. IEEE Transac- tions on Information Forensics and Security16, 1452–1466 (2020)
2020
-
[39]
In: Proceedings of the 32nd ACM International Conference on Multimedia
Zhou, F., Zhou, Q., Yin, B., Zheng, H., Lu, X., Ma, L., Ling, H.: Rethinking impersonation and dodging at- tacks on face recognition systems. In: Proceedings of the 32nd ACM International Conference on Multimedia. pp. 2487–2496 (2024)
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.