{"id":"c9bd7905-09cf-4df0-af97-88467cbe7c9c","arxiv_id":"2504.21420","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A fixed, pre-optimized set of transferable adversarial face images can estimate the robustness of unseen face recognition systems about 200 times faster than white-box attack evaluation.","lead":"The paper builds a fixed set of tricky face images designed to fool face recognition systems, then uses the set to score how robust any new face recognition system is without running expensive attacks. If the benchmark is trustworthy, robustness checks that currently take hours could take minutes, helping developers test models before deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"System-agnostic claim is under-supported: held-out validation uses only three systems from a single random split, with no confidence intervals; the paper's own ablation shows the suite can overfit (held-out corr -0.83 without regularization).","rationale":"ROBFACE is a practical and well-executed benchmark: the code and data are released, the ablation in Figure 6 is honest, and the adaptive-attack experiment in Section IV.C addresses an obvious gaming concern. The central claim, however, is stronger than the evidence. The paper explicitly frames ROBFACE as 'system-agnostic' and 'consistent with existing approaches,' which requires that the pre-optimized suite rank unseen systems correctly. The only direct evidence is RQ2 with a single split and three held-out systems. Since the optimization objective (Eq. 12) fits the suite to PGD robust accuracy on the tuning systems, the in-sample correlations in RQ1 are expected and do not validate the method. The load-bearing assumption is therefore that adversarial patterns found on five tuning systems (plus generator System-II) transfer to all face recognition systems. The paper cites transferability literature but does not establish the breadth of the phenomenon for this task. The ablation showing -0.83 held-out correlation without regularization demonstrates that the selected subset is not intrinsically transferable; it is regularized into transferability on one split. With n = 3, the confidence interval on r is very wide, so the claim is not statistically grounded. The low CLEVER correlation also qualifies the 'consistent with formal analysis' part of the claim. These are addressable concerns, not fatal flaws; the method may be useful as an efficient screening tool. The appropriate verdict remains CONDITIONAL, conditional on stronger held-out validation. My read therefore does not change the reader's verdict.","tokens_in":23907,"tokens_out":7063,"duration_ms":70765,"concrete_test":"Perform a pre-registered replication of RQ2 with 50 random tuning/testing splits over an expanded pool of at least 20 face recognition systems (including the existing nine plus models trained on different data and, where feasible, black-box commercial APIs). For each perturbation gamma in Table III, compute the held-out-only Pearson correlation between ROBFACE-01 robustness scores and independently computed PGD robust accuracy (or a query-based attack success rate for black-box systems). Report the median and 95% bootstrap confidence interval across splits. Pre-register a pass criterion, e.g., median held-out correlation >= 0.8 and lower 95% CI > 0 across all eight perturbations. If the criterion is not met, the system-agnostic generalization claim in Section I is not supported by the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section I) is that ROBFACE-01 is the first search-free, system-agnostic robustness test suite and that its estimates are consistent with PGD and CLEVER on unseen systems. That claim rests on transferability of adversarial samples. RQ2 tests this with one random split into a tuning group of five and a testing group of three systems from Table I. The text is internally inconsistent about the split (Section IV.A says a tuning group is sampled from all nine systems and 'the remaining systems' are tested, which would be four; RQ2 says 'the remaining three systems'), and a three-system held-out set cannot support a population-level system-agnostic claim. No confidence intervals or significance tests are reported, and the reported testing-group correlations 0.95 +/- 0.05 are computed on n = 3 systems, where Pearson r is extremely noisy. The fragility is visible in the paper's own ablation (Section IV.B, Figure 6): with regularization removed, tuning correlation is 1.0 but testing correlation drops to -0.83, showing the optimization can strongly overfit and that one favorable regularized split is weak evidence. Furthermore, the suite is explicitly optimized to maximize correlation with PGD robust accuracy on the tuning systems (Problem 12, Section III.B), so consistency with PGD on held-out systems is an unproven transferability property, not a guaranteed one. The paper reports only 0.48 and 0.63 correlation with CLEVER (Section IV.B), which undermines the sub-claim of consistency with formal analysis. Transferability is cited from prior work but not established for the population of face recognition systems, and the held-out systems are all standard academic models, not a diverse population.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ROBFACE, a pre-optimized test suite of transferable adversarial face images intended to estimate the robustness of face recognition systems without per-system iterative search. The suite covers eight perturbation types; each category is generated by a gradient-based algorithm (ROBFACE-GEN) and then subsampled by a genetic algorithm whose objective maximizes the Pearson correlation with PGD-based robust accuracy on a tuning group of face recognition systems. The authors claim that the resulting fixed suite is system-agnostic, consistent with empirical and formal references, and more than 200 times faster. Experiments on nine face recognition systems report correlations of 0.90 to 0.99, an ablation, an efficiency comparison, and a randomization scheme against adaptive overfitting.","tokens_in":24199,"tokens_out":7016,"duration_ms":76599,"significance":"The artifact is valuable: ROBFACE-01, the code, and raw results are publicly released; the evaluation procedure is simple to apply; and the speedup is substantial. The perturbation coverage beyond p-norm is a genuine practical advantage. However, the central generalization claim currently rests on weak evidence: the tuning-group correlations are partly constructed by the optimization objective, and the held-out validation uses a single split with only three systems. With stronger validation, such as multiple splits and confidence intervals, the contribution would be a solid engineering benchmark; as written, the headline claims outrun the evidence.","major_comments":[{"comment":"The optimization objective as written is inconsistent with the prose. Equation (12) minimizes phi1, but phi1 is defined as the Pearson correlation coefficient; minimizing it drives the correlation toward -1, whereas the text states the objective 'maximises the Pearson correlation coefficient'. Similarly, phi3 = Mean - lambda2*Std is added with a positive lambda1, but the text says the regularization should maximize the averaged value and the standard deviation, which would require subtracting the term. This is a load-bearing correctness issue because the entire construction of the suite depends on this objective. Please correct the sign (likely minimize -phi1 - lambda1*phi3, or define phi1 as negative correlation), or clarify the intended formulation.","section":"Section III.B, Eq. (12)"},{"comment":"The RQ1 correlations of 0.95 to 0.99 are computed on the tuning systems after the subset has been optimized to maximize correlation with the reference on exactly those systems; these numbers should be presented as fitting quality, not as independent evidence of consistency. The independence of the claim rests entirely on RQ2, but RQ2 uses one random split with only three held-out systems, and no confidence intervals or significance tests are reported. With n=3, a Pearson correlation is extremely noisy, and the paper's own ablation (Figure 6) shows that without regularization the tuning correlation is 1.0 while the testing correlation drops to -0.83, demonstrating that the optimization can strongly overfit. A single favorable regularized split is not sufficient to support a population-level 'system-agnostic' claim. Please provide repeated random splits (or leave-one-out evaluation), report confidence intervals, and consider tempering the generalization claim to the systems studied.","section":"Section IV.B, RQ1 and RQ2"},{"comment":"The number of held-out systems is internally inconsistent. Section IV.A states that a tuning group of five systems is randomly selected and 'the test group with the remaining systems' is used in RQ2. Section IV.B (RQ2) then says 'a testing group consists of the remaining three systems'. Since Table I lists nine systems, five tuning systems should leave four testing systems. Please state the exact composition of the tuning and testing groups and resolve this discrepancy, because the reported testing-group correlation depends on which three (or four) systems are held out.","section":"Section IV.A and Section IV.B"},{"comment":"The computation of the PGD reference for non-p-norm perturbations is not specified. Table III marks Reference-1 as applicable to glasses, mask, illumination, radial, age, and pose perturbations, but the PGD algorithm described in Section II is defined for norm-constrained perturbations. The necessary extension of PGD to these transformation-based perturbations is not given. Because all correlations are calibrated against this reference, the missing description threatens both reproducibility and the validity of the robustness values being predicted. Please specify the reference algorithm used for each non-p-norm perturbation, or state where this is defined.","section":"Section IV.A, Table III"},{"comment":"The reported CLEVER correlations of 0.48 and 0.63 are far below the 0.90 to 0.99 range reported for Reference-1, yet the paper's central claim includes consistency with 'formal analysis' (Section I). The text does not state over which systems, which perturbation types, or which tuning/testing groups these CLEVER correlations are computed, nor does it explain why they are not considered to contradict the consistency claim. Please clarify these numbers and, if appropriate, limit the consistency claim to the empirical PGD reference or qualify it accordingly.","section":"Section IV.B, RQ1"}],"minor_comments":[{"comment":"The phrase 'subjective to adaptive attacks' should be 'subject to adaptive attacks'.","section":"Section IV.C"},{"comment":"The contribution statement says 'more than 200 times faster' while RQ3 says 'around 200 times'; please report a single aggregate speedup ratio computed from Table IV so the claim is reproducible.","section":"Section I and RQ3"},{"comment":"The sentence 'the correlation with each reference was higher than the within-reference correlation' is never defined or supported; please define the within-reference correlation and report it, or remove the sentence.","section":"Section IV.B, RQ1"},{"comment":"The caption says testing systems are underlined, but the group assignment is not verifiable from the text given the 5/3 split ambiguity; please make the tuning/testing membership explicit, for example by listing the system indices in a table.","section":"Figure 5"},{"comment":"The phrase 'Here, n = 6.254' appears to be a typo; the input dimension should be an integer (75,264 as stated later), so please correct or remove the value.","section":"Example 2, Section III.A"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a useful and potentially reusable engineering artifact, and the public release of the test suite and code is a strength. My main concern is that the 'system-agnostic' and 'consistent with formal analysis' claims are not yet supported by the evidence: the held-out validation is underpowered and internally inconsistent, and the optimization objective contains a sign error that must be fixed. If the authors can add multi-split validation with confidence intervals and correct the identified issues, the paper could be acceptable; without that, the claims should be substantially downgraded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful artifact paper whose headline claim outruns its evidence. The suite and the construction method are real; the system-agnostic generalization claim is not yet supported by the experiments.\n\nWhat is actually new: the paper generates candidate adversarial face pairs under eight perturbation families, then solves a discrete optimization to pick a subset whose pass rates correlate with PGD robust accuracy on a tuning group of face recognition systems. That gives a precomputed test suite that runs black-box at inference time. The speedup is real; the methodology is described in enough detail to reproduce; code and data are public; and Figure 6 honestly shows the overfitting failure mode (tuning corr 1.0, testing corr -0.83 without regularization). I don't see a problem with the math; the formulation is clear and the genetic-algorithm details are not hidden.\n\nThe soft spot is validation, not construction. RQ1 correlations of 0.95-0.99 are in-sample and basically the optimizer's objective; I don't count them as evidence either way. The held-out evidence in RQ2 is one random split into five tuning and three held-out systems (the text also says four), with no repeated splits, no confidence intervals, and no significance tests. A Pearson r on three points is too noisy to support 'system-agnostic.' The paper's own ablation shows that one favorable regularized split is fragile. The reported correlations with CLEVER, 0.48 and 0.63, are low, so the abstract's claim of consistency with formal analysis is overstated. Transferability is cited from prior work but not established across a diverse population; all systems are standard academic models.\n\nMinor issues: the text is inconsistent on the held-out count, and whether the generator system g† is excluded from tuning/testing should be stated explicitly. 'First search-free and system-agnostic' is too strong; the optimization-to-correlation construction is the genuine novelty. The citation pattern is fine: the paper acknowledges CIFAR-10C and the transferability literature.\n\nWho should read this: robustness-evaluation researchers and practitioners who want a quick black-box estimate for face recognition. It deserves a serious referee. A revision should add more held-out systems or repeated splits with intervals, clean up the train/test separation, and soften the generalization language to match the evidence.","headline":"A useful and reproducible test suite for face recognition robustness, but the system-agnostic generalization claim rests on one thin split and needs stronger validation.","tokens_in":24793,"tokens_out":5175,"would_cite":true,"duration_ms":57124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ROBFACE estimates the robustness of unseen face recognition systems from one fixed set of transferable adversarial face images, matching PGD-based rankings at about 200x lower evaluation time.","keywords":["face recognition","robustness evaluation","adversarial examples","transferability","test suite","system-agnostic testing","perturbation robustness","black-box evaluation"],"falsifier":"Run ROBFACE-01 on a face recognition system that is architecturally distant from the nine systems studied, for example a vision-transformer-based encoder not seen during tuning, and compare its predicted robustness ranking to PGD robust accuracy on the same eight perturbations. If the Pearson or Spearman correlation falls substantially below 0.90, or the ranking reverses on several perturbation dimensions, the system-agnostic generality claim is refuted.","tokens_in":23647,"feed_emoji":"⚡","tokens_out":8675,"duration_ms":84416,"temperature":0.7,"pith_summary":"This paper tries to establish that the robustness of a face recognition system can be estimated without running any attack at evaluation time: by simply measuring how many pre-crafted, transferable adversarial face images the system classifies correctly. It introduces ROBFACE, a test suite in which each perturbation type has an optimised subset of adversarial samples, and the fraction of samples that flip a system's prediction becomes the robustness score. Across nine face recognition systems and eight perturbation types, these scores correlate with PGD-based robust accuracy (Pearson 0.90 to 0.99) and, on p-norm perturbations, with CLEVER, while taking about 200 times less runtime than either reference. If this holds, developers gain a fast, black-box, architecture-agnostic way to rank candidate systems before deployment and to re-check robustness after every model update. The load-bearing assumption is that adversarial perturbations transfer across face recognition systems.","feed_headline":"A pre-made test suite ranks face recognition resistance 200x faster","feed_subtitle":"Transferable adversarial images match PGD and CLEVER rankings across eight perturbation types, in minutes.","key_machinery":"The central object is the test suite $\\mathcal{T}=\\{T_\\gamma\\}$, one subset $T_\\gamma$ per perturbation scheme $(d,\\epsilon)$, where each element is an ordered triple $(x,x',y)$ containing the original input, a carefully perturbed input, and the correct label. The suite is built in two phases: ROBFACE-GEN generates, by gradient descent starting from random perturbation sizes, a large pool of candidates that span a range of prediction losses; then a constrained binary optimisation selects a small subset whose per-system misclassification counts maximise the Pearson correlation with known PGD robustness on a tuning group, regularised to keep the selected examples strong and spread out. At evaluation time, the robustness estimate is simply the fraction of triples in $T_\\gamma$ on which the system predicts the same label as $y$.","core_discovery":"The central discovery is that robustness evaluation for face recognition can be reduced to a single forward pass over a pre-optimised test suite. The paper constructs ROBFACE-01, covering eight perturbation spaces, by generating candidate adversarial examples with a gradient-descent process and then solving a binary subset-selection problem that maximises the Pearson correlation between the suite's per-system score and PGD-based robust accuracy on a tuning group. Across nine face recognition systems, with five randomly selected for tuning and the rest held out, the resulting scores correlate with PGD robust accuracy between 0.90 and 0.99 per perturbation dimension, and runtime drops from thousands of seconds to tens of seconds. ROBFACE is presented as the first search-free, system-agnostic robustness test suite, applicable also to perturbations such as glasses, masks, illumination, age, pose, and radial distortion that formal Lipschitz-based methods cannot handle.","pith_inferences":["Editorial inference: the transferability premise suggests that the suite's validity is population-dependent, so a prudent deployment would periodically regenerate the suite against a fresh random sample of contemporary systems rather than relying on one fixed instance forever.","Editorial inference: because ROBFACE correlates strongly with PGD but less strongly with CLEVER on p-norm perturbations, practitioners should treat ROBFACE scores as an empirical robustness proxy rather than a formal certificate; worst-case guarantees would still require verification.","Editorial inference: a natural next experiment is to extend the validation well beyond nine systems, including architecturally distant encoders such as vision-transformer-based models, to test whether the reported 0.90 to 0.99 correlations persist when the face-recognition architecture distribution shifts."],"forward_implications":["A developer can re-evaluate robustness after every model update in minutes by running the model once on ROBFACE-01, without configuring an attack or accessing gradients.","Because the suite is system-agnostic, the same pre-optimised samples can rank robustness across heterogeneous face recognition systems, making fair comparison between versions and vendors straightforward.","The suite covers eight perturbation dimensions, with four of them outside the reach of the formal reference method, so robustness under natural transformations such as glasses, masks, illumination, age, pose, and radial distortion can be measured where theoretical bounds are unavailable.","Rotation of a secret seed lets an evaluator generate alternative suite instances, so a developer who overfits to the public ROBFACE-01 sees its score collapse from 0.98 to 0.52 on a re-seeded suite.","The same subset-optimisation methodology can be applied to build robustness test suites for tasks other than face recognition, since the procedure only requires pair inputs, a binary label, and a reference attack."],"supporting_citations":[{"why":"Supplies the PGD attack whose robust accuracy is the main reference value that ROBFACE's scores are optimised to correlate with and compared against.","marker":"[9]"},{"why":"Supplies CLEVER, the formal Lipschitz-based reference used as Reference-2 on p-norm perturbations.","marker":"[20]"},{"why":"Establishes that adversarial perturbations transfer across models, providing the theoretical basis for ROBFACE's system-agnostic premise.","marker":"[17]"},{"why":"Documents transferability of adversarial samples between machine learning systems, supporting black-box reuse of the pre-optimised suite.","marker":"[19]"},{"why":"Provides the glasses and accessories perturbation and the physical adversarial attack scenario used as one of the eight perturbation dimensions.","marker":"[2]"},{"why":"Supplies the LFW face-pair dataset used as part of the original input pool for generating candidate adversarial test cases.","marker":"[51]"},{"why":"Defines the radial distortion perturbation space included in ROBFACE-01.","marker":"[16]"}],"fun_headline_variants":["Pre-made adversarial suite tests face recognition robustness in minutes","Test suite ranks face recognition robustness 200x faster than PGD","One-pass adversarial test suite matches formal robustness rankings","First system-agnostic robustness suite for face recognition","Test suite cuts face recognition robustness checks from hours to minutes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The suite's estimates are trustworthy only if adversarial examples that fool a handful of tuning systems also, on average, fool unseen face recognition systems with different architectures, training sets, and loss functions; the paper assumes this transferability rather than proving it for the whole population of face recognition systems.","fun_headline_variants_meta":{"raw":{"variants":["Pre-made adversarial suite tests face recognition robustness in minutes","Test suite ranks face recognition robustness 200x faster than PGD","One-pass adversarial test suite matches formal robustness rankings","First system-agnostic robustness suite for face recognition","Test suite cuts face recognition robustness checks from hours to minutes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":2995,"prompt_tokens":923,"completion_tokens":2072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":539,"tokens_out":2072,"duration_ms":15124,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:03:52.421574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ROBFACE-01 on a face recognition system that is architecturally distant from the nine systems studied, for example a vision-transformer-based encoder not seen during tuning, and compare its predicted robustness ranking to PGD robust accuracy on the same eight perturbations. If the Pearson or Spearman correlation falls substantially below 0.90, or the ranking reverses on several perturbation dimensions, the system-agnostic generality claim is refuted.","supporting_citations":[{"cited_title":"Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments,","cited_arxiv_id":null,"evidence_quote":"Supplies the LFW face-pair dataset used as part of the original input pool for generating candidate adversarial test cases."}],"review_version":1}