{"id":"b37413dc-f9cf-404f-9164-e9b855847f29","arxiv_id":"2411.15210","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PMA, a probability-margin loss attack, consistently outperforms existing individual white-box attacks, and a one-million-image evaluation shows much lower robust accuracy than small-scale tests.","lead":"This paper introduces Probability Margin Attack (PMA), a white-box adversarial attack that defines its loss in probability space instead of logit space, and reports small but consistent gains over existing individual attacks on RobustBench models. It also builds a one-million-image evaluation set and reports a much lower robust accuracy than the standard ImageNet test suggests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Million-scale robustness gap is confounded by dataset shift: CC1M differs from ImageNet-1k in both scale and distribution, and pseudo-label relative robustness conflates accuracy with robustness, so the headline 59%-to-19% drop does not isolate scale.","rationale":"The paper makes two separable claims: (1) PMA is a stronger individual attack and PMA+ is a faster effective ensemble; (2) million-scale evaluation reveals a large robustness gap. Claim 1 is supported by a correct gradient derivation (Eq. 3: the PM-loss gradient is a weighted sum of untargeted and targeted CE gradients) and consistent, if small, improvements across 29 RobustBench models. The main tables do contain some apparent inconsistencies in the 'diff' columns (e.g., several rows do not match the row minimum), which should be corrected, but the direction of the attack results is consistent. Claim 2 is the paper's distinctive novelty and is under-specified: CC1M is not a larger sample of ImageNet; it is a filtered web-captioned dataset, so the 40-point relative-robustness drop confounds dataset shift with scale. The pseudo-label relative-robustness metric further complicates interpretation because it depends on the model's clean accuracy, which is not reported for CC1M. The reader's weakest assumption identified this same confound, and I agree. The fix is straightforward—a 50k CC1M subset control and clean-accuracy reporting—so the conditional verdict stands rather than a rejection.","tokens_in":21658,"tokens_out":9704,"duration_ms":92646,"concrete_test":"Run PMA and PGDce under the same relative-robustness protocol on three sets: (a) a random 50k subset of CC1M, (b) the full 1M CC1M, and (c) the standard 50k ImageNet-1k validation set. Also report clean top-1 accuracy on each set. If 50k CC1M already shows ~19% relative robustness, the gap is due to distribution shift, not scale; if 50k CC1M is near 59% and only the 1M set drops to 19%, the scale effect is real. Additionally, condition the metric on clean-correct predictions to separate accuracy from adversarial robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 compares relative robustness on ImageNet-1k (50k, in-distribution) with CC1M (1M, out-of-distribution web images from CC3M). Scale and distribution change simultaneously, so the drop from ~59% to below 19% cannot be attributed to evaluation scale. This is not merely a missing control: expected robust accuracy is a population average, so drawing a larger i.i.d. sample from the same distribution would not move it by 40 points. The observed gap must come from CC1M's different image distribution, the pseudo-label protocol, or the filtering procedure. The paper reports no clean top-1 accuracy on CC1M and no numeric LID/MAD thresholds, and the relative-robustness metric counts any change in the model's own top-1 prediction as attack success. If clean accuracy on CC1M is low, the metric drops for reasons unrelated to adversarial robustness. Without a same-size subset control and clean-accuracy reporting, the 'first million-scale evaluation reveals a significant robustness gap' claim is not established. The PMA/PMA+ RobustBench results are independent of this and appear sound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Probability Margin Attack (PMA), a white-box individual attack that uses a probability margin loss within a two-stage margin-decomposition pipeline. It claims that PMA outperforms existing individual attacks on 29 RobustBench models, and that the PMA+APGDT ensemble (PMA+) outperforms AutoAttack while taking only 25% of its runtime. The paper also constructs CC1M, a filtered 1M-image subset of CC3M, and evaluates a newly defined 'relative robustness' metric on ImageNet-robust models, reporting that relative robustness drops from above 59% on ImageNet-1k to below 19% on CC1M, which the authors interpret as a small-scale versus large-scale robustness gap.","tokens_in":21892,"tokens_out":16860,"duration_ms":149811,"significance":"The PMA loss is a simple and well-motivated modification of existing margin-based losses, and the derivation of its gradient as a weighted combination of targeted and untargeted cross-entropy gradients is correct. The empirical pattern across 29 models is consistent: PMA lowers robust accuracy compared with the best baseline in nearly every row, and the efficiency gain over AutoAttack is substantial. If the attack results hold, PMA and PMA+ are practically useful tools for robustness evaluation. The million-scale dataset and the relative robustness metric are steps toward larger-scale evaluation, but the current experimental design does not isolate the effect of scale from distribution shift, and the metric itself is not calibrated without clean-accuracy reporting.","major_comments":[{"comment":"Section 4.4 compares relative robustness on the ImageNet-1k test set (50k, in-distribution) with CC1M (1M, out-of-distribution web images). Because robust accuracy is a population average, a larger i.i.d. sample from the same distribution would not move the estimate by roughly 40 percentage points; the observed drop from above 59% to below 19% must therefore be driven by the change in image distribution, the pseudo-label protocol, or the LID/MAD filtering. The paper does not provide a same-size (e.g., 50k) subset control drawn from CC1M, does not report clean top-1 accuracy on CC1M, and does not give numeric LID/MAD thresholds. As a result, the claim of a small-scale versus large-scale robustness gap is not established. Please add a subset control and clean-accuracy reporting, or reframe the claim as an out-of-distribution robustness gap.","section":"§4.4, Figure 1"},{"comment":"The 'relative robustness' metric counts an attack as successful whenever the model's prediction on the adversarial image differs from its own prediction on the clean image, regardless of whether the clean prediction was correct. An attack that flips an incorrect prediction to the correct class is therefore scored as a success, and the metric is strongly coupled to the model's clean accuracy on the test distribution. Without reporting clean top-1 accuracy on CC1M, the stated 3.56% to 11.86% gap between PMA and PGDce on CC1M cannot be interpreted as an adversarial-robustness effect; it may partly reflect lower confidence on out-of-distribution inputs. Please report clean accuracy on CC1M and, if feasible, standard robust accuracy on a labeled subset.","section":"§4.4, Relative Robustness definition"},{"comment":"There are inconsistencies between the main text and the appendix. For Swin-B (ImageNet), Table 3 reports PMA robust accuracy of 54.41%, while Table 10 reports the SGD+sign PMA value as 54.57%. For RWRN-70-16[27] (CIFAR-10), Table 2 reports PGDpm as 71.76%, while Table 10 reports the SGD+sign PGDpm as 71.16%. These discrepancies affect the exact claims about the magnitude of improvement and must be reconciled or explained (e.g., different seeds, perturbation budgets, or evaluation subsets).","section":"Tables 3 and 10"}],"minor_comments":[{"comment":"The notation 'zπi represents the i-th largest logit value (except zy)' is ambiguous; please define π explicitly (e.g., π1 is the index of the maximum non-true-class logit) in the text.","section":"Section 3.1, Eq. (2)"},{"comment":"The cosine step-size schedule appears only in Algorithm 1; the main text in Section 3.2 should mention that PMA uses a cosine-decayed step size to make the method self-contained.","section":"Algorithm 1"},{"comment":"Tables 4 and 5 are visually dense, with numbers and signs run together; reformatting them into clearer sub-columns would substantially improve readability.","section":"Tables 4 and 5"},{"comment":"The statement 'PMA is the strongest individual attack in the current literature' is stronger than what the experiments show, given the finite set of baselines (though AAA and ACG are covered in the appendix); consider phrasing this as 'strongest among the evaluated individual attacks.'","section":"Section 4.2"},{"comment":"Several typos appear in the abstract and introduction, including 'defeines' and 'aslo'; a copy-edit is recommended.","section":"General"},{"comment":"The small robustness differences (often below 0.1 percentage points) are reported without error bars or statistical tests; a paired test or confidence intervals would help support the claim of consistent improvement.","section":"Tables 2, 3, and 6"}],"recommendation":"major_revision","confidential_remarks":"The PMA attack contribution is solid and the empirical pattern is consistent across many models; the main risk is the million-scale evaluation section, whose design cannot separate scale from distribution shift. If the authors add the missing controls (a same-size subset of CC1M, clean accuracy on CC1M, and numeric thresholds) or reframe the contribution as an out-of-distribution robustness study, the paper is likely acceptable. The inconsistencies between Tables 3 and 10 should be fixed before publication. The hyperparameter selection on the evaluation models is a minor concern, but the ablations suggest it is not decisive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PMA is a real but modest improvement to individual white-box attacks, and the PMA+APGDT ensemble is a practical speedup over AutoAttack with matching or better robustness numbers. The novel piece is the probability margin loss; the gradient identity in Eq. (3) connecting it to a weighted sum of targeted and untargeted CE is correct and useful. The empirical sweep over 29 RobustBench models is broad and internally consistent (with one exception I'll mention), and the consistent ~0.03-0.55% improvement over the Margin Decomposition attack is credible. The K' and n ablations show the choice is not brittle, which helps.\n\nNow the soft spots, in order of seriousness. The million-scale evaluation on CC1M does not isolate scale. The comparison is between ImageNet-1k (50k, in-distribution) and CC1M (1M, web images from CC3M, out-of-distribution). Scale and distribution change at the same time. Relative robustness is computed against the model's own clean predictions, which conflates clean accuracy with robustness; if clean accuracy on CC1M is low, the metric drops regardless of attack. The paper doesn't report clean accuracy on CC1M or the numeric LID/MAD thresholds used for filtering. A same-size ImageNet subset control and a clean-accuracy number would make the claim testable. Without those, the 'scale reveals a gap' headline is not established. The stress-test note is right: a larger i.i.d. sample from the same distribution would not move robust accuracy by 40 points, so the gap is almost certainly distribution shift, not scale.\n\nSmaller issues: no error bars or seed variation; a Swin-B row differs between Table 3 (54.41) and Appendix Table 10 (54.57) under the same PMA/SGD+sign setting, which should be corrected. CC1M and code are not released, which limits reproducibility. These are addressable.\n\nWho is this for: people who build or benchmark adversarial attacks. The attack part is worth taking seriously. The million-scale section is not ready as-is.\n\nRecommendation: send to peer review, but expect major revision on the CC1M claim. The attack contribution alone justifies referee time.","headline":"PMA is a real but modest attack improvement; the million-scale robustness claim is confounded by dataset shift and needs controls.","tokens_in":22427,"tokens_out":2058,"would_cite":true,"duration_ms":19921,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Probability-space margin attack beats every individual attack tested.","keywords":["adversarial robustness","white-box attack","probability margin loss","robustness evaluation","million-scale evaluation","relative robustness","CIFAR-10","ImageNet"],"falsifier":"Take a random 50,000-image subset of CC1M and a 50,000-image subset of the ImageNet test set, run the same PMA and PGD-CE attacks with the same pseudo-label relative-robustness metric on both; if the relative robustness on the CC1M subset is close to the ImageNet subset or the claimed >59%-to-<19% gap disappears, the scale effect has not been isolated from the dataset shift.","tokens_in":21438,"feed_emoji":"🎯","tokens_out":8462,"duration_ms":74296,"temperature":0.7,"pith_summary":"Probability Margin Attack (PMA) tries to establish that measuring an adversarial margin in probability space, rather than logits space, makes an individual white-box attack reliably stronger. Across 29 leading adversarially trained models on CIFAR-10, CIFAR-100, and ImageNet, PMA reports robust accuracy 0.03% to 0.55% lower than the best existing individual attack, and when paired with one targeted attack it outperforms the standard four-method ensemble at 25% of the runtime. The paper also builds a million-image evaluation set, CC1M, and reports that the relative robustness of top ImageNet models collapses from above 59% on the standard test set to below 19% at million scale. If these results hold, standard benchmark numbers overstate real-world robustness, and cheap individual attacks can replace expensive ensembles in large-scale evaluation.","feed_headline":"Probability-space margin attack beats every individual attack tested","feed_subtitle":"It also matches the standard four-method ensemble at 25% runtime.","key_machinery":"The load-bearing object is the probability margin loss $L_{pm}=p_{\\max}-p_y$, which replaces the logits difference $z_{\\max}-z_y$ of the classic margin loss with a softmax-probability difference. Its gradient decomposes as $p_y\\nabla L_{ce} + p_{\\max}\\nabla L_{ce}^{\\mathrm{targeted}}$, making it a single formula that behaves as an adaptive blend of untargeted and targeted cross-entropy. PMA wraps this loss in a two-stage attack that alternates between maximizing $p_{\\max}$ and minimizing $p_y$ in stage one, then optimizes the full difference in stage two, with a cosine step-size schedule and multiple restarts; the alternation is what lets the attack explore different wrong-class directions before committing to the full margin.","core_discovery":"The central claim is that the probability margin $L_{pm}(z,y)=p_{\\max}-p_y$, the gap between the top non-true-class softmax probability and the true-class probability, is a better adversarial objective than logits margin or cross-entropy. The paper's key identity is that its gradient is a probability-weighted combination of the untargeted and targeted cross-entropy gradients, $\\nabla_x L_{pm}=p_y\\nabla_x L_{ce}+p_{\\max}\\nabla_x L_{ce}^{t}$, so the attack inherits both broad exploration and focus on the strongest wrong class. Built into a two-stage margin-decomposition pipeline with alternating objectives and restarts, PMA is reported as the strongest individual white-box attack on the 29-model test bed. The paper further claims that the one-attack combination, PMA+APGDT, beats the standard ensemble attack on all tested models while taking about a quarter of the runtime, and that million-scale evaluation on CC1M reveals a far lower robustness than the ImageNet test set suggests.","pith_inferences":["Editorial inference: Because the gradient identity expresses the PM loss as a positive combination of two CE gradients, a testable extension is whether any weight pair beyond $(p_y, p_{\\max})$ yields even stronger attacks; the paper's own ablation only varies a single scalar weight.","Editorial inference: The million-scale comparison conflates scale with distribution shift: a 50k-image control drawn from CC1M and evaluated with the same pseudo-label metric would isolate whether the drop from >59% to <19% is caused by the number of images or by the CC1M distribution itself.","Editorial inference: Relative robustness with model-generated pseudo-labels is a weaker metric than ground-truth robustness; if pseudo-labels drift on out-of-distribution images, the reported gap may partly reflect label noise rather than true vulnerability."],"forward_implications":["PMA lowers the measured robust accuracy of leading models by 0.03% to 0.55% compared with the best existing individual attack, so previously reported individual-attack numbers are slightly optimistic.","The PMA+APGDT ensemble achieves lower robust accuracy than the standard four-method ensemble on every tested model with about 25% of its runtime, making strong evaluation much cheaper.","The CC1M million-scale evaluation puts relative robustness of top ImageNet models below 19%, versus above 59% on the ImageNet test set, implying small-scale leaderboard evaluations substantially overstate robustness on broader data.","PMA is roughly 3.56% to 11.86% lower in relative robustness than PGD with cross-entropy on CC1M, so the gap between attacks grows at scale."],"supporting_citations":[{"why":"Supplies the two-stage margin-decomposition attack pipeline that PMA reuses with a replaced loss.","marker":"[7]"},{"why":"Provides the APGD update strategy, the DLR loss baseline, and the ensemble attack baseline PMA is compared against.","marker":"[5]"},{"why":"Defines the logits margin loss and targeted cross-entropy objective that the probability margin identity generalizes.","marker":"[3]"},{"why":"Supplies the projected gradient descent baseline and the L-infinity attack formulation.","marker":"[4]"},{"why":"Source dataset for Conceptual Captions 3 Million, from which the million-scale CC1M evaluation set is derived.","marker":"[14]"},{"why":"Provides the standardized leaderboard and evaluation protocol whose top models are the 29-model test bed.","marker":"[20]"}],"fun_headline_variants":["Probability margin attack beats all individual attacks tested","Million-scale attack eval shows ImageNet models more fragile","New attack matches ensemble at a quarter of runtime","Probability-space attack outperforms every individual method"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The million-scale finding assumes that relative robustness measured with model-predicted labels on the filtered CC1M images is a fair proxy for real-world robustness, so the large observed drop is due to evaluation scale rather than to the different image distribution or the filtering procedure.","fun_headline_variants_meta":{"raw":{"variants":["Probability margin attack beats all individual attacks tested","Million-scale attack eval shows ImageNet models more fragile","New attack matches ensemble at a quarter of runtime","Probability-space attack outperforms every individual method"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2567,"prompt_tokens":983,"completion_tokens":1584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1537}},"tokens_in":599,"tokens_out":1584,"duration_ms":13421,"temperature":1.0,"reasoning_tokens":1537,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:43:42.991724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random 50,000-image subset of CC1M and a 50,000-image subset of the ImageNet test set, run the same PMA and PGD-CE attacks with the same pseudo-label relative-robustness metric on both; if the relative robustness on the CC1M subset is close to the ImageNet subset or the claimed >59%-to-<19% gap disappears, the scale effect has not been isolated from the dataset shift.","supporting_citations":[{"cited_title":"Imbalanced gradients: a subtle cause of overestimated adversarial robustness,","cited_arxiv_id":null,"evidence_quote":"Supplies the two-stage margin-decomposition attack pipeline that PMA reuses with a replaced loss."},{"cited_title":"Reliable evaluation of adversarial ro- bustness with an ensemble of diverse parameter-free attacks,","cited_arxiv_id":null,"evidence_quote":"Provides the APGD update strategy, the DLR loss baseline, and the ensemble attack baseline PMA is compared against."},{"cited_title":"Towards evaluating the robustness of neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the logits margin loss and targeted cross-entropy objective that the probability margin identity generalizes."},{"cited_title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,","cited_arxiv_id":null,"evidence_quote":"Source dataset for Conceptual Captions 3 Million, from which the million-scale CC1M evaluation set is derived."},{"cited_title":"Robust- bench: a standardized adversarial robustness benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides the standardized leaderboard and evaluation protocol whose top models are the 29-model test bed."}],"review_version":1}