{"id":"d35a090f-e7e5-45c7-aa35-378bf6cffba0","arxiv_id":"2511.01724","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Across 222 models, standard adversarial training matches or beats dedicated probabilistic-robustness training on probabilistic robustness, while probabilistic methods show lower generalization gaps but near-zero adversarial robustness.","lead":"This paper builds a benchmark comparing two families of training methods for making image classifiers robust: adversarial training (worst-case) and probabilistic robustness training (average-case under random noise). It reports that standard adversarial training also delivers high probabilistic robustness, while dedicated probabilistic-robustness methods generalize better but remain weak against attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that AT outperforms PR-targeted training in improving PR may be an artifact of conditioning PR on clean-correct inputs; under the abstract's unconditional PR reading, Corruption/CVaR exceed PGD at γ=0.03.","rationale":"The reader's verdict already flags the GE metric inconsistency and hyperparameter budgets. I focus instead on a more direct threat to the central claim: the PR metric is conditional on clean-correct classification, which is model-dependent. Because AT lowers clean accuracy, its conditional PR is computed on a smaller subset, and the published numbers show that the headline ordering reverses when PR is made unconditional. This is a correctness risk in the benchmark's key comparison, not a matter of consensus: the abstract's own definition of PR is unconditional. A simple recomputation from Table 3 suffices to show the issue, so it is the single most load-bearing concern. The GE impossibility noted by the reader is serious but affects the secondary 'lower GE' finding rather than the 'AT wins on PR' claim. Verdict remains CONDITIONAL: the benchmark is valuable but the central claim needs re-analysis under an accuracy-adjusted PR metric.","tokens_in":47899,"tokens_out":13553,"duration_ms":155601,"concrete_test":"Recompute Table 3 (and Table 9) PRD columns as Acc × PRD/100 for every dataset/model, or rerun PR evaluation over the full test set with PR=0 for clean-misclassified inputs. If the ranking changes so that Corruption/CVaR exceed PGD/TRADES at γ=8/255 (as the published numbers already imply), then the central claim 'AT improves PR more than PR-targeted methods' is not supported by the stated PR definition and must be revised or re-scoped.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1's headline finding ('AT outperforms PR-targeted training in improving PR') rests on PRD(γ), which Eq. 22 defines over M test samples originally classified correctly. Since AT methods sacrifice clean accuracy (Table 3: PGD 83.83% vs ERM 94.85% and Corruption 94.17% on CIFAR-10/ResNet-18), this conditioning evaluates AT on a smaller, easier subset, so the comparison is not between models on the same population. Recomputing the abstract's PR notion—probability that a randomly drawn input remains correctly predicted under perturbation—as unconditional PRD = (Acc/100)×PRD_correct gives, at γ=0.03: Corruption 93.34, ERM 92.61, CVaR 88.72, KL-PGD 87.23, AT-PR 86.08, PGD 83.53, TRADES 82.97. PGD no longer outperforms the PR-targeted Corruption/CVaR; the ranking flips. At larger radii (γ=0.12) PGD/AT-PR do win, so the 'in most cases' claim is radius-dependent and metric-dependent. The paper's Remark 1 limits the claim to 'our current evaluation setting', but the abstract and introduction state PR as the unconditional probability that predictions remain correct. The conditional choice is defended via citations (Li et al.; Chen & Lee), but that defense supports a definition of robustness for clean-correct examples, not the benchmark's cross-model ranking claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PRBench, a benchmark for probabilistic robustness (PR) training methods. It compares ERM, six adversarial training (AT) methods, four risk-based training (RT / PR-targeted) methods, and the hybrid AT-PR method across 7 datasets and 10 architectures, reporting clean accuracy, AR, PR, generalization error (GE), and training time. The main empirical claim is that AT methods improve PR 'for free' and generally outperform PR-targeted methods on both AR and PR, while PR-targeted methods have lower GE and higher clean accuracy. The paper also derives uniform-stability GE bounds and attempts to explain the observed GE ordering theoretically. The benchmark is a substantial resource, but the central comparison is sensitive to the chosen conditional PR definition, the reported GE values are arithmetically inconsistent with the stated definition, and the theoretical results contain mismatches with their proofs.","tokens_in":48310,"tokens_out":7881,"duration_ms":84842,"significance":"If the results hold, PRBench would be a valuable standardized resource: it is the first dedicated PR training benchmark, covers a wide range of methods, and makes a clear, falsifiable claim that AT is a generally effective way to improve PR. The public leaderboard and extensible codebase are useful community assets. The paper also attempts a theoretical account of GE differences, which is a useful step. However, the validity of the headline claim depends on a metric choice that changes the empirical ranking, and the GE numbers are not internally consistent. The benchmark's significance is therefore currently contingent on resolving these issues.","major_comments":[{"comment":"The GE metric as defined is inconsistent with the reported values. Eq. (24) defines GE_PR = PR_train − PR_test, and Table 2 reports GE in percent. Yet for CIFAR-10 ResNet-18 PGD, PR_D(0.03)=99.63% and GE_PR(0.03)=11.4, which would imply PR_train=111.03%, an impossibility. The same pattern appears in Tables 3, 7, 9, and 11. Either GE is computed with a different formula (e.g., relative difference or over a different subpopulation) or the PR values are on a different scale. Because the GE comparisons in §4.2 and the theoretical discussion rely on these numbers, this inconsistency invalidates the empirical GE analysis and must be corrected before the benchmark results can be trusted.","section":"Table 2 / Eq. (24) / Table 3"},{"comment":"The headline ranking of AT over PR-targeted methods depends on a conditional PR measure that is inconsistent with the paper's own Definition 1 and abstract. Eq. (22) restricts M to clean-correct test inputs, so PR_D(γ)=99.63% for PGD on CIFAR-10 means '99.63% of correctly classified inputs remain correct', not '99.63% of all inputs remain correct'. Since AT methods reduce clean accuracy substantially (PGD 83.83% vs ERM 94.85% and Corruption 94.17% in Table 3), conditioning favors AT. Recomputing the unconditional quantity as Acc/100×PR_D_correct at γ=0.03 gives Corruption 93.34% vs PGD 83.53%, changing the §4.1 claim that AT outperforms PR-targeted methods. The ranking is radius-dependent (at γ=0.12 PGD and AT-PR do win) and metric-dependent. Remark 1 limits the claim to the 'current evaluation setting', but the abstract and introduction state PR as the unconditional probability. The pap","section":"§3.3, Eq. (22), §4.1"},{"comment":"Theorem 2 in the main text does not match its proof. Main-text Eq. (14) gives eϕ = φ + 2λγ(ν^2β + 2νLLθ) − 2Lθ(1−ν), while Appendix Theorem 3 derives eϕ = φ + 12λγ(ν^2β + 2νLLθ) (Eq. 84) with no −2Lθ(1−ν) term. Similarly, Eq. (15) gives ψ + 6λ(ν^2βθ + 4νLθ^2), while Eq. (85) gives ψ + λ(6ν^2βθ + 24νLθ^2) — a factor-of-λ discrepancy. These are not merely cosmetic: Theorem 2 is used in §4.2 to explain GE differences among AT methods (PGD vs TRADES/MART, ALP vs CLP). The proof must be corrected and the implications for the stated conclusions re-examined.","section":"Theorem 2 and Appendix E.2"},{"comment":"The theoretical support for the claim 'RT methods consistently yield lower GE' is not established. Prop. 1 states that CVaR training is φ-Lipschitz and max{φ,ψ}-smooth, but the proof in Appendix E.3 treats the term [ℓ(fθ(x+δ),y)−α]_+ with α fixed, even though Algorithm 1 updates α jointly with θ. No rigorous bound is given for the CVaR objective as a function of θ with the coupled (α,θ) dynamics. The statement that corruption training 'can be seen as a special case of CVaR' is also not derived. Given that the GE metric itself is inconsistent (see first major comment), the empirical observation of lower GE is not yet trustworthy. A complete proof or a clearly labeled conjecture is needed.","section":"Appendix E.3 / Proposition 1"}],"minor_comments":[{"comment":"The abstract states '229 trained models' while Section 1 and Section 3.2 state '222 trained models'. Please reconcile the count.","section":"Abstract vs. Section 1"},{"comment":"The abstract gives https://wellzline.github.io/PRBenchLeaderboard/ while the main text gives https://tmpspace.github.io/PRBenchLeaderboard/. Please use one consistent URL.","section":"Abstract / Leaderboard URL"},{"comment":"The composite robustness score is a weighted sum with user-chosen weights. The paper presents equal-weighting results, but Fig. 6 shows sensitivity to weights. This is acceptable as an exploratory aggregate, but the main conclusions should not rely on it without a sensitivity discussion.","section":"Fig. 2(b) and composite score"},{"comment":"The Monte Carlo evaluation uses N=100 perturbations per test input. This sampling size and the choice of perturbation distribution should be justified or at least noted as a possible source of variance, especially when reporting PR values to two decimal places.","section":"Appendix A.3 / Eq. (22)"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a bold and interesting claim, but the load-bearing metric definition and the GE arithmetic need to be fixed. The conditional-PR issue changes the ranking at small radii, so the 'AT outperforms PR-targeted' summary in the abstract is currently not supported by the unconditional PR reading. The Theorem 2 mismatch with its appendix proof is also a red flag that the theoretical section has not been carefully checked. I would encourage the editor to ask for a full revision and re-review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: PRBench is the first dedicated probabilistic-robustness training benchmark, and that alone earns it attention. The systematic comparison of 13 methods across 7 datasets and 10 architectures is real work, and the leaderboard artifact is valuable. The KL-PGD variant is a nice diagnostic idea, and Theorem 2's extension of the Xiao et al. stability bounds to regularized AT objectives is a legitimate theoretical contribution, even if it leans on prior machinery.\n\nThe central finding — AT improves PR more than PR-targeted training, so PR comes \"for free\" — is plausible and worth discussing, but it reads as stronger than the evidence supports. Two soft spots, in order.\n\nFirst, the GE metric is numerically impossible under its own definition. Table 2 defines GE_PR = PR_train − PR_test. For CIFAR-10 ResNet-18 PGD, PR_test(0.03)=99.63% and GE=11.4, which would require PR_train > 111%. That's not a rounding issue. The same pattern appears across many rows. Since the paper's second headline claim (PR-targeted methods generalize better) rests on GE, this is a load-bearing flaw. It needs re-derivation: maybe the GE was computed on a different base set or with a different formula, but as written it's simply wrong.\n\nSecond, the abstract and intro define PR as the probability that a random perturbation preserves the correct label, but the actual metric PR_D (Eq. 22) is conditioned on clean-correct inputs. That choice is defensible in isolation (the citations to Li et al. and Chen & Lee support it), but it biases the cross-method ranking. AT methods trade clean accuracy for robustness, so measuring only on the clean-correct subset evaluates AT on a smaller, easier population. Recomputing unconditional PR with the numbers in Table 3 flips the γ=0.03 ranking between PGD and Corruption. The paper's own Remark 1 restricts the claim to \"our current evaluation setting,\" but the abstract doesn't carry that caveat. At minimum, both versions should be reported.\n\nMinor issues: the per-method optimization budgets differ (10-step PGD vs single-sample corruption vs 20-sample CVaR), so fairness of the comparison is arguable though not obviously broken; code is not public; no seeds are reported; and EVaR is described but not included in the benchmark. These are fixable.\n\nBottom line: the paper deserves a serious referee, but not acceptance as-is. The benchmark artifact and the AT-vs-PR comparison are worth publishing after the GE issue is corrected, the conditional/unconditional choice is handled transparently, and the code is released.","headline":"Worth reviewing: a genuinely useful PR benchmark with an internally inconsistent GE metric and a headline claim that depends on a conditioning choice.","tokens_in":48816,"tokens_out":2395,"would_cite":true,"duration_ms":27559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard adversarial training improves probabilistic robustness as a free by-product, according to a new benchmark.","keywords":["probabilistic robustness","adversarial robustness","adversarial training","benchmark","generalization error","risk-based training","robust overfitting","deep learning"],"falsifier":"Retrain the PR-targeted methods in PRBench using the same 10-step PGD-style search over perturbations (or a matched number of sampled perturbations) and re-evaluate on the same leaderboard; if any PR-targeted method then matches or exceeds PGD's PR and AR, the central claim fails.","tokens_in":47783,"feed_emoji":"🛡️","tokens_out":4360,"duration_ms":43643,"temperature":0.7,"pith_summary":"The paper introduces PRBench, the first benchmark dedicated to evaluating training methods for probabilistic robustness (PR), and uses it to test a bold hypothesis: PR comes 'for free' when models are adversarially trained (AT) for adversarial robustness (AR). Across 222 models, 7 datasets, and 10 architectures, AT methods consistently outperform dedicated PR-targeted training methods on both AR and PR, at little or no extra cost. PR-targeted methods (risk-based training) retain advantages in cleaner accuracy and lower generalization error, a result the paper supports with uniform stability bounds. The upshot is that, under this benchmark's settings, there may be limited practical need for separate PR-specific training methods, though the paper cautions this is a hypothesis to be tested further.","feed_headline":"Adversarial training gets probabilistic robustness for free","feed_subtitle":"A 222-model benchmark shows standard AT beats dedicated PR training on both robustness metrics.","key_machinery":"PRBench itself is the key instrument: a unified evaluation protocol with a general formulation of risk-based training (Definition 2), a composite robustness score, and a uniform stability analysis (Theorems 1 and 2) that bounds generalization error via Lipschitz and smoothness constants. The theoretical lens explains why RT methods have lower GE and why penalty-based AT methods like TRADES have lower GE than plain PGD.","core_discovery":"The central discovery is a one-way free lunch: applying standard adversarial training (e.g., PGD or TRADES) to improve worst-case robustness also drives probabilistic robustness to above 99% PRD at the training radius, outperforming the dedicated PR-targeted methods (corruption training, CVaR-based training, and the hybrid AT-PR) on both PR and AR metrics. The reverse does not hold: PR-targeted methods yield near-zero AR under strong attacks. The paper derives generalization-error bounds showing why RT methods generalize better (they are smoother), while AT's robust overfitting is controlled by an extra term in the stability bound.","pith_inferences":["Matched optimization budgets: giving PR-targeted methods the same per-sample attack steps as PGD (e.g., 10-step random search) might close the PR/AR gap; the benchmark's current comparison may understate PR-targeted methods' potential.","The 'PR for free' claim likely depends on the perturbation radius: AT's advantage is largest near the training radius; at much larger radii, the gap may shrink or invert.","The GE advantage of RT methods suggests that PR-targeted training could serve as a regularizer in high-noise domains, even if AR is not the objective.","A testable extension: apply AT-PR's all-AE-region selection to standard AT candidates (not just PGD) to see if the GE/smoothness benefit can be combined with low cost."],"forward_implications":["If AT truly buys PR for free, future PR research can adopt standard AT (e.g., TRADES) as a strong baseline, instead of designing bespoke PR losses.","PR-targeted methods remain viable when clean accuracy and low generalization error are the priority, since they sacrifice AR but keep accuracy high.","The hybrid AT-PR method offers a balanced trade-off but with heavy computational cost, pointing to efficiency as the next bottleneck.","The benchmark's extensible codebase and public leaderboard standardize PR evaluation, allowing new methods to be compared transparently."],"fun_headline_variants":["AT beats PR-specific training on robustness metrics","PRBench: AT yields PR, but PR doesn't yield AR","Adversarial training provides PR for free, PRBench shows","PR-targeted methods generalize better, but AT is more robust"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's headline comparison gives standard AT a 10-step PGD attack to generate training perturbations, while PR-targeted methods draw only one random perturbation per sample; if the PR-targeted methods were given a comparable optimization budget, the 'PR for free' conclusion could change.","fun_headline_variants_meta":{"raw":{"variants":["AT beats PR-specific training on robustness metrics","PRBench: AT yields PR, but PR doesn't yield AR","Adversarial training provides PR for free, PRBench shows","PR-targeted methods generalize better, but AT is more robust"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00095,"raw_usage":{"total_tokens":3910,"prompt_tokens":786,"completion_tokens":3124,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3056}},"tokens_in":530,"tokens_out":3124,"duration_ms":27442,"temperature":1.0,"reasoning_tokens":3056,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:15:49.799955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the PR-targeted methods in PRBench using the same 10-step PGD-style search over perturbations (or a matched number of sampled perturbations) and re-evaluate on the same leaderboard; if any PR-targeted method then matches or exceeds PGD's PR and AR, the central claim fails.","supporting_citations":[],"review_version":1}