{"id":"847f7c85-4e8d-45a6-b455-404a5bb187ec","arxiv_id":"2411.09265","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"BEARD is a benchmark and leaderboard evaluating adversarial robustness of dataset distillation methods with new normalized metrics across multiple datasets and images-per-class settings.","lead":"This paper introduces BEARD, a benchmark for measuring how well models trained on distilled, synthetic datasets resist adversarial attacks. It proposes three normalized metrics and applies them to six dataset distillation methods across CIFAR-10, CIFAR-100, and TinyImageNet.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 reports identical RRM values for targeted and untargeted attacks, and appendices state contradictory training protocols; the central empirical claim is therefore not reproducible.","rationale":"The reader's weakest assumption is essentially correct: the empirical conclusion presupposes a single, consistent measurement protocol with correctly separated targeted and untargeted runs. The manuscript contradicts this in at least two places. Table 1's identical RRM values across attack types are the most direct evidence that the reported numbers were not generated by separate attack evaluations. The prose in Appendix B.1 even claims that untargeted attacks are less pronounced than targeted attacks, but the table shows no difference in RRM, so the text and numbers are internally inconsistent. The conflicting training protocols in Appendices A.1.1 and A.3 further undermine reproducibility, since the model pool is the foundation for every metric. The unspecified alpha in Definition 8 is a third, independent reproducibility defect. Any one of these would be a major issue for a benchmark paper; together they make the central claim unverifiable from the manuscript as written. I considered whether the identical RRM values could be a harmless typesetting error, but the wholesale duplication across every method and dataset, combined with the contradictory narrative, makes that unlikely. I therefore agree with the reader's REJECT verdict and do not recommend changing it. A major revision with corrected tables, a single stated training protocol, a specified alpha, and released code could change this assessment, but the current version does not support the headline claim.","tokens_in":17567,"tokens_out":5298,"duration_ms":59757,"concrete_test":"Download the released BEARD code and the provided distilled datasets; retrain the model pool using one explicitly chosen protocol (or both Appendix protocols separately); then run targeted and untargeted attacks as separate, logged scripts and recompute Table 1 from raw outputs. If the RRM values for targeted and untargeted attacks still match exactly, or if the CREI values do not satisfy CREI = alpha*RRM + (1-alpha)*AEM for any stated alpha (e.g., alpha=0.5), the central empirical claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that models trained on distilled datasets exhibit higher Multi-Adversary Robustness Ratio (RRM) and lower Multi-Adversary Attack Efficiency Ratio (AEM) than full-data models. The quantitative evidence for this is Table 1. In Table 1, the RRM values under 'Targ. Att.' are exactly identical to those under 'Untarg. Att.' for every method and every dataset (e.g., CIFAR-10 DM has RRM 46.01 in both columns). Targeted and untargeted attacks are distinct threat models with different success rates, so exact equality across all entries is implausible; the columns appear to have been copied rather than produced by separate attack runs. The problem is compounded by Appendix A.1.1, which states that models are trained with Adam (lr 1e-4, batch 128), while Appendix A.3 states that the same model training uses SGD (lr 0.01, momentum 0.9, weight decay 5e-4, batch 256). These are incompatible protocols for the model pool on which all metrics are computed. In addition, Definition 8 defines CREI = alpha*RR + (1-alpha)*AE but never specifies alpha, so the reported CREI values cannot be independently recomputed. Because the contribution is an empirical benchmark and ranking, this internal inconsistency is load-bearing: if the attack-type separation or training protocol is wrong, the claimed robustness ordering of DD methods is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BEARD, a benchmark for evaluating the adversarial robustness of dataset distillation (DD) methods. It proposes three metrics — Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive Robustness-Efficiency Index (CREI) — within an adversarial game framework, and reports experiments comparing six DD methods (DC, DSA, DM, MTT, IDM, BACON) on CIFAR-10, CIFAR-100, and TinyImageNet under targeted and untargeted attacks, across multiple IPC settings and with and without adversarial training. The central empirical claim is stated in Section 5.1: models trained on synthetic datasets generated by DD methods exhibit higher Multi-Adversary Robustness Ratio (RRM) under both targeted and untargeted attacks, and lower Multi-Adversary Attack Efficiency Ratio (AEM), compared to full-data models.","tokens_in":17867,"tokens_out":7529,"duration_ms":68847,"significance":"If its empirical results were reliable, BEARD would be a useful community resource: standardized robustness metrics for DD methods, a leaderboard, and an extensible code base would help researchers compare distillation algorithms under adversarial threat models. The adversarial-game framing and the separation of attack effectiveness from attack efficiency are reasonable design choices. However, the paper's contribution is primarily empirical, and the internal inconsistencies in the reported tables and training protocols undermine the credibility of all quantitative claims. The paper does not ship machine-checked proofs or a reproducible artifact that would let readers verify the results independently; the unspecified CREI weighting and the absence of variance estimates further limit the scientific value in its current form.","major_comments":[{"comment":"The RRM values under 'Targ. Att.' and 'Untarg. Att.' are identical for every method and every dataset (e.g., CIFAR-10 DM has RRM 46.01 in both columns). Targeted and untargeted attacks are distinct threat models with different success rates, so exact equality across all entries is implausible and indicates that the two columns were copied rather than produced by separate attack runs. This directly contradicts the central claim in Section 5.1 that DD methods exhibit higher RRM under both targeted and untargeted attacks, and it is also contradicted by the paper's own prose in Section B.1, which describes different robustness behavior for the two attack types.","section":"Table 1; Section 5.1"},{"comment":"The model training protocol is described inconsistently. Appendix A.1.1 states that models are trained with the Adam optimizer (learning rate 1e-4, batch size 128), while Appendix A.3 states that the same model training phase uses SGD (learning rate 0.01, momentum 0.9, weight decay 5e-4, batch size 256). Since all RRM, AEM, and CREI values are computed on models from this model pool, the reported results cannot be reproduced without knowing which protocol was actually used, and any comparison across methods is ambiguous.","section":"Appendix A.1.1 vs. Appendix A.3"},{"comment":"CREI is defined as alpha * RR + (1 - alpha) * AE, but the value of alpha is never specified anywhere in the manuscript, including in the appendix where CREI values are reported in Tables 1-4. Without alpha, the CREI scores cannot be independently recomputed or compared across tables, which is a load-bearing omission for the paper's headline metric.","section":"Definition 8, Eq. (9)"},{"comment":"The key observation 'Increasing the IPC decreases adversarial robustness' is supported for targeted attacks in Table 2, but Table 3 (untargeted attacks) shows non-monotonic or increasing CREI with IPC for several methods (e.g., CIFAR-10 BACON: 25.79, 27.09, 28.48; CIFAR-100 MTT: 17.91, 19.97, 20.54). As written, the claim overgeneralizes and is contradicted by the data in the same appendix; the text should either restrict the claim to targeted attacks or account for the non-monotonic cases.","section":"Section 5.2; Table 3"},{"comment":"The paper states that the model with the highest validation accuracy on the original test set is selected, which uses the test set for model selection and biases the subsequent robustness evaluation. Additionally, the paper says results are averaged over 5 runs, but no standard deviations or error bars are reported anywhere. For a benchmark whose contribution is comparative ranking, the lack of variance estimates makes it impossible to assess whether small differences in the tables (e.g., DM 46.01 vs. DSA 45.22 in Table 1) are meaningful.","section":"Appendix A.1.1; Tables 1-4"}],"minor_comments":[{"comment":"The reference list appears to have the titles for [46] and [47] swapped: [46] is cited for DM but listed as 'Dataset condensation with differentiable siamese augmentation' (which is actually DSA), while [47] is cited for DSA but listed as 'Dataset condensation with distribution matching' (which is actually DM). This will mislead readers who try to reproduce the methods.","section":"References [46] and [47]"},{"comment":"The paper says the leaderboard integrates 'six adversarial attack strategies' but lists only five named attacks (FGSM, PGD, C&W, DeepFool, AutoAttack). Please clarify whether a sixth attack is used and name it, or correct the count.","section":"Section 4.2"},{"comment":"The code is referenced only as 'BEARD' with no URL or permanent identifier. A stable link (e.g., a DOI or an archived repository) is needed for reproducibility.","section":"Abstract / Section 6"},{"comment":"Missing entries for IDM and BACON on TinyImageNet are marked with '/' but no explanation is given. State whether these runs were infeasible, omitted, or unavailable.","section":"Tables 2 and 3"},{"comment":"The thresholds gamma and beta in Definition 5 are declared conceptual and never assigned values, and they do not appear in the computed metrics. This is acceptable, but the text should state explicitly that gamma and beta are not used in the evaluation to avoid implying that the inequalities are ever checked.","section":"Definition 5, Remark 1"},{"comment":"The sentence 'with RRM values increasing as dataset size expands' is unclear because Figure 4 has no dataset-size axis; it plots methods. Please rephrase to describe what Figure 4 actually shows.","section":"Section 5.1"},{"comment":"The 'w/o AT' rows in Table 4 are identical to the CREI entries in Table 1. Confirm that this is intended and note it in the text, or explain the duplication.","section":"Table 4 vs. Table 1"}],"recommendation":"reject","confidential_remarks":"The duplicated RRM values between targeted and untargeted attacks and the contradictory training protocols in the appendices suggest that the manuscript may have been assembled from multiple experimental campaigns without internal verification. These are not mere presentation issues: they strike at the central empirical claim. If the authors can provide a corrected table and a single, clearly specified training protocol, plus error bars and the missing alpha value, a revised submission could be reconsidered. I would also ask for the code to be archived with a stable identifier rather than only a floating leaderboard page."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the benchmark apparatus: three datasets, six distillation methods, three IPC settings, five attacks, and an efficiency axis that the prior DD-RobustBench lacked. The leaderboard and the released code are genuine value-adds for the dataset distillation community. The RR, AE, and CREI formulas are simple aggregations of standard ASR/AST, but the empirical sweep itself is new and worth having.\n\nThat said, the empirical core does not hold together. Table 1 reports literally identical RRM values for targeted and untargeted attacks for every method and every dataset. That is not a plausible measurement outcome; targeted and untargeted attacks are different threat models with different success rates. Either the columns were copied or the attack-type separation was never actually performed. Since the central claim — DD methods improve robustness — rests on these numbers, this is load-bearing.\n\nThe training protocol is also ambiguous. Appendix A.1.1 says the model pool uses Adam, lr 1e-4, batch 128. Appendix A.3 says SGD, lr 0.01, momentum 0.9, weight decay 5e-4, batch 256. Those are incompatible. The model pool is the substrate for every metric, so this contradiction makes the whole benchmark hard to trust as written. And CREI's alpha is never specified, so the headline index cannot be independently recomputed. No error bars are given despite a claim of five runs.\n\nThese are all fixable in revision, but they are not cosmetic. The paper is a benchmark paper; its contribution is measurements and rankings. When the measurements are internally inconsistent, the rankings are unsupported. The circularity burden is low, to be fair: the metrics are openly normalized to the benchmark maximum and no target result is assumed. The related work is cited properly, including the prior DD-RobustBench.\n\nWho is this for? Researchers working on dataset distillation robustness who want a standardized evaluation suite. The concept deserves a serious referee — I would not desk-reject it. But the current version should not be accepted without a corrected Table 1, a single explicitly stated training protocol, a specified alpha, and ideally released model checkpoints with seeds. I would not cite the numbers in their present form.","headline":"Useful benchmark concept, but the numbers that carry the main robustness claim are internally inconsistent as printed; a revision could make this a solid community resource.","tokens_in":18363,"tokens_out":1627,"would_cite":false,"duration_ms":16134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on distilled datasets makes models harder to fool by adversarial attacks, the BEARD benchmark finds, at the cost of attack speed and some clean accuracy.","keywords":["dataset distillation","adversarial robustness","benchmark","robustness ratio","attack efficiency","adversarial attacks","images per class","adversarial training"],"falsifier":"Re-run the BEARD evaluation on CIFAR-10 with a single fixed training configuration (one optimizer, one batch size), reporting RRM separately for targeted and untargeted attacks: if the two columns match exactly, or if distilled models no longer beat the full-size baseline, the paper's central robustness claim fails.","tokens_in":17392,"feed_emoji":"🛡️","tokens_out":7809,"duration_ms":71369,"temperature":0.7,"pith_summary":"BEARD is a standardized benchmark asking whether models trained on distilled, synthetic datasets are more resistant to adversarial attacks than models trained on the full dataset. Across three datasets (CIFAR-10, CIFAR-100, TinyImageNet), six distillation methods, and five attack families, the paper claims the answer is yes on the robustness side: distilled-data models post a higher Multi-Adversary Robustness Ratio under both targeted and untargeted attacks, although attacks that do succeed take less time against them, as captured by a lower Multi-Adversary Attack Efficiency Ratio. The paper also reports that the robustness advantage shrinks as the number of images per class grows, and that adversarial training lifts full-size models more than distilled ones. If these results hold, dataset distillation is not only a way to save compute but a security-relevant design choice for the resulting model.","feed_headline":"Distilled data trains models that resist attacks better","feed_subtitle":"Benchmark across three datasets finds synthetic training sets beat full data on targeted and untargeted robustness.","key_machinery":"The load-bearing machinery is the adversarial game framework, formalized as attacker and defender functions, together with three derived metrics. Robustness Ratio is $RR = 100\\,\\left(1 - \\bar{ASR}/\\max ASR\\right)$, where $\\bar{ASR}$ averages the attack success rate over models and attacks; Attack Efficiency Ratio is $AE = 100\\,\\left(\\bar{AST}/\\max AST\\right)$, where $AST$ is the time to a successful attack; and the Comprehensive Robustness-Efficiency Index blends them as $CREI = \\alpha\\,RR + (1-\\alpha)\\,AE$. Single- and multi-adversary variants ($RRS/RRM$, $AES/AEM$) adapt the same formulas. These metrics convert the game's win conditions, low attack success rate and high attack success time, into comparable numbers that let the paper rank distillation methods on a leaderboard.","core_discovery":"The paper's central claim is that dataset distillation confers an adversarial-robustness advantage: models trained on synthetic sets produced by DD methods obtain higher Multi-Adversary Robustness Ratio (RRM) than models trained on the full-size dataset, under both targeted and untargeted attacks, across CIFAR-10, CIFAR-100, and TinyImageNet. This advantage is strongest at the smallest compression setting (IPC-1) and tapers off as the number of images per class increases. The authors interpret the accompanying lower Multi-Adversary Attack Efficiency Ratio (AEM) as part of the same trade-off: the distilled models are harder to fool, but the attacks that do succeed take less time. They further claim that adversarial training produces larger CREI gains for full-size datasets than for distilled ones, and that the robustness comes partly at the expense of clean-data accuracy, consistent with distillation exposing models to a mix of robust and non-robust features.","pith_inferences":["If the robustness advantage survives a cleanly separated re-run, the most direct testable extension is to check the paper's feature-level explanation directly: distillation should leave models with fewer non-robust features, which representation analyses could verify.","The result that adversarial training boosts full-size models more than distilled ones suggests the two defenses are partly redundant; a practical corollary the authors do not draw is that a fixed robustness budget spent on AT may be better allocated as the dataset grows toward full size.","Because the advantage is strongest at IPC-1, the findings point to a regime rule the authors only imply: extreme compression is where distillation's robustness benefit is most worth harvesting, and future distillation methods could be explicitly optimized for CREI rather than clean accuracy alone.","The identical RRM entries for targeted and untargeted attacks in Table 1 suggest the reported robustness metric may be insensitive to attack goal; if a rerun with separated attack runs contradicts the paper's ranking, the headline ordering of methods could shift."],"forward_implications":["Practitioners can treat distilled datasets as a robustness-enhancing choice: at IPC-1 the gains are largest, with distribution-matching methods (DSA, DM, BACON) posting the highest RRM values.","The robustness benefit declines as images per class increase, so the compression level should be part of any security analysis of a distilled model.","Adversarial training remains worthwhile for distilled models, but it pays off most on full-size datasets, so combining extreme compression with AT may waste part of the robustness budget.","The CREI metric gives a single ranking number that balances robustness and attack efficiency, which the BEARD leaderboard applies uniformly across IPC settings, datasets, and attack types.","Robustness gains come with a cost in clean-data accuracy, so downstream users should weigh the accuracy drop against the security gain when choosing a distillation method."],"supporting_citations":[{"why":"Supplies the adversarial game framework that the BEARD metrics are built on.","marker":"[7]"},{"why":"The prior dataset-distillation robustness benchmark that BEARD extends by adding multiple IPC settings and an attack-efficiency dimension.","marker":"[41]"},{"why":"DC-Bench, the source of the uniform IPC settings and hyperparameter conventions used for the distilled dataset pool.","marker":"[5]"},{"why":"The Torchattacks library that implements every adversarial attack used in the evaluation.","marker":"[17]"},{"why":"Provides the robust-versus-non-robust feature account that the paper uses to explain the robustness-accuracy trade-off.","marker":"[16]"},{"why":"BACON supplies the software foundation for the benchmark and is itself one of the six distillation methods evaluated.","marker":"[52]"},{"why":"AutoAttack is part of the multi-adversary attack suite, supporting the comprehensive robustness evaluation.","marker":"[4]"}],"fun_headline_variants":["Distilled data hardens models against attacks","Small distilled sets beat full datasets on robustness","Synthetic data improves adversarial defense","Compression boosts attack resistance in trained models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of distillation methods rests on Tables 1-4 all being produced under one consistent training-and-attack protocol, but the paper's appendix describes two different training setups and its main table shows identical robustness numbers for targeted and untargeted attacks, so that uniformity is not yet demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Distilled data hardens models against attacks","Small distilled sets beat full datasets on robustness","Synthetic data improves adversarial defense","Compression boosts attack resistance in trained models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1458,"prompt_tokens":956,"completion_tokens":502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":449}},"tokens_in":572,"tokens_out":502,"duration_ms":6801,"temperature":1.0,"reasoning_tokens":449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:50:30.570817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the BEARD evaluation on CIFAR-10 with a single fixed training configuration (one optimizer, one batch size), reporting RRM separately for targeted and untargeted attacks: if the two columns match exactly, or if distilled models no longer beat the full-size baseline, the paper's central robustness claim fails.","supporting_citations":[{"cited_title":"Multirobustbench: Benchmarking robustness against multiple attacks","cited_arxiv_id":null,"evidence_quote":"Supplies the adversarial game framework that the BEARD metrics are built on."},{"cited_title":"Dc- bench: Dataset condensation benchmark","cited_arxiv_id":null,"evidence_quote":"DC-Bench, the source of the uniform IPC settings and hyperparameter conventions used for the distilled dataset pool."},{"cited_title":"Adver- sarial examples are not bugs, they are features","cited_arxiv_id":null,"evidence_quote":"Provides the robust-versus-non-robust feature account that the paper uses to explain the robustness-accuracy trade-off."},{"cited_title":"BACON: Bayesian Optimal Condensation Framework for Dataset Distillation","cited_arxiv_id":"2406.01112","evidence_quote":"BACON supplies the software foundation for the benchmark and is itself one of the six distillation methods evaluated."},{"cited_title":"Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks","cited_arxiv_id":null,"evidence_quote":"AutoAttack is part of the multi-adversary attack suite, supporting the comprehensive robustness evaluation."}],"review_version":1}