{"id":"25193db1-95bd-46dd-9cb7-9efe8d7fc216","arxiv_id":"2411.11200","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A large benchmark of 16 backdoor mitigation methods shows high performance variability across settings, with only FT-SAM and SAU outperforming their baselines, and most newer methods failing to beat FP and FT.","lead":"This paper surveys and benchmarks 16 backdoor-defense methods against 8 attack types across 3 datasets, 4 model architectures, and 3 data-availability levels, using 122,236 experiments. It finds that most defenses work inconsistently across settings and that two 2018 baselines are not clearly beaten by newer methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline negative result is confounded by fixed defense hyperparameters: methods are compared off-the-shelf, not after the per-setting tuning their own papers used, so 'no improvement' and 'inconsistent' may be artifacts.","rationale":"The reader identified fixed hyperparameters as the weakest assumption, and I agree: it is the most load-bearing because the headline is explicitly a comparative negative result about overall performance and consistency. If the benchmark protocol handicaps newer methods by freezing hyperparameters at values from their original, often CIFAR-10-centric evaluations, then both the 'no substantial improvement' and 'inconsistent across settings' conclusions could be artifacts of configuration rather than genuine properties of the methods. The paper does give a practical justification for not tuning ASR/RDR, since these are defender-unobservable, but the claim as stated is about the methods themselves; at minimum a sensitivity analysis is needed. I see no internal inconsistency in the central argument; the concern is about external validity, not circularity or formal soundness. The paper has independent value in broadening the evaluation grid and in highlighting that RDR is rarely reported, but those contributions do not rescue the comparative claim from the hyperparameter confound. A targeted re-run on a reduced grid, with per-dataset and per-SPC tuning on defender-observable accuracy metrics, would settle whether the ranking changes. Since the reader's CONDITIONAL verdict already captures this uncertainty, I recommend no change.","tokens_in":30227,"tokens_out":5979,"duration_ms":61229,"concrete_test":"Re-run the comparison for FP, FT, FT-SAM, SAU, and one currently mid-ranking method (e.g., NPD or FST) on a reduced grid: CIFAR-10 and Tiny-ImageNet, BadNets and Blended, SPC=2 and SPC=100, ResNet and VGG. For each defense, select hyperparameters (FT-SAM epsilon; SAU lambda1/lambda2/lambda3; FST lambda; NPD lambda1-lambda3) on a held-out clean split using only ACC/ARR, over a 5-point log/linear grid per dataset and SPC, then evaluate ASR/RDR/ARR on the test set. Recompute the Figure 7 median rectangle. If FT-SAM/SAU stop separating from FT, or if any method currently outside the rectangle enters it, the abstract's negative conclusion is not robust to reasonable per-scenario tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('most newer approaches do not demonstrate substantial improvements in overall performance or consistency across diverse settings') rests on Figure 7 medians over 8 attacks x 3 datasets x 4 architectures x 3 poisoning ratios x 3 SPC values. Section V-B fixes hyperparameters: BackdoorBench defaults are used for included methods, and AWN, MM-BD, RNP, FST, and PBE use values from the authors' code, with only FST and NPD receiving dataset-specific lambda values in Supplementary Table I. This is not the configuration under which most defenses were proposed: original papers typically tune lambda and epsilon per dataset or architecture, and several methods (AWN, MM-BD, NPD, SAU) have three coupled loss weights. At SPC=2 or Tiny-ImageNet, fixed CIFAR-10-centric values can produce high ARR or high ASR for reasons unrelated to the method's capacity. The consistency claim is especially exposed, since variance across settings conflates method instability with poor hyperparameter transfer. The paper's Section VII-A argues that defenders cannot tune ASR/RDR because these are unobservable, but a benchmark can and should tune on ACC/ARR or report sensitivity; without that, the negative result is conditional on a particular default-configuration protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript combines a survey of backdoor-mitigation methods for image classification with a large independent benchmark. The authors evaluate 16 mitigation strategies against 8 backdoor attacks across 3 datasets, 4 architectures, 3 poisoning ratios, and 3 data-availability settings, reporting 122,236 individual experiments built on the BackdoorBench toolkit. The central empirical claim is that most recent mitigation methods do not substantially improve over the two seminal baselines, Fine-Pruning (FP) and Fine-Tuning (FT), in terms of median ASR, RDR, and ARR; FT-SAM and SAU are the notable exceptions. The survey portion categorizes methods into pruning and fine-tuning families and discusses the assumptions and limitations of each approach.","tokens_in":30347,"tokens_out":6013,"duration_ms":56420,"significance":"If the benchmarking protocol is accepted, this is a valuable contribution. It substantially widens the scenario coverage of BackdoorBench, reports distributions rather than cherry-picked settings, and highlights RDR as an under-reported metric that matters for practical deployment. The negative result about recent defenses is a useful corrective to single-setting evaluations in the original papers, and the detailed enumeration of experimental parameters supports reproducibility. The paper also offers a structured taxonomy and identifies hyperparameter sensitivity and limited-data overfitting as key open problems. The main risk is that the headline comparative claim is conditional on a fixed-default hyperparameter protocol, so the paper needs either a sensitivity analysis or a more carefully scoped conclusion.","major_comments":[{"comment":"The central comparison fixes all defense hyperparameters to BackdoorBench defaults or values from the authors' codebases, with only FST and NPD receiving dataset-specific lambda values. Several methods (AWN, MM-BD, NPD, SAU) have multiple coupled loss weights and perturbation bounds, so the reported ASR/RDR/ARR can reflect hyperparameter transfer failure rather than the method's capability, especially at SPC=2 or on Tiny-ImageNet. The abstract and Section VI.A.3 claim that newer approaches do not show \"substantial improvements in overall performance or consistency,\" but this claim is conditional on the default-configuration protocol. Please either add a sensitivity analysis (e.g., tune observable hyperparameters on a held-out validation split per dataset, architecture, and SPC using ACC/ARR, and report the resulting range of outcomes) or explicitly reframe the conclusion as applying to default configurations. The protocol is also internally heterogeneous: FST and NPD receive per-dataset values while AWN, MM-BD, and SAU do not, so it is unclear whether the comparison is purely off-the-shelf or partially tuned.","section":"Section V-B; Supplementary Table I"},{"comment":"The Pareto-style comparison against the FP/FT rectangle is sensitive to how \"improvement\" is defined. A method that reduces ASR but increases ARR is outside the rectangle even if it dominates on the most security-relevant metric, and the box plots show that ANP, BNP, CLP, and NPD are described as \"comparable\" to FP rather than clearly worse. The distinction between \"comparable\" and \"not substantially better\" is not quantified. Please report per-scenario win rates or a multi-metric dominance analysis, and provide uncertainty estimates (e.g., confidence intervals across the 10 iterations) so that the negative claim is not driven by the choice of pooled medians.","section":"Section VI.A.3 and Figure 7"},{"comment":"LIRA is excluded because it \"performed poorly during an initial set of experiments,\" which is a selection-on-outcome decision. If LIRA attacks are systematically weaker, their exclusion may change the distribution of attack difficulty and affect the relative ranking of defenses. In addition, NC is evaluated only on CIFAR-10 and i-BAU is incompatible with MobileNet, so the \"diverse settings\" claim is not uniform across methods. Please report the initial LIRA results or justify the exclusion on computational or scope grounds, and clearly mark every setting in which each method is absent.","section":"Section V-A; Supplementary Figures 4 and 6"}],"minor_comments":[{"comment":"The sentence listing FST, NAD, PBE, NC, and i-BAU as fine-tuning methods that do not surpass FT is inconsistent with Figure 6 and Table IV, where NC is classified as a pruning approach; this should be corrected.","section":"Section VI.A.2"},{"comment":"The reference placeholders appear as \"[?]\" instead of actual citation numbers; this should be fixed before publication.","section":"Supplementary Table I"},{"comment":"The statement \"ASR + RA ≤ 1\" is asserted without proof; because ASR excludes samples whose original label is the target class, the inequality is not self-evident and should be clarified or derived.","section":"Section VII-B"},{"comment":"There is a typo in \"we assess then under three data settings\"; it should read \"we assess them under three data settings.\"","section":"Section V-C"},{"comment":"The TAC definition averages over a set of clean/backdoor pairs, but the pairing construction is not specified; please define the joint set over which the average is taken.","section":"Section III-A.3, Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and would be a useful contribution after revision. The main risk is that the headline negative result overgeneralizes from a fixed-hyperparameter protocol; the editor should ask for either a sensitivity analysis or a softened claim. There is no indication of circularity: the benchmark uses a public toolkit and external baselines, and the paper does not fit constants to produce its own target result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a large, careful evaluation of 16 backdoor mitigation methods across a wider grid than BackdoorBench, including data availability levels. The main finding — that most post-2018 methods do not clearly beat the 2018 FP and FT baselines, with only FT-SAM and SAU beating FT — is a genuinely useful measurement for the field, even if it is less definitive than the abstract suggests.\n\nWhat's new and good: the paper extends BackdoorBench with a broader evaluation grid, especially the SPC (data availability) axis, and uses recovery accuracy (RDR) as a separate metric, which most prior work ignored. The experimental protocol is described in enough detail that the main results can be sanity-checked: attacks, architectures, datasets, poisoning ratios, SPC values, and hyperparameters are all specified. The box plots and scatter summaries in Figures 6 and 7 support the claim that most methods have high variance and that FT-SAM/SAU are the standouts. The discussion of why RDR is systematically poor is a useful contribution by itself.\n\nSoft spots: the biggest one is the fixed-hyperparameter protocol. Methods are evaluated with configurations from their original papers, not tuned per dataset, architecture, or data availability. At SPC=2 on Tiny-ImageNet, a lambda tuned for CIFAR-10 can produce high variance unrelated to the method's actual capacity. The stress-test concern is legitimate: the consistency claim conflates method instability with poor hyperparameter transfer. The authors partially anticipate this in Section VII-A, arguing that defenders can't tune on ASR/RDR because those are unobservable, but a benchmark can and should tune on ACC/ARR or at least report sensitivity. Without that, the negative result is conditional on a default-configuration protocol. That doesn't kill the paper, but it tempers the headline.\n\nOther soft spots: no code or data release (they reuse BackdoorBench but implemented five methods themselves), LIRA excluded after initial poor performance, MESA/BAERASER/BCU omitted, and NC only run on CIFAR-10. These are reasonable choices but should be clearly flagged in a final version. The survey part is competent but mostly a re-packaging of existing taxonomies.\n\nWho it's for: practitioners choosing defenses and researchers working on backdoor mitigation. It deserves a serious referee, but the central claim needs to be reworded to be explicitly about default-configuration performance, and the authors should be asked to provide some hyperparameter sensitivity analysis (e.g., a small grid on a subset of settings) or at least to release code for reproducibility.\n\nRecommendation: send to peer review. Conditional accept with a request for sensitivity analysis and code release.","headline":"A broad and careful benchmark that delivers a useful negative result, but the headline claim is conditional on off-the-shelf hyperparameters; the paper deserves review but the authors should be pushed on sensitivity analysis.","tokens_in":31002,"tokens_out":2298,"would_cite":true,"duration_ms":21523,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most published backdoor defenses, when tested across many attacks, datasets, and models, do not clearly outperform two baseline methods from 2018.","keywords":["backdoor attacks","backdoor mitigation","image recognition","model pruning","fine-tuning","benchmark evaluation","attack success rate","recovery accuracy"],"falsifier":"Re-run the same 288-scenario grid with per-scenario hyperparameter tuning, grid-searching each method's published hyperparameters on a held-out split for every dataset, architecture, and sample-per-class level, and check whether several newer methods' median ASR, ARR, and RDR jointly move inside the rectangle defined by the Fine-Pruning and fine-tuning baselines; if they do, the claim that newer methods offer no substantial improvement is overturned.","tokens_in":29918,"feed_emoji":"🛡️","tokens_out":14522,"duration_ms":117888,"temperature":0.7,"pith_summary":"This paper argues that recently published defenses against backdoor attacks are far less proven than their own papers suggest. Backdoor attacks hide a trigger in training data so a model misbehaves only when the trigger appears, and mitigation defenses aim to remove that hidden behavior without hurting normal performance. In a uniform benchmark of 16 methods against 8 attack types, 3 datasets, 4 model architectures, and 3 data-availability levels, 122,236 experiments in all, most newer methods do not deliver a substantial or consistent improvement over two baselines from 2018, filter pruning and plain fine-tuning. Only two fine-tuning methods, FT-SAM and SAU, beat the fine-tuning baseline, and both pay with greater clean-accuracy loss, while no pruning method beats the 2018 pruning baseline. The authors also find that restoring correct classification of trigger-carrying images is the unsolved core of the field, since most defenses lower attack success without restoring correct behavior, and that reported performance depends heavily on evaluation conditions.","feed_headline":"122,236 tests: most backdoor defenses don't beat 2018 baselines","feed_subtitle":"Performance swings wildly with attack, model, and data availability, so reported defense gains may not travel.","key_machinery":"The machinery is the evaluation protocol. Every defense is tested under one threat model, the Outsourced Training setting, where the defender holds only the trained model and a small clean dataset, and scored by three normalized measures: ASR (does the trigger still cause misclassification), ARR (how much clean accuracy did mitigation cost), and RDR (do trigger-bearing images get classified under their true labels again). The load-bearing device is the joint plot of median ARR and RDR against median ASR, with FP and FT marking the rectangle a method must enter to count as an improvement; because no pruning method enters it and only FT-SAM and SAU enter it on the fine-tuning side, the rectangle defines the paper's negative verdict. The second device is scale: 8 attacks, 3 datasets, 4 architectures, 3 poisoning ratios, and 3 data-availability levels turn the comparison into a generalizability test rather than a showcase.","core_discovery":"The central discovery is a negative comparative result. In a benchmark of 122,236 experiments organized as 288 attack scenarios, the authors find that no evaluated pruning method jointly beats the median performance of Fine-Pruning (FP), the 2018 method that prunes the least-activated filters of the final convolutional layer, on attack success rate (ASR), accuracy reduction ratio (ARR), and recovery difference ratio (RDR). Among fine-tuning methods, only FT-SAM and SAU beat the plain fine-tuning (FT) baseline, and both do so at the cost of higher median accuracy reduction, while their recovery performance stays poor. The evaluation also shows that dynamic-trigger attacks are generally easier to mitigate than static ones, that Blended and SSBA are the hardest attacks to defend, that most methods' performance is highly variable across architectures and datasets, and that the poisoning ratio has little effect on most defenses, contrary to the earlier benchmark [17]. The authors conclude that the observational basis many defenses are built on is not universal, that the field's progress relative to the 2018 baselines is mostly incremental, and that future work should focus on recovery accuracy and on constraining overfitting when clean data is scarce.","pith_inferences":["If the fixed-hyperparameter protocol understates tunable newer methods, the paper's central claim is really about off-the-shelf robustness, not ceiling performance; a per-scenario tuned re-run could reorder the rankings without contradicting the paper's data.","Because attack success alone has been the headline metric, the field may have optimized the wrong number; mandating recovery accuracy in defense evaluations could reorder the literature more than any new method.","The excluded methods mark the edge of the claim: approaches that use unlabeled or out-of-distribution data, or that assume knowledge of trigger position and target label, were not benchmarked, so the negative result does not cover defenses that relax the scarce-clean-data assumption.","The finding that dynamic-trigger attacks are easier to mitigate than static ones suggests attack and defense are co-evolving: methods that model the trigger distribution, like SAU, may keep looking strong against the newest attacks while static-trigger attacks like BadNets remain surprisingly effective against most defenses."],"forward_implications":["A defense that looks strong in a single attack and dataset test should not be trusted to transfer, since the benchmark shows ASR, RDR, and ARR swing widely across settings for most methods.","Real-world defenders face an extra burden the papers rarely discuss: after mitigation, ASR and RDR are unobservable, and tuning hyperparameters on a validation set shrinks already-scarce clean data.","RDR is the field's open core: suppressing the attack without restoring correct classification of backdoored inputs still leaves a misbehaving model, and only SAU's proposers even report the metric.","The practical standouts are FT-SAM and SAU, and both sacrifice clean accuracy to gain consistency, so deployment choices involve a real accuracy trade-off.","A low poisoning ratio of 1 to 10 percent does not automatically make an attack harder to mitigate, contrary to what the earlier benchmark [17] suggested."],"supporting_citations":[{"why":"Supplies the benchmark toolkit, default attack configurations, and model architectures that the evaluation builds on, and is the earlier benchmark whose mitigation-only coverage this work extends.","marker":"[17]"},{"why":"Introduces Fine-Pruning (FP) and plain fine-tuning (FT), the two 2018 baselines that all sixteen evaluated methods are compared against.","marker":"[31]"},{"why":"Defines the BadNets attack and the clean/backdoor component-decomposition hypothesis that motivates most pruning-based mitigation approaches.","marker":"[4]"},{"why":"Proposes BNP, the distribution-based pruning method that is among the best-performing pruning defenses in the benchmark.","marker":"[32]"},{"why":"Proposes ANP, the masking-based pruning baseline that AWN and RNP were designed to improve but, in this benchmark, do not.","marker":"[34]"},{"why":"Proposes FT-SAM, one of only two fine-tuning methods found to beat the FT baseline.","marker":"[42]"},{"why":"Proposes SAU, the other method that beats FT and the only evaluated approach whose proposers report recovery accuracy, the basis of RDR.","marker":"[46]"},{"why":"Proposes i-BAU, a recent adversarial-unlearning defense that the benchmark finds underperforms FT, supporting the paper's no-substantial-improvement claim.","marker":"[19]"}],"fun_headline_variants":["Backdoor defenses: 122k tests, few beat 2018 baselines","Most backdoor defenses don't beat 2018 baselines, per 122k tests","No major wins: backdoor defenses vs 2018 baselines from 122k tests","Backdoor defenses rarely beat 2018 baselines in 122k tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that running each defense with the hyperparameter values from its own paper or released code, with no tuning per dataset, architecture, or data-availability level, is a fair representation of how well that defense can perform.","fun_headline_variants_meta":{"raw":{"variants":["Backdoor defenses: 122k tests, few beat 2018 baselines","Most backdoor defenses don't beat 2018 baselines, per 122k tests","No major wins: backdoor defenses vs 2018 baselines from 122k tests","Backdoor defenses rarely beat 2018 baselines in 122k tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2841,"prompt_tokens":1031,"completion_tokens":1810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":1720}},"tokens_in":647,"tokens_out":1810,"duration_ms":12758,"temperature":1.0,"reasoning_tokens":1720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:47:56.161116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 288-scenario grid with per-scenario hyperparameter tuning, grid-searching each method's published hyperparameters on a held-out split for every dataset, architecture, and sample-per-class level, and check whether several newer methods' median ASR, ARR, and RDR jointly move inside the rectangle defined by the Fine-Pruning and fine-tuning baselines; if they do, the claim that newer methods offer no substantial improvement is overturned.","supporting_citations":[{"cited_title":"Backdoorbench: A comprehensive benchmark of backdoor learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark toolkit, default attack configurations, and model architectures that the evaluation builds on, and is the earlier benchmark whose mitigation-only coverage this work extends."},{"cited_title":"Pre-activation distributions expose backdoor neurons,","cited_arxiv_id":null,"evidence_quote":"Proposes BNP, the distribution-based pruning method that is among the best-performing pruning defenses in the benchmark."},{"cited_title":"Enhancing fine- tuning based backdoor defense with sharpness-aware minimization,","cited_arxiv_id":null,"evidence_quote":"Proposes FT-SAM, one of only two fine-tuning methods found to beat the FT baseline."},{"cited_title":"Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples,","cited_arxiv_id":null,"evidence_quote":"Proposes SAU, the other method that beats FT and the only evaluated approach whose proposers report recovery accuracy, the basis of RDR."},{"cited_title":"Adversarial unlearning of backdoors via implicit hypergradient,","cited_arxiv_id":null,"evidence_quote":"Proposes i-BAU, a recent adversarial-unlearning defense that the benchmark finds underperforms FT, supporting the paper's no-substantial-improvement claim."}],"review_version":1}