{"id":"9ae3ac33-4d65-432f-9761-522f52610e8d","arxiv_id":"2411.11006","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BackdoorMBTI is the first backdoor security benchmark and toolkit that covers image, text, and audio modalities with a unified evaluation pipeline.","lead":"A new open-source toolkit, BackdoorMBTI, lets researchers test backdoor attacks and defenses on images, text, and audio using one unified pipeline. It is the first benchmark to cover all three data types, with 11 datasets, 17 attacks, and 7 defenses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The text/audio adaptations of attacks and defenses are not validated against the originals; benchmark conclusions may describe modified variants rather than the named methods.","rationale":"The reader's weakest_assumption — cross-modal fidelity of attacks and defenses — is exactly the load-bearing concern for the central claim. A fidelity failure would not merely weaken a single result; it would invalidate the benchmark's purpose as a comparison tool, because every cross-modal ranking and the noise robustness conclusion would describe modified re-implementations rather than the named published methods. The other issues noted by the reader (single runs, omission of retraining-after-detection, and the uncorrected Mann-Whitney U test) are real but secondary: they reduce confidence in specific comparisons but do not threaten the core coverage claim. My read does not change the reader's CONDITIONAL verdict; it reinforces it. The proposed concrete test is deliberately narrow: it checks whether the published code reproduces original-method behavior on the original modality and whether the claimed-but-unevaluated attacks actually run. Passing that check would substantially strengthen the paper; failing it would require re-labeling the migrated methods as adapted variants and tempering the first-benchmark claim accordingly.","tokens_in":21118,"tokens_out":8890,"duration_ms":165538,"concrete_test":"Run BackdoorMBTI's released implementations of BadNets on CIFAR-10 and STRIP on SST-2 and SpeechCommands, and compare ASR, CAC, RAC, and detection F1 against the original public implementations under identical dataset, model, poison ratio, and hyperparameter settings. Also run each of the five attacks absent from Table 5 on a small subset to confirm they successfully poison. If the adapted methods differ materially (e.g., ASR shift by more than 2 points) or the absent attacks fail to run, the coverage and fidelity claims need revision; if they match, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BackdoorMBTI is the first benchmark supporting image, text, and audio backdoor attack and defense evaluation. For that claim to be useful, the text/audio implementations must be faithful to the named methods. The paper does not establish this. Section 5.3 says trigger generation was implemented 'by referring to open-source backdoor attacks', and Section 6.2.2 concedes 'defense methods often require modifications to achieve improved performance after migration'. No diff, configuration list, or validation against original published numbers is provided. Table 5 reports only 4 attacks per modality (12 of the claimed 17) and uses a single run without error bars, so there is no evidence that the unshown attacks or the re-implemented methods behave as published. If a migrated STRIP or BadNets differs in trigger placement, entropy threshold, pruning criterion, or fine-tuning schedule, every attack/defense comparison and the Q2 noise conclusion describes the re-implementation, not the original method. The Section 7 'Limited Migrations' note acknowledges incompatibilities but does not identify which modifications were made. This makes the benchmark's value for defense evaluation unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BackdoorMBTI, a benchmark and open-source toolkit for evaluating backdoor attacks and defenses across three data modalities: image, text, and audio. The authors claim this is the first benchmark to cover all three modalities, integrating eleven datasets, seventeen attacks, and seven defenses, along with a unified pipeline for data processing, poisoning, training, and evaluation. The toolkit also includes a noise generator that adds data noise and label noise to simulate real-world conditions, and the paper reports experiments on CIFAR-10, SST-2, and SpeechCommands with four attacks per modality and seven defenses. The central empirical claims are that migrated attacks remain highly effective across modalities, that certain migrated defenses (AC, STRIP, FT, FP) generalize well, and that label noise significantly improves defense detection accuracy, based on a Mann-Whitney U test.","tokens_in":21299,"tokens_out":5085,"duration_ms":49496,"significance":"If the fidelity of the migrated attacks and defenses were established and the results were reliable, BackdoorMBTI would provide a useful, extensible testbed for multimodal backdoor evaluation, with concrete assets including open-source code, poisoned datasets, and trained models. The inclusion of a noise generator is a genuinely useful contribution, as most prior benchmarks ignore data-quality and label-error factors. The paper's coverage claim is plausible and important: no prior benchmark supports image, text, and audio in one framework. However, the current manuscript does not verify that the text/audio implementations faithfully represent the original published methods, and the experimental evidence is too thin to support the quantitative conclusions. These issues are load-bearing for the benchmark's core value as a defense-evaluation tool.","major_comments":[{"comment":"The central value of BackdoorMBTI is that it evaluates the named attacks and defenses after migration to text and audio. However, the paper does not establish that the migrated implementations are faithful to the original methods: §5.3 says trigger generation was implemented \"by referring to open-source backdoor attacks\", and §6.2.2 concedes that \"defense methods often require modifications to achieve improved performance after migration\", but no list of modifications, configuration diffs, or validation against original published accuracy/ASR numbers is provided. Section 7's \"Limited Migrations\" paragraph acknowledges that ABL, CLP, and NC could not be adapted to all modalities, yet it does not say which modifications were made for the methods that were migrated. Without this, every benchmark result and the Q2 noise conclusion may describe re-implementations rather than the published methods, which undermines the benchmark's stated purpose of comparing backdoor defenses.","section":"§5.3, §6.2.2, §7"},{"comment":"Table 5 is the only source of Q1 evidence, but it reports a single run per attack-defense pair with no error bars, no seeds, and no variance estimates. It also covers only 12 of the 17 implemented attacks (four per modality), so the unimplemented or unreported attacks (e.g., DynaTrigger, BITE, SBAT, PNoise) are never evaluated and cannot support the coverage claim. Claims such as \"attacks migrated to the text and audio domains demonstrate excellent effectiveness compared to those in the original domain\" (Section 6.2) are not supported without run-to-run variability information.","section":"Table 5, §6.1.2"},{"comment":"The Q2 finding that label noise improves defense detection accuracy is based on a single Mann-Whitney U test with p=0.012, uncorrected for multiple comparisons, and the paper does not report the sample composition (which attacks/defenses are included), an effect size, or confidence intervals. The average DAC values (62.8% normal, 65.5% data noise, 68.2% label noise) are close, and the data-noise comparison is not significant (p=0.197); the broader statement \"noise factors enhance model robustness, thereby improving defense performance\" is therefore not supported by the evidence presented.","section":"§6.3"},{"comment":"Section 7 states that BackdoorMBTI \"currently supports a diverse range of single-modality applications, including image, text, audio, video, and contrastive learning, as well as the multimodal application VQA,\" which contradicts the Abstract and Table 1, where only image, text, and audio are listed as supported modalities and no VQA dataset appears in Table 2. This inconsistency matters for the \"first multimodal benchmark\" claim: either video/VQA are supported and should be described and evaluated, or the sentence should be corrected to avoid overstating the current scope.","section":"§7 vs Abstract/Table 1"},{"comment":"The paper excludes retraining after detection because \"the retraining's efficacy is quite low and lacks meaningful reference value\" (Section 6.2), but no experiment or analysis is shown to support this claim. Retraining on sanitized data is a standard part of the defense evaluation pipeline in the backdoor literature, and its omission without evidence weakens the benchmark's completeness for defense evaluation.","section":"§6.2"}],"minor_comments":[{"comment":"The section heading \"Limititions\" is a typo and should read \"Limitations\".","section":"§7"},{"comment":"The column header \"NCAttack\" in Table 5 should be formatted as \"NC Attack\" for consistency with the other defense names.","section":"Table 5"},{"comment":"Reference [2] cites \"Blind backdoors in deep learning models\" but is used in Table 1 to denote the Backdoor101 benchmark; the citation appears mismatched. Reference [41] is a duplicate of reference [40].","section":"References"},{"comment":"The noise settings (25% Gaussian noise with mean 0 and variance 1, text CER 0.1) are chosen without sensitivity analysis for text and audio; Appendix C studies noise-level variation only on CIFAR-10, so the generality of the chosen settings across modalities is not demonstrated.","section":"§6.1.3, Appendix C"},{"comment":"The phrase \"the security of these models remains a crucial concern. Specifically, there are no existing backdoor benchmarks targeting multimodal applications\" is slightly misleading because the paper's own Section 7 claims support for VQA and video; please align the abstract with the actual scope.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's header identifies it as a KDD '25 camera-ready, but the manuscript as provided contains a significant scope contradiction (Section 7 claims video/VQA support while the abstract and Table 1 list only three modalities) and a reference mismatch for Backdoor101. These issues, together with the missing validation of migrated attacks/defenses, should be resolved before the benchmark is used as a reference standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First off: the core claim checks out. I don't know of another benchmark that covers image, text, and audio backdoor attacks and defenses in one pipeline. The Table 1 comparison is accurate as far as I can tell. The open-source release and the unified pipeline are genuinely useful. The noise generator is a nice addition—real deployments have noisy data and bad labels, and existing benchmarks mostly ignore that.\n\nWhat's new here is integration, not algorithms. The attacks and defenses are largely existing methods ported across modalities. That's fine for a benchmark; the value is in having a shared testbed.\n\nThe soft spots are about evidence, not the idea. The biggest one: they never show that the text/audio adaptations behave like the originals. Section 5.3 says triggers were implemented “by referring to open-source backdoor attacks,” and Section 6.2.2 concedes defenses “often require modifications.” But there's no diff, no config list, no validation against published numbers. The limited-migrations footnote in Section 7 acknowledges constraints but doesn't say which modifications were made. For a benchmark whose purpose is comparing methods, that's a real gap. Table 5 also reports single runs without error bars, and only 4 of the 17 attacks appear in the main results—the rest are waved to the GitHub repo. The Q2 noise conclusion (label noise improves detection accuracy) rests on one uncorrected Mann-Whitney test with p=0.012; that's suggestive, not strong. And the decision to exclude retraining-after-detection because “efficacy is quite low” is defensible but too thinly justified.\n\nNone of this sinks the central claim. The paper is honest about its limitations, the related-work coverage is solid, and the citation pattern looks normal. It's a useful toolkit paper that needs a validation pass and proper statistics before its empirical conclusions can be trusted.\n\nI'd send this to a serious referee. It deserves a major revision, not a desk reject. If you work on backdoor defenses, it's worth a read.","headline":"First benchmark to cover image, text, and audio backdoor learning in one pipeline—a real gap and a useful open-source toolkit, but the unvalidated cross-modal adaptations and single-run statistics keep the empirical findings provisional.","tokens_in":21880,"tokens_out":2786,"would_cite":true,"duration_ms":31272,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces BackdoorMBTI, which it claims is the first backdoor-learning benchmark and toolkit to cover image, text, and audio in a single unified evaluation pipeline.","keywords":["backdoor attack","backdoor defense","multimodal benchmark","data poisoning","trojan detection","multimodal evaluation","noise robustness"],"falsifier":"Compare a migrated attack, such as BadNets, executed inside BackdoorMBTI against the original public implementation on the same dataset, model, and poison ratio; if the trigger pattern, attack success rate, or clean accuracy diverges substantially, the benchmark's cross-modal numbers cannot be read as faithful measurements of the original methods.","tokens_in":20865,"feed_emoji":"🛡️","tokens_out":6848,"duration_ms":65866,"temperature":0.7,"pith_summary":"The paper introduces BackdoorMBTI, a toolkit and benchmark that claims to be the first to cover backdoor attack and defense evaluation across image, text, and audio in one unified pipeline. It assembles eleven datasets, seventeen attacks, and seven defenses, and supplies poisoned datasets and trained models so defense methods can be compared on a common footing. The authors also add controlled data noise and label noise to mimic real-world conditions, and report that attacks stay effective under noise while defenses tend to improve, especially with label noise. If the benchmark works as intended, researchers get a standard multimodal testbed for backdoor defense evaluation.","feed_headline":"Toolkit unifies backdoor defense tests across image, text, audio","feed_subtitle":"Eleven datasets, seventeen attacks, seven defenses, one pipeline—with real-world noise built in","key_machinery":"The unifying mechanism is the four-stage pipeline: data processing, data poisoning, backdoor training, and evaluation. Its load-bearing piece is the backdoor poisoner, implemented as a dataset wrapper that standardizes trigger generation, sample synthesis, and label modification for each attack while still exposing custom training procedures for training-control attacks. The noise generator, placed before poisoning, injects Gaussian data noise and random label noise with controllable intensity, giving the benchmark a standardized way to test defenses under real-world conditions.","core_discovery":"BackdoorMBTI's central claim is that a single extensible benchmark can host backdoor learning across all three modalities, image, text, and audio, rather than treating each modality's attacks and defenses in separate pipelines. The authors claim to be the first to design such a benchmark, integrating eleven datasets, seventeen attacks, and seven defenses into a unified pipeline from data processing, data poisoning, backdoor training, to evaluation. They report that migrated attacks achieve high success rates, generally above 80% and above 95% for text, while defense performance varies: universal methods like fine-tuning and STRIP transfer well, whereas pruning-based methods fail on text models like BERT because BERT lacks batch normalization layers. They also report that noise factors, Gaussian data noise and mislabeling, do not weaken attacks and can improve defense detection accuracy, with label noise showing a statistically significant improvement.","pith_inferences":["Editorial inference: the paper's reported benefit of label noise for detection is consistent with noisy labels making poisoned samples easier to separate, but the paper does not establish that mechanism; a follow-up could test whether detection gains vanish when noise is applied after poisoning rather than before.","Editorial inference: because the benchmark freezes poisoned datasets before training, reproducibility depends on the exact artifact versions; future re-evaluations should record which dataset and model release they used, since trigger placement and noise seed can affect comparisons.","Editorial inference: a unified multimodal leaderboard would make cross-modal transferability a first-class selection criterion for defense methods, potentially shifting research incentives away from image-only backdoor defense."],"forward_implications":["For the first time, the same poisoning ratios, metrics, and evaluation pipeline can compare image, text, and audio backdoor defenses directly.","Defenses with universal mechanisms, such as fine-tuning and STRIP, are the ones the paper finds consistently effective across modalities; pruning methods that rely on batch normalization layers do not transfer to BERT-style text models.","Backdoor attacks migrated to text and audio retain high success rates, above 95% for text in the experiments, so multimodal deployments should assume the same poisoning risk as image models.","Benchmark evaluations should report noise conditions, since the paper finds label noise significantly improves detection accuracy, from an average of 62.8% under normal conditions to 68.2%, while data noise does not.","New attacks and defenses can be added to the toolkit and evaluated against existing results through the same pipeline, making cross-modal comparison a reusable community resource."],"supporting_citations":[{"why":"Existing benchmark that already covers image and text but integrates them separately; it is the main comparison point for the paper's first-to-cover-all-three claim.","marker":"[82]"},{"why":"Image-only benchmark whose architecture the paper says is hard to extend to new modalities, motivating the unified design.","marker":"[56]"},{"why":"Existing image-and-text benchmark with separate integrations, used to establish that no prior benchmark covers audio.","marker":"[2]"},{"why":"BadNets supplies the classic visible-trigger attack that anchors migrated attack results in all three modalities.","marker":"[20]"},{"why":"STRIP is the entropy-based detection defense whose near-perfect recall in text supports the defense-transfer findings.","marker":"[18]"},{"why":"Fine-tuning and fine-pruning are the post-training defenses whose consistent transfer across modalities is a main experimental result.","marker":"[42]"},{"why":"ABL is the poison-suppression defense that performs well on images but fails on text and audio, supporting the modality-dependence finding.","marker":"[37]"},{"why":"CLP's pruning-based defense fails on BERT because it depends on batch normalization layers, evidence for the architecture-dependence conclusion.","marker":"[92]"},{"why":"Neural Cleanse is the trigger-reverse defense whose failure against BPP underlines the conditions under which detection defenses transfer.","marker":"[77]"}],"fun_headline_variants":["First multimodal backdoor benchmark: image, text, audio","BackdoorMBTI: one toolkit for backdoor attacks and defenses across modalities","Unified backdoor benchmark spans image, text, audio with 11 datasets","First backdoor defense toolkit to cover three modalities in one pipeline","BackdoorMBTI: multimodal attacks, defenses, and noise robustness in one suite"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the attacks and defenses migrated across modalities still implement the same algorithms as the originals; if the adaptations change how they work, the benchmark numbers no longer represent the methods being compared.","fun_headline_variants_meta":{"raw":{"variants":["First multimodal backdoor benchmark: image, text, audio","BackdoorMBTI: one toolkit for backdoor attacks and defenses across modalities","Unified backdoor benchmark spans image, text, audio with 11 datasets","First backdoor defense toolkit to cover three modalities in one pipeline","BackdoorMBTI: multimodal attacks, defenses, and noise robustness in one suite"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3543,"prompt_tokens":1025,"completion_tokens":2518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2423}},"tokens_in":641,"tokens_out":2518,"duration_ms":19406,"temperature":1.0,"reasoning_tokens":2423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:00:52.650715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare a migrated attack, such as BadNets, executed inside BackdoorMBTI against the original public implementation on the same dataset, model, and poison ratio; if the trigger pattern, attack success rate, or clean accuracy diverges substantially, the benchmark's cross-modal numbers cannot be read as faithful measurements of the original methods.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Existing benchmark that already covers image and text but integrates them separately; it is the main comparison point for the paper's first-to-cover-all-three claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Image-only benchmark whose architecture the paper says is hard to extend to new modalities, motivating the unified design."},{"cited_title":"In 2021 IEEE Symposium on Security and Privacy (SP)","cited_arxiv_id":null,"evidence_quote":"CLP's pruning-based defense fails on BERT because it depends on batch normalization layers, evidence for the architecture-dependence conclusion."}],"review_version":1}