{"id":"1d5850db-d057-4f5e-b334-621c717779cc","arxiv_id":"2412.16254","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A dynamic ensemble of BERT, RoBERTa, and ALBERT with randomized smoothing, masked inference, and TextFooler adversarial training is reported to keep 82-87% accuracy under attack on AG News and IMDB, far above prior defenses.","lead":"This paper describes ARDEL, a defensive scheme that teams up three language models (BERT, RoBERTa, ALBERT), smooths their outputs, and reweights them per input to resist adversarial text attacks. The authors report much higher accuracy under attack than prior defenses, but the dynamic weighting and attack-detection pieces are never tested on their own.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The full ARDEL system is never evaluated: Tables V-VII test only 'RSMI + Ensemble' without the meta-model or detector, so the reported gains may come entirely from a static ensemble.","rationale":"The most defensible reading of the paper is that ensembling three RSMI-trained models improves robustness; that claim has external plausibility and is independently grounded in the RSMI baseline. What is not supported is the specific claim of the title and abstract: that dynamic ensemble learning, via meta-model weighting plus adversarial-pattern detection, is what delivers the gains. The ablations in Tables V-VII stop at 'RSMI + Ensemble', which omits the two components that define ARDEL. On the main datasets (Tables I-IV), there is no static-ensemble row at all, so the only contrast is ARDEL versus single-model defenses. This is a classic confound: the ensemble itself, not the dynamic mechanism, can explain the accuracy jump. The reader's verdict (REJECT) is appropriate because the evidence structure cannot support the central claim as stated, though the rejection is about missing controls, not about the impossibility of the concept. I agree with the reader's weakest assumption; the missing full-ARDEL ablation and absent static-ensemble control are the same core defect. No formal verification or code is provided, and the per-row ARDEL entries are ambiguous, compounding the reproducibility problem. A single controlled experiment, full ARDEL versus equal-weight ensemble versus RSMI, would settle whether the dynamic components matter; if a static ensemble reproduces the gains, the paper's contribution reduces to a known effect.","tokens_in":10175,"tokens_out":5637,"duration_ms":50410,"concrete_test":"Run a controlled comparison on AG News and IMDB using the same random splits and attack settings: (i) full ARDEL as specified in Section III/IV, (ii) an equal-weight ensemble of the same three RSMI-smoothed, masked-inference models with no meta-model and no detector, and (iii) RSMI alone. If condition (ii) matches or exceeds (i) within one point on accuracy under attack, the dynamic weighting and detection modules are not load-bearing and the central claim is unsupported. Also re-run FreeLB++, InfoBERT, and RSMI on these exact splits to rule out split-induced inflation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ARDEL's dynamic ensemble mechanism yields state-of-the-art robustness is not supported by any evaluation of the full system. In Section V, Tables I-IV report ARDEL (our) with roughly 82-87% accuracy under TextFooler, BERT-Attack, and TextBugger, but the ablations in Tables V-VII compare only Fine-Tuned, RSMI, and 'RSMI + Ensemble' on QNLI and MNLI; the meta-model weighting (Section IV, Step 3) and OneClassSVM detector (Section IV, Step 7) are never included. Thus the 20-30 point gains over RSMI may be entirely an ensemble-size effect: three smoothed and masked-inference models voting can match these numbers without dynamic reconfiguration. A static-ensemble control is absent from Tables I-IV, so the load-bearing novelty of the paper, the dynamic adjustment, is untested. Additionally, listing ARDEL separately under BERT-, RoBERTa-, and ALBERT-base rows is ambiguous: Section III defines ARDEL as a single ensemble of all three architectures, which would give one accuracy per dataset per attack, not one per base LM. Without a precise configuration and without a static-ensemble baseline, the reported improvements cannot be attributed to the proposed method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ARDEL, a defense for pre-trained language models that combines randomized smoothing and masked inference (following Moon et al.'s RSMI) with an ensemble of BERT, RoBERTa, and ALBERT whose weights are adjusted per input by a RandomForest meta-model, together with a OneClassSVM adversarial-pattern detector and adversarial training using TextFooler examples. Evaluation on AG News and IMDB (Tables I–IV) reports large gains in accuracy under TextFooler, BERT-Attack, and TextBugger over Fine-Tuned, FreeLB++, InfoBERT, and RSMI baselines, as well as increased attack-query counts. An ablation on QNLI and MNLI (Tables V–VII) compares only Fine-Tuned, RSMI, and 'RSMI + Ensemble.' The paper claims that ARDEL significantly improves robustness over existing methods.","tokens_in":10242,"tokens_out":5578,"duration_ms":47909,"significance":"If the reported numbers were produced by the full ARDEL pipeline on splits matched to the baselines, the contribution would be practically important: it would show roughly 82–87% accuracy under attack versus 52–74% for RSMI, and 1.2–1.6× more attack queries, a substantial advance over the tested 2019–2023 defenses. The paper is also interesting as an application of dynamic ensemble weighting to adversarial NLP. However, as presented, the evidence does not support attributing these gains to the dynamic components, and the missing static-ensemble control and baseline-match documentation leave the central claim unsupported. No code or hyperparameter details are provided, so the results are not reproducible from the manuscript alone.","major_comments":[{"comment":"The ablation compares only 'Fine-Tuned,' 'RSMI,' and 'RSMI + Ensemble' on QNLI and MNLI. The meta-model weighting (Section IV, Step 3) and OneClassSVM detection module (Section IV, Step 7) are never evaluated in any table. Consequently, the 'ARDEL (our)' entries in Tables I–IV cannot be distinguished from a static equal-weight ensemble of three RSMI-trained models. The paper's load-bearing novelty—dynamic reconfiguration—is untested; a static-ensemble control must be included in Tables I–IV, and the full ARDEL system must be added to the ablation.","section":"Section V, Tables V–VII"},{"comment":"The text says 'We create a random train, validation, and test splits' but does not state that FreeLB++, InfoBERT, and RSMI were re-run on those same splits or on the same 1000-example attack subset. If the baseline figures are taken from prior publications using standard splits, the comparison is confounded and the 20–30 point gains would be inflated. The authors must state explicitly whether all baselines were re-run under identical conditions and should make the evaluation scripts available for verification.","section":"Section V, experimental setup"},{"comment":"ARDEL is defined in Section III as a single ensemble of BERT, RoBERTa, and ALBERT, but Tables I–IV list separate 'ARDEL (our)' results under each of BERT-Base, RoBERTa-Base, and ALBERT-Base rows. This is internally inconsistent: one would expect one ARDEL row per dataset per attack, not one per base LM. The exact ensemble composition for each row, the noise standard deviation, the masking probability, the meta-model input features, and the OneClassSVM features are not specified, so the configuration behind each reported number is ambiguous and not reproducible.","section":"Sections III and V, Tables I–IV"},{"comment":"Adversarial training is performed with TextFooler-generated examples only, yet TextFooler is also one of the evaluation attacks. The reported TextFooler robustness is therefore partially an artifact of training-distribution overlap. The evaluation should include attacks not used in training (or an ablation that removes the training attack from the test set) to demonstrate generalization to unseen attack patterns.","section":"Section IV, Steps 8–9"},{"comment":"All reported values are point estimates averaged over '5 different random seeds' without standard deviations or confidence intervals, while the accuracy-under-attack metric is computed on a randomly selected 1000-example subset. Given the random-split design, this makes it impossible to judge whether differences between methods (e.g., 83.4 vs 83.5 in Table II) are meaningful. The paper should report error bars and specify the query-count protocol for cases where an attack fails to succeed.","section":"Section V, metrics and statistics"}],"minor_comments":[{"comment":"The caption reads 'NUMBER OF ADVERSARIAL QURIES'; 'QURIES' should be 'QUERIES'.","section":"Table III caption"},{"comment":"Reference [10] contains a typo: 'Estimating raining data influence' should be 'Estimating training data influence'.","section":"References"},{"comment":"References [27] and [28] are listed in the bibliography but are not cited anywhere in the text; please add appropriate citations or remove them.","section":"References"},{"comment":"Both the MNLI accuracy table and the MNLI query-count table are numbered 'TABLE VII'; renumber the second table as TABLE VIII.","section":"Tables VI and VII"},{"comment":"The text 'TextFoolerJin219' appears to be a typo for 'TextFoolerJin2019'; please correct it.","section":"Section IV, Step 8"},{"comment":"The affiliation line contains 'Praxis Businesss School'; it should be 'Praxis Business School'.","section":"Author affiliation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for acceptance in its current form. The central evaluation gap—no evaluation of the full ARDEL system and no static-ensemble control—is substantial, but it could in principle be repaired by adding the missing experiments, re-running baselines on matched splits, and providing code and hyperparameters. I would be willing to review a revised version. There is also a secondary concern about the incremental novelty relative to the cited RSMI and ensemble works and to the authors' own prior attack papers (refs [3], [4]), though this is not the main basis for my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nThe short version: the paper reports a big robustness win (82–87% accuracy under attack vs 52–67% for RSMI), but the evidence as presented doesn't support its own headline. The load-bearing flaw is that the full ARDEL system is never evaluated. Tables I–IV report 'ARDEL (our)' numbers, but the ablations in Tables V–VII only test 'RSMI + Ensemble' — no meta-model weighting, no OneClassSVM detector. So the claimed 20–30 point gains may be entirely from having three smoothed, masked-inference models vote instead of one. A static equal-weight ensemble is the obvious control, and it's missing.\n\nWhat's genuinely here: the idea of per-input weighting with an adversarial-pattern detector is a plausible extension of RSMI, and the paper cites the relevant prior work (RSMI, TextFooler, BERT-Attack, etc.). It also reports query counts, which is a useful secondary metric. The writing is readable and the method is described in enough detail to be re-implementable in principle.\n\nBut the soft spots are structural, not cosmetic. The baselines (Fine-Tuned, FreeLB++, InfoBERT, RSMI) are compared against a three-model ensemble, and the paper never shows they were re-run on the same random splits. It says 'We create a random train, validation, and test splits' but doesn't say the baselines were re-run on them; if the baseline numbers come from standard splits, the comparison is inflated. There are no error bars despite claiming 5 seeds, and no hyperparameters for the meta-model, smoothing noise, or masking probability. The tables list ARDEL separately under BERT-, RoBERTa-, and ALBERT-base rows, which is ambiguous: if ARDEL is a single ensemble of all three, there should be one number per dataset per attack, not one per base model.\n\nFor someone working on adversarial NLP defenses, this is a useful case study in evaluation pitfalls; the fix is clear. As submitted, I wouldn't cite it. But the potential impact is real and the hypothesis is testable, so I'd send it to review rather than desk-reject, with instructions that the control experiment is mandatory.","headline":"ARDEL's reported gains look like an ensemble-size effect because the full system is never evaluated; this needs a proper control before the numbers mean anything.","tokens_in":10957,"tokens_out":3231,"would_cite":false,"duration_ms":27512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dynamic language-model ensemble reports 83-87% accuracy under attack.","keywords":["adversarial robustness","ensemble learning","pre-trained language models","dynamic weighting","randomized smoothing","masked inference","text classification","adversarial attacks"],"falsifier":"Re-run the baselines on ARDEL's exact random train, validation, and test splits and re-run ARDEL with the meta-model and detection module disabled: if the baseline gap collapses or the equal-weight ensemble matches the full pipeline, the dynamic components and the claimed advantage are not doing the work.","tokens_in":9780,"feed_emoji":"🛡️","tokens_out":5336,"duration_ms":42576,"temperature":0.7,"pith_summary":"This paper proposes ARDEL, a defense that combines three pre-trained language models into an ensemble whose per-input weights are set by a meta-model. The ensemble also uses randomized smoothing with masked inference, adversarial pattern detection, and adversarial training on generated attack examples. The authors claim that under TextFooler, BERT-Attack, and TextBugger, ARDEL keeps classification accuracy around 82-87% on AG News and IMDB, whereas the strongest prior defense tested stays near 52-67%, and that attacks need roughly 1.2-1.6 times more queries to succeed. If these numbers hold, the scheme offers a practical way to harden NLP classifiers without changing the underlying model architectures.","feed_headline":"Dynamic language-model ensemble reports 83-87% accuracy under attack","feed_subtitle":"A per-input meta-model reweights BERT, RoBERTa, and ALBERT; reported attack success drops sharply versus prior defenses.","key_machinery":"The load-bearing mechanism is ARDEL's full pipeline: three diverse pre-trained language models (BERT, RoBERTa, ALBERT); a meta-model that predicts per-input ensemble weights; an adversarial pattern detection module built on attention distribution analysis and syntactic anomaly detection; randomized smoothing with masked inference; and adversarial training augmented with TextFooler, TextBugger, and BERT-Attack examples. The detector and meta-model are the dynamic part: for each input they reweight the ensemble in real time so models more vulnerable to the detected attack pattern contribute less. Dropout regularization is applied during training. The paper's own ablation, however, evaluates only randomized smoothing plus masked inference with an equal-weight ensemble, omitting the meta-model and the detector.","core_discovery":"On its own terms, the paper's central discovery is that dynamically reconfiguring an ensemble of BERT, RoBERTa, and ALBERT models per input, down-weighting models judged susceptible to detected attack patterns and up-weighting more robust ones, yields substantially higher accuracy under attack than static defenses such as adversarial training alone, information-theoretic regularization, or randomized smoothing with masked inference. The reported Tables I-IV show ARDEL at roughly 82-87% accuracy under TextFooler, BERT-Attack, and TextBugger on AG News and IMDB, compared with roughly 52-67% for RSMI, the best-performing baseline, and show ARDEL forcing more adversarial queries before a successful attack. The authors interpret this as evidence that model diversity plus input-dependent weighting, rather than any single model's strength, is what carries adversarial robustness.","pith_inferences":["The provided ablation suggests an equal-weight ensemble of RSMI-trained models already produces most of the reported improvement; the meta-model and detection module may contribute less than the dynamic-weighting narrative implies.","A direct test would compare the full ARDEL pipeline against its components on the same random splits; if equal-weight ensembling alone matches the full system, the extra machinery is optional.","Because the baselines' splits are not described as re-run on the same random splits, a re-evaluation under matched conditions could shrink the 20-30 point gap; until then the headline comparison should be read cautiously."],"forward_implications":["If ARDEL's numbers reproduce, NLP classifiers can be made substantially more attack-resistant by combining already-pretrained models rather than retraining a single architecture.","Attackers would need roughly 1.2-1.6 times more queries per successful attack on the tested datasets, raising the cost of black-box attacks.","Because the ensemble reweights per input, robustness gains are not tied to a single model family; the same pipeline should extend to stronger base models as they appear.","The modular design means new attack detectors or base models could be dropped in without redesigning the defense."],"supporting_citations":[{"why":"Supplies the randomized smoothing and masked inference method that ARDEL builds on and that is the RSMI baseline in the comparisons.","marker":"[15]"},{"why":"TextFooler is one of the three attack recipes used to generate adversarial examples during training and to evaluate robustness.","marker":"[16]"},{"why":"TextBugger is the character-level attack used in both adversarial training and evaluation.","marker":"[17]"},{"why":"BERT-Attack is the subword-level attack used in evaluation; the paper adopts this work's evaluation constraints.","marker":"[18]"},{"why":"The BERT base model that ARDEL fine-tunes as one ensemble member and as a baseline.","marker":"[23]"},{"why":"The RoBERTa base model that ARDEL fine-tunes as one ensemble member and as a baseline.","marker":"[24]"},{"why":"The ALBERT base model that ARDEL fine-tunes as one ensemble member.","marker":"[25]"},{"why":"InfoBERT is one of the baseline defenses compared in Tables I and II.","marker":"[26]"},{"why":"FreeLB provides the adversarial training baseline that FreeLB++ extends and that is compared in the tables.","marker":"[28]"}],"fun_headline_variants":["Dynamic ensemble reweights models per input to beat attacks","Per-input model weighting lifts adversarial accuracy to 87%","Adaptive ensemble thwarts text attacks on BERT, RoBERTa, ALBERT","Input-aware ensemble cuts attack success for NLP models","Dynamic ensemble boosts robustness against adversarial text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the ARDEL numbers in Tables I-IV come from the full pipeline, meta-model weighting and adversarial pattern detection included, tested on the same splits as the baselines; the paper's ablation only evaluates an equal-weight ensemble, so this is not shown.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic ensemble reweights models per input to beat attacks","Per-input model weighting lifts adversarial accuracy to 87%","Adaptive ensemble thwarts text attacks on BERT, RoBERTa, ALBERT","Input-aware ensemble cuts attack success for NLP models","Dynamic ensemble boosts robustness against adversarial text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3840,"prompt_tokens":893,"completion_tokens":2947,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2867}},"tokens_in":509,"tokens_out":2947,"duration_ms":20765,"temperature":1.0,"reasoning_tokens":2867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:18:31.748166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the baselines on ARDEL's exact random train, validation, and test splits and re-run ARDEL with the meta-model and detection module disabled: if the baseline gap collapses or the equal-weight ensemble matches the full pipeline, the dynamic components and the claimed advantage are not doing the work.","supporting_citations":[{"cited_title":"Randomized smoothing with masked inference for adversarially robust text classifications,","cited_arxiv_id":null,"evidence_quote":"Supplies the randomized smoothing and masked inference method that ARDEL builds on and that is the RSMI baseline in the comparisons."},{"cited_title":"Is bert really robust? A strong baseline for natural language attack on text classification and entailment,","cited_arxiv_id":null,"evidence_quote":"TextFooler is one of the three attack recipes used to generate adversarial examples during training and to evaluate robustness."},{"cited_title":"TextBugger: Generating adversarial text against real -world applications,","cited_arxiv_id":null,"evidence_quote":"TextBugger is the character-level attack used in both adversarial training and evaluation."},{"cited_title":"BERT-Attack: Adversarial attack against BERT using BERT,","cited_arxiv_id":null,"evidence_quote":"BERT-Attack is the subword-level attack used in evaluation; the paper adopts this work's evaluation constraints."},{"cited_title":"BERT: Pre - training of deep bi -directional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"The BERT base model that ARDEL fine-tunes as one ensemble member and as a baseline."},{"cited_title":"ALBERT: A lite BERT for self -supervised learning of language representations,","cited_arxiv_id":null,"evidence_quote":"The ALBERT base model that ARDEL fine-tunes as one ensemble member."},{"cited_title":"InfoBERT: Improving robustness of lang uage models from an information theoretic perspective,","cited_arxiv_id":null,"evidence_quote":"InfoBERT is one of the baseline defenses compared in Tables I and II."}],"review_version":1}