{"id":"ce7462fd-7fbe-4cc4-8ffb-cad64fbea4f6","arxiv_id":"2509.09706","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An empirical study claiming near-perfect adversarial robustness for RoBERTa and Flan-T5 and high vulnerability for BERT, based on small and internally inconsistent test runs.","lead":"This paper tested how well three language models (BERT, RoBERTa, Flan-T5) survive two known text-attack methods. It claims RoBERTa and Flan-T5 were never fooled while BERT was often fooled, but the evidence is based on very small, inconsistently reported samples.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0% ASR claim for RoBERTa/Flan-T5 rests on 7 and 35 attack attempts; with 7 successes the 95% CI upper bound is ~41%, so 'perfect robustness' is not supported.","rationale":"The reader's weakest-assumption pinpoint is correct and load-bearing: the 0% ASR is computed over a tiny number of correctly classified examples, making the headline 'perfect robustness' claim statistically unsupportable. My own reading of the manuscript confirms this is the central quantitative assertion, and Equation 3 indeed defines ASR over the subset D' of originally correct examples, so the tiny sample size is the number of attack attempts, not the full dataset. The internal contradiction between Table 3 (35% original accuracy) and Table 4 (20% original accuracy) for the same RoBERTa/BERTAttack condition is a further aggravating factor, but the sample-size problem alone is sufficient to invalidate the central claim. I therefore agree with the reader's REJECT verdict; no adjustment is needed. I am not raising a different concern about external consensus or novelty, which are secondary. The recommendation to strengthen the evaluation with a larger sample and exact confidence intervals is the natural way to test the claim.","tokens_in":7079,"tokens_out":3652,"duration_ms":35882,"concrete_test":"Reproduce the BERTAttack evaluation on RoBERTa-base over a held-out sample of at least 200 correctly classified SQuAD v2 examples (and a comparable sample for Flan-T5), and report the ASR with an exact binomial confidence interval. If the observed ASR is 0 out of 200, the 95% upper bound is about 1.5%, which would support high robustness. Also reconcile the Table 3 vs. Table 4 discrepancy in RoBERTa's original accuracy; if the accuracy cannot be reproduced, reject the reported scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RoBERTa-Base and Flan-T5 are 'perfectly robust' with 0% attack success rates, repeated in Equations 9 and 14-15 as R=1.0. This is the paper's main quantitative conclusion. But the evidence for it is statistically negligible. In Table 3, RoBERTa-Base under BERTAttack shows 0 successful attacks out of only 7 attempted attacks (7 failed, 13 skipped, original accuracy 35% of the 20 examples). With 0/7 successes, the 95% Clopper-Pearson confidence interval for the true ASR is (0, 0.41): the data are compatible with an ASR as high as 41%. For Flan-T5, the paper does not even report the number of attack attempts; if the 35% original accuracy corresponds to 35 correct examples, the upper bound is about 10%. Therefore, the reported 0% ASR cannot distinguish perfect robustness from substantial vulnerability. Additionally, Table 3 and Table 4 directly contradict each other for the same condition: Table 3 gives RoBERTa's original accuracy under BERTAttack as 35%, while Table 4 gives 20% for 'BERTAttack on RoBERTa'. This internal inconsistency further undermines the reliability of the 0% numbers. Since the abstract, Equations 8-15, and the discussion all build on this 0% ASR, the central conclusion is not supported by the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of BERT-base, RoBERTa-base, and Flan-T5 under TextFooler and BERTAttack attacks, using PromptBench on SST-2 and SQuAD v2. It defines attack success rate and a robustness score R = 1 - ASR, reports that BERT-base is highly vulnerable (93.75% ASR under TextFooler, accuracy dropping from 48% to 3%), and claims that RoBERTa-base and Flan-T5 show 'perfect robustness' with 0% ASR. It also introduces a computational efficiency metric and proposes several defense heuristics.","tokens_in":7470,"tokens_out":5629,"duration_ms":56508,"significance":"If the 0% ASR results were reliable, the paper would provide a useful comparative data point on transformer robustness and on the computational cost of adversarial queries. The authors use established attacks, report query counts, include an explicit evaluation algorithm, and candidly list limitations. However, the central empirical claim is not supported by the evidence as presented: the 0% ASR for RoBERTa rests on 7 attack attempts, Flan-T5's attempt count is not reported, and there is a direct internal inconsistency between Tables 3 and 4. These issues undermine the headline conclusion and the derived robustness scores R = 1.0.","major_comments":[{"comment":"The claim that RoBERTa-base has 0% attack success rate and R = 1.0 is based on only 7 attack attempts (0 successes, 7 failures, 13 skips). With 0/7 successes, the 95% Clopper-Pearson upper confidence bound for the true ASR is about 41%, so the data are compatible with substantial vulnerability. Flan-T5's 0% ASR in Table 4 is accompanied by no attempt count at all. The paper must report the number of attack attempts for every model–attack pair and provide confidence intervals; words such as 'perfect robustness' and 'complete resilience' are not warranted by these sample sizes.","section":"§5.2, Table 3, Algorithm 4.2"},{"comment":"The same condition, BERTAttack on RoBERTa, gives original accuracy 35% and accuracy under attack 35% in Table 3, but Table 4 reports original accuracy 20% and accuracy under attack 20% for the same condition. Since both tables report a 0% ASR, the discrepancy changes the number of correct examples and hence the number of attack attempts (7 vs 4). If these are different test runs or different subsets, the paper must say so; otherwise one of the tables contains an error. This inconsistency directly affects the reliability of the central 0% ASR claim.","section":"Tables 3 and 4"},{"comment":"The formal definition of ASR in Eq. (3) divides by the full dataset size |D|, while Algorithm 4.2 and Section 4.3 define ASR as successful attacks divided by the number of attack attempts, i.e., the number of originally correct examples. These definitions differ whenever original accuracy is below 100%. For example, Table 2 would give 45/100 = 45% by Eq. (3), not the reported 93.75%. The definitions must be reconciled, since all robustness scores and conclusions depend on this metric.","section":"Eq. (3) vs §4.2/§4.3"},{"comment":"The cross-model comparison is confounded by task and dataset. BERT-base is tested on SST-2 for sentiment classification, while RoBERTa-base is tested on SQuAD v2 for question answering; Flan-T5 appears in both. TextFooler and BERTAttack are also applied to different model–dataset combinations. A claim of 'significant variations in model robustness' across BERT, RoBERTa, and Flan-T5 cannot be supported when the models are not evaluated on the same tasks, datasets, and attack conditions. At minimum, all models should be tested on a common benchmark, and the paper should explicitly report which model–dataset–attack triples were run.","section":"§4.1 and Table 4"}],"minor_comments":[{"comment":"The term 'Differential Robustness' in the title is never defined or used in the body. Please clarify what makes the robustness 'differential'.","section":"Title/§1"},{"comment":"BERTAttack is cited as (Jin et al., 2020), but the BERTAttack paper is Li et al. (2020). The reference list correctly has Li et al.; the in-text citation should be corrected.","section":"§4.1"},{"comment":"Column headers are ambiguous: 'BERTAttack on BERT' is not discussed in Section 5.1, and the number of failed/skipped attacks is not reported for any row. Include N, failed counts, and dataset identifiers in the table.","section":"Table 4"},{"comment":"The typeset output at the end of the algorithm includes a stray '=0'. Remove it.","section":"Algorithm 4.2"},{"comment":"Minor typographical issues: 'FlanT5' should be 'Flan-T5', and 'In contrast. BERT-Base' has a misplaced period.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is not ready in its current form because the central quantitative claim (0% ASR / perfect robustness) rests on a handful of examples and is contradicted by an internal table inconsistency. However, these are fixable by rerunning the experiments on a proper test sample, reporting confidence intervals, and correcting the tables and metric definitions. The paper's scope is modest but acceptable for a venue that values empirical robustness studies; the authors should be given the opportunity to provide the missing statistical support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nPunchline: this is an empirical robustness study with a headline result that falls apart on inspection. The 0% ASR for RoBERTa and Flan-T5, repeated in the abstract, equations, and conclusion, is based on tiny samples, and the paper's own numbers contradict each other.\n\nWhat's useful: the paper defines the evaluation pipeline clearly, uses PromptBench to run TextFooler and BERTAttack on three off-the-shelf models, and reports query counts and perturbed word percentages. That structure is fine as a starting point for a replication exercise.\n\nThe problems are not minor. RoBERTa's 0% ASR under BERTAttack comes from 0 successes out of 7 attempted attacks. With 7 attempts, the 95% Clopper-Pearson upper bound is about 41%, so the data are compatible with substantial vulnerability. Flan-T5's 0% ASR under TextFooler is not even broken down by attempts; if the 35% original accuracy corresponds to 35 correct examples, the upper bound is about 10%. Calling this \"perfect robustness\" or assigning R=1.0 in Equations 9–15 is not supported.\n\nWorse, the tables disagree. Table 3 gives RoBERTa's original accuracy under BERTAttack as 35%, while Table 4 gives 20% for the same condition. That is an internal contradiction in the central numbers. The \"accuracy under attack\" metric is also computed as original accuracy times (1 − ASR) rather than measured from post-attack predictions, so any error in the original accuracy propagates directly. The BERTAttack citation is wrong (they cite Jin et al. 2020, but the method is Li et al. 2020). No code or data are released, so the inconsistencies cannot be checked externally.\n\nThe paper does have an explicit limitations section, which is good, but it does not address these statistical and consistency problems.\n\nBottom line: the comparative robustness results are unreliable as reported, and the sample sizes are too small to support the claims. The reader's REJECT verdict is right. I would not send this to a serious referee as is; it needs a full re-run with adequate sample sizes, a single consistent evaluation script, and corrected citations. For now, it's a desk reject.\n\nWould I cite it? No. Reading group? Probably not.\n\nRecommendation: desk reject.","headline":"The headline '0% attack success' for RoBERTa and Flan-T5 is unsupported: it rests on 7 and 35 attempts, and the paper's own tables contradict each other.","tokens_in":7941,"tokens_out":3272,"would_cite":false,"duration_ms":30024,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that RoBERTa-Base and Flan-T5 withstand TextFooler and BERTAttack with a 0% attack success rate, while BERT-Base's accuracy collapses from 48% to 3% under TextFooler.","keywords":["adversarial robustness","transformer language models","TextFooler","BERTAttack","attack success rate","PromptBench","sentiment classification","question answering"],"falsifier":"Run the same attacks on a larger sample of correctly classified examples—say 1,000 for each model. If any attack succeeds on RoBERTa-Base or Flan-T5, the claimed 0% attack-success rate is false; a binomial confidence interval on 0 successes out of 7 attempts already has an upper bound around 41%, so a single success in a larger run would be sufficient to overturn the perfect-robustness claim.","tokens_in":6971,"feed_emoji":"🛡️","tokens_out":8407,"duration_ms":72901,"temperature":0.7,"pith_summary":"This study asks whether three transformer-based language models—BERT-Base, RoBERTa-Base, and Flan-T5—differ in their resistance to two established adversarial text attacks, TextFooler and BERTAttack. Using the PromptBench evaluation library on SST-2 sentiment classification and SQuAD v2 question answering, the paper reports that RoBERTa-Base and Flan-T5 keep their original accuracy under attack, with an attack success rate of 0%, while BERT-Base loses almost all accuracy under TextFooler (93.75% attack success, accuracy dropping from 48% to 3%). If these results hold, they suggest that the choice of model architecture and pretraining can itself provide strong adversarial defenses, though the paper also notes such defenses can be computationally expensive, with hundreds of queries per attack attempt. The main conclusions rest on small numbers of attack attempts, so the 'perfect' robustness numbers are statistically fragile.","feed_headline":"RoBERTa and Flan-T5 resist both attacks on every try","feed_subtitle":"If the numbers hold, word-swap attacks can't flip these models; BERT's accuracy collapses to 3% under TextFooler.","key_machinery":"The load-bearing machinery is the pair of attack algorithms and the robustness metric. TextFooler identifies important words by the change in prediction when a word is removed, then replaces them with semantically similar substitutes (cosine-similar word embeddings). BERTAttack masks each token and asks a BERT model to propose alternative tokens, filtering for semantic similarity, then greedily swaps tokens. Robustness is quantified by R = 1 − ASR, where ASR is the fraction of attack attempts that flip a correctly classified example; the paper also defines a computational efficiency score C = R / (average query count) × 100 to compare models per unit of computational cost.","core_discovery":"On the paper's own terms, the central discovery is a sharp asymmetry in attack success rates. Under TextFooler, BERT-Base's accuracy falls from 48% to 3%, an attack success rate of 93.75%, yielding a robustness score R=0.0625. Under BERTAttack, RoBERTa-Base shows a 0% attack success rate (R=1.0) on the 7 correctly classified examples it was tested on, and Flan-T5 likewise shows 0% under TextFooler (R=1.0). The paper also reports BERT-Base suffers a 100% attack success rate under BERTAttack. It presents these numbers as evidence that RoBERTa-Base and Flan-T5 are completely resilient to these adversarial techniques, while BERT-Base is highly susceptible. It also introduces a computational effi","pith_inferences":["The 0% ASR for RoBERTa-Base is derived from only 7 attack attempts and Flan-T5 from 35; with such small denominators, the data cannot statistically rule out a true attack success rate well above zero, so the 'perfect' robustness claim should be treated as provisional until tested on larger samples.","If larger replications confirm the asymmetry, it would imply that robustness to word-swap attacks can be inherited from pretraining or model scale rather than requiring dedicated adversarial training, which would change how practitioners select models for security-sensitive NLP.","A natural extension is to test the same three models against gradient-based or generation-based attacks that do not rely on word substitution; the pattern seen here may not generalize across attack families.","The paper's efficiency metric C could be used as a regularizer in model selection or as a target for AutoML, balancing robustness and query cost rather than treating robustness as the only objective."],"forward_implications":["If the 0% attack success rates are accurate, word-substitution attacks in the style of TextFooler and BERTAttack are ineffective against RoBERTa-Base and Flan-T5 in these benchmark settings.","BERT-Base, by contrast, is highly vulnerable to TextFooler, so deployments of BERT for sentiment classification would need adversarial defenses or a different model choice.","The robustness score R and efficiency metric C provide a simple, comparable way to rank models by security per unit of computational cost.","The high average query counts reported for the robust models suggest that evaluating or defending against attacks on these models is computationally expensive, which shifts the cost of security to the evaluator.","The proposed multi-objective training objective in the recommendations (accuracy + robustness + computational cost) could guide future defenses for models like BERT."],"supporting_citations":[{"why":"Supplies the TextFooler attack algorithm used to perturb inputs and generate the paper's central vulnerability measurements.","marker":"(Jin et al., 2020)"},{"why":"Supplies the BERTAttack algorithm, the second attack family tested against BERT and RoBERTa.","marker":"(Li et al., 2020)"},{"why":"Provides the PromptBench evaluation library that runs the attacks and collects the robustness metrics.","marker":"(Zhu et al., 2024)"},{"why":"Defines the BERT-Base model, the vulnerable model in the comparison.","marker":"(Devlin et al., 2019)"},{"why":"Defines RoBERTa-Base, the model reported as perfectly robust under BERTAttack.","marker":"(Liu et al., 2019)"},{"why":"Defines Flan-T5, the model reported as perfectly robust under TextFooler.","marker":"(Chung et al., 2024)"},{"why":"Provides the SST-2 sentiment classification dataset used for the BERT and Flan-T5 evaluations.","marker":"(Socher et al., 2013)"},{"why":"Provides the SQuAD v2 question-answering dataset used for the RoBERTa and Flan-T5 evaluations.","marker":"(Rajpurkar et al., 2018)"}],"fun_headline_variants":["RoBERTa and Flan-T5 resist both attacks; BERT's accuracy hits 3%","Zero attack success for RoBERTa and Flan-T5; BERT drops to 3%","TextFooler wrecks BERT but can't touch Flan-T5 or RoBERTa","BERT's accuracy plummets to 3% while RoBERTa and Flan-T5 stay perfect","Adversarial attacks fail on RoBERTa and Flan-T5, BERT collapses"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusion that RoBERTa-Base and Flan-T5 are perfectly robust rests on treating 0% attack success rates computed from only 7 (RoBERTa) and 35 (Flan-T5) attack attempts as reliable estimates of true robustness; with that few attempts, the data cannot statistically rule out much higher underlying success rates.","fun_headline_variants_meta":{"raw":{"variants":["RoBERTa and Flan-T5 resist both attacks; BERT's accuracy hits 3%","Zero attack success for RoBERTa and Flan-T5; BERT drops to 3%","TextFooler wrecks BERT but can't touch Flan-T5 or RoBERTa","BERT's accuracy plummets to 3% while RoBERTa and Flan-T5 stay perfect","Adversarial attacks fail on RoBERTa and Flan-T5, BERT collapses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000819,"raw_usage":{"total_tokens":3417,"prompt_tokens":736,"completion_tokens":2681,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":2566}},"tokens_in":480,"tokens_out":2681,"duration_ms":18024,"temperature":1.0,"reasoning_tokens":2566,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:23:10.584746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same attacks on a larger sample of correctly classified examples—say 1,000 for each model. If any attack succeeds on RoBERTa-Base or Flan-T5, the claimed 0% attack-success rate is false; a binomial confidence interval on 0 successes out of 7 attempts already has an upper bound around 41%, so a single success in a larger run would be sufficient to overturn the perfect-robustness claim.","supporting_citations":[],"review_version":1}