{"id":"57d28ba9-3db4-46ec-8ed2-f89e1a1b0869","arxiv_id":"2412.21016","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A beam-search-based testing method (ABS) generates adversarial text perturbations that flip LLM classifier labels more often and more cheaply than five existing attack methods across three datasets.","lead":"This paper presents AORTA, a framework for automatically testing how robust LLM-based text classifiers are to small input changes, and ABS, a beam-search method that generates such test cases. The authors report that ABS outperforms five existing methods in flipping model labels, while using fewer queries and less time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline average success rate (86.138%) cannot be derived from Table I; recomputing the 15 ABS S-rate entries gives ~81.45%, so the paper's central quantitative claim is internally inconsistent.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and I agree with that overall disposition. The reader's weakest_assumption was the stability of LLM confidence scores; that is a genuine threat, and the authors themselves flag it in Section VI-A for smaller LLMs. However, the most load-bearing issue for the central claim is more elementary and more concrete: the advertised average success rate is not reproducible from the paper's own Table I. If the number 86.138% is wrong, the main summary of the paper is inaccurate as stated. A recomputation settles this without any modeling judgment. This is not an attack on the authors; the most plausible explanation is a typo or an aggregation not documented. Because ABS still outperforms PWWS in every one of the 15 table entries, the qualitative ABS-is-better conclusion may hold even after correction; therefore I would keep the CONDITIONAL verdict, requiring the authors to correct the headline number, report variance or error bars, and clarify hyperparameter tuning. The confidence-stability concern should also be addressed by calibration or ablation, but it does not replace the arithmetic check as the single decisive test.","tokens_in":24688,"tokens_out":9927,"duration_ms":99107,"concrete_test":"Recompute the averages from the raw result data in the public repository or from Table I: sum the 15 ABS S-rate values and divide by 15; also compute dataset-wise means for ABS and PWWS using the same convention that gives PWWS = 68.177% (mean of the FP, AG's News, and MR dataset means). If the ABS average is not 86.138%, the abstract and conclusion must be corrected or the intended aggregation must be stated explicitly. As a secondary check, re-run one small-LLM setting with a frozen or calibrated confidence oracle to see whether the ABS advantage over PWWS changes materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion assert an average test success rate of 86.138% for ABS versus 68.177% for PWWS. The PWWS figure is exactly the mean of the three dataset-wise PWWS means in Table I (79.395%, 54.868%, 70.270%) = 68.177%. Applying the identical aggregation to the 15 ABS S-rate entries in Table I: FP sum = 431.993 (mean 86.399%), AG's News sum = 391.077 (mean 78.215%), MR sum = 398.710 (mean 79.742%); total mean = 1221.780/15 = 81.452%. No standard aggregation (plain mean, mean of dataset means, mean of model means) reproduces 86.138%. The paper gives no subset or weighting that would yield this number. This matters because the abstract and conclusion present 86.138% as the headline evidence that ABS enables a more comprehensive and accurate robustness assessment. The number is not an arithmetic consequence of the reported results. The qualitative comparison ABS > PWWS survives the correction (81.45% vs 68.18%, and ABS beats PWWS in all 15 settings), so this is a correctness/correction issue rather than a rejection. The paper's own Section VI-A caveat about unstable confidence outputs in smaller LLMs is a real secondary limitation for Eq. (4), (6), and (7), but it is less decisive because success is defined by label flip and PWWS shares the same soft-label oracle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes AORTA, a modular black-box robustness testing framework for LLM-based NLP software that reformulates testing as a combinatorial optimization problem over the combined prompt and example input. The framework embeds 17 existing DNN-era test methods plus a new method, ABS, which uses beam search with an adaptive beam width and a backtracking/refill mechanism. The authors evaluate ABS against five baselines on three datasets and five open-source LLMs, reporting that ABS achieves a higher success rate, lower time and query overhead, and better naturalness and transferability than the strongest baseline, PWWS.","tokens_in":25047,"tokens_out":3504,"duration_ms":35318,"significance":"If the reported results hold, the paper makes a useful contribution to software testing for LLM-based systems: it provides a unified framework for adapting existing NLP test methods to the black-box LLM setting, and it proposes a plausible search strategy that outperforms the best DNN-era baseline in all 15 settings of Table I. The open-source repository and the modular design are concrete strengths that support reproduction and extension. However, the paper's headline quantitative claim—an average success rate of 86.138%—is not derivable from the paper's own Table I, and the hyperparameter tuning procedure described in Section V is applied on the same test data used for evaluation. These issues bear directly on the central claims and must be resolved before the paper can be recommended for acceptance.","major_comments":[{"comment":"The abstract and the conclusion state that ABS achieves an average test success rate of 86.138%, compared with 68.177% for PWWS. Table I reports 15 ABS S-rate values: their sum is 1221.780, giving a plain mean of 81.452%; the mean of the three dataset-wise means is also 81.452%. No standard aggregation (plain mean, dataset-wise mean, model-wise mean, or any subset) reproduces 86.138%. The delivered PWWS figure of 68.177% is exactly the mean of the three PWWS dataset means (79.395%, 54.868%, 70.270%), so the inconsistency is specific to the ABS claim. Because this number is presented as the headline evidence of ABS's superiority, please correct the aggregate statistic and recompute all dependent statements (including the query-reduction factor of 218.762 times).","section":"Abstract and Conclusion vs. Table I"},{"comment":"The paper states that the maximum beam width bmax is set based on 'the datasets' text lengths and the threat models' performance' and that bmax is kept within [6, 10] across experiments. If this tuning is performed on the same 1000-example test sets that are later used to compute S-rate, T-O, and Q-N, the evaluation is optimistically biased: the method is effectively selected on the test data. The paper's own internal-validity discussion in Section VII-B acknowledges hyperparameter sensitivity but does not address this circularity. Please either tune bmax on a held-out validation set, report a sensitivity analysis varying bmax over its range for each dataset/model, or provide variance and per-configuration results so readers can assess the stability of the reported advantages.","section":"Section V, Experiment Setup"},{"comment":"All S-rate, C-rate, and PPL entries in Table I are point estimates, even though the text says each experiment was repeated three times and averaged. No standard deviations, confidence intervals, or significance tests are reported. Consequently, assertions such as 'In 10 out of 15 experiments in RQ1, ABS outperformed the best-performing method by more than 5%' and the general claim that ABS 'consistently surpasses all baselines' cannot be distinguished from noise. Please report per-setting variance across the three repeats and, if feasible, a paired test over the 1000 examples or the three repeats to support the qualitative comparison.","section":"Section VI-A, Table I"}],"minor_comments":[{"comment":"In Eq. (4), the word importance score multiplies a softmax term by the same confidence difference; please clarify what the softmax is computed over and what this multiplication adds beyond the raw difference, since the current formulation is easy to misread as a squared difference with a normalization factor.","section":"Section IV, Eq. (4)"},{"comment":"The symbol 'n' is used both for the number of candidates in the current beam and for the number of words in the input text; using distinct symbols (e.g., b for beam size and m for text length) would improve clarity and avoid confusion in the adaptive beam-width formula.","section":"Section IV, Algorithm 1 and Eq. (5)"},{"comment":"The subcaptions in Figure 4 contain evident typos, including 'G rammatical error' and '(want ↓)'; these should be corrected and the y-axis labels made consistent across all panels.","section":"Figure 4"},{"comment":"The five selected LLMs differ not only in parameter count but also in architecture, tokenizer, and training data; the paper attributes performance differences mainly to parameter scale, so please acknowledge this confound explicitly in the discussion of generalizability.","section":"Section V, Threat Models"},{"comment":"The authors note in Section VI-A that smaller LLMs exhibit hallucination effects leading to unstable confidence outputs. Since Eq. (4), (6), and (7) all rely on confidence values, it would be helpful to add a brief discussion of how this limitation affects ABS's search guidance relative to the baselines, which use the same soft-label oracle.","section":"Section VI-A, confidence oracle"}],"recommendation":"major_revision","confidential_remarks":"The manuscript benefits from a reproducible repository and a clear modular design, and the qualitative claim that ABS outperforms PWWS in all 15 settings appears to survive the arithmetic correction. The main reservations are the internally inconsistent aggregate success rate and the test-set tuning of bmax. I would encourage the editor to ask the authors to recompute summary statistics, report variance, and either move hyperparameter tuning to a validation set or provide a sensitivity analysis before inviting a revised submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth reading for its systematic comparison, but the headline number is wrong. The abstract and conclusion say ABS achieves an average success rate of 86.138%, yet Table I's own ABS entries average to about 81.45% under any standard aggregation. The PWWS 68.177% figure does match the table, so the discrepancy is specific to the ABS claim. This is a correction, not a rejection: ABS still beats PWWS in all 15 settings and wins by a wide margin on most.\n\nWhat's actually new: AORTA reworks DNN-era testing methods into a unified framework that perturbs the combined prompt+example input, and ABS adapts beam search with dynamic width and backtracking. The empirical sweep is broad—five LLMs, three datasets, five baselines—and they report success rate, change rate, perplexity, grammatical errors, time, and query count. The code and data are open-sourced, which is a real plus.\n\nThe paper does a few things well. The framework design is clean and extensible, and the adaptation of existing methods looks faithful. The efficiency gains over PWWS are plausible and the transferability experiments are a nice addition. They also honestly flag the hallucination problem with smaller LLMs in Section VI-A, which is a real limitation for the confidence-based guidance.\n\nSoft spots, in order of importance. First, the 86.138% number cannot be derived from the table. The authors need to either correct it or explain the computation. Second, bmax is tuned based on dataset and model performance on the same test set. That is a form of test-set tuning that can inflate the results; it should be reported as such or done on a validation set. Third, they run each experiment three times but report no variance or significance tests, which makes it hard to judge whether the differences are meaningful. Fourth, the 'first' claims are overstated: beam search with backtracking has been used in adversarial prompt generation (Sadasivan et al.), and the novelty is in applying it to joint prompt+example testing, not in the search mechanism per se. The naturalness evidence is also indirect (PPL and grammar errors), but that's standard in this literature.\n\nOverall: the central qualitative result stands after the arithmetic correction, so this is a useful contribution to software testing and adversarial robustness. It deserves peer review, and I think a good referee would send it back for a correction of the headline number and a discussion of the hyperparameter tuning. I would bring it to a reading group focused on robustness testing, and I'd cite the empirical comparison if I worked in this area.","headline":"Solid empirical study with a sloppy headline number: ABS beats the baselines, but the abstract's 86.138% average doesn't match Table I.","tokens_in":25517,"tokens_out":3256,"would_cite":true,"duration_ms":30660,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a new automated testing method, Adaptive Beam Search (ABS), can find adversarial inputs for LLM-based text classification software far more effectively than existing methods designed for older neural networks.","keywords":["robustness testing","LLM-based NLP software","adaptive beam search","adversarial test cases","text classification","black-box testing","software testing"],"falsifier":"A direct test would be to run ABS on the same datasets and threat models but replace the confidence-based word importance ranking and beam-width indicator with a random or fixed ordering of words; if the success rate stays near 86%, the confidence signal is not doing the work, and the reported efficiency gains would need a different explanation. Alternatively, testing ABS on families of LLMs known to produce miscalibrated or unstable confidence scores (e.g., small instruction-tuned models) and observing a drop toward the 68% baseline would confirm the dependency on confidence reliability.","tokens_in":24531,"feed_emoji":"🧪","tokens_out":5239,"duration_ms":49944,"temperature":0.7,"pith_summary":"The paper claims that a new automated testing method, Adaptive Beam Search (ABS), can find adversarial inputs for LLM-based text classification software far more effectively than existing methods designed for older neural networks. The method treats the entire input—the prompt and the example together—as a single object to be perturbed, rather than testing prompts and examples separately. Across three datasets and five LLMs, ABS reports an average test success rate of 86.138% at inducing misclassification, compared with 68.177% for the best existing baseline (PWWS). It also claims to generate more natural and more transferable test cases while using fewer queries and less time per successful test case. If correct, this gives developers a practical way to probe LLM-based software for robustness defects before deployment.","feed_headline":"Adaptive beam search finds LLM text flaws at 86% success","feed_subtitle":"It tests whole prompt-plus-example inputs, using fewer queries and less time per successful test case.","key_machinery":"The central object is Adaptive Beam Search (ABS), a search over the perturbation space of the combined prompt-plus-example text. Standard beam search keeps a fixed-width set of the most promising candidate texts at each step; ABS makes the beam width $b$ adaptive through Eq. (5), which expands the beam when the indicator function $I(s_i, s'_i)$ in Eq. (6) shows that many candidates have lowered the model's confidence on the ground-truth label, and contracts it otherwise. Backtracking is implemented by maintaining a historical best candidate, updated via Eq. (7), and after each iteration replacing the beam's worst-scoring member with that historical best if the historical best has lower confidence. The word importance ranking in Eq. (4), which measures the confidence drop when each word is masked, sets the order in which words are perturbed. These mechanisms turn beam search from a fixed greedy heuristic into a search that widens when exploration pays off and rescues promising candidates that would otherwise be discarded.","core_discovery":"The central claim is that robustness testing of LLM-based NLP software should be reformulated as a combinatorial optimization problem over the full input (prompt plus example), and that an adaptive beam search is the right search algorithm for that problem. The paper introduces AORTA, a framework that supplies the goal function, perturbation space, linguistic constraints, and search method needed to run such tests in a black-box setting, and uses it to adapt 17 existing DNN-era testing methods to LLMs. Within this framework, ABS ranks words by how much masking them lowers the model's confidence in the ground-truth label, then performs synonym replacement guided by a beam search whose width grows when perturbations are successfully lowering confidence and shrinks otherwise, with a backtracking mechanism that preserves the historically best candidate. The reported result is an average success rate of 86.138%, about 18 points above the best baseline, together with lower query counts, lower time overhead, fewer grammatical errors, and better transferability of test cases across different LLMs.","pith_inferences":["The method's reliance on confidence scores suggests it may work better for large models with well-calibrated confidence; for small or hallucination-prone models, a calibration step or an alternative uncertainty signal would be a natural extension, since the paper itself notes unstable confidence outputs in smaller LLMs.","The same adaptive beam search with backtracking could be extended beyond classification to other safety-critical LLM tasks such as toxic content moderation or log generation, wherever a scalar confidence or likelihood score is available to guide the search.","The high transferability of ABS test cases across model sizes hints that the method may be surfacing input-level vulnerabilities that are intrinsic to the data distribution rather than to a particular model, which could support cross-model robustness certification.","A testable extension would be to compare ABS against a version that uses a random word-order ranking instead of confidence-based ranking; if success rates remain similar, the confidence signal is not the source of the advantage, and the method could be simplified."],"forward_implications":["If ABS works as reported, LLM-based text classification software can be stress-tested automatically before deployment, with success rates near 86% on standard benchmarks, giving a quantitative robustness score rather than a manual audit.","Because ABS test cases are more natural (fewer grammatical errors, lower perplexity) and more transferable across models, the same test suite can be reused on different LLM backends, reducing the cost of repeated robustness assessment.","The AORTA framework shows that existing DNN-era testing methods can be ported to LLM-based software, but the ported methods have limited effectiveness (e.g., CheckList reaches only 12.097% on AG's News), implying that a search strategy specialized for LLMs is needed, not just a software wrapper.","Ablation results indicate that adaptive beam width and backtracking each contribute independently to the gains, so the two enhancements can be tuned separately for different datasets and model scales."],"supporting_citations":[{"why":"PWWS is the best-performing baseline, defining the 68.177% average success rate that ABS claims to beat by 17.96 points.","marker":"[26]"},{"why":"TextFooler provides the word importance ranking and synonym replacement approach that ABS adapts for its perturbation order.","marker":"[54]"},{"why":"TextAttack's modular design inspires the four-component AORTA framework and supplies the evaluation metrics used in the experiments.","marker":"[48]"},{"why":"CheckList is one of the five baselines, showing the low success rate of behavioral testing methods when ported to LLM-based software.","marker":"[39]"},{"why":"LEAP represents the meta-heuristic search approach whose excessive computational overhead (over 500 minutes per case) motivates ABS's efficiency-focused design.","marker":"[38]"},{"why":"Financial Phrasebank is one of the three datasets used to evaluate ABS and the baselines.","marker":"[27]"},{"why":"AG's News is the long-text dataset used to test the search methods under a larger feature space.","marker":"[28]"},{"why":"MR is the movie-review sentiment dataset used as the third evaluation benchmark.","marker":"[29]"},{"why":"PromptRobust establishes the formulation of LLM inputs as prompt plus example, which AORTA's overall-input testing builds on.","marker":"[3]"},{"why":"WordNet supplies the synonym replacements used by ABS during perturbation.","marker":"[56]"}],"fun_headline_variants":["Adaptive beam search tests LLM inputs at 86% success","AORTA: first automated framework for LLM-based NLP robustness","New method tests full LLM prompts, uses 218x fewer queries","Adaptive beam search: 86% success, 218x fewer LLM queries","AORTA enables efficient robustness testing for LLM software"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the confidence score an LLM returns for the ground-truth label is a stable and meaningful measure of the model's uncertainty under perturbation, so that lowering that confidence reliably guides the search; the paper itself notes that hallucination effects in smaller LLMs produce unstable or incorrect confidence outputs, which could undermine the search guidance.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive beam search tests LLM inputs at 86% success","AORTA: first automated framework for LLM-based NLP robustness","New method tests full LLM prompts, uses 218x fewer queries","Adaptive beam search: 86% success, 218x fewer LLM queries","AORTA enables efficient robustness testing for LLM software"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2233,"prompt_tokens":1061,"completion_tokens":1172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1078}},"tokens_in":677,"tokens_out":1172,"duration_ms":10617,"temperature":1.0,"reasoning_tokens":1078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:04:33.371006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run ABS on the same datasets and threat models but replace the confidence-based word importance ranking and beam-width indicator with a random or fixed ordering of words; if the success rate stays near 86%, the confidence signal is not doing the work, and the reported efficiency gains would need a different explanation. Alternatively, testing ABS on families of LLMs known to produce miscalibrated or unstable confidence scores (e.g., small instruction-tuned models) and observing a drop toward the 68% baseline would confirm the dependency on confidence reliability.","supporting_citations":[{"cited_title":"Is bert really robust? a strong baseline for natural language attack on text classification and entailment,","cited_arxiv_id":null,"evidence_quote":"TextFooler provides the word importance ranking and synonym replacement approach that ABS adapts for its perturbation order."},{"cited_title":"Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp,","cited_arxiv_id":null,"evidence_quote":"TextAttack's modular design inspires the four-component AORTA framework and supplies the evaluation metrics used in the experiments."},{"cited_title":"Beyond accuracy: Behavioral testing of nlp models with checklist (extended abstract),","cited_arxiv_id":null,"evidence_quote":"CheckList is one of the five baselines, showing the low success rate of behavioral testing methods when ported to LLM-based software."},{"cited_title":"Leap: Efficient and automated test method for nlp software,","cited_arxiv_id":null,"evidence_quote":"LEAP represents the meta-heuristic search approach whose excessive computational overhead (over 500 minutes per case) motivates ABS's efficiency-focused design."},{"cited_title":"Character-level convolutional networks for text classification,","cited_arxiv_id":null,"evidence_quote":"AG's News is the long-text dataset used to test the search methods under a larger feature space."},{"cited_title":"Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales,","cited_arxiv_id":null,"evidence_quote":"MR is the movie-review sentiment dataset used as the third evaluation benchmark."}],"review_version":1}