{"id":"b9575751-c9d9-4c4f-8b18-081bccc397d9","arxiv_id":"2506.17812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BouncerBench evaluates whether LLM coding agents can abstain from acting on underspecified tickets and incorrect patches, and shows current models rarely abstain correctly.","lead":"This paper introduces BouncerBench, a benchmark that tests whether AI coding agents can refuse to work on vague bug tickets and refuse to hand over patch fixes that are likely wrong. It finds that current models, including reasoning models, usually fail these refusal tests, so there is room for improvement before agents can be trusted in real workflows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Input-bouncing ground truth relies on 'any annotator says vague' ensemble; without agreement stats, low recall may not show true abstention failure.","rationale":"The reader's weakest assumption matches the concern I find most load-bearing. The benchmark's purpose is to measure abstention on underspecified inputs; the only source of truth for 'underspecified' is the human annotation ensemble. The 'highest-severity' rule is a deliberate conservative choice, but it has a clear direction of bias: it inflates the vague class whenever annotators disagree. Without per-annotator agreement statistics or a sensitivity analysis under majority/unanimous ensembles, the benchmark cannot distinguish between 'models cannot recognize vagueness' and 'models disagree with the most aggressive human label.' The low input-bounce recall values (4-27%) are the key evidence for the paper's headline claim; if the label set is even moderately noisy, those numbers are not a valid measure. This is more fundamental than the secondary issues the reader lists: the O-Score's attainable range is a metric-design concern that does not affect the binary bouncing results, and the output-bouncing selection bias for vague tickets affects only 109/642 tasks and does not change the input-bouncing evidence. The paper is honest about this limitation in Section VII, but a benchmark that cannot provide inter-annotator agreement or ensemble robustness should be accepted only on the condition that these analyses are added. The central direction is sound and the empirical results are plausible, so I do not recommend rejection; the verdict stays CONDITIONAL.","tokens_in":16574,"tokens_out":7896,"duration_ms":81946,"concrete_test":"Obtain the per-annotator severity labels for the 1699 SWE-Bench instances from the OpenAI annotation data. Recompute the input-bouncing ground truth under two alternative ensembles: majority vote (vague iff at least 2 of 3 annotators assign label 2/3) and unanimous (vague iff all 3 assign label 2/3). Re-score the six bouncer configurations from Tables II and III on these alternative label sets, or at minimum recompute bounce-class recall and macro-F if per-item model predictions are released. If the relative ordering of models and the qualitative finding of low recall persist under majority/unanimous labels, the conservative-ensemble concern is mitigated; if recall rises substantially (e.g., >60% on unanimous-vague tickets), the headline overstates the abstention failure and the benchmark's input task needs re-labeling or a sensitivity analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section II-A, BouncerBench defines the input-bouncing ground truth by taking the highest-severity label across three annotators: a ticket is 'underspecified' if any one annotator assigns label 2 or 3. This yields 650/1699 (38.26%) vague tickets, a high fraction for tickets that were otherwise curated enough to reach SWE-Bench Verified. The paper reports no inter-annotator agreement for the binary vague/clear distinction, so we cannot tell how many tickets are vague by majority or unanimous vote versus by a single dissenting annotator. The central empirical claim—that LLM-based agents fail to abstain from underspecified inputs—is supported mainly by low recall on this over-inclusive label set (e.g., best model bounces only 26.8% of the 650). If many of those 650 are borderline tickets that two or three annotators considered well-specified, then the models' 'failures' may reflect disagreement with an aggressive annotation policy rather than an inability to detect underspecification. Section VII acknowledges reliance on the annotation soundness but provides no sensitivity analysis. Because this assumption directly determines the ground truth of the benchmark's primary new task, it is the most load-bearing point: if the labels are noisy, BouncerBench's input-bouncing scores and the headline conclusion are not a valid measurement of abstention ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BouncerBench, a benchmark for measuring whether LLM-based software agents can abstain from acting on underspecified issue reports (input bouncing) and from returning incorrect patches (output bouncing). The input task uses 1,699 SWE-Bench issues with human annotation labels 0-3, where ground truth 'should bounce' is assigned if any of three annotators gave the highest severity labels 2 or 3, yielding 650 vague tickets. The output task draws 642 LLM-generated patches from leaderboard submissions and labels each as correct or incorrect from unit-test outcomes, with the caveat that only incorrect patches are sampled for the 109 input-vague instances. The paper proposes macro-averaged F-measure, I-Score, and O-Score, and evaluates six LLM baselines plus a Codex CLI agent. Results show low recall on vague tickets (best 26.8% for o4-mini) and output-bouncing recall up to 61% for Codex, leading the authors to conclude that current models cannot yet abstain reliably.","tokens_in":16799,"tokens_out":10249,"duration_ms":95962,"significance":"BouncerBench addresses an overlooked and practically important failure mode in code agents, and the public leaderboard and replication package are good contributions. The paper ships a concrete benchmark with a clear task formulation and baseline implementations that will likely be useful to the community. The headline finding that current models seldom abstain, even when instructions ask them to, is plausible and largely supported by the low absolute recall values. However, several construction and metric choices currently weaken the force of the quantitative conclusions; these are fixable in revision.","major_comments":[{"comment":"The input-bouncing ground truth is defined as the maximum severity across three annotators, which labels 650 of 1,699 tickets (38.26%) as vague, yet the paper reports no inter-annotator agreement for the vague/clear distinction. Because a single dissenting annotator is sufficient to mark a ticket as 'should bounce', the low recall values in Table II may reflect disagreement with an aggressive labeling policy rather than an inability to detect underspecification. Please report agreement statistics (e.g., Fleiss' kappa for the binary decision) and a sensitivity analysis using majority-vote or unanimous-vote thresholds; the threats-to-validity section acknowledges reliance on annotation soundness but does not quantify it.","section":"II-A1 and VII"},{"comment":"For the output-bouncing task, the 109 instances that are input-vague are populated exclusively with incorrect patches, and 6 instances with no incorrect submissions are discarded. This means that for 109 of 642 tasks (17%), the correct label is 'bounce' by construction rather than by inspection of the patch, so a model that uses ticket vagueness as a shortcut will appear to perform well for reasons unrelated to output evaluation. The conditional sampling also changes the class distribution relative to the full population of patches. Please either include correct patches for vague tickets (and model the decision as needing both input and output judgments) or report the output-bouncing results separately for the well-specified subset.","section":"II-A2"},{"comment":"The two fine-grained scores are not calibrated so that 0 corresponds to the natural baseline, and the stated O-Score range is incorrect. For the input task, an 'accept all' policy achieves I-Score = (2/3)(1.5 - mean_label) ≈ 0.191 under the Table I label distribution, so Claude-3.7-sonnet (0.209) and Gemma3 (0.198) are barely above a trivial constant classifier. For the output task, an incorrect patch cannot have PassedTests/TotalTests = 1, so the maximum O-Score is strictly below +1 whenever the dataset contains incorrect patches, contradicting the claim of a [-1, +1] range; moreover, a constant-accept policy has a nonzero expected O-Score. Please report always-accept and always-bounce baselines for all metrics and reinterpret the scores against these baselines.","section":"II-B and IV"},{"comment":"Model comparisons are reported as point estimates without confidence intervals or significance tests. For example, the input macro-F gap between o4-mini (0.592) and Codex (0.561), and the output gap between Codex (0.690) and o4-mini (0.612), are used to support claims of improvement, but on a fixed task set a bootstrap or McNemar test would be inexpensive and appropriate. Without such tests, the relative statements in Sections IV-B, IV-D, and IV-E are not statistically supported; the central 'low absolute performance' claim is less affected by this issue.","section":"Tables II-III and IV-E"}],"minor_comments":[{"comment":"The sentence 'their is no human annotation for the patches' contains a typo; it should read 'there is no human annotation for the patches'.","section":"IV-C"},{"comment":"The related-work section contains the typo 'acheiving' in 'acheiving over 90% alignment with Human Judges'; it should be 'achieving'.","section":"VI"},{"comment":"The abstract says BouncerBench 'aims to improve precision', but as a benchmark it measures rather than improves systems; consider rewording to 'measure precision'.","section":"Abstract"},{"comment":"The term 'BouncerBench Lite' is introduced in the text but never explicitly defined; please state up front that it is the 642-task subset with equal numbers of input and output tasks.","section":"II-A2"},{"comment":"The statement 'We sample at temperature 0 for all the non-reasoning models' is immediately followed by exceptions for reasoning models; please consolidate the temperature and sampling settings for each model into a single clear specification.","section":"IV-A"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a useful contribution and the central negative finding is plausible, but the metric-calibration and task-construction problems are too central for acceptance in the current form. I would consider a revised version that adds baselines, sensitivity analysis, and significance tests. The work fits the venue's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. BouncerBench is the first benchmark I have seen that evaluates whether LLM-based coding agents abstain — reject vague issue tickets and refuse to ship their own bad patches — rather than always acting, and the authors ship the dataset, replication package, and leaderboard, so it is usable now. The headline finding, that current models rarely abstain well, is plausible and mostly holds up, but the input-side ground truth is softer than the paper admits and one metric's range claim is wrong.\n\nThe good parts are real. Output-bouncing ground truth is objective, defined by SWE-Bench test outcomes rather than an LLM judge. The construction is careful: instances with faulty test oracles are discarded, the awkward decision to sample only incorrect patches for the 109 vague-ticket output tasks is disclosed, and the paper shows the costs of bouncing honestly — their own output bouncer loses 30% of OpenHands' correct patches, and the flow diagram shows 22 clear tickets bounced along with 20 vague ones. The agreement analysis with human annotators (kappa below 0.15 for every model) is a strong addition, and the cost figures are transparent.\n\nThe soft spots, in proportion. The biggest is the input ground truth: 650 of 1699 tickets (38%) count as should-be-bounced if any one of three annotators called them vague. That union rule is aggressive, no inter-annotator agreement is reported, and the input prompt deliberately mirrors the OpenAI annotation instructions, so the input task partly measures how well a model can simulate those annotators. The stress-test calls this load-bearing; I think that overstates it. Output bouncing, with objective ground truth, shows the same failure mode — best recall is 61% at a 17% false-bounce rate, and non-reasoning models stay below 10% — and the low kappa values are independent of the ensemble rule. The qualitative conclusion would survive a stricter label set. Still, the input numbers are not calibrated, and the paper needs agreement statistics and a sensitivity analysis using majority or unanimous ensembles. Section VII flags the dependence on annotations but does not test it.\n\nThe metric problem is minor but real. O-Score is claimed to range from -1 to +1, but on this data the upper bound sits below +1: an incorrect patch that is correctly bounced contributes only its passed-test fraction, and incorrect patches by definition fail at least one test. Neither score demonstrably centers at zero for a random policy. Evaluations are single-run with no error bars; the model gaps are large enough that I suspect they are real, but a benchmark aiming to become standard should not skip this.\n\nBottom line: a genuinely useful artifact for anyone building or evaluating coding agents, with honest discussion and fixable flaws. It deserves a serious referee. I would send it to review and require the annotation agreement analysis, the sensitivity check, and corrected metric claims first.","headline":"First abstention benchmark for coding agents with a real artifact and a plausible headline finding, but the input ground truth needs agreement statistics and the O-Score's range claim is wrong.","tokens_in":17334,"tokens_out":9476,"would_cite":true,"duration_ms":87792,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM coding agents almost never refuse to act on vague bug reports or wrong patches, and BouncerBench is a new benchmark built to measure that failure.","keywords":["LLM agents","abstention","BouncerBench","software engineering benchmark","vague issue tickets","patch validation","trustworthy AI","selective prediction"],"falsifier":"Re-run the input-bouncing task with ground truth defined by majority vote instead of the highest severity label, and check whether the model rankings and the 38% 'should bounce' rate change substantially; if they do, the input task is an artifact of the ensemble rule.","tokens_in":16367,"feed_emoji":"🚪","tokens_out":10387,"duration_ms":95456,"temperature":0.7,"pith_summary":"The paper's central claim is that current LLM-based software agents are untrustworthy in a specific, measurable way: they rarely abstain. They generate a patch for every issue, even when the report is too vague to act on, and they ship patches even when those patches are wrong. To make this failure visible, the authors introduce BouncerBench, with two tasks: input bouncing, which asks a system to reject underspecified tickets, and output bouncing, which asks it to reject incorrect patches. On the benchmark's 1,699 input tasks and 642 output tasks, the best configurations still catch only a fraction of the cases that should be bounced, and the paper concludes that large language models cannot yet be trusted to make correct decisions in real software-engineering workflows.","feed_headline":"AI coders rarely abstain from vague bug reports or bad patches","feed_subtitle":"BouncerBench measures this; best models catch 27% of underspecified tickets and 61% of bad patches.","key_machinery":"The central object is the bouncer, a gatekeeping agent placed around an LLM-based coding system. An input bouncer decides whether an issue ticket is specific enough to work on; an output bouncer decides whether a generated patch should reach the user. BouncerBench turns both decisions into binary classification tasks with ground truth: for inputs, a highest-severity ensemble of three human annotators' labels; for outputs, the pass/fail behavior of the dataset's unit tests. The paper's main metric is macro-averaged F-measure, chosen so an 'always bounce' or 'always accept' strategy cannot inflate scores, with fine-grained I-Score and O-Score that weight each decision by ambiguity or by the fraction of tests passed.","core_discovery":"In the paper's own terms, the discovery is that abstention is a missing capability in current coding agents, and BouncerBench is the first benchmark to isolate and score it. The input side treats a ticket as underspecified if human annotators rated it vague or nearly impossible to solve, and the output side treats a patch as incorrect unless it passes all fail-to-pass and pass-to-pass tests. Across the evaluated LLMs, the strongest input bouncer bounced only 26.8% of the tickets that should be bounced, while the strongest output bouncer caught 61.0% of incorrect patches but also rejected 16.9% of correct ones. Combining both bouncers on a balanced subset raised the fraction of correct patches in the final output from about 36.9% to 57.4%, yet still left 142 wrong patches among 333 responses. The paper concludes that, while bouncing helps, current models are far from the reliability needed for real workflows.","pith_inferences":["Beyond the paper's claims, the highest-severity ensemble rule likely inflates the number of 'vague' tickets, so a majority-vote version of the benchmark could rank models differently.","Beyond the paper's claims, the output oracle is the dataset's own unit tests, which the annotations themselves flag as imperfect; output-bouncer scores therefore partly measure test quality rather than pure patch trustworthiness.","Beyond the paper's claims, the bouncer pattern generalizes: any LLM agent taking consequential actions could be gated by analogous input and output checks, and BouncerBench-style tasks could quantify abstention in those settings.","Beyond the paper's claims, a natural next experiment is to measure developer time saved by bouncing, not just the accuracy of the filtered patch set."],"forward_implications":["Resolution-rate leaderboards overstate agent usefulness because they reward answering every ticket and ignore the cost of wrong patches, which BouncerBench is designed to correct.","Reasoning models and codebase access improve bouncing relative to plain non-reasoning models, but even the best configuration agrees with human annotators only slightly beyond chance.","Deploying both bouncers in sequence raises the share of correct patches in the final output and cuts the number of wrong patches a developer reviews, while discarding some correct patches.","Abstention behaves like a distinct capability: models that resolve tickets well are not automatically good at deciding when to abstain."],"supporting_citations":[{"why":"Provides the real-world issue tickets, codebase checkpoints, and generated patch submissions that BouncerBench is built from.","marker":"[5]"},{"why":"Supplies the human specificity and test-validity annotations that define the ground truth for both bouncing tasks.","marker":"[19]"},{"why":"The annotation instructions the paper adapts into its input-bouncer prompt, tying the benchmark's labels to the original human judgment task.","marker":"[20]"}],"fun_headline_variants":["BouncerBench: AI coders can't tell when to abstain","Most AI coders can't refuse vague tickets — benchmark shows","Coding agents: no abstain button — BouncerBench scores the gap","AI coders fail the abstain test — BouncerBench reveals","BouncerBench: AI agents rarely know when to say no"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth for a vague ticket is the harshest label among three human annotators, so any ticket that one annotator found ambiguous is counted as underspecified; if those labels are noisy or the aggregation is too aggressive, the input-bouncing scores are not a clean measure of abstention.","fun_headline_variants_meta":{"raw":{"variants":["BouncerBench: AI coders can't tell when to abstain","Most AI coders can't refuse vague tickets — benchmark shows","Coding agents: no abstain button — BouncerBench scores the gap","AI coders fail the abstain test — BouncerBench reveals","BouncerBench: AI agents rarely know when to say no"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3088,"prompt_tokens":1049,"completion_tokens":2039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1943}},"tokens_in":665,"tokens_out":2039,"duration_ms":14519,"temperature":1.0,"reasoning_tokens":1943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:00:20.348480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the input-bouncing task with ground truth defined by majority vote instead of the highest severity label, and check whether the model rankings and the 38% 'should bounce' rate change substantially; if they do, the input task is an artifact of the ensemble rule.","supporting_citations":[{"cited_title":"Introducing SWE-bench Verified — openai.com,","cited_arxiv_id":null,"evidence_quote":"Supplies the human specificity and test-validity annotations that define the ground truth for both bouncing tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The annotation instructions the paper adapts into its input-bouncer prompt, tying the benchmark's labels to the original human judgment task."}],"review_version":1}