{"id":"b496978b-62ab-4598-8b5d-820491ca0d71","arxiv_id":"2507.02825","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Agentic benchmarks frequently mis-grade agents, and the new ABC checklist helps identify and correct such errors in ten popular benchmarks.","lead":"This paper examines how AI agent benchmarks score agents and shows that several widely used ones credit agents for doing nothing or for gaming the grading, sometimes by 38% or more. It introduces a 33-item checklist, ABC, to detect such flaws and demonstrates fixes on a cybersecurity benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ABC's construct validity is untested: an internally synthesized checklist with equal-weight 0/1 scoring is load-bearing, and the CVE-Bench 33% reduction rests on a self-audit.","rationale":"The reader's weakest assumption correctly identifies ABC's construct validity as the load-bearing issue, and my analysis agrees: the quantitative headline numbers, especially the CVE-Bench 33% reduction, depend on ABC being a valid instrument, and the evidence for that validity is internal to the authors' process. The paper's concrete exploits are independent and well-evidenced: SWE-Lancer's unauthenticated ZIP overwrite producing a 100% pass rate, tau-bench's 38% do-nothing success, and OSWorld's broken selectors are direct demonstrations of real flaws and do not depend on ABC's validity. Those findings alone support the paper's general claim that agentic benchmarks have serious evaluation issues. However, the 'systematic' claim about ABC and the precise reduction magnitude do depend on the unvalidated checklist construction. The equal-weight 0/1 aggregation is especially arbitrary: two benchmarks with the same score can have very different true error magnitudes, and no evidence is given that the score predicts anything. The CVE-Bench case study compounds this because the benchmark and the checklist share authors, and the expert confirmation is unspecified. These concerns do not overturn the paper's practical value, but they justify the reader's CONDITIONAL verdict: accept the direction and the documented exploits, while requiring external validation of ABC and a more careful quantitative framing. Verdict should remain CONDITIONAL, hence UNCHANGED relative to the reader's assessment.","tokens_in":34648,"tokens_out":3618,"duration_ms":44675,"concrete_test":"Pre-register the ABC items and have two independent teams, blind to each other and to the original authors' scores, apply ABC to CVE-Bench (pre-fix and post-fix) plus tau-bench and WebArena; compute Cohen's kappa on item-level scores. Then have an independent red team manually verify, on a held-out set of agent trajectories, whether the pre/post CVE-Bench success-rate drop corresponds to a real change in actual vulnerability exploitation (rather than a change in the checklist's definition of success). If inter-rater kappa is below 0.6, or if the 33% reduction does not reproduce under independent ground-truth verification, the systematicity claim for ABC and the precise magnitude of the CVE-Bench reduction are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires ABC to be a valid, systematic detector of evaluation flaws, but the checklist is synthesized from prior failure modes and the authors' own experience (Appendix C), with no external criterion for completeness or correctness. The paper itself concedes in Appendix A that the checklist may not be exhaustive. Item-level scoring is 0/1 equal-weight applied by the authors to each benchmark, with no reported inter-rater reliability, no weighting rationale, and no demonstration that the aggregate score correlates with true evaluation error. This matters most for the flagship quantitative claim: 'ABC reduces CVE-Bench's performance overestimation by 33%.' CVE-Bench was built by the same research group that built ABC, the 'cybersecurity experts' confirmation is described only as 'confirmed by cybersecurity experts' without a protocol, and the fix is judged against the same checklist that defined the flaw. There is therefore a real circularity risk: the checklist determines what counts as an issue, and then the same checklist measures the improvement. Separately, the KernelBench 31% overestimation is derived from an o3-mini-generated test set on 3 sampled kernels per level-1 task, a small and non-prespecified sample. If ABC is merely a codification of already-known failure modes, its 'systematic' status is not established against unseen or future benchmark flaws.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that many existing agentic benchmarks suffer from validity flaws in task setup or reward design, leading to under- or overestimation of agent performance by up to 100% in relative terms. To address this, the authors introduce the Agentic Benchmark Checklist (ABC), organized into task validity, outcome validity, and reporting criteria. They apply ABC to ten popular agentic benchmarks, identifying issues such as τ-bench's do-nothing agent success, SWE-Lancer's test-file overwrite vulnerability, KernelBench's insufficient fuzzing, and OSWorld's broken website selectors. They also present a case study in which applying ABC to CVE-Bench reduces a measured performance overestimation by 33% in absolute terms. The paper includes open-source code and detailed appendix materials.","tokens_in":34860,"tokens_out":3645,"duration_ms":41592,"significance":"If the central claims hold, this paper provides a valuable and timely contribution: concrete demonstrations of evaluation flaws in widely used benchmarks, a structured checklist for future benchmark construction, and quantitative evidence of how much reported agent performance can be distorted. The strengths include reproducible open-source code, multiple independent benchmark assessments (mostly third-party), and the identification of previously unknown issues such as the SWE-Lancer test-overwrite attack and OSWorld selector decay. The qualitative finding that many agentic benchmarks have validity problems is well supported. However, the paper's systematic status and flagship quantitative claims rest on the ABC checklist's construct validity and on a self-audit of CVE-Bench, both of which need stronger evidence before the checklist can be regarded as a validated instrument rather than a codification of known failure modes.","major_comments":[{"comment":"The construct validity of ABC is not established. The checklist items are synthesized from prior failure modes and the authors' own benchmark-building experience (Appendix C, Table 4), and the scoring rule is a 0/1 equal-weight average over applicable items (Section 5.1, Figure 5) with no reported inter-rater reliability, no weighting rationale, and no demonstration that the aggregate score correlates with true evaluation error. As the authors themselves concede in Appendix A, the checklist may not be exhaustive. Since the central claim is that ABC 'systematically' identifies evaluation flaws, the paper should provide external validation (e.g., application to benchmarks with known, independently documented flaws) or at least a sensitivity analysis showing that conclusions are robust to alternative item weightings.","section":"Sections 4.1-4.3, 5.1"},{"comment":"The flagship quantitative claim that ABC reduces CVE-Bench's performance overestimation by 33% is a self-assessment: CVE-Bench and ABC share the same research group, the fixes are judged against the very checklist items that defined the flaws, and the 'confirmed by cybersecurity experts' statement in Section 1 is not accompanied by any protocol or independent evaluation. This creates a circularity risk: the checklist determines what counts as an issue, and then the same checklist measures the improvement. The authors should address this by, for example, having an independent group apply ABC to CVE-Bench, pre-registering the evaluation criteria, or validating against an external measure of benchmark error.","section":"Section 5.3, Figure 6"},{"comment":"The KernelBench overestimation figure of 31% is derived from a small and non-prespecified sample: the authors sampled 3 generated kernels per level-1 task, asked o3-mini to generate additional test cases, manually verified them, and then re-evaluated all generations from Lange et al. No confidence interval or uncertainty quantification is reported for this point estimate. Given that the number of level-1 tasks is also not stated in the main text, the 31% figure is not sufficiently supported; the authors should either report the full sampling distribution, increase the sample size, or provide bounds on the estimate.","section":"Section 5.2, Appendix E.6"},{"comment":"Several quantitative overestimation figures are presented without sufficient experimental detail. For WebArena, the claimed 1.4-5.2% overestimate due to substring matching and LLM-as-a-judge issues is not accompanied by a protocol, the number of tasks evaluated, or the variance across runs. Similarly, the OSWorld 28% underestimation claim in the chrome section is based on 13/46 broken problems and one agent (UI-TAR), but no experimental setup or repeatability information is provided. These numbers are central to the abstract's 'up to 100% in relative terms' claim and should be reproducible from the appendix or the released code.","section":"Section 5.2, Appendix E.5"}],"minor_comments":[{"comment":"The confidence intervals in the modified BIRD leaderboard are inconsistent with the point estimates: for example, the first row reports 74.9% accuracy but a confidence interval of [66.8, 71.4], which does not contain the point estimate. Since this table is intended as an example of rigorous reporting, the errors undermine its illustrative value.","section":"Appendix F, Table 15"},{"comment":"The text 'An agentic benchmark mush involve tasks' contains a typo; 'mush' should be 'must'.","section":"Appendix B"},{"comment":"The phrase 'evaluation subjective' should be 'evaluation subject'; the same error appears in the assessment report for WebArena in Table 11.","section":"Section 4.3, R.6"},{"comment":"The sentence 'we identified significant evaluation issues that cases up to 100% errors' should read 'causes up to 100% errors'.","section":"Section 6"},{"comment":"The table header 'SWE-Bench-Lancer' is inconsistent with the 'SWE-Lancer' naming used throughout the rest of the paper.","section":"Table 5"},{"comment":"The label 'Avereage Score' in panel (c) should be 'Average Score'.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong qualitative core, but the checklist's validity and the CVE-Bench self-audit are load-bearing and need to be addressed before publication. I would also gently note that several of the paper's own reporting practices fall short of ABC's standards (e.g., no confidence intervals for the 31% and 28% figures), which the authors should reconcile in revision. The scope fits the journal, and the open-source release is a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. The paper does something concrete: it documents evaluation flaws in ten widely used agentic benchmarks, several of which are new and easily reproduced — a do-nothing agent passes 38% of TAU-bench Airline tasks, SWE-Lancer can be gamed to 100% by overwriting encrypted test archives, OSWorld's altered websites break 13/46 Chrome-task selectors, and WebArena's LLM judge accepts 'N/A' empty replies. Those findings alone justify the paper. The ABC checklist is a reasonable codification of known failure modes plus some new ones, and the three-way split (task validity, outcome validity, reporting) is a solid organizing frame. They shipped code and data, which makes the exploits checkable.\n\nThe soft spots are real but mostly in the packaging, not the core. The checklist scoring is 0/1 equal-weight applied by the authors themselves, with no inter-rater reliability reported; that limits any claim that ABC is a 'systematic' instrument. The KernelBench 31% overestimate rests on o3-mini-generated tests on three sampled kernels per level-1 task — fine as an upper-bound hint, but presented without sampling variance. The NeurIPS checklist's claim that all numbers are deterministic is odd given that sampling. And the CVE-Bench 33% reduction is a self-audit: same group built both the benchmark and the checklist, and the 'confirmed by cybersecurity experts' line has no protocol. That circularity doesn't undermine the individual fixes (the SQL-injection and outbound-server fixes are concrete), but it should be flagged as a demonstration, not independent validation.\n\nNone of this changes the main thrust: many agentic benchmarks have validity flaws that shift leaderboards, and the paper gives benchmark builders a practical way to look for them. The construct validity of ABC itself is the weakest link — the items were synthesized from known failures, so they may miss novel attack surfaces — but the authors acknowledge this in their limitations section. It is a tool, not a theorem.\n\nWho is it for: anyone building or consuming agentic benchmark scores. It deserves a serious referee: the empirical findings are verifiable, the checklist is actionable, and the weaknesses are addressable in revision (report inter-rater agreement, add uncertainty around sampled numbers, seek external confirmation for CVE-Bench). I'd send it out with that revision path in mind.","headline":"A practical, well-evidenced critique of agentic benchmarks; the checklist is useful but not yet a validated instrument.","tokens_in":35540,"tokens_out":2017,"would_cite":true,"duration_ms":21936,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Many widely used agentic benchmarks over- or underestimate AI agent performance by up to 100% in relative terms, and this paper's Agentic Benchmark Checklist (ABC) pinpoints and fixes such flaws.","keywords":["agentic benchmarks","evaluation rigor","task validity","outcome validity","benchmark checklist","reward design","AI agent evaluation","benchmark reporting"],"falsifier":"Audit a benchmark that passes every ABC item by running a do-nothing agent, an answer-dumping agent, and a fresh human re-grading of a random sample; if any of these still uncovers a material gap between reported and true performance, the claim that ABC captures the relevant failure modes is falsified.","tokens_in":34401,"feed_emoji":"📋","tokens_out":8173,"duration_ms":72780,"temperature":0.7,"pith_summary":"The paper argues that many widely used agentic benchmarks — evaluations in which AI agents complete multi-step, real-world tasks — contain flaws in task setup or reward design that skew reported performance by up to 100% in relative terms. It introduces the Agentic Benchmark Checklist (ABC), a set of yes/no checks organized around two validity conditions: task validity (a task is solvable if and only if the agent has the target capability) and outcome validity (a positive evaluation result genuinely means the task succeeded), plus a reporting standard. Auditing ten popular benchmarks, the authors find concrete violations: a do-nothing agent passes 38% of τ-bench airline tasks, SWE-Lancer can be scored 100% by overwriting its own test archive, and KernelBench overstates kernel correctness by about 31%. As a construction-time test, applying ABC to CVE-Bench cut a 32.5% overestimation from naive state matching and reduced the benchmark's total performance overestimation by 33%. The stakes are that leaderboard numbers drive model and product decisions, so knowing which numbers can be trusted is the point.","feed_headline":"Checklist exposes agentic benchmarks skewing AI scores up to 100%","feed_subtitle":"The Agentic Benchmark Checklist audits task and reward design; applied to CVE-Bench it cut overestimation by 33%.","key_machinery":"The Agentic Benchmark Checklist (ABC): roughly thirty yes/no checks grouped into three parts — task validity (tool versioning, API availability, environment isolation from ground truth, verified annotations, oracle solvers, implementation vulnerabilities), outcome validity (handling of semantically equivalent answers, negation, exhaustive listing, test-case quality, fuzz coverage, state spaces, guessing resistance, metric hacking), and benchmark reporting (open-sourcing, contamination prevention, flaw disclosure, trivial-agent and human baselines, confidence intervals). The load-bearing idea is the two-step decomposition: a benchmark is rigorous only if task success is equivalent to the target capability (task validity) and a positive evaluation result is equivalent to task success (outcome validity). Each item in the checklist is a concrete question targeting one of these equivalences, and the paper validates every violation it flags with an experiment — do-nothing agents, answer-dumping agents, overwriting test archives, and targeted fuzz inputs such as non-contiguous tensors.","core_discovery":"The central claim is that rigorous agentic evaluation consists of two equivalences that must both hold: the target capability must be equivalent to task success (task validity), and task success must be equivalent to a positive evaluation result (outcome validity). The paper shows these equivalences break in existing benchmarks in systematic, findable ways — insufficient unit tests in SWE-bench Verified, empty responses graded as success on impossible tasks in τ-bench, a password-protected test archive in SWE-Lancer whose contents can be overwritten without the password, and narrow fuzz inputs in KernelBench that miss failing memory layouts. It then claims that a checklist of concrete checks can surface these issues reliably: of the ten benchmarks assessed, seven violate task validity, seven violate outcome validity, and all ten fall short in reporting. The direct evidence for the checklist's usefulness is the CVE-Bench case study, where fixing the flagged items reduced performance overestimation by 33%.","pith_inferences":["A natural next step the paper does not take is validating ABC itself: measuring whether a benchmark's ABC score predicts its independently measured estimation error would test whether the checklist items are sufficient, rather than merely consistent with known failure modes.","The equal-weight 0/1 scoring used in the assessments is the authors' aggregation choice; since the paper reports per-issue error magnitudes, items could instead be weighted by measured impact, which would change which benchmarks look worst.","The taxonomy implies that agentic evaluation, unlike static benchmarks, is a game the agent plays against a dynamic environment, so reward-design flaws are likely to be discovered faster than label noise as agents improve — shortcut-finding should be treated as a first-class evaluation risk.","Several fixes suggested by ABC, such as pinned website snapshots, frozen tool versions, and trivial-agent regression tests in the harness, would turn ABC from a manual audit into a continuous evaluation-quality regression test run alongside every benchmark update."],"forward_implications":["Published leaderboard numbers on existing agentic benchmarks should be read with suspicion until trivial-agent, spamming-agent, and shortcut baselines are ruled out; in SWE-bench Verified, 24% of the top 50 leaderboard positions are incorrect.","Benchmark developers can run ABC during construction, not only after release; the CVE-Bench case shows that the checklist catches evaluation bugs, such as a 32.5% inflation from time-based injection state matching, before a benchmark is published.","The taxonomy diagnoses which layer of the evaluation chain is broken — design (wrong ground-truth state) versus implementation (agents can reach the ground truth) — which determines whether the fix requires re-annotation or re-engineering the harness.","Reporting discipline is itself part of validity: benchmarks that publish trivial-agent baselines, confidence intervals, and quantitative impact of known flaws, as illustrated with BIRD, give readers the information needed to discount the residual error.","Benchmarks that depend on live external resources, such as OSWorld's websites or WebArena's rate-limited APIs, degrade over time; OSWorld's chrome section currently underestimates a state-of-the-art agent by 28% because HTML selectors broke."],"supporting_citations":[{"why":"Supplies SWE-bench Verified, the paper's example of outcome invalidity from insufficient unit tests that pass incorrect patches.","marker":"[14]"},{"why":"Supplies τ-bench, where empty responses count as success on impossible tasks and a do-nothing agent passes 38% of airline tasks.","marker":"[86]"},{"why":"Supplies SWE-Lancer, shown to be scorable at 100% by an agent that overwrites the contents of the protected test archive.","marker":"[48]"},{"why":"Supplies KernelBench, whose fuzz testing is shown to overestimate kernel correctness by about 31%.","marker":"[59]"},{"why":"Supplies WebArena, whose substring matching and unvalidated LLM judge overestimate performance by 1.4–5.2%.","marker":"[93]"},{"why":"Supplies OSWorld, where live-website changes broke HTML selectors, underestimating a state-of-the-art agent by 28%.","marker":"[83]"},{"why":"Supplies CVE-Bench, the case study where ABC's items on state matching and oracle verification cut overestimation by 33%.","marker":"[96]"},{"why":"Prior work documenting that SWE-bench patches can pass tests without resolving issues; the paper builds its outcome-validity analysis on it.","marker":"[87]"},{"why":"Prior analysis of KernelBench grading flaws and ground-truth leakage that the paper extends with additional fuzz-coverage experiments.","marker":"[35]"}],"fun_headline_variants":["Agentic benchmark flaws skew scores up to 100% — checklist helps","ABC checklist: rigorous agentic benchmarks, less overestimation","Benchmark issues inflate agent scores by 100%; new checklist","Agentic Benchmark Checklist: expose flaws, cut overestimation 33%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that ABC's yes/no items are the right and sufficient indicators of evaluation rigor, since the items were synthesized from documented failures and the authors' experience and were never checked against an independent measure of benchmark quality.","fun_headline_variants_meta":{"raw":{"variants":["Agentic benchmark flaws skew scores up to 100% — checklist helps","ABC checklist: rigorous agentic benchmarks, less overestimation","Benchmark issues inflate agent scores by 100%; new checklist","Agentic Benchmark Checklist: expose flaws, cut overestimation 33%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2487,"prompt_tokens":896,"completion_tokens":1591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1514}},"tokens_in":512,"tokens_out":1591,"duration_ms":13950,"temperature":1.0,"reasoning_tokens":1514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:20:04.920366+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a benchmark that passes every ABC item by running a do-nothing agent, an answer-dumping agent, and a fresh human re-grading of a random sample; if any of these still uncovers a material gap between reported and true performance, the claim that ABC captures the relevant failure modes is falsified.","supporting_citations":[{"cited_title":"tau-bench: A bench- mark for tool-agent-user interaction in real-world domains","cited_arxiv_id":null,"evidence_quote":"Supplies τ-bench, where empty responses count as success on impossible tasks and a do-nothing agent passes 38% of airline tasks."},{"cited_title":"Kernelbench: Can llms write efficient gpu kernels? In ICLR 2025 Third Workshop on Deep Learning for Code (Best paper award), 2025","cited_arxiv_id":null,"evidence_quote":"Supplies KernelBench, whose fuzz testing is shown to overestimate kernel correctness by about 31%."},{"cited_title":"Webarena: A realistic web environment for build- ing autonomous agents","cited_arxiv_id":null,"evidence_quote":"Supplies WebArena, whose substring matching and unvalidated LLM judge overestimate performance by 1.4–5.2%."},{"cited_title":"Utboost: Rigorous evaluation of coding agents on swe-bench","cited_arxiv_id":null,"evidence_quote":"Prior work documenting that SWE-bench patches can pass tests without resolving issues; the paper builds its outcome-validity analysis on it."}],"review_version":1}