{"id":"6b076ad0-1d22-4039-a48f-2922753d048b","arxiv_id":"2607.27518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM transcript scanners can surface real validity flaws in agentic benchmarks, including ground-truth leakage and answer-format ambiguity, but sensitivity varies by criterion, benchmark, and model.","lead":"Researchers built AI scanners that read agent transcripts to flag four types of benchmark flaws—ground truth leakage, tool failures, guessable tasks, and unclear answer formats—and tested them against human judgments across eleven agentic benchmarks. The scanners found several confirmed quality problems in widely used benchmarks, but performance varied widely and human validation was not fully independent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set human labels were not independent: reviewers saw scanner outputs (§3.3), so reported F1/QWK may be inflated by anchoring; blinded re-validation is required.","rationale":"The reader's weakest assumption—that human post-validation labels are a reliable, independent ground truth—is exactly the load-bearing concern. §3.3 explicitly gives test-set reviewers access to scanner outputs, and §5 acknowledges human grades are not definitive. Without this assumption, the F1/QWK metrics that underpin the central claim (scanners 'can be usefully implemented') are not trustworthy. This is more fundamental than the secondary overclaim about 'random manual inspection,' because it affects the entire quantitative validation. The paper is otherwise a careful proof-of-concept with open-source artifacts and credible qualitative examples, and the flaw is addressable via blinded re-grading, so the reader's CONDITIONAL verdict is appropriate—no change is needed.","tokens_in":31754,"tokens_out":4982,"duration_ms":54531,"concrete_test":"Re-grade a stratified random sample of 150–200 test-set transcripts (oversampling scanner-flagged cases as in §3.9) with human reviewers who are blinded to scanner outputs but have access to the same rubrics and supporting tools. Compute inter-rater agreement (Cohen's kappa/QWK) between blind and original labels, and recompute scanner sensitivity, specificity, F1, and QWK against the blind labels using the same inverse-probability weighting. If F1 or QWK drops by more than ~0.1 for any criterion, the reported performance is inflated by the validation loop; if the metrics are within the original confidence intervals, the concern is resolved. Reporting human-human agreement on the blinded set is also essential.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that scanners track human judgments with F1 up to 0.74 and can prioritize review—rests entirely on human post-validation labels from the test set. But §3.3 states these reviewers had access to scanner-generated outputs. This creates a validation loop: if human graders were anchored by scanner flags, their labels are not an independent ground truth, and the agreement metrics (sensitivity, specificity, F1, QWK) in Table 4 and Figure 2 are systematically inflated. The paper itself concedes in §5 that human grades 'should not be taken as a definitive judgement' and, in the limitations, that graders 'often had reasonable disagreements'—yet no inter-rater reliability is reported for the test set. Under this condition, the performance numbers that support 'usefully implemented' are not independently confirmed. A secondary but related issue is the unsupported claim that found flaws 'would be difficult to find through random manual inspection alone' (Abstract, §5); no random-manual baseline was run. The qualitative examples (e.g., CORE-Bench leakage) are compelling, but they do not rescue the quantitative validation, which is the paper's basis for claiming scanners are broadly useful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops AI 'scanners' that analyze agentic benchmark transcripts for four classes of validity violations: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. Scanners are built on Inspect Scout, developed using human-graded and synthetically manipulated transcripts, then frozen and evaluated on a held-out set of transcripts across several Inspect-based benchmarks. Performance is measured against human post-validation labels using sensitivity, specificity, F1, and quadratic-weighted Cohen's kappa. The authors also report verified benchmark issues found in CORE-Bench, SWE-Bench-Verified, KernelBench, CVE-Bench, and Terminal-Bench-2.0, and they open-source their scanners and dataset. The central claim is that automated transcript analysis can serve as a practical, supplemental quality-assurance layer for agentic benchmarks, with scanner performance sufficient to prioritize human review.","tokens_in":32012,"tokens_out":3832,"duration_ms":44393,"significance":"If the quantitative validation held, this would be a useful contribution to a growing area: scalable audit of agentic benchmarks. The qualitative evidence is genuinely strong—the CORE-Bench example in Figure 4, where the answer is readable directly in the plotting script, and the SWE-Bench-Verified example in Appendix A.1, where the agent applies an upstream git patch verbatim, are compelling and independently verifiable from the transcripts. The open-sourced scanners, rubrics, and dataset are concrete assets that other researchers can build on. However, the reported F1/QWK numbers are not independently confirmed because of the validation loop described below; the paper's own limitations section concedes that human grades are not definitive. With a blinded re-validation or a reframing as 'scanner-human agreement' rather than 'validated performance,' the contribution would be solid: the qualitative findings and the open tooling alone justify publication as a proof of concept in a specialized venue.","major_comments":[{"comment":"The quantitative validation is not independent. The manuscript states in §3.3 that for the test set 'human reviewers had access to the scanner generated outputs as well as the same supporting tools'. Since human labels are the referent for sensitivity, specificity, F1, and QWK in Table 4, scanner flags can anchor human graders toward agreement, inflating all agreement metrics. The inverse-probability weighting in §3.10 corrects for stratified sampling but cannot remove this dependence. This is compounded by the absence of any inter-rater reliability statistic for the test set and by the paper's own §5 caveat that 'human grades should therefore not be taken as a definitive judgement.' I would ask for a blinded re-validation on a sample (human graders labeling without seeing scanner output), or, at minimum, a reframing of Table 4 and Figure 3 as scanner–human agreement rather than validate","section":"§3.3, §3.10, Table 4"},{"comment":"The claim that found flaws 'would be difficult to find through random manual inspection alone' is not supported by any experimental baseline. No random-manual inspection condition was run, so there is no evidence about the detection rate of unassisted human review. If this claim is retained, a simple baseline should be added—for example, having human graders review a random sample of transcripts without scanner assistance. Otherwise the claim should be removed or weakened to something like 'not previously reported by manual audits' for the specific examples.","section":"Abstract, §5, Conclusion"},{"comment":"The sensitivity and F1 estimates rest on very small numbers of confirmed positives. For example, in Table 6 the Tool Failure GPT-5.4 row has 9 true positives, and the Ground Truth Access rows have 52 and 32 unweighted true positives; correspondingly, the confidence intervals in Table 4 are wide (e.g., tool failure F1 0.13–0.41). The paper acknowledges this difficulty in §3.9 ('difficult to attain our sample size target of 100 verified positives'), but the abstract's 'F1 up to 0.74' and the Discussion's 'usefully implemented' should be tempered by the fact that most scanner/criterion combinations have substantially wider and lower estimates. This is not fatal to the proof-of-concept, but the strength of the quantitative conclusions should be scaled to the precision actually achieved.","section":"§3.9, Table 4, Table 6"}],"minor_comments":[{"comment":"Typo: 'There where several cases' should be 'There were several cases.'","section":"§4.2.2"},{"comment":"SWE-Bench-Verified is attributed to Yu et al. (2025), but the cited Yu et al. reference is 'UTBoost: Rigorous evaluation of coding agents on SWE-bench,' not the original SWE-Bench-Verified dataset paper. Please correct or clarify the citation.","section":"§1, References"},{"comment":"Figure 2's caption notes that some cells had perfect agreement and therefore no bootstrap CI. It would be helpful to state explicitly how many transcripts are in those cells, since with very small counts the absence of a CI can be misleading.","section":"§4.1, Figure 2"},{"comment":"The number of human reviewers (n=5) is reported for the development set, but not separately for the test set post-validation. Please report the test-set reviewer count and the adjudication procedure in the main text.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The central problem is a fixable validation-loop issue: test-set human graders saw scanner outputs, so the reported agreement metrics are not independent confirmation. I would not reject, because the qualitative findings and the released tooling are substantive contributions. If the authors add even a modest blinded re-validation sample and adjust the claims accordingly, the paper would likely meet the bar for acceptance. The 'random manual inspection' claim also needs a baseline or removal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a real contribution: four working scanners for benchmark validity flaws, open-sourced, with a dataset and a sampling framework. The qualitative finds—CORE-Bench answers sitting in plotting code, SWE-Bench agents pulling gold patches from git history—are concrete and valuable. Second, the quantitative validation rests on human labels that were not produced blind: §3.3 says test-set graders saw scanner outputs before grading. That means the reported F1/QWK numbers could be inflated by anchoring, and the claim that scanners track human judgment is not independently confirmed. The authors even concede that human grades aren't definitive, but they don't report inter-rater reliability.\n\nWhat's new: prior tools (Docent, BenchGuard, Auto Benchmark Audit) did transcript auditing, but none of them evaluated criteria-specific scanners against human labels on a held-out set with synthetic development data. The scanners, rubrics, and dataset are useful resources. The sample-size calculations are a nice practical addition.\n\nSoft spots beyond the validation loop: the claim that issues would be \"unlikely to be caught by random manual inspection\" has no baseline behind it—that's an overclaim. Scanner performance is very uneven; the tool failure scanner is close to useless, especially with Sonnet. The paper honestly shows this, but it means the headline F1 of 0.74 is the best case, not the norm. The per-benchmark validation samples are small, and confidence intervals are wide.\n\nWho it's for: people building or auditing agentic benchmarks, and anyone who needs a lower-bound QA layer. It's a proof of concept, not an assurance tool. As a referee, I'd ask for a blinded re-validation with inter-rater reliability and a formal baseline for \"difficult to find manually.\" Those are addressable.\n\nYes, send to peer review. The core proof-of-concept holds qualitatively, and the flaws are fixable. The authors should be pushed to address the anchoring concern head-on and soften the manual-inspection claim.","headline":"A genuinely useful proof-of-concept for automated benchmark auditing, but the quantitative validation is undermined by non-independent human labels; fixable, worth refereeing.","tokens_in":32501,"tokens_out":2279,"would_cite":true,"duration_ms":22901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated transcript scanners can flag validity flaws in agentic benchmarks, from leaked answers to guessable tasks, when paired with human confirmation.","keywords":["agentic benchmarks","benchmark validity","transcript analysis","AI scanners","ground truth leakage","guessing vulnerability","answer format ambiguity","benchmark auditing"],"falsifier":"A blinded replication in which a new set of human graders, who never see scanner outputs, independently grades a sample of the same transcripts using the published rubrics; if the scanner's 'verified issues' are not confirmed or the F1 drops sharply, the human-label ground truth was not independent.","tokens_in":31644,"feed_emoji":"🔍","tokens_out":3655,"duration_ms":43225,"temperature":0.7,"pith_summary":"The paper tries to show that automated scanners—AI models that read agent rollout transcripts—can surface validity flaws in agentic benchmarks, including flaws that are hard to catch by random manual inspection. It defines four scanner-viable criteria (ground truth access, tool failure, guessing vulnerability, answer format ambiguity), builds prompt-and-rubric scanners for them, and validates the scanners against human labels on held-out transcripts. On the test set, scanner flag rates tracked human severity ratings, and the best scanner configuration reached an F1 of 0.74 for guessing vulnerability. The scanners verified real issues in five widely used benchmarks, but performance varied by criterion, benchmark, and judge model, so the authors position the result as a proof of concept for scalable audit triage rather than a finished assurance tool.","feed_headline":"AI scanners catch benchmark flaws that manual review misses","feed_subtitle":"Automated transcript analysis flags leaked answers and guessable tasks in five agentic benchmarks, with human check as backup.","key_machinery":"The carrying mechanism is a set of four scanner prompts, each paired with a 0–3 severity rubric, that operate on agent transcripts and output a severity grade. Human graders use the same rubrics as the reference. A stratified post-validation design over-samples scanner-flagged transcripts, and inverse-probability weighting corrects reported performance back to population-level estimates. Sample-size calculations accompany the method, specifying how many transcripts are needed to rule out high violation rates. The scanners themselves are transferable across benchmarks; only the supplementary context given to them changes.","core_discovery":"The central claim is that LLM-based transcript scanners, built from grading rubrics and run over agent rollout logs, can detect a meaningful fraction of validity violations that compromise agentic benchmarks. Against human post-validation labels, scanner flag rates rose with human-assigned severity, and the strongest scanner (guessing vulnerability) achieved an F1 of 0.74 with a sensitivity of 0.93. The scanners also surfaced verified issues humans had not noticed unaided, such as answers exposed verbatim in source code, models retrieving upstream gold patches from git history, and answer spaces small enough to guess. The authors emphasize that these results are a lower bound: no model fine-","pith_inferences":["A natural extension the paper points toward is combining scanners with deliberately adversarial red-team agents; recording agents that attempt reward-hacking while scanners watch could surface vulnerabilities faster than scanning honest transcripts alone.","Cross-model scanner disagreement may itself be diagnostic: the paper's development data show transcripts where two scanners agree align better with human grades, suggesting disagreement flags genuinely ambiguous cases worthy of human review.","If scanners generalize across benchmarks that standardize the context they provide, benchmark release could routinely include a scanned-transcript check, turning quality assurance from an occasional audit into a regular step.","The four textual criteria likely transfer to non-agentic evaluations: the paper's own non-agentic explorations show answer-format and guessing scanners flagging issues in multiple-choice benchmarks, implying a common auditing pipeline across evaluation types."],"forward_implications":["Scanner flag rates that track human severity ratings support using scanners to triage transcripts, concentrating scarce human review on the most suspicious cases.","Low scanner flag rates for a benchmark/model combination are evidence of few validity violations, since humans rarely found violations in transcripts scanners did not flag.","Roughly 75 human-graded or 120 scanner-graded transcripts suffice to rule out a greater-than-5% violation rate with 95% confidence, assuming no violations are found and positives are verified.","Composite scoring across two judge models can trade sensitivity for specificity: taking the max flag raises recall, while requiring agreement raises precision, giving users a tunable audit","Scanners can catch issues that are difficult to see by reading code or datasets alone, such as answers embedded in plotting scripts or exact solutions recovered from upstream git history."],"fun_headline_variants":["AI scanners expose hidden flaws in agentic benchmarks","Automated audit catches benchmark leaks humans overlook","LLM-based scanners spot benchmark cheating vulnerabilities","AI transcript analysis finds benchmark validity gaps","Benchmark auditing with AI flags answer leaks and guessable tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Human labels are treated as the ground truth for scanner performance, but the human graders could see the scanner outputs and the authors concede their grades are subjective, so a bias toward agreeing with the scanner would inflate the reported sensitivity and F1.","fun_headline_variants_meta":{"raw":{"variants":["AI scanners expose hidden flaws in agentic benchmarks","Automated audit catches benchmark leaks humans overlook","LLM-based scanners spot benchmark cheating vulnerabilities","AI transcript analysis finds benchmark validity gaps","Benchmark auditing with AI flags answer leaks and guessable tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3427,"prompt_tokens":728,"completion_tokens":2699,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2629}},"tokens_in":472,"tokens_out":2699,"duration_ms":20435,"temperature":1.0,"reasoning_tokens":2629,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:20:42.614910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blinded replication in which a new set of human graders, who never see scanner outputs, independently grades a sample of the same transcripts using the published rubrics; if the scanner's 'verified issues' are not confirmed or the F1 drops sharply, the human-label ground truth was not independent.","supporting_citations":[],"review_version":1}