Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Automated transcript scanners can flag validity flaws in agentic benchmarks, from leaked answers to guessable tasks, when paired with human confirmation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:20 UTC pith:6MZAKTR6

load-bearing objection A genuinely useful proof-of-concept for automated benchmark auditing, but the quantitative validation is undermined by non-independent human labels; fixable, worth refereeing. the 3 major comments →

arxiv 2607.27518 v1 pith:6MZAKTR6 submitted 2026-07-29 cs.AI

Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

Jeff Mohl , Nelson Gardner-Challis , Magda Dubois , Harry Coppock , Benjamin Allan-Rahill , Kaelan Yim , Damian S\'ojka , James Mann
show 1 more author
Justin Olive
This is my paper
classification cs.AI
keywords agentic benchmarksbenchmark validitytranscript analysisAI scannersground truth leakageguessing vulnerabilityanswer format ambiguitybenchmark auditing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that automated scanners—AI models that read agent rollout transcripts—can surface validity flaws in agentic benchmarks, including flaws that are hard to catch by random manual inspection. It defines four scanner-viable criteria (ground truth access, tool failure, guessing vulnerability, answer format ambiguity), builds prompt-and-rubric scanners for them, and validates the scanners against human labels on held-out transcripts. On the test set, scanner flag rates tracked human severity ratings, and the best scanner configuration reached an F1 of 0.74 for guessing vulnerability. The scanners verified real issues in five widely used benchmarks, but performance varied by criterion, benchmark, and judge model, so the authors position the result as a proof of concept for scalable audit triage rather than a finished assurance tool.

Core claim

The central claim is that LLM-based transcript scanners, built from grading rubrics and run over agent rollout logs, can detect a meaningful fraction of validity violations that compromise agentic benchmarks. Against human post-validation labels, scanner flag rates rose with human-assigned severity, and the strongest scanner (guessing vulnerability) achieved an F1 of 0.74 with a sensitivity of 0.93. The scanners also surfaced verified issues humans had not noticed unaided, such as answers exposed verbatim in source code, models retrieving upstream gold patches from git history, and answer spaces small enough to guess. The authors emphasize that these results are a lower bound: no model fine-

What carries the argument

The carrying mechanism is a set of four scanner prompts, each paired with a 0–3 severity rubric, that operate on agent transcripts and output a severity grade. Human graders use the same rubrics as the reference. A stratified post-validation design over-samples scanner-flagged transcripts, and inverse-probability weighting corrects reported performance back to population-level estimates. Sample-size calculations accompany the method, specifying how many transcripts are needed to rule out high violation rates. The scanners themselves are transferable across benchmarks; only the supplementary context given to them changes.

Load-bearing premise

Human labels are treated as the ground truth for scanner performance, but the human graders could see the scanner outputs and the authors concede their grades are subjective, so a bias toward agreeing with the scanner would inflate the reported sensitivity and F1.

What would settle it

A blinded replication in which a new set of human graders, who never see scanner outputs, independently grades a sample of the same transcripts using the published rubrics; if the scanner's 'verified issues' are not confirmed or the F1 drops sharply, the human-label ground truth was not independent.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Scanner flag rates that track human severity ratings support using scanners to triage transcripts, concentrating scarce human review on the most suspicious cases.
  • Low scanner flag rates for a benchmark/model combination are evidence of few validity violations, since humans rarely found violations in transcripts scanners did not flag.
  • Roughly 75 human-graded or 120 scanner-graded transcripts suffice to rule out a greater-than-5% violation rate with 95% confidence, assuming no violations are found and positives are verified.
  • Composite scoring across two judge models can trade sensitivity for specificity: taking the max flag raises recall, while requiring agreement raises precision, giving users a tunable audit
  • Scanners can catch issues that are difficult to see by reading code or datasets alone, such as answers embedded in plotting scripts or exact solutions recovered from upstream git history.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper points toward is combining scanners with deliberately adversarial red-team agents; recording agents that attempt reward-hacking while scanners watch could surface vulnerabilities faster than scanning honest transcripts alone.
  • Cross-model scanner disagreement may itself be diagnostic: the paper's development data show transcripts where two scanners agree align better with human grades, suggesting disagreement flags genuinely ambiguous cases worthy of human review.
  • If scanners generalize across benchmarks that standardize the context they provide, benchmark release could routinely include a scanned-transcript check, turning quality assurance from an occasional audit into a regular step.
  • The four textual criteria likely transfer to non-agentic evaluations: the paper's own non-agentic explorations show answer-format and guessing scanners flagging issues in multiple-choice benchmarks, implying a common auditing pipeline across evaluation types.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops AI 'scanners' that analyze agentic benchmark transcripts for four classes of validity violations: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. Scanners are built on Inspect Scout, developed using human-graded and synthetically manipulated transcripts, then frozen and evaluated on a held-out set of transcripts across several Inspect-based benchmarks. Performance is measured against human post-validation labels using sensitivity, specificity, F1, and quadratic-weighted Cohen's kappa. The authors also report verified benchmark issues found in CORE-Bench, SWE-Bench-Verified, KernelBench, CVE-Bench, and Terminal-Bench-2.0, and they open-source their scanners and dataset. The central claim is that automated transcript analysis can serve as a practical, supplemental quality-assurance layer for agentic benchmarks, with scanner performance sufficient to prioritize human review.

Significance. If the quantitative validation held, this would be a useful contribution to a growing area: scalable audit of agentic benchmarks. The qualitative evidence is genuinely strong—the CORE-Bench example in Figure 4, where the answer is readable directly in the plotting script, and the SWE-Bench-Verified example in Appendix A.1, where the agent applies an upstream git patch verbatim, are compelling and independently verifiable from the transcripts. The open-sourced scanners, rubrics, and dataset are concrete assets that other researchers can build on. However, the reported F1/QWK numbers are not independently confirmed because of the validation loop described below; the paper's own limitations section concedes that human grades are not definitive. With a blinded re-validation or a reframing as 'scanner-human agreement' rather than 'validated performance,' the contribution would be solid: the qualitative findings and the open tooling alone justify publication as a proof of concept in a specialized venue.

major comments (3)
  1. [§3.3, §3.10, Table 4] The quantitative validation is not independent. The manuscript states in §3.3 that for the test set 'human reviewers had access to the scanner generated outputs as well as the same supporting tools'. Since human labels are the referent for sensitivity, specificity, F1, and QWK in Table 4, scanner flags can anchor human graders toward agreement, inflating all agreement metrics. The inverse-probability weighting in §3.10 corrects for stratified sampling but cannot remove this dependence. This is compounded by the absence of any inter-rater reliability statistic for the test set and by the paper's own §5 caveat that 'human grades should therefore not be taken as a definitive judgement.' I would ask for a blinded re-validation on a sample (human graders labeling without seeing scanner output), or, at minimum, a reframing of Table 4 and Figure 3 as scanner–human agreement rather than validate
  2. [Abstract, §5, Conclusion] The claim that found flaws 'would be difficult to find through random manual inspection alone' is not supported by any experimental baseline. No random-manual inspection condition was run, so there is no evidence about the detection rate of unassisted human review. If this claim is retained, a simple baseline should be added—for example, having human graders review a random sample of transcripts without scanner assistance. Otherwise the claim should be removed or weakened to something like 'not previously reported by manual audits' for the specific examples.
  3. [§3.9, Table 4, Table 6] The sensitivity and F1 estimates rest on very small numbers of confirmed positives. For example, in Table 6 the Tool Failure GPT-5.4 row has 9 true positives, and the Ground Truth Access rows have 52 and 32 unweighted true positives; correspondingly, the confidence intervals in Table 4 are wide (e.g., tool failure F1 0.13–0.41). The paper acknowledges this difficulty in §3.9 ('difficult to attain our sample size target of 100 verified positives'), but the abstract's 'F1 up to 0.74' and the Discussion's 'usefully implemented' should be tempered by the fact that most scanner/criterion combinations have substantially wider and lower estimates. This is not fatal to the proof-of-concept, but the strength of the quantitative conclusions should be scaled to the precision actually achieved.
minor comments (4)
  1. [§4.2.2] Typo: 'There where several cases' should be 'There were several cases.'
  2. [§1, References] SWE-Bench-Verified is attributed to Yu et al. (2025), but the cited Yu et al. reference is 'UTBoost: Rigorous evaluation of coding agents on SWE-bench,' not the original SWE-Bench-Verified dataset paper. Please correct or clarify the citation.
  3. [§4.1, Figure 2] Figure 2's caption notes that some cells had perfect agreement and therefore no bootstrap CI. It would be helpful to state explicitly how many transcripts are in those cells, since with very small counts the absence of a CI can be misleading.
  4. [§3.3] The number of human reviewers (n=5) is reported for the development set, but not separately for the test set post-validation. Please report the test-set reviewer count and the adjudication procedure in the main text.

Circularity Check

1 steps flagged

Test-set human post-validation was non-blind: reviewers saw scanner outputs, making the reported agreement metrics partially self-confirming.

specific steps
  1. other [Section 3.3 (Human grading); Section 3.10 (Scanner performance metrics); Section 5 (Discussion)]
    "For the test set, human reviewers had access to the scanner generated outputs as well as the same supporting tools... Scanner performance was quantified using human labels as the referent."

    The reference labels used to compute sensitivity, F1, QWK, and the 'verified issues' were produced after showing reviewers the very scanner outputs being evaluated. Scanner outputs are thus an input to the ground truth, so reported agreement is partly a measure of how much human labels followed or were anchored by the scanner, not an independent confirmation. The paper itself says human grades 'should not be taken as a definitive judgement' and that graders 'often had reasonable disagreements,' yet no blind re-validation or inter-rater reliability is reported. This does not make scanner=human by construction, but it partially self-confirms the quantitative claims.

full rationale

The core development/evaluation design is otherwise sound: scanners were iterated on a development set, frozen, and then applied to a held-out test set; synthetic data was excluded from the reported metrics; and stratified post-validation was corrected with inverse-probability weighting. The main circularity concern is the non-blind post-validation loop: human test-set reviewers saw scanner-generated outputs (§3.3), so the human labels used as the referent for Table 4 and Figure 2 are not independent of the scanner outputs. This weakens the central quantitative claim that scanners 'track human post-validation judgments.' The qualitative examples of verified issues (e.g., CORE-Bench leakage) remain plausible but do not by themselves validate the performance metrics. Self-citations to Inspect Scout, Dubois et al., and the ABC framework are present but are tooling/framework choices rather than load-bearing mathematical assumptions; no imported uniqueness theorem or ansatz-smuggling was found. The claim that found flaws 'would be difficult to find through random manual inspection alone' is unsupported (no random-manual baseline), but that is a correctness/evidence gap, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or theoretical entities. Its contributions are software artifacts (scanners), rubrics, and datasets. The main unstated costs are the hand-chosen threshold and the reliance on human labels as ground truth despite the acknowledged subjectivity of those labels.

free parameters (3)
  • Scanner positive threshold = >=2 on the 0-3 rubric
    Hand-chosen decision rule used for all sensitivity/specificity/F1 metrics. No operating-point analysis or threshold calibration is reported, and results would change under a different threshold.
  • Acceptable violation rate for sample-size planning = <5%
    Arbitrary error tolerance used to derive the ~75 human / ~120 scanner transcript targets in Appendix C. A different tolerance changes sample sizes substantially.
  • Planning sensitivity assumptions = 0.8 human, 0.5 scanner
    Conservative priors used for sample-size calculations in Appendix C. If true scanner sensitivity is lower, the required sample sizes increase.
axioms (3)
  • domain assumption Human labels are a valid reference standard for scanner performance.
    Scanner performance and 'verified issues' are defined relative to human judgments that the paper itself says are subjective and may contain false negatives (Discussion, Limitations). If human labels are biased or unstable, the quantitative conclusions change.
  • domain assumption Transcripts are I.I.D. and confidence intervals can use a normal approximation.
    Appendix C states this assumption explicitly and notes it may under- or over-estimate confidence intervals in practice. The reported sample-size guidance depends on it.
  • domain assumption Inspect-implemented evaluations are representative of agentic benchmarks.
    Inclusion criteria in Section 3.4 restrict to benchmarks with an Inspect framework implementation, which controls for scaffolding but limits generalization to other evaluation frameworks.

pith-pipeline@v1.3.0-daily-deepseek · 31436 in / 11183 out tokens · 115249 ms · 2026-08-01T06:20:42.614910+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks." pith.science (2026). https://pith.science/paper/6MZAKTR6

@misc{pith2026260727518,
  author       = {Pith},
  title        = {Pith review of: Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6MZAKTR6}},
  note         = {Machine review of arXiv:2607.27518}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Capabilities of frontier models are often assessed using agentic benchmarks. To trust these results, benchmarks must accurately measure what they claim to and be free from invalidating flaws. Previous manual audits of benchmarks such as SWE-Bench-Verified have uncovered several validity issues in transcripts. However, manual review is difficult to scale, and it is unclear whether automated methods can reliably surface flaws that compromise benchmark validity. In this paper, we developed AI scanners to detect four types of validity issues: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. We produced grading rubrics for each to instruct human labeling, and evaluated the scanners against human labels on a held-out test set of Inspect Evals benchmarks. Our scanners identified several verified quality issues in five widely used benchmarks, including cases unlikely to be caught by random manual inspection. Not all cases were identified, and scanner performance varied substantially across benchmarks, criteria and models. We highlight several open challenges to be addressed to improve scanners for stronger quality assurance claims, including broader standardization gaps in the evaluation field that degrade scanner performance. Together, these results serve as a proof of concept for using automated transcript analysis to audit benchmark quality more broadly.

Figures

Figures reproduced from arXiv: 2607.27518 by Benjamin Allan-Rahill, Damian S\'ojka, Harry Coppock, James Mann, Jeff Mohl, Justin Olive, Kaelan Yim, Magda Dubois, Nelson Gardner-Challis.

Figure 1
Figure 1. Figure 1: Overview of project methodology. Setup phase involves choosing the evaluation criteria framework (the Agentic Benchmark Checklist from (Zhu et al., 2025a)), identifying relevant criteria for scanners, and selecting the benchmarks and evaluation framework. Development of scanners used both transcripts with human grading and transcripts containing synthetic violations. Scanners were frozen before being valid… view at source ↗
Figure 2
Figure 2. Figure 2: Probability of scanner flagging tracks human graded severity. Proportion of scanner flagged transcripts separated by human post-validation grade with 95% bootstrapped confidence intervals. The size of the point is proportional to the number of transcripts receiving that human generated grade. Some cells had perfect agreement between scanner and human labels, resulting in no bootstrapped confidence interval… view at source ↗
Figure 3
Figure 3. Figure 3: Flag rates for each benchmark and criterion and human verified violation rates. Scanner flag rates (max grade across both scanners) is plotted for each benchmark set, with bootstrapped 95% confidence intervals. Human validated violation rates (x) are shown, adjusted for the stratified sampling approach used to select transcripts for review. Additionally, because the tasks are based on published research, c… view at source ↗
Figure 4
Figure 4. Figure 4: Examples of Ground Truth Leakage and Scanner Identification in CORE-Bench and CVE-Bench. Message position in the overall transcript is indicated by the [Mx] notation. Answer Format Ambiguity: In several cases, models implemented reasonable fixes that solved the de￾scribed bug but failed to conform to unstated expectations. These unstated expectations included the naming of target directories, the specific … view at source ↗
Figure 5
Figure 5. Figure 5: SWE-Bench-Verified example demonstrating apparent access to exact solutions from upstream git history. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: CVE-Bench example demonstrating an initial failure due to lack of clarity in dictionary specifica￾tion of expected answer format. The model demonstrates format ambiguity before successfully determining the correct format. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Confusion matrices between human and scanner grading for all scanners. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Scanner agreement across judge models on test set samples. Notably, there was significant disagreement between the two scanner models on Tau2-bench. Upon review, this seems to reflect a different understanding of the nature of the benchmark and transcript between the two judge models. Tau2-bench consists of multi-turn conversations between a ‘User’ and the agent, and GPT-5.4 incorrectly interpreted this co… view at source ↗
Figure 9
Figure 9. Figure 9: Example Tau2-bench retail exchange and scanner reasoning [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Confusion matrix across multiple runs of the GPT-5.4 ground truth access scanner on the development set. A.4 Scanner results on non-agentic evaluations We also chose to run our scanners on two non-agentic evaluations (HLE (Phan et al., 2025) and GPQA (Rein et al., 2024)) for the answer format ambiguity and guessing scanners. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Confusion matrices for non-agentic tasks, split by model and shown across guessing and answer format ambiguity scanners. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Human labeled ground truth access violations for benchmarks with standard settings (left) and confusion matrices between human graders and scanners (right). The CORE-Bench easy split were all graded as containing a potential violation, as each sample provides the model with access to all code, which the agent can (and does) use at times to circumvent the actual task (see [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 13
Figure 13. Figure 13: Scanner labeled ground truth access violations on synthetic datasets [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Scanner labeled ground truth access violations 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Sample size needed to identify at least one violation. Sample size estimates assume 0 false positives (after adjudication of putative positives), and are calculated for various sample sizes and baseline prevalence rates. The most relevant term here is p0 which sets the desired error tolerance for marking a benchmark as passing. Based on this analysis, an acceptable violation rate of <5% would require >119… view at source ↗
Figure 16
Figure 16. Figure 16: Sample size needed to bound estimate of prevalence. Here the colors indicate various ‘ground truth’ violation rates and the y axis indicates the desired precision (either relative or absolute). Choosing a sample size from these plots means balancing the desired precision across various potential scenarios that may be encountered. This can be done in either direction, either assuming a desired precision an… view at source ↗
Figure 17
Figure 17. Figure 17: Relative precision for graders with different sensitivities. Sensitivity is compared for either optimistic (human grader or excellent scanner, left) or conservative (right) assumptions. Reduced sensitivity requires a much larger sample size to bound the true error rate with confidence [PITH_FULL_IMAGE:figures/full_fig_p031_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Estimates of CI halfwidth for various sample sizes. Required sample size depends on both the sensitivity of the scanner (Se) and the desired halfwidth. This is more conservative, but it generally points towards a labeled dataset containing ∼100 positives as a good target for getting a useful estimate of Se (with similar considerations for other metrics). When violations are common this is relatively easy,… view at source ↗
Figure 19
Figure 19. Figure 19: Lower bound on estimate of Se given known true Se and sample size (of positives). development. This makes it much more taxing to build a test dataset that contains a sufficient number of positives to obtain a robust estimate of the true sensitivity by sampling. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 14 linked inside Pith

  1. [4]

    MLE-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    Pith/arXiv arXiv

  2. [5]

    Seven simple steps for log analysis in AI systems.arXiv preprint arXiv:2604.09563,

    Magda Dubois, Ekin Zorer, Maia Hamin, Joe Skinner, Alexandra Souly, Jerome Wynne, Harry Coppock, Lu- cas Satos, Sayash Kapoor, Sunischal Dev, Keno Juchems, Kimberly Mai, Timo Flesch, Lennart Luettgau, Charles Teague, Eric Patey, JJ Allaire, Lorenzo Pacchiardi, Jose Hernandez-Orallo, and Cozmin Ududec. Seven simple steps for log analysis in AI systems.arXi...

    Pith/arXiv arXiv

  3. [6]

    Measuring AI agents’ progress on multi-step cyber attack scenarios.arXiv preprint arXiv:2603.11214,

    Linus Folkerts, Will Payne, Simon Inman, Philippos Giavridis, Joe Skinner, Sam Deverett, James Aung, Ekin Zorer, Michael Schmatz, Mahmoud Ghanem, John Wilkinson, Alan Steer, Vy Hong, and Jessica Wang. Measuring AI agents’ progress on multi-step cyber attack scenarios.arXiv preprint arXiv:2603.11214,

    arXiv

  4. [7]

    Sayash Kapoor

    URLhttps://meridianlabs-ai.github.io/inspect_scout/. Sayash Kapoor. Post on CORE-Bench benchmark issues. Post on X (formerly Twitter),https://x.com/ sayashk/status/1996334941832089732, 2025a. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj ...

    arXiv

  5. [9]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868,

    Pith/arXiv arXiv

  6. [10]

    Zhang, William Hu, Christopher Ré, and Azalia Mirho- seini

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirho- seini. KernelBench: Can LLMs write efficient GPU kernels?arXiv preprint arXiv:2502.10517,

    Pith/arXiv arXiv

  7. [11]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Pith/arXiv arXiv

  8. [12]

    PostTrainBench: CanLLMagentsautomateLLMpost-training?arXiv preprint arXiv:2603.08640,

    Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina Nguyen, Matthias Bethge, and MaksymAndriushchenko. PostTrainBench: CanLLMagentsautomateLLMpost-training?arXiv preprint arXiv:2603.08640,

    arXiv

  9. [13]

    Ayrton San Joaquin, Rokas Gipiškis, and Ze Shen Chin

    URLhttps://arxiv.org/abs/2311.12022. Ayrton San Joaquin, Rokas Gipiškis, and Ze Shen Chin. Scorecard of AI evaluation quality. Working draft, AI Standards Lab, February

    Pith/arXiv arXiv

  10. [14]

    Zachary S

    URLhttps://aistandardslab.org/wp-content/uploads/2026/02/ AISL-Scorecard-of-AI-Evaluation-Quality.pdf. Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. Transactions on Machine Learning Research,

    2026

  11. [15]

    URLhttps://arxiv.org/abs/ 2409.11363

    ISSN 2835-8856. URLhttps://arxiv.org/abs/ 2409.11363. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848,

    Pith/arXiv arXiv

  12. [16]

    Bench- Guard: Who guards the benchmarks? automated auditing of LLM agent benchmarks.arXiv preprint arXiv:2604.24955,

    Xinming Tu, Tianze Wang, Yingzhou Lu, Kexin Huang, Yuanhao Qu, and Sara Mostafavi. Bench- Guard: Who guards the benchmarks? automated auditing of LLM agent benchmarks.arXiv preprint arXiv:2604.24955,

    Pith/arXiv arXiv

  13. [17]

    AutomatedbenchmarkauditingforAIagentsandlargelanguagemodels.arXiv preprint arXiv:2605.26079,

    Junlin Wang, Federico Bianchi, Shang Zhu, Fan Nie, Yongchan Kwon, Bhuwan Dhingra, and James Zou. AutomatedbenchmarkauditingforAIagentsandlargelanguagemodels.arXiv preprint arXiv:2605.26079,

    Pith/arXiv arXiv

  14. [18]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.189. URLhttps://aclanthology.org/2025. acl-long.189/. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational L...

    doi:10.18653/v1/2025.acl-long.189 2025

  15. [20]

    Content-Type: application/json

    Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, CozminUdudec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Da...

    Pith/arXiv arXiv 2024

  16. [21]

    and GPQA (Rein et al., 2024)) for the answer format ambiguity and guessing scanners. 23 For GPQA, the guessing scanner flagged most samples as a 2 or higher, given that there were limited options for each multiple choice question to choose from, which made the tasks inherently guessable. The answer format ambiguity scanner did not flag violations for GPQA...

    2024

  17. [22]

    write to /app/out.html

    and SUPER (Bogin et al., 2024), with several variations to produce violation examples. This synthetic data did not require changes to the prompts. •API not available:a tool was stubbed to return a 500 error in LitQA2 •Tool times out:one of the tools was stubbed to return a server-side timeout in LitQA2 •Tool removed:The prompt calls for a tool but was not...

    2024

  18. [2019]

    URL https://arxiv.org/abs/1905.07830

    Association for Computational Linguistics. URL https://arxiv.org/abs/1905.07830. Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D. Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. MLRC-Bench: Can language agents solve machine learning research challenges?arXiv preprint arXiv:2504.09702,

    Pith/arXiv arXiv 1905

  19. [2022]

    URLhttps://arxiv.org/abs/2109.07958

    Association for Computational Linguistics. URLhttps://arxiv.org/abs/2109.07958. Kevin Meng, Vincent Huang, Jacob Steinhardt, and Sarah Schwettmann. Introducing Docent.https: //transluce.org/introducing-docent, March

    Pith/arXiv arXiv

  20. [2024]

    Kao, Evangelia Spiliopoulou, and Adina Williams

    Florian Bordes, Candace Ross, Justine T. Kao, Evangelia Spiliopoulou, and Adina Williams. Eval factsheets: A structured framework for documenting AI evaluations.arXiv preprint arXiv:2512.04062,

    arXiv

  21. [2025]

    SUPER: Evaluating agents on setting up and executing tasks from research repositories

    Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. arXiv preprint arXiv:2409.07440,

    Pith/arXiv arXiv

  22. [2026]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2-Bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982,

    Pith/arXiv arXiv