Pith. sign in

REVIEW 3 major objections 7 minor 77 references

The paper introduces a unified corpus of 957,253 agent-evaluation records and shows that the choice of aggregation rule—all-pass versus partial credit—can change both measured capability and agent rankings.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:10 UTC pith:J7EY7EQ6

load-bearing objection Useful corpus with a real caveat: the HarveyAI-Lab counterfactual numbers rest on an unvalidated local verifier, so treat them as illustrative until the authors verify against official scoring and actually release the data. the 3 major comments →

arxiv 2607.25891 v1 pith:J7EY7EQ6 submitted 2026-07-28 cs.AI cs.DB

Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

Stefan Krsteski , Charlotte Meyer , Guillaume Allegre , Tony O'Halloran , Alexandre Sallinen This is my paper
classification cs.AI cs.DB
keywords agent evaluationbenchmark corpusverifier-level recordsaggregation rulesitem response theorycapability scalingcounterfactual rescoringtask difficulty prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper builds Messier, a corpus of 957,253 trial records spanning 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers, with each record standardized by model, scaffold, environment, task, verifier, and aggregation rule. Using verifier-level granularity, it argues that a benchmark's aggregation rule is not a neutral reporting choice: on a long-horizon legal-workflow benchmark, the official all-pass rule reports 0.2% task success even though the same trials satisfy 28.8% of criteria on average, and the two scoring methods reorder agents (Spearman ρ = 0.667). The harmonized records also support fitting a 1PL item-response model whose ability rankings match an established cross-benchmark capability index at ρ = 0.81 across 75 matched agents without any new evaluation campaign. A sympathetic reader would care because the corpus makes existing evaluations reusable, auditable, and re-scorable at the criterion level, and because the aggregation-sensitivity results temper how single-number leaderboards should be read.

Core claim

Messier's central claim is that per-criterion, per-verifier records—not just task-level pass/fail—are the right unit for comparing AI agents across benchmarks. From those records, the authors demonstrate counterfactual rescoring: rescoring the same trials under soft (mean fraction of criteria passed) or majority-pass rules instead of the official all-pass rule raises measured success substantially and, on some benchmarks, changes agent orderings. They then fit a 1PL Rasch model to the consolidated agent-task response matrix and show that the resulting ability estimates align with the published ranking of an external capability index at Spearman ρ = 0.81 (95% CI [0.68, 0.89]) for general capa

What carries the argument

The corpus itself is the central object: a reconciled schema in which every trial is decomposed as (model, scaffold, environment, task, verifier, aggregation rule), with 74,205 verifiers and per-criterion outcomes for multi-verifier tasks. Two mechanisms carry the argument. (1) Counterfactual rescoring: because records preserve each verifier's binary outcome, any aggregation rule—all-pass, majority-pass, or soft mean—can be applied post hoc to the same trials, revealing how much of measured capability is an artifact of the rule. (2) A 1PL Rasch (item-response) model fitted to the consolidated agent-task binary matrix, from which per-agent ability θ and per-task difficulty β are estimated on

Load-bearing premise

The load-bearing premise is that the per-criterion verifier outcomes for the newly contributed benchmarks, especially the legal-workflow benchmark whose rubric was rebuilt locally as an LLM judge, faithfully reproduce the upstream benchmark's own grading; the paper reports no validation of this local verifier against official scores.

What would settle it

Take a multi-verifier benchmark that publicly releases both official task-level scores and individual criterion outcomes, and check whether the paper's rescoring pipeline exactly reproduces the official all-pass scores from the criterion outcomes; a mismatch would indicate the pipeline (or its local judge) diverges from the upstream grader. Concretely, for the six contributed benchmarks, run the upstream benchmarks' original verifiers on the same recorded traces and compare per-criterion outcomes with the local LLM judge's judgments—if they disagree on a large fraction of criteria, the headlin

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If evaluations release criterion-level outcomes, any aggregation rule can be computed post hoc, making leaderboards re-auditable without rerunning agents.
  • Strict all-pass scoring should be read as a lower bound on progress: on multi-verifier tasks it can hide substantial partial completion, as the 0.2% vs 28.8% gap illustrates.
  • Open, harmonized records can reproduce an established capability ordering (ρ = 0.81) without a new evaluation campaign, lowering the cost of capability scaling.
  • Capability scales can be specialized by occupation, action space, or verifier type, enabling domain-specific rather than global comparisons.
  • Frontier progress is uneven: function calling is near saturation, programming shows the largest gains, and enterprise workflows remain the hardest frontier.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if per-verifier records become a release standard, audits for reward hacking, sandbagging, and evaluation awareness could run on archived trials at the exact granularity where those failure modes occur, without new compute.
  • Editorial extension: the observed rank reordering under different aggregation rules implies that any single-number leaderboard is partly a choice of scoring philosophy; competitive pressure on all-pass leaderboards may push agents to satisfy every criterion rather than to maximize partial progress, a behavior that may not match deployment value.
  • Editorial extension: the modest within-benchmark difficulty-prediction gains suggest task text encodes hardness beyond benchmark identity; a testable next step is using such estimates to build stratified test sets with calibrated difficulty for future benchmarks.
  • Editorial extension: a direct check of the verifier-transfer assumption—comparing the locally rebuilt rubric judge's per-criterion outcomes against the upstream benchmark's official scores on a shared subset—would quantify how much of the aggregation-effect magnitudes depend on the local judge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. Messier is proposed as a unified corpus of 957,253 standardized trial outcomes spanning 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers, assembled from public releases plus new five-agent runs on six benchmarks (HarveyAI-Lab, MedAgentBench, DABStep, QCircuitBench, ScienceAgentBench, ReplicationBench) under a fixed OpenHands/Harbor configuration. On this corpus the paper reports three analyses: (i) frontier progress, finding function-calling near saturation, programming fastest-improving, and enterprise workflows most challenging (Figure 1f); (ii) counterfactual rescoring, showing that all-pass aggregation can obscure partial progress and reorder agents (Table 3); and (iii) 1PL IRT capability scales correlating with Epoch ECI at Spearman ρ=0.81 over 75 matched agents (Figure 2), plus a difficulty-prediction experiment (Table 4).

Significance. With the caveat below, the corpus would be a valuable community asset: roughly eight times HAL in task count, verifier-level rather than task-level, with careful per-source license/access tables, deterministic hygiene checks, and modular per-benchmark builders. The ECI alignment is a genuine external anchor: ρ=0.81 over 75 matched agents is meaningful, although it partly validates the reconciliation pipeline given shared source benchmarks. The qualitative aggregation result is also supported by tau2-bench and TheAgentCompany, whose per-criterion outcomes come from public releases. The paper is honest about the difficulty-prediction baseline and includes a thoughtful Limitations section. The central risk in this manuscript, fidelity of the locally constructed HarveyAI-Lab verifier, does land on reading §A.5 and is detailed in Major Comment 1.

major comments (3)
  1. [§5.2 / §A.5 / Table 3] The HarveyAI-Lab row rests on a verifier constructed by the authors, not on the upstream benchmark grading. For each of 1,000 tasks, anthropic/claude-sonnet-4-6 issues 33-65 binary criterion judgments (§A.5), so Table 3 all-pass 0.002 vs soft 0.288, near-pass 0.080, and per-agent ρ=0.667 are internal properties of that judge. No validation against official Harvey scores, human labels, or a second judge is reported. The Limitations sentence that the authors largely assume the validity of underlying datasets does not cover a newly constructed judge. Because Harvey supplies 1,000 of 2,572 enterprise tasks and 221k of 957k outcomes, the enterprise-frontier point 0.57 (Figure 1f) inherits this bias. Please validate the judge on a sample against official scoring and re-derive these figures, or explicitly label the Harvey counterfactual as illustrative.
  2. [§5.2, Table 3] The claim that aggregation artificially alters agent rankings is only weakly supported by the HarveyAI-Lab row: ρ=0.667 is computed across n=5 agents (Table 2), each with one trial per task (§A.5), with no confidence interval or significance test, and most all-pass means cluster near zero, making the estimate tie-dominated. Moreover, with roughly 60 criteria per task and mean criterion pass 0.288, an all-pass rate near zero is close to a mathematical consequence (0.288^60 ≈ 0), so the empirical content of that row is the per-criterion rate of the unvalidated judge (Major Comment 1). The stronger reordering evidence is TheAgentCompany (n=18, ρ=0.981) and tau2-bench (n=7, ρ=0.857). The paper calls for broader validation in §5.2; the abstract should reflect that caution.
  3. [§5.1, Figure 1f] The per-task frontier is the maximum score over eligible agents, so its expected value grows with the number of eligible agents, which differs by an order of magnitude across the groups compared: function-calling has 108 agents, programming up to 192, while enterprise benchmarks have 5-18 (Table 2). The headline that enterprise workflows remain the most challenging at 0.57 is therefore confounded with agent coverage and, for the 1,000 Harvey tasks, with the unvalidated verifier of Major Comment 1. The §5.1 disclaimer that the analysis is not intended to estimate field-wide progress is appropriate but does not protect the abstract statement. Please add a robustness check on a common agent subset or an explicit coverage adjustment, and show per-benchmark eligible-agent counts alongside the curves.
minor comments (7)
  1. [§A.1 / Table 2] BFCL-Multi-Turn is described as matching the agent per-turn method-call trace turn by turn (§A.1) but Table 2 lists one verifier per task (800 verifiers / 800 tasks). Clarify whether trace matching is a composite single verifier; otherwise it may qualify as a multi-verifier all-pass benchmark and should be considered for §5.2.
  2. [Figure 1f] The dumbbell markers and +Δ annotations are not defined in the caption; state which endpoint is the first versus final observed quarter, and what the delta values are.
  3. [Table 4] The z-scored-β baseline value of -0.081 is confusing; state that the baseline predicts the training-fold benchmark mean and can be negative when the target is within-benchmark z-scored difficulty.
  4. [§A.5] Each (agent, task) cell is run once. Add a sentence acknowledging that Table 3 and Figure 1f estimates carry no sampling error and would benefit from repeated trials or interval estimates.
  5. [§A.3] SOC/NAICS classification reports only 2-of-3 voter agreement (88.3%/86.8%); consider validating the labels against a small human-annotated sample, since agreement among the same-family voters is not accuracy.
  6. [Limitations] Extend the validity-assumption sentence to cover newly introduced verifiers (in particular the Harvey judge), and note the single-trial design, so readers can calibrate trust in the contributed runs.
  7. [§5.3] Because many Messier benchmarks overlap Epoch's source pool, ρ=0.81 partly verifies the reconciliation pipeline rather than a fully independent scale; mention this explicitly and, if feasible, add a robustness check excluding shared benchmarks.

Circularity Check

0 steps flagged

No significant circularity: the ECI alignment uses an external benchmark, the aggregation and difficulty analyses are empirical or cross-validated, and no load-bearing self-citation appears.

full rationale

I walked the derivation chain in each analytical section. Section 5.1 is explicitly descriptive and disclaims causal progress claims. Section 5.2 rescoring is a direct empirical comparison of formally defined aggregation rules (Section 3): the direction that all-pass ≤ soft is a mathematical fact about the definitions, but the paper does not disguise this as a fitted prediction; the empirical content is the Table 3 magnitudes, which are computed from verifier-level records. Section 5.3 compares a 1PL fit to Epoch's ECI, an external published benchmark ranking; there is no author overlap, and the paper openly states that it follows Epoch's naming and model-level aggregation protocol (§4, §B.2). This is a replication/validation exercise, not a derivation from the target itself. Section 5.4 uses cross-validated regression with a benchmark-mean baseline, so no leakage equates the predictor to the label. Appendix B.1 explicitly labels its BRIDGE/METR agreement as an 'implementation check confirming pipeline consistency on overlapping data,' which is transparent rather than circular. The Limitations section and Appendix A.5 flag that upstream dataset validity is assumed and that the HarveyAI-Lab verifier is a locally built LLM judge without reported validation against official Harvey scores. These are external-validity concerns, not circularity: they affect whether the numbers are trustworthy, not whether the reasoning reduces to its inputs. No load-bearing self-citations or uniqueness claims imported from the authors' prior work appear in the paper.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The paper is largely a data-asset paper. Its claims rest on the faithfulness of the normalization, the choice of best-scaffold per model, the IRT fitting setup, and the local verifier for HarveyAI-Lab. The headline counterfactual figures are constructed from the paper's own per-criterion records, so they are internally consistent but can be sensitive to how those records were built. The ECI alignment is the main external anchor, but it depends on which models appear in both corpora and how scaffolds are selected.

free parameters (3)
  • IRT priors and optimization hyperparameters = mean 0, std 1; ClippedAdam lr 0.01; 1000-2000 epochs; seed 42
    Choice of priors and optimizer settings in B.2. Reasonable defaults but without sensitivity analysis; the claim of ρ=0.81 could be sensitive to them.
  • Best-scaffold aggregation protocol = highest-scoring scaffold per (model,benchmark)
    §5.3: 'select the highest-scoring scaffold configuration' — a modeling choice that can only inflate θ estimates for models with more scaffolds. Given the goal of matching ECI, it is plausible.
  • Domain slicing via SOC = SOC 15-1200 for programming, SOC 15-2000 for math
    §5.3: domain comparisons are defined by SOC codes assigned by a three-voter LLM ensemble with 88.3%/86.8% agreement. Classification errors weaken the comparison, and the author's own agreement rates are only ~87%.
axioms (5)
  • domain assumption The 30 selected benchmarks are a representative sample of interactive agent evaluation, and their releases are mutually comparable after normalization.
    Entire §4. The corpus is a convenience sample of public releases plus six new runs; cross-benchmark comparability depends on the normalization mapping being faithful to each benchmark's original semantics.
  • standard math Rasch (1PL) item-response theory adequately describes trial success, P(Y=1)=σ(θ_a−β_t), for heterogeneous tasks and agents.
    Eq. 1. IRT is a standard tool, but for a corpus with 5-90 tasks on some benchmarks and dominated by a few benchmarks (HarveyAI-Lab 221k of 960k outcomes), the fit quality is not established beyond a single 2PL comparison.
  • domain assumption Binary per-verifier outcomes are sufficient to encode task success; the SOC/NAICS classifier ensemble is trustworthy enough for domain slicing.
    A.3: 88.3% SOC majority agreement and 86.8% NAICS; the remaining ~12% are adjudicated. That does not validate the labels against any gold standard.
  • domain assumption The response matrix can be formed by taking each model's best scaffold per benchmark, and constant-outcome tasks can be dropped.
    B.2 drops items with constant outcomes; this is standard for IRT but changes the target of θ and could bias comparisons if the corpus has missingness correlated with scaffold choices.
  • ad hoc to paper The anonymized repository will actually release the code and data at the stated URL.
    Footnote 1 cites anon.4open.science/messier-d3, not a dated public release. Reproducibility depends on this external resource.
invented entities (2)
  • Capability scales from MESSIER θ and β independent evidence
    purpose: Provide 'open psychometric calibration' for agent capability, specialized by domain/occupation/action space.
    The θ and β are fitted from the corpus; their correlation with Epoch ECI is a falsifiable external check (ρ=0.81). The scales are not separate entities beyond the corpus.
  • HarveyAI-Lab's local rubric verifier no independent evidence
    purpose: Acts as the official Harvey benchmark grading for the whole corpus.
    §A.5 describes a local LLM judge with 33-65 criteria per task, triggering the headline all-pass vs soft result. No comparison against Harvey's official scores is given, so this entity is internally defined rather than externally validated.

pith-pipeline@v1.3.0-alltime-deepseek · 18392 in / 11100 out tokens · 83889 ms · 2026-08-01T01:10:09.169380+00:00 · methodology

0 comments
read the original abstract

Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.

Figures

Figures reproduced from arXiv: 2607.25891 by Alexandre Sallinen, Charlotte Meyer, Guillaume Allegre, Stefan Krsteski, Tony O'Halloran.

Figure 1
Figure 1. Figure 1: Composition and frontier analysis. Panels summarize the corpus across the dimensions defined in §3: (a) tasks by benchmark group, (b) action space, (c) environment state, (d) aggregation rule, and (e) verifier type. Counts in (b)–(e) are on a log scale. Panel (f) tracks frontier pass rate by release quarter and benchmark group, with dumbbell markers connecting each group’s first and final observed quarter … view at source ↗
Figure 2
Figure 2. Figure 2: Capability orderings from MESSIER vs. Epoch’s ECI. Each panel plots MESSIER-fit per-agent θ (logit scale) against Epoch’s published ECI for the matched subset, with Spearman ρ, 95% bootstrap CI over 1000 resamples, pairwise concordance, and matched-agent count. The General panel uses Epoch’s overall ECI (n = 75); Coding uses SWE-ECI (n = 59); Math uses Math-ECI (n = 63). A 1PL fit on MESSIER alone aligns w… view at source ↗
Figure 3
Figure 3. Figure 3: Entity-relationship view of MESSIER. An agent is the composition of a model and a scaffold. A task specifies one or more verifiers v1, . . . , vN . A trial is one execution of an agent on a task; each verifier produces an outcome vi(τ ) ∈ {0, 1}, and the aggregation rule f combines these into the trial score S. (uniformly passed or uniformly failed) are dropped before fitting, since they carry no informati… view at source ↗
Figure 4
Figure 4. Figure 4: Distributions of fitted IRT 1PL agent ability [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 21 linked inside Pith

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

    1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    Florian Brand and Jean-Stanislas Denain , year=

  9. [9]

    Artificial intelligence , volume=

    Planning and acting in partially observable stochastic domains , author=. Artificial intelligence , volume=. 1998 , publisher=

    1998

  10. [10]

    Proceedings of the 26th annual international conference on machine learning , pages=

    Curriculum learning , author=. Proceedings of the 26th annual international conference on machine learning , pages=

  11. [11]

    European review , volume=

    ‘Improving ratings’: audit in the British University system , author=. European review , volume=. 1997 , publisher=

    1997

  12. [12]

    arXiv preprint arXiv:2211.09110 , year=

    Holistic evaluation of language models , author=. arXiv preprint arXiv:2211.09110 , year=

    Pith/arXiv arXiv

  13. [13]

    Transactions on machine learning research , year=

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year=

  14. [14]

    International Conference on Learning Representations , volume=

    Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume=

  15. [15]

    International Conference on Learning Representations , volume=

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Theagentcompany: benchmarking llm agents on consequential real world tasks , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Webvoyager: Building an end-to-end web agent with large multimodal models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  19. [19]

    International Conference on Learning Representations , volume=

    Cybench: A framework for evaluating cybersecurity capabilities and risks of language models , author=. International Conference on Learning Representations , volume=

  20. [20]

    Barres, Victor and Dong, Honghua and Ray, Soham and Si, Xujie and Narasimhan, Karthik , journal=

  21. [21]

    Forty-second International Conference on Machine Learning , year=

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models , author=. Forty-second International Conference on Machine Learning , year=

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Measuring AI ability to complete long software tasks , author=. Advances in Neural Information Processing Systems , volume=

  23. [23]

    arXiv preprint arXiv:2512.00193 , year=

    A Rosetta Stone for AI Benchmarks , author=. arXiv preprint arXiv:2512.00193 , year=

    arXiv

  24. [24]

    arXiv preprint arXiv:2510.04374 , year=

    Gdpval: Evaluating ai model performance on real-world economically valuable tasks , author=. arXiv preprint arXiv:2510.04374 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2510.21652 , year=

    Astabench: Rigorous benchmarking of ai agents with a scientific research suite , author=. arXiv preprint arXiv:2510.21652 , year=

    Pith/arXiv arXiv

  26. [26]

    Nejm Ai , volume=

    MedAgentBench: a virtual EHR environment to benchmark medical LLM agents , author=. Nejm Ai , volume=. 2025 , publisher=

    2025

  27. [27]

    arXiv preprint arXiv:2510.01179 , year=

    Toucan: Synthesizing 1.5 m tool-agentic data from real-world mcp environments , author=. arXiv preprint arXiv:2510.01179 , year=

    arXiv

  28. [28]

    arXiv preprint arXiv:2510.11977 , year=

    Holistic agent leaderboard: The missing infrastructure for ai agent evaluation , author=. arXiv preprint arXiv:2510.11977 , year=

    arXiv

  29. [29]

    arXiv preprint arXiv:2504.08942 , year=

    Agentrewardbench: Evaluating automatic evaluations of web agent trajectories , author=. arXiv preprint arXiv:2504.08942 , year=

    arXiv

  30. [30]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  31. [31]

    arXiv preprint arXiv:2602.07267 , year=

    BRIDGE: Predicting Human Task Completion Time From Model Performance , author=. arXiv preprint arXiv:2602.07267 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2604.00594 , year=

    Agent psychometrics: Task-level performance prediction in agentic coding benchmarks , author=. arXiv preprint arXiv:2604.00594 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2504.12516 , year=

    Browsecomp: A simple yet challenging benchmark for browsing agents , author=. arXiv preprint arXiv:2504.12516 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2504.01382 , year=

    An illusion of progress? assessing the current state of web agents , author=. arXiv preprint arXiv:2504.01382 , year=

    arXiv

  35. [35]

    arXiv preprint arXiv:2601.11868 , year=

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year=

    Pith/arXiv arXiv

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    Gso: Challenging software optimization tasks for evaluating swe-agents , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    Science , volume=

    Rethink reporting of evaluation results in AI , author=. Science , volume=. 2023 , publisher=

    2023

  38. [38]

    arXiv preprint arXiv:2508.20453 , year=

    Mcp-bench: Benchmarking tool-using llm agents with complex real-world tasks via mcp servers , author=. arXiv preprint arXiv:2508.20453 , year=

    Pith/arXiv arXiv

  39. [39]

    2026 , month =

    Niko Grupen and Gabe Pereyra and Julio Pereyra , title =. 2026 , month =

    2026

  40. [40]

    International Conference on Learning Representations , volume=

    Ai sandbagging: Language models can strategically underperform on evaluations , author=. International Conference on Learning Representations , volume=

  41. [41]

    arXiv preprint arXiv:2505.23836 , year=

    Large language models often know when they are being evaluated , author=. arXiv preprint arXiv:2505.23836 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2510.20487 , year=

    Steering Evaluation-Aware Language Models to Act Like They Are Deployed , author=. arXiv preprint arXiv:2510.20487 , year=

    arXiv

  43. [43]

    arXiv preprint arXiv:2511.21654 , year=

    EvilGenie: A Reward Hacking Benchmark , author=. arXiv preprint arXiv:2511.21654 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2603.29357 , year=

    BenchScope: How Many Independent Signals Does Your Benchmark Provide? , author=. arXiv preprint arXiv:2603.29357 , year=

    arXiv

  45. [45]

    arXiv preprint arXiv:2501.12851 , year=

    Acebench: Who wins the match point in tool usage? , author=. arXiv preprint arXiv:2501.12851 , year=

    arXiv

  46. [46]

    arXiv preprint arXiv:2602.12670 , year=

    SkillsBench: Benchmarking how well agent skills work across diverse tasks , author=. arXiv preprint arXiv:2602.12670 , year=

    Pith/arXiv arXiv

  47. [47]

    arXiv preprint arXiv:2602.11898 , year=

    Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences , author=. arXiv preprint arXiv:2602.11898 , year=

    arXiv

  48. [48]

    arXiv preprint arXiv:2603.00039 , year=

    CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation , author=. arXiv preprint arXiv:2603.00039 , year=

    arXiv

  49. [49]

    arXiv preprint arXiv:2605.02964 , year=

    Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use , author=. arXiv preprint arXiv:2605.02964 , year=

    Pith/arXiv arXiv

  50. [50]

    2025 , url =

    Johannes Gasteiger and Akbir Khan and Sam Bowman and Misha Wagner and Vladimir Mikulik and Ethan Perez and Fabien Roger , title =. 2025 , url =

    2025

  51. [51]

    arXiv preprint arXiv:2508.00828 , year=

    Finance agent benchmark: Benchmarking llms on real-world financial research tasks , author=. arXiv preprint arXiv:2508.00828 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2603.11337 , year=

    RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents , author=. arXiv preprint arXiv:2603.11337 , year=

    arXiv

  53. [53]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Energy and policy considerations for deep learning in NLP , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  54. [54]

    Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval , pages=

    C-pack: Packed resources for general chinese embeddings , author=. Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval , pages=

  55. [55]

    arXiv preprint arXiv:2602.18998 , year=

    Benchmark test-time scaling of general LLM agents , author=. arXiv preprint arXiv:2602.18998 , year=

    arXiv

  56. [56]

    Georg Rasch , title =

  57. [57]

    Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H. Tran and Fuqiang Li and Ren Ma and Mingzhang Zheng and Bill Qian and Yanjun Shao and Niklas Muennighoff and Yizhe Zhang and Binyuan Hui and Junyang Lin and Robert Brennan and Hao Peng and H...

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2509.16941 , year=

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? , author=. arXiv preprint arXiv:2509.16941 , year=

    Pith/arXiv arXiv

  59. [59]

    Advances in Neural Information Processing Systems , volume=

    QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design , author=. Advances in Neural Information Processing Systems , volume=

  60. [60]

    arXiv preprint arXiv:2411.15114 , year=

    Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts , author=. arXiv preprint arXiv:2411.15114 , year=

    Pith/arXiv arXiv

  61. [61]

    International Conference on Learning Representations , volume=

    Mle-bench: Evaluating machine learning agents on machine learning engineering , author=. International Conference on Learning Representations , volume=

  62. [62]

    International Conference on Learning Representations , volume=

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery , author=. International Conference on Learning Representations , volume=

  63. [63]

    arXiv preprint arXiv:2510.24591 , year=

    ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? , author=. arXiv preprint arXiv:2510.24591 , year=

    arXiv

  64. [64]

    arXiv preprint arXiv:2501.14249 , year=

    Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year=

    Pith/arXiv arXiv

  65. [65]

    Advances in Neural Information Processing Systems , volume=

    Matharena: Evaluating llms on uncontaminated math competitions , author=. Advances in Neural Information Processing Systems , volume=

  66. [66]

    arXiv preprint arXiv:2410.04698 , year=

    Mathhay: An automated benchmark for long-context mathematical reasoning in llms , author=. arXiv preprint arXiv:2410.04698 , year=

    Pith/arXiv arXiv

  67. [67]

    arXiv preprint arXiv:2506.23719 , year=

    Dabstep: Data agent benchmark for multi-step reasoning , author=. arXiv preprint arXiv:2506.23719 , year=

    Pith/arXiv arXiv

  68. [68]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  69. [69]

    International Conference on Learning Representations , volume=

    Gaia: a benchmark for general ai assistants , author=. International Conference on Learning Representations , volume=

  70. [70]

    Advances in neural information processing systems , volume=

    Agentboard: An analytical evaluation board of multi-turn llm agents , author=. Advances in neural information processing systems , volume=

  71. [71]

    Advances in neural information processing systems , volume=

    Are emergent abilities of large language models a mirage? , author=. Advances in neural information processing systems , volume=

  72. [72]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Show your work with confidence: Confidence bands for tuning curves , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  73. [73]

    arXiv preprint arXiv:2406.10229 , year=

    Quantifying variance in evaluation benchmarks , author=. arXiv preprint arXiv:2406.10229 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2512.03438 , year=

    Multimodal Reinforcement Learning with Agentic Verifier for AI Agents , author=. arXiv preprint arXiv:2512.03438 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2602.03130 , year=

    FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation , author=. arXiv preprint arXiv:2602.03130 , year=

    arXiv

  76. [76]

    2018 , organization =

    2018 Standard Occupational Classification Manual , author =. 2018 , organization =

    2018

  77. [77]

    2022 , organization =

    North American Industry Classification System: United States, 2022 , author =. 2022 , organization =

    2022