Pith.

Artificial Intelligence · arXiv:2610.22350 v1 · submitted 15 Oct 2026 · 40 references

FDE-Gym: The Last Mile of AI Delivery

Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao

show 18 more authors

Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang

The paper says

“Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints.”

more of the abstract

We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A genuinely careful benchmark with credible headline numbers, but the abstract's 'constraint discovery is the primary bottleneck' claim is weaker than the data support because L0 also hands the agent the scoring criteria and the business metric. This line is wrong

1 major objection · 3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 40 references say.

The paper says, at §5.2 / Table 1 / Eq. (1)

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

The L0-to-L1 comparison does not isolate constraint discovery. Table 1 states that the L0 instruction provides the underlying problem, applicable rules and constraints, and "how the result is scored," while L1 provides only a clear statement of the problem. Since Eq.

more of objection 1

(1) defines the score as normalized improvement on the customer's KPI, removing the scoring information at the same step as the constraints means the reported 0.28 drop mixes failure to discover unstated rules with uncertainty about which business metric will be measured. Telling an agent which KPI is evaluated changes the objective it can optimize, so the abstract's claim that "discovering unstated constraints is the primary bottleneck" overstates what the comparison can establish. The manuscript does not acknowledge this confound in §5.2 or in Section 7. I recommend either adding a control condition that supplies the business metric and scoring criteria without the constraints, or revising the claim to refer to withheld task-relevant information generally, with constraint discovery presented as one component rather than the cleanly identified primary bottleneck.

3 minor objections

The paper says, at §2.3, Eqs. (4)–(5)

Pith says: minor objection 1 of 3

The average FDE-Gym score in Eq. (5) uses the same symbol S as the single-run score in Eq. (4); using an overbar or a different symbol for the average would prevent ambiguity.

The paper says, at §5.2

Pith says: minor objection 2 of 3

The paired 95% interval of 0.17 to 0.39 for the L0-to-L1 drop is reported without describing the resampling procedure; please specify whether the bootstrap resamples tasks under the paired design and report the corresponding per-model intervals.

show 1 more

The paper says, at Figure 2b caption

Pith says: minor objection 3 of 3

The caption states that L0-L2 are evaluated on the 20 original tasks and L3 on all 95 tasks, but the curve in Figure 2b appears to show the level progression; please clarify whether the final point is the original-task L3 mean or the full 95-task mean, since Table 3 reports both separately.

What would settle it

On the same 20 original tasks used for L0-L2, add an L1+ condition whose task instruction contains the L1 underlying-problem statement plus the L0 scoring sentence ('how the result is scored': the customer KPI and the normalization in Eq. 1), while still omitting the applicable rules and constraints. Run the three leading configurations three times as in Section 4 and compare the paired L0-to-L1 drop with the L0-to-L1+ drop. If the L0-to-L1+ drop is much smaller than 0.28 (for example below about 0.10), the claimed constraint-discovery penalty is substantially a scoring-criterion effect; if the drop remains near 0.28, the concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 48 s · 17,986 tokens read, 5,332 written
  3. Editor wrote the verdict above 33 s · 19,599 tokens read, 1,161 written
  4. Referee major revision: 1 major objection, 3 minor 1 min 29 s · 18,350 tokens read, 10,270 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Worth a serious look. FDE-Gym is one of the more carefully constructed agent benchmarks I have seen in a while, and the headline results are probably about right. The combination is new: 95 tasks from 20 real delivery projects, scored on the customer's own business metric with the do-nothing baseline at 0 and the reference solution at 1, literal compliance capped at 0.2, scripted stakeholders instead of an LLM judge, three runs per configuration, bootstrapped confidence intervals, and admission anchors tested against 270 shortcut strategies. That is real methodology work, and the paper earns credit for it. The central empirical findings — best configuration at 0.56, resolving 14 of 95 tasks; weak models below zero on most tasks; stakeholders' facts recovered but not converted into uplift — are credible and useful on their own. The soft spot the stress-test note identifies is real, and the paper does not acknowledge it. Finding 2 attributes the L0-to-L1 drop of about 0.28 to withheld constraints, but Table 1 shows L0 also provides the scoring criteria and the business metric. Telling an agent what KPI will be measured changes the objective it optimizes; withholding it forces inference. The drop therefore mixes constraint discovery with objective-function uncertainty, so the abstract's 'discovering unstated constraints is the primary bottleneck' is not cleanly established. Notice the paper's own conclusion shifts emphasis to 'translating those facts into an effective system' and stakeholder contact cost — which is consistent with Finding 3 and with the L1-to-L3 flatness, but it is a different claim from the abstract. The fix is easy: give the scoring criteria at L1, or reframe the claim. Two smaller issues. First, 75 of 95 tasks are held out, so the full results cannot be independently reproduced; the authors disclose this, but it still limits verification of the ranking. Second, Section 5.7's 'one in seven engagements' line leans hard on comparing success rates with normalized uplift; they do disclaim the mismatch, so I would call it minor. Who is this for: anyone building agent benchmarks, anyone evaluating agents for real deployments, and procurement-minded readers. The benchmark deserves a serious referee, and the conditional verdict is the right stance. I would send it to review and ask the authors to address the L1 confound before it is cited as evidence about constraint discovery.

The strongest objection

The most load-bearing comparison is Finding 2 in Section 5.2: the L0-to-L1 drop of 0.28 is attributed to 'withholding the constraints,' and this is the evidence for the abstract's claim that discovering unstated constraints is the primary bottleneck. But Table 1 states that L0 includes 'the underlying problem, applicable rules and constraints, and how the result is scored,' while L1 includes only a clear statement of the underlying problem. The scoring criteria and the customer's business metric are therefore removed at the same step as the constraints. This matters because the FDE-Gym score is defined as improvement on the customer's KPI (Eq. 1); telling an agent which KPI will be measured changes the objective it can optimize, while withholding it forces the agent to infer what is valued from documents and stakeholders. The observed 0.28 drop thus mixes constraint discovery with objective-function uncertainty. The paper does not acknowledge this confound in Section 5.2 or in the Limitations section. The empirical headline result of 0.56 and 14/95 resolved is unaffected, but the causal claim about the primary bottleneck is not cleanly established by the reported comparison.

The reader's weakest assumption already identifies this confound, and I agree with it. The benchmark construction is otherwise unusually careful: admission anchors, three runs per configuration, scripted stakeholders, and cost reporting all give independent support to the quantitative measurement of 0.56 mean score and 14/95 resolved tasks. Those headline numbers do not depend on the L0-to-L1 attribution. What is load-bearing is the separate causal claim, repeated in the abstract and Section 5.2, that discovering unstated constraints is the primary bottleneck. That claim rests entirely on comparing L0 with L1, and the two conditions differ in the scoring information as well as in the constraints. If the proposed L1+ ablation shows a much smaller drop, the central bottleneck statement would need to be revised to distinguish 'not knowing the business metric' from 'not knowing the constraints'; if the drop is unchanged, the concern is empirically resolved. Because the paper is already CONDITIONAL and this is the same concern, no verdict change is needed; the condition should explicitly require the L1+ ablation or an equivalent isolation of scoring-criterion effects before the bottleneck claim is accepted.

The referee's report

If the empirical results are accepted, FDE-Gym is a substantial contribution to agent evaluation. Its strengths are real: scoring on the customer's own business metric rather than binary task completion, predefined admission anchors that cap literal compliance and shortcut strategies, held-out grading, three runs per configuration, key-fact tracking, a large set of cross-sector scenarios, and an open-source harness and experiment runner. The headline quantitative result—best configuration 0.56 mean score and 14/95 resolved—is a striking and falsifiable measurement, as is the finding that leading models obtain most stakeholder facts yet still fall well short of reference uplift. The main load-bearing weakness is that the L0-to-L1 comparison, which carries the abstract's causal claim about constraint discovery, also removes the scoring criteria and business metric, so the paper's strongest interpretive claim is not cleanly identified by the reported experiment.

Pith's reading of the paper (a machine's summary, not the authors')

FDE-Gym is a benchmark that grades an AI agent the way a customer would: not by whether a system was delivered, but by how much the customer's own business metric improved, on a scale where doing nothing scores 0 and an expert reference solution scores 1. The paper's central finding is that current frontier agents fall far short on this scale: the best of 13 configurations averages 0.56 and resolves only 14 of 95 realistic delivery tasks, even though it gathers 72 to 85 percent of the key facts stakeholders hold. The largest measured performance drop occurs when agents must discover unstated constraints on their own, while a misdirected sponsor request costs them almost nothing beyond that. The paper argues this reveals a gap between solving well-specified benchmark tasks and delivering real customer value, with weak models frequently leaving the customer worse off than no delivery at all.

What carries the argument. The mechanism carrying the argument is the FDE-Gym score, $S = U_{\mathrm{adj}} - C$, defined as incident-adjusted normalized business uplift minus stakeholder contact cost, where uplift is $U = (\mathrm{KPI}_{\mathrm{delivered}} - \mathrm{KPI}_{\mathrm{baseline}})/(\mathrm{KPI}_{\mathrm{reference}} - \mathrm{KPI}_{\mathrm{baseline}})$ evaluated on held-out production data. The four task instruction levels (L0 to L3) vary only the instruction, from a fully specified problem to the sponsor's original request, while the environment, scripted stakeholders, and grader stay fixed, so score differences across levels are meant to isolate the effect of withheld information. Admission anchors guarantee the scale is meaningful: the reference solution must score at least 0.95, the do-nothing baseline 0, the literal solution at most 0.2, and shortcut strategies at most 0.8, with graders tested against more than 270 bypass strategies. Scripted stakeholders rather than LLM-simulated users make interactions reproducible, and the grader's record of which key facts an agent obtained lets the paper separate information discovery from system building in its analysis.

Core claim. At L3, the benchmark's realistic setting, the sponsor's original request may target the wrong problem and the agent must decide whether the request even addresses the underlying issue. The paper reports that Claude Opus 5.5, the best of the 13 configurations, reaches a mean FDE-Gym score of 0.56 (95 percent interval 0.50 to 0.62) and resolves 14 of 95 tasks, where resolved means scoring at least 0.8 with no operational incident; five configurations resolve none, and even an oracle that picks the best configuration per task reaches only 0.70 with 30 tasks resolved. The score decomposes into three losses: withholding the constraints costs the leading models about 0.28 on the same tasks (the L0 to L1 step), making the request vague and then misdirected costs only a further 0.04, and stakeholder contact costs consume 0.14 to 0.22 of the delivered uplift. The paper's central claim about the residual gap is that information discovery alone does not explain it: on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. Weaker models are worse than nothing, with Claude Haiku 4.5 scoring below the do-nothing baseline on 79 of 95 tasks and Mistral Small 4 doing so on 51 even before contact costs are deducted.

The premise it rests on. The claim that discovering unstated constraints is the primary bottleneck assumes that the only meaningful difference between the fully specified level and the next level down is the presence of the rules, but the fully specified level also tells the agent what will be measured, so part of the measured drop could simply be the agent not knowing the scoring target.

If the paper is right (Pith's inferences, not the authors')

A model's score on component benchmarks is a weak guide to delivery: the same models that solve up to 88 to 90 percent of terminal, web-app, and finance-research tasks improve the customer's business metric on FDE-Gym in at most one of seven engagements.

Even an oracle that selects the best configuration for each task reaches only 0.70 with 30 of 95 tasks resolved, so no single evaluated model or simple selection rule closes the gap.

Stakeholder contact cost is a large hidden deduction: GPT-6 Astra's delivered systems reach 0.8 uplift on 36 tasks, but only 9 remain resolved after contact costs, meaning agents that ask as if questions were free are penalized for it.

Weak models are not a cheaper version of strong ones: Mistral Small 4's delivered systems fall below the do-nothing baseline on 51 of 95 tasks before any contact cost is deducted, so deploying them can be expected to harm the customer.

Performance is strongly fact-dependent: on five of 20 scenarios no model averages 0.5, and across tasks within a scenario the median score range for a leading model is 0.34 to 0.44, so a solution that works on one variant can fail on a sibling with different key facts.

The reported L0-to-L1 drop conflates two things: the fully specified level also hands the agent the scoring criteria and the business metric, so part of the 0.28 loss may be 'agent did not know what would be measured' rather than 'agent could not find the rules'; an intermediate level that states the metric but withholds the constraints would settle this.

The score definition suggests a concrete training target that the paper does not propose: optimize $S = U_{\mathrm{adj}} - C$ directly, which would reward agents for asking fewer, better questions and stopping when the delivery is good enough, rather than maximizing uplift alone.

The finding that several models routinely score below zero implies a cheap deployment safeguard: measure delivered uplift on a holdout slice before committing to production and roll back negative-uplift deliveries.

The promised human baseline will be the natural calibration of the scale: if experienced engineers also score well below 0.8 on many L3 tasks, the resolved threshold is stricter than 'competent delivery'; if they score near the reference, the 0.56 result is a genuine model capability gap.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 1.00

“These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.80 · human participants 0.68 · controlled comparison 0.53 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.55 · mentions code or data availability 0.14

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. background 0.99 0.04 0.02 asserted 1.00
Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. gap 0.52 0.08 0.05 asserted 1.00
We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. contribution 0.94 0.93 0.04 asserted 1.00
In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. method 0.95 0.60 0.03 asserted 0.99
FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). method 0.97 0.79 0.04 asserted 1.00
An online platform lets humans practice FDE skills under the same grading criteria. contribution 0.72 0.71 0.03 asserted 1.00
We evaluate 13 model configurations from seven providers. method 0.88 0.90 0.02 asserted 0.96
In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). result 1.00 0.96 0.22 asserted 1.00
Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. result 0.51 0.90 0.22 asserted 1.00
Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. result 0.90 0.96 0.14 asserted 1.00
These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers. interpretation 1.00 0.87 0.85 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 8,311 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Construction of FDE-Gym. Experts adapt 20 real-world delivery projects from 18 sectors into operational customer environments, alter key facts to create variant tasks, and retain the 95 tasks whose graders have been validated against the predefined scoring anchors
Figure 2 of the paper
Figure 2. Model performance across task instruction levels. (a) Mean L3 score for each model on all 95 tasks, on a normalized scale where 0 represents the do-nothing baseline and 1 represents the reference solution. Solid bars show scores after deducting stakeholder contact cost; lighter bars show normalized business uplift before this deduction. (b) Scores as progressively less information is provided in the task instruction. L0-L2 are evaluated on the 20 original tasks and L3 on all 95 tasks. 1 arXiv:2610.22350v1 [cs.AI] 15 Oct 2026
Figure 3 of the paper
Figure 3. Components of an FDE-Gym task, showing the inputs, agent actions and deliverable, deployment and evaluation of the delivered system, and interpretation of the resulting score. The agent reads the task inputs and exchanges messages with customer stakeholders, who answer questions and may also initiate contact. A task is included only if the scoring anchors shown on the axis are satisfied. 2.1 Environment and Available Information The agent starts in a workspace containing the task instruction (the description of the work assigned by the sponsor), a copy of the customer’s production…
Figure 4 of the paper
Figure 4. A worked example: the drilling scenario (oil_npt), original task. where K is the set of chargeable contacts in the run and ck is the charge for contact k under the task’s predefined pricing and contact rules. Each stakeholder has a hidden per-message rate, with higher charges when the agent exceeds their patience or violates a contact restriction. Unanswered messages are not charged. In scenarios with a trial environment or a calendar, each trial run and each day of waiting is also a chargeable contact. FDE-Gym Score (S). The FDE-Gym Score measures delivery performance in a single…
Figure 5 of the paper
Figure 5. (a) Share of key stakeholder facts obtained against normalized business uplift, at L3. Leading models learn most of the facts and deliver 0.60–0.70 of the reference uplift. (b) Mean model usage cost per run against score at L3. results for the same 20 tasks and averaging across the three models, the decline from L0 to L1 is 0.28 (95% interval 0.17 to 0.39) and the decline from L1 to L3 is 0.04 (−0.03 to 0.14). The first step therefore costs more than the two that follow, although a further loss of up to 0.14 cannot be excluded, and for an individual model the upper bound is as large…
Figure 6 of the paper
Figure 6. Distribution of each model’s 95 tasks at L3 by score band. Scores of 0.2 or below correspond to the cap on literal compliance with the sponsor’s task instruction. The second component of the finding is stakeholder contact cost. GPT-6 Astra’s delivered systems reach 0.8 on 36 of 95 tasks, but only 9 remain resolved once stakeholder contact cost is deducted; for Claude Opus 5.5 the figures are 40 and 14. Agents are not informed that questions incur a cost, and they ask as if questions were free. Stakeholder contact cost consumes 0.14 of the 0.70 uplift delivered by Claude Opus 5.5 and…
Figure 7 of the paper
Figure 7. Mean score at L3 on each scenario, for each model; scenarios are ordered by the best model’s score, and the highest￾scoring cell in each row is outlined. Color runs from red (below the do-nothing baseline) to blue (at the reference solution)
Figure 8 of the paper
Figure 8. Model performance on Terminal-Bench 2.1, Vibe Code Bench, Finance Agent, and FDE-Gym. 5.7 Comparison with other benchmarks

Figures are the authors', reproduced from the paper.

References (40)
  1. [1] Carlos E. Jimenez, John Y ang, Alexander Wettig, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations (ICLR) , 2024 on Pith
  2. [2] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv preprint arXiv:2601.11868, 2026 on PithPith's reviewarXiv
  3. [3] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? In International Conference on Machine Learning (ICML) , 2025 on Pith
  4. [4] Weihang Ding, Junfei Zhan, Yueting Li, and Qirong Guo. FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration. arXiv preprint arXiv:2609.27571, 2026 on PithPith's reviewarXiv
  5. [5] Frank F. Xu, Yufan Song, Boxuan Li, et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , 2025 on Pith
  6. [6] Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, et al. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. Transactions on Machine Learning Research (TMLR), 2025 on Pith
  7. [7] Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Y erukola, Maarten Sap, and Graham Neubig. Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering. In International Conference on Learning Representations (ICLR) , 2026 on Pith
  8. [8] George Pu, Michael S. Lee, Udari Madhushani Sehwag, et al. LHA W: Controllable Underspecification for Long-Horizon Tasks. arXiv preprint arXiv:2602.10525, 2026 on PitharXiv
  9. [9] Quan Shi, Keshav Dhandhania, Karthik Narasimhan, and Victor Barres. τ τ -Bench: An Environment for End-To-End, Realistic Agent Construction. arXiv preprint arXiv:2609.04611, 2026 on PithPith's reviewarXiv
  10. [10] Weihang Ding and Junfei Zhan. Trains but Doesn’t Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Industry Track, 2026 on Pith
  11. [11] Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim. XYEval: Agents say yes to bad advice. arXiv preprint arXiv:2609.23939, 2026 on PithPith's reviewarXiv
  12. [12] Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development. In ACM CAIS, 2026 on Pith
  13. [13] Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks. arXiv preprint arXiv:2508.00828, 2025 on PithPith's reviewarXiv
  14. [14] Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations. In Annual Meeting of the Association for Computational Linguistics (AC... on Pith
  15. [15] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. Transactions on Machine Learning Research (TMLR), 2025 on Pith
  16. [16] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, et al. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. In International Conference on Learning Representations (ICLR) , 2026 on Pith
  17. [17] Xin Zhou, Chun Y ong Chong, Kisub Kim, et al. A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents. arXiv preprint arXiv:2608.09072, 2026 on PithPith's reviewarXiv
  18. [18] Cheng Qian, Zuxin Liu, Akshara Prabhakar, et al. UserBench: An Interactive Gym Environment for User-Centric Agents. arXiv preprint arXiv:2507.22034, 2025 on PithPith's reviewarXiv
  19. [19] Dongming Jin, Zhi Jin, Zheng Fang, et al. ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation. arXiv preprint arXiv:2602.18306, 2026 on PitharXiv
  20. [20] Zirui Tang, Xuanhe Zhou, Yumou Liu, et al. Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies. arXiv preprint arXiv:2605.03596, 2026 on PithPith's reviewarXiv
show 20 more
  1. [21] Fabian Wenz, Felix Treutwein, Çagatay Demiralp, and Michael Stonebraker. RUBICON: Agentic AI for Messy Enterprise Data. arXiv preprint arXiv:2604.21413, 2026 on PithPith's reviewarXiv
  2. [22] Amirhossein Abaskohi, Tianyi Chen, Miguel Muñoz-Mármol, et al. DRBench: A Realistic Benchmark for Enterprise Deep Research. arXiv preprint arXiv:2510.00172, 2025 on PitharXiv
  3. [23] Ved Sirdeshmukh and Marc Wetter. Implicit Intelligence: Evaluating Agents on What Users Don’t Say. arXiv preprint arXiv:2602.20424, 2026 on PitharXiv
  4. [24] Léo Boisvert, Megh Thakkar, Maxime Gasse, et al. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. In Advances in Neural Information Processing Systems (NeurIPS) , 2024 on Pith
  5. [25] Min Y ang, Yichen Pan, Jinghua Piao, Dandan Song, Y ongshun Gong, and Y ong Li. EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making. arXiv preprint arXiv:2609.37658, 2026 on PithPith's reviewarXiv
  6. [26] Bertie Vidgen, Austin Mann, Abby Fennelly, et al. APEX-Agents. arXiv preprint arXiv:2601.14242, 2026 on PitharXiv
  7. [27] Xiang Deng, Jeff Da, Edwin Pan, et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv preprint arXiv:2509.16941, 2025 on PithPith's reviewarXiv
  8. [28] Saurabh Jha, Rohan Arora, Yuji Watanabe, et al. ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. In International Conference on Machine Learning (ICML) , 2025 on Pith
  9. [29] Yinfang Chen, Manish Shetty, Gagan Somashekar, et al. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds. In Conference on Machine Learning and Systems (MLSys) , 2025 on Pith
  10. [30] Axel Backlund and Lukas Petersson. Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents. arXiv preprint arXiv:2502.15840, 2025 on PithPith's reviewarXiv
  11. [31] Wei Fan, Xinjie Shen, Xudong Guo, et al. E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation. arXiv preprint arXiv:2608.30730, 2026 on PithPith's reviewarXiv
  12. [32] Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, et al. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv preprint arXiv:2510.04374, 2025. 13 on PithPith's reviewarXiv
  13. [33] Mantas Mazeika, Alice Gatti, Cristina Menghini, et al. Remote Labor Index: Measuring AI Automation of Remote Work. arXiv preprint arXiv:2510.26787, 2025 on PitharXiv
  14. [34] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025 on PithPith's reviewarXiv
  15. [35] Zimo Ji, Zekai Zhang, Congying Xu, et al. Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions. arXiv preprint arXiv:2607.02294, 2026 on PithPith's reviewarXiv
  16. [36] Shunyu Y ao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024 on PithPith's reviewarXiv
  17. [37] Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a Science of AI Agent Reliability. In International Conference on Machine Learning (ICML) , 2026 on Pith
  18. [38] Yuxuan Zhu, Tengjun Jin, Y ada Pruksachatkun, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv preprint arXiv:2507.02825, 2025 on PithPith's reviewarXiv
  19. [39] Thomas Kwa, Ben West, Joel Becker, et al. Measuring AI Ability to Complete Long Software Tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2025 on Pith
  20. [40] Yijia Shao, Vinay Samuel, Yucheng Jiang, John Y ang, and Diyi Y ang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. In International Conference on Learning Representations (ICLR) , 2026. Image credits (Figure 1). Photographs from Wikimedi... on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment