Artificial Intelligence · arXiv:2610.22350 v1 · submitted 15 Oct 2026 · 40 references
FDE-Gym: The Last Mile of AI Delivery
The paper says
“Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints.”
more of the abstract
We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuinely careful benchmark with credible headline numbers, but the abstract's 'constraint discovery is the primary bottleneck' claim is weaker than the data support because L0 also hands the agent the scoring criteria and the business metric. This line is wrong
1 major objection · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 40 references say.
The paper says, at §5.2 / Table 1 / Eq. (1)
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The L0-to-L1 comparison does not isolate constraint discovery. Table 1 states that the L0 instruction provides the underlying problem, applicable rules and constraints, and "how the result is scored," while L1 provides only a clear statement of the problem. Since Eq.
more of objection 1
(1) defines the score as normalized improvement on the customer's KPI, removing the scoring information at the same step as the constraints means the reported 0.28 drop mixes failure to discover unstated rules with uncertainty about which business metric will be measured. Telling an agent which KPI is evaluated changes the objective it can optimize, so the abstract's claim that "discovering unstated constraints is the primary bottleneck" overstates what the comparison can establish. The manuscript does not acknowledge this confound in §5.2 or in Section 7. I recommend either adding a control condition that supplies the business metric and scoring criteria without the constraints, or revising the claim to refer to withheld task-relevant information generally, with constraint discovery presented as one component rather than the cleanly identified primary bottleneck.
3 minor objections
The paper says, at §2.3, Eqs. (4)–(5)
Pith says: minor objection 1 of 3
The average FDE-Gym score in Eq. (5) uses the same symbol S as the single-run score in Eq. (4); using an overbar or a different symbol for the average would prevent ambiguity.
The paper says, at §5.2
Pith says: minor objection 2 of 3
The paired 95% interval of 0.17 to 0.39 for the L0-to-L1 drop is reported without describing the resampling procedure; please specify whether the bootstrap resamples tasks under the paired design and report the corresponding per-model intervals.
show 1 more
The paper says, at Figure 2b caption
Pith says: minor objection 3 of 3
The caption states that L0-L2 are evaluated on the 20 original tasks and L3 on all 95 tasks, but the curve in Figure 2b appears to show the level progression; please clarify whether the final point is the original-task L3 mean or the full 95-task mean, since Table 3 reports both separately.
What would settle it
On the same 20 original tasks used for L0-L2, add an L1+ condition whose task instruction contains the L1 underlying-problem statement plus the L0 scoring sentence ('how the result is scored': the customer KPI and the normalization in Eq. 1), while still omitting the applicable rules and constraints. Run the three leading configurations three times as in Section 4 and compare the paired L0-to-L1 drop with the L0-to-L1+ drop. If the L0-to-L1+ drop is much smaller than 0.28 (for example below about 0.10), the claimed constraint-discovery penalty is substantially a scoring-criterion effect; if the drop remains near 0.28, the concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 1 major objection, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Worth a serious look. FDE-Gym is one of the more carefully constructed agent benchmarks I have seen in a while, and the headline results are probably about right. The combination is new: 95 tasks from 20 real delivery projects, scored on the customer's own business metric with the do-nothing baseline at 0 and the reference solution at 1, literal compliance capped at 0.2, scripted stakeholders instead of an LLM judge, three runs per configuration, bootstrapped confidence intervals, and admission anchors tested against 270 shortcut strategies. That is real methodology work, and the paper earns credit for it. The central empirical findings — best configuration at 0.56, resolving 14 of 95 tasks; weak models below zero on most tasks; stakeholders' facts recovered but not converted into uplift — are credible and useful on their own. The soft spot the stress-test note identifies is real, and the paper does not acknowledge it. Finding 2 attributes the L0-to-L1 drop of about 0.28 to withheld constraints, but Table 1 shows L0 also provides the scoring criteria and the business metric. Telling an agent what KPI will be measured changes the objective it optimizes; withholding it forces inference. The drop therefore mixes constraint discovery with objective-function uncertainty, so the abstract's 'discovering unstated constraints is the primary bottleneck' is not cleanly established. Notice the paper's own conclusion shifts emphasis to 'translating those facts into an effective system' and stakeholder contact cost — which is consistent with Finding 3 and with the L1-to-L3 flatness, but it is a different claim from the abstract. The fix is easy: give the scoring criteria at L1, or reframe the claim. Two smaller issues. First, 75 of 95 tasks are held out, so the full results cannot be independently reproduced; the authors disclose this, but it still limits verification of the ranking. Second, Section 5.7's 'one in seven engagements' line leans hard on comparing success rates with normalized uplift; they do disclaim the mismatch, so I would call it minor. Who is this for: anyone building agent benchmarks, anyone evaluating agents for real deployments, and procurement-minded readers. The benchmark deserves a serious referee, and the conditional verdict is the right stance. I would send it to review and ask the authors to address the L1 confound before it is cited as evidence about constraint discovery.
The strongest objection
The most load-bearing comparison is Finding 2 in Section 5.2: the L0-to-L1 drop of 0.28 is attributed to 'withholding the constraints,' and this is the evidence for the abstract's claim that discovering unstated constraints is the primary bottleneck. But Table 1 states that L0 includes 'the underlying problem, applicable rules and constraints, and how the result is scored,' while L1 includes only a clear statement of the underlying problem. The scoring criteria and the customer's business metric are therefore removed at the same step as the constraints. This matters because the FDE-Gym score is defined as improvement on the customer's KPI (Eq. 1); telling an agent which KPI will be measured changes the objective it can optimize, while withholding it forces the agent to infer what is valued from documents and stakeholders. The observed 0.28 drop thus mixes constraint discovery with objective-function uncertainty. The paper does not acknowledge this confound in Section 5.2 or in the Limitations section. The empirical headline result of 0.56 and 14/95 resolved is unaffected, but the causal claim about the primary bottleneck is not cleanly established by the reported comparison.
The reader's weakest assumption already identifies this confound, and I agree with it. The benchmark construction is otherwise unusually careful: admission anchors, three runs per configuration, scripted stakeholders, and cost reporting all give independent support to the quantitative measurement of 0.56 mean score and 14/95 resolved tasks. Those headline numbers do not depend on the L0-to-L1 attribution. What is load-bearing is the separate causal claim, repeated in the abstract and Section 5.2, that discovering unstated constraints is the primary bottleneck. That claim rests entirely on comparing L0 with L1, and the two conditions differ in the scoring information as well as in the constraints. If the proposed L1+ ablation shows a much smaller drop, the central bottleneck statement would need to be revised to distinguish 'not knowing the business metric' from 'not knowing the constraints'; if the drop is unchanged, the concern is empirically resolved. Because the paper is already CONDITIONAL and this is the same concern, no verdict change is needed; the condition should explicitly require the L1+ ablation or an equivalent isolation of scoring-criterion effects before the bottleneck claim is accepted.
The referee's report
If the empirical results are accepted, FDE-Gym is a substantial contribution to agent evaluation. Its strengths are real: scoring on the customer's own business metric rather than binary task completion, predefined admission anchors that cap literal compliance and shortcut strategies, held-out grading, three runs per configuration, key-fact tracking, a large set of cross-sector scenarios, and an open-source harness and experiment runner. The headline quantitative result—best configuration 0.56 mean score and 14/95 resolved—is a striking and falsifiable measurement, as is the finding that leading models obtain most stakeholder facts yet still fall well short of reference uplift. The main load-bearing weakness is that the L0-to-L1 comparison, which carries the abstract's causal claim about constraint discovery, also removes the scoring criteria and business metric, so the paper's strongest interpretive claim is not cleanly identified by the reported experiment.
Pith's reading of the paper (a machine's summary, not the authors')
FDE-Gym is a benchmark that grades an AI agent the way a customer would: not by whether a system was delivered, but by how much the customer's own business metric improved, on a scale where doing nothing scores 0 and an expert reference solution scores 1. The paper's central finding is that current frontier agents fall far short on this scale: the best of 13 configurations averages 0.56 and resolves only 14 of 95 realistic delivery tasks, even though it gathers 72 to 85 percent of the key facts stakeholders hold. The largest measured performance drop occurs when agents must discover unstated constraints on their own, while a misdirected sponsor request costs them almost nothing beyond that. The paper argues this reveals a gap between solving well-specified benchmark tasks and delivering real customer value, with weak models frequently leaving the customer worse off than no delivery at all.
What carries the argument. The mechanism carrying the argument is the FDE-Gym score, $S = U_{\mathrm{adj}} - C$, defined as incident-adjusted normalized business uplift minus stakeholder contact cost, where uplift is $U = (\mathrm{KPI}_{\mathrm{delivered}} - \mathrm{KPI}_{\mathrm{baseline}})/(\mathrm{KPI}_{\mathrm{reference}} - \mathrm{KPI}_{\mathrm{baseline}})$ evaluated on held-out production data. The four task instruction levels (L0 to L3) vary only the instruction, from a fully specified problem to the sponsor's original request, while the environment, scripted stakeholders, and grader stay fixed, so score differences across levels are meant to isolate the effect of withheld information. Admission anchors guarantee the scale is meaningful: the reference solution must score at least 0.95, the do-nothing baseline 0, the literal solution at most 0.2, and shortcut strategies at most 0.8, with graders tested against more than 270 bypass strategies. Scripted stakeholders rather than LLM-simulated users make interactions reproducible, and the grader's record of which key facts an agent obtained lets the paper separate information discovery from system building in its analysis.
Core claim. At L3, the benchmark's realistic setting, the sponsor's original request may target the wrong problem and the agent must decide whether the request even addresses the underlying issue. The paper reports that Claude Opus 5.5, the best of the 13 configurations, reaches a mean FDE-Gym score of 0.56 (95 percent interval 0.50 to 0.62) and resolves 14 of 95 tasks, where resolved means scoring at least 0.8 with no operational incident; five configurations resolve none, and even an oracle that picks the best configuration per task reaches only 0.70 with 30 tasks resolved. The score decomposes into three losses: withholding the constraints costs the leading models about 0.28 on the same tasks (the L0 to L1 step), making the request vague and then misdirected costs only a further 0.04, and stakeholder contact costs consume 0.14 to 0.22 of the delivered uplift. The paper's central claim about the residual gap is that information discovery alone does not explain it: on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. Weaker models are worse than nothing, with Claude Haiku 4.5 scoring below the do-nothing baseline on 79 of 95 tasks and Mistral Small 4 doing so on 51 even before contact costs are deducted.
The premise it rests on. The claim that discovering unstated constraints is the primary bottleneck assumes that the only meaningful difference between the fully specified level and the next level down is the presence of the rules, but the fully specified level also tells the agent what will be measured, so part of the measured drop could simply be the agent not knowing the scoring target.
If the paper is right (Pith's inferences, not the authors')
A model's score on component benchmarks is a weak guide to delivery: the same models that solve up to 88 to 90 percent of terminal, web-app, and finance-research tasks improve the customer's business metric on FDE-Gym in at most one of seven engagements.
Even an oracle that selects the best configuration for each task reaches only 0.70 with 30 of 95 tasks resolved, so no single evaluated model or simple selection rule closes the gap.
Stakeholder contact cost is a large hidden deduction: GPT-6 Astra's delivered systems reach 0.8 uplift on 36 tasks, but only 9 remain resolved after contact costs, meaning agents that ask as if questions were free are penalized for it.
Weak models are not a cheaper version of strong ones: Mistral Small 4's delivered systems fall below the do-nothing baseline on 51 of 95 tasks before any contact cost is deducted, so deploying them can be expected to harm the customer.
Performance is strongly fact-dependent: on five of 20 scenarios no model averages 0.5, and across tasks within a scenario the median score range for a leading model is 0.34 to 0.44, so a solution that works on one variant can fail on a sibling with different key facts.
The reported L0-to-L1 drop conflates two things: the fully specified level also hands the agent the scoring criteria and the business metric, so part of the 0.28 loss may be 'agent did not know what would be measured' rather than 'agent could not find the rules'; an intermediate level that states the metric but withholds the constraints would settle this.
The score definition suggests a concrete training target that the paper does not propose: optimize $S = U_{\mathrm{adj}} - C$ directly, which would reward agents for asking fewer, better questions and stopping when the delivery is good enough, rather than maximizing uplift alone.
The finding that several models routinely score below zero implies a cheap deployment safeguard: measure delivered uplift on a holdout slice before committing to production and roll back negative-uplift deliveries.
The promised human baseline will be the natural calibration of the scale: if experienced engineers also score well below 0.8 on many L3 tasks, the resolved threshold is stricter than 'competent delivery'; if they score near the reference, the 0.56 result is a genuine model capability gap.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 1.00
“These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.80 · human participants 0.68 · controlled comparison 0.53 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.55 · mentions code or data availability 0.14
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. | background 0.99 | 0.04 | 0.02 | asserted 1.00 |
| Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. | gap 0.52 | 0.08 | 0.05 | asserted 1.00 |
| We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. | contribution 0.94 | 0.93 | 0.04 | asserted 1.00 |
| In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. | method 0.95 | 0.60 | 0.03 | asserted 0.99 |
| FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). | method 0.97 | 0.79 | 0.04 | asserted 1.00 |
| An online platform lets humans practice FDE skills under the same grading criteria. | contribution 0.72 | 0.71 | 0.03 | asserted 1.00 |
| We evaluate 13 model configurations from seven providers. | method 0.88 | 0.90 | 0.02 | asserted 0.96 |
| In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). | result 1.00 | 0.96 | 0.22 | asserted 1.00 |
| Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. | result 0.51 | 0.90 | 0.22 | asserted 1.00 |
| Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. | result 0.90 | 0.96 | 0.14 | asserted 1.00 |
| These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers. | interpretation 1.00 | 0.87 | 0.85 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.92
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 8,311 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (40)
- [1] Carlos E. Jimenez, John Y ang, Alexander Wettig, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations (ICLR) , 2024 on Pith
- [2] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv preprint arXiv:2601.11868, 2026 on PithPith's reviewarXiv
- [3] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? In International Conference on Machine Learning (ICML) , 2025 on Pith
- [4] Weihang Ding, Junfei Zhan, Yueting Li, and Qirong Guo. FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration. arXiv preprint arXiv:2609.27571, 2026 on PithPith's reviewarXiv
- [5] Frank F. Xu, Yufan Song, Boxuan Li, et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , 2025 on Pith
- [6] Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, et al. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. Transactions on Machine Learning Research (TMLR), 2025 on Pith
- [7] Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Y erukola, Maarten Sap, and Graham Neubig. Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering. In International Conference on Learning Representations (ICLR) , 2026 on Pith
- [8] George Pu, Michael S. Lee, Udari Madhushani Sehwag, et al. LHA W: Controllable Underspecification for Long-Horizon Tasks. arXiv preprint arXiv:2602.10525, 2026 on PitharXiv
- [9] Quan Shi, Keshav Dhandhania, Karthik Narasimhan, and Victor Barres. τ τ -Bench: An Environment for End-To-End, Realistic Agent Construction. arXiv preprint arXiv:2609.04611, 2026 on PithPith's reviewarXiv
- [10] Weihang Ding and Junfei Zhan. Trains but Doesn’t Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Industry Track, 2026 on Pith
- [11] Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim. XYEval: Agents say yes to bad advice. arXiv preprint arXiv:2609.23939, 2026 on PithPith's reviewarXiv
- [12] Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development. In ACM CAIS, 2026 on Pith
- [13] Antoine Bigeard, Langston Nashold, Rayan Krishnan, and Shirley Wu. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks. arXiv preprint arXiv:2508.00828, 2025 on PithPith's reviewarXiv
- [14] Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb-Tarrant. Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations. In Annual Meeting of the Association for Computational Linguistics (AC... on Pith
- [15] Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. Transactions on Machine Learning Research (TMLR), 2025 on Pith
- [16] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, et al. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. In International Conference on Learning Representations (ICLR) , 2026 on Pith
- [17] Xin Zhou, Chun Y ong Chong, Kisub Kim, et al. A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents. arXiv preprint arXiv:2608.09072, 2026 on PithPith's reviewarXiv
- [18] Cheng Qian, Zuxin Liu, Akshara Prabhakar, et al. UserBench: An Interactive Gym Environment for User-Centric Agents. arXiv preprint arXiv:2507.22034, 2025 on PithPith's reviewarXiv
- [19] Dongming Jin, Zhi Jin, Zheng Fang, et al. ReqElicitGym: An Evaluation Environment for Interview Competence in Conversational Requirements Elicitation. arXiv preprint arXiv:2602.18306, 2026 on PitharXiv
- [20] Zirui Tang, Xuanhe Zhou, Yumou Liu, et al. Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies. arXiv preprint arXiv:2605.03596, 2026 on PithPith's reviewarXiv
show 20 more
- [21] Fabian Wenz, Felix Treutwein, Çagatay Demiralp, and Michael Stonebraker. RUBICON: Agentic AI for Messy Enterprise Data. arXiv preprint arXiv:2604.21413, 2026 on PithPith's reviewarXiv
- [22] Amirhossein Abaskohi, Tianyi Chen, Miguel Muñoz-Mármol, et al. DRBench: A Realistic Benchmark for Enterprise Deep Research. arXiv preprint arXiv:2510.00172, 2025 on PitharXiv
- [23] Ved Sirdeshmukh and Marc Wetter. Implicit Intelligence: Evaluating Agents on What Users Don’t Say. arXiv preprint arXiv:2602.20424, 2026 on PitharXiv
- [24] Léo Boisvert, Megh Thakkar, Maxime Gasse, et al. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. In Advances in Neural Information Processing Systems (NeurIPS) , 2024 on Pith
- [25] Min Y ang, Yichen Pan, Jinghua Piao, Dandan Song, Y ongshun Gong, and Y ong Li. EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making. arXiv preprint arXiv:2609.37658, 2026 on PithPith's reviewarXiv
- [26] Bertie Vidgen, Austin Mann, Abby Fennelly, et al. APEX-Agents. arXiv preprint arXiv:2601.14242, 2026 on PitharXiv
- [27] Xiang Deng, Jeff Da, Edwin Pan, et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv preprint arXiv:2509.16941, 2025 on PithPith's reviewarXiv
- [28] Saurabh Jha, Rohan Arora, Yuji Watanabe, et al. ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. In International Conference on Machine Learning (ICML) , 2025 on Pith
- [29] Yinfang Chen, Manish Shetty, Gagan Somashekar, et al. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds. In Conference on Machine Learning and Systems (MLSys) , 2025 on Pith
- [30] Axel Backlund and Lukas Petersson. Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents. arXiv preprint arXiv:2502.15840, 2025 on PithPith's reviewarXiv
- [31] Wei Fan, Xinjie Shen, Xudong Guo, et al. E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation. arXiv preprint arXiv:2608.30730, 2026 on PithPith's reviewarXiv
- [32] Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, et al. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv preprint arXiv:2510.04374, 2025. 13 on PithPith's reviewarXiv
- [33] Mantas Mazeika, Alice Gatti, Cristina Menghini, et al. Remote Labor Index: Measuring AI Automation of Remote Work. arXiv preprint arXiv:2510.26787, 2025 on PitharXiv
- [34] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025 on PithPith's reviewarXiv
- [35] Zimo Ji, Zekai Zhang, Congying Xu, et al. Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions. arXiv preprint arXiv:2607.02294, 2026 on PithPith's reviewarXiv
- [36] Shunyu Y ao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045, 2024 on PithPith's reviewarXiv
- [37] Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a Science of AI Agent Reliability. In International Conference on Machine Learning (ICML) , 2026 on Pith
- [38] Yuxuan Zhu, Tengjun Jin, Y ada Pruksachatkun, et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv preprint arXiv:2507.02825, 2025 on PithPith's reviewarXiv
- [39] Thomas Kwa, Ben West, Joel Becker, et al. Measuring AI Ability to Complete Long Software Tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2025 on Pith
- [40] Yijia Shao, Vinay Samuel, Yucheng Jiang, John Y ang, and Diyi Y ang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. In International Conference on Learning Representations (ICLR) , 2026. Image credits (Figure 1). Photographs from Wikimedi... on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment