Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CEO-Bench: Can Agents Play the Long Game?

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most state-of-the-art language-model agents go bankrupt when asked to run a simulated startup for 500 days.

desk verdict CEO-Bench's environment is a genuine step forward in long-horizon agent evaluation, but the single-seed best-of-three protocol and an abstract/table mismatch weaken the headline claims. read the letter →

arxiv 2606.18543 v2 pith:PPA4QS5E submitted 2026-06-16 cs.AI cs.CLcs.SE

Haozhe Chen , Karthik Narasimhan , Zhuang Liu This is my paper · ORCID
classification cs.AIcs.CLcs.SE
keywords long-horizonagentsagentbenchmarkstartupsimulationlanguage-modeldecision-makingunderuncertaintynon-stationaryenvironmentdelayedfeedbackevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CEO-Bench is an attempt to measure a capability that short-horizon benchmarks miss: sustaining coherent, adaptive decisions over a long time under noisy and delayed feedback. The paper constructs a 500-day simulation of a subscription-software startup—26 customer groups, 34 tools, a 19-table database, social media, competitors, and macro cycles—and hands control to a language-model agent through a Python terminal. The headline result is that current agents largely fail: most go bankrupt or finish below their $1M starting cash, a simple hand-tuned rule-based policy beats all but the two strongest agents, and the best runs stay far below the estimated achievable ceiling. The paper reads this as evidence that tool-level competence does not yet translate into long-horizon strategic control.

What carries the argument

The load-bearing object is the CEO-Bench simulator itself: a mechanistic, stochastic world in which every customer is an individual with a private price-quality tolerance curve, and subscribes only if perceived quality clears a price-dependent minimum requirement. Actions are taken through a composable Python API, and the world responds through coupled equations—reputation spillovers between customer groups, a competitor that raises quality expectations faster when the agent makes broad improvements, a hidden mean-reverting macro cycle, delayed R&D payoffs, and Poisson-distributed acquisition and churn. Because the agent observes only indirect traces, success depends on writing code to query

What would settle it

Using the released trajectories, check whether a model run with near-perfect four-week cash forecasts and fast competitor detection still goes bankrupt; if forecasting skill does not track survival, the claimed skill-success link weakens. Alternatively, re-run the simple rule-based baseline under many random seeds other than the single seed used; if its $15.76M result collapses under other seeds, the headline comparison against LLM agents is not stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that evaluating agents on isolated, short-horizon tasks overstates their readiness for real-world work, because the hard part of sustained decision-making is making many interdependent choices whose consequences arrive late, noisily, and through hidden state. CEO-Bench instantiates that challenge by making an agent run NovaMind, a fictional subscription-software company, for 500 simulated days; the agent sees only dashboards, databases, social-media posts, and negotiation histories, and must infer willingness-to-pay, satisfaction, and competitor moves. On this task, most state-of-the-art models cannot survive; the two strongest finish above the starting balance b

Load-bearing premise

The load-bearing premise is that the hand-built simulator—with its authored customer participation rule, competitor catch-up behavior, and macro cycle, none calibrated to real-world data—is a representative stand-in for the long-horizon decision-making the paper wants to measure; if the mechanics miss the essential difficulties of real sustained management, the benchmark's conclusions may not transfer beyond the simulation.

Editorial extensions

If this is right

  • If CEO-Bench measures what it claims, high scores on short-horizon agent benchmarks do not imply ability to steer a long-running organization; readiness must be tested over horizons where costs and benefits compound.
  • The four measured skills—allocating spend to hidden high-yield channels, forecasting cash four weeks ahead, detecting competitor moves quickly, and forming if-then plans—can serve as progress metrics for long-horizon agent research.
  • Because even the best agents sit far below the estimated $2.2B ceiling, the benchmark has room to discriminate between current and future models rather than saturating immediately.
  • A simple rule-based policy that makes no model calls outperforming nearly all LLM agents implies that the bottleneck is not knowledge but sustained, consistent execution of a coherent strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the benchmark's conclusions depend on its hand-authored equations and parameters (customer participation, competitor catch-up, macro cycle), which are not calibrated to real market data; a different simulator design could shift which models succeed.
  • Inference: a strong control experiment would be to put human startup operators on the same interface; if experienced humans beat the best agents, the benchmark captures a real gap, while if they do not, it may reward simulation-specific heuristics.
  • Inference: since a fixed playbook beats adaptive LLMs, part of the failure may be strategic instability—overreacting to recent noise instead of committing to a plan; testing whether prompt-level 'strategy lock-in' improves outcomes would be a cheap experiment.
  • Inference: the four skill metrics could be extracted as standalone diagnostic tasks, letting researchers identify which long-horizon skill a model lacks without running the full 500-day simulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. CEO-Bench introduces a 500-day simulated startup environment for evaluating LLM agents on long-horizon tasks. Agents manage a fictional company through a Python API, using SQL analytics, pricing, marketing, product development, infrastructure, enterprise sales, and social media to maximize final cash. The paper evaluates ten models (three runs each), reports that most fail to grow the $1M starting balance, identifies Claude Opus 4.8 and GPT-5.5 as the only models finishing above $1M, and compares against a rule-based baseline reaching $15.76M. It also presents ablations on competitor difficulty, time horizon, and agent harness, plus qualitative analyses of agent behavior.

Significance. The benchmark addresses a real gap: existing agent evaluations largely measure short, well-specified tasks, whereas CEO-Bench attempts to measure sustained strategic control under hidden state, delayed feedback, and non-stationarity. The environment is mechanistic, open-ended, and released with trajectories, which are strengths. However, the current experimental evidence is not yet sufficient to support the headline conclusions: all runs use a single environment seed, the rule-based baseline is tuned on that same seed, and results are reported via best-run selection. If the multi-seed evidence and reporting inconsistencies are addressed, the benchmark could be a useful contribution to long-horizon agent evaluation.

major comments (4)
  1. [§3.1, §3.2, Appendix B] All model runs use simulator seed 42, and the rule-based baseline is selected by grid search on the same seed (§3.1 and Appendix B). The paper then reports the best of three runs for each model (§3.1). This protocol does not support claims about model capability: GPT-5.5 bankrupts in 2 of 3 runs with mean survival 333.7±229.7 days, yet the text says it 'finishes above $1M.' Because the three runs are not independent environment realizations, the results provide no estimate of seed-to-seed variation. Please evaluate on multiple independent seeds, report full distributions, and tune the baseline on a separate seed or per-seed so the comparison is out-of-sample.
  2. [Abstract vs. §3.2/Table 3] The abstract at the top of the file states 'Only Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 finish above the $1M starting balance, and all evaluated models remain below the rule-based baseline.' The full-text abstract, §3.2, Table 3, and Fig. 2 state instead that Claude Opus 4.8 ($27.8M) and GPT-5.5 ($21.3M) exceed the $15.76M baseline. These statements are mutually incompatible and use different model names. Since the comparison to the rule-based baseline is a central quantitative claim, the manuscript must be reconciled to a single consistent set of numbers and model names.
  3. [Table 3, §3.2] For surviving models, the paper reports only max final cash and mean survival days; per-run final cash and variance are absent. Claude Opus 4.8 has three surviving runs, but readers cannot tell whether $27.8M is typical or a single favorable outcome. Please report all run-level final cash values, together with mean, median, and interquartile range, for every model. Without this, the qualitative claims about which models 'finish above $1M' rest on selected maxima rather than robust summaries.
  4. [§2.2, Appendix A] The simulator mechanics are entirely hand-authored and uncalibrated to empirical data: the customer participation rule (Eq. 5), competitor catch-up model (Eqs. 34–36), and macro Ornstein–Uhlenbeck process (Eq. 37) are plausible but untested abstractions. The abstract and §1 claim agents operate 'in the same environment and the same challenges as a human CEO,' which overstates external validity. Please either temper the realism claims to 'a synthetic environment designed to exercise...' or provide validation or sensitivity analysis (e.g., perturbation of key parameters, calibration to real SaaS/business datasets) showing that the main conclusions are not artifacts of specific parameter choices.
minor comments (5)
  1. [Abstract] The top-of-file abstract uses model names (Claude Fable 5, GPT-5.6 Sol) that do not appear anywhere in the body; the full-text abstract and body use Claude Opus 4.8 and GPT-5.5. This appears to be a stale or erroneous version and must be fixed.
  2. [Throughout] The typeset name 'CEO-BE N C H' has inconsistent spacing; use 'CEO-Bench' consistently.
  3. [Fig. 1, §6.1] The figure shows 'Code / Blog / Trajectory' but no URLs are given in the text. Please provide explicit links to the code, trajectory viewer, and any released data for reproducibility.
  4. [Appendix D, Table 3] The 'Estimated final cash upper bound' is not a mathematical upper bound: it depends on a chosen configuration and an ad hoc friction factor F=0.49. The appendix does caution that this is an estimate, but the label in Table 3 is misleading. Rename to something like 'Illustrative headroom estimate' and de-emphasize it in the main text.
  5. [Fig. 12] The plots show means and standard deviations, but the number of runs behind each bar is small (three, with many runs ending early). State the sample sizes and note that early-bankrupt models contribute few observations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CEO-Bench is an empirical evaluation whose environment equations and tuned baselines are design/comparison inputs, not fitted predictions.

full rationale

I walked the paper's derivation chain. The central claim is empirical: most evaluated models finish below $1M or bankrupt, with only Claude Opus 4.8 and GPT-5.5 exceeding the starting balance on their best runs. The simulator equations (Eqs. 1-44) define the environment; they are not fitted to, nor derived from, the agent outcomes they measure. The rule-based baseline in Appendix B is tuned on seed 42, but it is a comparison heuristic, not a 'prediction' whose input forces the reported conclusions. The $2.2B upper-bound estimate in Appendix D is explicitly labeled an approximate headroom calculation, uses a chosen friction factor F=0.49, and is not load-bearing for the main empirical result, which is directly observable in Table 3 and Figure 2. The only self-citations (SWE-bench, tau-bench) appear as background related-work benchmarks and do not carry the argument. Internal inconsistencies between the abstract, Table 3, and Figure 2 (e.g., the abstract says all models remain below the rule-based baseline, while Table 3 shows Claude Opus 4.8 at $27.8M and GPT-5.5 at $21.3M against a $15.76M baseline) and the single-seed best-run selection are experimental-validity and reporting-robustness concerns, not circularity. No step in the paper reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The benchmark's central claim rests on a large set of hand-chosen simulation parameters and behavioral rules. The most consequential choices are the customer participation model, competitor adaptation dynamics, and the macro cycle, none of which are calibrated to real-world data. The rule-based baseline and the upper-bound estimate also involve fitting or hand-tuning, which affects the interpretation of the benchmark results.

free parameters (3)
  • Rule-based baseline configuration = Best of 24 grid-search configs (mid price book, S1-only target, heavy spend, $100K cash floor)
    Selected by grid search over a template on seed 42; performance is conditional on this search and may not generalize to other seeds.
  • Friction factor F = 0.49
    Applied to the pre-friction upper-bound subtotal in Appendix D to obtain a 'conservative' estimate; chosen by hand, not derived from any theory or data.
  • Upper-bound configuration parameters = T3 inference, T7 capacity, $40K/day dev, 3x targeted development, etc.
    Selected from a small grid in Appendix D to keep all groups retained in the upper-bound estimate; thus the $2.2B figure is not a rigorously proven upper bound.
assumptions (4)
  • domain assumption Customers decide subscription based on a price-quality participation rule (Eq. 5, Mussa-Rosen).
    The entire acquisition and retention mechanics rest on this behavior rule; the authors chose it as a microeconomics-motivated model, but it is an assumption about how simulated customers behave.
  • domain assumption Competitors raise customer expectations through a stationary plus adaptive process (Eq. 34-36).
    The non-stationary difficulty of the environment depends on this parameterized competitor model; the specific distributions and coefficients are chosen by the authors rather than derived from data.
  • domain assumption Macroeconomic conditions evolve as an Ornstein-Uhlenbeck process around a sinusoidal PMI (Eq. 37).
    The macro cycle is a synthetic stochastic process; the assumption that this captures real economic dynamics is unverified.
  • domain assumption Reputation and social media dynamics follow the weighting formula in Eq. 26.
    The simulator's word-of-mouth and public feedback signals are generated by a hand-specified model, not validated against real data.
invented entities (1)
  • NovaMind simulation
    purpose: To evaluate agent performance on a simulated startup management task; provides the environment in CEO-Bench.
    It is a fictional company and market created by the authors; there is no external data validating the simulation's realism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CEO-Bench: Can Agents Play the Long Game?." pith.science (2026). https://pith.science/paper/PPA4QS5E

@misc{pith2026260618543,
  author       = {Pith},
  title        = {Pith review of: CEO-Bench: Can Agents Play the Long Game?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPA4QS5E}},
  note         = {Machine review of arXiv:2606.18543}
}
abstract

Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that forecasts churn regimes, billing timing, customer losses, and future cash under different scenarios. Even so, most state-of-the-art models struggle in this environment. Only Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 finish above the $1M starting balance, and all evaluated models remain below the rule-based baseline. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.

Figures

Figures reproduced from arXiv: 2606.18543 by the authors.

Figure 1
Figure 1. CEO-BENC H evaluates general long-horizon agent capabilities by simulating a startup over 500 days in a realistic and challenging environment. The agent operates through a programmable interface with access to business databases, company management tools, and social media. Outcomes are driven by a partially observable, noisy, and evolving market with delayed and coupled consequences. Abstract Language model agents a… view at source ↗
Figure 2
Figure 2. Cash on hand over time for each model’s best run [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Cash on hand over time for each of the three runs per model. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: Major design principles behind CEO-BENC H’s world mechanics and example designs that follow the principles. factor in Equation 3. We describe full details of each factor in Appendix A.3, A.4, and A.6. Q perc i,t | {z } quality perceived by customer i = mp |{z} model-ti…
Figure 6
Figure 6. Figure 6: Agents interact with CEO-BENC H through a versatile Python interface. Left: We give the agent access to diverse business databases to test its information acquisition capability through a realistic data analytics workflow. Middle: We widen agents’ opportunity space by …
Figure 7
Figure 7. Figure 7: Example memos written by Claude Opus 4.8 (top), GPT-5.5 (middle), and Claude Opus 4.7 (bottom) in their [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Example code files written by top-performing agents during their best trajectories. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Number of customers by customer group over time for the best runs of Claude Opus 4.8 and GPT-5.5. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Targeted development spending breakdown. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Average per-week tool usage frequency for the best runs of Claude Opus 4.8, GPT-5.5, and Claude Opus 4.7 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Better-performing models excel along four skill axes [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Examples of planning in GPT-5.5 and Claude Opus 4.8 memos. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Ablating simulator configurations. (a) Weaker or absent competitors make the task substantially easier. (b) Shortening the horizon to 50 days results in most models still unable to make profits. 4.2 Ablating Time-Horizon We examine whether agents behave differently wh…
Figure 15
Figure 15. Figure 15: Cash trajectories and action frequency when ablating agent harnesses. [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Example customer participation curves. Each curve represents the average participation behavior of a customer group. Each curve maps the offered monthly price C to the minimum accepted quality Q req i (C) for a customer with different willingness to pay, quality floor…
Figure 17
Figure 17. Figure 17: GPT-5.5 cash trajectories in Vending-Bench 2 and [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Business Arena, a data-grounded B2B marketplace simulator, ranks 15 LLM agents from $20,856 to $188,488 mean final net worth, well below expert-designed strategies that reach $436,195.

  2. Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?

    cs.AI 2026-08 conditional novelty 7.0 of 10

    C-SUITEBENCH shows that adding visual business evidence improves evidence-centric reasoning in nine multimodal LLMs but degrades constrained resource allocation in all nine, a pattern attributed to signal crowding.

Reference graph

Works this paper leans on

51 extracted references · 6 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Claude code overview

    Anthropic . Claude code overview. https://code.claude.com/docs/en/overview, 2026

    2026

  2. [2]

    Vending-bench: A benchmark for long-term coherence of autonomous agents

    Axel Backlund and Lukas Petersson. Vending-bench: A benchmark for long-term coherence of autonomous agents. arXiv preprint arXiv:2502.15840, 2025 a

    arXiv 2025

  3. [3]

    Vending-bench 2

    Axel Backlund and Lukas Petersson. Vending-bench 2. https://andonlabs.com/evals/vending-bench-2, 2025 b

    2025

  4. [4]

    LongBench : A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench : A bilingual, multitask benchmark for long context understanding. In ACL, 2024

    2024

  5. [5]

    MLE-bench : Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. MLE-bench : Evaluating machine learning agents on machine learning engineering. In ICLR, 2025

    2025

  6. [6]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [7]

    Laradji, Manuel Del Verme, Tom Marty, L \'e o Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, L \'e o Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena : How capable are web agents at solving common knowledge work tasks? In ICML, 2024

    2024

  8. [8]

    YC-Bench : Benchmarking AI agents for long-term planning and consistent execution

    Muyu He, Adit Jain, Anand Kumar, Vincent Tu, Soumyadeep Bakshi, Sachin Patro, and Nazneen Rajani. YC-Bench : Benchmarking AI agents for long-term planning and consistent execution. arXiv preprint arXiv:2604.01212, 2026 a

    arXiv 2026

Show all 51 references
  1. [9]

    MemoryArena : Benchmarking agent memory in interdependent multi-session agentic tasks

    Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland. MemoryArena : Benchmarking agent memory in interdependent multi-session agentic tasks. arXiv preprin...

    2026

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021

    2021

  3. [11]

    RULER : What's the real context size of your long-context language models? In COLM, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER : What's the real context size of your long-context language models? In COLM, 2024

    2024

  4. [12]

    Evaluating memory in LLM agents via incremental multi-turn interactions

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. In ICLR, 2026

    2026

  5. [13]

    SWE-bench : Can language models resolve real-world GitHub issues? In ICLR, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench : Can language models resolve real-world GitHub issues? In ICLR, 2024

    2024

  6. [14]

    Prospect theory: An analysis of decision under risk

    Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 47 0 (2): 0 263--291, 1979. doi:10.2307/1914185

    1979 doi

  7. [15]

    Evan F. Koenig. Using the purchasing managers' index to assess the economy's strength and the likely direction of monetary policy. Federal Reserve Bank of Dallas Economic and Financial Policy Review, 1 0 (6), 2002. URL https://fraser.stlouisfed.org/files/docs/publications/frbd...

    2002

  8. [16]

    In-context reinforcement learning with algorithm distillation

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillatio...

    2023

  9. [17]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. TMLR, 2023

    2023

  10. [18]

    AgentBench : Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench ...

    2024

  11. [19]

    Fung, Chun Yuan, and Li Shen

    Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, Zixuan Hu, Hongze Mi, Yibo Wang, Naiqiang Tan, Hong Chen, Yi R. Fung, Chun Yuan, and Li Shen. UltraHorizon : Benchmarking agent capabilities in ultra ...

    2025

  12. [20]

    AgentBoard : An analytical evaluation board of multi-turn LLM agents

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard : An analytical evaluation board of multi-turn LLM agents. In NeurIPS, 2024

    2024

  13. [21]

    James G. March. Exploration and exploitation in organizational learning. Organization Science, 2 0 (1): 0 71--87, 1991. doi:10.1287/orsc.2.1.71

    1991 doi

  14. [22]

    GAIA : A benchmark for general AI assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA : A benchmark for general AI assistants. In ICLR, 2024

    2024

  15. [23]

    SWE-Lancer : Can frontier LLMs earn \ 1 million from real-world freelance software engineering? In ICML, 2025

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer : Can frontier LLMs earn \ 1 million from real-world freelance software engineering? In ICML, 2025

    2025

  16. [24]

    LLMs are in-context bandit reinforcement learners

    Giovanni Monea, Antoine Bosselut, Kiant \'e Brantley, and Yoav Artzi. LLMs are in-context bandit reinforcement learners. arXiv preprint arXiv:2410.05362, 2024

    2024

  17. [25]

    Monopoly and product quality

    Michael Mussa and Sherwin Rosen. Monopoly and product quality. Journal of Economic Theory, 18 0 (2): 0 301--317, 1978

    1978

  18. [26]

    Allen Newell and Herbert A. Simon. Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ, 1972. ISBN 0-13-445403-0

    1972

  19. [27]

    Richard L. Oliver. A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research, 17 0 (4): 0 460--469, 1980. doi:10.1177/002224378001700405

    1980 doi

  20. [28]

    Codex cli

    OpenAI . Codex cli. https://developers.openai.com/codex/cli, 2026

    2026

  21. [29]

    Opencode: The open source ai coding agent

    OpenCode . Opencode: The open source ai coding agent. https://opencode.ai/, 2026

    2026

  22. [30]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In ICML, 2025

    2025

  23. [31]

    Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, Jerry Tworek, et al

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Sim \'o n Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, A...

    2025 arXiv

  24. [32]

    AccountingBench : Evaluating LLMs on real long-horizon business tasks

    Penrose AI . AccountingBench : Evaluating LLMs on real long-horizon business tasks. https://accounting.penrose.com/, 2025

    2025

  25. [33]

    Pi documentation

    Pi Contributors . Pi documentation. https://pi.dev/docs/latest, 2026

    2026

  26. [34]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof Q&A benchmark. In COLM, 2024

    2024

  27. [35]

    Herbert A. Simon. A behavioral model of rational choice. The Quarterly Journal of Economics, 69 0 (1): 0 99--118, 1955. doi:10.2307/1884852

    1955 doi

  28. [36]

    Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR, 2023

    2023

  29. [37]

    PaperBench : Evaluating AI 's ability to replicate AI research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench : Evaluating AI 's ability to replicate AI research. In ICML, 202...

    2025 arXiv

  30. [38]

    Testing for mean reversion in processes of Ornstein--Uhlenbeck type

    Alexander Szimayer and Ross Maller. Testing for mean reversion in processes of Ornstein--Uhlenbeck type. Statistical Inference for Stochastic Processes, 7: 0 95--113, 2004. doi:10.1023/B:SISP.0000026032.80363.59

    2004

  31. [39]

    Teece, Gary Pisano, and Amy Shuen

    David J. Teece, Gary Pisano, and Amy Shuen. Dynamic capabilities and strategic management. Strategic Management Journal, 18 0 (7): 0 509--533, 1997. doi:10.1002/(SICI)1097-0266(199708)18:7<509::AID-SMJ882>3.0.CO;2-Z

    1997 doi

  32. [40]

    AppWorld : A controllable world of apps and people for benchmarking interactive coding agents

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld : A controllable world of apps and people for benchmarking interactive coding agents. In ACL, 2024

    2024

  33. [41]

    Uhlenbeck and Leonard S

    George E. Uhlenbeck and Leonard S. Ornstein. On the theory of the Brownian motion. Physical Review, 36 0 (5): 0 823--841, 1930. doi:10.1103/PhysRev.36.823

    1930 doi

  34. [42]

    OdysseyBench : Evaluating LLM agents on long-horizon complex office application workflows

    Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor R \"u hle, and Saravan Rajmohan. OdysseyBench : Evaluating LLM agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09124, 2025

    2025 arXiv

  35. [43]

    TRACE : A comprehensive benchmark for continual learning in large language models

    Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, and Xuanjing Huang. TRACE : A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023

    2023 arXiv

  36. [44]

    LongMemEval : Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval : Benchmarking chat assistants on long-term interactive memory. In ICLR, 2025

    2025

  37. [45]

    Mitchell, and Yuanzhi Li

    Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. SmartPlay : A benchmark for LLMs as intelligent agents. In ICLR, 2024

    2024

  38. [46]

    TravelPlanner : A benchmark for real-world planning with language agents

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. TravelPlanner : A benchmark for real-world planning with language agents. In ICML, 2024 a

    2024

  39. [47]

    OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurIPS, 2024 b

    2024

  40. [48]

    Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. TheAgentCompany : Benchmarking LLM agents on consequential real world tasks. In NeurIPS, 2025

    2025

  41. [49]

    -bench: A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench: A benchmark for tool-agent-user interaction in real-world domains. In ICLR, 2025

    2025

  42. [50]

    AssistantBench : Can web agents solve realistic and time-consuming tasks? In EMNLP, 2024

    Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench : Can web agents solve realistic and time-consuming tasks? In EMNLP, 2024

    2024

  43. [51]

    WebArena : A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena : A realistic web environment for building autonomous agents. In ICLR, 2024

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.