Pith.

Computation and Language · arXiv:2611.05147 v1 · submitted 3 Nov 2026 · 23 references

GM League Arena: An Interactive Long-Horizon Task Benchmark Based on Professional Sports League Management

Junkai Zhou, Ziming Pan, Zhaoyi Zhang

The paper says

“Long-horizon decision-making under sustained competition requires large language models (LLMs) to balance immediate investment with future resource needs.”

more of the abstract

However, local success in multi-step interactions can not adequately capture cumulative decision effects or demonstrate long-term goal achievement under resource constraints and sustained competition. To address this gap, we introduce GM Bench, a benchmark for managing professional sports teams across seasons under resource constraints and competition among teams. We build the environment simulators for four sports and add financial constraints, player demands, and market mechanisms to establish a unified task across sports. Diverse rosters of comparable strength are constructed from real-world player data, and each LLM is evaluated as a team general manager to assess its long-term management capabilities. We evaluate competitive performance, cash reserves, and cross-season improvement across four sports. Over five NBA seasons, Claude Opus 5.5 ranks first among 29 LLMs, while GPT-6 Astra and GPT-6.1 Sol combine strong performance with larger cash reserves.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 4 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Does not hold up as stated. The benchmark design is a real extension, but the headline rankings are unsupported: the win/loss tables don't add up, and the evaluation rests on one unreplicated run per model. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 23 references say.

The paper says, at Appendix C.3 / Tables 1 and 3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.85):

“The tables do not report confidence intervals across independent full-horizon runs. 13”

Pith says: Objection 1 of 4 · major

Appendix C.3 states explicitly that 'each reported full-horizon model evaluation consists of one complete five-season run' and that 'the tables do not report confidence intervals across independent full-horizon runs' ([b221], [b227]). Section 3.1 makes the simulator stochastic (injuries, random events, game outcomes).

more of objection 1

Consequently the headline rankings in Tables 1 and 3 are point estimates from a single trajectory per model per sport, while Sections 6.2 and 6.4 treat rank separations and cross-sport top-seven stability as measured facts ('The six highest-ranked NBA models remain in the top seven in all three additional sports'). As the authors acknowledge, disclosure does not provide a variance estimate; without seed-to-seed variability, Elo differences of order 10-100 points and adjacent rank separations cannot be distinguished from realized luck. This is a load-bearing gap for the central empirical claim.

The paper says, at Tables 1 and 2 / Section 5.2

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The reported counts are internally inconsistent. In a 30-team NBA league, five regular seasons contain at most 5 times 30 times 82 divided by 2, that is, 6150 total regular-season wins, yet the wins column in Table 1 sums to 6568, and Opus alone is listed with 446 wins, exceeding the 410-game regular-season maximum for one team across five NBA seasons.

more of objection 2

Additionally, MiniMax M3 is listed with 193 regular-season wins in Table 1, but its season-by-season wins in Table 2 (39, 38, 38, 38, 41) sum to 194. These discrepancies make the headline data not internally checkable and undermine the reported rankings and the cash-versus-competitive comparisons in Section 6.3.

The paper says, at Section 5.2, Eq. (6)

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

Equation (6) defines Gi as the mean win rate of the last w seasons minus the mean win rate of the first w seasons, which is consistent with the Table 2 caption. However, Section 6.1 and the caption describe the seasons as spanning 2022-2026, while Table 2's columns are labeled 22, 23, 24, 25, 26.

more of objection 3

Please clarify whether the five seasons are calendar years or the 2022-23 through 2025-26 seasons, and ensure Equation (6) matches the definition actually used to compute the printed Gi values.

The paper says, at Section 6.4 / Table 3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.68):

“The reported full-horizon results remain single-run measurements; short-horizon tests are not counted as independent five-season replications.”

Pith says: Objection 4 of 4 · major

The claim that 'the six highest-ranked NBA models remain in the top seven in all three additional sports' is supported by the printed tables: in baseball the six NBA leaders are within the top seven, and in hockey and American football the correspondence also holds.

more of objection 4

However, because each sport's table reports only Elo and cash, and because the single-run limitation from Appendix C.3 applies to all four sports, the cross-sport stability claim inherits the unreplicated-measurement problem and cannot support the strength of the conclusion as currently phrased.

5 minor objections

The paper says, at Abstract, Section 6.1, Appendix B

Pith says: minor objection 1 of 5

The abstract says '29 LLMs' for the NBA, Section 6.1 says '31 LLMs from 12 families', and Appendix B states 29 shared models plus two additional models for football and hockey; the counts should be made consistent across these locations.

The paper says, at Table 3

Pith says: minor objection 2 of 5

The caption should state explicitly that the leading integer column is the rank and how ties or near-equal Elo values are ordered.

show 3 more

The paper says, at Section 5.2, Eq. (4)

Pith says: minor objection 3 of 5

The Elo update accounts for draws, but the definition of v_i,s in Eq. (6) excludes draws; please state explicitly how draws are handled in the season win-rate used for Gi, since hockey permits draws.

The paper says, at References

Pith says: minor objection 4 of 5

Several related works are cited as arXiv preprints without arXiv identifiers or version numbers; including identifiers would improve verifiability.

The paper says, at Figure 1

Pith says: minor objection 5 of 5

Figure 1 illustrates strategy trajectories but the mapping from the five strategy labels to the five roster types in Section 5.1 is not explicit; a sentence connecting them would help.

What would settle it

Run 5–10 independent five-season NBA replications for the six leading models (Claude Opus 5.5, GPT-6 Astra, GPT-6.1 Sol, Claude Sonnet 5.5, Claude Fable 5.1, Grok 4.7), keeping prompts, APIs, and initial rosters fixed and varying only the simulator seed. Compute per-model distributions of final Elo, terminal cash, championships, rank, and cross-sport top-seven consistency. If 95% rank intervals overlap by more than a few positions, or if the cross-sport pattern disappears in a majority of seeds, the single-run rankings in Tables 1–3 are not stable measurements of long-horizon management capability. The same replication logs should also be used to reconcile the Table 1 and Table 2 win columns.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 4 Nov 2026

  1. Reader does not hold up confidence high
  2. Skeptic agrees 2 min 32 s · 15,271 tokens read, 19,583 written
  3. Editor wrote the verdict above 55 s · 16,136 tokens read, 5,730 written
  4. Referee major revision: 4 major objections, 5 minor 25 s · 15,633 tokens read, 3,181 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This one is worth knowing about because it fills a real gap in the benchmark lineup, but the evaluation section is not fit for reporting rankings. The four-sport unified management environment—same GM loop across NBA, NFL, NHL, and MLB, with real player data, five roster archetypes, salary cap and luxury tax, and LLM-driven player and agent NPCs—is a genuine extension beyond FM-Bench and FromPitch2Board. The environment concept is useful; the problem is the results.

The headline tables don't add up. Table 1's regular-season wins sum to 6,568 across 30 NBA teams. Five 82-game seasons can produce at most 6,150 regular-season wins. That is not a scheduling nuance. Worse, Opus's Table 1 wins (446) don't match the sum of its Table 2 season wins (313); MiniMax M3 similarly has 193 in Table 1 and 194 in Table 2. With the season-by-season table right there, this should have been caught before submission. Because the paper's central contribution is the ranking and the cross-sport stability claim, this alone undermines the empirical section.

Second, Appendix C.3 says each full-horizon result is one five-season run and no confidence intervals are reported. The authors disclose this plainly, and I credit them for that. But disclosure doesn't make a single noisy trajectory an estimate. The environment is stochastic—injuries, random events, game outcomes—so without seed-to-seed variability we cannot tell whether Opus's three championships and the claim that the six leading NBA models stay in the top seven reflect management ability or one roll of the dice. The burden is on the authors to rerun.

Third, no code or data artifacts are released. For a benchmark paper, that is not a minor omission. The environment is the deliverable; readers need to inspect and run it.

What's good: the task design is coherent, the financial constraints and inter-season carryover are sensible, and the related work is accurate about what FM-Bench and FromPitch2Board do. The limitations section is honest. This paper would be useful to the LLM-agent community if these issues were fixed.

Recommendation: reject as submitted. I would not spend a full referee cycle on this version. But I'd encourage the authors to release the simulator and prompt/trajectory data, fix the win/loss aggregation, and rerun with several seeds; with those changes it becomes a serious benchmark paper and should go to review as a new submission.

The strongest objection

Appendix C.3 states that each reported full-horizon result is a single five-season run, that short-horizon tests are not independent replications, and that no confidence intervals are reported ([b221], [b227]). Section 3.1 makes the environment stochastic (injuries, random events, game outcomes), so a single trajectory is a realization, not an estimate. The central claim—Claude Opus 5.5 first with three championships, and the six leading NBA models staying in the top seven in all three additional sports—compares point estimates from one realization per model per sport. The authors disclose this limitation, but disclosure does not remove the load-bearing gap: without a variance estimate, Elo differences and rank separations in Tables 1–3 cannot be distinguished from realized luck. A secondary audit problem compounds this: Table 1's win total is 6,568, which exceeds the 6,150 regular-season wins possible in a 30-team, five-season NBA schedule, and MiniMax M3's Table 2 regular-season wins (194) exceed its Table 1 wins (193), so the reported data are not internally checkable. The burden is therefore on replication, not on the reader.

The reader and I identify the same load-bearing assumption: the reported rankings are treated as estimates of model capability, but each full-horizon result is a single stochastic trajectory. The paper is explicit about this in Appendix C.3 and in the Limitations, so this is not a concealed flaw; it is an unresolved reliability gap. The environment itself is described as stochastic (Section 3.1), and the conclusions draw precise comparative claims (first place, three championships, top-seven consistency across sports) from unreplicated runs. The concrete multi-seed test would settle whether adjacent ranks and cross-sport consistency are stable properties or artifacts of one seed. I also note the Table 1/Table 2 win-count discrepancy as a supporting audit problem, but I do not treat it as the primary objection. Because this concern fully supports the reader's REJECT verdict, no adjustment is needed.

The referee's report

If its claims held, the benchmark would be a useful contribution to long-horizon agent evaluation. The task design is a genuine strength: it combines multi-sport institutional variation, interactive player/agent roles, real-world data initialization, and outcome metrics that jointly track competitive results, terminal cash, and seasonal trajectories. The evaluation protocol is transparently specified (Elo with K=20, Eq. 4; improvement window w=2, Eq. 6). The paper also documents its own data-collection scale and states its limitations clearly, including the absence of replication.

Pith's reading of the paper (a machine's summary, not the authors')

GM Bench asks whether large language models can sustain a plan across five simulated seasons as a professional sports general manager, under salary caps, luxury taxes, player demands, trades, and rival teams that keep improving. The paper's claim is that this setting measures long-horizon decision-making that short multi-step benchmarks miss, because today's trades and signings show up only later as depleted cash, lost flexibility, or championships. Over the NBA season, Elo ratings spread from about 1071 to 1977 as management choices compound, and final-season win totals range from 8 to 76. The headline result is a ranking of 29 LLMs, led by Claude Opus 5.5 with three championships, with the six best NBA managers staying in the top seven when the same task moves to American football, ice hockey, and baseball. Why it matters: if the ranking holds, the benchmark gives a direct comparison of which models can turn short-term interactions into sustained competitive results while keeping future resources intact.

What carries the argument. The load-bearing mechanism is the five-season management loop itself, formalized as a partially observable stochastic game $G = (I, S, A, P, O, U, H)$ in which every GM action persists into later observations, so costs and rewards are separated in time. It is carried by three design choices: five roster archetypes (Balanced, Star, Twins, Offense, Defense) built from real player ratings with comparable starting strength, so strategy rather than initial talent decides outcomes; economic constraints (salary cap, luxury tax, cash carried across seasons) that force intertemporal trade-offs; and a common Elo protocol with $K = 20$, ratings reset to 1500 each season, that converts game results into a comparable competitive score. The environment mixes LLM GMs with LLM-driven players and agents who negotiate and leak trade rumors, so market signals and outcomes co-evolve. The paper's metrics, mean season-end Elo, terminal cash, and the improvement gap $G_i$ between early and late win rates, are what make the claim that cumulative effects are observable testable.

Core claim. The central discovery, on the paper's own terms, is that a unified multi-season management task can make cumulative decision effects observable and rank models by them. The paper builds GM Bench as a partially observable stochastic game in which each evaluated LLM acts as a general manager, drafting, signing, trading, and renewing contracts, while players and agents driven by a shared fixed LLM express demands and generate market rumors; a simulator enforces league rules and resolves games. Across five seasons of NBA play, initially comparable rosters diverge sharply: Elo spans 1071.0 to 1977.3, and teams that start with 35-47 regular-season wins end anywhere from 8 to 76. Claude Opus 5.5 ranks first with three championships, followed by GPT-6 Astra and GPT-6.1 Sol, and the paper reports that the six leading NBA models remain within the top seven in American football, ice hockey, and baseball. The paper further contrasts the joint criteria of competitive return and resource retention, showing for example that Opus spends its cash advantage on stars, GPT-6 models keep larger buffers, and Sonnet's championship run is followed by unaffordable renewal demands.

The premise it rests on. The paper treats each model's single five-season run in a stochastic simulator as a representative measurement of that model's ability, even though the paper itself reports no repeated runs and no confidence intervals.

If the paper is right (Pith's inferences, not the authors')

If the rankings hold, GM Bench gives the field a direct comparison of which models sustain a plan across many interacting decisions, rather than only winning short single-task rounds.

The joint reporting of Elo and terminal cash turns 'winning while going broke' and 'hoarding cash while losing' into detectable failure modes, so resource retention becomes part of the capability being measured.

The cross-sport consistency claim (six leading NBA models in the top seven everywhere) implies that the measured ability transfers across rule sets, not just within one simulator.

The widening win spreads across seasons (35-47 in year one, 8-76 in year five) mean the task becomes more discriminating the longer it runs, supporting five-season horizons over shorter evaluations.

The scripted Native GM baseline ranking 14th of 30 in the NBA gives a floor: stronger-than-rule-based management is now a documented property of the better models.

A direct test follows from the paper's own limitation: because each five-season result is a single run with no confidence intervals, re-running the top models with several random seeds would determine whether the headline ordering is measurement or draw; that replication is the natural next experiment.

The design suggests a scaled-down screening protocol: short-horizon seeded tests could serve as cheap filters, with full five-season runs reserved for the few models that pass, trading the paper's roughly USD 15,000 API budget for more statistical power.

Pairing identical seeded leagues across models, a control the paper does not use, would isolate management skill from the shared stochastic environment and could also expose whether one model's early losses simply hand favorable draft positions to another.

The same loop of delayed feedback, persistent resources, and negotiating partners applies to corporate planning or supply-chain roles, so the four-sport 'institutional adaptation' result is evidence for a broader claim about transferring long-horizon skill rather than just a sports result.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Long-horizon decision-making under sustained competition requires large language models (LLMs) to balance immediate investment with future resource needs. However, local success in multi-step interactions can not adequately capture cumulative decision effects or demonstrate long-term goal achievement under resource constraints and sustained competition. To address this gap, we introduce GM Bench, a benchmark for managing professional sports teams across seasons under resource constraints and competition among teams. We build the environment simulators for four sports and add financial constraints, player demands, and market mechanisms to establish a unified task across sports. Diverse rosters of comparable strength are constructed from real-world player data, and each LLM is evaluated as a team general manager to assess its long-term management capabilities. We evaluate competitive performance, cash reserves, and cross-season improvement across four sports. Over five NBA seasons, Claude Opus 5.5 ranks first among 29 LLMs, while GPT-6 Astra and GPT-6.1 Sol combine strong performance with larger cash reserves.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-04

The authors' main claim, as the classifier picked it out 0.96

“Over five NBA seasons, Claude Opus 5.5 ranks first among 29 LLMs, while GPT-6 Astra and GPT-6.1 Sol combine strong performance with larger cash reserves.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.16 · human participants 0.05 · controlled comparison 0.18 · randomized assignment 0.02 · states quantitative result 0.97 · compares with prior results 0.68 · mentions code or data availability 0.06

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Long-horizon decision-making under sustained competition requires large language models (LLMs) to balance immediate investment with future resource needs. background 1.00 0.06 0.05 asserted 1.00
However, local success in multi-step interactions can not adequately capture cumulative decision effects or demonstrate long-term goal achievement under resource constraints and sustained competition. gap 0.99 0.08 0.05 asserted 0.98
To address this gap, we introduce GM Bench, a benchmark for managing professional sports teams across seasons under resource constraints and competition among teams. contribution 1.00 0.90 0.09 asserted 0.99
We build the environment simulators for four sports and add financial constraints, player demands, and market mechanisms to establish a unified task across sports. method 0.95 0.95 0.03 asserted 1.00
Diverse rosters of comparable strength are constructed from real-world player data, and each LLM is evaluated as a team general manager to assess its long-term management capabilities. method 0.97 0.91 0.05 asserted 0.99
We evaluate competitive performance, cash reserves, and cross-season improvement across four sports. method 0.82 0.93 0.07 asserted 0.93
Over five NBA seasons, Claude Opus 5.5 ranks first among 29 LLMs, while GPT-6 Astra and GPT-6.1 Sol combine strong performance with larger cash reserves. result 1.00 0.93 0.54 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.60

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-04: 1 request, 5,559 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Illustrative performance trajectories of four team management strategies over five seasons show that early competitive gains can diminish, whereas rebuilding and sustained investment can yield delayed benefits. gains with larger cash reserves. Grok 4.7 achieves stronger competitive returns with less remaining cash, whereas Kimi K3 combines steady improvement with a larger financial buffer. MiMo-V2.6-Pro retains substantial cash despite declining performance. Sonnet faces unaffordable renewals after an NBA championship run. These cases highlight the need to coordinate acquisition,…
Figure 2 of the paper
Figure 2. Overview of GM Bench, illustrated with the NBA environment and extended to hockey, baseball, and American football. LLM general managers interact with players and their agents within a shared simulator that enforces league and financial rules, while evaluation tracks competitive performance, resources and finances, and cross-season improvement. 3 NBA Environment The NBA instance draws inspiration from ZenGM1 , and models a NBA environment with league rules, LLM-based management, financial constraints, and role interactions. 3.1 League Foundations Player Data and State. Initial…
Figure 3 of the paper
Figure 3. Schematic profiles of the five team types. Balanced emphasizes an evenly distributed roster; Star centers on one standout player; Twins features two leading players; Offense prioritizes offensive ability; and Defense emphasizes defensive ability. The axes represent roster balance (BL), single-star contrast (SC), dual-core advantage (DC), offensive index (OI), and defensive index (DI). Larger radial values indicate stronger emphasis on the corresponding characteristic. The final panel overlays all five profiles for comparison. capturing cumulative decision returns and future…

Figures are the authors', reproduced from the paper.

References (23)
  1. [1] Axel Backlund and Lukas Petersson. 2025. Vending-bench: A benchmark for long-term coherence of autonomous agents.arXiv preprint arXiv:2502.15840 on PithPith's reviewarXiv
  2. [2] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2-bench: Evaluating conversational agents in a dual-control environment.Preprint, arXiv:2506.07982 on PithPith's reviewarXiv
  3. [3] Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al on Pith
  4. [4] De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, et al. 2024. The browsergym ecosystem for web agent research.arXiv preprint arXiv:2412.05467 on PithPith's reviewarXiv
  5. [5] Arpad E Elo. 1978. The rating of chessplayers, past and present.(No Title) on Pith
  6. [6] Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang, Baohua Dong, Yangqiu Song, et al. 2026. E-commerce bench: Evaluating llm agents on long-horizon autonomous business operation.arXiv preprint arXiv:2608.30730 on PithPith's reviewarXiv
  7. [7] Yi Han, Yan Wang, Lingfei Qian, Haohang Li, Yupeng Cao, Yueru He, Xueqing Peng, Nanhan Shen, Yitao Xu, Yankai Chen, et al. 2026. Can llm agents be cfos? benchmarking long-horizon resource allocation in an uncertain enterprise environment. arXiv preprint arXiv:2603.23638 on PithPith's reviewarXiv
  8. [8] Xavier Hu, Jinxiang Xia, Shengze Xu, Kangqi Song, Yishuo Yuan, Guibin Zhang, JinCheng Ren, Boyu Feng, Li Lu, Tieyong Zeng, et al. 2026. Ecogym: Evaluating llms for long-horizon plan-and-execute in interactive economies.arXiv preprint arXiv:2602.09514 on PithPith's reviewarXiv
  9. [9] Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He on Pith
  10. [10] Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth V ora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2025. Discoverybench: Towards data-driven discovery with large language models. InInternational Co... on Pith
  11. [11] Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wołczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci´nski, Lerrel Pinto, Rob Fergus, et al. 2025. Balrog: Benchmarking agentic llm and vlm reasoning on games. In International Conference on Learning Re... on Pith
  12. [12] Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, and Takashi Ishida on Pith
  13. [13] Megh Thakkar, Nicolas Chapados, Christopher Pal, et al. 2025. Webarena verified: Reliable evaluation for web agents. In Workshop on Scaling Environments for Agents on Pith
  14. [14] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annu... on Pith
  15. [15] Tianyou Wang, Chongyang Gao, Kezhen Chen, Dong Chen, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, and Chi Li. 2026. Fm-bench: A benchmark for long-horizon management with competing agents.arXiv preprint arXiv:2608.18423 on PithPith's reviewarXiv
  16. [16] Frank Fangzheng Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. 2026. Theagentcompany: benchmarking llm agents on consequential real world tasks.Advances in Neural Information Processing Systems, 38 on Pith
  17. [17] Peiyu Zang. 2026. Frompitch2board: Benchmarking llm agents in long-horizon football management.arXiv preprint arXiv:2609.34710 on PithPith's reviewarXiv
  18. [18] Yushuo Zheng, Huiyu Duan, Zicheng Zhang, Yucheng Zhu, Xiongkuo Min, and Guangtao Zhai. 2026. Market-bench: Benchmarking large language models on economic and trade competition. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (V olume 1... on Pith
  19. [19] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2024. Webarena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, volume 202... on Pith
  20. [20] Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Daisy Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. 2025. Multiagentbench: Evaluating the collaboration and competition of llm agents. InProceedings of the 63rd Annual Meeting of the Assoc... on Pith
show 3 more
  1. [2024] Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362 on Pith
  2. [2025] Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. InInternational Conference on Learning Representations, volume 2025, pages 96934–96990 on Pith
  3. [2026] Coffeebench: Benchmarking long-horizon llm agents in heterogeneous multi-agent economies.arXiv preprint arXiv:2606.16613 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment