Pith. sign in

REVIEW 4 major objections 4 minor 50 references

A plan's apparent quality drops when the responder searches harder: best-of-eight response cuts BLUE advantage from .518 to .486 in AdvPlan-Bench.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:10 UTC pith:O4NZPGPI

load-bearing objection A well-scoped, honest synthetic benchmark; the headline numbers are mostly sanity checks, but the evaluation object itself is a real contribution. the 4 major comments →

arxiv 2608.00832 v1 pith:O4NZPGPI submitted 2026-08-01 cs.LG

AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents

Alina Kapanova , Arun Kanhai , Natan Vidra , Spurthi Setty This is my paper
classification cs.LG
keywords adversarial evaluationplan generationbenchmarkself-playresponse budgetcandidate frontiermulti-agent councilconstraint coherence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AdvPlan-Bench argues that structured plan-generation agents should not be scored as if a plan has quality in isolation. It builds a small offline benchmark in which each generated plan is judged against a responding plan that can sample many counter-plans and pick the strongest, and it asks how the original plan's apparent quality changes as the responder searches harder. Under its own synthetic metrics, the answer is clear: a single-sample response gives BLUE a .518 advantage and .900 win rate, but a best-of-eight response lowers those to .486 and .820, with the curve continuing to .480 at sixteen samples. The paper's contribution is methodological—an evaluation object made of typed plans, adversarial response sets, and inspectable traces—not an operational planner, and it explicitly says so.

Core claim

The paper's central claim is that adversarial plan generation can be turned into a reproducible evaluation object, and that doing so changes conclusions that static scoring would support. Defining each instance as a scenario, a BLUE plan, a sampled RED response set, an advantage signal, and constraint diagnostics, the benchmark measures how BLUE robustness degrades as RED's response budget grows. The headline empirical finding is that sampled best response is a tougher opponent: BLUE advantage falls from .518 to .486 and win rate from .900 to .820 when RED draws eight candidates instead of one, and to .480 at sixteen candidates. The paper further shows that selection objectives matter—a cons

What carries the argument

The load-bearing object is the evaluation tuple (s, c_B, C_R, U, D): a scenario, a BLUE plan, a set of RED counter-plans, an advantage signal, and a diagnostic vector, all written to per-scenario trace files. The mechanism that carries the argument is sampled best response: RED draws k candidates and selects the one that maximizes the adversarial score, so the benchmark can sweep the response budget k and observe a robustness curve. Around that mechanism, the benchmark wraps two selection objectives (quality-greedy vs. constraint-aware), a candidate-frontier analysis (diversity, quality spread, Pareto count), and a council loop of propose, stress-test, critique, repair, and adjudicate. The t

Load-bearing premise

The load-bearing premise is that the synthetic quality score (0.5·effectiveness − 0.3·cost − 0.2·risk) and the heuristic constraint rubric are acceptable stand-ins for real plan quality; the paper itself flags metric misspecification as its main internal-validity risk.

What would settle it

Re-run the 150-scenario policy comparison with three expert human judges scoring plan quality instead of the synthetic formula, and check whether the response-budget effect (BLUE advantage .518 at k=1 to .480 at k=16) still appears. If expert scores are uncorrelated with the synthetic score, or if the effect flips sign under expert adjudication, the benchmark's headline result is an artifact of its own scoring rule.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the benchmark's proxies are accepted, then any claim that a plan-generation policy is 'strong' must state the opponent's response budget; otherwise the claim is incomplete.
  • Apparent plan robustness is not a fixed property of the plan: the response-budget curve shows it is a function of the responder's search effort, so benchmarks should report the curve rather than a single win rate.
  • Candidate sets carry usable information: multi-candidate RED responses produce Pareto-nondominated alternatives, so surfacing options can be evaluated separately from picking one.
  • Multi-agent critique-and-revision changes selection behavior: with council generation, three quarters of final plans come from revised candidates, meaning revision, not initial proposal, drives the final choice.
  • Transparent trace artifacts catch silent benchmark failures: a framing variable had zero measured effect until it was wired into generated content, which shows why benchmark controls must perturb the artifact to be meaningful.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The evaluation object is metric-agnostic: nothing stops a future user from swapping the synthetic quality score for expert ratings or a learned reward model while keeping the same trace schema, which would test whether the .518-to-.480 budget effect survives contact with realistic quality judgments.
  • The near-zero worldline-completion score is a red flag that current synthetic plans rarely specify end states; if that gap were fixed, the advantage numbers could shift in ways the current formulas cannot see.
  • The paper's own framing no-op result generalizes into a design principle: in any benchmark, a control variable has no interpretable effect unless it changes the generated content; editing the scenario text alone is not an intervention.
  • A natural next stressor the paper does not run: an adversary that selects its response metric adversarially (e.g., switching between quality-greedy and constraint-aware to maximize damage to BLUE) rather than committing to one fixed objective.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. AdvPlan-Bench is an offline benchmark artifact for adversarial evaluation of structured plan-generation agents. The paper defines a typed plan representation, a synthetic quality score q = w_e e − w_c c − w_r r with default weights (.5, .3, .2), a BLUE-vs-RED advantage metric, a heuristic constraint-coherence rubric, candidate-frontier diagnostics, and a multi-stage proposal/revision/adjudication council protocol. Across 150 synthetic scenarios in five planning templates, it reports that a best-of-8 quality-greedy RED response lowers BLUE advantage from .518 to .486 and win rate from .900 to .820; an offline deterministic LLM-policy contract baseline reaches .496 advantage and .700 win rate; and a two-stage council reaches .509 advantage and .813 win rate. A rubric-sensitivity study with three deterministic validators reports .978 agreement, and a scenario-framing experiment shows the framing variable was a silent no-op until wired into generation. The paper explicitly limits its scope to a reproducible benchmark artifact and disclaims operational or real-world decision quality.

Significance. The proposed evaluation object—an explicit tuple of scenario, structured plan, adversarial response distribution, and traceable candidate-set diagnostics—is a genuinely useful complement to final-answer agent benchmarks and static planning suites. The paper is commendably transparent: it ships a code artifact, reports bootstrap intervals in several tables, labels the offline LLM-policy baseline as deterministic, and names its main validity risks rather than hiding them. If the metric-validation concerns below are addressed, the benchmark could serve as a reusable scaffold for comparing plan-generation policies under adversarial response search. The current empirical numbers are best viewed as demonstrations of the benchmark machinery rather than as substantive findings about planning quality, because the synthetic quality score and constraint rubric are unvalidated and partly circular with the benchmark's own selection rules.

major comments (4)
  1. [§4, Eq. (3); §5 Tables 3–5] The quality score q = .5e − .3c − .2r is the basis for every advantage number and policy comparison in the paper, yet no sensitivity analysis over the weights (w_e, w_c, w_r) is reported. Section 7 itself names metric misspecification as the main internal-validity risk. As written, Table 3's headline effect (.518→.486 advantage) and Table 4's budget curve could reorder under plausible alternative weights such as uniform (.333/.333/.333) or risk-heavy (.3/.3/.4). This is load-bearing because the paper's central empirical claims would then reflect the benchmark's arbitrary scoring convention rather than any property of the policies. Please report a weight-sweep analysis showing whether the direction and ranking of the main conclusions (best-of-k effect, constraint-aware vs quality-greedy, council recovery) are stable, or provide external grounding for the chosen weights.
  2. [§4 Eqs. (4)–(5); Table 4] The monotone decrease of BLUE advantage with response budget k is a mathematical consequence of the definitions, not an independent empirical discovery. Eq. (5) selects the RED candidate maximizing q(c), and Eq. (4) defines advantage as a decreasing linear function of the selected RED quality. Since the maximum of k i.i.d. samples is nondecreasing in k, BLUE advantage must decrease with k for any quality score. Thus Table 4's .518→.480 drop is a preselected-max artifact. The paper should distinguish this deductive property from empirical evidence of robustness; the non-automatic win-rate drop in Table 3 is more informative but lacks confidence intervals. Please either reframe the budget curve as a construction check or add non-monotone metrics that are not predetermined by the argmax definition.
  3. [§5 Rubric Sensitivity; §7] The rubric-sensitivity study uses three deterministic validators that are perturbations of the same heuristic rubric, and the paper candidly states that this is not expert human validation. The constraint-coherence score is nonetheless load-bearing: it appears in the constraint-aware selector (Eq. 6, λ=.7) and in the council score (Eq. 1), and the paper interprets the .099 structural-alignment gain as a meaningful trade-off. Agreement of .978 among variants of the same underlying rule does not establish that the rubric captures qualitative plan quality. The authors should either add a small expert-rating study or explicitly downgrade all constraint-alignment results to internal-consistency checks. The latter would be sufficient for the narrow methodological claim, but the current framing overstates the import of the reported structural gain.
  4. [Table 3; Table 4; §5 Protocol] The protocol states that percentile-bootstrap 95% intervals are reported, but Table 3 and Table 4 give bare point estimates for the headline metrics. With 150 scenarios, the standard error of a win-rate difference between .900 and .820 is roughly 4–5 percentage points per group, so the reported 8-point drop may be within noise. This matters because the win-rate drop is one of the few non-automatic consequences of best-of-k selection. Please add bootstrap confidence intervals for BLUE advantage, BLUE win rate, RED quality, and Pareto counts in Tables 3 and 4, or clearly state that only point estimates are shown and intervals are available in the artifact.
minor comments (4)
  1. [§3, Eq. (1); §5 Council] The adjudicator score S(b) = αQ + βD + γV − ρP is defined abstractly, but the values of α, β, γ, ρ (and the h=2 threshold) are never given. Table 6's council results are therefore not reproducible from the paper alone. Please report the hyperparameters or point to the configuration file in the artifact.
  2. [Abstract; §5 Rubric Sensitivity] Calling the three deterministic validators 'raters' and reporting 'inter-rater agreement' is misleading; these are programmatic scoring variants, not independent human raters. Suggest 'validator agreement' or 'perturbation agreement' throughout.
  3. [Table 5] The column headers 'Single', 'k=8 LLM Win', 'k=8 Council win' are ambiguous. It is unclear which columns report advantage and which report win rate. Please clarify with explicit subheadings or a table caption.
  4. [§5 Scenario-Framing Finding] The framing-sensitivity delta (.066) is reported for an 'expanded corpus' that differs from the original zero-delta corpus. Please describe how the corpus changed and why the two numbers are not directly comparable.

Circularity Check

1 steps flagged

The headline result that best-of-k RED reduces BLUE advantage is a mathematical consequence of the benchmark's own quality and advantage definitions, not an independent empirical discovery.

specific steps
  1. self definitional [Section 3, Eq. 4; Section 5, Eq. 5 and Table 4]
    "The quality-greedy response selects cQ R = arg max c∈CR q(c) (Eq. 5); Table 4: 'BLUE advantage drops monotonically from .518 at k=1 to .480 at k=16.'"

    Advantage is defined as (q_blue − q_red + 2)/4 (Eq. 4), a decreasing linear function of q_red. The best-of-k response (Eq. 5) selects the maximum q(c) over k samples, so for any scenario the selected q_red at k>1 is at least as large as the single-sample q_red at k=1; in expectation it is strictly larger. Thus the observed monotone decrease in BLUE advantage with k is entailed by the definitions of the quality score, the advantage metric, and the response policy. The paper reports this as an empirical result in Table 4, but the direction is guaranteed by construction; only the magnitudes are data-dependent.

full rationale

The paper's central contribution is a benchmarking methodology, and most of its diagnostics are transparent, reproducible, and not circular. There are no self-citations, no imported uniqueness theorems, and no fitted parameters renamed as predictions. The main internal-validity threat (unvalidated synthetic quality weights) is explicitly acknowledged in Section 7 and is a correctness concern, not a circularity one. However, the flagship empirical observation that sampled best-response reduces BLUE advantage is a direct formal consequence of the advantage definition (Eq. 4) and the max-of-k selection rule (Eq. 5); the direction of the effect is baked into the benchmark's scoring equations. Because this 'result' is presented as evidence of the benchmark's usefulness, it is a partial circularity: the benchmark demonstrates a property it defines into existence. The exact magnitudes and win-rate changes are still empirical, so the circularity is limited to the direction of the headline claim.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The benchmark rests on hand-chosen synthetic scoring weights and a heuristic constraint rubric; these are the prices paid for a synthetic, expert-unvalidated benchmark. No new physical entities are introduced.

free parameters (2)
  • synthetic quality weights = w_e=0.5, w_c=0.3, w_r=0.2
    Hand-selected default weights in Eq. 3 for synthetic quality; the paper states it is a benchmark metric, not asset-effect matching.
  • constraint-aware selector lambda = 0.7
    Hand-chosen tradeoff in Eq. 6 between synthetic quality and constraint alignment for the constraint-aware RED policy.
axioms (3)
  • domain assumption Synthetic quality score q = we e - wc c - wr r reflects plan quality
    Eq. 3; the paper explicitly acknowledges it is a benchmark metric and requires expert validation (Section 7).
  • domain assumption The constraint-coherence rubric captures qualitative plan coherence
    Section 4, 'Constraint and Wargame Diagnostics'; rubric is heuristic and not expert-validated.
  • domain assumption The stylized competitive simulator is a useful secondary diagnostic
    Section 2 and 4; the paper calls it deliberately stylized and reports it only as a secondary diagnostic.

pith-pipeline@v1.3.0-alltime-deepseek · 10536 in / 10192 out tokens · 97792 ms · 2026-08-05T00:10:25.484932+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents." pith.science (2026). https://pith.science/paper/O4NZPGPI

@misc{pith2026260800832,
  author       = {Pith},
  title        = {Pith review of: AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4NZPGPI}},
  note         = {Machine review of arXiv:2608.00832}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Structured plan-generation agents are often evaluated as if a plan has quality in isolation, yet many realistic planning tasks require asking how a candidate behaves when another agent can search for responses. We introduce AdvPlan-Bench, an offline benchmark for adversarial evaluation of structured plan-generation agents. The contribution is a general evaluation object: a typed plan, an adversarial response set, selector diagnostics, and traceable candidate-frontier metrics. AdvPlan-Bench represents plans as typed action chains with optional branches, assigns synthetic quality scores, compares opposing plans with BLUE-vs-RED advantage and Nash-gap diagnostics, and evaluates qualitative constraint coherence with a transparent heuristic rubric. In 150 synthetic scenarios spanning five planning templates, a sampled best-response policy that draws eight response candidates reduces BLUE advantage from .518 to .486 and BLUE win rate from .900 to .820 relative to a single-sample response. An offline LLM-policy contract baseline reaches .496 BLUE advantage and .700 BLUE win rate, while a two-stage multi-agent council obtains .509 BLUE advantage and .813 BLUE win rate. A three-rater rubric-sensitivity study over 600 rating records yields .978 inter-rater agreement. AdvPlan-Bench is not an operational planner and provides no evidence about real-world decision quality; it is a reproducible benchmark artifact for studying adversarial plan evaluation, response-budget sensitivity, candidate frontiers, and multi-agent critique-and-revision traces.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 34 canonical work pages

  1. [1]

    P.; et al

    Bakhtin, A.; Brown, N.; Dinan, E.; Farina, G.; Flaherty, C.; Fried, D.; Goff, A.; Gray, J.; Hu, H.; Jacob, A. P.; et al. 2022. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1067--1074

    work page 2022

  2. [2]

    BIG-bench authors . 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research

    work page 2023

  3. [3]

    Brown, N.; and Sandholm, T. 2019. Superhuman AI for multiplayer poker. Science, 365(6456): 885--890

    work page 2019

  4. [4]

    E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In International Conference on Learning Representations

    work page 2024

  5. [5]

    W.; Keutzer, K.; and Gholami, A

    Kim, S.; Moon, S.; Tabrizi, R.; Lee, N.; Mahoney, M. W.; Keutzer, K.; and Gholami, A. 2024. An LLM Compiler for Parallel Function Calling. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 24370--24391

    work page 2024

  6. [6]

    Kokel, H.; Katz, M.; Srinivas, K.; and Sohrabi, S. 2025. ACPBench : Reasoning About Action, Change, and Planning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(25): 26559--26568

    work page 2025

  7. [7]

    Lanchester, F. W. 1916. Aircraft in Warfare: The Dawn of the Fourth Arm

    work page 1916

  8. [8]

    D.; Re, C.; Acosta-Navas, D.; Hudson, D

    Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; Newman, B.; Yuan, B.; Yan, B.; Zhang, C.; Cosgrove, C.; Manning, C. D.; Re, C.; Acosta-Navas, D.; Hudson, D. A.; Zelikman, E.; Durmus, E.; Ladhak, F.; Rong, F.; Ren, H.; Yao, H.; Wang, J.; Santhanam, K.; Orr, L.; Zheng, L.; Yuksekgonul, M....

    work page 2023

  9. [9]

    Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. 2024. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations

    work page 2024

  10. [10]

    Lu, J.; Holleis, T.; Zhang, Y.; Aumayer, B.; Nan, F.; Bai, H.; Ma, S.; Ma, S.; Li, M.; Yin, G.; Wang, Z.; and Pang, R. 2025. T ool S andbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, 1160--1183. Association for Computational Linguistics

    work page 2025

  11. [11]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine : Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, volume 36

    work page 2023

  12. [12]

    Ou, T.; Guo, W.; Gandhi, A.; Neubig, G.; and Yue, X. 2025. A gent D iagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 207--215. Association for Computational Linguistics

    work page 2025

  13. [13]

    Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In International Conference on Learning Representations

    work page 2024

  14. [14]

    Schelling, T. C. 1960. The Strategy of Conflict. Cambridge, MA: Harvard University Press

    work page 1960

  15. [15]

    Schick, T.; Dwivedi-Yu, J.; Dessi, R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, volume 36

    work page 2023

  16. [16]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36

    work page 2023

  17. [17]

    Shridhar, M.; Yuan, X.; C \^o t \'e , M.-A.; Bisk, Y.; Trischler, A.; and Hausknecht, M. 2021. ALFWorld : Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representations

    work page 2021

  18. [18]

    Silver, D.; Hubert, T.; Schrittwieser, J.; Antonoglou, I.; Lai, M.; Guez, A.; Lanctot, M.; Sifre, L.; Kumaran, D.; Graepel, T.; Lillicrap, T.; Simonyan, K.; and Hassabis, D. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 1140--1144

    work page 2018

  19. [19]

    M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D

    Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.; Georgiev, P.; et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350--354

    work page 2019

  20. [21]

    L.; Cao, Y.; and Narasimhan, K

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023 a . Tree of Thoughts : Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems, volume 36

    work page 2023

  21. [22]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023 b . ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations

    work page 2023

  22. [23]

    Zhang, Y.; Ganapavarapu, G.; Jayaraman, S.; Agrawal, B.; Patel, D.; and Fokoue, A. 2026. SPIRAL : Symbolic LLM Planning via Grounded and Reflective Search. Proceedings of the AAAI Conference on Artificial Intelligence, 40(43): 36527--36535

    work page 2026

  23. [24]

    Zheng, Z.; Cui, T.; Xie, C.; Pan, J.; Chen, Q.; and He, L. 2025. P lanning A rena: A Modular Benchmark for Multidimensional Evaluation of Planning and Tool Learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 31047--31086. Association for Computational Linguistics

    work page 2025

  24. [25]

    F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; Alon, U.; and Neubig, G

    Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; Alon, U.; and Neubig, G. 2024. WebArena : A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations

    work page 2024

  25. [26]

    Science , volume=

    A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , author=. Science , volume=

  26. [27]

    Nature , volume=

    Grandmaster level in StarCraft II using multi-agent reinforcement learning , author=. Nature , volume=

  27. [28]

    Science , volume=

    Superhuman AI for multiplayer poker , author=. Science , volume=

    work page

  28. [29]

    Science , volume=

    Human-level play in the game of Diplomacy by combining language models with strategic reasoning , author=. Science , volume=

  29. [30]

    Aircraft in Warfare: The Dawn of the Fourth Arm , author=

    work page

  30. [31]

    The Strategy of Conflict , author=

    work page

  31. [32]

    FM 3-0: Operations , author=

    work page

  32. [33]

    International Conference on Learning Representations , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. International Conference on Learning Representations , year=

  33. [34]

    International Conference on Learning Representations , year=

    AgentBench: Evaluating LLMs as Agents , author=. International Conference on Learning Representations , year=

  34. [35]

    Advances in Neural Information Processing Systems , volume=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  35. [36]

    International Conference on Learning Representations , year=

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs , author=. International Conference on Learning Representations , year=

  36. [37]

    International Conference on Learning Representations , year=

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , author=. International Conference on Learning Representations , year=

  37. [38]

    Proceedings of the 41st International Conference on Machine Learning , series=

    An LLM Compiler for Parallel Function Calling , author=. Proceedings of the 41st International Conference on Machine Learning , series=. 2024 , url=

    work page 2024

  38. [39]

    arXiv preprint arXiv:2406.12045 , year=

    tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    Pith/arXiv arXiv

  39. [40]

    2025 , publisher=

    Ou, Tianyue and Guo, Wanyao and Gandhi, Apurva and Neubig, Graham and Yue, Xiang , booktitle=. 2025 , publisher=. doi:10.18653/v1/2025.emnlp-demos.15 , url=

    doi:10.18653/v1/2025.emnlp-demos.15 2025

  40. [41]

    2025 , doi=

    Kokel, Harsha and Katz, Michael and Srinivas, Kavitha and Sohrabi, Shirin , journal=. 2025 , doi=

    work page 2025

  41. [42]

    2026 , doi=

    Zhang, Yifan and Ganapavarapu, Giridhar and Jayaraman, Srideepika and Agrawal, Bhavna and Patel, Dhaval and Fokoue, Achille , journal=. 2026 , doi=

    work page 2026

  42. [43]

    2025 , publisher=

    Zheng, Zihan and Cui, Tianle and Xie, Chuwen and Pan, Jiahui and Chen, Qianglong and He, Lewei , booktitle=. 2025 , publisher=. doi:10.18653/v1/2025.acl-long.1499 , url=

    doi:10.18653/v1/2025.acl-long.1499 2025

  43. [44]

    2025 , publisher=

    Lu, Jiarui and Holleis, Thomas and Zhang, Yizhe and Aumayer, Bernhard and Nan, Feng and Bai, Haoping and Ma, Shuang and Ma, Shen and Li, Mengyu and Yin, Guoli and Wang, Zirui and Pang, Ruoming , booktitle=. 2025 , publisher=. doi:10.18653/v1/2025.findings-naacl.65 , url=

    doi:10.18653/v1/2025.findings-naacl.65 2025

  44. [45]

    Transactions on Machine Learning Research , year=

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on Machine Learning Research , year=

  45. [46]

    Transactions on Machine Learning Research , year=

    Holistic Evaluation of Language Models , author=. Transactions on Machine Learning Research , year=

  46. [47]

    International Conference on Learning Representations , year=

    Shridhar, Mohit and Yuan, Xingdi and C. International Conference on Learning Representations , year=

  47. [48]

    and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle=

    Zhou, Shuyan and Xu, Frank F. and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle=. 2024 , url=

    work page 2024

  48. [49]

    and Cao, Yuan and Narasimhan, Karthik , booktitle=

    Yao, Shunyu and Yu, Dian and Zhao, Jeffrey and Shafran, Izhak and Griffiths, Thomas L. and Cao, Yuan and Narasimhan, Karthik , booktitle=. 2023 , url=

    work page 2023

  49. [50]

    2023 , url=

    Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Yazdanbakhsh, Amir and Clark, Peter , booktitle=. 2023 , url=

    2023

  50. [51]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    work page 2023