Pith. sign in

REVIEW 4 major objections 10 minor 153 references

Even the strongest LLMs fail most of the time at reliable multi-step tool use on product-like state-changing tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 14:48 UTC pith:NDFKPE52

load-bearing objection Solid agent-eval engineering with a real reliability signal; the closed artifact and missing MCP-side solvability check are the main caveats, not the core story. the 4 major comments →

arxiv 2607.23722 v1 pith:NDFKPE52 submitted 2026-07-26 cs.AI

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Weihuang Zheng , Tianyuan Zou , Eileen Ye , Alphet Liu , Youyong Kong , Ya-Qin Zhang , Duran Zheng , Maxm Pan This is my paper
classification cs.AI
keywords multi-step tool useLLM agentssynthetic benchmarksstateful environmentsfunction callingdatabase-diff evaluationcode executionproduct scenarios
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

E-Bench argues that real agent work is not one-shot API calling but multi-step tool use: repeatedly finding hidden state, composing tools, and committing exact changes in a stateful product world. Existing benchmarks either stay short and isolated or sit on live systems that are hard to scale, control, or grade cleanly. The paper builds a fully synthetic alternative with 323 state-changing tasks in three product domains, built so agents cannot skip discovery or invent missing data. On this benchmark, top models still leave large headroom: average success stays well below ceiling and three-trial reliability stays under 60%, and giving agents code execution raises scores but does not fix consistency. A sympathetic reader cares because product agents that only sometimes leave the database correct are not yet safe to run against real software state.

Core claim

Multi-step tool use on stateful product tasks remains unsolved for current frontier LLMs. Across 323 deterministically graded tasks, the best model reaches only about 74% average trial success, with Pass^3 below 60%; adding code execution improves every model but keeps the best Pass^3 below 70%. Reliability, not one-shot peak skill, is the central bottleneck.

What carries the argument

Generator–solver asymmetry on reusable synthetic product databases: graph-guided, orphan-free environment filling creates a shared world; a privileged generator with SQL and code authors tasks that induce verified state diffs, while the solver sees only natural-language requests and restricted domain tools, forcing an information gap and a tool gap graded by exact database diffs.

Load-bearing premise

If synthetic product databases and privileged task generation do not create the same hard parts as real multi-step product work, the low scores may not transfer beyond this construction.

What would settle it

Re-run the same models on held-out or live product backends with the same style of multi-step state-changing requests and check whether Pass^3 stays below about 60% without code and below about 70% with code, under exact state-diff grading.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Exact database-diff grading can replace LLM judges for closed-loop tool-use evaluation.
  • Code execution mainly lifts computation-heavy retrieval, filtering, and aggregation, not reasoning-heavy boundary and cascade decisions.
  • Reliability metrics like Pass^3 matter more than single-trial success for agents that modify live product state.
  • Reusable synthetic environments can scale task generation without redesigning data fixtures per task.
  • Cost-efficient agents will need parallel tool calling and selective code batching, not only longer trajectories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training on synthetic generator–solver pairs with verified diffs could target consistency failures more directly than trajectory imitation.
  • Cross-domain tasks that chain game, music, and meeting state would likely widen the reliability gap further.
  • Product teams may need mandatory dry-run or diff-preview gates until Pass^3 on this class of work is much higher.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 10 minor

Summary. The paper introduces E-Bench, a fully synthetic benchmark of 323 state-changing, multi-step tool-use tasks over three mocked product domains (Honor of Kings, QQ Music, Tencent Meeting). Construction is decoupled: graph-guided database filling produces a reusable, referentially-integrity-checked product DB per domain; a privileged generator (Claude Opus 4.7) with query_sql/exec_code then authors tasks and records ground-truth state diffs, while solvers see only natural-language intents and domain MCP tools, inducing an 'information gap' and a 'tool gap.' Grading is deterministic exact DB-diff match with no partial credit, over three independent trials on fresh DB copies, with Avg@3 / Pass@3 / Pass^3 metrics. Eleven frontier models are evaluated, plus the E-Bench-Code ablation granting solvers exec_code. Findings: best Avg@3 is 73.79% (Kimi-K3), Pass^3 stays below 60% on E-Bench and below 70% on E-Bench-Code; exec_code raises every model's accuracy while cutting tool calls ~74% and cost; gains concentrate in computation-heavy capabilities rather than reasoning-heavy ones. Per-domain, per-capability, parallelism, and cost/Pareto analyses are included.

Significance. If the tasks are as solvable as claimed, E-Bench is a useful addition to agent evaluation, and several strengths deserve explicit credit: (i) deterministic database-diff grading with no LLM judge; (ii) fresh DB copies and three independent trials per task, enabling the Pass^3 reliability analysis — the Pass@3→Pass^3 collapse (87.62%→58.82% for the best model) is a substantive, well-measured finding; (iii) the E-Bench-Code ablation is a genuinely controlled comparison (tasks, tools, DBs, grader held fixed) that cleanly separates orchestration mechanics from information extraction, with falsifiable per-capability predictions that are borne out (Table 4); (iv) cost/Pareto and parallelism analyses (App. A) of practical value; (v) detailed failure-anatomy case studies. The qualitative conclusion — reliable multi-step tool use is unsolved — is robust. The precise difficulty numbers, however, currently rest on an unverified feasibility premise (Major Comment 1), which bounds their evidentiary value until the requested audit is reported.

major comments (4)
  1. [§3.3, 'Task Generation and Validation'] Solvability is validated only under privileged access. The validators receive 'the same full-information tools as the generator, query_sql and exec_code,' so the claim that tasks are 'deterministically solvable' is never checked through the MCP-only interface solvers actually use. Under exact-diff grading, an MCP-infeasible task is silently scored as agent failure, inflating the difficulty gap that is the paper's headline. A decisive, cheap audit exists: report the per-task union solve rate across the 11 evaluated models, separately for E-Bench and E-Bench-Code. Opus-4.8's 92.61% Pass@3 in E-Bench-Code is already constructive evidence that most tasks are reachable through the domain tools (exec_code only wraps them), but the residual — and the base setting specifically — is uncharacterized. Any never-solved task needs a manual feasibility pass or removal.
  2. [§3.3] Two aspects of the validation protocol are underspecified in ways that matter under exact-diff grading. (i) Do validators re-solve each task from the natural-language intent, or are they shown the recorded diff to confirm? The text ('focuses validation on the correctness of the recorded state change') suggests the latter, which leaves intent-level ambiguity untested. (ii) Tie-prone intents — 'the one with the highest intimacy' (App. B.1), 'the earliest common 1-hour slot in half-hour steps' (App. B.2) — admit multiple defensible end states under ties, and no-partial-credit grading scores an equally-correct alternative as failure. Please state how uniqueness of the target state is enforced (construction-time tie exclusion, or an explicit tie-break in the intent). Also, the validator panel includes the generator model itself (Claude Opus 4.7), so ≥2-of-3 agreement is not fully independent
  3. [§3.3 and abstract] The construction funnel is not reported. Tasks are discarded for 'no state change, unparseable intent, or excessive tool use,' then filtered by ≥2-of-3 validator agreement, then 'tasks solved easily by a weak baseline' are removed — but no counts are given at any stage, and the weak baseline and its threshold are unnamed. Because the headline is a difficulty claim ('multi-step tool use remains far from solved'), the size of the difficulty filter is load-bearing for interpretation: Avg@3/Pass^3 currently characterize a difficulty-filtered subset, not a representative distribution of product requests. Please report per-stage acceptance counts, name the baseline and the easy-task criterion, and add one caveat sentence to the abstract or §4.2.
  4. [§4.1–4.2, Table 3] No uncertainty quantification accompanies two-decimal rankings. With 969 trials per model, the top-table gap (Kimi-K3 73.79% vs GPT-5.5 72.03%, Δ≈1.8 pts) is within roughly one standard error of an unpaired test, and per-task Pass^3 is a four-level estimator, so Pass^3 differences of a few points are likewise noisy. Claims such as 'Opus-4.8 ... taking the top spot' (§4.2.2) are not established at n=3 trials/task. Please add task-clustered bootstrap confidence intervals for Avg@3 and Pass^3 differences (at minimum for adjacent ranks), and temper ordinal language where intervals overlap. The requested union-solve statistic in Major Comment 1 would partially reuse the same machinery.
minor comments (10)
  1. [§A.4] The paragraph around Eq. (1) is duplicated and garbled: 'The task averaged API cost is calculated following: Real-world deployments must balance...' is immediately followed by 'The task-averaged API cost is computed as: follows:', and the symbol definitions are given twice with slightly different wording. Please rewrite the passage once.
  2. [§A.3] Opening sentence: 'As discussed in Section A.3, under E-Bench...' is a self-reference; presumably Section 4.6 is meant.
  3. [§4.6] 'Because models can capability many domain-specific function calls into a single exec_code block' — 'capability' is a typo (presumably 'pack' or 'fold').
  4. [Table 2] Note reads 'there is not DB Diff of update type for QQ Music by environment desgin' — grammar and typo ('desgin'); also worth one sentence saying why updates are absent by design, since readers will otherwise suspect a data issue.
  5. [§3.3 / Table 1] Capability labels are annotated by the generator itself, and the per-capability conclusions of Table 4/§4.4 rest on them. Please report a spot-check (human or cross-model agreement on a sample) so the labels are not taken on faith.
  6. [§4.6 / §A.3] The Pearson correlations are computed over 33 aggregated <model,domain> points, mixing between-model and between-domain variation; the gloss 'a low call count often indicates premature action' is causal and should be softened or supported by a within-model analysis.
  7. [§2, Related Work] AppWorld is characterized as coupling environments to individual tasks; in fact it uses a shared app world with programmatically generated per-task initial states and programmatic checkers. The sharper differentiator for E-Bench is full automation of both environment and task synthesis — please adjust the sentence rather than overstate the contrast.
  8. [Footnote, p.1 (release policy)] The anti-overfitting rationale for non-release is defensible, but consider releasing schemas, MCP tool specifications/implementations, and a small dev task slice so tool semantics and the harness can be independently audited while test tasks stay private.
  9. [Table 5] API prices change over time and three rows are FX-converted from RMB; please state the price snapshot date so the Pareto analysis remains interpretable.
  10. [Appendix B.1, Case 1] The a_star_b/b_star_a failure reads, on inspection, as a genuine agent error (the relation record exposes the current user's side, and the model even self-corrects but leaves the stale write) rather than interface unfairness. One clarifying sentence to that effect would preempt readers citing it as evidence of an unnavigable schema.

Circularity Check

0 steps flagged

No circular derivation: E-Bench reports external empirical measurements on synthetic tasks, not predictions forced by fitted inputs or self-defined quantities.

full rationale

E-Bench is an empirical benchmark paper. Its central claim (frontier models reach at most ~74% Avg@3 and Pass^3 < 60%/70% on base/code settings) is a measured outcome on 11 external LLM APIs under a fixed harness, graded by deterministic database-state diffs—not a first-principles derivation, fitted-parameter “prediction,” or uniqueness argument. Environment synthesis (graph-guided filling, FK constraints, validation/repair) and task synthesis (generator–solver asymmetry, capability taxonomy, majority-vote privileged validators, easy-task filter) are construction choices that can bias difficulty or transfer, but they do not make the reported scores equal their inputs by definition: solvers never see the ground-truth diffs, models are not the generator, and success is exact state match on held-out trials. Related-work citations are ordinary positioning, not load-bearing self-cited uniqueness theorems. Concerns that validators only check privileged reachability (SQL/code) rather than MCP-only solvability, or that filtering easy tasks raises measured hardness, are validity/design risks—not circular reductions of the form Eq. X ≡ Eq. Y. Per the circularity criteria, score 0 with empty steps is the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

As an empirical systems/benchmark paper, load-bearing premises are design choices and evaluation conventions rather than fitted physical constants. The claim that multi-step tool use is unsolved rests on accepting the synthetic worlds, the withheld task set, exact diff grading, and the chosen model/harness snapshot as a fair probe.

free parameters (4)
  • Validator acceptance rule (≥2 of 3 strong validators) = ≥2 of 3
    Task inclusion depends on majority agreement of GPT-5.5, Claude Opus 4.7, and GLM-5.1 with full SQL/code tools; threshold and validator set are design choices that shape the final 323-task distribution.
  • Easy-task filter via weak baseline
    Surviving tasks are further filtered by removing those solved easily by a weak baseline (§3.3), which directly tunes difficulty upward without a published baseline identity or threshold curve.
  • Trials per task (n=3) and Pass^3 reliability definition = 3 trials
    Headline reliability uses three independent runs and requires all three successes; n and the strict all-success metric are conventional choices that strongly affect the <60%/<70% claims.
  • Domain and scale choices (3 products, 323 tasks, table/row counts) = 323 tasks; 76,317 rows
    Environment size and task count are author-chosen; results are conditional on this mix (HoK / QQ Music / Tencent Meeting) and capability stratification.
axioms (5)
  • domain assumption Exact match of final database state to a recorded ground-truth diff is the correct notion of task success (no partial credit, no LLM judge).
    Stated throughout §3 and §4.1; defines all reported Avg@3/Pass metrics.
  • domain assumption Referential integrity via topological graph-guided filling plus post-hoc repair scripts yields a sufficiently realistic product state space for agent evaluation.
    §3.2 environment construction; underpins ecological-validity claims versus task-local fixtures.
  • domain assumption Withholding SQL/raw DB from solvers while allowing only domain MCP tools (and optionally exec_code over those tools) isolates multi-step tool use rather than arbitrary programming against the schema.
    §3.3–3.4 information gap and tool gap design.
  • domain assumption Standard multi-turn tool-agent harness assumptions: models run at highest thinking effort, tools are correctly implemented MCP CRUD facades, and unit-tested tool behavior matches DB semantics.
    §3.2 tool construction and §4.1 evaluation setup.
  • standard math Relational schemas, foreign keys, and deterministic diff comparison behave as in ordinary database systems.
    Background CS assumptions for grading and integrity-by-construction.
invented entities (2)
  • E-Bench / E-Bench-Code benchmark worlds no independent evidence
    purpose: Provide reusable synthetic product environments and 323 graded state-changing tasks with information and tool gaps.
    Core artifact introduced by the paper; not an external public standard at time of writing, and deliberately unreleased.
  • Six-type capability taxonomy (Full-Data Acquisition, Multi-Condition Filtering, Aggregation, Cross-Step Dependency, Precise Boundary Judgment, Cross-Entity Cascade) no independent evidence
    purpose: Guide task synthesis and slice results by skill.
    Author-defined labels on generated tasks (Table 1); useful analytically but not independently validated constructs.

pith-pipeline@v1.2.0-grok45-kimik3 · 27076 in / 3572 out tokens · 73157 ms · 2026-07-30T14:48:02.369978+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios." pith.science (2026). https://pith.science/paper/NDFKPE52

@misc{pith2026260723722,
  author       = {Pith},
  title        = {Pith review of: E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NDFKPE52}},
  note         = {Machine review of arXiv:2607.23722}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

153 extracted references · 1 canonical work pages

  1. [1]

    The claude 3 model family: A new standard for intelligence, 2024

    Anthropic. The claude 3 model family: A new standard for intelligence, 2024. URL https://www.anthropic.com/news/claude-3-family

    2024

  2. [2]

    MCP -atlas: A large-scale benchmark for tool-use competency with real MCP servers

    Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, and Bing Liu. MCP -atlas: A large-scale benchmark for tool-use competency with real MCP servers. arXiv preprint arXiv:2602.00933, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    VitaBench : Benchmarking LLM agents with versatile interactive tasks in real-world applications

    Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, and Yunke Zhao. VitaBench : Benchmarking LLM agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025

    arXiv 2025

  4. [4]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  5. [5]

    SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  6. [6]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, 2024

    2024

  7. [9]

    MCP-Universe : Benchmarking large language models with real-world Model Context Protocol servers

    Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. MCP-Universe : Benchmarking large language models with real-world Model Context Protocol servers. arXiv preprint arXiv:2508.14704, 2025

    Pith/arXiv arXiv 2025

  8. [10]

    Gaia: a benchmark for general ai assistants

    Gr \'e goire Mialon, Cl \'e mentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  9. [11]

    LiveMCPBench : Can agents navigate an ocean of MCP tools? arXiv preprint arXiv:2508.01780, 2025

    Guozhao Mo, Wenliang Zhong, Jiawei Chen, Qianhao Yuan, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. LiveMCPBench : Can agents navigate an ocean of MCP tools? arXiv preprint arXiv:2508.01780, 2025

    arXiv 2025

  10. [12]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 0 126544--126565, 2024

    2024

  11. [13]

    Gonzalez

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=2GmDdhBdDk

    2025

  12. [14]

    Tool LLM : Facilitating large language models to master 16000+ real-world API s

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. Tool LLM : Facilitating large language models to master 16000+ real-world API s. In The Twelfth International Conference on Lear...

    2024

  13. [15]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98

    2024

  14. [17]

    Appworld: A controllable world of apps and people for benchmarking interactive coding agents

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024

  15. [18]

    MCP -bench: Benchmarking tool-using LLM agents with complex real-world tasks via MCP servers

    Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, and Eugene Siow. MCP -bench: Benchmarking tool-using LLM agents with complex real-world tasks via MCP servers. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=fe8mzHwMxN

    2026

  16. [19]

    MCPMark : A benchmark for stress-testing realistic and comprehensive MCP use

    Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, and Michael Qizhe Shieh. MCPMark : A benchmark for stress-testing realistic and comprehensive MCP use. In International Conference on Learning Representations, 2026. URL https://open...

    2026

  17. [21]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369--2380, 2018

    2018

  18. [22]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  19. [24]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=oKn9c6ytLx

    2024

  20. [25]

    2025 , month = dec, note =

    Gupta, Mehul , title =. 2025 , month = dec, note =

    2025

  21. [26]

    OpenAI o3 and o4-mini System Card , author=

  22. [27]

    arXiv preprint arXiv:2504.21801 , year=

    Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition , author=. arXiv preprint arXiv:2504.21801 , year=

    Pith/arXiv arXiv

  23. [28]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi k2: Open agentic intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    Pith/arXiv arXiv

  24. [29]

    arXiv preprint arXiv:2601.03267 , year=

    Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

    Pith/arXiv arXiv

  25. [30]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    Pith/arXiv arXiv

  26. [31]

    2: Pushing the frontier of open large language models , author=

    Deepseek-v3. 2: Pushing the frontier of open large language models , author=. arXiv preprint arXiv:2512.02556 , year=

    Pith/arXiv arXiv

  27. [32]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  28. [33]

    Nature , volume=

    A benchmark of expert-level academic questions to assess AI capabilities , author=. Nature , volume=. 2026 , publisher=

    2026

  29. [34]

    International Conference on Learning Representations , year=

    Memorizing Transformers , author=. International Conference on Learning Representations , year=

  30. [35]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Nested Learning: The Illusion of Deep Learning Architectures , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  31. [36]

    2025 , eprint=

    ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems , author=. 2025 , eprint=

    2025

  32. [37]

    2024 , url=

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle=. 2024 , url=

    2024

  33. [38]

    International conference on data intelligence and cognitive informatics , pages=

    Prompt engineering in large language models , author=. International conference on data intelligence and cognitive informatics , pages=. 2023 , organization=

    2023

  34. [39]

    arXiv preprint arXiv:2402.07927 , year=

    A systematic survey of prompt engineering in large language models: Techniques and applications , author=. arXiv preprint arXiv:2402.07927 , year=

    Pith/arXiv arXiv

  35. [40]

    ACM computing surveys , volume=

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  36. [41]

    arXiv preprint arXiv:2302.11382 , year=

    A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT , author=. arXiv preprint arXiv:2302.11382 , year=

    Pith/arXiv arXiv

  37. [42]

    Advances in Neural Information Processing Systems , volume=

    Language Models are Few-Shot Learners , author=. Advances in Neural Information Processing Systems , volume=

  38. [43]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    A survey on in-context learning , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  39. [44]

    2022 , journal=

    In-context Learning and Induction Heads , author=. 2022 , journal=

    2022

  40. [45]

    International Conference on Learning Representations , year=

    An Explanation of In-context Learning as Implicit Bayesian Inference , author=. International Conference on Learning Representations , year=

  41. [46]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages=

    Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages=

  42. [47]

    International Conference on Machine Learning , pages=

    Calibrate Before Use: Improving Few-shot Performance of Language Models , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  43. [48]

    arXiv preprint arXiv:2512.13564 , year=

    Memory in the Age of AI Agents , author=. arXiv preprint arXiv:2512.13564 , year=

    Pith/arXiv arXiv

  44. [49]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  45. [50]

    arXiv preprint arXiv:2507.13334 , year=

    A survey of context engineering for large language models , author=. arXiv preprint arXiv:2507.13334 , year=

    Pith/arXiv arXiv

  46. [51]

    Advances in Neural Information Processing Systems , volume=

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , author=. Advances in Neural Information Processing Systems , volume=

  47. [52]

    arXiv preprint arXiv:2310.08560 , year=

    MemGPT: Towards LLMs as Operating Systems , author=. arXiv preprint arXiv:2310.08560 , year=

    Pith/arXiv arXiv

  48. [53]

    2025 , eprint=

    Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG , author=. 2025 , eprint=

    2025

  49. [54]

    2024 , eprint=

    Retrieval-Augmented Generation for Large Language Models: A Survey , author=. 2024 , eprint=

    2024

  50. [55]

    2024 , url=

    The Claude 3 Model Family: A New Standard for Intelligence , author=. 2024 , url=

    2024

  51. [56]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  52. [57]

    arXiv preprint arXiv:2403.05530 , year=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. arXiv preprint arXiv:2403.05530 , year=

    Pith/arXiv arXiv

  53. [58]

    arXiv preprint arXiv:2406.12793 , year=

    ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools , author=. arXiv preprint arXiv:2406.12793 , year=

    Pith/arXiv arXiv

  54. [59]

    arXiv preprint arXiv:2407.21783 , year=

    The Llama 3 Herd of Models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  55. [60]

    Findings of EMNLP , year=

    ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding , author=. Findings of EMNLP , year=

  56. [61]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    L-eval: Instituting standardized evaluation for long context language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  57. [62]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  58. [63]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    2024

  59. [64]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Loogle: Can long-context language models understand long contexts? , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  60. [65]

    arXiv preprint arXiv:2402.13718 , year=

    -Bench: Extending Long Context Evaluation Beyond 100K Tokens , author=. arXiv preprint arXiv:2402.13718 , year=

    Pith/arXiv arXiv

  61. [66]

    2024 , url=

    Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Boris Ginsburg , booktitle=. 2024 , url=

    2024

  62. [67]

    arXiv preprint arXiv:2410.02694 , year=

    Helmet: How to evaluate long-context language models effectively and thoroughly , author=. arXiv preprint arXiv:2410.02694 , year=

    Pith/arXiv arXiv

  63. [68]

    Transactions of the Association for Computational Linguistics , volume=

    The NarrativeQA Reading Comprehension Challenge , author=. Transactions of the Association for Computational Linguistics , volume=

  64. [69]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    2021

  65. [70]

    Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    QuALITY: Question answering with long input texts, yes! , author=. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2022

  66. [71]

    Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing , pages=

    Docbench: A benchmark for evaluating llm-based document reading systems , author=. Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing , pages=

  67. [72]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics , pages=

    2021

  68. [73]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Efficient Attentions for Long Document Summarization , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  69. [74]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    SQuALITY: Building a Long-Document Summarization Dataset the Hard Way , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages=

    2022

  70. [75]

    2023 , howpublished=

    Needle in a Haystack - Pressure Testing LLMs , author=. 2023 , howpublished=

    2023

  71. [76]

    Advances in Neural Information Processing Systems , volume=

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack , author=. Advances in Neural Information Processing Systems , volume=

  72. [77]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  73. [78]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Longcite: Enabling llms to generate fine-grained citations in long-context qa , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  74. [79]

    Advances in Neural Information Processing Systems , volume=

    Many-shot in-context learning , author=. Advances in Neural Information Processing Systems , volume=

  75. [80]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-Following Evaluation for Large Language Models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  76. [81]

    12th International Conference on Learning Representations, ICLR 2024 , year=

    COLLIE: SYSTEMATIC CONSTRUCTION OF CONSTRAINED TEXT GENERATION TASKS , author=. 12th International Conference on Learning Representations, ICLR 2024 , year=

    2024

  77. [82]

    Advances in Neural Information Processing Systems , volume=

    Benchmarking complex instruction-following with multiple constraints composition , author=. Advances in Neural Information Processing Systems , volume=

  78. [83]

    Findings of the Association for Computational Linguistics: ACL 2024 , year=

    InFoBench: Evaluating Instruction Following Ability in Large Language Models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , year=

    2024

  79. [84]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Followbench: A multi-level fine-grained constraints following benchmark for large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  80. [85]

    12th International Conference on Learning Representations, ICLR 2024 , year=

    KITAB: EVALUATING LLMS ON CONSTRAINT SATISFACTION FOR INFORMATION RETRIEVAL , author=. 12th International Conference on Learning Representations, ICLR 2024 , year=

    2024

Showing first 80 references.