Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The Necessity of a Unified Framework for LLM-Based Agent Evaluation

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LLM-agent benchmark results cannot be attributed to the model until the field adopts a unified evaluation framework built on a deterministic sandbox and a standardized evaluation methodology.

desk verdict A well-organized position paper on agent-evaluation standardization with real examples, but the central 'necessity' claim is asserted, not demonstrated, and the stress-test concern about architecture bias is genuine; deserves a serious referee, not a desk reject. read the letter →

arxiv 2602.03238 v2 pith:UWESIHK4 submitted 2026-02-03 cs.AI

Pengyu Zhu , Li Sun , Philip S. Yu , Sen Su This is my paper
classification cs.AI
keywords LLMagentsagentevaluationbenchmarkstandardizationsandboxmethodologyreproducibilityconfoundingfactorsfailuretaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current LLM-agent benchmark scores are not trustworthy measures of the model's own capability because they are entangled with framework-level choices: system prompts, planning strategies, memory formats, tool definitions, and environmental dynamics. It claims that without a shared evaluation standard, reported performance gains cannot be attributed to the agent itself, making cross-benchmark comparisons effectively meaningless. The proposed remedy is a unified framework with two components: a sandbox that makes the environment deterministic and reproducible, and a standardized evaluation methodology that includes consistent metrics, pass@k protocols, and failure taxonomies. A sympathetic reader should care because the paper identifies a concrete, structural flaw in how agent progress is currently measured, and proposes a path toward comparable and auditable results.

What carries the argument

The sandbox is the central mechanism: defined not merely as a security container but as a unified evaluation substrate that encapsulates the agent system and its controllable environment, enforcing deterministic, versioned, and reproducible execution. It is paired with a standardized evaluation methodology that compares agent trajectories and environment-state changes against gold references, uses a benchmark-wide standardized pass@k, and adopts a unified failure taxonomy. Together these are proposed to convert agent evaluation from a fragmented, framework-dependent practice into a controlled measurement of agentic capability.

What would settle it

Take the same set of LLMs and run them on identical tasks in two different agent frameworks that differ only in system prompt wording and memory formatting. If the resulting scores are nearly identical across frameworks, the paper's central claim that framework choices heavily confound agent evaluation would be contradicted.

Watch

Extended reading notes

Core claim

The central claim is that a unified framework for LLM-based agent evaluation is not optional but necessary. The paper decomposes agent evaluation into two load-bearing parts: the sandbox, a unified evaluation substrate that enforces hermetic determinism and isolates the agent from external volatility, and the evaluation methodology, which standardizes metrics, pass@k choices, efficiency measures, and failure attribution. The paper argues that without both, evaluation outcomes reflect design choices rather than agentic capability, and that specific sources of variance—inference configuration, prompting and planning, memory mechanisms, tool invocation, and external environments—each systematic

Load-bearing premise

The paper assumes that a single standardized framework can be defined neutrally enough to measure agentic capability without privileging dominant architectures, and that the research community will actually adopt it.

Editorial extensions

If this is right

  • If the unified framework is adopted, benchmark results across different research groups would become directly comparable, because the agent instantiation and environment would be the same.
  • Standardizing pass@k across benchmarks would prevent reported scores from reflecting favorable sampling regimes rather than robust agent behavior.
  • Static, versioned sandbox environments would make failures traceable and reproducible, allowing errors to be attributed to the agent rather than to external volatility.
  • A unified failure taxonomy would enable cross-benchmark diagnosis of agent weaknesses, supporting large-scale and automated error attribution.
  • The framework would also provide a safe substrate for evaluating offensive and defensive agent capabilities without real-world side effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's thesis: run a fixed set of models on the same tasks under different agent frameworks while holding the model fixed; if scores vary widely across frameworks, the confounding claim is supported, but if they barely move, the urgency of unification weakens.
  • The paper's insistence on a deterministic sandbox implies an acceptance of reduced ecological validity; the editorially drawn consequence is that sandbox results should be treated as a lower bound on real-world performance, not a complete measure.
  • If a unified standard explicitly avoids prescribing agent architecture, it remains an open question whether standardization can be confined to evaluation-relevant interfaces without leaking into architectural assumptions—something the paper acknowledges as a risk in its alternative-view section.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This position paper argues that current LLM-based agent evaluations are confounded by framework-specific implementation choices—inference protocols, system prompts and planning strategies, memory mechanisms, tool representations, and external environment dynamics—so that benchmark scores cannot be reliably attributed to the model's agentic capability. It proposes a unified evaluation framework built on a sandbox (standardized dataset plus a unified agent architecture, e.g., smolagents) and a multidimensional evaluation methodology (state-based correctness comparison, pass@k with a fixed k, efficiency metrics, and a standardized failure taxonomy). Section 6 acknowledges limitations regarding architectural privilege and ecological validity.

Significance. The paper provides a useful and well-organized taxonomy of variance sources in LLM-based agent evaluation and a concrete, if high-level, proposal for standardization. It is honest about some limitations and grounds its arguments in real examples (BrowseComp/BrowseComp-Plus, BFCL, τ-bench). If the central claim holds, current agent leaderboards and cross-paper comparisons would need to be reinterpreted as measuring a joint system rather than the model alone. However, the paper is a normative analysis rather than an empirical study: it offers no measurements showing that a unified framework removes the identified confounds, and the proposed standardization itself introduces a new set of choices that may favor one architectural paradigm. The contribution is therefore a position statement with a clear agenda rather than a validated evaluation methodology.

major comments (3)
  1. [Section 4.1, 5.2, 6] The paper claims that standardizing the agent system architecture ensures that 'evaluation results reflect the LLM's agentic capability itself' (Section 4.1), and recommends adopting a specific open-source framework such as smolagents (Section 5.2). But standardizing on one framework does not eliminate framework-specific bias; it selects one configuration as the reference. Without evidence that model rankings are stable across equally reasonable architectures, the measured scores are at best capability conditional on that architecture. Section 6 concedes this risk, but 'continuously extensible' does not resolve it: extensibility and cross-benchmark comparability pull in opposite directions. The authors should either weaken the attribution claim to 'capability under a shared, explicitly specified substrate' or provide empirical evidence (e.g., a cross-architecture ranking-stability analys
  2. [Section 3, 4.1] The entire argument assumes that the 'primary object of interest is the agentic capability of the LLM' (Section 3) and treats all framework-level choices as nuisance variance. But 'agentic capability' is not defined independently of the framework that instantiates prompts, planning, memory, and tools. For many deployed systems the framework is part of the system, and there may be no model-independent latent quantity to isolate. If so, the proposed unification selects one among several possible measurement conventions without a principled basis. The authors should define the target construct more carefully and justify why cross-framework differences are noise rather than meaningful performance differences.
  3. [Section 5.3] The paper advocates pass@k as a 'robustness-aware evaluation protocol' and proposes benchmark-wide standardization of k. However, pass@k is sensitive to the per-trial variance of each model: two models with the same expected per-trial success probability can have very different pass@k if one is more deterministic. Fixing k across benchmarks does not make scores comparable if models differ in stochasticity. The authors should specify a more principled aggregation, such as reporting full pass@k curves, confidence intervals, or variance-adjusted scores, or justify why a single k is adequate.
minor comments (6)
  1. [Section 4.1] Typo: 'their necessary to achieve such consistency' should be 'their necessity to achieve such consistency'.
  2. [Section 2.2] Appears twice: 'plantforms' should be 'platforms'.
  3. [Section 5.2] Grammatical: 'we recommend adopting a unified open-source agent frameworks' should be 'a unified open-source agent framework'.
  4. [References] BFCL appears twice (Patil et al., 2025a and 2025b) with the same title and URL; consolidate or distinguish the two versions clearly.
  5. [Figure 1] The caption would benefit from a fuller explanation of how trajectory, final answer, and environment-state evaluation are represented in the diagram.
  6. [Section 5.3] Minor formatting: 'standardization ofk' lacks a space; consider introducing pass@k formally before use.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a position/analysis piece with no fitted parameters or derivation to reduce to its inputs.

full rationale

The manuscript makes a normative argument for a unified evaluation framework based on observed sources of variance (system prompts, tool schemas, environments, metrics) in existing benchmarks. It does not derive a numerical result, fit a parameter, or predict a benchmark outcome; its central claim is an argument about evaluation practice, supported by cited external benchmarks and examples. The proposal (sandbox + standardized agent architecture + multidimensional methodology) is presented as a recommendation, and Section 6 explicitly acknowledges the main limitation—that standardization may privilege dominant architectures and reduce ecological validity—rather than defining that limitation away. No equation in the paper equates an output to an input, no fitted value is relabeled as a prediction, and citations to the authors' own prior work are not load-bearing for the central premise. The paper is self-contained as an analytical position piece; any weakness is evidentiary (absence of empirical demonstration that rankings are stable across architectures), which is a correctness/evidence concern, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's argument rests on normative premises about what evaluation should measure and whether standardization is feasible. These are acknowledged in the text but not established empirically.

assumptions (3)
  • domain assumption The primary goal of agent evaluation is to measure the intrinsic agentic capability of the LLM, independent of framework components.
    This premise grounds the entire argument; if evaluation instead should measure system-level performance including framework choices, the confound problem largely disappears. Stated in Section 3: 'the primary object of interest is the agentic capability of LLM.'
  • domain assumption A deterministic, static sandbox can faithfully represent agentic capability without an unacceptable loss of ecological validity.
    Section 3.5 and Section 4.1 require deterministic, version-controlled environments; Section 6 concedes this reduces ecological validity. The trade-off is asserted, not measured.
  • domain assumption Community-wide adoption of a standardized framework is achievable and would not suppress methodological diversity.
    Section 5 states effectiveness depends on 'broad community adoption,' and Section 6 acknowledges the risk of privileging dominant paradigms but does not resolve it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Necessity of a Unified Framework for LLM-Based Agent Evaluation." pith.science (2026). https://pith.science/paper/UWESIHK4

@misc{pith2026260203238,
  author       = {Pith},
  title        = {Pith review of: The Necessity of a Unified Framework for LLM-Based Agent Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWESIHK4}},
  note         = {Machine review of arXiv:2602.03238}
}
read the original abstract

With the advent of Large Language Models (LLMs), general-purpose agents have seen fundamental advancements. However, evaluating these agents presents unique challenges that distinguish them from static QA benchmarks. We observe that current agent benchmarks are heavily confounded by extraneous factors, including system prompts, toolset configurations, and environmental dynamics. Existing evaluations often rely on fragmented, researcher-specific frameworks where the prompt engineering for reasoning and tool usage varies significantly, making it difficult to attribute performance gains to the model itself. Additionally, the lack of standardized environmental data leads to untraceable errors and non-reproducible results. This lack of standardization introduces substantial unfairness and opacity into the field. We propose that a unified evaluation framework is essential for the rigorous advancement of agent evaluation. To this end, we introduce a proposal aimed at standardizing agent evaluation.

Figures

Figures reproduced from arXiv: 2602.03238 by the authors.

Figure 1
Figure 1. Overview of LLM-based agent evaluation, where correct￾ness is assessed over trajectories, final answer, and environment state changes rather than a single output. 2. Background on Agent Evaluation 2.1. From LLM Evaluation to Agent Evaluation Classical LLM benchmarks evaluate models as static input– output systems, where performance is assessed by the cor￾rectness or quality of the response for a fixed prompt (Ni et … view at source ↗
Figure 2
Figure 2. A unified framework where the sandbox integrates a standardized dataset and a unified agent architecture, coupled with multidimensional evaluation. full capability. Additional metrics should include reasoning steps, resource consumption, failure analysis, and communi￾cation in multi-agent scenarios. However, many benchmarks rely on a narrow set of metrics, often focusing solely on task completion, which limits the c… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Unified Framework for the Evaluation of LLM Agentic Capabilities

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    A unified framework standardizes LLM agent benchmarks and demonstrates through 400K rollouts that scaffold and environment choices materially alter outcomes, enabling separation of intrinsic capabilities from artifacts.

Reference graph

Works this paper leans on

45 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    System card: Claude opus 4 claude sonnet 4, 2025

    Anthropic . System card: Claude opus 4 claude sonnet 4, 2025. URL https://www.anthropic.com/claude-4-system-card. Technical Report

    2025

  2. [2]

    ^2 -bench: Evaluating conversational agents in a dual-control environment, 2025

    Barres, V., Dong, H., Ray, S., Si, X., and Narasimhan, K. ^2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982

    arXiv 2025

  3. [3]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    arXiv 2021

  4. [4]

    E motion Q ueen: A benchmark for evaluating empathy of large language models

    Chen, Y., Yan, S., Liu, S., Li, Y., and Xiao, Y. E motion Q ueen: A benchmark for evaluating empathy of large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 2149--2176, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2...

    doi:10.18653/v1/2024.findings-acl.128 2024

  5. [5]

    Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent

    Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Sharifymoghaddam, S., Liu, A., Green, J., Patel, K., Meng, R., Su, M., Li, Y., Hong, H., Shi, X., Liu, X., Thakur, N., Zhang, C., Gao, L., Chen, W., and Lin, J. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. In First Workshop on Multi-Turn Interactions in Large Lang...

    2025

  6. [6]

    Agentic reinforced policy optimization, 2025

    Dong, G., Mao, H., Ma, K., Bao, L., Chen, Y., Wang, Z., Chen, Z., Du, J., Wang, H., Zhang, F., Zhou, G., Zhu, Y., Wen, J.-R., and Dou, Z. Agentic reinforced policy optimization, 2025. URL https://arxiv.org/abs/2507.19849

    arXiv 2025

  7. [7]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Hernandez, D., Hume, T., Jacobson, J., Johnston, S., Kravec, S., Olsson, C., Ringer, S., Tran-J...

    arXiv 2022

  8. [8]

    Goldberg, I., Wagner, D., Thomas, R., and Brewer, E. A. A secure environment for untrusted helper applications confining the wily hacker. In Proceedings of the 6th Conference on USENIX Security Symposium, Focusing on Applications of Cryptography - Volume 6, SSYM'96, pp.\ 1, USA, 1996. USENIX Association

    1996

Show all 45 references
  1. [9]

    Function calling | gemini api | google ai for developers

    Google. Function calling | gemini api | google ai for developers. https://ai.google.dev/gemini-api/docs/function-calling, 2026. Accessed: 2026-01-16

    2026

  2. [10]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  3. [11]

    Measuring mathematical problem solving with the MATH dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021...

    2021

  4. [12]

    Model context protocol (mcp): Landscape, security threats, and future research directions, 2025

    Hou, X., Zhao, Y., Wang, S., and Wang, H. Model context protocol (mcp): Landscape, security threats, and future research directions, 2025. URL https://arxiv.org/abs/2503.23278

    2025 arXiv

  5. [13]

    Understanding the planning of llm agents: A survey, 2024

    Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., and Chen, E. Understanding the planning of llm agents: A survey, 2024. URL https://arxiv.org/abs/2402.02716

    2024 arXiv

  6. [14]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  7. [15]

    Langgraph: Build resilient language agents as graphs

    LangChain AI . Langgraph: Build resilient language agents as graphs. https://github.com/langchain-ai/langgraph, 2024. Accessed: 2026-01-15

    2024

  8. [16]

    The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution, 2025

    Li, J., Zhao, W., Zhao, J., Zeng, W., Wu, H., Wang, X., Ge, R., Cao, Y., Huang, Y., Liu, W., Liu, J., Su, Z., Guo, Y., Zhou, F., Zhang, L., Michelini, J., Wang, X., Yue, X., Zhou, S., Neubig, G., and He, J. The tool decathlon: Benchmarking language agents for diverse, realisti...

    2025

  9. [17]

    Agentbench: Evaluating LLM s as agents

    Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating LLM s as agents. In The Twelfth Internati...

    2024

  10. [18]

    LangChain v0.3

    Mavroudis, V. LangChain v0.3 . working paper or preprint, December 2024. URL https://hal.science/hal-04817573

    2024

  11. [19]

    GAIA : a benchmark for general AI assistants

    Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., and Scialom, T. GAIA : a benchmark for general AI assistants. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=fibxvahvs3

    2024

  12. [20]

    Azure OpenAI service content filtering

    Microsoft. Azure OpenAI service content filtering. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter, 2026. Accessed: 2026-01-16

    2026

  13. [21]

    A survey on large language model benchmarks, 2025

    Ni, S., Chen, G., Li, S., Chen, X., Li, S., Wang, B., Wang, Q., Wang, X., Zhang, Y., Fan, L., Li, C., Xu, R., Sun, L., and Yang, M. A survey on large language model benchmarks, 2025. URL https://arxiv.org/abs/2508.15361

    2025 arXiv

  14. [22]

    Function calling - openai api documentation

    OpenAI. Function calling - openai api documentation. https://platform.openai.com/docs/guides/function-calling, 2026. Accessed: 2026-01-16

    2026

  15. [23]

    OpenAI, :, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., Mądry, A., Baker-Whitcomb, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A. T...

    2024 arXiv

  16. [24]

    G., Stoica, I., and Gonzalez, J

    Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards llms as operating systems, 2024. URL https://arxiv.org/abs/2310.08560

    2024 arXiv

  17. [25]

    S., O'Brien, J., Cai, C

    Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST '23, New York, NY, USA, 2023. Associ...

    2023

  18. [26]

    G., Mao, H., Yan, F., Ji, C

    Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. The berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025 a . UR...

    2025

  19. [27]

    G., Mao, H., Yan, F., Ji, C

    Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. The berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025 b . UR...

    2025

  20. [28]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98

    2024

  21. [29]

    V., Wolf, T., von Werra, L., and Kaunismäki, E

    Roucher, A., del Moral, A. V., Wolf, T., von Werra, L., and Kaunismäki, E. `smolagents`: a smol library to build great agentic systems. https://github.com/huggingface/smolagents, 2025

    2025

  22. [30]

    Significant Gravitas . Autogpt. https://github.com/Significant-Gravitas/AutoGPT, 2023. Accessed: 2026-01-21

    2023

  23. [31]

    Song, Y., Wang, G., Li, S., and Lin, B. Y. The good, the bad, and the greedy: Evaluation of LLM s should not ignore non-determinism. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association f...

    2025 doi

  24. [32]

    Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., Mariooryad, S., Ding, Y., Geng, X., Alcober, F., Frostig, R., Omernick, M., Walker, L., Paduraru, C., Sorokin, C., Tacchetti, A., Gaffney, C., Daruki, S., Sercinog...

    2024 arXiv

  25. [33]

    Gemini: A family of highly capable multimodal models, 2025

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., and others. Gemini: A family of highly capable multimodal models, 2025. URL https://arxiv.org/abs/2312.11805

    2025 arXiv

  26. [34]

    K.-W., and Lim, E.-P

    Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the As...

    2023 doi

  27. [35]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processin...

    2022

  28. [36]

    W., Passos, A

    Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., and Glaese, A. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504.12516

    2025 arXiv

  29. [37]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State...

    2020

  30. [38]

    The rise and potential of large language model based agents: a survey

    Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng, R., Fan, X., Wang, X., Xiong, L., Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng, R., Qin, W., Zheng, Y., Qiu, X., Huang, X., Zhang, Q., and Gui, T. T...

    2025 doi

  31. [39]

    Yao, S., Chen, H., Yang, J., and Narasimhan, K. R. Webshop: Towards scalable real-world web interaction with grounded language agents. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.n...

    2022

  32. [40]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X

    2023

  33. [41]

    Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. R. \ \ tau\ \ -bench: A benchmark for underline\ T\ ool- underline\ A\ gent- underline\ U\ ser interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openre...

    2025

  34. [42]

    A survey on trustworthy llm agents: Threats and countermeasures

    Yu, M., Meng, F., Zhou, X., Wang, S., Mao, J., Pan, L., Chen, T., Wang, K., Li, X., Zhang, Y., An, B., and Wen, Q. A survey on trustworthy llm agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ...

    2025

  35. [43]

    H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J

    Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGL ang: Efficient execution of structured language model programs. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  36. [44]

    M ulti A gent B ench : Evaluating the collaboration and competition of LLM agents

    Zhu, K., Du, H., Hong, Z., Yang, X., Guo, S., Wang, Z., Wang, Z., Qian, C., Tang, R., Ji, H., and You, J. M ulti A gent B ench : Evaluating the collaboration and competition of LLM agents. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Proceedings of the 63r...

    2025 doi

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.