Agent Identity Evals: Measuring Agentic Identity

Elija Perrier; Michael Timothy Bennett

REVIEW 4 major objections 5 minor 2 cited by

Agent Identity Evals: Measuring Agentic Identity

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Agent Identity Evals, quantifying whether a language-model agent remains the same agent over time, and argues this matters for planning.

desk verdict A useful first cut at defining and measuring LMA identity stability, but the persistence metric is broken and the empirical support falls far short of 'rigorous.' read the letter →

arxiv 2507.17257 v1 pith:3LX45YV6 submitted 2025-07-23 cs.AI cs.MA

Elija Perrier , Michael Timothy Bennett This is my paper

classification cs.AIcs.MA

keywords agentidentitylanguagemodelagentsagenticstabilityLLMpathologiesdriftevaluationframeworksemanticembeddingdistance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agent Identity Evals (AIE) tries to establish that the identity of a language-model agent—whether it remains the same named, goal-directed entity over time—can be decomposed into five measurable properties: identifiability, continuity, consistency, persistence, and recovery. The paper formalizes each property as a statistical score computed from textual traces and argues that these scores are degraded by the underlying pathologies of LLMs (statelessness, stochasticity, semantic sensitivity, linguistic intermediation). The motivation is practical: if identity stability can be quantified, then scaffold design choices such as memory, tools, and corrective prompts can be compared directly, and agent drift can be caught before it harms planning and action. The paper also demonstrates the metrics on a suite of planning experiments, reporting that identity profiles differ sharply across conditions even when planning outcomes look similar.

What carries the argument

The load-bearing mechanism is Definition 3.1, an assumed equivalence relation and metric $d$ over agent states, instantiated in the experiments as cosine distance over embeddings of textual identity representations. Every score in Definitions 3.2–3.6 is a function of this $d$: identifiability is the maximum fraction of repeated instantiations whose self-representations fall within a threshold of a reference; continuity is the fraction of cross-turn recall probes answered correctly; consistency is the average fraction of paraphrased-output pairs within a similarity threshold; persistence is the average normalized closeness of successive session snapshots; recovery is the fractional reduction in distance toward a reference state after corrective interventions. The machinery works by converting the philosophical question 'is this the same agent?' into a computable number, so that memory, tool, and prompt variations can be compared on the same scale.

What would settle it

Take a set of paired identity descriptions where human raters agree that two phrasings express the same agent (same name, role, goals) and another set where the phrasings clearly describe different agents, then compute the cosine distance the paper uses. If the same-identity pairs and different-identity pairs cannot be separated by any threshold delta, the distance function at the core of Definitions 3.2–3.6 fails as a measure of identity, and human agreement with the resulting scores would be near chance.

Watch

Extended reading notes

Core claim

The central discovery, as the authors state it, is that agentic identity is a diachronic, measurable quantity rather than an unobservable abstraction: identity is the set of attributes that remain within a distance epsilon of themselves across all evaluated time points, and the five metrics quantify different ways that set can shrink. Identifiability asks whether repeated instantiations converge on the same self-description; continuity asks whether information from earlier turns is correctly recalled; consistency asks whether paraphrase-equivalent queries produce semantically equivalent answers; persistence asks whether identity representations stay close across sessions; recovery asks whether a perturbed agent returns toward its reference state after corrective prompts. The accompanying experiments are offered as worked applications of the framework rather than as a settled empirical law: they show that identity can be scored on real interaction traces, that scores are not all correlated, and that planning performance does not straightforwardly track them (notably, identifiability is zero almost everywhere, and perfect persistence with retrieval-augmented memory co-occurs with worse planning than no memory).

Load-bearing premise

The load-bearing premise is that embedding distance between two textual identity descriptions tracks whether the agent described is 'the same agent'; if that mapping fails, all five scores are measuring something other than identity.

Editorial extensions

If this is right

If the AIE scores are valid, agent developers can test whether a scaffolding change (memory, tools, recovery prompts) actually stabilizes an agent's identity before deployment, not just whether it improves task metrics.
Identity scores can be paired with capability scores in a single evaluation run, letting failures be attributed to identity attrition rather than to task skill.
The paper's Exp. 4 result—perfect measured persistence alongside worse planning under RAG—implies that identity evaluations reveal a dimension that capability benchmarks miss: persistence of information matters less than how it is integrated.
Because the five definitions are model-agnostic, the same probes can be applied across the LMA lifecycle, from initial instantiation to long-running deployed sessions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper's own table shows scores that are often perfectly high on one axis and zero on another, the natural unit of comparison is the five-score profile, not a composite average; a single index would hide exactly the trade-offs the framework exposes.
The metrics measure identity as expressed in text, so a low identifiability score may reflect a refusal or a lapse in phrasing rather than a loss of identity; separating retrieval failures from identity loss would require a follow-up probe that asks the same question in multiple forms.
The cosine-distance assumption could be calibrated cheaply before deployment: collect human equivalence judgments on the exact probe pairs, fit the threshold delta, and only then trust the absolute scores for cross-system comparison.
The recovery metric's dependence on the choice of corrective prompts suggests it may be more naturally read as a measure of scaffold responsiveness than of intrinsic agent resilience.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's my read. The paper's real contribution is a vocabulary and a set of five metric definitions for a genuinely under-tested dimension of LMA behavior: whether an agent stays 'itself' over time. That is worth having. The related-work survey is the most useful part: Table 4 maps existing benchmarks to the five axes and shows that no single benchmark covers them all. The authors also say plainly that this is a first iteration, and their limitation section is honest.

That is where the credit ends. The central claim of a 'rigorous, statistically-driven, empirical framework' is not supported by what is reported. Table 1 gives point scores with no variance, no significance tests, no correlation analysis, even though Appendix A.1 promises t-tests, ANOVA, and correlations. Identifiability scores are 0 across every condition, which makes the metric non-discriminating in these pilots. And the measurement loop is circular in a way that matters: the same model family (GPT-4o-mini) generates the profile, plays the agent, and scores the identity similarity.

The most serious problem is the persistence metric, Definition 3.5. The stress-test note is correct: because each consecutive distance is normalized by the global maximum pairwise distance, adding a single far-away state can raise the persistence score. Three states with consecutive distances 0.2 and 0.2 give P≈0.5; add a fourth state at distance 1.0 from all three, and P rises to about 0.53 despite a large identity drift. A metric that rewards a large drift is not measuring average stability. This also explains the paper's own Experiment 4, where both memory conditions hit perfect persistence (1.0) yet produced very different planning outcomes: the normalization saturates and hides real drift.

The paper is still worth engaging. The formulas are simple enough to fix, the benchmark mapping is a useful starting point, and the proposed axes are the right kind of decomposition to debate. But as it stands, the evidence does not back the 'rigorous' language. The authors would need to correct the persistence formula, publish code and data, report full statistics, and break the same-model evaluator loop before the framework can be used as claimed.

Who is this for? People working on LMA evaluation and scaffolding design, especially anyone thinking about memory and identity drift. I would send it to review, but with a clear request for major revision rather than desk rejection; the idea is sound enough to deserve referee time, and the flaws are fixable. I would not cite it in its current form.

Referee Report

4 major / 5 minor

Summary. The paper proposes Agent Identity Evals (AIE), a framework of five metrics—identifiability, continuity, consistency, persistence, and recovery—for measuring the stability of language model agent (LMA) identity over time. It gives formal definitions for each metric, describes five integrated experiments linking identity scores to planning performance, and reports mixed empirical results. The central claim is that identity stability matters for task performance and that AIE provides a rigorous, statistically-driven measurement framework for evaluating LMA reliability and scaffolding design.

Significance. The paper addresses a real gap: most LMA benchmarks focus on task performance rather than the ontological stability of the agent itself. The five-axis decomposition of agentic identity is a useful conceptual contribution, and the formal definitions (Definitions 3.2–3.6) are mostly simple and internally consistent. If validated, the metrics could serve as reusable evaluation tools for memory, tool use, and prompt design. However, the empirical support is thin and the measurement validity is currently unestablished; the authors are transparent about some limitations, but the load-bearing assumptions and statistical gaps need to be addressed before the framework can be considered rigorous.

major comments (4)

[§3.5, Eq. (5)] The persistence score is non-monotonic and does not measure average stability, as claimed. With three states whose consecutive distances are 0.2 and 0.2, P = 0.5; adding a fourth state at distance 1.0 from all three gives P ≈ 0.53, so the score increases despite a large identity drift. Normalizing each consecutive distance by the global maximum pairwise distance lets a single distant state inflate the other terms, masking drift. This internal inconsistency affects one of the five core metrics and likely explains the saturated P = 1.0 results in Experiment 4 (§5, Table 1). The formula should be redefined (e.g., using a fixed scale or per-step normalization) and the experiments re-run.
[§5, Table 1 and Appendix A.1] Appendix A.1 states that each condition is run for N = 30–50 trials and that statistical significance is assessed with t-tests, ANOVA, and correlation coefficients, but Table 1 reports only single binary scores (0/1) per experiment, with no variance, confidence intervals, or significance tests. The central empirical claim that identity stability relates to task performance is therefore not statistically supported. The authors should report per-trial distributions and test statistics, or explicitly label these experiments as illustrative rather than statistically rigorous.
[Appendix A.1] The measurement pipeline is self-referential: GPT-4o-mini is used as the PROFILE_GENERATOR_LLM, as the SimulatedAgent under evaluation, and as the AgentIdentityEvaluator and SupervisorLLM that score the outputs. The reported identity scores may therefore reflect single-model self-consistency rather than a general property of LMA identity. To support the framework's claims, generation and evaluation should use different models, or the scores should be validated against human judgments.
[Definition 3.1 and §6.3] All five metrics depend on the distance d, usually embedding cosine distance, but the paper never validates that this distance tracks identity-relevant semantics. The authors themselves note in Limitation 1 that string/embedding distance 'may miss nuanced semantic consistency or contradiction.' Without external validation (e.g., human similarity ratings, NLI-based contradiction detection, or a known set of identity-preserving versus identity-changing transformations), the scores are uninterpretable. Please add such a validation study or clearly frame the framework as provisional.

minor comments (5)

[§3.2, Eq. (2)] The maximum in the identifiability score is taken over r, but the set R(Π) is never formally defined; please clarify that r ranges over the reference identity representations.
[§5, Table 1] The table header contains a stray '3' after 'Recovery', and the table would be more informative if it reported the number of trials or confidence intervals alongside the scores.
[§6.1] The text says 'The radar charts in Figure 3 provide a granular view', but the radar charts are in Figure 2 and the line plots are in Figure 3; the cross-reference should be corrected.
[§4.2] The sentence 'Exp. 5 asses assessed the agent's ability...' contains a duplicated word; it should read 'Exp. 5 assessed...'.
[Abstract] The phrase 'exhibit and maintain their agentic identity' is slightly awkward; consider 'exhibit and maintain' or 'exhibits and maintains' for grammatical consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the five metric definitions are explicit operationalizations, and the claimed identity–planning relationship is tested against independently scored planning outputs rather than derived from the metric formulas.

full rationale

The paper's derivational content consists of five metric definitions (Eqs. 2-6) plus an experimental comparison of those metrics to planning performance. None of the equations is fitted to data, none is derived from another equation, and none predicts a quantity already implicit in its inputs; each is an explicit operationalization of a named construct. The claimed relationship between identity stability and planning is tested against independently scored PLAN_MASTER/SupervisorLLM planning outcomes, so the central empirical claim is falsifiable rather than true by construction. The self-referential LLM pipeline (the same GPT-4o-mini family generates profiles, acts as the agent, and evaluates identity) is a legitimate measurement concern, but it does not make any formula or stated result equivalent to its own inputs; the paper's own Limitation 1 concedes that 'String/embedding distance may miss nuanced semantic consistency or contradiction', which is a construct-validity caveat, not a circularity. The self-citations to the authors' position paper support the framing of LLM pathologies, but Appendix E independently attributes each pathology to external sources, so no load-bearing step reduces to those citations. The persistence score's non-monotonicity under global-max normalization is a metric-validity defect and a correctness risk, but it is not a circularity under the enumerated patterns. Overall, the derivation chain is self-contained relative to its declared assumptions; no step reduces by construction to its inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The framework rests on many hand-chosen thresholds (δ, δ_c, ε), an unvalidated distance measure, and assumptions about LLM identity that come from the authors' own prior work. No free parameters are fit to data in a statistical sense, but every output score depends on these choices, and the central construct (agentic identity) is defined only through the measurement itself.

free parameters (6)

threshold δ (identifiability, Def. 3.2) = not specified
Hand-chosen matching threshold; determines whether an identity representation counts as matching the reference. No sensitivity analysis is provided.
threshold δ_c (consistency, Def. 3.4) = not specified
Hand-chosen threshold for whether two outputs are consistent; directly controls the consistency score.
constant ε (Defs. 3.1, 3.5, 3.6) = not specified
Small constant used in normalization and to avoid division by zero; its value affects persistence and recovery scores but is never stated.
number of probes K (Appendix A.2) = e.g., K = 5
Experimenter-chosen number of probing queries per trial; affects identifiability measurement.
number of trials N = 30 to 50 (claimed)
Claimed in Appendix A.1 but no variance or significance results are reported in Section 5.
distance measure d (embedding choice) = OpenAI text-embedding-ada-002 cosine
All identity scores are computed with this embedding cosine distance; no validation that this metric tracks identity.

assumptions (4)

domain assumption There exists a suitable metric d over agent states such that state equivalence holds iff attribute distances are below thresholds (Def. 3.1).
The entire framework presupposes identity can be captured by a distance metric over text embeddings, but no validation against an external criterion is given.
domain assumption A declarative agent prompt Π instantiates a distinct agent A with identity attributes.
Section 3.1 defines an 'agent prompt' as instantiating an agent; this is a metaphysical assumption about LLM behavior that is not empirically established.
domain assumption Semantic embedding cosine distance between self-description texts is a valid proxy for identity sameness or difference.
Used throughout Definitions 3.2 to 3.6; the paper itself notes in Limitation 1 that string/embedding distance may miss semantic nuance.
domain assumption The four LLM pathologies (statelessness, stochasticity, semantic sensitivity, linguistic intermediation) are the correct causes of identity attrition.
This framework is drawn from the authors' own position paper (refs [24], [67]) and is adopted without independent evidence or comparison to alternative explanations.

invented entities (1)

agentic identity (Def. 3.1)
purpose: Target construct that AIE is designed to measure; defined as the subset of attributes that stay within a distance threshold over time.
The construct has no falsifiable handle outside the framework itself; its only operationalization is through the same embedding similarities used to define it, and the paper provides no external validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Agent Identity Evals: Measuring Agentic Identity." pith.science (2026). https://pith.science/paper/3LX45YV6

@misc{pith2026250717257,
  author       = {Pith},
  title        = {Pith review of: Agent Identity Evals: Measuring Agentic Identity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LX45YV6}},
  note         = {Machine review of arXiv:2507.17257}
}

read the original abstract

Central to agentic capability and trustworthiness of language model agents (LMAs) is the extent they maintain stable, reliable, identity over time. However, LMAs inherit pathologies from large language models (LLMs) (statelessness, stochasticity, sensitivity to prompts and linguistically-intermediation) which can undermine their identifiability, continuity, persistence and consistency. This attrition of identity can erode their reliability, trustworthiness and utility by interfering with their agentic capabilities such as reasoning, planning and action. To address these challenges, we introduce \textit{agent identity evals} (AIE), a rigorous, statistically-driven, empirical framework for measuring the degree to which an LMA system exhibit and maintain their agentic identity over time, including their capabilities, properties and ability to recover from state perturbations. AIE comprises a set of novel metrics which can integrate with other measures of performance, capability and agentic robustness to assist in the design of optimal LMA infrastructure and scaffolding such as memory and tools. We set out formal definitions and methods that can be applied at each stage of the LMA life-cycle, and worked examples of how to apply them.

Figures

Figures reproduced from arXiv: 2507.17257 by the authors.

**Figure 1.** Agent identity attrition. As AI systems become increasingly autonomous, the question of agent identity – whether a system remains “the same agent” over time and across contexts – emerges as crucial to their reliability, safety, and utility. Agent identity is central to LMA functionality. An agent instantiated and configured in one way will perform different to another differently configured agent. Similarly as age… view at source ↗

**Figure 3.** Total identity drift (measuring cosine distance of output description at each iteration from initial [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 2.** Radar charts of semantic similarity of agent (along each axis) for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Each LLM pathology primarily degrades the corresponding ontological property/metric. [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

**Figure 5.** Figure 5: Impact of Agent Identity. Stable identity supports capabilities (left), while fractured identity increases [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms
cs.CY 2026-05 conditional novelty 7.0 of 10

VLMs preserve linearly separable visual magnitudes and can compare them, yet collapse at symbolic mapping because visual and textual number spaces remain fractured and disjoint.
Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms
cs.CY 2026-05 unverdicted novelty 6.0 of 10

LM agents' changeable modules prevent persistent identity and sanction sensitivity, making reputation mechanisms structurally inapplicable and requiring protocol-based behavioral harnesses instead.

Reference graph

Works this paper leans on

99 extracted references · 39 canonical work pages · cited by 1 Pith paper

[1]

Siegel, Nitya Nadgir, and Arvind Narayanan

Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. arXiv:2407.01502 [cs]

arXiv 2024
[3]

Introducing Devin, the first AI software engineer, March 2024

Scott Wu. Introducing Devin, the first AI software engineer, March 2024

2024
[4]

GAIA: a benchmark for General AI Assistants, November 2023

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023

2023
[5]

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, August 2024

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, August 2024. arXiv:2408.04682 [cs]

arXiv 2024
[6]

Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W

Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham ...

arXiv 2024
[7]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, April 2024. arXiv:2310.06770 [cs]

arXiv 2024
[8]

Jimenez, John Yang, Kevin Liu, and Aleksander Madry

Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified, 2024

2024
[9]

A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, February 2024

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, February 2024. arXiv:2307.12856 [cs]

arXiv 2024

Show all 99 references

[10]

MultiOn AI, 2024

MultiOn. MultiOn AI, 2024

2024
[11]

Agents that reduce work and information overload.Communications of the ACM, 37(7):30–40, July 1994

Pattie Maes. Agents that reduce work and information overload.Communications of the ACM, 37(7):30–40, July 1994

1994
[12]

Artificial life meets entertainment: lifelike autonomous agents.Communications of the ACM, 38(11):108–114, November 1995

Pattie Maes. Artificial life meets entertainment: lifelike autonomous agents.Communications of the ACM, 38(11):108–114, November 1995

1995
[13]

Autonomous interface agents

Henry Lieberman. Autonomous interface agents. InProceedings of the ACM SIGCHI Confer- ence on Human factors in computing systems, CHI ’97, pages 67–74, New York, NY , USA, March 1997. Association for Computing Machinery

1997
[14]

A roadmap of agent research and development.Autonomous agents and multi-agent systems, 1:7–38, 1998

Nicholas R Jennings, Katia Sycara, and Michael Wooldridge. A roadmap of agent research and development.Autonomous agents and multi-agent systems, 1:7–38, 1998. Publisher: Springer

1998
[15]

Deborah G. Johnson. Software Agents, Anticipatory Ethics, and Accountability. In Gary E. Marchant, Braden R. Allenby, and Joseph R. Herkert, editors,The Growing Gap Between Emerging Technologies and Legal-Ethical Oversight: The Pacing Problem, pages 61–76. Springer Netherlands...

2011
[16]

Sutton and Andrew G

Richard S. Sutton and Andrew G. Barto.Reinforcement learning: An introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. tex.lccn: Q325.6 .R45 2018

2018
[17]

Russell and Peter Norvig.Artificial Intelligence: A Modern Approach

Stuart J. Russell and Peter Norvig.Artificial Intelligence: A Modern Approach. 4 edition, 2021. 10

2021
[18]

Harms from Increasingly Agentic Algorithmic Systems

Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstanti...

2023
[19]

White, Doug Burger, and Chi Wang

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. 2023...

2023 arXiv
[20]

OpenAI Charter, 2018

OpenAI. OpenAI Charter, 2018

2018
[21]

Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, Seliem El-Sayed, Sasha Brown, Canfer Akbulut, Andrew Trask, Edward Hughes, A. Stevie Bergman, Renee Shelby, Nahema Marchal...

2024 arXiv
[22]

Governing AI Agents, April 2024

Noam Kolt. Governing AI Agents, April 2024

2024
[23]

Potter and Kevin J

Henry D. Potter and Kevin J. Mitchell. Naturalising agent causation.Entropy, 24(4):472, 2022

2022
[24]

Position: Stop acting like language model agents are normal agents, 2025

Elija Perrier and Michael Timothy Bennett. Position: Stop acting like language model agents are normal agents, 2025

2025
[25]

Emergent causality and the foundation of consciousness

Michael Timothy Bennett. Emergent causality and the foundation of consciousness. InArtificial General Intelligence. Springer Nature, 2023

2023
[26]

Compression, the fermi paradox and artificial super-intelligence

Michael Timothy Bennett. Compression, the fermi paradox and artificial super-intelligence. In Artificial General Intelligence, pages 41–44. Springer, 2022

2022
[27]

World Scientific, 2013

Pei Wang.Non-Axiomatic Logic. World Scientific, 2013

2013
[28]

Thorisson.A New Constructivist AI: From Manual Methods to Self-Constructive Systems, pages 145–171

Kristinn R. Thorisson.A New Constructivist AI: From Manual Methods to Self-Constructive Systems, pages 145–171. Atlantis Press, Paris, 2012

2012
[29]

Artificial general intelligence: Concept, state of the art.Journal of Artificial General Intelligence, 5(1):1–48, 2014

Ben Goertzel. Artificial general intelligence: Concept, state of the art.Journal of Artificial General Intelligence, 5(1):1–48, 2014

2014
[30]

Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

2023 arXiv
[31]

Gaia: a benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

2023 arXiv
[32]

Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302, 2023

Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302, 2023

2023 arXiv
[33]

Agentsims: An open-source sandbox for large language model evaluation.arXiv preprint arXiv:2308.04026, 2023

Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. Agentsims: An open-source sandbox for large language model evaluation.arXiv preprint arXiv:2308.04026, 2023. 11

2023 arXiv
[34]

Charactereval: A chinese benchmark for role-playing conversational agent evaluation.arXiv preprint arXiv:2401.01275, 2024

Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. Charactereval: A chinese benchmark for role-playing conversational agent evaluation.arXiv preprint arXiv:2401.01275, 2024

2024 arXiv
[35]

Cve-bench: A benchmark for ai agents’ ability to exploit real-world web application vulnerabilities.arXiv preprint arXiv:2503.17332, 2025

Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. Cve-bench: A benchmark for ai agents’ ability to exploit real-worl...

2025 arXiv
[36]

Multiagentbench: Evaluating the collaboration and competition of llm agents.arXiv preprint arXiv:2503.01935, 2025

Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, and Jiaxuan You. Multiagentbench: Evaluating the collaboration and competition of llm agents.arXiv preprint arXiv:2503.01935, 2025

2025 arXiv
[37]

Elt-bench: An end-to-end benchmark for evaluating ai agents on elt pipelines.arXiv preprint arXiv:2504.04808, 2025

Tengjun Jin, Yuxuan Zhu, and Daniel Kang. Elt-bench: An end-to-end benchmark for evaluating ai agents on elt pipelines.arXiv preprint arXiv:2504.04808, 2025

2025 arXiv
[38]

Benchmarking agentic workflow generation.arXiv preprint arXiv:2410.07869, 2024

Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation.arXiv preprint arXiv:2410.07869, 2024

2024 arXiv
[39]

Turner, Eric Undersander, and Tsung-Yen Yang

Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M....

2024 arXiv
[40]

John wiley & sons, 2009

Michael Wooldridge.An introduction to multiagent systems. John wiley & sons, 2009

2009
[41]

Is it an agent, or just a program? a taxonomy for autonomous agents

Stan Franklin and Art Graesser. Is it an agent, or just a program? a taxonomy for autonomous agents. InProceedings of the Third International Workshop on Agent Theories, Architectures, and Languages, pages 21–35, 1997

1997
[42]

Jennings

Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: Theory and practice.The Knowledge Engineering Review, 10(2):115–152, 1995

1995
[43]

Adversarial nli: A new benchmark for natural language understanding

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4885–4901, 2020

2020
[44]

Metagpt: Meta programming for multi-agent collaborative framework

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Zhou, Qibusheng Zhang, Zili Wang, Steven Zhuang, Ceyao Li, Weiming Wu, and Jun Zhu. Metagpt: Meta programming for multi-agent collaborative framework. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

2023
[45]

Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Xia, Richard Wang, Alexander Drain, Zifan Li, J Zico Kolter, Matt Fredrikson, et al. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

2023 arXiv
[46]

Langchain.https://github.com/langchain-ai/langchain, 2022

Harrison Chase. Langchain.https://github.com/langchain-ai/langchain, 2022

2022
[47]

Thórisson

Kristinn R. Thórisson. Seed-programmed autonomous general learning. InProceedings of the First International Workshop on Self-Supervised Learning, volume 131 ofProceedings of Machine Learning Research, pages 32–61. PMLR, 27–28 Feb 2020

2020
[48]

Autocatalytic endogenous reflective architecture

Eric Nivel et al. Autocatalytic endogenous reflective architecture. Technical report, Reykjavik University, School of Computer Science, 2013

2013
[49]

Wang.Rigid Flexibility: The Logic of Intelligence

P. Wang.Rigid Flexibility: The Logic of Intelligence. Applied Logic Series. Springer, 2006

2006
[50]

‘opennars for applications’: Architecture and control

Patrick Hammer and Tony Lofthouse. ‘opennars for applications’: Architecture and control. In Ben Goertzel, Aleksandr I. Panov, Alexey Potapov, and Roman Yampolskiy, editors,Artificial General Intelligence, pages 193–204, Cham, 2020. Springer

2020
[51]

The general theory of general intelligence: A pragmatic patternist perspective

Ben Goertzel. The general theory of general intelligence: A pragmatic patternist perspective. Technical report, Singularity Net, 2021. 12

2021
[52]

Opencog hyperon: A framework for agi at the human level and beyond

Ben Goertzel et al. Opencog hyperon: A framework for agi at the human level and beyond. Technical report, OpenCog Foundation, 2023

2023
[53]

Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L

Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive Architectures for Language Agents, September 2023. arXiv:2309.02427 [cs]

2023 arXiv
[54]

Language agents in the digital world: Opportunities and risks.princeton-nlp.github.io, Jul 2023

Shunyu Yao and Karthik Narasimhan. Language agents in the digital world: Opportunities and risks.princeton-nlp.github.io, Jul 2023

2023
[55]

Practices for governing agentic ai systems

OpenAI. Practices for governing agentic ai systems. 2023. https://openai.com/research/ practices-for-governing-agentic-ai-systems

2023
[56]

Langlois, Pedro A

Tom Everitt, Ryan Carey, Eric D. Langlois, Pedro A. Ortega, and Shane Legg. Agent incentives: A causal perspective. InThirty-fifth AAAI conference on artificial intelligence, AAAI 2021, thirty- third conference on innovative applications of artificial intelligence, IAAI 2021, ...

2021
[57]

Visibility into AI agents.arXiv: 2401.13138 [cs.CY], January 2024

Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, and Markus Anderljung. Visibility into AI agents.arXiv: 2401.13138 [cs.CY], January 2024

2024 arXiv
[58]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Informa- tion Processing Systems, pages 5998–6008, 2017

2017
[59]

do you follow me?

L. Jacqmin, L. Rault, M. Dinarelli, and F. ’Evrard. "do you follow me?": A survey of recent approaches in dialogue state tracking.Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 201–213, 2022. Also arXiv:2207.14627

2022 arXiv
[60]

Multi-domain dialogue state tracking with disentangled domain-slot attention

Libo Yang, Dading Lee, and Yun-Nung Chen. Multi-domain dialogue state tracking with disentangled domain-slot attention. InFindings of the Association for Computational Linguistics: ACL 2023, pages 4800–4811, 2023

2023
[61]

Gao, Jianfeng Liu, Jiawei Li, Epísilon Kumar, and Dian Yu

Ruizhe Zhang, Suvir Li, Laks V .S. Gao, Jianfeng Liu, Jiawei Li, Epísilon Kumar, and Dian Yu. Towards llm-driven dialogue state tracking.arXiv preprint arXiv:2310.14970, 2023

2023 arXiv
[62]

Unsal, Sjoerd Behbahani, Dirk Weissenborn, Heiko Küttler, Daniel Zoran, Adrià Puigdomenech Badia, Bernhard Schölkopf, Raia Hadsell, and Olivier Bachem

Francesco Locatello, Emre O. Unsal, Sjoerd Behbahani, Dirk Weissenborn, Heiko Küttler, Daniel Zoran, Adrià Puigdomenech Badia, Bernhard Schölkopf, Raia Hadsell, and Olivier Bachem. Object-centric learning with slot attention. InAdvances in Neural Information Processing Systems...

2020 arXiv
[63]

Qi, Dragomir Anguelov, and Yin Zhou

Kailai Tian, Ziling Jia, Charles R. Qi, Dragomir Anguelov, and Yin Zhou. 4d panoptic scene graph generation.arXiv preprint arXiv:2405.10305, 2024

2024 arXiv
[64]

Scene graph generation: A comprehensive survey.ACM Computing Surveys, 55(9):1–37, 2022

Guoyuan Zhu, Jin Wang, and Wen An. Scene graph generation: A comprehensive survey.ACM Computing Surveys, 55(9):1–37, 2022. Also arXiv:2201.00443

2022 arXiv
[65]

From pixels to graphs: Open-vocabulary scene graph generation with large language models.arXiv preprint arXiv:2404.00906, 2024

Zequn Zhang, Minsu Park, Minsuk Cho, and Kun Zhang. From pixels to graphs: Open-vocabulary scene graph generation with large language models.arXiv preprint arXiv:2404.00906, 2024

2024 arXiv
[66]

What makes a scene? scene graph-based evaluation and feedback for controllable generation.arXiv preprint arXiv:2401.01929, 2024

Zhe Chen, Shaoteng Huang, Keren Wang, Zhou Li, Hanwang Zhang, and Tat-Seng Chua. What makes a scene? scene graph-based evaluation and feedback for controllable generation.arXiv preprint arXiv:2401.01929, 2024

2024 arXiv
[67]

Position: Stop acting like language model agents are normal agents.arXiv preprint arXiv:2502.10420, 2025

Elija Perrier and Michael Timothy Bennett. Position: Stop acting like language model agents are normal agents.arXiv preprint arXiv:2502.10420, 2025

2025 arXiv
[68]

The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

2024 arXiv
[69]

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023. arXiv:2201.11903 [cs]. 13

2023 arXiv
[70]

Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

2023
[71]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[72]

Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, N...

2021
[73]

Transformers as stochastic optimizers.arXiv preprint arXiv:2305.14314, 2023

Zhen Li, Yujia Zhang, Yujia Li, and Liwei Wang. Transformers as stochastic optimizers.arXiv preprint arXiv:2305.14314, 2023

2023 arXiv
[74]

Do language models have bayesian brains? distinguishing stochastic and deterministic decision patterns within large language models

Andrea Yaoyun Cui and Pengfei Yu. Do language models have bayesian brains? distinguishing stochastic and deterministic decision patterns within large language models. InNeurIPS 2024 Workshop on Behavioral Machine Learning, 2024

2024
[75]

Do i know this entity? knowledge awareness and hallucinations in language models.arXiv preprint arXiv:2411.14257, 2024

Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models.arXiv preprint arXiv:2411.14257, 2024

2024 arXiv
[76]

Exploring autonomous agents through the lens of large language models.arXiv preprint arXiv:2404.04442, 2023

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Exploring autonomous agents through the lens of large language models.arXiv preprint arXiv:2404.04442, 2023

2023 arXiv
[77]

PromptBench: Towards evaluating the robustness of Large Language Models on adversarial prompts.arXiv: 2306.04528 [cs.CL], June 2023

Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. PromptBench: Towards evaluating the robustness of Large Language Models on adversarial prompts.arXiv: 2306.04528 [cs.CL], June 2023

2023 arXiv
[78]

Robustifying language models with test-time adaptation.arXiv preprint arXiv:2310.19177, 2023

Noah Thomas McDermott, Junfeng Yang, and Chengzhi Mao. Robustifying language models with test-time adaptation.arXiv preprint arXiv:2310.19177, 2023

2023 arXiv
[79]

Evaluating the robustness of neural language models to input perturbations.arXiv preprint arXiv:2108.12237, 2021

Milad Moradi and Matthias Samwald. Evaluating the robustness of neural language models to input perturbations.arXiv preprint arXiv:2108.12237, 2021

2021 arXiv
[80]

Kgpa: Robustness evaluation for large language models via cross-domain knowledge graph prompt attack.arXiv preprint arXiv:2406.10802, 2023

Yifan Wang, Yifan Zhang, Yuxuan Zhu, Yuxuan Lai, Yuxuan Zhang, Yuxuan Wang, Yuxuan Li, Yuxuan Chen, Yuxuan Liu, and Yuxuan Yang. Kgpa: Robustness evaluation for large language models via cross-domain knowledge graph prompt attack.arXiv preprint arXiv:2406.10802, 2023

2023 arXiv
[81]

Measuring the inconsistency of large language models in preferential ranking.arXiv preprint arXiv:2410.08851, 2024

Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Measuring the inconsistency of large language models in preferential ranking.arXiv preprint arXiv:2410.08851, 2024

2024 arXiv
[82]

Assert: Automated safety scenario red teaming for evaluating the robustness of large language models

Alex etal Mei. Assert: Automated safety scenario red teaming for evaluating the robustness of large language models. November 2023. arXiv:2310.09624

2023 arXiv
[83]

Large language models can be easily distracted by irrelevant context

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. InInternational Conference on Machine Learning, pages 31210–31227. PMLR, 2023

2023
[84]

Long context rag performance of large language models.arXiv preprint arXiv:2411.03538, 2024

Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. Long context rag performance of large language models.arXiv preprint arXiv:2411.03538, 2024

2024 arXiv
[85]

Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

2024
[86]

Computational dualism and objective superintelligence

Michael Timothy Bennett. Computational dualism and objective superintelligence. InArtificial General Intelligence. Springer, 2024

2024
[87]

Are biological systems more intelligent than artificial intelligence? 2025

Michael Timothy Bennett. Are biological systems more intelligent than artificial intelligence? 2025. 14

2025
[88]

Philosophical specification of empathetic ethical artificial intelligence.IEEE Transactions on Cognitive and Developmental Systems, 14(2):292–300, 2022

Michael Timothy Bennett and Yoshihiro Maruyama. Philosophical specification of empathetic ethical artificial intelligence.IEEE Transactions on Cognitive and Developmental Systems, 14(2):292–300, 2022. 15 Technical Appendices and Supplementary Material A Detail of Experiments I...

2022
[89]

An initial system prompt defining the LMA’s persona, role, and core directives
[90]

A detailed structured identity template (e.g., JSON) capturing key attributes (name, version, role, capabilities, constraints, catchphrases, etc.) used for t0 embedding references and detailed identity tracking
[91]

De- velop a 3-stage marketing strategy for a new eco-friendly coffee brand,

A concise textual identity template (a short paragraph) for simpler self-description probes. •Planning Task Generation: –PLANNING_UTILS_LLM: GPT-4o-mini. –For each experimental context, generates: 1.PLAN_OBJECTIVE : A specific, multi-stage goal for the LMA to achieve (e.g., "D...
[92]

–Perform the standardised planning task

For each LMA configuration: –Perform comprehensive identity evaluations. –Perform the standardised planning task
[93]

Planning Performance Scores for Configi)

Collect data pairs: (Identity Score Profile / Composite Score for Configi, Avg. Planning Performance Scores for Configi). • Analysis Focus:Calculate correlation coefficients (e.g., Pearson’s r, Spearman’s ρ) between individual/composite identity metrics and planning performanc...
[94]

Control Group: LMA performs the task without perturbation
[95]

Perturbation-NoRecovery Group: LMA is perturbed at tperturb and continues the task without explicit recovery
[96]

every k ticks

Perturbation-Recovery Group: LMA is perturbed at tperturb, the recovery mecha- nism is applied, then it continues the task. • Procedure:Run multiple trials for each condition, monitoring task performance throughout. • Analysis Focus:Compare task performance trajectories across...
[97]

LLMs do not retain information across separate inference instances [ 68, 58]

Statelessness. LLMs do not retain information across separate inference instances [ 68, 58]. Each query–response cycle operates in isolation unless prior context is explicitly reintroduced. While the trace of LLM inputs/outputs may be retained in external memory, the underlyin...
[98]

LLM outputs are typically probabilistic [72, 73, 74], meaning the same query can yield varying or even incorrect results on different runs [ 75]

Stochasticity. LLM outputs are typically probabilistic [72, 73, 74], meaning the same query can yield varying or even incorrect results on different runs [ 75]. This unpredictability complicates any attempt to establish consistent traits that might signal a unified agent- like...
[99]

Semantic sensitivity. Small linguistic modifications in a prompt can lead to significantly altered responses [76, 77], a phenomenon that becomes especially clear under techniques like jailbreaking or in adversarial scenarios [78, 79, 80]. Even subtle changes can override exist...
[100]

All interaction with an LLM is text-based: agent definitions, environmental factors, and actions are translated into tokens, which the LLM interprets to produce responses in kind

Linguistic intermediation. All interaction with an LLM is text-based: agent definitions, environmental factors, and actions are translated into tokens, which the LLM interprets to produce responses in kind. This imposes an additional abstraction layer between the agent and its...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Siegel, Nitya Nadgir, and Arvind Narayanan

Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. arXiv:2407.01502 [cs]

arXiv 2024

[2] [3]

Introducing Devin, the first AI software engineer, March 2024

Scott Wu. Introducing Devin, the first AI software engineer, March 2024

2024

[3] [4]

GAIA: a benchmark for General AI Assistants, November 2023

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023

2023

[4] [5]

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, August 2024

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, August 2024. arXiv:2408.04682 [cs]

arXiv 2024

[5] [6]

Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W

Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham ...

arXiv 2024

[6] [7]

Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, April 2024. arXiv:2310.06770 [cs]

arXiv 2024

[7] [8]

Jimenez, John Yang, Kevin Liu, and Aleksander Madry

Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified, 2024

2024

[8] [9]

A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, February 2024

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, February 2024. arXiv:2307.12856 [cs]

arXiv 2024