Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

The paper claims an agent is a Python object—methods as actions, docstrings as prompts, type annotations as contracts—and shows current models use it fluently enough to replace multi-component agent stacks without losing benchmark performan

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:34 UTC pith:L5HRLQKH

load-bearing objection Genuine engineering with solid capability evidence; the ARC-AGI-3 '6.4x harness effect' is an uncontrolled comparison and should be reframed before publication. the 3 major comments →

arxiv 2607.20709 v1 pith:L5HRLQKH submitted 2026-07-22 cs.AI cs.CL

NVIDIA-labs OO Agents: Native Python Object-Oriented Agents

Paul Furgale , Severin Klingler , James Nolan , Matt Staats , Gaia Di Lorenzo , Elisa Martinez Abad , Christian Sch\"uller , Razvan Dinu
show 7 more authors
Alessio Devoto Pascal Berard Gal Kaplun Elad Sarafian Riccardo Roveri Leon Derczynski Ricardo Silveira Cabral
This is my paper
classification cs.AI cs.CL
keywords agent-as-a-python-objectobject-oriented agentscode-as-actionpass-by-referencetyped contractsagentic loopslong-term memoryARC-AGI-3
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that agent development has overcomplicated itself: instead of prompt templates, tool schemas, callback stacks, and workflow graphs, an agent can be a single Python class. The paper's central claim is that language models already know this language, so the same object can serve as source code, prompt, type contract, tool interface, and state boundary. The authors support it with three results: 97.9% average pass rate on 88 targeted capability tests across ten models, 82.2% on SWE-bench Verified with a frontier model, and 85.1% on the interactive reasoning benchmark ARC-AGI-3 with a single agent plus a 50-line skill, compressing a previous multi-agent world-model system. If right, this removes the learning curve of agent frameworks and makes agent behavior testable, traceable, and refactorable like ordinary software.

Core claim

The discovery is the agent-as-a-Python-object programming model. A class mixes deterministic methods, whose bodies run as normal Python, with 'agentic' methods, whose ellipsis bodies are completed at runtime by an LLM-driven loop. The signature supplies typed inputs and a validated return; the docstring becomes the prompt; methods and state on self become callable tools; and large arguments are passed by reference with bounded previews instead of being serialized into context. On this surface the paper reports that current models are already fluent—97.9% on the capability suite, with failures concentrated in long-horizon stress cases rather than interface understanding—and that this ordinary

What carries the argument

The load-bearing object is the Agent class itself, whose two built-in strategies define the loop: PredictStrategy makes a single typed LLM call with output validation and local retry, and CodeActStrategy runs an iterative Python REPL in which the model writes and executes code, inspects live objects, and must finish with a validated return. Around this, bounded-variable previews implement pass-by-reference at scale, ContextManager and EventManager expose static/dynamic context and typed event history as model-callable APIs, and the optional MemoryManager lets the agent curate its own SQLite store, retrieved by a relevance/recency/importance activation score. The claim is that these six model

Load-bearing premise

The headline ARC-AGI-3 comparison pits the NOOA agent at a large interactive budget against the raw model at a different, smaller evaluation budget, so if unequal budgets explain the 6.4x gap, the claim that the interface itself advances the score–cost Pareto frontier does not hold.

What would settle it

Rerun the same 25 ARC-AGI-3 games for the raw frontier model at the NOOA fleet's exact budget—same 2-hour wall clock, same tokens for input/output/cache, same per-turn retry allowance—and record RHAE; if the raw model approaches 85.1%, the 'harness effect' collapses. Separately, run SWE-bench and Terminal-Bench with 5–10 repeated seeds per harness and report confidence intervals; if the NOOA margin over the two open general-purpose agents falls inside the overlap, the reported advantage is sampling noise.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent behavior becomes unit-testable, traceable, refactorable software: docstrings, signatures, and deterministic helpers replace prompt engineering, so the same CI practices apply to agents.
  • Scaling past the context window: because arguments are live objects with bounded previews, the prompt carries fixed-size views while the agent processes full multi-million-row tables in code.
  • Fewer false completions: validated typed returns turn termination into a programmatic contract, reducing premature 'done' responses in interactive benchmarks.
  • The ARC-AGI-3 compression result suggests multi-agent orchestration can sometimes be replaced by a single agent with a skill and memory, changing how world-model systems are built.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 6.4x ARC-AGI-3 harness effect is computed against a raw-model evaluation with different budgets; a matched-budget rerun (same wall-clock, tokens, and turn allowance) would tell whether the interface itself, rather than the evaluation scale, drives the gain.
  • The stress-test failure pattern—models transcribing results instead of returning the live variable—points to a fixable mismatch between model habits and the interface; trajectory-level RL or preference data over this surface could plausibly close most of the remaining 2% capability gap.
  • If self-curated SQLite memory generalizes beyond ARC-AGI-3, it opens a route to cross-task transfer for frozen-weight agents, effectively accumulating skill without parameter updates—an implication the paper gestures at but does not establish.
  • The six-capability taxonomy could serve as a neutral checklist for comparing future agent harnesses, since the paper's own survey shows the field converging on these features piecemeal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. NOOA is a Python agent framework built on the idea that an agent is a Python object: methods are actions, fields are state, docstrings are prompts, and type annotations are contracts. Methods whose body is '...' are executed by an LLM loop under Predict or CodeAct strategies; context, events, and an optional long-term memory subsystem are exposed as Python APIs. The paper's contributions are the programming model, a claim to be the first to combine six model-facing capabilities on one surface, and an empirical evaluation: 97.9% on 88 self-authored capability tests (4,400 records across 10 models), 82.2% on SWE-bench Verified with GPT-5.5 xhigh, 73.0% on Terminal-Bench 2.0, 86.8% on CyberGym L1, and an ARC-AGI-3 fleet at 85.1% RHAE vs 13.3% for raw GPT-5.6-sol, described as a '6.4x harness effect'. A 14-framework comparison and detailed appendices on harness scoring, memory design, and the ARC containment audit are included.

Significance. The paper is a serious systems contribution with a substantial and largely transparent evaluation on the capability side. If the benchmark claims held, the contribution would be important: a single compact object-oriented surface would let developers and models share one programming model while matching or beating specialized agent stacks. The paper deserves credit for shipping the source, pinning framework snapshots, reporting 4,400 capability records, and including a red-team audit for the ARC runs. However, the strongest interface-specific claims — the '6.4x harness effect' and the Pareto-frontier advantage over OpenCode/PI — rest on comparisons that are either not controlled (ARC baseline) or reported without uncertainty (SWE-bench, Terminal-Bench, CyberGym). The central empirical case is defensible but needs revision before those stronger claims can be accepted.

major comments (3)
  1. [Sec 4.4, Fig. 7, footnote 2] The headline '6.4x harness effect' (85.1% vs 13.3% RHAE) is not a controlled comparison. The 13.3% baseline is ARC Prize's raw-model evaluation, which has no agent loop, no tools, and no skill; footnote 2 concedes that evaluation budgets differ. The other curves in Fig. 7 are all NOOA variants, so none isolates the OO interface from the 50-line world-model skill or the memory subsystem. As reported, the result shows that a harness plus this skill plus memory reaches 85.1%, not that the OO interface itself advances the Pareto frontier. A matched run of the same skill in a non-OO CodeAct harness (e.g., smolagents) or in a JSON-tool harness is needed; without it, the paper's strongest interface-specific claim is not established.
  2. [Sec 4.2-4.4, Tables 3-5, Fig. 6-7] Benchmark scores are reported as single point estimates with no confidence intervals or repeated-seed variance. On SWE-bench (Table 3) NOOA's advantage over OpenCode/PI is 3-5 points; on Terminal-Bench (Table 4) it ranges from -2.3 to +12.3 points depending on configuration; the ARC ablations (Fig. 7) are single 25-game fleets. Known run-to-run variability on these benchmarks is comparable to or larger than several of these gaps, so the claims that NOOA 'defines most of the observed accuracy-cost frontier' and that memory contributes +11.8 RHAE points are not yet supported at the reported precision. Please provide repeated runs (at least 3-5 seeds or fleets) with intervals for the key comparisons, or explicitly state that only single runs were performed and soften the frontier claims.
  3. [Sec 4.1, Table 2] The stress-test analysis rests on 300 records total, with only 20 per cell for the small/efficient group (e.g., sentiment_batch 8/20, refinement 11/20). The conclusion that large models show only intermittent failures while small models exhibit systematic 0/5 failures is based on very few observations per model family. The qualitative reading of Appendix B is plausible and well illustrated, but the quantitative claims ('scale gap widens to 23 points', '12.5% of small-model stress pairs at 0/5') should be accompanied by uncertainty estimates or additional runs before being used to support design conclusions.
minor comments (5)
  1. [Table 4] The duplicate values in the GPT-5.5 row (73.0 for both high and xhigh) and the Opus row (64.0 and 65.2) should be verified; if deliberate, a note would avoid the appearance of a formatting artifact.
  2. [Table 5] The 'MDASH' placeholders in system names (e.g., 'Microsoft MDASHv2', 'MDASHunknown') are formatting artifacts and should be replaced with actual em-dashes or labels.
  3. [Header before abstract] The repository link '/githubnvidia-nemo/labs-OO-Agents' lacks a URL scheme; it should be 'https://github.com/nvidia-nemo/labs-OO-Agents' or similar.
  4. [Sec 4.1, final sentence] The phrase 'completely remove the interface friction' overstates the evidence: the stress-test pass rate is 84.7%, and even frontier models show intermittent failures. Suggest softening to 'substantially reduce interface friction'.
  5. [Sec 7, Limitations] The claim that models perform well 'despite never being trained on it' cannot be verified from the paper; models are trained on Python, CodeAct patterns, and similar tool-use formats. Suggest rephrasing to 'not trained on this specific interface' and pointing to the zero-shot capability results as evidence.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims are anchored on external benchmarks and internal ablations, with self-citations that are not load-bearing.

full rationale

I find no circular step in the paper's derivation chain. The central claim that an agent-as-a-Python-object interface is effective is evaluated against external benchmarks (SWE-bench Verified, Terminal-Bench 2.0, CyberGym L1, ARC-AGI-3) and via internal ablations that isolate the memory subsystem and the world-model skill. The ARC-AGI-3 comparison to raw GPT-5.6-sol is explicitly caveated in footnote 2 ('evaluation budgets differ, so the comparison is indicative'), which is a transparency limitation rather than a reduction of the result to its own inputs. The self-citations to companion work [49] motivate the world-model skill and memory design, but they are not load-bearing: the skill is separately compared against a baseline skill within the same harness, and the memory effect is measured by ablation against markdown-file notes. No fitted parameter is renamed as a prediction, no equation is reused as its own output, and no claimed result is defined in terms of the authors' own outputs. The paper is therefore self-contained in the sense required by the circularity analysis: its benchmark results stand or fall on external, reproducible evaluations.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

NOOA introduces no new physical entities or fitted physical constants. The load-bearing choices are engineering conventions (ellipsis-body methods, preview formats, memory scoring) plus domain assumptions about model behavior and benchmark comparability. The memory subsystem is a new software artifact, but it is not an invented theoretical entity.

free parameters (2)
  • pprint/context preview format = not disclosed (format iterated experimentally across models, Sec 3.2)
    The bounded preview format (head/tail sample lengths, truncation markers) is central to pass-by-reference, and the paper states it was changed based on experimentation across open and closed models. It is a hand-tuned interface parameter rather than a standard library behavior.
  • Memory importance mapping and ACT-R retrieval weights = not disclosed (verbal descriptors mapped to 0–10; ACT-R activation used)
    The memory subsystem's measured benefit (+11.8 RHAE in the ARC ablation) depends on how importance, recency, decay, injection budget, and pruning are scored. These are design choices not derived from first principles, and their numeric values are not fully specified.
axioms (4)
  • domain assumption A method with an ellipsis body is interpreted as an LLM-driven loop, with the docstring as prompt and type annotations as contracts.
    This is the core semantic convention of NOOA. The capability tests support it for the evaluated models, but the assumption is not guaranteed across all models or future model families.
  • domain assumption Code-as-action is immediately usable because Python is heavily represented in LLM training data.
    P4 states this directly. The stress tests (Table 2) show the assumption degrades for small/efficient models on long-horizon tasks, so it is a real load-bearing premise.
  • domain assumption Benchmark comparisons are performed under matched settings, and published leaderboard numbers are comparable.
    The SWE-bench, Terminal-Bench, and CyberGym tables assume this. The ARC-AGI-3 comparison explicitly violates it: footnote 2 says evaluation budgets differ, so the 6.4× claim depends on an assumption the paper itself admits is not fully met.
  • domain assumption Executing model-written code in-process is safe because an external sandbox surrounds the agent process.
    The Limitations section says the in-process validator protects the agent loop, not the host, and that sandboxing is external (OpenShell). The ARC red-team audit is evidence for the specific ARC run, but the general claim depends on this deployment assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 39047 in / 11164 out tokens · 101158 ms · 2026-08-01T09:34:42.103842+00:00 · methodology

0 comments
read the original abstract

Traditional agent development is split across prompt templates, tool schemas, callback code, and workflow graphs. We present NVIDIA Object-Oriented Agents (NOOA), a model-agnostic Python framework for building reliable AI agents. NOOA takes a simpler approach: an agent is a Python object. Its methods are the actions the model can take, fields are its state, docstrings are its prompts, and its type annotations are contracts. A method whose code body consists of "..." is completed at runtime by an LLM-driven agent loop, while methods with normal bodies remain standard deterministic Python. This gives developers and agents the same interface, so agent behavior can be tested, traced, refactored, and improved just like other software. This paper makes three contributions. (1) We present the agent-as-a-Python-object programming model and the design principles behind it. Where Python has existing abstractions, we adopt them directly. Agent-specific capabilities--context, events, state rendering, long-term memory, and validated LLM loops--are exposed through simple Pythonic APIs, so both developers and agents share one familiar programming model. (2) We identify six model-facing ideas that NOOA is, to our knowledge, the first to combine on a single surface: typed input/output, pass-by-reference over live objects, code as action, programmable loop engineering, explicit object state, and model-callable harness APIs for context and events. We find the community already converging on several of these ideas--often as experimental or partial features--and present the comparison to encourage further adoption. (3) We demonstrate that current models use this interface effectively, both in targeted capability tests and on agentic and reasoning benchmarks such as SWE-bench Verified and Terminal-Bench 2.0 and ARC-AGI-3.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3

    cs.AI 2026-07 conditional novelty 6.5

    Selective programmatic world modeling (actor-requested builder) yields 100 RHAE on all 183 public ARC-AGI-3 levels, while automatic repair is more transition-exact but weaker at play.

Reference graph

Works this paper leans on

59 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Agrawal, Shangyin Tan, Dilara Soylu, et al

    Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. InICLR, 2026. URLhttps://arxiv.org/abs/2507. 19457

    2026

  2. [2]

    Letta: Stateful agents framework (formerly memgpt), 2024

    Letta AI. Letta: Stateful agents framework (formerly memgpt), 2024. URLhttps://github. com/letta-ai/letta

    2024

  3. [3]

    Anderson, Daniel Bothell, Michael D

    John R. Anderson, Daniel Bothell, Michael D. Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. An integrated theory of the mind.Psychological Review, 111(4):1036–1060,

  4. [4]

    opencode.https://github.com/anomalyco/opencode, 2025

    anomalyco. opencode.https://github.com/anomalyco/opencode, 2025

    2025

  5. [5]

    Building effective agents, 2024

    Anthropic. Building effective agents, 2024. URLhttps://www.anthropic.com/engineering/ building-effective-agents

    2024

  6. [6]

    Introducing advanced tool use on the claude developer platform, 2025

    Anthropic. Introducing advanced tool use on the claude developer platform, 2025. URL https://www.anthropic.com/engineering/advanced-tool-use

    2025

  7. [7]

    Equipping agents for the real world with agent skills, 2025

    Anthropic. Equipping agents for the real world with agent skills, 2025. URL https://www.anthropic.com/engineering/ equipping-agents-for-the-real-world-with-agent-skills

    2025

  8. [8]

    Effective harnesses for long-running agents, 2025

    Anthropic. Effective harnesses for long-running agents, 2025. URLhttps://www.anthropic. com/engineering/effective-harnesses-for-long-running-agents

    2025

  9. [9]

    How we built our multi-agent research system, 2025

    Anthropic. How we built our multi-agent research system, 2025. URL https://www. anthropic.com/engineering/multi-agent-research-system

    2025

  10. [10]

    Introducing dynamic workflows in claude code, 2026

    Anthropic. Introducing dynamic workflows in claude code, 2026. URLhttps://claude. com/blog/introducing-dynamic-workflows-in-claude-code

    2026

  11. [11]

    Manage claude’s memory: CLAUDE.md and auto memory, 2026

    Anthropic. Manage claude’s memory: CLAUDE.md and auto memory, 2026. URLhttps: //code.claude.com/docs/en/memory

    2026

  12. [12]

    Cursor rules and memories, 2025

    Anysphere. Cursor rules and memories, 2025. URLhttps://cursor.com/docs/rules

    2025

  13. [13]

    Prompting is programming: A query language for large language models

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. InPLDI, 2023. URLhttps://arxiv.org/abs/ 2212.06094

    Pith/arXiv arXiv 2023

  14. [14]

    Langchain, 2023

    Harrison Chase. Langchain, 2023. URLhttps://github.com/langchain-ai/langchain

    2023

  15. [15]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588, 2022. URLhttps://arxiv.org/abs/2211.12588

    Pith/arXiv arXiv 2022

  16. [16]

    Cheng, Logan Weber, Tian Jin, and Michael Carbin

    Ellie Y. Cheng, Logan Weber, Tian Jin, and Michael Carbin. Sharing state between prompts and programs. InInternational Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2512.14805. 22 NVIDIA-labs OO Agents Native Python Object-Oriented Agents

    arXiv 2026

  17. [17]

    CrewAI memory documentation, 2024

    CrewAI. CrewAI memory documentation, 2024. URLhttps://docs.crewai.com/concepts/ memory

    2024

  18. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. URLhttps://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  19. [19]

    Pi: Ai agent toolkit.https://github.com/earendil-works/pi, 2025

    Earendil Works. Pi: Ai agent toolkit.https://github.com/earendil-works/pi, 2025

    2025

  20. [20]

    ARC-AGI-3: Interactive reasoning benchmark, 2026

    ARC Prize Foundation. ARC-AGI-3: Interactive reasoning benchmark, 2026. URLhttps: //arcprize.org/arc-agi/3

    2026

  21. [21]

    Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022

    Luyu Gao, Aman Madaan, Shuyan Zhou, et al. Pal: Program-aided language models.arXiv preprint arXiv:2211.10435, 2022. URLhttps://arxiv.org/abs/2211.10435

    Pith/arXiv arXiv 2022

  22. [22]

    Agent development kit (adk), 2025

    Google. Agent development kit (adk), 2025. URLhttps://google.github.io/adk-docs/

    2025

  23. [23]

    Gemini CLI: memory files and the save_memory tool, 2025

    Google. Gemini CLI: memory files and the save_memory tool, 2025. URLhttps://github. com/google-gemini/gemini-cli/blob/main/docs/tools/memory.md

    2025

  24. [24]

    Anpl: Towards natural programming with interactive decomposition

    Di Huang, Ziyuan Nan, Xing Hu, Pengwei Jin, Shaohui Peng, Yuanbo Wen, Rui Zhang, Zidong Du, Qi Guo, Yewen Pu, and Yunji Chen. Anpl: Towards natural programming with interactive decomposition. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. URLhttps://arxiv.org/abs/2305.18498

    Pith/arXiv arXiv 2023

  25. [25]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  26. [26]

    Joshi, Hanna Moazam, et al

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://arxiv.org/abs/2310. 03714

    2024

  27. [27]

    Langgraph, 2024

    LangChain. Langgraph, 2024. URLhttps://github.com/langchain-ai/langgraph

    2024

  28. [28]

    LangMem SDK for agent long-term memory, 2025

    LangChain. LangMem SDK for agent long-term memory, 2025. URL https://www. langchain.com/blog/langmem-sdk-launch

    2025

  29. [29]

    Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023

    Chengshu Li, Jacky Liang, Andy Zeng, et al. Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023. URL https: //arxiv.org/abs/2312.04474

    Pith/arXiv arXiv 2023

  30. [30]

    Instructor: Structured outputs for llms, 2023

    Jason Liu. Instructor: Structured outputs for llms, 2023. URL https://github.com/ 567-labs/instructor

    2023

  31. [31]

    Llamaindex, 2023

    Jerry Liu. Llamaindex, 2023. URLhttps://github.com/run-llama/llama_index

    2023

  32. [32]

    Recursive agent harnesses

    Elias Lumer, Sahil Sen, Kevin Paul, and Vamse Kumar Subbiah. Recursive agent harnesses. arXiv preprint arXiv:2606.13643, 2026. URLhttps://arxiv.org/abs/2606.13643

    Pith/arXiv arXiv 2026

  33. [33]

    Rich: Rich text and beautiful formatting in the terminal, 2026

    Will McGugan. Rich: Rich text and beautiful formatting in the terminal, 2026. URL https://rich.readthedocs.io/. 23 NVIDIA-labs OO Agents Native Python Object-Oriented Agents

    2026

  34. [35]

    AutoGen’steachableagents, 2023

    Microsoft. AutoGen’steachableagents, 2023. URLhttps://microsoft.github.io/autogen/ 0.2/blog/2023/10/26/TeachableAgent/

    2023

  35. [36]

    Typechat, 2023

    Microsoft. Typechat, 2023. URLhttps://github.com/microsoft/TypeChat

    2023

  36. [37]

    Microsoft agent framework, 2025

    Microsoft. Microsoft agent framework, 2025. URLhttps://learn.microsoft.com/en-us/ agent-framework/

    2025

  37. [39]

    Openshell: a safe, private runtime for autonomous ai agents, 2026

    NVIDIA. Openshell: a safe, private runtime for autonomous ai agents, 2026. URLhttps: //github.com/NVIDIA/OpenShell

    2026

  38. [40]

    Askit: Unified programming interface for programming with large language models

    Katsumi Okuda and Saman Amarasinghe. Askit: Unified programming interface for programming with large language models. InProceedings of the 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2024. URL https://arxiv.org/abs/2308.15645

    Pith/arXiv arXiv 2024

  39. [41]

    URLhttps://arxiv.org/abs/2605.18747

    Pith/arXiv arXiv

  40. [42]

    Codex memories and custom instructions with AGENTS.md, 2026

    OpenAI. Codex memories and custom instructions with AGENTS.md, 2026. URLhttps: //developers.openai.com/codex/memories

    2026

  41. [43]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2023. URLhttps://arxiv.org/abs/2310.08560

    Pith/arXiv arXiv 2023

  42. [44]

    Function calling – openai api documentation, 2024

    OpenAI. Function calling – openai api documentation, 2024. URLhttps://developers. openai.com/api/docs/guides/function-calling

    2024

  43. [45]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, et al. Pytorch: An imperative style, high- performance deep learning library. InAdvances in Neural Information Processing Systems (NeurIPS), 2019. URLhttps://arxiv.org/abs/1912.01703

    Pith/arXiv arXiv 2019

  44. [46]

    Pydanticai: Agent framework with type-safe structured outputs, 2024

    Pydantic. Pydanticai: Agent framework with type-safe structured outputs, 2024. URL https://github.com/pydantic/pydantic-ai

    2024

  45. [47]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023. URLhttps://arxiv.org/abs/2304.03442

    Pith/arXiv arXiv 2023

  46. [48]

    smolagents: A barebones library for agents that think in code, 2024

    Aymeric Roucher et al. smolagents: A barebones library for agents that think in code, 2024. URLhttps://github.com/huggingface/smolagents. 24 NVIDIA-labs OO Agents Native Python Object-Oriented Agents

    2024

  47. [49]

    Workspace optimization: How to train your agent.arXiv preprint arXiv:2605.09650, 2026

    Elad Sarafian, Gal Kaplun, Ron Banner, Daniel Soudry, and Boris Ginsburg. Workspace optimization: How to train your agent.arXiv preprint arXiv:2605.09650, 2026. URL https://arxiv.org/abs/2605.09650

    Pith/arXiv arXiv 2026

  48. [50]

    Taskweaver: A code-first agent framework.arXiv preprint arXiv:2311.17541, 2024

    Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, Minghua Ma, Pu Zhao, Saravan Rajmohan, et al. Taskweaver: A code-first agent framework.arXiv preprint arXiv:2311.17541, 2024. URLhttps://arxiv. org/abs/2311.17541

    Pith/arXiv arXiv 2024

  49. [51]

    Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. URLhttps://arxiv.org/abs/ 2305.16291

    Pith/arXiv arXiv 2024

  50. [52]

    Executable code actions elicit better llm agents.arXiv preprint arXiv:2402.01030, 2024

    Xingyao Wang, Yangyi Chen, et al. Executable code actions elicit better llm agents.arXiv preprint arXiv:2402.01030, 2024. URLhttps://arxiv.org/abs/2402.01030

    Pith/arXiv arXiv 2024

  51. [53]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2303. 11366

    2023

  52. [54]

    Cybergym: Evaluating AI agents’ real-world cybersecurity capabilities at scale

    Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating AI agents’ real-world cybersecurity capabilities at scale. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps:// openreview.net/forum?id=2YvbLQEdYt

    2026

  53. [55]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702, 2023. URLhttps://arxiv.org/abs/2307.09702

    Pith/arXiv arXiv 2023

  54. [56]

    Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Xingyao Wang, Boxuan Li, et al. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024. URLhttps://arxiv.org/abs/ 2407.16741

    Pith/arXiv arXiv 2024

  55. [57]

    Jimenez, et al

    John Yang, Carlos E. Jimenez, et al. Swe-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793, 2024. URL https://arxiv.org/ abs/2405.15793

    Pith/arXiv arXiv 2024

  56. [58]

    are not passed to the LLM

    Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive language models.arXiv preprint arXiv:2512.24601, 2025. URLhttps://arxiv.org/abs/2512.24601. A. Appendix: Harness comparison details This appendix expands the compact comparison in Table 7. Each subsection begins with the pinned source snapshot (repository, commit, and package version, all retrieved on...

    Pith/arXiv arXiv 2025

  57. [59]

    Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025

    Sikuan Yan, Xiufeng Yang, Zuchao Huang, et al. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025. URLhttps://arxiv.org/abs/2508.19828

    Pith/arXiv arXiv 2025

  58. [2004]

    URLhttps://doi.org/10.1037/0033-295X.111.4.1036

    doi:10.1037/0033-295x.111.4.1036

  59. [2026]

    URLhttps://arxiv.org/abs/2601.11868

    Pith/arXiv arXiv