Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MemoHarness claims that an LLM agent's control layer can be learned from its own past executions and adapted to each new task without test-time feedback.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:38 UTC pith:Y7P3T7TI

load-bearing objection Interesting framework, honest writing—but the headline 0.806 vs 0.722 claim is within sampling noise on an 18-task split. the 5 major comments →

arxiv 2607.14159 v1 pith:Y7P3T7TI submitted 2026-07-14 cs.AI cs.CL

MemoHarness: Agent Harnesses That Learn from Experience

Yue Huang , Wenjie Wang , Han Bao , Yuchen Ma , Xiaonan Luo , Yi Nian , Haomin Zhuang , Zheyuan Liu
show 2 more authors
Yue Zhao Xiangliang Zhang
This is my paper
classification cs.AI cs.CL
keywords agent harnessharness optimizationexperience banktest-time adaptationLLM agentsretrieval-augmented controlcost-aware selectiontransfer learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to show that the external control layer that turns a base LLM into an agent — context assembly, tool use, decoding, orchestration, memory, and output handling — can itself be optimized from execution experience, rather than fixed by hand or tuned only as a prompt. It proposes decomposing the harness into six editable dimensions, storing per-case diagnoses and distilled cross-case patterns in a dual-layer experience bank, and using that bank to adapt the harness to each new case without labels or feedback. Across shell-agent, code-generation, and analytical-reasoning tasks, the learned harness outperforms a strong fixed baseline, transfers selectively to unseen suites and base models, and stays cost-competitive when retrieved context is cacheable. If right, this shifts automated agent improvement from prompt and workflow tuning to learning the full control layer, making deployed agents self-specializing per case.

Core claim

The paper claims that an agent harness — the external control layer handling context, tools, generation, orchestration, memory, and output — can be decomposed into six editable dimensions and optimized by a search that records per-case diagnoses and distilled global patterns in a dual-layer experience bank. The resulting global harness is selected by a correctness-first lexicographic rule (reward primary, token cost as tiebreaker), and at test time the bank is retrieved by similarity to emit a case-specific harness with no labels, feedback, or extra search. The reported evidence on a shell-agent benchmark shows a roughly eight-point mean-success improvement over the strongest fixed harness,

What carries the argument

The central object is the six-dimensional harness space: context assembly, tool interaction, generation control, orchestration, memory management, and output processing. The mechanism that carries the argument is the dual-layer experience bank B=(E,G), pairing per-case execution entries (diagnosis, trajectory, reward, cost) with distilled global patterns about what works, what fails, and how dimensions interact. During search, a retrieval-conditioned controller proposes the next candidate, and selection is lexicographic: maximize mean reward first, minimize token cost second. At test time, TopK similar successful and failed entries plus feature-conditioned patterns adapt the learned global h

Load-bearing premise

The paper's evidence rests on an 18-task held-out split reported as point averages without confidence intervals or significance tests, so the headline gap of roughly eight points could in principle be sampling noise.

What would settle it

Run the same harness search on the same held-out split with several random seeds or bootstrap resampling of tasks and compute a confidence interval for the difference between MemoHarness and the strongest fixed harness; if the interval includes zero, the claimed systematic improvement is not established. A second check is to disable test-time case adaptation and keep only the global harness; if task success does not drop, the adaptation mechanism is not doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, automated improvement for LLM agents can move beyond prompts and workflows to the full control layer, making context assembly, tool exposure, orchestration, memory, and output validation jointly tunable.
  • A harness learned on one benchmark can transfer to unseen suites and to other base models without retraining, although selectively, which would let one search session amortize across many deployments.
  • Test-time adaptation from a frozen experience bank means deployed agents can specialize to each case using only the input and training history, without needing labels or interactive feedback at inference.
  • The cost analysis implies that retrieval-based adaptation can remain economically viable when most retrieved context is cacheable, so adaptivity need not require a large per-task price increase.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separately ablate the experience bank, global patterns, and case-specific adaptation; a reader might infer that test-time retrieval, rather than search alone, could be the main driver of the gains, and that an ablation would sharpen attribution.
  • Because transfer is selective and strongest when the source task is long-horizon and tool-centric, an inference is that what transfers is not the full harness but a few portable control decisions, such as robust instruction following or inspection operations; isolating those decisions could produce smaller, more transferable edits.
  • The operation-level diagnostic links newly added inspection and condition-checking shell commands to reward-improving transitions, suggesting that biasing search toward such low-level repair operations could accelerate optimization in new domains.
  • Given the explicit lack of confidence intervals, a sensible extension is a bootstrap or permutation test on the same held-out split; if the interval excludes zero, the result moves from suggestive to established.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MemoHarness, an adaptive agent-harness optimization framework. It decomposes a monolithic harness into six editable control dimensions (context, tool, generation, orchestration, memory, output), maintains a dual-layer experience bank of per-case execution entries and distilled global patterns, and at test time adapts a global harness learned by search into a case-specific harness using retrieved successes and failures, without test-time labels or further search. The authors report improvements on Terminal-Bench (0.722 vs. 0.806 against the strongest fixed baseline), LiveCodeBench, and FinanceAgent; selective transfer to six unseen suites; mean +0.098 task-success gain across six held-out base models; and cost-competitive inference when retrieved contexts are cacheable.

Significance. If the empirical claims are sustained, the paper makes a useful contribution: it broadens automatic optimization from prompts/workflows to the full control layer around an LLM, and it proposes a concrete mechanism for per-case test-time adaptation without feedback. The write-up is transparent about its limitations, and the design choices — validation-selected harnesses, correctness-first selection, cost tie-breaking separate from dollar accounting, released code, fixed seed splits — are sound and reproducible in spirit. The cross-dataset and cross-model transfer tests are also valuable sanity checks. The central weakness is statistical: the headline result rests on an 18-task split with point estimates only, which the authors themselves flag, and the strongest baseline is not a controlled scaffold transplant.

major comments (5)
  1. [§3.2, Figure 2, Appendix A] The primary claim that MemoHarness improves over Codex by +0.084 is not statistically supported. The held-out Terminal-Bench split is 18 tasks; the gap is about 1.5 tasks. A conservative two-proportion standard error for 0.806 vs 0.722 at n=18 is roughly 0.14, so the difference is below one standard error, and the 95% interval for 0.806 contains 0.722. Appendix A says the paper reports point estimates rather than confidence intervals or significance tests, and the number of repeated runs is not stated. To make the RQ1 claim load-bearing, the paper must report the number of independent runs, confidence intervals (e.g., bootstrap over tasks with the fixed split), or use a larger held-out split (e.g., cross-validation on all 89 tasks).
  2. [§3.1, Appendix C] The strongest baseline, Codex, is not a controlled scaffold transplant under the same base model and runtime surface. The paper acknowledges that product-specific baselines should be read as system-level comparisons. This matters because the headline +0.084 improvement is against that baseline; differences could stem from the model, tool interfaces, or product behavior rather than from the learned harness. The paper should either run a truly model-controlled version of Codex (same generator and tool surface, only harness configuration changed) or, failing that, temper the claim and add a sensitivity analysis that keeps the underlying generator fixed in the comparisons where that is possible.
  3. [RQ4, Table 3] The cross-model transfer result is also reported as point estimates without confidence intervals or significance tests. Mean gain +0.098 across six models is directionally suggestive, and the fact that all six models improve is a positive sign. But per-model gains range from +0.038 (GPT-4.1, roughly 0.7 tasks on n=18) to +0.233 (GLM-5, roughly 4.2 tasks). Without per-model intervals or a paired test across tasks, the smaller gains cannot be distinguished from sampling noise. Please report task-level paired comparisons or bootstrap intervals.
  4. [RQ5, Table 4] The cost advantage of MemoHarness ($6.89 vs. Codex $10.28) depends on the assumption that 13.32M of 14.18M input tokens are cached. The paper correctly flags this, but the abstract still claims the method 'can also remain cost-competitive when much of the retrieved experience is cacheable.' Because the entire cost claim hinges on that cache rate, the paper should include a sensitivity analysis: report cost if cache reuse were, say, 0%, 50%, and 90% of retrieved context. Without this, the reader cannot judge how brittle the cost conclusion is.
  5. [Appendix A and §2.4] The paper does not ablate the key components — the dual-layer experience bank, global patterns, and case-specific test-time adaptation. Since the method's novelty is precisely the combination of these components, the lack of ablations leaves component attribution open. At minimum, the paper should compare the full system against (i) the search-derived global harness with no test-time adaptation, and (ii) a version without distilled global patterns. Without these, the empirical gains cannot be assigned to the claimed mechanisms.
minor comments (6)
  1. [§2.6, Eq. (17)] The instruction representation ψ(u) is not defined. Please specify the embedding function, model, or feature representation used for the cosine similarity.
  2. [Figure 1] Figure 1 is dense and the notation B_t=(E_t,G_t) is easy to confuse with the per-case entry variables. Consider simplifying and clarifying in the caption what is a set, what is a pair, and which components are frozen at test time.
  3. [Appendix C] The paper says test metrics are 'averaged over repeated runs' but never states the number of runs. State n_runs explicitly; this is needed to interpret any interval or the stability of the point estimates.
  4. [Table 3] In the cross-model table, the 'Base' column should clarify that 'Base' means the fixed Codex harness applied to that model. As written, a reader may mistakenly infer that 'Base' means the model's default out-of-the-box behavior with no harness.
  5. [Appendix G, Table 6] The operation-level lift analysis relies on very small counts (n_add from 2 to 46) and many comparisons. 'cat' has only 11 newly-added occurrences. Please add confidence intervals or a warning that these lifts are unstable and not multiple-comparison corrected.
  6. [§2.4] The distillation triggers M and N are introduced in the main text but only instantiated in Appendix C. Add a forward reference to the hyperparameters, or state the defaults in the method section.

Circularity Check

0 steps flagged

No significant circularity: learning is from the training split; held-out, cross-dataset, and cross-model evaluations are independent of the search objective.

full rationale

MemoHarness's derivation chain is a standard train/test pipeline. The global harness W* is selected by lexicographic reward/cost on the labeled search set Dsearch (Eqs. 15-16), and test-time adaptation (Eqs. 17-20) conditions only on the test-visible input, the frozen experience bank, and retrieved training entries; the held-out evaluation set Dtest is disjoint and its labels are not used. No fitted parameter is renamed as a prediction: the search optimizes a harness configuration on training reward, and the reported gains are measured on the held-out split, on unseen suites (Table 2), and on unseen base models (Table 3), which are external to the search objective. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The limitations in Appendix A (18-task split, point estimates, no confidence intervals or significance tests) concern statistical robustness rather than definitional or construction-level circularity; likewise the caching assumption in Table 4 is a stated accounting assumption, not a circular reduction. The central claim therefore does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central claims depend on a handful of hand-picked hyperparameters and on four domain assumptions about diagnosis, retrieval similarity, cross-model portability, and cacheability. None of these is itself derived or benchmarked independently; they are empirical premises of the system design.

free parameters (5)
  • Train/eval split ratio (80/20, seed 42) = 80/20, seed 42
    Fixed random partition of the 89-task Terminal-Bench suite into 71 search / 18 held-out tasks; all reported gains depend on this split and seed.
  • Search iterations T = 10
    Controller–Bank loop runs for 10 outer iterations; FinanceAgent still improving at iterations 8–9, so results are sensitive to this budget.
  • Retrieval neighborhood sizes K_succ, K_fail = 10 each
    Number of recent successes/failures given to the controller and used in test-time adaptation; directly shapes the case-specific harness.
  • Distillation triggers M, N = M=5 bank entries or N=3 consecutive failures
    Schedule for distilling global patterns from per-case failures; changes the content of the experience bank.
  • Decoding determinism = temperature=0.0, top_p=1.0, candidate=1, max budget=8192
    All runs use deterministic decoding; different sampling budgets could change harness behavior and transfer.
axioms (4)
  • domain assumption Base-model harness experience transfers across model families without retraining.
    RQ4 assumes that a harness searched with GPT-5.3-Codex is applicable to Claude/Gemini/Qwen/GLM/DeepSeek; results are positive but without significance tests.
  • ad hoc to paper The diagnostic operator assigns failures to six harness dimensions accurately enough to guide search.
    Appendix B states diagnosis is 'intentionally coarse' and uses 'practical heuristics'; if diagnosis is unreliable, the search and distilled patterns lose grounding.
  • domain assumption Success/failure labels in the experience bank are a reliable basis for retrieving useful adaptation evidence.
    Test-time adaptation selects similar successful and failed training cases by cosine similarity of instruction embeddings; this assumes surface similarity correlates with useful harness edits.
  • domain assumption Retrieved experience remains largely cacheable at deployment.
    RQ5 cost comparison assumes 13.32M of 14.18M input tokens are cacheable; the paper warns that deployments with lower cache reuse may see a different cost profile.
invented entities (2)
  • Case-specific harness W(x_j) no independent evidence
    purpose: Per-instance specialization of the global harness generated at test time from retrieved experience
    Internal construct of the method; no falsifiable handle outside the reported benchmark outcomes.
  • Dual-layer experience bank B_t=(E_t,G_t) no independent evidence
    purpose: Stores per-case execution entries and distilled global patterns to guide search and test-time adaptation
    Internal memory structure; its contribution is not isolated by ablation in the reported experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 15180 in / 15288 out tokens · 149407 ms · 2026-08-02T05:38:29.121696+00:00 · methodology

0 comments
read the original abstract

An agent harness is the external control layer that turns a base LLM into an executable agent by managing context, tools, orchestration, memory, decoding, and output handling. While harness design strongly affects agent behavior, most automatic improvement methods optimize narrower artifacts such as prompts, pipelines, or workflows, and deployed agents usually reuse a single global harness for all cases. We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions. MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank, and adapts the learned harness to each test case using retrieved experience without test-time labels, feedback, or additional search. In our evaluation across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness improves over the fixed harnesses we compare against and shows selective transfer to unseen suites and base models. Its additional context can also remain cost-competitive when much of the retrieved experience is cacheable. These results provide evidence that execution experience is a practical substrate for building agent harnesses that are more adaptive than a single static configuration, while leaving broader claims about statistical robustness and component attribution to future work.

Figures

Figures reproduced from arXiv: 2607.14159 by Han Bao, Haomin Zhuang, Wenjie Wang, Xiangliang Zhang, Xiaonan Luo, Yi Nian, Yuchen Ma, Yue Huang, Yue Zhao, Zheyuan Liu.

Figure 1
Figure 1. Figure 1: Overview of MemoHarness. Phase A (Search, training-time optimization) runs a guided search over the six-dimensional harness space W = Î6 𝑑=1 W(𝑑) : each candidate harness 𝑊 is executed on labeled search cases, scored by a correctness-first reward 𝑟𝑖(𝑊 ) with token usage 𝑐𝑖(𝑊 ) = 𝑛 tok 𝑖 as tiebreaker, and the resulting trajectories are stored in a dual-layer experience bank B𝑡 = (𝐸𝑡 ,𝐺𝑡) of per-case entrie… view at source ↗
Figure 2
Figure 2. Figure 2: Baseline comparison on Terminal-Bench. Higher is better; Memo￾Harness (rightmost, using GPT-5.3-Codex) reaches the highest mean of 0.806 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Harness quality at six checkpoints (base, four intermediate iterations, and the final selected harness) across three benchmarks. The final score is higher than base on every benchmark, but it is not always equal to the in-training peak: LiveCodeBench touches 1.000 at iteration 3, and Terminal-Bench reaches 0.833 at iteration 4 because the shipped harness is selected by validation rather than by peeking at … view at source ↗
Figure 4
Figure 4. Figure 4: Per-iteration success rate on FinanceAgent (left) and LiveCodeBench (right) over 10 search rounds. FinanceAgent continues to benefit from additional iterations (rising from 42.5% to a 65.0% peak around iterations 8 and 9), whereas LiveCodeBench saturates almost immediately near the base-model ceiling and oscillates within a ∼4pt band. largest gain appears on FinanceAgent, where the base harness leaves the … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systems

    cs.CR 2026-07 conditional novelty 6.0

    Agent Harness Distillation extracts inference-time multi-agent orchestration from black-box AMAS and transfers it to weaker backbones, exposing IP leakage that a deception defense can blunt.

Reference graph

Works this paper leans on

66 extracted references · 24 linked inside Pith · cited by 1 Pith paper

  1. [1]

    2023 , eprint =

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines , author =. 2023 , eprint =

    2023

  2. [2]

    2023 , eprint =

    Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution , author =. 2023 , eprint =

    2023

  3. [3]

    2024 , eprint =

    AutoFlow: Automated Workflow Generation for Large Language Model Agents , author =. 2024 , eprint =

    2024

  4. [4]

    2024 , eprint =

    AFlow: Automating Agentic Workflow Generation , author =. 2024 , eprint =

    2024

  5. [5]

    2026 , eprint =

    Meta-Harness: End-to-End Optimization of Model Harnesses , author =. 2026 , eprint =

    2026

  6. [6]

    2026 , eprint =

    Natural-Language Agent Harnesses , author =. 2026 , eprint =

    2026

  7. [7]

    2025 , eprint =

    AlphaEvolve: A coding agent for scientific and algorithmic discovery , author =. 2025 , eprint =

    2025

  8. [8]

    2026 , eprint =

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author =. 2026 , eprint =

    2026

  9. [9]

    2023 , eprint =

    Large Language Models as Optimizers , author =. 2023 , eprint =

    2023

  10. [10]

    2023 , eprint =

    Self-Refine: Iterative Refinement with Self-Feedback , author =. 2023 , eprint =

    2023

  11. [11]

    2023 , eprint =

    Reflexion: Language Agents with Verbal Reinforcement Learning , author =. 2023 , eprint =

    2023

  12. [12]

    2024 , month = dec, howpublished =

    Building Effective Agents , author =. 2024 , month = dec, howpublished =

    2024

  13. [13]

    2025 , month = sep, howpublished =

    Writing Effective Tools for Agents with Agents , author =. 2025 , month = sep, howpublished =

    2025

  14. [14]

    2025 , month = jul, howpublished =

    Context Engineering , author =. 2025 , month = jul, howpublished =

    2025

  15. [15]

    2026 , month = feb, howpublished =

    Harness Engineering: Leveraging Codex in an Agent-First World , author =. 2026 , month = feb, howpublished =

    2026

  16. [16]

    doi:10.48550/arXiv.2210.03629 , url =

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik Narasimhan and Yuan Cao , year =. doi:10.48550/arXiv.2210.03629 , url =. 2210.03629 , archivePrefix=

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629

  17. [17]

    doi:10.48550/arXiv.2302.04761 , url =

    Timo Schick and Jane Dwivedi-Yu and Roberto Dessi and Roberta Raileanu and Maria Lomeli and Luke Zettlemoyer and Nicola Cancedda and Thomas Scialom , year =. doi:10.48550/arXiv.2302.04761 , url =. 2302.04761 , archivePrefix=

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.04761

  18. [18]

    2023 , eprint =

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author =. 2023 , eprint =

    2023

  19. [19]

    2023 , eprint =

    Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models , author =. 2023 , eprint =

    2023

  20. [20]

    2023 , eprint =

    Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search , author =. 2023 , eprint =

    2023

  21. [21]

    2024 , eprint =

    Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs , author =. 2024 , eprint =

    2024

  22. [22]

    Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , year =

    John Yang and Carlos E. Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , year =. doi:10.48550/arXiv.2405.15793 , url =. 2405.15793 , archivePrefix=

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.15793

  23. [23]

    2025 , howpublished =

    Codex. 2025 , howpublished =

    2025

  24. [24]

    2025 , howpublished =

    Claude. 2025 , howpublished =

    2025

  25. [25]

    2024 , howpublished =

    2024

  26. [26]

    Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H

    Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H. Tran and Fuqiang Li and Ren Ma and Mingzhang Zheng and Bill Qian and Yanjun Shao and Niklas Muennighoff and Yizhe Zhang and Binyuan Hui and Junyang Lin and Robert Brennan and Hao Peng and H...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.16741

  27. [27]

    Proceedings of the International Conference on Learning Representations (ICLR) , year =

    Measuring Massive Multitask Language Understanding , author =. Proceedings of the International Conference on Learning Representations (ICLR) , year =

  28. [28]

    2024 , url =

    Niklas Muennighoff and Qian Liu and Armel Randy Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro Von Werra and Shayne Longpre , booktitle =. 2024 , url =

    2024

  29. [29]

    Alexandra Souly and Qingyuan Lu and Dillon Bowen and Tu Trinh and Elvis Hsieh and Sana Pandey and Pieter Abbeel and Justin Svegliato and Scott Emmons and Olivia Watkins and Sam Toyer , booktitle =. A. 2024 , url =

    2024

  30. [30]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year =

    Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards , author =. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year =

  31. [32]

    He and Charles Ide and Kanak Garg and Niklas Lauffer and Andrew Park and Chetan Rane and Karmini Sampath and Maya Krishnan and Srivatsa R Kundurthy and Sean M

    Xiang Deng and Jeff Da and Edwin Pan and Yannis Y. He and Charles Ide and Kanak Garg and Niklas Lauffer and Andrew Park and Chetan Rane and Karmini Sampath and Maya Krishnan and Srivatsa R Kundurthy and Sean M. Hendryx and Zifan Wang and Chen Bo Calvin Zhang and Noah Jacobson and Bing Liu and Brad Kenstler , year =

  32. [33]

    International Conference on Learning Representations , year =

    Large Language Models as Optimizers , author =. International Conference on Learning Representations , year =

  33. [34]

    Differentiation

    TextGrad: Automatic "Differentiation" via Text , author=. 2024 , eprint=

    2024

  34. [35]

    Building effective agents

    Anthropic . Building effective agents. Anthropic Engineering Blog, December 2024. URL https://www.anthropic.com/engineering/building-effective-agents. Published December 19, 2024

    2024

  35. [36]

    Claude Code : Agentic coding in the terminal

    Anthropic . Claude Code : Agentic coding in the terminal. https://github.com/anthropics/claude-code, 2025 a . GitHub repository, accessed 2026-04

    2025

  36. [37]

    Writing effective tools for agents with agents

    Anthropic . Writing effective tools for agents with agents. Anthropic Engineering Blog, September 2025 b . URL https://www.anthropic.com/engineering/writing-tools-for-agents. Published September 11, 2025

    2025

  37. [38]

    He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Y. He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa R Kundurthy, Sean M. Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE -bench pro: Can AI agents solve long-horizon software engineering tasks?, 2026. URL https:/...

    2026

  38. [39]

    LawBench : Benchmarking legal knowledge of large language models

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. LawBench : Benchmarking legal knowledge of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    doi:10.18653/v1/2024.emnlp-main.452 2024

  39. [40]

    Promptbreeder: Self-referential self-improvement via prompt evolution, 2023

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt\"aschel. Promptbreeder: Self-referential self-improvement via prompt evolution, 2023. URL https://arxiv.org/abs/2309.16797

    Pith/arXiv arXiv 2023

  40. [41]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  41. [42]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines, 2023. URL https://arxiv.org/abs/2310.03714

    Pith/arXiv arXiv 2023

  42. [43]

    Context engineering

    LangChain . Context engineering. LangChain Blog, July 2025. URL https://blog.langchain.com/context-engineering-for-agents/. Published July 2, 2025

    2025

  43. [44]

    Meta-harness: End-to-end optimization of model harnesses, 2026

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses, 2026. URL https://arxiv.org/abs/2603.28052

    Pith/arXiv arXiv 2026

  44. [45]

    Autoflow: Automated workflow generation for large language model agents, 2024

    Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents, 2024. URL https://arxiv.org/abs/2407.12821

    Pith/arXiv arXiv 2024

  45. [46]

    Harness engineering: Leveraging codex in an agent-first world

    Ryan Lopopolo. Harness engineering: Leveraging codex in an agent-first world. OpenAI Engineering Blog, February 2026. URL https://openai.com/index/harness-engineering/. Published February 11, 2026

    2026

  46. [47]

    Self-refine: Iterative refinement with self-feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651

    Pith/arXiv arXiv 2023

  47. [48]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. URL https://arxiv.org/abs/2601.11868

    Pith/arXiv arXiv 2026

  48. [49]

    OctoPack : Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. OctoPack : Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mw1PWNSWZP

    2024

  49. [50]

    Alexander Novikov, Ng\^an V \ u , Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and ...

    Pith/arXiv arXiv 2025

  50. [51]

    Codex CLI : Lightweight coding agent for the terminal

    OpenAI . Codex CLI : Lightweight coding agent for the terminal. https://github.com/openai/codex, 2025. GitHub repository, accessed 2026-04

    2025

  51. [52]

    Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab

    Krista Opsahl-Ong, Michael J. Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs, 2024. URL https://arxiv.org/abs/2406.11695

    Pith/arXiv arXiv 2024

  52. [53]

    Natural-language agent harnesses, 2026

    Linyue Pan, Lexiao Zou, Shuo Guo, Jingchen Ni, and Hai-Tao Zheng. Natural-language agent harnesses, 2026. URL https://arxiv.org/abs/2603.25723

    Pith/arXiv arXiv 2026

  53. [54]

    Automatic prompt optimization with ``gradient descent'' and beam search, 2023

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with ``gradient descent'' and beam search, 2023. URL https://arxiv.org/abs/2305.03495

    Pith/arXiv arXiv 2023

  54. [55]

    Toolformer : Language models can teach themselves to use tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer : Language models can teach themselves to use tools, 2023. URL https://arxiv.org/abs/2302.04761

    Pith/arXiv arXiv 2023

  55. [56]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366

    Pith/arXiv arXiv 2023

  56. [57]

    A StrongREJECT for empty jailbreaks

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=KZLE5BaaOH

    2024

  57. [58]

    OpenCode : An open-source ai coding agent for the terminal

    SST . OpenCode : An open-source ai coding agent for the terminal. https://github.com/sst/opencode, 2024. GitHub repository, accessed 2026-04

    2024

  58. [59]

    Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards

    Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas K \"o pf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/fo...

    2025

  59. [60]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands : An open platform for ai sof...

    Pith/arXiv arXiv 2024

  60. [61]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2023. URL https://arxiv.org/abs/2309.03409

    Pith/arXiv arXiv 2023

  61. [62]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent : Agent-computer interfaces enable automated software engineering, 2024. URL https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  62. [63]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023 a . URL https://arxiv.org/abs/2305.10601

    Pith/arXiv arXiv 2023

  63. [64]

    ReAct : Synergizing reasoning and acting in language models, 2023 b

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models, 2023 b . URL https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2023

  64. [65]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text, 2024. URL https://arxiv.org/abs/2406.07496

    Pith/arXiv arXiv 2024

  65. [66]

    Aflow: Automating agentic workflow generation, 2024

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation, 2024. URL https://arxiv.org/abs/2410.10762

    Pith/arXiv arXiv 2024

  66. [67]

    Language agent tree search unifies reasoning acting and planning in language models, 2023

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023. URL https://arxiv.org/abs/2310.04406

    Pith/arXiv arXiv 2023