Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Agent-optimizer gains compound only when regression control lives inside the search loop.

desk verdict A genuinely useful evaluation protocol, but the causal headline is confounded by search-space breadth and thin statistics; read it as a preliminary methodology, not a mechanism test. read the letter →

arxiv 2607.14004 v1 pith:X72QQE25 submitted 2026-07-15 cs.AI cs.CLcs.LG

Wenxiao Wang , Priyatham Kattakinda , Soheil Feizi This is my paper
classification cs.AIcs.CLcs.LG
keywords agentoptimizationcontinuallearningcatastrophicforgettingregression-awaresearchharnesstransfergeneralizationbenchmarkoverfittingtwo-phaseevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agent-optimization results are usually reported as one-shot benchmark scores, which do not tell whether an optimized agent can be optimized again as new tasks arrive without losing earlier gains. This paper studies that compounding question with a two-phase protocol on hard terminal tasks: optimize on an initial set, test transfer to a new set, then re-optimize on the union. All three optimizers beat the unoptimized baseline in the static first phase, but only the regression-aware method transfers positively to unseen tasks and then continues improving in the second phase; the other two either overfit the first task set or stall when re-optimized. The paper's central observation is that gains compounded only when a no-regression constraint was enforced inside the optimization loop, rejecting candidate harness edits that trade new-task gains for regressions on tasks already solved. A reader should care because deployed agents are optimized recursively, and this protocol measures whether optimizer improvements survive that reality.

What carries the argument

The load-bearing machinery is the two-phase protocol itself, which separates three conflated properties: static optimization strength (Phase 1 score), transfer to unseen tasks (score on the expanded task union after optimizing only on the initial set), and continued improvement (final score after re-optimizing on the union). The mechanism the paper credits for the only method that passes all three is an in-loop no-regression constraint: candidates that improve newly targeted tasks while regressing on previously solved tasks are rejected during search, biasing the optimizer toward small, generalizable harness edits. In this evaluation those edits were generic additions—a task-type classifier,

What would settle it

Repeat the two-phase protocol with many trials per task (say 20) and report confidence intervals on Phase-1, transfer, and final pass rates; if the regression-aware method's transfer and continued-improvement advantages over the other two shrink to noise, the compounding claim fails. A sharper test: run the same regression-aware search with the no-regression rejection disabled but the same search space and budget; if transfer and second-phase gains disappear, the mechanism is confirmed.

Watch

Extended reading notes

Core claim

In a two-phase evaluation on 22 hard terminal tasks, each optimizer receives 200 rollouts to improve the same baseline agent, then another 200 after new tasks arrive. All three improve on the baseline's 62.5% Phase-1 pass rate (70.8%, 66.6%, 79.2%). Without new-task optimization, the prompt-evolution method falls to 54.5% on the 22-task union (below the 56.8% baseline), the harness-code method rises to 68.2%, and the regression-aware method reaches 72.7%. After re-optimization, the harness-code method drops to 59.1% because every second-round candidate scored worse than the existing agent; the regression-aware method improves to 77.3% and has the highest lifelong average (76.4% vs 66.0%, 64.

Load-bearing premise

The load-bearing premise is that the reported pass-rate differences reflect real optimizer behavior rather than sampling noise, since each task is scored from only two trials and comparisons are made without confidence intervals or significance tests.

Editorial extensions

If this is right

  • A high static benchmark score does not predict transfer: the top Phase-1 method transferred below the unoptimized baseline once new tasks appeared.
  • Transfer alone does not imply compounding: the method that transferred well stalled in Phase 2, with every generated candidate worse than the existing agent.
  • Repeated optimization alone does not imply compounding: the method that overfit Phase 1 regained performance in Phase 2 only by optimizing directly against the new tasks.
  • Regression control placed inside the search loop, not checked afterward, was associated with both positive transfer and continued improvement in this task population.
  • The lifelong average is informative only when decomposed into Phase 1, transfer, and final components; similar averages can hide opposite failure modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The no-regression constraint may function as an implicit complexity prior: in this task population, an edit that fixes a new task without breaking old ones is more likely to be general; this could be tested by removing the constraint while holding search space and budget fixed.
  • Because the benchmark tasks are only loosely related, the observed differences are not a stress test; on a stream of correlated failures within one application domain, shortcut exploitation could make both overfitting and stall effects more extreme.
  • The two-trial-per-task scoring leaves the reported gaps between close methods (e.g., 66.0% vs 64.6% lifelong average) potentially within sampling noise; a re-run with more trials per task would sharpen which differences are real.
  • For production settings with imperfect verifiers and single-trajectory feedback, an in-loop no-regression constraint would need a probabilistic or approximate form; the paper's clean verifier-based setting may overstate how easily the mechanism transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-phase continual-learning evaluation protocol for agent-harness optimizers, instantiated on 22 hard Terminal-Bench 2.0 tasks split into an initial 12-task set and a 10-task set. Three optimizers—GEPA, Meta Harness, and RELAI-VCL—are compared against a shared baseline under equal per-phase rollout budgets (200 rollouts per phase). The authors report static Phase 1 pass rates, transfer to unseen tasks after Phase 1, final pass rates after Phase 2 re-optimization, and a lifelong average. They find that all three optimizers improve over the baseline in the static setting, but only RELAI-VCL both transfers positively and continues to improve under re-optimization. The abstract attributes this compounding behavior to regression control being enforced inside the search loop.

Significance. The protocol is a useful methodological contribution: it separates static optimization strength, transfer to unseen tasks, and the ability to continue improving under re-optimization—properties that a single benchmark score conflates. The paper also releases its artifacts and implementation details, which supports reproducibility. If the causal attribution were established, the result would be practically important for designing continual agent-optimization systems. However, as presented, the headline causal claim is not supported: RELAI-VCL differs from the comparison methods in both the breadth of its search space and the presence of in-loop regression control, and all pass-rate comparisons rest on point estimates from only two trials per task. The paper's own limitations section explicitly acknowledges that the protocol cannot fully distinguish the proposed mechanism from other explanations, yet the abstract states it as the key finding.

major comments (3)
  1. [§4.2–4.4, Table 2] The central attribution—that compounding gains arise specifically from in-loop regression control—is confounded by search-space breadth. Per Table 2, RELAI-VCL searches over prompts, tools, workflow, memory, skills, and code, whereas GEPA (prompt) searches only prompts and Meta Harness only harness code. The one arm that would have provided a broad-search/no-constraint comparison, GEPA-code, failed to produce a valid candidate during Phase 1 and was excluded (§4.2). The observed differences could therefore reflect expressiveness of the search space rather than the no-regression mechanism. A direct manipulation of the constraint, holding the search space fixed, is needed to support the abstract's causal claim.
  2. [§3.3, Tables 4–8] Every task is scored from only two trials (R=2), and the paper compares point estimates without confidence intervals or significance tests. With 12–22 tasks, the headline gaps are small on a per-task basis: on transfer, RELAI-VCL's 72.7% vs. Meta Harness's 68.2% on the 22-task union is a difference of one task out of 22. Phase 1 differences are similarly within roughly one to two tasks. Without variance estimates or more trials, the claim that RELAI-VCL 'leads at every stage' is not statistically supported. At minimum, bootstrap confidence intervals or per-task score breakdowns should be reported.
  3. [§6.6, §7.1–7.3; abstract; §8] The paper's own limitations state that the current protocol 'cannot fully distinguish' the generalization-filter interpretation from other explanations, and note that the tasks are only loosely related and that only two optimization rounds are studied. Despite these caveats, the abstract and conclusion assert, as the 'key observation,' that gains compounded only when regression control was built into the optimization loop. This overstates the evidentiary strength of a single confounded three-way comparison. The conclusion should be reframed as an observed association, with the causal claim explicitly deferred until a controlled manipulation is performed.
minor comments (4)
  1. [Table 3] The 'Max. agent timeout' entry for T1∪T2 is listed as '900s / 1800s'; this is ambiguous. Since the union's maximum timeout is 1800s, reporting a single value would be clearer.
  2. [§4.2, Appendix B.2] GEPA-code's failure to produce a valid candidate is described only briefly. A short explanation of the failure mode (e.g., invalid syntax, evaluation crashes, budget exhaustion) would help readers interpret the exclusion.
  3. [§6.2] The pass rate on the unseen Phase 2 tasks alone (65.0%) is reported only in text. Including it in Table 5 or a separate column would make the transfer result more transparent.
  4. [References] Several references are to 2026 preprints or software repositories. Please confirm that all cited items are publicly accessible and include version/accession information where applicable.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; central claim is empirical, though the causal attribution is confounded and one component of 'compounding' overlaps with RELAI-VCL's definition by construction.

full rationale

The paper's central result is an empirical three-way comparison on Terminal-Bench 2.0, not a derivation from its own inputs. The per-stage pass rates in Tables 4-6 are measurements against an external benchmark; there is no fitted parameter being renamed as a prediction. The LifelongAvg metric is explicitly defined as the unweighted mean of Phase-1, Transfer, and Final, and the paper checks that this reproduces the reported numbers, so it is a summary rather than a hidden input. The only self-citation (Maestro, ref [23]) appears in related work and is not used to justify the paper's conclusions, so it is not load-bearing. One partial definitional overlap exists: the paper defines compounding as including 'without regressing on tasks it had already solved' (§1.2), and RELAI-VCL's no-regression constraint enforces exactly that by construction (§4.4); Section 6.6 even says 'by construction.' But the empirical content—positive transfer and continued improvement—is not guaranteed by the constraint, and the paper explicitly hedges the generalization-filter interpretation as 'consistent with the evidence' rather than a general claim. The main threat to the abstract's 'only when regression control was built into the optimization loop' is confounding, not circularity: RELAI-VCL differs from GEPA and Meta Harness in search-space breadth as well as the constraint, and GEPA-code (the arm that would disentangle these) failed. That is a correctness risk requiring a controlled manipulation, not a reduction of the derivation chain to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on experimental assumptions about task representativeness, trial count sufficiency, and faithful optimizer execution. There are no fitted free parameters or invented entities.

assumptions (3)
  • domain assumption The task split (T1: 12 hard tasks with 900s timeout; T2: 10 hard tasks with 1800s timeout; excluding 8 hard tasks with longer timeouts) is representative of a continual-learning stream.
    Section 5.1 defines the split by agent timeout; the choice of which tasks appear in T1 vs T2 is not randomized or justified beyond time constraints, and could affect transfer and re-optimization results.
  • domain assumption Two trials per task yield sufficiently reliable pass-rate estimates.
    Section 3.3 scores each task as the fraction of 2 trials the agent solves; with 12–22 tasks, the standard error on pass rate is roughly 5–7 percentage points, so small differences between methods may be noise.
  • domain assumption The optimizers are implemented and executed faithfully under identical budgets.
    Section 5.2 states a 200-rollout budget per phase, but implementation details like GEPA-code's failure and exclusion (Section 4.2) may reduce comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0." pith.science (2026). https://pith.science/paper/X72QQE25

@misc{pith2026260714004,
  author       = {Pith},
  title        = {Pith review of: Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X72QQE25}},
  note         = {Machine review of arXiv:2607.14004}
}
read the original abstract

Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.

Figures

Figures reproduced from arXiv: 2607.14004 by the authors.

Figure 1
Figure 1. Lifelong average pass rate by agent: RELAI-VCL reaches 76.4%, ahead of GEPA (66.0%), Meta Harness (64.6%), and the unopti￾mized baseline (58.7%). 1.3 PREVIEW OF PROTOCOL AND FINDINGS We construct a two-phase continual-learning evaluation from hard tasks in Terminal-Bench 2.0. In Phase 1, each optimizer is given a fixed rollout budget to optimize a common baseline agent on an initial task set T1. We then evaluate the… view at source ↗
Figure 2
Figure 2. Pass rate on the 12 Phase 1 tasks after Phase 1 optimiza [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Pass rate on the 22-task union T1 ∪ T2 after Phase 1 optimization only (transfer setting). Agent Transfer pass rate Baseline 56.8% GEPA 54.5% Meta Harness 68.2% RELAI-VCL 72.7% [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pass rate on the 22-task union T1 ∪ T2 after Phase 2 re￾optimization on the combined task set. Agent Final pass rate Baseline (not re-opt.) 56.8% GEPA 72.7% Meta Harness 59.1% RELAI-VCL 77.3% [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Lifelong average pass rate: the mean of the Phase-1, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 13 linked inside Pith

  1. [1]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    arXiv 2025

  2. [2]

    Claude code.https://www.anthropic.com/claude-code, 2025

    Anthropic. Claude code.https://www.anthropic.com/claude-code, 2025. Terminal-based agentic coding tool, 2025–2026

    2025

  3. [3]

    Dreaming.https://platform.claude.com/docs/en/managed-agents/dreams,

    Anthropic. Dreaming.https://platform.claude.com/docs/en/managed-agents/dreams,

  4. [4]

    HALO: An RLM-based automatic agent optimization loop.https://github.com/ context-labs/halo, 2026

    context-labs. HALO: An RLM-based automatic agent optimization loop.https://github.com/ context-labs/halo, 2026. Software. Accessed 2026-07-14

    2026

  5. [5]

    Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026

    Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URLhttps://doi.org/10.5281/zenodo.20953922

    doi:10.5281/zenodo.20953922 2026

  6. [6]

    Karpathy

    A. Karpathy. LLM wiki.https://gist.github.com/karpathy/ 442a6bf555914893e9891c11519de94f, 2026. Gist

    2026

  7. [7]

    Khattab, A

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023. doi: 10.48550/arXiv.2310.03714. URL https://arxiv.org/abs/2310.03714

    arXiv doi:10.48550/arxiv.2310.03714 2023

  8. [8]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic for- getting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. URL https://arxiv.org/abs/1612.00796. ar...

    arXiv 2017

Show all 33 references
  1. [9]

    Y . Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026. URLhttps://arxiv.org/abs/2603.28052

    2026 arXiv

  2. [10]

    Letta (formerly MemGPT): Stateful agents with long-term memory.https://github.com/ letta-ai/letta, 2023

    Letta AI. Letta (formerly MemGPT): Stateful agents with long-term memory.https://github.com/ letta-ai/letta, 2023. Software

    2023

  3. [11]

    H. Li, R. He, Q. Zhang, C. Ji, Q. Mang, X. Chen, L. A. Agrawal, W.-L. Liao, E. Yang, A. Cheung, J. Zou, K. Olukotun, I. Stoica, and J. E. Gonzalez. Combee: Scaling prompt learning for self-improving language model agents.arXiv preprint arXiv:2604.04247, 2026. URLhttps://arxiv....

    2026 arXiv

  4. [12]

    Motus.https://github.com/lithos-ai/motus, 2026

    LithosAI. Motus.https://github.com/lithos-ai/motus, 2026. Software

    2026

  5. [13]

    Manheim and S

    D. Manheim and S. Garrabrant. Categorizing variants of goodhart’s law.arXiv preprint arXiv:1803.04585, 2018. URLhttps://arxiv.org/abs/1803.04585

    2018 arXiv

  6. [14]

    mem0: Memory layer for AI agents.https://github.com/mem0ai/mem0, 2024

    mem0.ai. mem0: Memory layer for AI agents.https://github.com/mem0ai/mem0, 2024. Software

    2024

  7. [15]

    M. A. Merrill et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. URLhttps://arxiv.org/abs/2601.11868

    2026 arXiv

  8. [16]

    Hermes.https://github.com/nousresearch/hermes-agent, 2026

    Nous Research. Hermes.https://github.com/nousresearch/hermes-agent, 2026. Software

    2026

  9. [17]

    Codex.https://openai.com/index/introducing-codex/, 2025

    OpenAI. Codex.https://openai.com/index/introducing-codex/, 2025. Cloud-based au- tonomous coding agent, 2025–2026

    2025

  10. [18]

    Opsahl-Ong, M

    K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab. Optimizing instructions and demonstrations for multi-stage language model programs.arXiv preprint arXiv:2406.11695, 2024. doi: 10.48550/arXiv.2406.11695. URLhttps://arxiv.org/abs/2406.11695...

    2024 doi arXiv

  11. [19]

    Pyatkin, S

    V . Pyatkin, S. Malik, V . Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi. Generalizing verifiable instruction following.arXiv preprint arXiv:2507.02833, 2025

    2025 arXiv

  12. [20]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. doi: 10.48550/arXiv.2402.03300. URLhttps://arxiv.org/abs/2...

    2024 doi arXiv

  13. [21]

    Shinn, F

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366, 2023. doi: 10.48550/arXiv.2303.11366. URL https://arxiv.org/abs/2303.11366

    2023 doi arXiv

  14. [22]

    Tannyhill

    B. Tannyhill. Introducing LangSmith engine.https://www.langchain.com/blog/ introducing-langsmith-engine, 2026. LangChain Blog, May 13, 2026

    2026

  15. [23]

    W. Wang, P. Kattakinda, and S. Feizi. Maestro: Joint graph & config optimization for reliable ai agents.arXiv preprint arXiv:2509.04642, 2025. URLhttps://arxiv.org/abs/2509.04642

    2025 arXiv

  16. [24]

    P. Xia, J. Chen, X. Yang, H. Tu, J. Liu, K. Xiong, S. Han, S. Qiu, H. Ji, Y . Zhou, Z. Zheng, C. Xie, and H. Yao. Metaclaw: Just talk – an agent that meta-learns and evolves in the wild.arXiv preprint arXiv:2603.17187, 2026. URLhttps://arxiv.org/abs/2603.17187

    2026

  17. [25]

    Zhang, B

    J. Zhang, B. Zhao, W. Yang, J. Foerster, J. Clune, M. Jiang, S. Devlin, and T. Shavrina. Hyperagents.arXiv preprint arXiv:2603.19461, 2026. URLhttps://arxiv.org/abs/2603.19461

    2026

  18. [26]

    Zhang et al

    Q. Zhang et al. Agentic context engineering: Evolving contexts for self-improving language models.arXiv preprint arXiv:2510.04618, 2025. URLhttps://arxiv.org/abs/2510.04618

    2025 arXiv

  19. [27]

    TerminusKira

    H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vuli ´c, A. Korhonen, and S. ¨O. Arık. Multi-agent design: Optimizing agents with better prompts and topologies.arXiv preprint arXiv:2502.02533, 2025. doi: 10.48550/ arXiv.2502.02533. URLhttps://arxiv.org/abs/2502.02533. 12 Tec...

    2025 doi

  20. [29]

    Infer the externally visible deliverable, not just the process liveness condition

  21. [30]

    Inspect nearby task files, tests, verifier scripts, or obvious validation assets when present so you know what a checker will request over the network, filesystem, or CLI

  22. [31]

    Treat HTTP 200 on a root page, an open port, a running service, or a clean daemon status as intermediate evidence unless the task explicitly says that is the final deliverable

  23. [32]

    Before finishing, perform one end-to-end audit that exercises the published artifact or workflow result the verifier is likely to check

  24. [33]

    If the task depends on a user workflow step such as clone/push/fetch/request, simulate that workflow or validate the exact resulting externally served artifact before stopping. """ •GEPA-code’s invalid-candidate outcome (Section 4.2).The code-mutation variant of GEPA produced ...

  25. [2026]

    Managed agents documentation

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.