Pith.

Artificial Intelligence · arXiv:2609.37834 v1 · submitted 29 Sep 2026 · 78 references

Mixture of Self-Improving Branches For Agent Harness Optimization

Haoyu Dong, Yuhang Zhou, Zihao Lin, Yifan Wu, Bo Peng, Mingyi Wang, Xiangjun Fan, Lizhu Zhang

show 1 more author

Zhuokai Zhao

The paper says

“Harness optimization provides a practical setting for recursive self-improvement (RSI), where agent-generated modifications inform subsequent changes through execution feedback.”

more of the abstract

Recent work such as Meta-Harness implements this process through iterative code generation and evaluation, but retains a fixed development set and proposal policy. These constraints channel evolution along a single search trajectory, increasing the risk of converging to a local optimum. We make the improvement process itself adaptive by organizing search into branches with evolving development subsets and proposal policies. Each branch retains development cases solved by more of its leading harnesses than by those of other branches, drops cases solved by every leading harness across all branches, and revises its proposal policy using its own search history. To deploy the resulting complementary harnesses, we propose a router to select one development-selected branch head for each new input before execution. Across mathematical reasoning and agentic coding benchmarks, our system achieves relative improvements over Meta-Harness of 34.8% on Olympiad-level mathematical reasoning, 11.6% on Terminal-Bench 2.0, and 3.8% on SWE-bench Lite, with harness selection and router configuration based solely on development data. These results show that evolving branch objectives and proposal policies can yield complementary harnesses whose strengths a router combines without access to test outcomes.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 30 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The branching mechanism is a genuine extension of Meta-Harness and the evaluation design is honest, but three of four headline deltas are within sampling noise and the one large gain rests on a single unreplicated ablation. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 78 references say.

The paper says, at §4.1, Table 1

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The paper reports no seeds, error bars, or significance tests, and on the stated test sizes three of the four headline deltas over Meta-Harness are consistent with no effect: Math-Sonnet +3/200 (29.0% to 30.5%), Terminal-Bench 2.0 +3/58 (44.8% to 50.0%), and SWE-bench Lite +6/250 (63.6% to 66.0%).

more of objection 1

Only Math-Gemini +32/200 (46.0% to 62.0%) is a substantial, credible gain. The abstract's claim [b6] of improvements 'across mathematical reasoning and agentic coding benchmarks' therefore rests on essentially a single unseeded run, and the paper's framing of the TB2 and SWE gains as supporting broad applicability (Section 4.2) is not supported by the point estimates alone. The authors should either add replication with variance estimates and significance tests for these comparisons, or temper the claims to state clearly that only the Math-Gemini result is established as a statistically robust gain and that the other settings show consistent-direction but within-noise improvements.

The paper says, at §4.4 Table 2, §4.5 Table 3b

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.77):

“The component ablations support the combined design, but do not establish that each observed trajectory change caused its 9 Preprint subsequent test gain.”

Pith says: Objection 2 of 3 · major

The router's value and the component ablations are single-setting, single-run, and unreplicated. By Table 2, the router exceeds the stronger branch head by 4.0% on Math-Gemini and 1.7% on Terminal-Bench 2.0 and trails by one test case on Math-Sonnet and SWE-bench Lite — all within a few cases of sampling noise given test sizes as small as 58.

more of objection 2

Table 3b's 62.0% versus 51.0% (neither), 50.0% (optimizer-only), and 54.0% (pruning-only) is one unreplicated ablation on a single setting. The paper acknowledges at [b199] that the ablations do not establish that each observed trajectory change caused its subsequent test gain, but this caveat is placed in the conclusion while Sections 1 and 4.5 present the combined design as the validated configuration. The claim that pruning and proposal adaptation jointly drive performance needs replication (multiple seeds, ideally more than one benchmark) or an explicit re-framing of what the ablation supports.

The paper says, at §3, Eq. (2)

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The coverage votes c_b^t(x) that drive ownership reassignment in PRUNEDEVELOPMENT (Algorithm 1) are single-sample estimates of the Eq. (1) expectation over execution randomness, taken over a frontier of only q=5 harnesses per branch. If trajectories are stochastic, cases can move between branches on the basis of noise, and the drop rule for cases solved by every frontier harness assumes such cases carry no latent signal for future proposals.

more of objection 3

The paper reports no development-set variance, so the subset divergence documented in Figure 3 and Appendix B.1 is not yet established to reflect genuine complementary strengths rather than sampling noise. A variance analysis of development scores or a sensitivity check perturbing the pruning updates would directly address this, and would also clarify how the single-trajectory estimates in Eq. (2) interact with the expectation objective in Eq. (1).

5 minor objections

The paper says, at Table 1 and Table 2 captions

Pith says: minor objection 1 of 5

Each entry in Tables 1 and 2 is a single unseeded run; the captions should state this explicitly so readers do not infer precision from the reported point estimates.

The paper says, at Appendix D.2, Table 6

Pith says: minor objection 2 of 5

In the fourth data row the Terminal-Bench and SWE-bench values appear as '46.666.4', which reads as two numbers run together; the row should be split into '46.6' and '66.4'.

show 3 more

The paper says, at Figure 2

Pith says: minor objection 3 of 5

The Venn diagrams in the right panel are labeled 'schematic' but the coverage percentages are exact; either draw areas to scale or add a note explaining which quantities the areas encode.

The paper says, at §4.2

Pith says: minor objection 4 of 5

The claim that the system 'surpasses the strongest fixed harness in every setting' is true for the reported point estimates, but several of these deltas are also small (e.g., 66.0% versus 62.8% for mini-swe-agent on 250 tasks); the statistical caveat raised in the major comments applies here as well and should be stated in the results section rather than only in the conclusion.

The paper says, at §5, Table 5

Pith says: minor objection 5 of 5

The token-usage asymmetry is acknowledged at [b197], but the headline gains are not cost-normalized (ours consumes roughly 2x total tokens on Math and 1.5x-1.7x on the coding benchmarks); a one-sentence reminder in Section 4.2 that the reported gains come with this overhead would make the comparison easier to interpret.

What would settle it

Compute an exact/bootstrap test from the per-test-case pass/fail outcomes underlying Table 1 (paired, since both systems run on the identical held-out sets) for Math-Sonnet (N=200), TB2 (N=58), and SWE-bench Lite (N=250); if the 95% CI on the routed-minus-Meta-Harness difference includes zero for any of the three, restrict the headline claim to Math-Gemini. Then replicate the Table 3b four-cell ablation on Math-Gemini with at least 3 independent seeds of the full search; if the 62.0%-vs-51.0% gap does not reproduce within its CI, the branching mechanism's causal claim is unsupported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 30 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 58 s · 21,516 tokens read, 9,645 written
  3. Editor wrote the verdict above 13 s · 22,662 tokens read, 1,874 written
  4. Referee major revision: 3 major objections, 5 minor 31 s · 22,026 tokens read, 5,488 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The mechanism is the contribution here, and it is real. Organizing harness search into branches that move development cases between themselves by comparative frontier coverage, revise branch-local proposal guidance, and feed a development-trained router is a genuine extension of Meta-Harness, which keeps a fixed set and a single trajectory. The writing is clear, and the appendix traces (subset histories, guidance diffs, the iteration-13 head replacement) give the mechanism a concrete feel rather than a hand-waved one. I read the branching design as honestly novel against the cited quality-diversity and skill-adaptation work.

The strongest part is the evaluation design. Harness selection, guidance updates, router labeling, and GEPA optimization all touch development data only; the test sets are held out, and they explicitly cite Wang et al. on the search-task overstatement problem and avoid it. That is more discipline than most papers in this line show, and the limitation statements in the conclusion are candid as far as they go.

Now the soft spots, and they are load-bearing. There are no seeds, no error bars, no significance tests anywhere. From the stated set sizes, the routed-vs-Meta-Harness deltas are +32/200 on Math-Gemini, +3/200 on Math-Sonnet, +3/58 on TB2, and +6/250 on SWE. The latter three are consistent with no effect, and the router's own contribution over the stronger head is also within a few cases on three of four settings. So the abstract's claim of gains 'across mathematical reasoning and agentic coding' effectively rests on the single Math-Gemini run, and that run's causal story is carried by Table 3b: one unreplicated ablation on one setting, no seeds. The token accounting shows ours uses up to 2.3x total tokens on SWE, so the comparison is not compute-matched either. The paper states unequal token usage and limited repetitions as efficiency/reliability caveats, which is accurate but understates how much they threaten the headline numbers themselves. I also note no code or data is shipped.

I judge the coverage-vote concern secondary: the ownership margin over a q=5 frontier and the drop rule requiring all ten harnesses to solve a case aggregate enough to be reasonably noise-tolerant, unlike a three-case delta on N=58. And there is no test leakage, which keeps the circularity burden genuinely low.

Who is this for: anyone working on harness evolution or recursive self-improvement. The mechanism deserves referee time, and the fixes—seeded runs, error bars, a compute-matched Meta-Harness baseline, shipped code—are all feasible. As it stands it is a strong mechanism paper with an under-supported empirical claim; I would engage with it and push for the additional runs rather than desk-reject.

The strongest objection

All reported accuracies are single-pass, no-seed runs. From the stated test-set sizes in §4.1, the four routed-vs-Meta-Harness deltas are: Math-Gemini +32/200 (46.0%→62.0%), Math-Sonnet +3/200 (29.0%→30.5%), TB2 +3/58 (44.8%→50.0%), SWE +6/250 (63.6%→66.0%). Treated as independent proportions — even under the more favorable paired reading given a few-case counts — the latter three are far from significant; e.g. 26/58 vs 29/58 on TB2 is around p≈0.5, and 159/250 vs 165/250 on SWE is likewise null. Table 2 adds that the router's margin over the single stronger head is +4.0% (Math-Gemini), +1.7% (TB2), and minus one test case on both Math-Sonnet and SWE-bench Lite; again within a few cases. So three of four settings' headline claims, and the router contribution itself, are consistent with no effect, and the abstract's 'improvements over Meta-Harness across mathematical reasoning and agentic coding' rests effectively on a single Math-Gemini run. That run's causal attribution is carried by Table 3b (62.0% with both components vs 51.0% neither / 50.0% optimizer-only / 54.0% pruning-only) — a single unreplicated ablation on one setting with no seeds. The paper honestly states 'limited repeated evaluations... constrain conclusions about efficiency and reliability' [b197] and that ablations 'do not establish that each observed trajectory change caused its subsequent test gain' [b199], but frames these as efficiency/reliability caveats rather than as a threat to the headline numbers themselves. I judged the single-sample coverage-vote concern secondary: the ownership margin (Δ=2 over q=5 frontiers) and especially the drop rule (all 10 frontier harnesses must solve a case) aggregate over multiple harnesses and are reasonably noise-tolerant, whereas a three-case delta on N=58 is not.

I read the paper in good faith. The design is honest: harnesses selected on development data only, held-out test sets (§4.1), full token accounting (Appendix A), and explicit limitation statements ([b197], [b199]). The branching construction is clearly specified (Eq. 1–2, Algorithm 1), and Table 4 partly weakens a pure search-budget explanation of the Math-Gemini gain: Meta-Harness's test-best over its own pool is 49.0%, well below the routed 62.0%, suggesting the gain is not simply 'more candidates.' That is genuine supporting evidence and I credit it. My single most load-bearing concern is that the central empirical claim — deployable held-out gains over Meta-Harness across both mathematical reasoning and agentic coding — is carried by unseeded point estimates, and three of the four deltas are within a few test cases of zero effect. The reader's weakest_assumption (single-sample coverage votes in Eq. 2) is real but secondary: the ownership and drop rules aggregate over q=5–10 frontier harnesses and are comparatively noise-tolerant, while a six-case delta on N=250 is not. I therefore partially agree with the reader, who listed missing error bars and unreplicated ablations in the rationale; my attack sharpens the consequence — absent replication, the paper's own reported counts imply no significant improvement on three of four settings, so the abstract's generalization claim cannot stand as stated. This does not justify REJECT: the method is coherent, the evaluation design is careful, the Math-Gemini result (with Table 4) is promising, and every gap is fixable by adding seeds, error bars, and significance tests. I keep the reader's CONDITIONAL verdict and make the per-setting significance requirement explicit.

The referee's report

If the central claim holds, that branch-adaptive objectives and proposal policies yield complementary harnesses that a development-informed router can combine without test access, the paper would be a meaningful advance in practical recursive self-improvement. The evaluation design is a genuine strength: development/test separation is enforced throughout, and the paper directly confronts the search-task overstatement concern raised by Wang et al. by reporting only held-out performance. The mechanism-level analysis on Math-Gemini (Appendix B.2), including the recorded head replacement at iteration 13 and the concrete answer-check example in Section 4.3, is specific and falsifiable, and the paper's own limitation statements in the conclusion are candid. However, the statistical support for the headline claims is thin: no seeds, error bars, or significance tests are reported anywhere, and on the stated test sizes three of the four setting-level deltas over Meta-Harness are consistent with no effect.

Pith's reading of the paper (a machine's summary, not the authors')

Harness optimization — automatically redesigning the retrieval, tool interfaces, and control flow wrapped around an LLM — is a practical route to recursive self-improvement, but prior systems evolve along a single search trajectory with a fixed evaluation set and fixed proposal instructions. This paper claims that splitting the search into branches, each with a development subset and proposal policy that adapt to its own emerging strengths, discovers complementary harnesses that a development-trained router can deploy. The reported result: routed, development-selected systems beat the single-trajectory Meta-Harness baseline by 34.8% on Olympiad-level math (Gemini 3 Flash), 11.6% on Terminal-Bench 2.0, and 3.8% on SWE-bench Lite. If correct, the mechanism shows that the improvement process itself can be made adaptive without any access to test outcomes, and that no single best harness bounds what a routed mixture can achieve.

What carries the argument. The load-bearing object is adaptive branching search driven by a coverage-vote ownership rule. For each development case x, branch b computes the number of its top-q frontier harnesses that solve the case; the case is reserved exclusively for the branch whose coverage vote exceeds every other's by at least delta, and is removed from all subsets when every frontier solves it. This updates the branch objective, the expected reward over the branch's current development subset, which changes which candidate is the branch head and therefore which harness the proposer extends next. Alongside this, a periodic guidance update revises each branch's SKILL.md from local search records, separating repeated evidence from isolated successes. At deployment, a router optimized with GEPA on head-exclusive development cases picks one branch head before execution.

Core claim. The central claim is that changing which cases each branch is scored on — and letting each branch revise its own proposal guidance — redirects search toward harnesses with complementary strengths. Each branch counts how many of its top-q harnesses solve each development case; a case is owned exclusively by the branch whose count exceeds every other's by the margin delta = 2, and cases solved by every branch's frontier are dropped from all subsets. The paper argues this both makes each branch specialize, avoiding cases that give little signal. Branch-specific guidance, stored in SKILL.md and revised by the proposer, turns local successes and unresolved failures into distinct exploration priorities. The reported outcome is that the two final branch heads solve partly disjoint held-out cases — combined oracle coverage 4.0 to 9.0 percent above the stronger head — and a router trained on head-exclusive development cases selects among them before execution, approaching or surpassing the stronger head in each setting without test feedback.

The premise it rests on. The ownership rule treats each harness's solve or no-solve of a development case as a fixed fact, but that fact is estimated from a single trajectory per harness, so on stochastic executions the reassignment that redirects search could be driven by sampling noise rather than real capability differences.

If the paper is right (Pith's inferences, not the authors')

Harness evolution no longer needs a fixed evaluation set: search objectives can be re-targeted mid-run toward each branch's emerging strengths, reducing the risk of converging on a single local optimum.

Deployable gains are achievable with development-only selection: the router needs no test outcomes, and in two settings it exceeds the strongest individual branch head's test accuracy.

More frequent development-subset updates improve the routed system: 62.0 percent accuracy at every-iteration updates versus 53.0 percent at every-five-iterations on Math-Gemini, consistent with timely specialization.

Development pruning and proposal adaptation are complementary: enabling both yields 62.0 percent, while each alone yields 54.0 and 50.0 percent, showing that distinct subsets make local guidance more useful.

Routing can exceed the test-best individual harness: on Math-Gemini the router reaches 62.0 percent against 58.0 percent for the best harness in the candidate pool, so no single harness bounds the routed system.

Because the coverage votes that reassign development cases come from a single trajectory per harness, the method's benefit plausibly depends on how deterministic execution is; on highly stochastic environments, ownership reassignment could be partly noise-driven, and a multi-sample estimate of solve probabilities would test this directly.

The two hyperparameters of the ownership rule — the margin delta = 2 and the drop rule that removes cases solved by all frontiers — are not ablated, so they are the most likely place the reported gains depend on hand-tuned choices.

A natural extension is scaling beyond two branches; the paper only runs two branches, and it is unclear whether more branches continue to add complementary coverage or merely duplicate heads.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Harness optimization provides a practical setting for recursive self-improvement (RSI), where agent-generated modifications inform subsequent changes through execution feedback. Recent work such as Meta-Harness implements this process through iterative code generation and evaluation, but retains a fixed development set and proposal policy. These constraints channel evolution along a single search trajectory, increasing the risk of converging to a local optimum. We make the improvement process itself adaptive by organizing search into branches with evolving development subsets and proposal policies. Each branch retains development cases solved by more of its leading harnesses than by those of other branches, drops cases solved by every leading harness across all branches, and revises its proposal policy using its own search history. To deploy the resulting complementary harnesses, we propose a router to select one development-selected branch head for each new input before execution. Across mathematical reasoning and agentic coding benchmarks, our system achieves relative improvements over Meta-Harness of 34.8% on Olympiad-level mathematical reasoning, 11.6% on Terminal-Bench 2.0, and 3.8% on SWE-bench Lite, with harness selection and router configuration based solely on development data. These results show that evolving branch objectives and proposal policies can yield complementary harnesses whose strengths a router combines without access to test outcomes.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-30

The authors' main claim, as the classifier picked it out 1.00

“These results show that evolving branch objectives and proposal policies can yield complementary harnesses whose strengths a router combines without access to test outcomes.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.41 · human participants 0.03 · controlled comparison 0.90 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Harness optimization provides a practical setting for recursive self-improvement (RSI), where agent-generated modifications inform subsequent changes through execution feedback. background 1.00 0.07 0.03 asserted 1.00
Recent work such as Meta-Harness implements this process through iterative code generation and evaluation, but retains a fixed development set and proposal policy. background 0.95 0.05 0.02 asserted 1.00
These constraints channel evolution along a single search trajectory, increasing the risk of converging to a local optimum. gap 0.64 0.09 0.03 asserted 0.69
We make the improvement process itself adaptive by organizing search into branches with evolving development subsets and proposal policies. contribution 0.81 0.86 0.08 asserted 0.98
Each branch retains development cases solved by more of its leading harnesses than by those of other branches, drops cases solved by every leading harness across all branches, and revises its proposal policy using its own search history. method 0.99 0.87 0.07 asserted 0.98
To deploy the resulting complementary harnesses, we propose a router to select one development-selected branch head for each new input before execution. method 0.90 0.71 0.06 asserted 0.85
Across mathematical reasoning and agentic coding benchmarks, our system achieves relative improvements over Meta-Harness of 34.8% on Olympiad-level mathematical reasoning, 11.6% on Terminal-Bench 2.0, and 3.8% on SWE-bench Lite, with harness selection and router configuration based solely on development data. result 1.00 0.96 0.22 asserted 1.00
These results show that evolving branch objectives and proposal policies can yield complementary harnesses whose strengths a router combines without access to test outcomes. interpretation 1.00 0.92 0.85 asserted 0.99

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.91

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 6,329 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Pipeline overview. Left: each branch proposes and evaluates harnesses using its current development subset, branch history, and proposal guidance. Periodic guidance updates (upper inset) translate local search experience into priorities for subsequent proposals. Right: development sub￾sets are updated by comparing how many leading harnesses in each branch solve each case. Bottom: a router selects one development-best branch harness for each new input before execution. Diversity through adaptive search inputs. LLM-driven evolutionary systems encourage novelty through diverse…
Figure 2 of the paper
Figure 2. Left: per-iteration test-best accuracy and discovered mechanisms on Math–Gemini over
Figure 3 of the paper
Figure 3. Development-subset changes across iterations in four settings. Filled segments show exact
Figure 4 of the paper
Figure 4. Branch-specific SKILL.md changes on Math–Gemini from iteration-15 gradients. Math (Sonnet 4.5), Branch 1 SKILL.md diff at iteration 5 1 Accuracy is the primary objective. 2 Implement three mechanism-level candidates. 3 Use branch-local scores and traces as evidence. 4 ## Retrieval Guidance 5 + Geometry-only BM25 retrieval is the established foundation for this branch. Mixing indexes 6 + has been tried twice without benefit. 7 ## Reasoning Structure Guidance 8 + Dual-path independent solving with an agreement shortcut and conditional adjudication is the 9 + strongest validated…
Figure 5 of the paper
Figure 5. Branch-specific SKILL.md changes on Math–Sonnet from iteration-5 gradients. 16
Figure 6 of the paper
Figure 6. Branch-specific SKILL.md changes on Terminal-Bench 2.0 from iteration-5 gradients. SWE-bench Lite (Sonnet 4.5), Branch 1 SKILL.md diff at iteration 5 1 Change mechanisms rather than tuning constants. 2 Use branch-local scores and traces as evidence. 3 Implement the required candidate set. 4 - Best-of-N is the highest-value direction. 5 + Best-of-N is cautionary. Iteration 3 regressed to 2/30 because dual-attempt and 6 + test-selection overhead did not pay off at this task difficulty. 7 + Context-window trimming or summarization is harmful. Iteration 5 regressed to 1/30 after 8 +…
Figure 7 of the paper
Figure 7. Branch-specific SKILL.md changes on SWE-bench Lite from iteration-5 gradients. 17

Figures are the authors', reproduced from the paper.

References (78)
  1. [1] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA : Refle... on Pith
  2. [2] Anthropic . Claude Sonnet 4.5 System Card . Technical report, Anthropic, September 2025. URL https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf on Pith
  3. [3] Anthropic . Claude Opus 4.6 System Card . Technical report, Anthropic, February 2026. URL https://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf on Pith
  4. [4] Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT : How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023 on PithPith's reviewarXiv
  5. [7] Tulsee Doshi. Gemini 3 Flash : Frontier intelligence built for speed. Google Blog, December 2025. URL https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/ on Pith
  6. [9] Hanghui Guo, Weijie Shi, Zhangze Chen, Shengxiang Xu, Yishu Wang, Yimei Zhang, Wangze Ni, Jia Zhu, and Shimin Di. DREvo : Distilling recalibrated historical experience for harness self-evolution. arXiv preprint arXiv:2607.26722, 2026 on PithPith's reviewarXiv
  7. [12] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3 0 (1): 0 79--87, 1991 on Pith
  8. [13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench : Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024 on Pith
  9. [15] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy : Compiling declarative language model calls into self-impr... on PithPith's reviewarXiv
  10. [16] KRAFTON AI and Ludo Robotics . Terminus-KIRA : Boosting frontier model performance on Terminal-Bench with minimal harness, 2026. URL https://github.com/krafton-ai/KIRA on Pith
  11. [18] Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19 0 (2): 0 189--223, 2011 on Pith
  12. [19] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances i... on Pith
  13. [24] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. RouteLLM : Learning to route llms with preference data. arXiv preprint arXiv:2406.18665, 2024 on PithPith's reviewarXiv
  14. [25] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with ``gradient descent'' and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023 on Pith
  15. [26] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3 0 (4): 0 333--389, 2009. doi:10.1561/1500000019 on Pithdoi:10.1561/1500000019
  16. [27] Biswa Sengupta and Jinhua Wang. HARBOR : Automated harness optimization. arXiv preprint arXiv:2604.20938, 2026 on PithPith's reviewarXiv
  17. [28] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023 on Pith
  18. [31] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024 a on Pith
  19. [32] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent : Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, 2024 b . URL https://arxiv.org/abs/... on PithPith's reviewarXiv
  20. [33] Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. SkillOpt : Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026 on PithPith's reviewarXiv
show 58 more
  1. [34] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023 on Pith
  2. [35] Jiaqi Yuan, Jialu Wang, Zihan Wang, Qingyun Sun, Ruijie Wang, and Jianxin Li. AgenticGEO : A self-evolving agentic system for generative engine optimization. arXiv preprint arXiv:2603.20213, 2026 on PitharXiv
  3. [36] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad : Automatic ``differentiation'' via text. arXiv preprint arXiv:2406.07496, 2024 on PithPith's reviewarXiv
  4. [37] Hanqing Zeng, Yinglong Xia, Zhuokai Zhao, Chuan Jiang, Qiang Zhang, Jiayi Liu, Qunshu Zhang, Lizhu Zhang, Xiangjun Fan, and Benyu Zhang. S'MoRE : Structural mixture of residual experts for parameter-efficient LLM fine-tuning. In Advances in Neural Information Processing System... on Pith
  5. [39] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow : Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024 on PithPith's reviewarXiv
  6. [40] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
  7. [41] Self-Evolving Multi-Agent Systems via Decentralized Memory , author=. arXiv preprint arXiv:2605.22721 , year= on PithPith's reviewarXiv
  8. [42] Zeng, Hanqing and Xia, Yinglong and Zhao, Zhuokai and Jiang, Chuan and Zhang, Qiang and Liu, Jiayi and Zhang, Qunshu and Zhang, Lizhu and Fan, Xiangjun and Zhang, Benyu , booktitle= on Pith
  9. [43] DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts , author=. arXiv preprint arXiv:2606.01062 , year= on PithPith's reviewarXiv
  10. [44] Token-Level LLM Collaboration via FusionRoute , author=. arXiv preprint arXiv:2601.05106 , year= on PithPith's reviewarXiv
  11. [45] Rethinking Transfer in Continual Learning: A Replay-Based Realisation , author=. arXiv preprint arXiv:2607.15587 , year= on PithPith's reviewarXiv
  12. [46] Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume = on Pith
  13. [47] Deep learning , author=. 2016 , publisher= on Pith
  14. [48] Meta-Harness: End-to-End Optimization of Model Harnesses , author=. arXiv preprint arXiv:2603.28052 , year= on PithPith's reviewarXiv
  15. [49] Self-Harness: Harnesses That Improve Themselves , author=. arXiv preprint arXiv:2606.09498 , year= on PithPith's reviewarXiv
  16. [50] Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents , author=. arXiv preprint arXiv:2607.22688 , year= on PithPith's reviewarXiv
  17. [51] HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems , author=. arXiv preprint arXiv:2606.01779 , year= on PithPith's reviewarXiv
  18. [52] Guo, Hanghui and Shi, Weijie and Chen, Zhangze and Xu, Shengxiang and Wang, Yishu and Zhang, Yimei and Ni, Wangze and Zhu, Jia and Di, Shimin , journal= on Pith
  19. [53] Sengupta, Biswa and Wang, Jinhua , journal= on Pith
  20. [54] AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design , author=. arXiv preprint arXiv:2608.13560 , year= on PithPith's reviewarXiv
  21. [55] Continual Harness: Online Adaptation for Self-Improving Foundation Agents , author=. arXiv preprint arXiv:2605.09998 , year= on PithPith's reviewarXiv
  22. [56] Huang, Lisheng and Yang, Chen and Zhou, Hao and Song, Huatong and Chen, Zongchao and Le, Ran and Song, Yang and Zhao, Wayne Xin and Zhang, Tao , journal= on Pith
  23. [57] Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents , author=. arXiv preprint arXiv:2605.30621 , year= on PithPith's reviewarXiv
  24. [58] Yang, Yifan and Gong, Ziyang and Huang, Weiquan and Yang, Qihao and Zhou, Ziwei and Huang, Zisu and Li, Yan and Gao, Xuemei and Dai, Qi and Liu, Bei and Qiu, Kai and Yang, Yuqing and Chen, Dongdong and Yang, Xue and Luo, Chong , journal= on Pith
  25. [59] Code as Agent Harness , author=. arXiv preprint arXiv:2605.18747 , year= on PithPith's reviewarXiv
  26. [60] Automated Design of Agentic Systems , author=. arXiv preprint arXiv:2408.08435 , year= on PithPith's reviewarXiv
  27. [61] Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xionghui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , journal= on Pith
  28. [62] Agrawal, Lakshya A. and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J. and Jiang, Meng and Potts, Christopher and Sen, Koushik and Dimakis, Alexandros G. and Stoica, Ion and ... on Pith
  29. [63] Illuminating Search Spaces by Mapping Elites , author=. arXiv preprint arXiv:1504.04909 , year= on PithPith's reviewarXiv
  30. [64] Curriculum Learning , author=. Proceedings of the 26th International Conference on Machine Learning , pages= on Pith
  31. [65] Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design , author=. arXiv preprint arXiv:2012.02096 , year= on PithPith's reviewarXiv
  32. [66] Chen, Lingjiao and Zaharia, Matei and Zou, James , journal= on Pith
  33. [67] Ong, Isaac and Almahairi, Amjad and Wu, Vincent and Chiang, Wei-Lin and Wu, Tianhao and Gonzalez, Joseph E. and Kadous, M. Waleed and Stoica, Ion , journal= on Pith
  34. [68] Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , year= on Pith
  35. [69] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle= on Pith
  36. [70] Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , year= on Pith
  37. [71] Automatic Prompt Optimization with ``Gradient Descent'' and Beam Search , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year= on Pith
  38. [72] Large Language Models as Optimizers , author=. International Conference on Learning Representations , year= on Pith
  39. [73] Yuksekgonul, Mert and Bianchi, Federico and Boen, Joseph and Liu, Sheng and Huang, Zhi and Guestrin, Carlos and Zou, James , journal= on Pith
  40. [74] Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , journal= on Pith
  41. [75] Abandoning Objectives: Evolution Through the Search for Novelty Alone , author=. Evolutionary Computation , volume= on Pith
  42. [76] Population Based Training of Neural Networks , author=. arXiv preprint arXiv:1711.09846 , year= on PithPith's reviewarXiv
  43. [77] Self-Paced Learning for Latent Variable Models , author=. Advances in Neural Information Processing Systems , year= on Pith
  44. [78] Adaptive Mixtures of Local Experts , author=. Neural Computation , volume= on Pith
  45. [79] Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah D. , booktitle= on Pith
  46. [80] Self-Refine: Iterative Refinement with Self-Feedback , author=. arXiv preprint arXiv:2303.17651 , year= on PithPith's reviewarXiv
  47. [81] Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution , author=. arXiv preprint arXiv:2309.16797 , year= on PithPith's reviewarXiv
  48. [82] Voyager: An Open-Ended Embodied Agent with Large Language Models , author=. arXiv preprint arXiv:2305.16291 , year= on PithPith's reviewarXiv
  49. [83] Rethinking the Evaluation of Harness Evolution for Agents , author=. arXiv preprint arXiv:2607.12227 , year= on PithPith's reviewarXiv
  50. [84] Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle= on Pith
  51. [85] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. arXiv preprint arXiv:2601.11868 , year= on PithPith's reviewarXiv
  52. [86] Robertson, Stephen and Zaragoza, Hugo , journal=. The Probabilistic Relevance Framework:. 2009 , doi= on Pith
  53. [87] Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , booktitle=. 2024 , url= on Pith
  54. [88] Novikov, Alexander and V. arXiv preprint arXiv:2506.13131 , year= on PithPith's reviewarXiv
  55. [89] Yuan, Jiaqi and Wang, Jialu and Wang, Zihan and Sun, Qingyun and Wang, Ruijie and Li, Jianxin , journal= on Pith
  56. [90] Doshi, Tulsee , year= on Pith
  57. [91] 2025 , month=sep, url= on Pith
  58. [92] 2026 , month=feb, url= on Pith
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment