Pith.

Artificial Intelligence · arXiv:2610.02826 v1 · submitted 2 Oct 2026 · 40 references

Scaling Trajectories for Complex Tasks through Recursive Self-Rewrite

Zongxia Li, Yucheng Shi, Zhongzhi Li, Junyao Yang, Ruhan Wang, Chengsong Huang, Fuxiao Liu, Haitao Mi

show 2 more authors

Jordan Boyd-Graber, LeoweiLiang

The paper says

“Successful trajectories on difficult tasks provide valuable supervision for model improvement, but specialized harnesses introduce interventions that may be unavailable during deployment.”

more of the abstract

We propose Recursive Self-Rewrite (RSR), a framework that uses one base model, Qwen-3.8-27B, to discover successful solutions under diverse harnesses and reconstruct them as training trajectories under a general harness. A planner extracts procedures into runbooks, a critic screens for verifier and solution leakage and guides recursive revision, and an executor follows qualified runbooks in fresh sandboxes. Across approximately 3K self-curated terminal tasks, three harnesses jointly solve 759 tasks, 34.3% more than the strongest individual harness in the recorded pool. RSR expands 2,001 successful source trajectories into 11,094 rewritten trajectories for supervised finetuning. Training on these trajectories outperforms both the base model and direct trajectory SFT. Compared with the base model, pass@3 increases from 57.0% to 74.2% on Terminal-Bench 2, from 1.5% to 9.1% on Terminal-Bench 4, from 39.0% to 63.0% on our self-curated Terminal-Bench Hard, and from 3.0% to 6.0% on our Software Terminal-Bench. Process reward on Long-Horizon Terminal-Bench rises from 0.21 to 0.29. These results show how diverse harness-assisted experiences can be reconstructed into reusable capabilities for a model operating under a general harness.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The rewrite mechanism is genuinely new and the external benchmark gains hold up; the same-model leakage critic that is never audited is a real weakness that chiefly inflates the self-curated numbers, not the whole claim. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 40 references say.

The paper says, at §2.2 (Critic: Filtering Leakage; Trajectory Filtering)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.60):

“Verifier success does not imply quality-audit acceptance.”

Pith says: Objection 1 of 4 · major

The entire data-integrity guarantee rests on the same Qwen-3.8-27B model acting as critic: it screens runbooks for verifier and solution leakage and later flags 'values that appear in a trajectory but cannot be derived from the task or the environment.'

more of objection 1

No independent audit of either filter is reported: no critic rejection statistics, no agreement rate against a human or a stronger model, and no adversarial test in which deliberately leaked runbooks are shown to be caught. This matters because the 86% rewrite pass rate (11,094/12,893) is roughly six times the 13.7% pooled discovery rate from Table 2; with M=4 fresh executions at temperature 0.7 per accepted runbook, a runbook that retains the finished deliverable or the verifier's checks could reproduce that rate trivially. As the authors themselves note, 'Verifier success does not imply quality-audit acceptance' (b150) — but no quality audit of the surviving trajectories is shown either. At minimum, the paper should report critic rejection rates per leakage category and an independent (human or stronger-model) audit of a sample of accepted runbooks and rewritten trajectories. This is load-bearing for the TBH (39→63) and SWR100 (3→6) gains, which come from the same task families used in training.

The paper says, at §3.1, §3.5, Table 5 (train/eval overlap)

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The training pool is built from RST tasks (420) and self-constructed SWR tasks (~2,509), while TBH is 'self-curated Terminal-Bench Hard' from RST [11] and SWR100 and LHTB are also self-curated and labeled 'ours' in Table 5. The paper nowhere states that the TBH, SWR100, and LHTB evaluation tasks are disjoint from the tasks whose trajectories appear in the RSR training set (11,094 rewrites covering 975 unique tasks).

more of objection 2

If any training trajectories concern the same tasks that are later evaluated, the pass@3 gains on these three benchmarks measure retrieval of memorized procedures rather than the claimed reusable general-harness capability (b236). The external benchmarks (TB2, TB4, TB3) are not subject to this concern and do support the claim, but the paper must either quantify the overlap or report the self-curated results only on provably held-out splits; the process-reward gain on LHTB (0.21→0.29, with 0/46 completions in all groups) likewise uses the paper's own dense-reward scheme and would need the same hold-out guarantee.

The paper says, at §3.6, Table 5 (Direct SFT comparison)

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The headline comparison 'RSR vs Direct SFT' is confounded by data volume: Direct SFT trains on the 2,001 passing source rollouts, while RSR trains on 11,094 rewritten trajectories plus 766 Terminus 2 passes — roughly six times more data. The paper frames this as 'rewriting improves experience scaling,' which is legitimate as a design goal, but the claim in the …

more of objection 3

… abstract and conclusion (b11, b236) that rewriting per se, rather than simply more data, is what drives the gains is not isolated. A size-matched or data-abundance-matched control (e.g., Direct SFT on a comparably large pool of raw or lightly cleaned single-harness trajectories) would separate the rewriting mechanism from the scaling effect. The TB2 regression of Direct SFT relative to Base (−3.6 pts) is attributed to 'circular and deadend behaviors' on the basis of an inspection of three trajectories (b192-b193); this is anecdotal and should be quantified.

The paper says, at §3.4, Figure 5 (case-study statistics)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.70):

“Shown errors do not establish the final failure cause.”

Pith says: Objection 4 of 4 · major

The case studies in Figure 5 report pass rates of 7/12 (58.3%) and 7/8 (87.5%) on two tasks, and the paper correctly hedges that shown errors do not establish the final failure cause (b146, b149). These are illustrative and are not the basis of the main claims, so this is a minor concern overall; however, the selection of these two …

more of objection 4

… tasks is not described as random, and the 86% aggregate rewrite pass rate cited in §3.4 would be more informative if accompanied by a distribution over the 975 rewritten tasks rather than a single pooled figure, since a high pooled rate with heavy mass on easy tasks would weaken the claim that runbooks transfer 'difficult' procedures.

5 minor objections

The paper says, at Throughout

Pith says: minor objection 1 of 5

Typographical issues: 'T rajectories' in the title, 'Experince' in the §2.2 heading, 'doamin' in the Figure 4 caption, 'filted' in §3.1, and the author byline 'LeoweiLiang' without a space. These should be corrected in revision.

The paper says, at §3.2, footnote 1

Pith says: minor objection 2 of 5

The parenthetical that 'This part can be done with Jev for faster inference [3]' is an unexplained aside that reads like an advertisement; it should either be removed or supported by an actual experiment showing Jev is used anywhere in the reported pipeline.

show 3 more

The paper says, at Table 6

Pith says: minor objection 3 of 5

The 'Union of all three' row in Table 6 reports pooled success rates across harnesses (68.5/70.0/4.3) but the column header says 'success rates (%)'; it should be clarified that this is a union-over-harnesses count per task, not a per-run pass rate, and whether it is pass@1 or pass@3 analogues should be stated.

The paper says, at §3.4.1, Table 4

Pith says: minor objection 4 of 5

The overlap metrics are computed on only two tasks (12 and 8 rewrites), and the OpenFOAM within-runbook vs different-runbook differences are inconclusive on most metrics; the paragraph's conclusion that 'shared runbooks are associated with more similar command-level behavior' is supported mainly by the Markdown task. A sentence acknowledging the small sample size would be appropriate.

The paper says, at §3.3, Table 1

Pith says: minor objection 5 of 5

The equal-rollout control is a strength, but it is reported only for task coverage; per-harness rollout counts still differ in the full pool (5,405/3,777/5,416), so the domain-level percentages in Figure 4 should state whether they use the pooled or the equalized rollout counts, or both.

What would settle it

Independently audit a random sample of ~150 retained runbooks (spanning tasks in TBH and SWR100) with an auditor who has access to the task verifier and the reference solution, classifying each as procedural-only or solution-leaking (i.e., the runbook plus the public instruction reveals the deliverable or the verifier's checks to the executor). Then, restricted to the auditor-clean subset, recompute the rewrite pass rate and, if feasible, retrain the RSR checkpoint and re-measure pass@3 and mean per-run accuracy on TBH and SWR100. If the auditor flags a large fraction as leaking, or if the clean-subset rewrite pass rate collapses toward the 13.7% base rate, or if the clean-subset training gains over Direct SFT shrink substantially, the critic's screening is the load-bearing failure and the headline self-curated gains are inflated.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 4 s · 18,743 tokens read, 6,474 written
  3. Editor wrote the verdict above 30 s · 19,931 tokens read, 2,782 written
  4. Referee major revision: 4 major objections, 5 minor 1 min 17 s · 21,552 tokens read, 2,340 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The actual new thing here is the recursive rewrite pipeline: planner extracts runbooks, a critic screens them for leakage, and an executor re-solves under a general harness in fresh sandboxes, converting multi-harness successes into deployable demonstrations. Harness-Zero distills behaviors directly; this reconstructs through screened procedures, and the distinction is substantive. If it works, harnesses become discovery instruments for self-improvement, which is a genuine within-field contribution for agent training.

What the paper does well deserves credit. The multi-harness discovery analysis is careful, and the equal-rollout control (1,247 tasks with matched counts, complementarity still holding at 23.5%) is the kind of check that usually gets skipped. The authors report results that don't flatter them: Direct SFT actively hurts TB2, and they say so, with a plausible dead-end diagnosis. The case studies show real failures (Markdown 7/12, OpenFOAM 7/8), which is evidence rewrites aren't trivially passing. The arithmetic is consistent, and the gains on external benchmarks — TB2 57→74.2, TB4 1.5→9.1, TB3 0→9.5 — are real and can't be waved away.

Now the soft spot, and the stress test has it right. The leakage critic is the same 27B base model that later trains on the surviving data, and it is never independently audited. The 86% rewrite pass rate against the 13.7% discovery base rate is a sharp internal signal that runbooks carry more than abstract procedure; with M=4 fresh executions per accepted runbook, acceptance is cheap, and the only gate is a self-critic whose rejection statistics and agreement with any reference are not reported. If runbooks retain the deliverable or the verifier's checks, the self-curated gains (TBH 39→63, SWR100 3→6, LHTB process reward 0.21→0.29) are inflated, since those benchmarks come from the same task pool and the same group.

But this tempers rather than voids. The external benchmarks still move, and the failure cases plus the command-similarity table suggest runbooks aren't producing token-identical trajectories. The paper acknowledges verifier success does not imply quality acceptance and the distribution-mismatch issue, but nowhere addresses critic reliability. That is the gap: they need an independent audit of the critic — a held-out leakage set, human-checked runbooks, or agreement with a second model — plus confidence intervals on pass@3, and ideally the harnesses made public.

This deserves a serious referee. Send it out, ask hard questions about the critic, expect a revision. Anyone working on agent self-improvement, harness scaling, or trajectory distillation gets value from it.

The strongest objection

The central claim (b11) is that RSR's rewritten trajectories confer reusable general-harness capability, so the pass@3 gains on TBH (39→63) and SWR100 (3→6) — benchmarks drawn from the same task families as the training pool (RST [11] and the self-constructed SWR) — measure genuine re-solution. The integrity of that claim rests entirely on the critic in Section 2.2 (screening runbooks for verifier and solution leakage) and on the trajectory filter, both executed by the same Qwen-3.8-27B model that is later trained on the surviving data, and neither is independently audited anywhere in the paper. The sharpest internal red flag: the rewritten-trajectory pass rate is 86% (11,094/12,893 passing rewrites), about six times the 13.7% pooled discovery rate, implying runbooks carry far more solution-specific information than raw attempts. The executor samples M=4 fresh executions per accepted runbook at temperature 0.7, so acceptance is cheap; a runbook that retains the finished deliverable or the verifier's checks could reproduce the 86% rate trivially. The only gate is the critic, whose rejection statistics and agreement with any reference are not reported, and the trajectory filter ('We use the model itself to flag values that appear in a trajectory but cannot be derived from the task or the environment') is the same circular mechanism. The paper acknowledges verifier-vs-quality separation (b150) and distribution mismatch (b27, b29), but nowhere addresses critic reliability or the contamination of a training pool that is also the source of the self-curated evaluation sets. If leakage passes this gate, the most dramatic self-curated numbers are inflated; the external TB2 (57→74.2), TB4 (1.5→9.1), and TB3 gains would still indicate a genuine effect, which is why this tempers rather than voids the claim.

The reader's weakest_assumption — that the same 27B base model acting as critic reliably screens runbooks and rewritten trajectories for verifier and solution leakage, so that the verified training data reflects genuine re-solution rather than transferred answers — is precisely the concern I find most load-bearing, and I agree with it fully. My reading sharpens it in two ways. First, the 86% rewrite pass rate versus a 13.7% raw discovery rate is the strongest internal evidence that accepted runbooks carry substantial task-specific content; the paper reports this disparity without any decomposition or independent check, and Section 2.2 gives no critic rejection statistics. Second, the trajectory-level leakage filter is the same model judging underived values, so the audit is circular by construction. The consequence is concrete: the headline gains on TBH and SWR100 come from benchmarks drawn from the same task families as the ~2.9K training pool (RST and self-constructed SWR), so a critic that misses leakage inflates exactly the numbers that make the abstract's claim vivid. I do not escalate to REJECT because the external benchmarks (TB2, TB4, TB3) lie outside the training pool, and the gains there — plus the plausible mechanism of transferring procedural, verification, and recovery behaviors — provide independent support that some genuine capability transfer occurs. The concern therefore tempers the magnitude and the cleanliness of the self-curated component rather than voiding the central mechanism. I also verified the reader's arithmetic (759 = 565+194 with 34.3% relative gain; 2,001 source trajectories expanding to 11,094 passing rewrites at 86% of 12,893; K=4 runbooks and M=4 executions consistent with the reported counts) and found the pipeline internally consistent. The absence of confidence intervals on pass@3 and the non-public harnesses and self-curated benchmarks are additional, secondary gaps that the reader already noted. No independent support (machine-checked proofs, released pipeline, or public benchmarks) offsets the missing external audit of the critic, so CONDITIONAL remains the right verdict: the recipe should not be broadly adopted until an independent leakage audit and clean-subset re-evaluation either confirm or refute the 86%-pass-rate interpretation.

The referee's report

If the central claim holds, the paper makes a genuinely useful contribution: it shows that diverse harnesses can serve as discovery tools whose by-products can be converted into general-harness supervision. The numbers are internally consistent (759 = 565+194; 2,001 = 766+599+636; 11,094/12,893 = 86%), and the multi-harness complementarity result is strengthened by an equal-rollout-count control (b108-b111). The command-similarity analysis in Table 4 lends some support to the claim that runbooks alter execution structure. The external-benchmark gains (TB2, TB3, TB4) are the most convincing evidence that the trained model genuinely re-solves under a general harness. However, two load-bearing pillars are under-supported: the integrity of the leak filter (the critic is the same 27B model, never independently audited) and the absence of any stated hold-out policy for the self-curated eval benchmarks, which come from the same task families as the training pool. Three of the six headline results (TBH, SWR100, LHTB) therefore cannot currently be separated from data-memorization or leakage effects.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that successful trajectories a weak model reaches only under specialized execution harnesses can be distilled back into the model's own capabilities. It proposes Recursive Self-Rewrite (RSR), where one base model plans, critiques, and re-executes its own successes under a single general harness, producing cleaner demonstrations that survive deployment without the original scaffolding. The strongest result is that finetuning on these rewritten trajectories outperforms both the base model and direct finetuning on harness-collected rollouts, with pass@3 rising from 57.0% to 74.2% on Terminal-Bench 2 and from 39.0% to 63.0% on the self-curated hard split. If correct, this makes diverse harnesses discovery instruments that expand what a fixed model can learn, rather than merely inference-time controllers.

What carries the argument. The load-bearing object is the runbook and the planner–critic–executor loop around it. A runbook is a structured, harness-free description of how a task was solved: the required end state, key milestones, useful checks, recovery strategies, and pitfalls, deliberately stripped of the finished deliverable. The critic screens each candidate runbook for verifier leakage (revealing what the hidden task checker checks) and solution leakage (revealing answers the executor should derive itself), rejecting candidates and feeding critique back to the planner for up to N revisions; a model-based filter later flags trajectory values that cannot be derived from the task or environment. The claim is that this screening is what lets harness-specific success transfer into general-harness behaviour: the runbook carries the procedure, the critic keeps out the answer, and the executor must re-derive the solution in a fresh sandbox, with private runbooks excluded from training data.

Core claim. Recursive Self-Rewrite claims that solutions a model can only reach with specialized external control can be reconstructed as general-harness demonstrations and learned by the same model. The pipeline runs Qwen-3.8-27B in three roles: a planner turns each source trajectory into a runbook of milestones and checks, a critic rejects runbooks that leak verifier internals or solution content and feeds critique back for recursive revision, and an executor follows an approved runbook in a fresh sandbox under a single general harness, keeping only passing executions. Across approximately 3K self-curated terminal tasks, three harnesses jointly solve 759 tasks, 34.3% more than the strongest individual harness, and RSR expands 2,001 successful source trajectories into 11,094 verified training trajectories. Supervised finetuning on those rewrites beats both the base model and direct trajectory SFT: pass@3 rises from 57.0% to 74.2% on Terminal-Bench 2, 1.5% to 9.1% on Terminal-Bench 4, 39.0% to 63.0% on the self-curated hard split, and 3.0% to 6.0% on the software split, while process reward on the long-horizon benchmark rises from 0.21 to 0.29 (with all groups still completing zero of its tasks).

The premise it rests on. The same 27B model that solved the tasks, acting as critic and leakage filter, reliably distinguishes legitimately derivable procedure from verifier leakage and hidden answer transfer, so the 11,094 'verified' trajectories reflect genuine re-solution rather than transferred answers.

If the paper is right (Pith's inferences, not the authors')

Three harnesses on the same base model solve 759 of roughly 3K terminal tasks, 34.3% more than the strongest individual harness, and pairwise unions beat every single harness even at equal rollout counts, so harness diversity expands the set of tasks a fixed model can solve.

No harness leads in every domain, and each contributes tasks no other harness solves, so in this framework specialised harnesses function as discovery tools rather than just inference-time controllers.

Training on rewritten trajectories beats direct SFT by 20.8, 4.1, 4.6, 7.0, and 3.0 pass@3 points on five benchmarks, implying standardised re-execution converts harness-assisted success into supervision the model can actually absorb.

The process multiplies experience without new human annotation, expanding 2,001 source successes into 11,094 verified trajectories, so a single verified solve can yield many effective training demonstrations.

Executions sharing a runbook show higher command overlap than executions under different runbooks, indicating the runbook itself shapes the rewritten behaviour rather than the task alone determining it.

If RSR is right, the practical ceiling on self-improvement shifts from how hard a task is to how reliably the critic blocks leakage: any scaffolding that lets a weak model reach a solution it otherwise could not becomes a data-collection instrument for its own training set.

The rewrite pass rate of 86% against a 13.7% raw discovery rate implies runbooks carry substantial solution-specific information, and that gap is the strongest available test of whether the critic's leakage screen is working; a held-out audit of that gap would settle the method's integrity.

The recipe — discover under permissive control, then re-derive under the deployment harness — should transfer beyond terminals to any domain with a verifier and a definable notion of leakage, such as software engineering, web agents, or robotics.

A testable extension is to vary the discovery harness's mechanical support while holding the model fixed and measure whether rewriting gains shrink as harness support shrinks, isolating whether RSR distills a reusable procedure or simply replays transferred answers.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Successful trajectories on difficult tasks provide valuable supervision for model improvement, but specialized harnesses introduce interventions that may be unavailable during deployment. We propose Recursive Self-Rewrite (RSR), a framework that uses one base model, Qwen-3.8-27B, to discover successful solutions under diverse harnesses and reconstruct them as training trajectories under a general harness. A planner extracts procedures into runbooks, a critic screens for verifier and solution leakage and guides recursive revision, and an executor follows qualified runbooks in fresh sandboxes. Across approximately 3K self-curated terminal tasks, three harnesses jointly solve 759 tasks, 34.3% more than the strongest individual harness in the recorded pool. RSR expands 2,001 successful source trajectories into 11,094 rewritten trajectories for supervised finetuning. Training on these trajectories outperforms both the base model and direct trajectory SFT. Compared with the base model, pass@3 increases from 57.0% to 74.2% on Terminal-Bench 2, from 1.5% to 9.1% on Terminal-Bench 4, from 39.0% to 63.0% on our self-curated Terminal-Bench Hard, and from 3.0% to 6.0% on our Software Terminal-Bench. Process reward on Long-Horizon Terminal-Bench rises from 0.21 to 0.29. These results show how diverse harness-assisted experiences can be reconstructed into reusable capabilities for a model operating under a general harness.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-05

The authors' main claim, as the classifier picked it out 0.99

“These results show how diverse harness-assisted experiences can be reconstructed into reusable capabilities for a model operating under a general harness.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.27 · human participants 0.05 · controlled comparison 0.95 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Successful trajectories on difficult tasks provide valuable supervision for model improvement, but specialized harnesses introduce interventions that may be unavailable during deployment. background 0.88 0.09 0.05 hedged possibility 0.63
We propose Recursive Self-Rewrite (RSR), a framework that uses one base model, Qwen-3.8-27B, to discover successful solutions under diverse harnesses and reconstruct them as training trajectories under a general harness. contribution 0.83 0.88 0.12 asserted 0.97
A planner extracts procedures into runbooks, a critic screens for verifier and solution leakage and guides recursive revision, and an executor follows qualified runbooks in fresh sandboxes. method 1.00 0.79 0.04 asserted 0.98
Across approximately 3K self-curated terminal tasks, three harnesses jointly solve 759 tasks, 34.3% more than the strongest individual harness in the recorded pool. result 1.00 0.95 0.09 asserted 1.00
RSR expands 2,001 successful source trajectories into 11,094 rewritten trajectories for supervised finetuning. method 0.77 0.95 0.06 asserted 1.00
Training on these trajectories outperforms both the base model and direct trajectory SFT. result 1.00 0.94 0.26 asserted 1.00
Compared with the base model, pass@3 increases from 57.0% to 74.2% on Terminal-Bench 2, from 1.5% to 9.1% on Terminal-Bench 4, from 39.0% to 63.0% on our self-curated Terminal-Bench Hard, and from 3.0% to 6.0% on our Software Terminal-Bench. result 1.00 0.95 0.13 asserted 1.00
Process reward on Long-Horizon Terminal-Bench rises from 0.21 to 0.29. result 1.00 0.94 0.06 asserted 1.00
These results show how diverse harness-assisted experiences can be reconstructed into reusable capabilities for a model operating under a general harness. interpretation 1.00 0.83 0.86 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-05: 1 request, 7,117 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (40)
  1. [11] Zhongzhi Li, Yucheng Shi, Zongxia Li, Ruhan Wang, Anhao Li, Zixun Huang, Junyao Yang, Lei Ke, Ninghao Liu, Haitao Mi, et al. Recursive synthesis for long-horizon terminal tasks.arXiv preprint arXiv:2608.05466, 2026 on PithPith's reviewarXiv
  2. [1] Anthropic. Claude 3.7 Sonnet and Claude Code.https://www.anthropic.com/news/claude-3-7-sonnet, 2025. Accessed October 1, 2026 on Pith
  3. [2] Anthropic. Effective harnesses for long-running agents. https://www.anthropic.com/engineering/ effective-harnesses-for-long-running-agents, 2025. Accessed October 1, 2026 on Pith
  4. [3] Boyuan Deng, Shuyi Fan, Hongyang Zhang, and Xinhong Xie. Jev for scientific decisions: Evaluating semantic choices and their consequences.arXiv preprint arXiv:2609.24965, 2026 on PithPith's reviewarXiv
  5. [4] EverMind AI. Raven: The harness of harnesses for composable agentic intelligence.arXiv preprint arXiv:2609.33439, 2026. URLhttps://arxiv.org/abs/2609.33439 on PithPith's reviewarXiv
  6. [5] Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URLhttps://doi.org/10.5281/zenodo.20953922 on PithPith's reviewdoi:10.5281/zenodo.20953922
  7. [6] Yicheng He, Chengsong Huang, Zongxia Li, Jiaxin Huang, and Yonghui Yang. Visplay: Self-evolving vision- language models from images, 2025. URLhttps://arxiv.org/abs/2511.15661 on PitharXiv
  8. [7] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors,International Conferenc... on Pith
  9. [8] KRAFTON AI and Ludo Robotics. Terminus-kira: Boosting frontier model performance on terminal-bench with minimal harness, 2026. URLhttps://github.com/krafton-ai/kira on Pith
  10. [10] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses.https://arxiv.org/abs/2603.28052, 2026 on PithPith's reviewarXiv
  11. [12] Zongxia Li, Hongyang Du, Chengsong Huang, Xiyang Wu, Lantao Yu, Yicheng He, Jing Xie, Xiaomin Wu, Zhichao Liu, Jiarui Zhang, and Fuxiao Liu. Mm-zero: Self-evolving multi-model vision language models from zero data, 2026. URLhttps://arxiv.org/abs/2603.09206 on PitharXiv
  12. [13] Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, et al. Long-horizon-terminal-bench: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading.arXiv preprint arXiv:2607.08964, 2026 on PithPith's reviewarXiv
  13. [14] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026 on PithPith's reviewarXiv
  14. [15] Haozhe Liu, Tian Ye, Sensen Gao, Qihang Cao, Yitong Li, Mingchen Zhuge, Duomin Wang, Ruihua Zhang, Ping Luo, Jiawang Bian, et al. Sol-pi: Recursively scaling auto-research loops for efficient agent harness.arXiv preprint arXiv:2609.20519, 2026. 14 on PithPith's reviewarXiv
  15. [16] Xinghua Lou, Miguel Lázaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P Murphy. Autoharness: improving llm agents by automatically synthesizing a code harness.arXiv preprint arXiv:2603.03329, 2026 on PitharXiv
  16. [17] Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Skill0: In-context agentic reinforcement learning for skill internalization.arXiv preprint arXiv:2604.02268, 2026 on PithPith's reviewarXiv
  17. [18] Ryan Marten. Terminal-Bench 4.0. Terminal-Bench release announcement, 2026. URLhttps://www.tbench.ai/ news/terminal-bench-4-0. Accessed October 1, 2026 on Pith
  18. [19] Mike A Merrill, Alexander Glenn Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Jenia Jitsev, Marianna Nezhurina, Di Lu, Orfeas Menis Mast... on Pith
  19. [20] Atsuyuki Miyai, Kiyoharu Aizawa, and Toshihiko Yamasaki. Task-coevolve: Efficient harness optimization via adaptive validation task selection.arXiv preprint arXiv:2608.20169, 2026 on PithPith's reviewarXiv
  20. [21] Xuying Ning, Dongqi Fu, Tianxin Wei, Hanqing Zeng, Yuanchen Bei, Bingxuan Li, Zihao Li, Qifan Wang, Xiang Shen, Yifan Wu, et al. Evoharness-rl: Learning self-evolving runtime harness for long-horizon llm agents.arXiv preprint arXiv:2608.05446, 2026 on PithPith's reviewarXiv
show 20 more
  1. [22] OpenAI. Codex.https://github.com/openai/codex, 2025. OpenAI coding agent / CLI on Pith
  2. [23] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024 on PithPith's reviewarXiv
  3. [24] Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, et al. Autosaddler: Automatic harness optimization with durable updates from agent execution traces.arXiv preprint arXiv:2608.23041, 2026 on PithPith's reviewarXiv
  4. [25] Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, and Shelby Heinecke. Ai4ai at test-time: Strong-to-weak capability transfer via harnesses.arXiv preprint arXiv:2608.12307, 2026 on PithPith's reviewarXiv
  5. [26] Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, and Kai Wang. StateM: Reaching 95.3% raw accuracy, or a $15 frontier run, on Terminal-Bench 2.1 via harness scaling, 2026. URLhttps://arxiv.org/abs/2608.15089 on PithPith's reviewarXiv
  6. [27] Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, August 2026. URLhttps://qwen.ai/blog?id= qwen3.8 on Pith
  7. [28] Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, and Zhengzhou Zhu. Pats: Policy-aware training scaffolding for agentic reinforcement learning.arXiv preprint arXiv:2607.21419, 2026 on PithPith's reviewarXiv
  8. [29] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023 on Pith
  9. [30] Kishanthan Thangarajah, Boyuan Chen, and Ahmed E Hassan. Dcas: Decoupling cli agent scaffolding to internalize planning across scaffolds.arXiv preprint arXiv:2608.06113, 2026 on PithPith's reviewarXiv
  10. [31] Vivek Trivedy. Improving deep agents with harness engineering. LangChain engineering blog.https://www. langchain.com/blog/improving-deep-agents-with-harness-engineering , February 2026. Published February 17, 2026; accessed September 23, 2026. 15 on Pith
  11. [32] Naman Vats and Oleg Golev. The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation.arXiv preprint arXiv:2607.22585, 2026 on PithPith's reviewarXiv
  12. [33] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. InCOLM 2026 The 2nd Workshop on Lifelong Agents: Learning, Aligning, an... on Pith
  13. [34] Siwei Wu, Jincheng Ren, Yizhi Li, Haau-Sing Li, Chengran Yang, Yuxuan Zhang, Weicheng Gu, Jian Yang, Riza Batista-Navarro, Chuanyi Zhang, et al. Modularrsi: Modular and generalizable recursive harness self-improvement. arXiv preprint arXiv:2609.14857, 2026 on PithPith's reviewarXiv
  14. [35] Shuhan Xue, Jianyuan Zhong, Ziyuan Nan, Wenbin Li, Zhaochen Yu, Jinchao Ding, Qiang Gao, Pengyu Zhan, Yuntong Zhang, Tian Cheng, et al. Sciencebuddy: Recursive-in-recursive self-improvement for interactive scientific agents.arXiv preprint arXiv:2609.17523, 2026 on PithPith's reviewarXiv
  15. [36] Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, et al. Harness-bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922, 2026 on PithPith's reviewarXiv
  16. [37] Haoran Ye, Yuxing Lu, Haonan Dong, Zhaochen Su, and Guojie Song. Harness-zero: Harness distillation via agent-as-harness, 2026. URLhttps://arxiv.org/abs/2609.24974 on PithPith's reviewarXiv
  17. [38] Zhou Yu, Bin Bi, Shiva Kumar Pentyala, Shubham Mehrotra, Sougata Chaudhuri, Shilpa Bhagavath, Zeyuan Chen, Ran Xu, Phil Mui, James Zhu, et al. Co-evolving harnesses and models: On-policy correction helps weaker models catch up where imitation fails.arXiv preprint arXiv:2609.09... on PithPith's reviewarXiv
  18. [39] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026 on PithPith's reviewarXiv
  19. [40] A Zhao, D Huang, Q Xu, M Lin, YJ Liu, and G Huang. Expel: Llm agents are experiential learners. arxiv, 2023 on Pith
  20. [41] Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, et al. Dream-rsi: Recursive self-improvement through evolving worlds.arXiv preprint arXiv:2609.14858, 2026. 16 on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment