REVIEW 4 major objections 4 minor 3 cited by
Recursive Synthesis for Long-Horizon Terminal Tasks
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Recursive verified synthesis produces 37,484 long-horizon terminal tasks at about $0.05 each; difficulty climbs until a fixed solver passes only 2.5%, and training on the tasks lifts agents on three terminal benchmarks.
desk verdict RST is a serious, reproducible recipe for cheap verified terminal-task synthesis; the headline RL numbers conflict, and the contract-validity audit is load-bearing but only LLM-checked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the recursive verified task bundle: each task carries a public instruction, runtime metadata, an environment definition, a reference solution script, and a private verifier. The synthesis loop works solution-first, extending solve.sh with additional operations, then aligning the environment, verifier, and instruction to that workflow, then validating the complete candidate in a fresh sandbox. Acceptance rests on two checks: oracle validity, meaning the reference solution passes the private verifier, and contract validity, meaning every verifier requirement is stated in the public instruction or discoverable from workspace files. A taxonomy of forty rewrite operators in five families (configuration and control state; data, manifest, and schema state; filesystem and resource binding; build, cache, and artifact state; runtime, tooling, and diagnostics) supplies the transformations that make later rounds longer, more state-dependent, and harder for a fixed solver.
What would settle it
Take a random sample of tasks from rounds ten and fifteen, hide the private verifier and reference solution, and have independent human annotators label every verifier assertion as stated in the public instruction, inferable from workspace files, or hidden; if more than a small fraction are hidden, the reported difficulty collapse is partly an artifact of under-specification.
Extended reading notes
Core claim
The central claim is that a solution-first, verifier-aligned recursion can generate a growing pool of executable terminal tasks that are genuinely harder and useful for training. Each rewrite begins by extending the reference solution with more executable work, then updates the environment, private verifier, and public instruction to match the new workflow, and accepts a candidate only if its reference solution passes the verifier in a fresh sandbox (oracle validity) and every verifier requirement is stated in the instruction or inferable from the workspace (contract validity). The paper reports that this loop stays stable for fifteen rounds: synthesis yield and candidate pass rates remain flat, structural difficulty metrics climb several-fold, and the same fixed solver's success drops from 90% to 2.5% while mean partial credit falls from 0.970 to 0.170. Training on rollouts from these tasks yields consistent gains on all three evaluated terminal benchmarks, with verifier-based PPO producing the largest relative improvement on a benchmark constructed independently of the training pool.
Load-bearing premise
The load-bearing premise is that the automatic consistency checks truly guarantee that nothing the verifier tests is hidden from the agent; if some verifier requirements are neither stated in the instruction nor discoverable from the workspace, the later-round tasks could look harder than they actually are.
Editorial extensions
If this is right
- The pipeline should keep producing harder tasks beyond round fifteen, since synthesis yield and validation rates stay flat while structural difficulty continues to climb.
- Each accepted task is directly usable as a verifier-based reinforcement learning environment, because it bundles an instruction, a runnable sandbox, and a dense verifier.
- Training on recursively synthesized trajectories transfers to independently constructed terminal benchmarks, with the largest relative gain on a benchmark built from a different data source.
- Long-horizon terminal training data becomes dramatically cheaper, dropping from the usual hundreds or thousands of dollars per task to roughly five cents per accepted task.
- Diversity caps on parent lineage, category, rewrite family, and cohort keep later rounds from collapsing onto a small set of ancestors or transformation patterns.
Reading between the lines
- If contract validity is genuinely enforced, the same recursion recipe could transfer to other agent domains such as web, desktop, or code editing, wherever instruction-environment-verifier agreement can be checked automatically; the paper itself demonstrates only terminal tasks.
- Because the difficulty curve is measured with a fixed solver from the same generation family, a stronger test would evaluate late-round tasks with solvers of a different family or with independent human checklists to separate 'harder' from 'less well-specified'.
- The reported absence of a ceiling suggests a natural curriculum loop in which each round's accepted tasks become practice for the next round's policy, a self-improving dynamic the paper does not fully explore.
- The rising nearest-neighbor similarity tail at round fifteen implies that continued scaling should include explicit deduplication, a limitation the authors themselves flag.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Recursive Synthetic Terminal Tasks (RST), a framework that starts from 639 verified terminal-agent tasks and recursively rewrites the reference solution, verifier, instruction, and environment over fifteen rounds, producing 37,484 accepted tasks at an estimated $0.05 per task. The authors report that later-round tasks are structurally larger and harder for fixed solvers, that DeepSeek-V4-Pro pass@4 drops from 90% at R1 to 2.5% at R15, and that supervised fine-tuning and agentic PPO on trajectories from these tasks improve Qwen3.5 models on Terminal-Bench 2, Terminal-Bench Hard, and Long-Horizon Terminal Bench. The paper also presents diversity, duplication, leakage, and contamination analyses, a case study of one task lineage, and detailed implementation appendices.
Significance. If the central claims hold, RST would be a significant practical contribution: it provides a low-cost, scalable, sandbox-validated pipeline for generating long-horizon terminal-agent training tasks, with evidence that self-collected trajectories improve held-out benchmark performance. The paper is stronger than many synthetic-data papers in that it reports both SFT and PPO results, includes per-round diversity and contamination analyses, ships detailed prompt templates and operator taxonomies, and validates oracle solvability in fresh sandboxes. The main limitations are that the contract-validity guarantee is enforced by LLM-driven audits rather than by an independent check, that one of the three evaluation benchmarks is co-authored by the same group, and that a headline RL number is reported inconsistently in Table 4 and the Conclusion. These issues are load-bearing for the interpretation of the difficulty curves and the training-utility claim, but they are addressable with additional analysis and disclosure.
major comments (4)
- [Table 4 vs. Conclusion] Table 4 reports Qwen3.5-27B-RL as 49.44% on Terminal-Bench 2 with a relative gain of +20.00%, while the Conclusion reports 46.07% and +11.82% for the same benchmark. Since 46.07/41.20 - 1 = 11.82% and 49.44/41.20 - 1 = 20.0%, exactly one of these pairs is consistent; the abstract repeats the 49.44% value. This inconsistency directly affects the headline PPO result and must be corrected, with the authors specifying which value corresponds to the three-run average reported in Table 4.
- [Section 4.4 and Figures 10-11] The acceptance criterion in Section 4.4 and Appendix D.1 is that every requirement checked by the verifier must be stated in the public instruction or discoverable from the workspace. The audit metrics in Figures 10-11, however, show hidden-check protection at only 63.5% at R15, short-instruction risk at 7.5%, and median requirement coverage at 0.57. If these metrics describe tasks that passed the acceptance gate, then a substantial fraction of accepted tasks do not satisfy the stated contract-validity guarantee, and the observed collapse in DeepSeek-V4-Pro pass@4 from 90% to 2.5% could partly reflect broken, under-specified, or over-constrained tasks rather than genuinely increasing difficulty. This would also confound the interpretation of the PPO gains. Please clarify how Figures 10-11 relate to the acceptance gate and provide an independent stratified audit, such as human annotation or a second model not involved in generation, of verifier requirements across rounds.
- [Section 5.2 and reference [21]] Long-Horizon Terminal Bench (LHTB) is co-authored by several of the present paper's authors, and the paper does not disclose this overlap. In addition, the difficulty curves in Section 5.2 use DeepSeek-V4-Pro, the same model family used to generate tasks and reference solutions, as the fixed solver. These two facts create possible circularity channels: the LHTB improvement could reflect verifier or task-structure similarity with the training pool, and the difficulty decline could reflect generator-specific blind spots rather than general task hardness. The lexical contamination analysis in Table 2 addresses textual overlap but not structural or verifier-level overlap. Please disclose the author overlap, report full-round difficulty results for at least one independent solver (GPT-5.6-sol appears in Figure 1 but not in Section 5.2), and discuss whether the LHTB reward and verifier design is sufficiently different from the synthesized training tasks.
- [Section 5.5] The PPO training description states that reward is computed from each task's built-in verifier 'with customized reward shaping,' but the reward-shaping function is never specified. Without this information, the reported reward increase from 0.11 to 0.14 and the downstream benchmark gains cannot be reproduced or fully interpreted. Please provide the exact reward-shaping formula, the PPO hyperparameters beyond clipping and GAE, the number of rollouts per update, and the evaluation protocol used for Table 4.
minor comments (4)
- [Abstract and Section 6] The phrase 'no ceiling' is stronger than the evidence supports; stable yield through fifteen rounds is not the same as demonstrating an absence of saturation, and the paper itself notes a growing high-similarity tail that motivates future deduplication.
- [Title and captions] The title contains an apparent spacing artifact ('T erminal Tasks'), and Figure 1's caption includes unexplained shorthand ('TB2 Ep. Tmax Ep. LHTB Ep.') that should be expanded.
- [Section 5.1] The cost figure of approximately $0.05 per task should state which cost components are included (model inference, sandbox compute, repair retries, etc.) and whether the cost is per accepted task or per seed attempt.
- [Section 5.2] The difficulty analysis reports DeepSeek-V4-Pro pass@4 with matched subsets, but the matching procedure and subset sizes per round are not specified; reporting the number of tasks and confidence intervals would strengthen the comparison.
Circularity Check
No central circularity; one minor self-citation in an evaluation benchmark.
-
other
[Section 1 (Related Work, Long-Horizon Terminal Bench) and Reference [21] author list]
"Long-Horizon Terminal Bench evaluates persistent, multi-stage terminal workflows with many dependent interactions and dense partial-credit grading [21]."
Reference [21], the Long-Horizon Terminal Bench, is authored by Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Anhao Li, Ninghao Liu, Haitao Mi, and Leowei Liang, who are also authors of this paper. Using this same-group benchmark as one of the three headline evaluation targets means that this particular leg of the training-utility claim is not independently constructed relative to the present authors. This is a minor self-citation rather than a forced reduction, because Terminal-Bench 2 and Terminal-Bench Hard are externally constructed benchmarks and also show consistent gains, so the central claim does not rest solely on the self-authored benchmark.
full rationale
The paper's central derivation chain is not circular. Accepted tasks require oracle validity (reference solution passes the private verifier in a fresh sandbox) and contract validity (verifier requirements are stated or discoverable), and these are executable checks rather than fitted predictions. The training-utility claim is supported by gains on Terminal-Bench 2 and Terminal-Bench Hard, which are external benchmarks; the Long-Horizon Terminal Bench is co-authored by the same group and is a minor self-referential element, but it is not load-bearing because the other two benchmarks independently confirm the transfer. The difficulty claim is corroborated by two different solvers: DeepSeek-V4-Pro pass@4 falls from 90% to 2.5% and GPT-5.6-sol falls from 72.2% to 7%, so the decline is not an artifact of a single self-measuring model. Structural growth in solution lines, commands, and assertions is partly designed into the rewrite contract, which imposes minimum changed-line gates, but the reported pass-rate and partial-credit declines are empirical outcomes rather than consequences of those gates. No equation-level or definitional reduction of a prediction to its inputs was found, and no load-bearing argument reduces to a self-citation chain.
Assumptions & free parameters
free parameters (3)
- generation quality gate thresholds =
3 files, 8 solution lines, 12 verifier lines, 180-word instruction cap, 1.6x length ratio
- seed selection caps =
4 descendants per parent, 160 per category, 320 per family, 280 per cohort
- bounded repair rounds =
2
assumptions (5)
- domain assumption Oracle validity (reference solution passes verifier in a fresh sandbox) implies the task is solvable by a capable agent and is not biased toward the generating model.
- ad hoc to paper Automatic instruction-verifier consistency checks correctly detect hidden requirements, leakage, and weak grounding at scale.
- domain assumption Pass@4 decline for DeepSeek-V4-Pro and GPT-5.6-sol measures task difficulty rather than task breakage or instruction degradation.
- domain assumption The 13-token sliding-window and 5-gram Jaccard audit is sufficient to rule out benchmark leakage.
- domain assumption Terminal-Bench 2, TMax-derived Terminal-Bench Hard, and LHTB are valid external measures of terminal-agent capability.
Cite this review
Pith. "Pith review of Recursive Synthesis for Long-Horizon Terminal Tasks." pith.science (2026). https://pith.science/paper/YHEG6MS4
@misc{pith2026260805466,
author = {Pith},
title = {Pith review of: Recursive Synthesis for Long-Horizon Terminal Tasks},
year = {2026},
howpublished = {\url{https://pith.science/paper/YHEG6MS4}},
note = {Machine review of arXiv:2608.05466}
}
abstract
High-quality long-horizon training data for terminal agents is expensive to produce, often costing hundreds to thousands of dollars per task, because each task must keep the instruction, environment, reference solution, and verifier mutually consistent. Human authoring does not scale, and direct generation with large language models (LLMs) often breaks these dependencies. We present Recursive Synthetic Terminal Tasks (RST), a recursive verified synthesis framework for constructing long-horizon terminal-agent tasks at scale. Starting from verified seed tasks, RST extends the reference solution, realigns the verifier and instruction to the new workflow, validates the result in a fresh sandbox, and reuses accepted tasks as seeds for subsequent rounds. Across fifteen recursive rounds, RST produces 37,484 synthesized terminal-agent tasks at roughly $0.05 per task. Task difficulty increases substantially over rounds: the median reference solution grows from 67 to 374 lines, the median number of executed commands grows from 40 to 244, and DeepSeek-V4-Pro pass@4 drops from 90% at R1 to 2.5% at R15. To demonstrate training utility, we collect rejection-sampled Qwen3.5 trajectories on the synthesized tasks and use them for supervised fine-tuning. Fine-tuning on these trajectories improves Qwen3.5-27B and Qwen3.5-122B-A10B by up to 10 points on Terminal-Bench 2, Terminal-Bench Hard, and Long-Horizon Terminal Bench, while agentic PPO lifts Qwen3.5-27B to 49.44%, 32.00%, and 22.07% on the three benchmarks, corresponding to relative gains of 20.0%, 41.2%, and 21.9% over the base model. Moreover, after 15 rounds, the recursion shows no ceiling: synthesis yield and validation rates remain stable as difficulty keeps climbing, indicating that the process can continue well beyond the scale reported here.
Forward citations
Cited by 3 Pith papers
-
FrogNano: Training a 4B Coding Agent via Online Task Synthesis
FrogNano, a 4B coding agent trained via RL on adaptively synthesized tasks, achieves competitive SWE-bench results without distillation from larger models.
-
Learning Generalizable Behaviors for Terminal Agents
Reinforcement learning for terminal agents generalizes by composing existing skills through high-level behaviors, and filtering environments plus penalizing repetition improves RL gains while using 30% of the training data.
-
Terminal Agents: A Survey of AI Agents in Command-Line Environments
Terminal agents are defined by a command-execution action-observation loop, analyzed via seven competence dimensions, with diagnostics showing benchmark-dependent process exposure and attribution limits.
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.