Computation and Language · arXiv:2610.13166 v2 · submitted 9 Oct 2026
Scaling Agentic Data for Long-Horizon Terminal Intelligence
The paper says
“Terminal agents are moving beyond local operations toward completing real-world professional tasks.”
more of the abstract
Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 12 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Impressive data efficiency, but the load-bearing filtering and decontamination rely on an unvalidated AI reviewer. This line is wrong
2 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Appendices F and G; Section 2.2
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The trajectory filtering (Appendix F) and benchmark decontamination (Appendix G) rely entirely on a single AI reviewer (gpt-5.6-luna) with no reported human validation or inter-rater reliability assessment. The filtering criteria are qualitative, and the paper provides no evidence that the AI's decisions correlate with human expert judgments.
more of objection 1
Similarly, the decontamination claim of zero contamination rests wholly on the AI's semantic assessment. Since the quality of the training data and the absence of test-set leakage are load-bearing for the central claim (Table 1, Table 2), the lack of human verification weakens the evidence. A human-validated sample (e.g., 100 trajectory-filter decisions, 50 decontamination candidate pairs) would substantiate the reliability of these procedures; otherwise, the filtering and decontamination steps should be acknowledged as unvalidated automated checks.
The paper says, at Appendix C, Table 5
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The four RSI case studies are each conducted with a single run per model per case. While the authors present detailed qualitative analysis, a single run does not provide statistical evidence that the observed behavioral differences are reproducible or that TerminalHorizon-27B systematically outperforms the base model in RSI settings. The paper should either report multiple runs (with mean and variance) or clearly state that these are anecdotal illustrations and not controlled experiments.
4 minor objections
The paper says, at Section 3, Figure 3c
Pith says: minor objection 1 of 4
The line 'Figure 3c shows a median chain length of 25 turns' is clear, but the caption for Figure 3c does not define 'feedback chain'; consider adding a brief definition in the caption or a footnote.
The paper says, at Tables 1 and 2
Pith says: minor objection 2 of 4
The base scores for Qwen3.5-27B on Terminal-Bench 2.0 and 2.1 are reported with a dagger (†) in Table 1 but without in Table 2. Unify the notation or clarify which scores were provided by the official model card versus the authors' own evaluation.
show 2 more
The paper says, at Appendix E, Tables 6 and 7
Pith says: minor objection 3 of 4
The 27 behavioral patterns are well-defined, but the paper never specifies how they are automatically detected in trajectories for the analysis in Figure 3d. If detection is automated, the method should be described; if manual, the paper should note that Figure 3d is based on human annotation (and by whom).
The paper says, at AI Use Statement, Section 6
Pith says: minor objection 4 of 4
The AI Use Statement says 'We manually reviewed all AI-assisted content' but does not specify whether the trajectory filtering and decontamination outputs (Appendices F, G) were manually reviewed. Clarify this point to address the concern about human validation.
What would settle it
Randomly sample 200 trajectory-filtering decisions (100 KEEP, 100 DROP) and 100 decontamination candidate pairs (the top 50 by n-gram overlap and top 50 by TF-IDF similarity, excluding those already reviewed). Have three human experts independently apply the same criteria (Appendix F and Appendix G) to each sample, blind to the AI verdict. Measure inter-rater agreement (Fleiss' kappa) and human–AI agreement (Cohen's kappa). If human–AI agreement is below 0.7 for either task, the AI reviewer's reliability is insufficient to support the claim without further validation.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 12 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does something genuinely useful: it identifies a subtle failure mode in terminal-task synthesis—environments that inadvertently complete the very work the agent is supposed to learn—and builds a two-stage engine to detect and fix it. The core idea (challenge-guided reconstruction via independent execution) is well motivated, and the results across 12 benchmarks are strikingly consistent for a 3K-trajectory SFT run. The data-efficiency plot (Fig. 9) is the strongest piece of evidence in the paper.
The soft spot flagged by the stress test is real and load-bearing. Trajectory filtering (Appendix F) and benchmark decontamination (Appendix G) both use gpt-5.6-luna as the sole reviewer, with no human validation reported. The filtering criteria are sensible—validity, grounded progress, multi-turn value—but the AI decides whether a trajectory is kept, and the contamination verdict is entirely its output. If the reviewer systematically discards useful trajectories or misses near-duplicates, the claimed gains could be inflated. The paper does not ablate the filter or compare against even a small human-judged sample. Given that the whole empirical case rests on the quality of those ~3K trajectories, this is not a minor concern; it is the main uncertainty.
Other weaknesses are minor: the RSI case studies are single runs (interesting but not statistically meaningful), and the synthesis code is not released, which limits reproducibility of the data generation itself. The authors acknowledge most of these in the reproducibility statement, but the AI-reviewer reliance is not called out as a limitation.
Who this is for: anyone working on synthetic data for agent training, especially in terminal or tool-use settings. The method is a clear step beyond prior work on environment synthesis. The paper deserves a serious referee. A good review would ask for a human validation sample on the filtering and decontamination decisions, plus an ablation where the filter is removed or replaced by a simpler heuristic.
Bottom line: interesting approach, solid engineering, but the filtering stack needs external calibration before the results can be taken at face value.
The strongest objection
The paper's central claim—that SFT on ~3K long-horizon trajectories yields strong gains across 12 benchmarks—rests on the assumption that the training data is high-quality and free from test contamination. Both quality assurance steps (trajectory filtering and benchmark decontamination) are performed by a single AI reviewer (gpt-5.6-luna; see §F, §G) with no human validation reported. The filtering criteria in Appendix F are process-oriented and require grounding in the trajectory, but the reviewer's reliability for this task is unknown. Similarly, the decontamination procedure (Appendix G) uses gpt-5.6-luna to semantically classify candidate pairs as contaminated or clean; the paper reports zero contamination, but this conclusion depends entirely on the AI's judgment. If the reviewer systematically over-filters (discarding useful trajectories) or under-filters (retaining low-quality or contaminated trajectories), the reported data efficiency and benchmark gains could be artifacts. The paper acknowledges using GPT for automated filtering but does not ablate the filtering step or compare against human expert evaluation. A single missed contamination event could explain part of the improvement on any benchmark. This concern is load-bearing because if the data quality assurance is flawed, the entire empirical foundation is weakened.
The reader's weakest assumption already identifies the trajectory filtering criteria and AI-based reviewer as a key condition. This stress-test focuses on a more precise version: the same AI reviewer is used for both filtering and decontamination, and there is no human validation for either. This is the most load-bearing concern because it threatens the validity of all empirical results: if contamination exists or data quality is low, the reported gains may be spurious. The paper acknowledges the use of GPT for these tasks but does not provide a human baseline. The concrete test (human evaluation of a stratified sample) directly addresses whether the AI judgments can be trusted. The verdict remains CONDITIONAL because the results are plausible and the method is well-defined, but the condition on independent validation of these AI-based checks is now explicit. I agree with the reader that this is the weakest point, and I extend it to cover both filtering and decontamination as a unified concern.
The referee's report
If the reported gains hold, the paper makes a strong case that carefully constructed long-horizon trajectories provide a data-efficient route to improving terminal agents: ~3K trajectories outperform prior work using up to 10x more data on the same base model (Qwen3.5-27B). The core idea of challenge-guided reconstruction—using independent execution to identify where environment synthesis has inadvertently removed problem-solving work—is novel and could influence future data-generation pipelines. The public release of the data and model would facilitate reproducibility and follow-up research.
Pith's reading of the paper (a machine's summary, not the authors')
The paper introduces TerminalHorizon, a fully automated engine for synthesizing long-horizon terminal-agent trajectories that connect exploration, diagnosis, implementation, verification, and strategy revision into sustained problem-solving processes. By constructing 1,535 task environments and collecting 2,852 average-52.8-turn trajectories, then supervised fine-tuning Qwen3.5-27B, the authors obtain TerminalHorizon-27B, which outperforms the base model on all 12 benchmarks spanning coding (Terminal-Bench 2.0 +14.2%), real-world productivity (AutomationBench Pass Rate +11.7%, Partial +23.3%), and general reasoning (AIME 2025 +6.6%). The key claim is that long-horizon trajectories—when designed so that environments force agents to actually perform the intended analysis, implementation, and verification—carry much higher information density than shorter or less-structured data, enabling strong gains with far fewer examples.
What carries the argument. The central mechanism is a two-stage automated pipeline: (1) Challenge-Guided Long-Horizon Synthesis, in which an independent agent attempts each candidate task and a horizon-gap analysis identifies where the environment has already completed the intended work, guiding environment reconstruction that restores that work; (2) Progressive Horizon Expansion, which extends accepted tasks under the same goal by adding requirements (broader roles, deeper mechanisms, richer evidence, method upgrades) that build on earlier results and feedback, so agentic behavioral patterns continue across stages. Together, these stages ensure that the collected trajectories exhibit 27 defined agentic behavioral patterns (across five families: information exploration and judgment, diagnosis and error correction, construction and work progress, verification and assurance, strategy and execution coordination) connected over extended horizons.
Core claim. The core discovery is that supervised fine-tuning on approximately 3,000 long-horizon trajectories—averaging 52.8 interaction turns with sustained feedback chains—improves terminal agent performance across every evaluated benchmark, often with double-digit percentage-point gains, while also preserving and strengthening general reasoning. The authors attribute this to the fact that their challenge-guided synthesis and progressive expansion preserve the substantive problem-solving work (exploration, diagnosis, construction, verification, and strategy revision) that conventional synthesis methods often complete in advance, thereby providing richer training signal per trajectory.
The premise it rests on. The trajectory filtering criteria (Appendix F) and the AI-based reviewer (gpt-5.6-luna) reliably identify trajectories that provide useful training signal; if the filtering introduces systematic bias or fails to exclude low-quality trajectories, the reported gains may not hold for other model sizes or domains.
If the paper is right (Pith's inferences, not the authors')
Terminal agents trained on long-horizon, behaviorally rich trajectories may require orders of magnitude less data to achieve competitive performance, reducing the cost and complexity of agent training.
The approach's transfer to real-world productivity benchmarks (AutomationBench, VitaBench, τ²-bench) suggests that terminal-task training can generalize to enterprise automation and customer-service domains where sustained feedback-driven work is required.
The preservation and small improvement of general reasoning (MATH-500 +2.0%, AIME 2025 +6.6%) indicates that long-horizon interaction training need not come at the cost of pure reasoning capability, and may even strengthen it.
The RSI case studies (12-hour research budgets) demonstrate that the trained model can autonomously pursue iterative improvement, diagnose its own proxy-metric failures, and retain effective solutions in final deliverables—capabilities critical for long-horizon autonomous work.
The automated filtering criteria (Appendix F) and AI-based reviewer (gpt-5.6-luna) are not validated against human judgment; if the filtering systematically discards trajectories with informative but unsuccessful episodes, the reported data-efficiency may be partly an artifact of conservative curation rather than a property of long-horizon data alone.
The paper's focus on terminal environments (command-line interface) raises the question of whether similar long-horizon, behaviorally-rich synthesis can be applied to other agent interfaces (web, GUI, robotics) where exploration and feedback loops are equally critical.
The observation that longer feedback chains (median 25 turns) correlate with better task completion suggests that reward or preference optimization methods (RLHF/DPO) could further benefit from these trajectories, beyond the supervised fine-tuning demonstrated here.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Terminal agents are moving beyond local operations toward completing real-world professional tasks. Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-12
The authors' main claim, as the classifier picked it out 0.99
“We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.25 · human participants 0.10 · controlled comparison 0.19 · randomized assignment 0.01 · states quantitative result 0.97 · compares with prior results 0.35 · mentions code or data availability 0.06
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Terminal agents are moving beyond local operations toward completing real-world professional tasks. | background 1.00 | 0.06 | 0.03 | asserted 1.00 |
| Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. | gap 0.86 | 0.13 | 0.04 | asserted 1.00 |
| High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. | background 0.89 | 0.11 | 0.09 | asserted 1.00 |
| However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. | gap 0.71 | 0.08 | 0.03 | hedged possibility 1.00 |
| To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. | contribution 0.96 | 0.92 | 0.05 | asserted 1.00 |
| Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. | method 0.99 | 0.89 | 0.04 | asserted 1.00 |
| To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. | method 0.99 | 0.83 | 0.05 | asserted 1.00 |
| Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. | method 0.76 | 0.97 | 0.09 | asserted 1.00 |
| We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. | result 1.00 | 0.96 | 0.51 | asserted 1.00 |
| Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables. | result 0.94 | 0.94 | 0.18 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.92 · evidence from observation or experiment 0.93
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-12: 1 request, 7,667 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (15)
Showing 14 of the paper's 15 figures. Figures are the authors', reproduced from the paper.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment