Artificial Intelligence · arXiv:2610.27435 v1 · submitted 20 Oct 2026 · 40 references
Harness Self-Evolution for Long-Horizon Terminal Agents
The paper says
“Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution.”
more of the abstract
We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid, transparent harness evolution paper with a load-bearing but acknowledged transfer reporting issue that needs fixing in revision. This line is wrong
1 major objection · 8 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 40 references say.
The paper says, at Table 1, Appendix C.3 (p. 35), Appendix D.2 (Table 4)
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The headline transfer result for DeepSeek-V4-Flash on TB2.1 (Table 1: +7.30% Pass@1, +8.99% Pass4) is obtained by selecting the version that performed best on TB2.1 itself (+OO), not the final version from the TBH evolution (+IR). As stated in Appendix C.3, 'TB2.1 does not inform harness modifications; selection by its observed score is nevertheless part of this reporting protocol.'
more of objection 1
This is non-standard: a held-out benchmark is used to choose which version to present, which introduces over-optimism. The paper's own Appendix D.2 (Table 4) shows that the direct transfer of the final +IR version yields smaller gains: +3.09% Pass@1 and +6.74% Pass4 versus the seed. The difference is substantial. The central claim that 'frozen harnesses retain gains on a benchmark not used for evolution' (b244) is therefore weaker than the headline suggests. The authors should (a) present the direct-transfer (final version) results as the primary evidence for RQ4 in the main table, with the best-observed version as a supplementary note; (b) clearly state the selection protocol in all figure/table captions, including the risk of over-optimism; and (c) discuss what the gap between the two numbers implies about the robustness of transfer.
8 minor objections
The paper says, at Section 4.1 (Metrics)
Pith says: minor objection 1 of 8
The definition of Flip = Pass@r − Pass_r is standard but Figure 2 shows a stacked bar chart with '1–3/4' tasks; these are exactly the flipping tasks. Naming the Flip metric in the figure or caption would help readability.
The paper says, at Figures 2 and 3
Pith says: minor objection 2 of 8
The y-axis ranges differ between subplots (e.g., Figure 2 TBH vs TB2.1). For ease of comparison across benchmarks, consider using common y-axis ranges or explicitly justifying the different scales.
show 6 more
The paper says, at Table 1 (GPT-6-sol row)
Pith says: minor objection 3 of 8
The coral annotations for RiseHarness show absolute changes from Terminus-2. For consistency, the same annotation style could be applied to the ModularRSI and AHE Harness rows to let readers quickly see their deltas from the seed.
The paper says, at Appendix A.4
Pith says: minor objection 4 of 8
The case studies are thorough but very long. A one-paragraph summary in the main text (e.g., after Section 4.4) illustrating one concrete problem-to-change path would help non-specialist readers grasp the operational flavor of the method.
The paper says, at General
Pith says: minor objection 5 of 8
Several sentences are long and packed with parenthetical clauses, e.g., the first sentence of Section 3.3. Consider splitting to improve readability.
The paper says, at Appendix C.3
Pith says: minor objection 6 of 8
The phrase 'selection by its observed score is nevertheless part of this reporting protocol' is honest but reads as defensive. Suggest rewording to explicitly state that the best-observed version is reported for completeness but that the direct transfer of the frozen version is the primary evidence for generalization.
The paper says, at Appendix D.2
Pith says: minor objection 7 of 8
Table 4 presents cumulative results for TBH and TB2.1. The version labels (Base, +SE, +DI, +OO, +IR) are clear, but the TB2.1 row for +IR shows a drop in Pass@1 and Pass4 relative to +OO. This is key data; consider calling it out in the main text when discussing version ranking differences.
The paper says, at Appendix C.2 (Eq. 4)
Pith says: minor objection 8 of 8
The notation bp_confirmation_B(D) is used in Algorithm 1 but its full definition (Eq. 4 with historical vs fresh baseline) appears only in the appendix. Consider moving or summarizing this in the main text (Section 3.5) to clarify the adoption rule.
What would settle it
Recompute all TB2.1 results in the main table (Table 1) using only the final harness version selected on TBH (e.g., +IR for DeepSeek-V4-Flash) for both models, without any version selection on TB2.1. Compare the resulting Pass@1 and Pass4 gains to the seed harness. If any gain falls below 3% or becomes non-positive, the central claim of robust transfer should be explicitly qualified.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 21 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 1 major objection, 8 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
RiseHarness is well-engineered work. The core idea—iteratively improving a fixed-model agent harness by contrasting successful and failed runs, mapping behavioral differences to code via a Handbook, and looking up external design patterns—is sensible and fairly implemented. The empirical base is honest: two models, ablations, case studies, and a clear separation between the evolution benchmark (TBH) and the transfer benchmark (TB2.1). The improvements in Pass@1 and Pass4 on TBH are real and replicable from the data shown. The authors also do a good job documenting their own limitations (Appendix E), including the key one that repeated development-set selection can exploit noise.
Now the soft spot that matters most. The headline transfer numbers for DeepSeek-V4-Flash on TB2.1 (+7.30% Pass@1, +8.99% Pass4) come from the version that performed best on TB2.1 itself, not from the final version produced by the TBH evolution. The paper's own Table 4 shows that the final version yields +3.09% and +6.74%—still positive but roughly half the headline gain. Appendix C.3 acknowledges the selection protocol. That is transparent but does not change the fact that the central transfer claim is materially weaker than the abstract and Table 1 suggest. This is not a fatal flaw because the paper reports both numbers, but the primary presentation conflates what should be two separate claims: (1) harness evolution improves the target benchmark, (2) the final frozen harness retains moderate gains on an unseen benchmark.
Who is this for? People building or studying agent harnesses for long-horizon tasks. The framework is practical, the method is replicable, and the ablation data is useful. The paper deserves refereeing.
Recommendation: Accept after revision. Require that the abstract and Table 1 present the final-version transfer as the primary result, with the best-observed version moved to a clearly marked supplementary note. This aligns the headline with the actual strength of the evidence.
The strongest objection
The paper's headline transfer result (Table 1, DeepSeek-V4-Flash on TB2.1: +7.30% Pass@1, +8.99% Pass4) is obtained by selecting the version that performed best on TB2.1 itself (+OO), not the final version from the TBH evolution (+IR). Appendix C.3 (p. 35) states: 'TB2.1 does not inform harness modifications; selection by its observed score is nevertheless part of this reporting protocol.' This is non-standard: a held-out benchmark is used to choose which version to present, which introduces over-optimism. The paper's own data (Table 4, p. 36) shows that the final +IR version yields smaller gains on TB2.1: +3.09% Pass@1, +6.74% Pass4 (vs. seed). The difference between +7.30% and +3.09% is substantial. The central claim that 'frozen harnesses retain gains on a benchmark not used for evolution' [b244] is therefore weaker than the headline suggests. The reader flagged this risk; the paper's acknowledgement does not resolve it. This is the most load-bearing concern because the claim of generalization to unseen tasks is a key part of the contribution (RQ4). If the primary evidence for transfer is derived from version selection on the test set, the conclusion that harness improvements transfer robustly is not fully supported by the reported data.
The reader's verdict (CONDITIONAL, high confidence) identifies the best-version selection as a key concern. My stress-test confirms this is the most load-bearing unresolved issue. The paper acknowledges the practice but still presents the best-performance version as the primary result, which inflates the apparent transfer benefit. The paper's own supplementary data shows that using the final version reduces gains substantially. This does not invalidate the framework's overall value, but it weakens the specific claim of cross-benchmark transfer without overfitting. The reader's condition ('using a fixed final version for all transfer evaluations') directly addresses this. Therefore the verdict remains CONDITIONAL as the reader stated. I agree with the reader's weakest_assumption: the version selection introduces a risk of over-optimism. Other potential concerns (behavioral checks not run, causality of patterns) are either acknowledged by the paper or less directly load-bearing for the central claim of empirical improvement. The concrete test would settle whether the transfer conclusion is robust to proper out-of-sample evaluation.
The referee's report
If the results hold, RiseHarness represents a substantial step in automated harness self-evolution for LLM-based agents. The framework is one of the first to systematically combine dual-trace behavioral analysis (agent trajectory + internal harness trace), behavioral mapping via a Handbook, and external design reuse into an iterative improvement loop. The experimental design includes two models, ablations, transfer tests, and model-swap checks, and the paper is transparent about limitations. The main strengths are the evidence-driven diagnosis and the demonstration that harness improvements can transfer to tasks not seen during evolution, even with a fixed model. The approach does not require ground-truth answers or model fine-tuning, making it complementary to training-based methods.
Pith's reading of the paper (a machine's summary, not the authors')
This paper introduces RiseHarness, a framework that systematically improves the harness—the runtime layer controlling command execution, feedback, context management, and completion decisions—for long-horizon terminal agents without updating model weights. The core idea is to focus on 'flipping tasks' (those with both successful and failed runs under the same configuration), compare the paired trajectories, combine agent actions with internal harness traces from a runtime monitor, and use a Harness Handbook to connect execution evidence to specific harness behaviors and code. RiseHarness then searches other agent frameworks for relevant designs, adapts them to the current model and harness structure, validates changes through behavioral checks and repeated task evaluations, and iteratively retains measured improvements. The paper reports that on Terminal-Bench Hard this method raises Pass@1 by up to 7.25% and Pass4 (all four runs successful) by up to 14% for DeepSeek-V4-Flash, and that gains transfer to a held-out benchmark (TB2.1) with the same model and harness frozen. These results suggest that careful, evidence-driven design of the interaction layer can make existing model capabilities more reliable, complementing model-level improvements.
What carries the argument. The key machinery is the combination of (a) paired trajectory contrast on flipping tasks (same task, same configuration, differing outcomes); (b) a runtime monitor that captures harness-internal execution traces aligned with agent actions; (c) a Harness Handbook that links behavioral responsibilities to implementing code; and (d) an iterative search-and-adapt loop that retrieves relevant designs from other agent frameworks and tailors them to the current harness's structure, available signals, and model behavior. These components together enable evidence-driven diagnosis of harness weaknesses and targeted, validated modifications that improve execution reliability while keeping model weights fixed.
Core claim. The central claim is that iteratively improving the agent harness—guided by contrastive analysis of successful and failed runs, internal execution traces, and external design references—can significantly boost both average task success and repeated-run reliability for long-horizon terminal agents, even when the underlying model weights remain unchanged. The paper demonstrates this through the RiseHarness framework, which uses a runtime monitor to record harness-level execution (including function calls, arguments, and returns) alongside agent trajectories, and a Harness Handbook that maps these records to behavioral responsibilities and source code. By identifying recurring problems in shared interaction mechanisms (such as completion confirmation, observation feedback, and session recovery) and adapting proven designs from other frameworks, RiseHarness achieves measurable gains across multiple models and benchmarks. The authors treat the specific model as the unit of adaptation and show that evolved harnesses are model-specific: swapping harnesses between models reduces performance, while frozen harnesses retain gains on unseen benchmarks.
The premise it rests on. That the behavioral patterns identified by contrasting successful and failed dual traces are causally related to specific harness design flaws, and that the same harness modification will benefit other tasks sharing those patterns, rather than reflecting sampling noise, environment variation, or task difficulty.
If the paper is right (Pith's inferences, not the authors')
If harness evolution can consistently improve reliability without retraining, then deployment of LLM agents in production may benefit from a dedicated, model-specific harness optimization phase before or alongside model updates.
The approach suggests that many failure modes in long-horizon tasks are attributable to the interface between model and environment, not the model itself, opening a complementary line of attack to training-based methods.
Evidence-driven harness design could be extended to other agent frameworks (e.g., SWE-agent, OpenHands) and to different interaction modalities (e.g., GUI, API) by adapting the monitoring and handbook components.
The finding that different models benefit from different harness changes implies that a one-size-fits-all harness is suboptimal; future agent systems might ship with model-specific harness configurations derived through similar self-evolution processes.
The reliance on flipping tasks means that consistently failing or consistently succeeding tasks provide no signal; a complete harness evolution strategy may need to incorporate active exploration or synthetic failures to cover blind spots.
The method's dependence on the quality of the runtime monitor and the Harness Handbook suggests that in environments where instrumentation is costly or incomplete (e.g., legacy closed-source systems), the approach may yield diminishing returns. An inference is that deployability hinges on observability engineering.
The paper's model-specificity finding hints at a deeper principle: the optimal interaction protocol is a function of the model's stochastic generation patterns, not just the task. Harness self-evolution could be extended to meta-learn a policy for adapting harnesses on the fly per model instance.
Because the evolution loop uses the same model to propose and evaluate changes, there is a risk of overfitting to the model's quirks. The authors acknowledge benchmark overfitting controls but do not test on models unseen during evolution; future work could assess whether a harness evolved for one model transfers to a sibling checkpoint.
The reported gains on TB2.1, while positive, are smaller than on TBH and the best version differs between benchmarks. This suggests that version selection is task-distribution-sensitive; a practical harness service might need to maintain multiple candidate harnesses and select per task cluster.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-21
The authors' main claim, as the classifier picked it out 0.85
“RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.”
How the main claim is stated: asserted 0.98
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.32 · human participants 0.03 · controlled comparison 0.29 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.48 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. | background 0.99 | 0.19 | 0.04 | hedged possibility 1.00 |
| Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. | background 0.98 | 0.10 | 0.04 | asserted 0.78 |
| We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. | contribution 0.96 | 0.89 | 0.09 | asserted 1.00 |
| It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. | method 0.94 | 0.74 | 0.06 | asserted 0.99 |
| RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. | method 0.99 | 0.85 | 0.04 | asserted 0.99 |
| A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. | method 0.93 | 0.81 | 0.04 | asserted 0.98 |
| RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. | method 0.99 | 0.84 | 0.03 | asserted 1.00 |
| Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. | method 1.00 | 0.89 | 0.08 | asserted 0.99 |
| This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. | method 0.99 | 0.82 | 0.10 | asserted 1.00 |
| We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. | method 0.52 | 0.95 | 0.12 | asserted 0.96 |
| RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks. | contribution 0.75 | 0.70 | 0.28 | asserted 0.98 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 8,078 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Showing 5 of the paper's 7 figures. Figures are the authors', reproduced from the paper.
References (40)
- [1] Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Cheng- song Huang, Ergan Shang, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, and Leowei Liang. Long-Horizon- Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Ta... on PithPith's reviewarXiv
- [2] Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, and Leowei Liang. Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Constru... on PithPith's reviewarXiv
- [3] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045 , 2024. URL https://arxiv.org/abs/2406. 12045 on PithPith's reviewarXiv
- [4] Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Towards a science of AI agent reliability. arXiv preprint arXiv:2602.16666 , 2026. URL https://arxiv.org/abs/2602. 16666 on PithPith's reviewarXiv
- [5] Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, and Leowei Liang. T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks. arXiv preprint arXiv:2609.11042 , 2026. URL https: //arxiv.org/abs/2609.11042 on PithPith's reviewarXiv
- [6] Zhongzhi Li, Yucheng Shi, Zongxia Li, Ruhan Wang, Anhao Li, Zixun Huang, Junyao Yang, Lei Ke, Ning- hao Liu, Haitao Mi, and Leowei Liang. Recursive Synthesis for Long-Horizon Terminal Tasks. arXiv preprint arXiv:2608.05466, 2026. URL https://arxiv.org/abs/2608.05466 on PithPith's reviewarXiv
- [8] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052 , 2026. URL https://arxiv.org/ abs/2603.28052 on PithPith's reviewarXiv
- [9] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhen- hua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850 , 20... on PithPith's reviewarXiv
- [10] Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, and Qing Wang. From failed trajectories to reliable LLM agents: Diagnosing and repairing harness flaws. arXiv preprint arXiv:2606.06324 , 2026. URL https://arxiv.org/abs/2606.06324 on PithPith's reviewarXiv
- [11] Nikita Agarwal and Nivedit Jain. FIRE: Failure-informed runtime engineering for reliable language-model agents. arXiv preprint arXiv:2609.26048 , 2026. URL https://arxiv.org/abs/2609.26048 on PithPith's reviewarXiv
- [13] Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, and Leoweiliang. Harness handbook: Making evolving agent harnesses readable, navigable, and editable. arXiv preprint arXiv:2607.13285 , 2026. URL https://arxiv.org... on PithPith's reviewarXiv
- [14] Zihao Cheng, Zeming Liu, Yingyu Shan, Xinyi Wang, Xiangrong Zhu, Yunpu Ma, Hongru Wang, Yuhang Guo, Wei Lin, and Yunhong Wang. Mem 2Evolve: Towards self-evolving agents via co-evolutionary capability expansion and experience distillation. arXiv preprint arXiv:2604.10923 , 2026... on PithPith's reviewarXiv
- [15] Xiaotian Luo, Dizhan Xue, Fengxingyu Wang, Chuanrui Hu, and Yafeng Deng. HarnessBank: Semantic gene- bank search with gated verification for agent-harness self-evolution. arXiv preprint arXiv:2607.13683 , 2026. URL https://arxiv.org/abs/2607.13683 on PithPith's reviewarXiv
- [16] Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, and Wenxuan Zhang. HarnessDev: Can LLMs create and e... on PithPith's reviewarXiv
- [17] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022. URL https: //arxiv.org/abs/2210.03629 on PithPith's reviewarXiv
- [18] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.118... on PithPith's reviewarXiv
- [19] Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You. Where LLM agents fail and how they can learn f... on PitharXiv
- [20] Shraddha Barke, Arnav Goyal, Alind Khare, A valjot Singh, Suman Nath, and Chetan Bansal. AgentRx: Diagnosing AI agent failures from execution trajectories. arXiv preprint arXiv:2602.02475 , 2026. URL https://arxiv.org/abs/2602.02475 on PithPith's reviewarXiv
- [21] Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, and Graham Neubig. The OpenHands software agent SDK: A composable and extensible foundation for production agents. arXiv preprint arXi... on PithPith's reviewarXiv
- [22] Chengkai Huang, Junda Wu, Yu Xia, Zixu Yu, Ruhan Wang, Tong Yu, Ruiyi Zhang, Ryan A. Rossi, Branislav Kveton, Dongruo Zhou, Julian McAuley, and Lina Yao. Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models. arXiv preprint arXiv:2503.16734 , 2025.... on PithPith's reviewarXiv
show 20 more
- [23] Adam AlSayyad, Kelvin Yuxiang Huang, and Richik Pal. AgentTrace: A structured logging framework for agent system observability. arXiv preprint arXiv:2602.10133 , 2026. URL https://arxiv.org/abs/2602.10133 on PitharXiv
- [24] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christo- pher Potts. DSPy: Compiling declarative language model calls into self-imp... on PithPith's reviewarXiv
- [25] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflect... on PithPith's reviewarXiv
- [26] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, 13 Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762 , 2024. URL ... on PithPith's reviewarXiv
- [28] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel machine: Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954 , 2025. URL https://arxiv.org/abs/2505. 22954 on PithPith's reviewarXiv
- [29] Ruhan Wang, Zhiyong Wang, Chengkai Huang, Rui Wang, Tong Yu, Lina Yao, John C. S. Lui, and Dongruo Zhou. Federated In-Context Learning: Iterative Refinement for Improved Answer Quality. arXiv preprint arXiv:2506.07440, 2025. URL https://arxiv.org/abs/2506.07440 on PithPith's reviewarXiv
- [30] URL https://arxiv.org/abs/2408.08435 on PithPith's reviewarXiv
- [31] Zongxia Li, Yucheng Shi, Zhongzhi Li, Junyao Yang, Ruhan Wang, Chengsong Huang, Fuxiao Liu, Haitao Mi, Jordan Boyd-Graber, and LeoweiLiang. Scaling trajectories for complex tasks through recursive self-rewrite. arXiv preprint arXiv:2610.02826 , 2026. URL https://arxiv.org/abs/... on PithPith's reviewarXiv
- [32] Junyao Yang, Zongxia Li, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Ruiwen Zhou, Haitao Mi, Shuicheng Yan, and Leowei Liang. Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine. arXiv preprint arXiv:2610.22192 , 2026. URL https://arxiv... on PithPith's reviewarXiv
- [33] Ruhan Wang, Chengkai Huang, Zhiyong Wang, Junda Wu, Rui Wang, Tong Yu, Julian McAuley, Lina Yao, and Dongruo Zhou. FERA: Uncertainty-Aware Federated Reasoning for Large Language Models. arXiv preprint arXiv:2605.10082, 2026. URL https://arxiv.org/abs/2605.10082 on PithPith's reviewarXiv
- [34] Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang, Haofei Yang, Rui Yang, Yang Liu, Tao Lv, and Fangming Li. HarnessEvolve: Learning from reference trajectories for reliable agent self-evolution. arXiv preprint arXiv:2609.00829, 2026. URL https://arxiv.org/abs/2609.00829 on PithPith's reviewarXiv
- [35] Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. AgentSquare: Automatic LLM agent search in modular design space. arXiv preprint arXiv:2410.06153 , 2024. URL https://arxiv.org/abs/2410. 06153 on PithPith's reviewarXiv
- [36] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-Harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498 , 2026. URL https://arxiv. org/abs/2606.09498 on PithPith's reviewarXiv
- [37] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Noah A. Smith, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. arXiv preprint arXiv:2607.12227 , 2026. URL https://arxi... on PithPith's reviewarXiv
- [38] The Terminal-Bench Team. Terminal-Bench 2.1. Official benchmark release announcement, 2026. URL https: //www.tbench.ai/news/terminal-bench-2-1 . Accessed October 19, 2026 on Pith
- [39] Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhuang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, and Chen-Yu Lee. RRSI: Regularized recursive self-improvement of agent harnesses. arXiv preprint arXiv:2609.... on PithPith's reviewarXiv
- [40] Junyao Yang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Ruhan Wang, Xiangxin Zhou, Kishan Panaganti, Haitao Mi, and Leowei Liang. Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Rein- forcement Learning. arXiv preprint arXiv:2607.18722 , 2026. URL ht... on PithPith's reviewarXiv
- [42] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2309.14393 on PithPith's reviewarXiv
- [2024] URL https://arxiv.org/abs/2405.15793 on PithPith's reviewarXiv
- [2026] URL https://arxiv.org/abs/2609.14857 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment