Computation and Language · arXiv:2610.05039 v1 · submitted 4 Oct 2026 · 47 references
Causal Improvement Graph for Agentic Harness Optimization
The paper says
“Agentic Harness is the runtime that constructs task context and controls execution flow, thereby shaping overall agent performance.”
more of the abstract
Given a fixed model and external evaluation, automated Harness optimization seeks to improve this runtime through an iterative proposal–evaluation loop to better solve target tasks. Existing meta-harness methods mainly adopt proposer-centric discovery, in which an LLM-based proposer integrates accumulated experimental findings to determine subsequent Harness revisions. This places the burden of maintaining the evolving improvement state on the proposer as history expands and its underlying experimental logic becomes harder to discern. In this paper, we introduce the Causal Improvement Graph (CIG), a graph-governed meta-harness framework that externalizes the evolving improvement state in a persistent graph, allowing prior findings to directly govern subsequent Harness optimization through local proposer operations. CIG grows and links Evidence, Hypothesis, Intervention, and Outcome nodes to represent what was observed, how it may be explained, how to test that explanation, and what the evaluation reveals. Their structural relations preserve how the improvement state changes across iterations, allowing local proposers to build directly on relations among prior findings rather than recover them from raw history. Across various agent tasks, CIG discovers stronger Harnesses than previous meta-harness baselines and remains robust to the choice of task solver and proposer. Structural ablations further support the design of an explicit improvement state with graph-governed evolution.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 6 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuine structural contribution — graph-externalized improvement state with careful matched ablations — whose cleanest evidence sits on SWE-bench, while the headline AppWorld/TB2.0 gains are less controlled and one benchmark runs on a reconstructed partition. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 47 references say.
The paper says, at §4.1, Table 1, A.7
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The Full History control is not matched to CIG on the staged-call protocol or proposer token budget. Section 4.1 states matching in 'initialization, observable evidence, and candidate-evaluation budget,' but this omits the two factors that the skeptic's concern hinges on.
more of objection 1
The only token-matched evidence (A.7, Table 7) is a single representative configuration on the three classification tasks, reconstructed from a trajectory checkpoint, and it does not cover the agent benchmarks. Since AppWorld (+4.4 pp) and Terminal-Bench (+5.9 pp) are the two largest CIG-over-Full-History gaps in Table 1, the gains on those benchmarks cannot yet be attributed to the graph structure rather than to the staged calling protocol or token allocation. The matched structural ablation (Table 4/A.3) does fix these confounds, but only on SWE-bench Verified. Please extend a staged-call- and token-matched Full History control to AppWorld and Terminal-Bench 2.0, or restrict the corresponding headline claims.
The paper says, at §A.8, Table 1
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The Terminal-Bench 2.0 comparison is computed on a reconstructed partition. Appendix A.8 discloses that the partition is 'reconstructed following the reference protocol using the public split sampler with seed 20260522,' while the published HarnessFix row (26.5) in Table 1 is presumably on the canonical partition.
more of objection 2
Because Terminal-Bench shows the largest gain over the published HarnessFix baseline (+2.9 pp), the comparison is only valid if the reconstructed split coincides exactly with the canonical one. Please confirm partition equality (for example, by listing the exact task IDs or the sampler configuration used by HarnessFix) or rerun the comparison on the canonical split.
The paper says, at §A.8
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The manuscript states 'We will release the code publicly upon acceptance,' and the Full History baseline is the authors' own implementation with no shipped artifact. Given the non-canonical Terminal-Bench partition raised above and the unmatched controls on AppWorld and Terminal-Bench, the central Table 1 comparison cannot be independently verified from the manuscript as submitted. At revision, please include the code, the exact split-sampler configuration, and the Full History matching script as supplementary material.
4 minor objections
The paper says, at §A.6
Pith says: minor objection 1 of 4
Appendix A.6 states that portfolio selection is 'a proposer judgment under these constraints, not a deterministic maximization of a weighted score.' The reported 3- and 5-run means (Tables 1, 6) therefore contain an unquantified stochastic component from the selector. Reporting repetition of the selection step (for example, re-running PlanSelect on the same plan set) or a variance decomposition would strengthen the robustness claims.
The paper says, at §A.7
Pith says: minor objection 2 of 4
The token-budget checkpoint in Table 7 is 'reconstructed from the search trajectory.' Since only one checkpoint is reported, consider reporting the range over plausible checkpoints (for example, the earliest checkpoint at or above the 0.530M-token mark) to rule out ex-post checkpoint selection.
show 2 more
The paper says, at §3.2
Pith says: minor objection 3 of 4
The term 'causal' is used operationally and the paper explicitly excludes formal causal identification, which is appropriate; a one-line clarification in the abstract or introduction would reduce the risk of misreading, since the headline framing invites a stronger interpretation.
The paper says, at Figure 3
Pith says: minor objection 4 of 4
The Figure 3 insets showing node counts before and after graph updates are too small to read in print; consider enlarging them or moving the node-count transitions into a table or caption.
What would settle it
Run the M0-style staged Full History control (Appendix A.3) and a token-matched CIG on AppWorld and Terminal-Bench 2.0 under the HarnessFix protocol of Table 1 (GPT-5 mini, identical partitions, identical 8-candidate budget, same TB2.0 seed 20260522), and report validation-selected test means with run SDs. If CIG's gains over the staged Full History drop below roughly 2 pp on these benchmarks, the headline advantage is not attributable to graph-state structure and the per-benchmark claim should be restricted to SWE-bench; if the +4 to +6 pp gaps persist at matched tokens and staged calls, the concern is resolved. Additionally, re-run the TB2.0 published-baseline comparison using the canonical Terminal-Bench sampler seed if recoverable from HarnessFix's released protocol, to confirm that the +2.9 pp over the published 26.5 is not a partition artifact.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 6 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The one thing to know: CIG deserves a serious look. The core idea — externalize the improvement state in a persistent graph of Evidence/Hypothesis/Intervention/Outcome nodes and run local, role-scoped proposers against it instead of one proposer re-reading raw history — is a real structural step beyond the proposer-centric methods it cites.
The paper earns most of its claims. The strongest part is Table 4's structural ablation, where the authors fix the staged-call protocol, round schedule, and portfolio size across M0–M4 and run honest side branches: B1 shuffles relations to test semantics against edge count, B2 separates persistence from revision, B3 removes cross-record edges while keeping revision. Five runs with standard deviations in A.4. M4 beats M0 by 5.6 points test on SWE-V with both proposers, each cumulative component adding a point or two. That is real attribution evidence, not a pile of single-score comparisons. The robustness tables across five solvers and five proposers are consistent, and the A.9 case study distinguishes a failed joint target from a locally supported prediction — care this subfield doesn't always get.
Now the soft spots, in proportion. The stress-test concern half-lands. The matched staged-call and token control exists only on SWE-bench Verified. On AppWorld and Terminal-Bench 2.0, the two largest Table 1 gains, Full History is matched on initialization, evidence, and candidate budget but not on staged-call structure or proposer tokens. The only token-matched evidence (A.7) is one checkpoint reconstructed from a trajectory on classification tasks, not the agent benchmarks. And TB2.0 runs on a reconstructed partition — disclosed, fair, but it means the +5.9 pp there cannot be checked against the published HarnessFix 26.5. So the per-benchmark generality claim outside SWE-V is under-supported. I would stop short of calling it load-bearing: the SWE-V matched ablation carries the structural claim, and the macro picture holds. But the biggest headline numbers come from the least controlled settings, and that deserves fixing.
On the plus side, the paper is honest. The "causal" term is operational and stated as such; failed joint targets do not get spun as refuted hypotheses. The real blocker is that code ships only on acceptance, so the matching and the reconstructed partition cannot be verified now. My call: send it out. A referee should require code and data release, a token-matched control on at least one agent benchmark, and either a canonical TB2.0 partition or the comparison re-run and downgraded. The method is serious; the verification is what is missing.
The strongest objection
The paper's matched structural ablation (Table 4, Appendix A.3) fixes the staged-call protocol, round schedule, and portfolio size across M0–M4 and shows +1.0 to +1.8 pp per cumulative component and +5.6 pp overall for full CIG over the staged Full History on SWE-bench Verified (proposers A/B, Qwen3.7-Flash). That control partially resolves the staged-call and token confound, but only on SWE-V. The headline Table 1 shows AppWorld (+4.4 pp over Full History: 44.4 vs 40.0) and TB2.0 (+5.9 pp: 29.4 vs 23.5) as the two largest CIG gaps, and for neither is any staged-call- or token-matched Full History control reported; Section 4.1 states Full History is matched in 'initialization, observable evidence, and candidate-evaluation budget,' omitting staged-call structure and proposer tokens. The only matched-token evidence (A.7, Table 7) is a single configuration on the three classification tasks, reconstructed from a trajectory checkpoint, and does not cover the agent benchmarks. Independently, Appendix A.8 discloses that the Terminal-Bench partition is 'reconstructed following the reference protocol using the public split sampler with seed 20260522,' so the largest relative gain in Table 1 is computed on a non-canonical partition that cannot be verified against the published HarnessFix 26.5. Because the central claim is precisely that CIG's per-benchmark advantage over proposer-centric discovery reflects the graph-governed state, the uncontrolled computation and reconstructed partition on AppWorld and TB2.0 are load-bearing. If the gaps shrink to within noise under a staged, token-matched control, the per-benchmark claim is unsupported outside SWE-V-class settings, and only the macro-level direction would remain.
The reader identified the matching of the Full History control and the missing code artifact as the weakest assumption; I agree and sharpen it. The paper supplies real independent support: Table 4 with Appendix A.3 fixes the staged-call protocol, round schedule, and portfolio size across M0–M4, showing consistent per-component effects and +5.6 pp overall on SWE-bench, with B1/B2/B3 directional contrasts supporting relation semantics, revision, and cross-record relations; A.7 gives a matched-proposer-token comparison ahead at equal budget on three classification tasks. Those controls are the right idea. The unresolved residue is that neither control extends to AppWorld or Terminal-Bench 2.0, the two benchmarks with the largest Table 1 gaps over Full History (+4.4 and +5.9 pp), where the stated matching covers initialization, observable evidence, and candidate-evaluation budget but not staged-call structure or proposer tokens. Appendix A.8 additionally discloses that the TB2.0 partition is reconstructed with a self-chosen seed, so the largest relative gain rests on a non-canonical partition. These are not internal contradictions, and the paper is transparent about them, but they are the least-secure supports of the central claim. The proposed test — a staged, token-matched Full History on AppWorld and TB2.0 plus a seed check on the TB2.0 published comparison — would settle the attribution. My read does not change the verdict: CONDITIONAL remains appropriate, with release of code, quantification of run variance on the matched controls, and these two controls as the conditions to satisfy.
The referee's report
If the headline results hold, CIG is a genuinely useful step: it converts the formatless history of proposer-centric Harness search into an explicit, revisable improvement state that local proposers can build on, and the structural-ablation design is a real strength. The M0-M4 cumulative construction with the B1 (shuffled relations), B2 (append-only), and B3 (no cross-record relations) alternatives isolates representation, relation semantics, persistence-with-revision, and selection under a single matched workflow; the shuffle control in particular is a clean test of the claim that relations carry meaning, and the run-to-run statistics in Table 6 help assess effect size. The SWE-bench Verified ablation provides the strongest evidence for the central structural claim because it fixes the staged-call protocol, round schedule, and portfolio size. However, the headline Table 1 comparisons on AppWorld and Terminal-Bench — exactly the two largest CIG gaps over Full History — rest on a control not matched on staged-call structure or proposer tokens, and the Terminal-Bench partition is a seed-20260522 reconstruction rather than the canonical split used for the published 26.5 baseline.
Pith's reading of the paper (a machine's summary, not the authors')
Automated Harness optimization searches over the runtime that turns a fixed model into task behavior, using an iterative propose-and-evaluate loop. The paper claims this search is hampered because a single LLM proposer must reconstruct the evolving experimental state from raw history, and that the fix is to store that state explicitly as a growing causal graph. The Causal Improvement Graph (CIG) links Evidence, Hypothesis, Intervention, and Outcome nodes, keeps revisable assessments of each hypothesis, and invokes proposers only for locally scoped operations on that graph. On SWE-bench Verified, AppWorld, and Terminal-Bench 2.0, CIG reaches a macro average of 44.6 percent versus 42.3 for the strongest published baseline and outperforms a matched graph-free control across five solvers and five proposers. The reusable claim is that a persistent, revisable record of experimental reasoning, rather than the raw history itself, is the asset that should guide further optimization.
What carries the argument. The central object is the Causal Improvement Graph, a persistent typed graph whose four node types are Evidence (an observed behavior with provenance), Hypothesis (a falsifiable explanation with a revisable assessment), Intervention (an executable Harness change testing that hypothesis), and Outcome (the externally measured result and verdict). Its relations record support, contradiction, and Harness lineage, forming the basic causal path Evidence → Hypothesis → Intervention → Outcome. The graph is the persistent improvement state: externally evaluated outcomes revise hypothesis assessments, and subsequent Observe, Plan, PlanSelect, Implement, and Update operations read and extend the graph, with proposers invoked only for locally scoped operations that never own the full state.
Core claim. The paper's central claim is that graph-governed discovery, in which a persistent improvement graph of linked Evidence, Hypothesis, Intervention, and Outcome nodes directly guides subsequent Harness optimization, finds stronger Harnesses than proposer-centric discovery, in which a single LLM proposer reconstructs the improvement state from raw history. The graph is updated by externally evaluated outcomes that revise hypothesis assessments, so later proposals build on updated judgments and explicit experimental dependencies instead of re-deriving them. The empirical claim is that CIG beats the published HarnessFix baseline and a Full History control on three agent benchmarks, and that the advantage holds across five task solvers and five proposer models. Structural ablations attribute the gains to meaningful relations (shuffled relations lose performance), revision-aware persistence (append-only records lose performance), and graph-guided plan selection (removing cross-record relations loses performance), supporting the design principle that both accumulated experience and its evolving interpretation should be maintained.
The premise it rests on. The central comparison rests on whether the Full History control is genuinely matched to CIG in initialization, observable evidence, and evaluation budget, since the paper does not ship code and the plan-selector is a proposer judgment rather than a deterministic rule, leaving portfolio-choice variance unquantified.
If the paper is right (Pith's inferences, not the authors')
Graph-governed discovery improves on proposer-centric search across repository repair, application interaction, and terminal tasks, with a macro average of 44.6 percent versus 42.3 for HarnessFix and 40.1 for the Full History control.
The benefit appears robust to the task solver: CIG (Wide) outperforms Full History on all three classification tasks across all five solvers tested.
The benefit appears robust to the proposer: all five proposer models show positive gains over Full History on SWE-bench Verified, ranging from +2.4 to +8.8 percentage points.
Broader candidate portfolios beat deeper ones in the tested settings: the two-wave, four-candidate Wide schedule outperforms the four-wave, two-candidate Deep schedule for every solver with lower mean selection regret.
Structural ablations indicate each design component contributes: correct relations beat shuffled relations, revision-aware persistence beats append-only records, and graph-guided plan selection adds a further gain over revisable local records alone.
An updated versus retained hypothesis assessment changes both the chosen intervention and the measured outcome, as shown by a 3.11-point difference (60.44 versus 57.33) in a state-intervention comparison.
If the graph is what carries the gain, the effect should grow as search history lengthens: the reconstruction burden the paper identifies increases with history size, so CIG's advantage over Full History may widen on longer searches or larger tasks, an extension the paper does not test.
Because the seven-criteria plan selector is a proposer judgment rather than a deterministic rule, portfolio choice introduces run-to-run variance; a fixed or learned selection policy would isolate whether the graph structure alone, or its interaction with proposer judgment, produces the reported gains.
The paper leaves transfer of persistent improvement state across tasks and models as future work; a natural corollary is that a mature CIG graph could serve as a reusable experimental prior, letting a new search inherit resolved hypotheses and retired dead ends from a related task.
The token-budget comparison, at a common proposer budget, suggests CIG is more compute-efficient per candidate; if replicated across configurations, this would point to graph-scoped proposer calls, not raw search budget, as the key efficiency lever.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agentic Harness is the runtime that constructs task context and controls execution flow, thereby shaping overall agent performance. Given a fixed model and external evaluation, automated Harness optimization seeks to improve this runtime through an iterative proposal–evaluation loop to better solve target tasks. Existing meta-harness methods mainly adopt proposer-centric discovery, in which an LLM-based proposer integrates accumulated experimental findings to determine subsequent Harness revisions. This places the burden of maintaining the evolving improvement state on the proposer as history expands and its underlying experimental logic becomes harder to discern. In this paper, we introduce the Causal Improvement Graph (CIG), a graph-governed meta-harness framework that externalizes the evolving improvement state in a persistent graph, allowing prior findings to directly govern subsequent Harness optimization through local proposer operations. CIG grows and links Evidence, Hypothesis, Intervention, and Outcome nodes to represent what was observed, how it may be explained, how to test that explanation, and what the evaluation reveals. Their structural relations preserve how the improvement state changes across iterations, allowing local proposers to build directly on relations among prior findings rather than recover them from raw history. Across various agent tasks, CIG discovers stronger Harnesses than previous meta-harness baselines and remains robust to the choice of task solver and proposer. Structural ablations further support the design of an explicit improvement state with graph-governed evolution.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-06
The authors' main claim, as the classifier picked it out 0.99
“Across various agent tasks, CIG discovers stronger Harnesses than previous meta-harness baselines and remains robust to the choice of task solver and proposer.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.08 · human participants 0.03 · controlled comparison 0.81 · randomized assignment 0.02 · states quantitative result 0.04 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agentic Harness is the runtime that constructs task context and controls execution flow, thereby shaping overall agent performance. | background 1.00 | 0.07 | 0.02 | asserted 1.00 |
| Given a fixed model and external evaluation, automated Harness optimization seeks to improve this runtime through an iterative proposal--evaluation loop to better solve target tasks. | background 1.00 | 0.07 | 0.03 | asserted 0.96 |
| Existing meta-harness methods mainly adopt proposer-centric discovery, in which an LLM-based proposer integrates accumulated experimental findings to determine subsequent Harness revisions. | background 0.99 | 0.04 | 0.02 | asserted 1.00 |
| This places the burden of maintaining the evolving improvement state on the proposer as history expands and its underlying experimental logic becomes harder to discern. | gap 0.68 | 0.06 | 0.02 | asserted 1.00 |
| In this paper, we introduce the Causal Improvement Graph (CIG), a graph-governed meta-harness framework that externalizes the evolving improvement state in a persistent graph, allowing prior findings to directly govern subsequent Harness optimization through local proposer operations. | contribution 0.97 | 0.84 | 0.20 | asserted 1.00 |
| CIG grows and links Evidence, Hypothesis, Intervention, and Outcome nodes to represent what was observed, how it may be explained, how to test that explanation, and what the evaluation reveals. | method 0.95 | 0.77 | 0.05 | asserted 0.99 |
| Their structural relations preserve how the improvement state changes across iterations, allowing local proposers to build directly on relations among prior findings rather than recover them from raw history. | method 0.95 | 0.59 | 0.06 | asserted 1.00 |
| Across various agent tasks, CIG discovers stronger Harnesses than previous meta-harness baselines and remains robust to the choice of task solver and proposer. | result 1.00 | 0.94 | 0.72 | asserted 1.00 |
| Structural ablations further support the design of an explicit improvement state with graph-governed evolution. | result 0.83 | 0.91 | 0.09 | asserted 0.94 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-06: 1 request, 6,943 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (4)
Figures are the authors', reproduced from the paper.
References (47)
- [1] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA : Refle... on Pith
- [2] Alibaba Cloud . GLM-5.2-Fast-Preview , 2026. URL https://www.alibabacloud.com/help/en/model-studio/glm-5-2-fast on Pith
- [3] Alibaba Group . Alibaba open-sources Qwen3.5 , a natively multimodal model built for high-efficiency inference, 2026. URL https://www.alibabagroup.com/en-US/document-1960233590314762240 on Pith
- [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Conference on Neural Information Processing Systems, 2020 on Pith
- [5] Jiashun Chen, Yirong Mao, and Wenhui Que. EviGraph : Towards verifiable evidence construction for information-seeking agents. arXiv preprint arXiv:2608.24667, 2026 a on PithPith's reviewarXiv
- [6] Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, and Qing Wang. From failed trajectories to reliable LLM agents: Diagnosing and repairing harness flaws. arXiv preprint arXiv:2606.06324, 2026 b on PithPith's reviewarXiv
- [7] Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, Yuan Xie, Heng Qu, Kun Shao, and Jian Luan. HarnessX : A composable, adaptive, and evolvable agent harness foundry. arXiv preprint arXiv:2606.14249, 2026 c on PithPith's reviewarXiv
- [8] DeepSeek-AI . DeepSeek V4 model card, 2026. URL https://fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdf on Pith
- [9] Yuetian Du, Yucheng Wang, He Xu, Jiexu Xu, Shanwen Tan, Bing Zhao, Boyu Yang, Zhijie Xu, Ming Kong, Hu Wei, et al. Living-harness is an interactive-agent evolver. arXiv preprint arXiv:2607.26598, 2026 on PithPith's reviewarXiv
- [10] Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. LawBench : Benchmarking legal knowledge of large language models. Conference on Empirical Methods in Natural Language Processing, 2024 on Pith
- [11] Gretel AI . Symptom to diagnosis dataset, 2023. URL https://huggingface.co/datasets/gretelai/symptom_to_diagnosis on Pith
- [12] Zhezheng Hao, Hong Wang, Jian Luo, Jianqing Zhang, Yuyan Zhou, Qiang Lin, Can Wang, Hande Dong, and Jiawei Chen. ReCreate : Reasoning and creating domain agents driven by experience. Annual Meeting of the Association for Computational Linguistics, 2026 on Pith
- [13] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. International Conference on Learning Representations, 2025 on Pith
- [14] Yoichi Ishibashi, Taro Yano, and Masafumi Oyamada. Effective harness engineering for algorithm discovery with coding agents. arXiv preprint arXiv:2605.15221, 2026 on PithPith's reviewarXiv
- [15] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench : Can language models resolve real-world GitHub issues? International Conference on Learning Representations, 2024 on Pith
- [16] Kimi Team et al. Kimi K3 : Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026 on PitharXiv
- [17] Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang. Recursive harness self-improvement. arXiv preprint arXiv:2607.15524, 2026 a on PithPith's reviewarXiv
- [18] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness : End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026 b on PithPith's reviewarXiv
- [19] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026 on PithPith's reviewarXiv
- [20] Ryan Lopopolo. Harness engineering: Leveraging Codex in an agent-first world, 2026. URL https://openai.com/index/harness-engineering/ on Pith
show 27 more
- [21] Xinghua Lou, Miguel L \'a zaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P. Murphy. AutoHarness : Improving LLM agents by automatically synthesizing a code harness. arXiv preprint arXiv:2603.03329, 2026 on PitharXiv
- [22] Yaxin Luo, Haobin Jiang, Jialv Zou, Xu Huang, Wenhao Yan, Haodong Li, Zhengrong Yue, Jing Li, Xiaofu Chen, Xiaohan Zhao, Jiacheng Liu, Jiacheng Cui, Zhiqiang Shen, and Xiaotong Li. AutoDesign : Meta-harness optimization for long-horizon agentic design. arXiv preprint arXiv:260... on PithPith's reviewarXiv
- [23] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-Bench : Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11... on PithPith's reviewarXiv
- [24] MiniMax . MiniMax M2.5 : Built for real-world productivity, 2026. URL https://www.minimax.io/news/minimax-m25 on Pith
- [25] OpenAI . GPT-5 mini model, 2025. URL https://developers.openai.com/api/docs/models/gpt-5-mini on Pith
- [26] OpenAI . Introducing GPT-5.5 , 2026 a . URL https://openai.com/index/introducing-gpt-5-5/ on Pith
- [27] OpenAI . GPT-5.6 : Frontier intelligence that scales with your ambition, 2026 b . URL https://openai.com/index/gpt-5-6/ on Pith
- [28] OpenClaw . OpenClaw : Your assistant, on your devices, in your chats, 2026. URL https://github.com/openclaw/openclaw on Pith
- [29] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Conference on Neural Information Processing Systems, 2022 on Pith
- [30] Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. AutoSaddler : Automatic harness optimization with durable updates from agent execution trac... on PithPith's reviewarXiv
- [31] Zehua Pei, Hui-Ling Zhen, Shixiong Kai, Sinno Jialin Pan, Yunhe Wang, Mingxuan Yuan, and Bei Yu. SCOPE : Prompt evolution for enhancing agent effectiveness. arXiv preprint arXiv:2512.15374, 2025 on PithPith's reviewarXiv
- [32] Qwen Team . Qwen3.7-Flash model release, 2026. URL https://docs.qwencloud.com/changelog/models on Pith
- [33] Zhenjiang Ren, Ruiji Li, Xujing Zhang, Ziliang Pang, Shuo Ren, and Jiajun Zhang. EviGraph : Evidence-guided autonomous research agents. arXiv preprint arXiv:2608.04738, 2026 on PithPith's reviewarXiv
- [34] Nadine Schneider, Nikolaus Stiefl, and Gregory A. Landrum. What's what: The (nearly) definitive guide to reaction role assignment. Journal of Chemical Information and Modeling, 2016 on Pith
- [35] Thariq Shihipar. A harness for every task: Dynamic workflows in Claude Code , 2026. URL https://claude.com/blog/a-harness-for-every-task-dynamic-workflows-in-claude-code on Pith
- [36] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld : A controllable world of apps and people for benchmarking interactive coding agents. Annual Meeting of the Association f... on Pith
- [37] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. arXiv preprint arXiv:2607.12227, 2026 a on PithPith's reviewarXiv
- [38] Zefeng Wang, Minxi Yan, Jinhe Bi, Sikuan Yan, Volker Tresp, and Yunpu Ma. MetaSkill-Evolve : Recursive self-improvement of LLM agents via two-timescale meta-skill evolution. arXiv preprint arXiv:2607.05297, 2026 b on PithPith's reviewarXiv
- [39] Z.ai . GLM-5.3 : Overview, 2026. URL https://docs.z.ai/guides/llm/glm-5.3 on Pith
- [40] Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer ( STOP ): Recursively self-improving code generation. Conference on Language Modeling, 2024 on Pith
- [41] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-Harness : Harnesses that improve themselves. arXiv preprint arXiv:2606.09498, 2026 a on PithPith's reviewarXiv
- [42] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin G \"o del Machine : Open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954, 2025 a on PithPith's reviewarXiv
- [43] Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. Hyperagents. arXiv preprint arXiv:2603.19461, 2026 b on PithPith's reviewarXiv
- [44] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow : Automating agentic workflow generation. International Conference on Learning Represen... on Pith
- [45] Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, Ran Xu, and Zeyuan Chen. DarwinX : Evolving agent harnesses through natural selection. arXiv preprint arXiv:2608.07545, 2026 c on PithPith's reviewarXiv
- [46] Tailin Zhou. Hierarchical self-improvement: A framework for task-specific evolvable agent harnesses. arXiv preprint arXiv:2608.08466, 2026 on PithPith's reviewarXiv
- [47] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J \"u rgen Schmidhuber. GPTSwarm : Language agents as optimizable graphs. International Conference on Machine Learning, 2024 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment