Software Engineering · arXiv:2610.34938 v1 · submitted 26 Oct 2026 · 10 references
How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation
The paper says
“Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear.”
more of the abstract
We study e̊tro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from e̊tro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid empirical study: first to isolate individual harness adaptations and measure whether they reduce repeated work on later tasks; the core finding that coverage predicts effectiveness is well-supported despite the small sample. This line is wrong
4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 10 references say.
4 minor objections
The paper says, at Table II
Pith says: minor objection 1 of 4
The column 'Consulted' reports whether adapted sessions consulted the adaptation. Consider adding a footnote or clarifying in the caption what 'consulted' means (e.g., file read or command executed), as the text distinguishes 'loaded' from 'consulted' but the table uses only 'Consulted'.
The paper says, at §III, Transfer paragraph
Pith says: minor objection 2 of 4
The distinction between 'encountered' and 'covered' is central to the analysis but introduced somewhat late. A brief definition earlier in the design section (e.g., after describing the behavioral measure) would help readers follow the results in Table II.
show 2 more
The paper says, at §IV, first paragraph
Pith says: minor objection 3 of 4
The sentence 'Only nine of the 38 executable checks could report failure in the task environment as written' could be clarified: does 'report failure' mean the hook would actually fail, or merely detect a failure? The following sentences explain, but rewording would improve readability.
The paper says, at §VI, Conclusion
Pith says: minor objection 4 of 4
The conclusion mentions 'retained changes also need continued evaluation' (b147), which is well supported by the control-task results. Consider adding a concrete example from the control tasks to reinforce this point in the conclusion.
What would settle it
Reproduce the analysis of the seven informative transfer events using an independent set of issues and sessions, with preregistered criteria for coverage and recurrence, to confirm that the observed pattern (recurrence reduction only when coverage is present) replicates with new data and blinded labeling.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is the most careful empirical study I've seen on coding-agent harness self-evolution. The novelty is real: prior work evaluated bundled changes or overall task success. This paper isolates one adaptation at a time, runs five fresh sessions with and without it on targeted later issues, and checks whether the targeted action actually recurs. That design alone is an upgrade.
The headline result is clean: two adaptations that covered the exact instances encountered later eliminated the repeated work; four that didn't cover those instances showed no reduction; one incomplete known-failures file increased verification work on unrelated tasks. The authors deserve credit for transparency — they flag the small sample (seven informative events), the post-hoc grouping, and that the design doesn't establish causation (b140). They also acknowledge that the evaluation likely overestimates effectiveness because issues were selected to reproduce the problem (b137). None of this is buried.
The soft spots are the ones they name. Seven informative events is thin for strong conclusions, and the grouping by coverage is post-hoc. The study uses one agent (Claude Code with Opus 5.5), one version of /retro, and only environment problems. The LLM-based coding of proposals (b55) introduces uncertainty, though the agreement stats are reasonable. The labeling of recurrence on traces wasn't fully blind (b139). None of these are fatal; they limit the scope of what we can claim.
This paper is for researchers building or evaluating self-evolution loops in coding agents. It makes a clear, empirically grounded point: generating and implementing changes is not enough; you need evaluation mechanisms. The artifact is complete, and the writing is honest. I'd cite this in work on agent harness design, especially the finding about incomplete known-failures files adding work.
Recommendation: definitely send to peer review. The study is well-designed, its claims are appropriately cautious, and the result is actionable. Minor revisions should ask for clearer reporting of the rationale for issue selection in the second sample and perhaps a table showing which adaptations were excluded at each eligibility step. No major issues.
The strongest objection
The paper is transparent about its limitations, small sample size, and post-hoc grouping of results. The central claim—that harness self-evolution requires evaluation mechanisms—is well supported by the data, even though the sample is small and the evidence for coverage as the explanatory factor is correlational. The authors do not overclaim and explicitly note that the design does not establish causation (b140). The reader's weakest assumptions (representativeness of SWE-bench, LLM labeling) are already acknowledged in the paper (b45, b55, b137). I find no additional load-bearing concern that would undermine the main conclusion or warrant a change in verdict.
The paper is careful, limitations are well-stated, and the central claim is modest and appropriately qualified. The reader's weakest assumptions are already covered by the paper's own threats-to-validity section. The stress-tester does not identify any unaddressed concern that would reduce confidence in the paper's conclusions. However, the reader's weakest assumption focused on representativeness and labeling, while the most subtle unresolved issue is the post-hoc grouping of adaptations by coverage; but this is explicitly acknowledged in b140 and does not threaten the paper's main argument, which is about the need for evaluation. Agreement is partial because the reader's assumptions were correct but not the only possible concerns; the paper addresses them adequately. Verdict unchanged: ACCEPT remains appropriate.
The referee's report
If the result holds, this is a significant contribution to the understanding of coding-agent self-evolution. It provides the first empirical evidence that retrospective harness proposals often target environment problems (known-failures files, test-runner instructions) and that transferring such changes across tasks is fragile: adaptations help only when they cover the exact instances encountered on later tasks, and can add unnecessary work otherwise. The study is carefully scoped to one agent, one retrospective skill, and environment-related problems, and the authors are transparent about limitations including the small sample, targeted issue selection, and post-hoc grouping. The central claim—that self-evolution requires evaluation mechanisms—is well supported by the data and has practical implications for the design of self-evolving coding systems.
Pith's reading of the paper (a machine's summary, not the authors')
This paper studies whether coding agents can improve their own behavior across tasks by retaining changes to their harness (software, configuration, and instructions) based on experience from earlier sessions. Using the /retro skill, the authors collected 125 proposals from 30 coding sessions, implemented 24, and evaluated 12 on later SWE-bench Verified issues from the same repositories. The central finding is that simply generating and implementing a harness change does not guarantee it will reduce repeated work: in the seven evaluations where later sessions encountered the targeted problem, only two changes eliminated the targeted action, one reduced it, four showed no reduction, and some even added work on unrelated issues. The determining factor was whether the adaptation covered the specific instances encountered later, such as listing all failing tests rather than a partial set. The paper concludes that harness self-evolution requires not only proposing and applying changes, but also evaluating their effect on later tasks and revising or removing ineffective ones.
What carries the argument. The central object is the concept of coverage: an adaptation's ability to address the particular problem instances that later sessions encounter. The /retro skill generates proposals for harness changes after a completed session. The study operationalizes coverage by comparing the set of problems listed or addressed by an adaptation (e.g., specific failing tests, test-runner instructions) against the problems that actually arise in later transferred sessions. The key mechanism is the transfer experiment: each adaptation is evaluated one at a time on a later issue chosen to reproduce the targeted problem, with baseline and adapted sessions run in parallel to isolate the effect of the change.
Core claim. The paper establishes that the effectiveness of a retrospective harness adaptation depends on whether it covers the exact problem instances the agent later encounters. Two adaptations that did so—a CLAUDE.md line directing to the correct test runner and a known-failures file listing all tests that failed in the session—eliminated repeated work in every subsequent session. In contrast, five adaptations that covered only a subset of the failures encountered later left repeated work unchanged or only partially reduced. This result is grounded in an empirical design where each adaptation was transferred individually to later issues from the same repository and evaluated with five sessions per harness, with careful recording of whether the adaptation was loaded, consulted, and whether the targeted action recurred.
The premise it rests on. The evaluation issues were selected specifically to reproduce the problem targeted by each adaptation, which the authors acknowledge may overestimate how often adaptations would help on unselected tasks; additionally, the proposal coding was done by two LLMs rather than humans, and the agreement reflects consistency across LLM inputs, not human inter-rater reliability.
If the paper is right (Pith's inferences, not the authors')
Harness self-evolution systems should include an evaluation loop that checks whether each retained change actually reduces targeted actions in subsequent sessions, and should discard changes that do not.
Designers of coding agents should prioritize adaptations that capture repository-level invariants (e.g., the correct test command) over adaptations that list environment-specific failures, because the latter often do not cover future encountered instances.
Lint-related proposals that recommend a practice regardless of session events (18 of 43) are unlikely to be loaded or consulted in practice and may add overhead without benefit.
A retained change that only partially covers later encountered failures can mislead the agent into performing additional verification work, increasing session time without improving task success.
The finding that coverage determines effectiveness suggests a natural scalability constraint: as the set of encountered problems grows, a harness must either expand to list all possible failures or adopt a strategy that generalizes (e.g., learning test-runner conventions rather than recording specific test outcomes). Partial lists become liabilities.
The study design could be replicated with other agents (e.g., GPT-based agents) and other versions of /retro to test whether the coverage condition is an invariant or specific to Claude Code and the current skill implementation.
An automated post-session evaluation that compares agent actions under original and adapted harnesses, without requiring human labeling of traces, would be a direct engineering extension of the paper's finding that evaluation is the missing piece.
The paper implicitly assumes that reducing repeated work is beneficial, but in two control tasks the adapted harness increased verification work; a cost-benefit analysis that accounts for task success, time, and resource usage would strengthen future work.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear. We study e̊tro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from e̊tro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 1.00
“Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.82 · human participants 0.12 · controlled comparison 0.94 · randomized assignment 0.03 · states quantitative result 0.98 · compares with prior results 0.31 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear. | gap 0.74 | 0.04 | 0.04 | asserted 0.80 |
| We study \retro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. | contribution 0.99 | 0.82 | 0.04 | unclear 0.72 |
| We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from \retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. | method 0.99 | 0.98 | 0.03 | asserted 0.96 |
| Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. | result 1.00 | 0.93 | 0.03 | asserted 1.00 |
| Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. | result 1.00 | 0.96 | 0.12 | asserted 1.00 |
| Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not. | interpretation 0.93 | 0.38 | 0.87 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 5,118 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (10)
- [1] H. Zhou, H. Hu, T. Luo, Y . Shang, C. Fang, Z. Chen, L. Xiao, and Q. Zhang, “Self-evolving coding agents,” arXiv:2608.03392, 2026 on PithPith's reviewarXiv
- [2] “skills: Agent skills for engineering, productivity, and writing,” https://github.com/mattpocock/skills, commit b0618bc, 2026, accessed 2026-10-08 on Pith
- [3] M. Galster, S. Mohsenimofidi, J. L. Lulla, M. A. Abubakar, C. Treude, and S. Baltes, “Harness engineer- ing for agentic AI coding tools: An exploratory study,” arXiv:2602.14690, 2026 on PithPith's reviewarXiv
- [4] J. L. Lulla, S. Mohsenimofidi, M. Galster, J. M. Zhang, S. Baltes, and C. Treude, “On the impact of AGENTS.md files on the efficiency of AI coding agents,” arXiv:2601.20404, 2026, presented at the Journal Ahead Workshop (JAWs) at ICSE 2026 on PitharXiv
- [5] X. Zhang, G. Wang, Y . Cui, W. Qiu, Z. Li, B. Zhu, and P. He, “Guardrails beat guidance: A large-scale study of rules, skills, and persistent configuration for coding agents,” arXiv:2604.11088, 2026 on PithPith's reviewarXiv
- [6] J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, Z. Xi, X. Huang, H. Yan, Z. Han, T. Gui, and Y .-G. Jiang, “Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses,” arXiv:2604.25850, 2026 on PithPith's reviewarXiv
- [7] S. Wu, J. Ren, Y . Li, H.-S. Li, C. Yang, X. Ma, Y . Zhang, W. Gu, J. Yang, R. Batista-Navarro, C. Zhang, X. Liu, M. Zhou, B. Dai, and C. Lin, “ModularRSI: Modular and generalizable recursive harness self-improvement,” arXiv:2609.14857, 2026 on PithPith's reviewarXiv
- [8] M. Lin, J. Wu, Z. Wang, Z. Shi, Y . Sang, B. He, Z. Liu, T. Wei, Z. Wu, Z. Zhang, D. Wang, X. Zhang, B. Dumoulin, C. Xie, Y . Zhou, S. Wang, and H. Lu, “Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving LLM agents,” arXiv:2605.30621, 2026 on PithPith's reviewarXiv
- [9] J. Huang, F. Cheng, J. Jiang, Z. Yu, and A. Aizawa, “BenchTrace: A benchmark for testing reflection ability and controlled evolution in LLM agents,” arXiv:2605.29225, 2026 on PithPith's reviewarXiv
- [10] OpenAI, “Introducing SWE-bench Verified,” https: //openai.com/index/introducing-swe-bench-verified/, 2024, accessed 2026-10-10 on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment