Computation and Language · arXiv:2609.14857 v2 · submitted 14 Sep 2026
ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement
The paper says
“Recent work extends recursive self-improvement (RSI) to agent harnesses for long-horizon coding and terminal tasks, enabling agents to improve execution mechanisms from experience. However, generalizable harness RSI remains challenging.”
more of the abstract
First, evolving harnesses on evaluation benchmarks or their subsets makes it difficult to distinguish reusable improvements from benchmark-specific adaptation. Second, single-trajectory updates can conflate systematic harness deficiencies with instance-specific reasoning and solution details, producing modifications that transfer poorly to unseen tasks. Third, localizing recurring behavioral deficiencies within monolithic harnesses is difficult, while whole-harness optimization can entangle unrelated mechanisms and complicate attribution and validation. We propose ModularRSI, a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution. ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies. It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection. Each module evolves independently within a restricted modification scope, followed by an integration stage that combines the evolved modules into a unified harness and resolves potential conflicts. To support benchmark-disjoint evolution, we curate 2,000 executable evolution tasks from external sources that are disjoint from downstream evaluation benchmarks. Experiments on TB2.0 and SWE-Bench Verified show consistent improvements on unseen in-domain and cross-domain tasks, with the evolved harness also transferring across different foundation models.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Worth refereeing: a genuinely useful benchmark-disjoint evolution protocol and dataset with plausible transfer results, but the headline numbers rest on a post-hoc difficulty-distribution choice and the central contrastive mechanism is explicitly unablated. This line is wrong
4 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §6.6; Tables 2 and 7
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The headline SWE-Bench Verified result in Table 2 (76.45) is the Medium-centered evolution-data configuration, and §6.6 presents that configuration as one of two alternatives (Medium-centered vs. Hard & Easy, the latter scoring 74.25). The manuscript never states whether the difficulty distribution was fixed before any downstream evaluation, and Table 7 is a single comparison with no variance or significance information.
more of objection 1
If the configuration was selected with knowledge of downstream SWE-Bench performance, then the benchmark-disjoint protocol is violated at the hyperparameter level and the abstract's claim that evolution is disjoint from downstream benchmarks is too strong; if the choice was made a priori or on the evolution-set evidence shown in Fig. 4, the paper should say so explicitly. Either way, the 2.20-point gap between the two configurations in Table 7 is comparable to the 3.05-point gain of the chosen configuration over the no-evolution baseline (73.40), so this selection dominates the reported improvement. I ask the authors to report both configurations with proper uncertainty, state the selection chronology, and qualify the benchmark-disjoint claim accordingly.
The paper says, at §6.1–6.4; Tables 2, 3, 4, 7
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
All reported numbers are single-run point estimates from stochastic rollouts, with no error bars, repeated seeds, or significance tests. This matters concretely at the reported granularity: TerminalBench 2.0 contains 89 tasks, so the headline +4.86 accuracy points (47.57 to 52.43) and the Pass 3 gain (30.34 to 35.96) each correspond to only a few tasks changing outcome, while the SWE-Bench Verified gain of 3.05 points corresponds to roughly 15 of 500 tasks.
more of objection 2
Under three-rollout evaluation, per-task outcomes are noisy, and as presented none of the reported differences can be distinguished from sampling variation. I request multi-seed runs or, at minimum, bootstrap confidence intervals computed over paired per-task outcomes for the main comparisons in Tables 2, 3, 4, and 7, together with a statement of the number of rollouts underlying each reported accuracy.
The paper says, at §7; §6.5; Tables 4 and 5
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The paper states in §7 that 'we do not conduct a dedicated ablation that isolates the contribution of contrastive trajectory analysis,' and this gap is load-bearing rather than cosmetic: Contribution 1 and §6.5 attribute the framework's generalizability specifically to the combination of same-task contrast, cross-task aggregation, and module-restricted evolution, yet Table 4 varies only the modularity axis (non-modular, joint, modular) with contrastive analysis present in every arm, and Fig. 3's decreasing usable-pair ratio is a correlation, not a causal test.
more of objection 3
The case studies in Appendix D are explicitly non-isolating ('These runs also contain other updates'). I ask for an ablation that removes or randomizes the contrastive pairing (for example, single-sided failed-trajectory diagnosis for all tasks, or pairing successful and failed trajectories from different tasks) while keeping modular evolution and the validation gates fixed; without such an arm, the observed gains cannot be attributed to the contrastive mechanism as opposed to the modular decomposition or the validation gates.
The paper says, at §3.5; §7
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The protocol's defense against task-specific overfitting rests on a reward-blind diff review performed by the same model that authored the modification, plus execution validation on only two tasks randomly sampled from the current evolution batch. Two tasks from the current batch cannot reveal batch-level overfitting, and a self-review by the editing model carries a systematic risk of leniency; the manuscript reports no statistics on gate rejection or rollback rates and no independent check of review reliability.
more of objection 4
Since the introduction's contribution list states that modifications are retained only after 'validation for correctness, executability, and task-specific overfitting,' I ask for quantitative gate-level evidence (for example, rejection rates per gate across the evolution runs and a small independent-model or human audit of diff-review verdicts), or a clearly scoped statement that the overfitting guard is as yet unvalidated. This is a support gap rather than a demonstrated failure, but it is central to the generalization claim.
6 minor objections
The paper says, at §6.1; Table 3; §6.4
Pith says: minor objection 1 of 6
Model naming is inconsistent across the paper: §6.1 attributes Table 2 to DeepSeek-V4-Flash-Preview, Table 3 labels one row DeepSeek-V4-Flash, and §6.4 uses DeepSeek-V4-Flash-0731; the authors should spell out the model versions per table and explain why the Terminus-2 baseline differs between Table 2 (47.57) and Table 6 (61.79).
The paper says, at §5.2
Pith says: minor objection 2 of 6
The metrics 'Pass@3' and 'Pass 3' are nearly identical in notation but mean different things (at least one of three rollouts succeeds vs. all three rollouts succeed); consider renaming them (for example, Any@3 and All@3) to avoid ambiguity, and fix the spacing/colon typo in the Pass 3 definition.
show 4 more
The paper says, at §6.5; Fig. 3; §5.3
Pith says: minor objection 3 of 6
The text says the contrastive-pair ratio is computed 'in each epoch on TerminalBench 2.0'; since evolution runs on the disjoint TB-related evolution set, this statement contradicts the benchmark-disjoint protocol on its face, and it should be clarified that Fig. 3 refers to the evolution set. The rollout count K defined in Eq. (1) is also never specified in the implementation details and should be stated.
The paper says, at Title page; Abstract; Fig. 2
Pith says: minor objection 4 of 6
Typography and presentation: the title and parts of the abstract have words run together ('ANDGENERALIZABLERECURSIVEHARNESSSELF-IMPROVEMENT' and 'demonstratinggeneralizable'), and the task-category distribution is reported twice in Fig. 2 (panel (a) and the embedded table); these should be cleaned up.
The paper says, at §4–§5.3
Pith says: minor objection 5 of 6
The contributions highlight the 2,000-instance curated pool, while the experiments use 120 instances per domain (240 total); this discrepancy is acknowledged in §7, but the main results sections should remind the reader that all headline comparisons use the 240-task subset, and Table 2 should also state the rollout policy and seed policy used for evaluation.
The paper says, at §4; Table 1
Pith says: minor objection 6 of 6
Table 1's categorical 'not use benchmark data' claim should be reconciled with §4's statement that downstream benchmark category labels are used to guide external resource retrieval; the text says this is high-level guidance only, but the caveat should appear next to the table, and the same precision should be applied to the difficulty-distribution construction in Appendix E.
What would settle it
Run ModularRSI on the same 120 SWE-related evolution tasks, but replace the contrastive trajectory analysis (Sec. 3.3) with a single-sided diagnosis for all groups: for every task, regardless of reward pattern, the Code-Modify Agent analyzes only the failed trajectories (or only the successful ones for efficiency) without cross-trajectory comparison. Keep all other components (modular decomposition, module-wise evolution, validation gates) identical. If the resulting harness achieves similar accuracy on TerminalBench 2.0 and SWE-Bench Verified, then the contrastive analysis is not the driver of improvements. If accuracy drops significantly, the claim is supported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Quick take: credible engineering, and the benchmark-disjoint protocol is the most useful part of the paper, but the abstract oversells the evidence. Two specific things matter. First, Section 6.6 shows the SWE headline (76.45) comes from the Medium-centered difficulty distribution, and the same number is the main result in Table 2. The paper never states the difficulty mix was chosen before seeing downstream results, so on the written record it looks like selection on the evaluation metric. That weakens the benchmark-disjoint claim. Second, Section 7 admits there is no ablation isolating the contrastive trajectory analysis, which is the mechanism the abstract leads with. The stress-test concern lands there.
What is genuinely new and good: the 2,000-task evolution set, curated from external sources with similarity filtering against the benchmarks and released with code, is a real community resource. The evaluation design is the right one for the claim: frozen evolved harness, tested in-domain, cross-domain, and cross-model. Gains are directionally consistent in every setting (Tables 2 and 3), a reasonable pattern argument against pure noise even without error bars. The AHE and Meta-Harness comparison under a unified protocol is fair-minded, and the Appendix D case studies are concrete, with the authors noting the comparisons do not isolate any single change.
Soft spots, in proportion. The difficulty-distribution choice is the one I would push hardest on; reporting both settings is honest, but the better one should not headline without a pre-registered protocol or a clearly a priori justification. The missing contrastive ablation is disclosed, but the central mechanism deserves a test with pairing removed or randomized. No significance tests is real but endemic here, and TB2.0's 89 tasks means a couple of points move easily. Matching compute across methods by epoch count in Table 6 is rough. The evolution experiments use only 240 of the 2,000 tasks, fine for the claims but worth stating more prominently.
Who it is for: anyone working on harness evolution or agent evaluation methodology will want the dataset and protocol. It deserves a serious referee; the revision path is clear: variance or significance estimates, an a priori rule for the difficulty mix, and a contrastive-pairing ablation even on a subset.
The strongest objection
The paper's central mechanism is contrastive trajectory pairing (Sec. 3.3), where successful and failed trajectories of the same task are compared to identify harness deficiencies. The authors argue this is essential for generalizable improvements. However, Section 7 explicitly states no dedicated ablation isolates the contribution of this contrastive analysis. The ablations in Table 4 compare modular vs. joint vs. non-modular evolution, but all used the same contrastive analysis. Without a version that removes or randomizes the contrastive pairing, the gains could arise from the modular decomposition or the validation gates rather than from the specific contrastive mechanism. The case studies (Appendix D) are illustrative but not systematic. Therefore, the central causal claim is undersupported.
The reader's weakest_assumption identified the reliability of LLM self-diagnosis as a core assumption. The paper's own limitation (Sec. 7) admits the lack of an ablation isolating contrastive analysis. This is the most load-bearing gap because it directly tests whether the claimed mechanism drives improvements. The proposed concrete test is a clean ablation that isolates the contrastive pairing while keeping the rest of the pipeline intact. The current evidence (Tables 2-7, case studies) supports a CONDITIONAL verdict but cannot confirm causation. Thus the verdict remains CONDITIONAL pending this test.
The referee's report
If the reported results hold up, ModularRSI is a useful step for harness RSI: it is one of the few frameworks that does not evolve on downstream benchmark instances, it releases a sizable curated evolution pool, and its structured-finding format with explicit abstention (Appendix B) and staged validation gates (static checks, reward-blind diff review, execution validation) are concrete protocol contributions. The controlled comparison of modular, joint, and non-modular evolution (Table 4) and the unified-protocol comparison against AHE and Meta-Harness (Table 6) are appropriately scoped, and the paper is transparent in shipping code and data and in stating its limitations in Section 7. However, the significance is currently tempered by three issues: the single-seed stochastic evaluations with no uncertainty quantification, the unstated selection chronology for the Medium-centered difficulty distribution that underlies the headline SWE-Bench number, and the absence of any controlled test of the contrastive mechanism that the abstract names as a core ingredient. These are fixable, but they currently prevent confident acceptance of the central generalization claim.
Pith's reading of the paper (a machine's summary, not the authors')
The paper introduces ModularRSI, a framework for recursive self-improvement of coding-agent harnesses. Its central claim is that by decomposing the harness into five independent modules and evolving each one separately using contrastive analysis of successful and failed trajectories, the system learns reusable improvements that generalize to unseen tasks and even to different underlying language models. The authors curate a benchmark-disjoint dataset of 2,000 tasks to avoid overfitting to evaluation benchmarks. Experiments on TerminalBench 2.0 and SWE-Bench Verified show consistent accuracy gains, with the evolved harness also improving performance when used with models not seen during evolution.
What carries the argument. The central mechanism is the combination of a five-module decomposition of the agent harness (Agent Loop, Tool Use, Observation Management, Context Management, Task Completion Detection), contrastive trajectory analysis that pairs successful and failed rollouts of the same task to identify reusable behavioral deficiencies, independent module-wise evolution with restricted modification scope, and cross-module integration with validation gates (program checks, diff review, execution validation) that reject task-specific overfitting.
Core claim. ModularRSI establishes that harness-level recursive self-improvement can generalize beyond the specific tasks used during evolution when the improvement process is decomposed into independent module-level changes guided by contrastive trajectory pairs. The key empirical result is that a harness evolved on TB-related tasks improves SWE-Bench Verified accuracy from 73.40 to 75.80 (out-of-domain transfer), and a harness evolved on SWE-related tasks improves TerminalBench 2.0 from 47.57 to 49.40. Additionally, the improvements transfer across different foundation models: a harness evolved with one model also boosts accuracy on two other models. The paper argues that modularity prevents cross-mechanism interference, contrastive analysis identifies generalizable deficiencies, and the benchmark-disjoint protocol ensures that gains reflect genuine capability improvements rather than benchmark-specific adaptation.
The premise it rests on. The framework assumes that the LLM acting as diagnostician and editor can reliably distinguish generalizable harness deficiencies from task-specific patterns using contrastive trajectory analysis, and that the validation gates (self-review, two-task execution test) catch overfitting without introducing false rejections.
If the paper is right (Pith's inferences, not the authors')
Harness self-improvement can be conducted on a curated, benchmark-disjoint dataset rather than on task-specific evaluation data, making the process more scalable and less prone to overfitting.
The modular evolution approach allows different aspects of harness behavior to be improved in parallel and then combined, potentially reducing the computational cost of full-harness evolution.
The cross-model transfer of improvements suggests that some harness mechanisms are model-agnostic, meaning a harness optimized for one LLM can be reused with others without re-evolution.
The difficulty distribution of evolution tasks matters: moderately difficult tasks provide more informative contrastive signals than mixtures of very easy and very hard tasks, guiding dataset construction for future self-improvement systems.
If the contrastive trajectory analysis reliably identifies generalizable deficiencies, the same approach could be extended to other agent-environment interaction domains, such as robotics or game-playing, by defining analogous modular decompositions.
The paper does not fully ablate the contribution of contrastive analysis; a stronger test would compare against random pairing of trajectories or single-trajectory analysis with the same total evidence, which could reveal whether the contrastive mechanism is necessary for the observed gains.
The evolution uses only 240 instances out of the 2000 curated; scaling to the full dataset might yield further gains or reveal diminishing returns, and future work should explore the effect of evolution-set size.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Recent work extends recursive self-improvement (RSI) to agent harnesses for long-horizon coding and terminal tasks, enabling agents to improve execution mechanisms from experience. However, generalizable harness RSI remains challenging. First, evolving harnesses on evaluation benchmarks or their subsets makes it difficult to distinguish reusable improvements from benchmark-specific adaptation. Second, single-trajectory updates can conflate systematic harness deficiencies with instance-specific reasoning and solution details, producing modifications that transfer poorly to unseen tasks. Third, localizing recurring behavioral deficiencies within monolithic harnesses is difficult, while whole-harness optimization can entangle unrelated mechanisms and complicate attribution and validation. We propose ModularRSI, a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution. ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies. It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection. Each module evolves independently within a restricted modification scope, followed by an integration stage that combines the evolved modules into a unified harness and resolves potential conflicts. To support benchmark-disjoint evolution, we curate 2,000 executable evolution tasks from external sources that are disjoint from downstream evaluation benchmarks. Experiments on TB2.0 and SWE-Bench Verified show consistent improvements on unseen in-domain and cross-domain tasks, with the evolved harness also transferring across different foundation models.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.98
“Experiments on TB2.0 and SWE-Bench Verified show consistent improvements on unseen in-domain and cross-domain tasks, with the evolved harness also transferring across different foundation models.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.14 · human participants 0.03 · controlled comparison 0.18 · randomized assignment 0.02 · states quantitative result 0.72 · compares with prior results 0.40 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Recent work extends recursive self-improvement (RSI) to agent harnesses for long-horizon coding and terminal tasks, enabling agents to improve execution mechanisms from experience. | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| However, generalizable harness RSI remains challenging. | gap 0.88 | 0.06 | 0.02 | asserted 0.99 |
| First, evolving harnesses on evaluation benchmarks or their subsets makes it difficult to distinguish reusable improvements from benchmark-specific adaptation. | gap 0.75 | 0.08 | 0.02 | asserted 0.97 |
| Second, single-trajectory updates can conflate systematic harness deficiencies with instance-specific reasoning and solution details, producing modifications that transfer poorly to unseen tasks. | gap 0.73 | 0.11 | 0.02 | asserted 0.89 |
| Third, localizing recurring behavioral deficiencies within monolithic harnesses is difficult, while whole-harness optimization can entangle unrelated mechanisms and complicate attribution and validation. | gap 0.60 | 0.09 | 0.02 | asserted 0.99 |
| We propose ModularRSI, a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution. | contribution 1.00 | 0.84 | 0.27 | asserted 0.99 |
| ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies. | method 0.95 | 0.80 | 0.09 | asserted 1.00 |
| It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection. | method 0.93 | 0.87 | 0.04 | asserted 1.00 |
| Each module evolves independently within a restricted modification scope, followed by an integration stage that combines the evolved modules into a unified harness and resolves potential conflicts. | method 0.99 | 0.69 | 0.05 | asserted 0.99 |
| To support benchmark-disjoint evolution, we curate 2,000 executable evolution tasks from external sources that are disjoint from downstream evaluation benchmarks. | method 0.97 | 0.95 | 0.03 | asserted 1.00 |
| Experiments on TB2.0 and SWE-Bench Verified show consistent improvements on unseen in-domain and cross-domain tasks, with the evolved harness also transferring across different foundation models. | result 1.00 | 0.95 | 0.74 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,218 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
Cited by 15 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- HandRSI: Recursive Self-Improvement for Hand Pose Estimation cs.CV · Nov 2026
- Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback cs.AI · Nov 2026
- The Trinity: Capability-Driven Recursive Self-Improvement for Multimodal Agents cs.AI · Oct 2026
- Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims cs.LG · Oct 2026
- AtomSearch: Evidence-Grounded Tree Search for Agent Skill Optimization cs.CL · Oct 2026
- VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness cs.CV · Oct 2026
- KnowMe Paper Radar: Lightweight Discovery of Promising Papers in the Age of Autoresearch cs.AI · Nov 2026
- From Recursion to Convolution: Parallel Self-Improvement for Multi-Agent Systems cs.AI · Nov 2026
- Veritas: A Verifier Generation Harness for Agent Tasks cs.AI · Nov 2026
- TactiNavi: Serving and Revising an LLM Game Coach with Explicit Strategy Conditions cs.AI · Nov 2026
- CoHARL: Co-Evolving Agent Policies and Executable Harnesses with Reinforcement Learning cs.LG · Oct 2026
- How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation cs.SE · Oct 2026
- Harness Self-Evolution for Long-Horizon Terminal Agents cs.AI · Oct 2026
- DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration cs.AI · Oct 2026
- Scaling Trajectories for Complex Tasks through Recursive Self-Rewrite cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment