{"id":"2af77da0-4614-4d80-92ae-b3c0a019bddf","arxiv_id":"2609.14857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ModularRSI reports that contrastive, module-wise, benchmark-disjoint harness self-improvement raises accuracy on TerminalBench 2.0 and SWE-Bench Verified and transfers across base models.","lead":"ModularRSI is a framework that lets an AI coding agent improve its own execution harness by comparing successful and failed runs, editing separate harness modules, and testing changes on tasks kept separate from the evaluation benchmarks. The authors report consistent accuracy gains on terminal and software-engineering benchmarks and transfer of the improved harness across different base models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No dedicated ablation isolates the contribution of contrastive trajectory analysis, leaving open whether improvements stem from the contrastive mechanism or from other design choices.","rationale":"The reader's weakest_assumption identified the reliability of LLM self-diagnosis as a core assumption. The paper's own limitation (Sec. 7) admits the lack of an ablation isolating contrastive analysis. This is the most load-bearing gap because it directly tests whether the claimed mechanism drives improvements. The proposed concrete test is a clean ablation that isolates the contrastive pairing while keeping the rest of the pipeline intact. The current evidence (Tables 2-7, case studies) supports a CONDITIONAL verdict but cannot confirm causation. Thus the verdict remains CONDITIONAL pending this test.","tokens_in":21014,"tokens_out":2325,"duration_ms":23844,"concrete_test":"Run ModularRSI on the same 120 SWE-related evolution tasks, but replace the contrastive trajectory analysis (Sec. 3.3) with a single-sided diagnosis for all groups: for every task, regardless of reward pattern, the Code-Modify Agent analyzes only the failed trajectories (or only the successful ones for efficiency) without cross-trajectory comparison. Keep all other components (modular decomposition, module-wise evolution, validation gates) identical. If the resulting harness achieves similar accuracy on TerminalBench 2.0 and SWE-Bench Verified, then the contrastive analysis is not the driver of improvements. If accuracy drops significantly, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is contrastive trajectory pairing (Sec. 3.3), where successful and failed trajectories of the same task are compared to identify harness deficiencies. The authors argue this is essential for generalizable improvements. However, Section 7 explicitly states no dedicated ablation isolates the contribution of this contrastive analysis. The ablations in Table 4 compare modular vs. joint vs. non-modular evolution, but all used the same contrastive analysis. Without a version that removes or randomizes the contrastive pairing, the gains could arise from the modular decomposition or the validation gates rather than from the specific contrastive mechanism. The case studies (Appendix D) are illustrative but not systematic. Therefore, the central causal claim is undersupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"ModularRSI is a framework for recursive self-improvement of LLM agent harnesses that combines three design principles: benchmark-disjoint evolution data, contrastive trajectory analysis, and modular harness decomposition. The harness, starting from Terminus-2, is split into five functional modules (Agent Loop, Tool Use, Observation Management, Context Management, Task Completion Detection). For each evolution task, the system performs K rollouts, groups tasks by reward pattern (Positive, Contrastive, Negative), and uses a Code-Modify Agent to produce structured diagnoses from same-task success/failure pairs, with explicit abstention when no module change would plausibly affect the outcome. Candidate modifications pass static program checks, a reward-blind diff review, and a two-task execution sample, after which independently evolved modules are integrated in a cross-module epoch. The authors curate 2,000 external tasks claimed disjoint from TerminalBench 2.0 and SWE-Bench Verified and use 120 tasks per domain for evolution. The paper reports consistent point-estimate gains across settings: in-domain accuracy from 47.57 to 52.43 on TerminalBench 2.0 and from 73.40 to 76.45 on SWE-Bench Verified; cross-domain transfer of 75.80 and 49.40; cross-model gains on TerminalBench 2.0 (Table 3); and modular evolution outperforming joint and non-modular variants (Table 4).","tokens_in":21350,"tokens_out":13642,"duration_ms":105224,"significance":"If the reported results hold up, ModularRSI is a useful step for harness RSI: it is one of the few frameworks that does not evolve on downstream benchmark instances, it releases a sizable curated evolution pool, and its structured-finding format with explicit abstention (Appendix B) and staged validation gates (static checks, reward-blind diff review, execution validation) are concrete protocol contributions. The controlled comparison of modular, joint, and non-modular evolution (Table 4) and the unified-protocol comparison against AHE and Meta-Harness (Table 6) are appropriately scoped, and the paper is transparent in shipping code and data and in stating its limitations in Section 7. However, the significance is currently tempered by three issues: the single-seed stochastic evaluations with no uncertainty quantification, the unstated selection chronology for the Medium-centered difficulty distribution that underlies the headline SWE-Bench number, and the absence of any controlled test of the contrastive mechanism that the abstract names as a core ingredient. These are fixable, but they currently prevent confident acceptance of the central generalization claim.","major_comments":[{"comment":"The headline SWE-Bench Verified result in Table 2 (76.45) is the Medium-centered evolution-data configuration, and §6.6 presents that configuration as one of two alternatives (Medium-centered vs. Hard & Easy, the latter scoring 74.25). The manuscript never states whether the difficulty distribution was fixed before any downstream evaluation, and Table 7 is a single comparison with no variance or significance information. If the configuration was selected with knowledge of downstream SWE-Bench performance, then the benchmark-disjoint protocol is violated at the hyperparameter level and the abstract's claim that evolution is disjoint from downstream benchmarks is too strong; if the choice was made a priori or on the evolution-set evidence shown in Fig. 4, the paper should say so explicitly. Either way, the 2.20-point gap between the two configurations in Table 7 is comparable to the 3.05-point gain of the chosen configuration over the no-evolution baseline (73.40), so this selection dominates the reported improvement. I ask the authors to report both configurations with proper uncertainty, state the selection chronology, and qualify the benchmark-disjoint claim accordingly.","section":"§6.6; Tables 2 and 7"},{"comment":"All reported numbers are single-run point estimates from stochastic rollouts, with no error bars, repeated seeds, or significance tests. This matters concretely at the reported granularity: TerminalBench 2.0 contains 89 tasks, so the headline +4.86 accuracy points (47.57 to 52.43) and the Pass 3 gain (30.34 to 35.96) each correspond to only a few tasks changing outcome, while the SWE-Bench Verified gain of 3.05 points corresponds to roughly 15 of 500 tasks. Under three-rollout evaluation, per-task outcomes are noisy, and as presented none of the reported differences can be distinguished from sampling variation. I request multi-seed runs or, at minimum, bootstrap confidence intervals computed over paired per-task outcomes for the main comparisons in Tables 2, 3, 4, and 7, together with a statement of the number of rollouts underlying each reported accuracy.","section":"§6.1–6.4; Tables 2, 3, 4, 7"},{"comment":"The paper states in §7 that 'we do not conduct a dedicated ablation that isolates the contribution of contrastive trajectory analysis,' and this gap is load-bearing rather than cosmetic: Contribution 1 and §6.5 attribute the framework's generalizability specifically to the combination of same-task contrast, cross-task aggregation, and module-restricted evolution, yet Table 4 varies only the modularity axis (non-modular, joint, modular) with contrastive analysis present in every arm, and Fig. 3's decreasing usable-pair ratio is a correlation, not a causal test. The case studies in Appendix D are explicitly non-isolating ('These runs also contain other updates'). I ask for an ablation that removes or randomizes the contrastive pairing (for example, single-sided failed-trajectory diagnosis for all tasks, or pairing successful and failed trajectories from different tasks) while keeping modular evolution and the validation gates fixed; without such an arm, the observed gains cannot be attributed to the contrastive mechanism as opposed to the modular decomposition or the validation gates.","section":"§7; §6.5; Tables 4 and 5"},{"comment":"The protocol's defense against task-specific overfitting rests on a reward-blind diff review performed by the same model that authored the modification, plus execution validation on only two tasks randomly sampled from the current evolution batch. Two tasks from the current batch cannot reveal batch-level overfitting, and a self-review by the editing model carries a systematic risk of leniency; the manuscript reports no statistics on gate rejection or rollback rates and no independent check of review reliability. Since the introduction's contribution list states that modifications are retained only after 'validation for correctness, executability, and task-specific overfitting,' I ask for quantitative gate-level evidence (for example, rejection rates per gate across the evolution runs and a small independent-model or human audit of diff-review verdicts), or a clearly scoped statement that the overfitting guard is as yet unvalidated. This is a support gap rather than a demonstrated failure, but it is central to the generalization claim.","section":"§3.5; §7"}],"minor_comments":[{"comment":"Model naming is inconsistent across the paper: §6.1 attributes Table 2 to DeepSeek-V4-Flash-Preview, Table 3 labels one row DeepSeek-V4-Flash, and §6.4 uses DeepSeek-V4-Flash-0731; the authors should spell out the model versions per table and explain why the Terminus-2 baseline differs between Table 2 (47.57) and Table 6 (61.79).","section":"§6.1; Table 3; §6.4"},{"comment":"The metrics 'Pass@3' and 'Pass 3' are nearly identical in notation but mean different things (at least one of three rollouts succeeds vs. all three rollouts succeed); consider renaming them (for example, Any@3 and All@3) to avoid ambiguity, and fix the spacing/colon typo in the Pass 3 definition.","section":"§5.2"},{"comment":"The text says the contrastive-pair ratio is computed 'in each epoch on TerminalBench 2.0'; since evolution runs on the disjoint TB-related evolution set, this statement contradicts the benchmark-disjoint protocol on its face, and it should be clarified that Fig. 3 refers to the evolution set. The rollout count K defined in Eq. (1) is also never specified in the implementation details and should be stated.","section":"§6.5; Fig. 3; §5.3"},{"comment":"Typography and presentation: the title and parts of the abstract have words run together ('ANDGENERALIZABLERECURSIVEHARNESSSELF-IMPROVEMENT' and 'demonstratinggeneralizable'), and the task-category distribution is reported twice in Fig. 2 (panel (a) and the embedded table); these should be cleaned up.","section":"Title page; Abstract; Fig. 2"},{"comment":"The contributions highlight the 2,000-instance curated pool, while the experiments use 120 instances per domain (240 total); this discrepancy is acknowledged in §7, but the main results sections should remind the reader that all headline comparisons use the 240-task subset, and Table 2 should also state the rollout policy and seed policy used for evaluation.","section":"§4–§5.3"},{"comment":"Table 1's categorical 'not use benchmark data' claim should be reconciled with §4's statement that downstream benchmark category labels are used to guide external resource retrieval; the text says this is high-level guidance only, but the caveat should appear next to the table, and the same precision should be applied to the difficulty-distribution construction in Appendix E.","section":"§4; Table 1"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Quick take: credible engineering, and the benchmark-disjoint protocol is the most useful part of the paper, but the abstract oversells the evidence. Two specific things matter. First, Section 6.6 shows the SWE headline (76.45) comes from the Medium-centered difficulty distribution, and the same number is the main result in Table 2. The paper never states the difficulty mix was chosen before seeing downstream results, so on the written record it looks like selection on the evaluation metric. That weakens the benchmark-disjoint claim. Second, Section 7 admits there is no ablation isolating the contrastive trajectory analysis, which is the mechanism the abstract leads with. The stress-test concern lands there.\n\nWhat is genuinely new and good: the 2,000-task evolution set, curated from external sources with similarity filtering against the benchmarks and released with code, is a real community resource. The evaluation design is the right one for the claim: frozen evolved harness, tested in-domain, cross-domain, and cross-model. Gains are directionally consistent in every setting (Tables 2 and 3), a reasonable pattern argument against pure noise even without error bars. The AHE and Meta-Harness comparison under a unified protocol is fair-minded, and the Appendix D case studies are concrete, with the authors noting the comparisons do not isolate any single change.\n\nSoft spots, in proportion. The difficulty-distribution choice is the one I would push hardest on; reporting both settings is honest, but the better one should not headline without a pre-registered protocol or a clearly a priori justification. The missing contrastive ablation is disclosed, but the central mechanism deserves a test with pairing removed or randomized. No significance tests is real but endemic here, and TB2.0's 89 tasks means a couple of points move easily. Matching compute across methods by epoch count in Table 6 is rough. The evolution experiments use only 240 of the 2,000 tasks, fine for the claims but worth stating more prominently.\n\nWho it is for: anyone working on harness evolution or agent evaluation methodology will want the dataset and protocol. It deserves a serious referee; the revision path is clear: variance or significance estimates, an a priori rule for the difficulty mix, and a contrastive-pairing ablation even on a subset.","headline":"Worth refereeing: a genuinely useful benchmark-disjoint evolution protocol and dataset with plausible transfer results, but the headline numbers rest on a post-hoc difficulty-distribution choice and the central contrastive mechanism is explicitly unablated.","tokens_in":22015,"tokens_out":6835,"duration_ms":52755},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"ModularRSI demonstrates that contrastive, modular evolution of agent harnesses, using a benchmark-disjoint dataset, produces improvements that transfer to unseen tasks and different foundation models.","keywords":["recursive self-improvement","agent harness","modularity","contrastive trajectory analysis","benchmark-disjoint evolution","terminal agents","software engineering","generalization"],"falsifier":"If the evolved harness were evaluated on a new set of tasks from a third domain (e.g., data science workflows) and failed to show improvement over the baseline, that would contradict the claim of cross-domain generalizability. More specifically, if the accuracy gain on a held-out benchmark disappeared when the benchmark's task distribution was balanced to exclude any tasks similar to the evolution set, that would indicate the improvements are not general.","tokens_in":22524,"feed_emoji":"🧩","tokens_out":4832,"duration_ms":46173,"temperature":0.7,"pith_summary":"The paper introduces ModularRSI, a framework for recursive self-improvement of coding-agent harnesses. Its central claim is that by decomposing the harness into five independent modules and evolving each one separately using contrastive analysis of successful and failed trajectories, the system learns reusable improvements that generalize to unseen tasks and even to different underlying language models. The authors curate a benchmark-disjoint dataset of 2,000 tasks to avoid overfitting to evaluation benchmarks. Experiments on TerminalBench 2.0 and SWE-Bench Verified show consistent accuracy gains, with the evolved harness also improving performance when used with models not seen during evolution.","feed_headline":"Contrastive modular evolution improves agent harness generalization","feed_subtitle":"A benchmark-disjoint approach produces harness updates that transfer across tasks and foundation models.","key_machinery":"The central mechanism is the combination of a five-module decomposition of the agent harness (Agent Loop, Tool Use, Observation Management, Context Management, Task Completion Detection), contrastive trajectory analysis that pairs successful and failed rollouts of the same task to identify reusable behavioral deficiencies, independent module-wise evolution with restricted modification scope, and cross-module integration with validation gates (program checks, diff review, execution validation) that reject task-specific overfitting.","core_discovery":"ModularRSI establishes that harness-level recursive self-improvement can generalize beyond the specific tasks used during evolution when the improvement process is decomposed into independent module-level changes guided by contrastive trajectory pairs. The key empirical result is that a harness evolved on TB-related tasks improves SWE-Bench Verified accuracy from 73.40 to 75.80 (out-of-domain transfer), and a harness evolved on SWE-related tasks improves TerminalBench 2.0 from 47.57 to 49.40. Additionally, the improvements transfer across different foundation models: a harness evolved with one model also boosts accuracy on two other models. The paper argues that modularity prevents cross-mechanism interference, contrastive analysis identifies generalizable deficiencies, and the benchmark-disjoint protocol ensures that gains reflect genuine capability improvements rather than benchmark-specific adaptation.","pith_inferences":["If the contrastive trajectory analysis reliably identifies generalizable deficiencies, the same approach could be extended to other agent-environment interaction domains, such as robotics or game-playing, by defining analogous modular decompositions.","The paper does not fully ablate the contribution of contrastive analysis; a stronger test would compare against random pairing of trajectories or single-trajectory analysis with the same total evidence, which could reveal whether the contrastive mechanism is necessary for the observed gains.","The evolution uses only 240 instances out of the 2000 curated; scaling to the full dataset might yield further gains or reveal diminishing returns, and future work should explore the effect of evolution-set size."],"forward_implications":["Harness self-improvement can be conducted on a curated, benchmark-disjoint dataset rather than on task-specific evaluation data, making the process more scalable and less prone to overfitting.","The modular evolution approach allows different aspects of harness behavior to be improved in parallel and then combined, potentially reducing the computational cost of full-harness evolution.","The cross-model transfer of improvements suggests that some harness mechanisms are model-agnostic, meaning a harness optimized for one LLM can be reused with others without re-evolution.","The difficulty distribution of evolution tasks matters: moderately difficult tasks provide more informative contrastive signals than mixtures of very easy and very hard tasks, guiding dataset construction for future self-improvement systems."],"supporting_citations":[{"why":"Provides the Terminus-2 harness used as the starting point for modular decomposition and evolution.","marker":"Merrill et al., 2026b"},{"why":"Supplies the SWE-Bench Verified benchmark used to evaluate cross-domain generalization.","marker":"Jimenez et al., 2024b"},{"why":"Implements the AHE method that serves as a baseline for comparison under the same benchmark-disjoint protocol.","marker":"Lin et al., 2026"},{"why":"Implements the Meta-Harness method used as another baseline for comparison.","marker":"Lee et al., 2026"},{"why":"Defines the TerminalBench family used to guide evolution-task category construction and as an evaluation benchmark.","marker":"Merrill et al., 2026a"}],"fun_headline_variants":["ModularRSI contrasts trajectories to evolve generalizable harnesses","Benchmark-disjoint modular evolution transfers across tasks and models","Separate modules plus contrastive pairs yield transferable harness gains","Contrastive modular evolution boosts out-of-domain agent performance"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The framework assumes that the LLM acting as diagnostician and editor can reliably distinguish generalizable harness deficiencies from task-specific patterns using contrastive trajectory analysis, and that the validation gates (self-review, two-task execution test) catch overfitting without introducing false rejections.","fun_headline_variants_meta":{"raw":{"variants":["ModularRSI contrasts trajectories to evolve generalizable harnesses","Benchmark-disjoint modular evolution transfers across tasks and models","Separate modules plus contrastive pairs yield transferable harness gains","Contrastive modular evolution boosts out-of-domain agent performance"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"cost":0.000580328,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000580328,"upstream_inference_prompt_cost":0.000107128,"upstream_inference_completions_cost":0.0004732},"total_tokens":2660,"prompt_tokens":970,"completion_tokens":1690,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1620}},"tokens_in":970,"tokens_out":1690,"cost_source":"openrouter_usage","duration_ms":16216,"image_pages":[],"temperature":1.0,"reasoning_tokens":1620,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-15T13:10:11.418215+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If the evolved harness were evaluated on a new set of tasks from a third domain (e.g., data science workflows) and failed to show improvement over the baseline, that would contradict the claim of cross-domain generalizability. More specifically, if the accuracy gain on a held-out benchmark disappeared when the benchmark's task distribution was balanced to exclude any tasks similar to the evolution set, that would indicate the improvements are not general.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}