{"id":"e62f27f6-296f-4592-8fd6-f846c9494833","arxiv_id":"2608.10178","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Self-evolved coding-agent harnesses act as legible compensation layers: they close the gap between what a model can do and what it does, share a portable playbook across languages, and still require native re-evolution for language-specific ecosystem details.","lead":"This paper runs one fixed self-evolution recipe across eight programming languages and three base models, and shows that the resulting agent harnesses mostly fix simple execution mistakes, not deeper reasoning gaps. It also shows which parts of an evolved harness transfer to other languages and which parts must be re-learned from scratch.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 88% recovery statistic counts any test-file edit, not only edits that blocked grading; if many were incidental, the main compensation evidence is overstated.","rationale":"The paper is unusually honest: it reports limitations, a planted-defect probe, a meta-agent ablation, and per-cell numbers, and the held-out protocol plus contract manifests are real evidence. My concern is not about missing coverage in general—the paper explicitly admits the detectors cover only part of the recoverable space—but about the precision of the detector used for the headline statistic. The 88% figure depends on classifying any test-path touch as a recoverable defect, whereas the paper's own causal story requires that the edit actually blocks grading (zero tests captured). The authors have the precise artifact-based signal in Sec. A.2 (patch_broke_tests / broke_test_build) and use it in the worked example of Sec. A.6, so the check is a re-analysis of existing logs rather than new experiments. If the stricter signal reproduces the 88%, the compensation mechanism is substantially supported; if not, the correlation and the null-region explanations need re-examination. This is an addressable verification condition, so it does not change the reader's conditional verdict; it sharpens the condition that should be met before the central claim is accepted.","tokens_in":27773,"tokens_out":10227,"duration_ms":107539,"concrete_test":"Recompute the D.1 classification using the deterministic patch_broke_tests bucket (submitted patch touches a test path AND grader fix stage captured zero tests) instead of the lexical test-path matcher, and additionally report how often the seed's source patch matches the reference fix among recovered instances. If the detector-covered share of recovered instances falls substantially below 88%, or if a large fraction of test-edit recoveries had runnable tests and a wrong source fix, then C2's aggregate support is not established and the paper should be reworded to claim only the demonstrated subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section D.1's headline statistic—88% of 256 recovered instances carried a detector-covered defect under the minimal seed—is computed with a lexical test-path matcher (Sec. B.4) that fires on any submitted patch touching a test path. The causal mechanism the paper actually needs is narrower: an edit is a recoverable execution defect when it collides with the hidden gold test patch so that the grader's fix stage captures zero tests (the deterministic patch_broke_tests bucket of Sec. A.2, and the worked example of Sec. A.6). The broad matcher also counts test-path edits that do not conflict with the gold patch and leave the target tests runnable; in those rollouts the seed failed because the source fix was wrong, not because of an execution defect, and a 'never modify test files' rule would not recover the instance. Table 6 and Fig. 9 do not report the intersection of test-edit with zero-tests-graded or with a source patch matching the reference, so the 81%/88% attribution conflates a true blocking defect with an incidental action. Since this statistic is the main quantitative support for C2 and also feeds the defect-mass correlation of Sec. D.2, the compensation account is weaker than presented unless the stricter causal signal is applied.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies what self-evolving coding-agent harnesses encode. The authors fix a single evolution recipe (TRIAGE), which routes every harness edit through typed failure signals and records each edit as a falsifiable contract, and run it on a grid of eight Multi-SWE-Bench languages by three base models, with 20-instance evolution splits and disjoint 50-instance held-out splits. Four findings are reported: (1) the loop improves held-out solve rates over a minimal seed and over mini-SWE-agent in most cells, with two null regions (Python across models, GPT-5-mini across languages) identified post hoc; (2) gains compensate recoverable execution defects (chiefly test-file edits that collide with the hidden gold test patch, non-compiling submissions, and unverified submissions), with detector-covered defect mass correlated with gain (pooled r=0.81) and the dominant defect varying by cell; (3) evolved harnesses converge on abstract concepts (mean off-diagonal Jaccard about 0.55) but diverge in ecosystem machinery (about 0.12), with 20-40% of each harness ecosystem-specific; (4) a distilled language-agnostic harness and cross-language transplants transfer a disciplinary core (positive in 18 of 20 pairs) but leave an ecosystem margin on Java, C++, and TypeScript (32-52% of native gain lost).","tokens_in":27975,"tokens_out":30091,"duration_ms":249510,"significance":"The potential contribution is substantial if the compensation account holds: it moves the field from aggregate 'self-evolving harnesses work' results toward a mechanistic, partly falsifiable statement of what the gains consist of, when they transfer, and when they do not, with the Python and GPT-5-mini nulls, the JS-to-TS negative transfer (-2.30), and the export/import asymmetry as the kind of structure that makes the account testable. The methodology is a genuine strength: a recipe held fixed across 24 cells, held-out splits used only once, contract-style predictions written before the next round with honest reporting that most individual predictions missed (Sec. A.7), a non-regression gate, bounded memory that forces distillation, detailed compute accounting, and negative results reported rather than suppressed. Circularity risk is low by design: gains are measured on disjoint splits, no constant is fitted to the target result, and defect mass is measured by fixed artifact-based detectors on the seed's rollouts.","major_comments":[{"comment":"The headline attribution that 88% of the 256 recovered held-out instances carried a detector-covered defect under the minimal seed is computed with the broad lexical test-edit matcher of Sec. B.4, which fires on any submitted patch that touches a test path, rather than with the causal condition defining the deterministic patch_broke_tests bucket of Sec. A.2 (fix stage captures zero tests and the patch touches a test path). A test-path edit that does not collide with the hidden gold test patch leaves the target tests runnable, so the seed then failed because the source fix was wrong, not because of an execution defect, and a 'never modify test files' rule would not recover such an instance. Table 6 and Fig. 9 report no intersection of the test-edit signal with zero-tests-graded, so the 81% (and the pooled 88%) conflates a blocking defect with an incidental action. Because this statistic is the principal quantitative support for C2 and also feeds the defect-mass correlation of Sec. D.2, the authors should re-report Table 6 against the patch_broke_tests bucket (or fix-stage-zero-tests), which the stage-level grader records of Sec. A.5 make directly computable.","section":"Sec. D.1 / Table 6 (with Sec. B.4 and Sec. A.2)"},{"comment":"The per-language defect-composition evidence for 'no single rule is the mechanism' is not self-consistent with the paper's own worked example. Sec. A.6 reports that 16 of 18 round-0 failures in the C++ times Haiku evolution split were patch_broke_tests (test-file edits colliding with the gold test patch), yet Sec. 4.3 and Fig. 3 report C++ as 70% non-compiling source and at most 10% test-file edits; if Fig. 3 averages over models, the composition should be shown per model for the flagship C++ time Haiku cell, whose +74.3 pp held-out gain is the paper's strongest result. In addition, the two detectors of Sec. B.4 are not mutually exclusive (a test-file collision also zeroes the fix stage and fires the build-break detector), so Fig. 3's 'relative mix' needs an explicit priority or exclusive-classification rule, and the 5.6% Python defect rate cited in Sec. 4.3 is a single-channel test-edit number compared against two-channel numbers for the other languages.","section":"Sec. A.6 vs Sec. 4.3 / Fig. 3"},{"comment":"The abstract and Sec. 4.3 state that gains size-tracks the defect mass a cell exhibits, supported by the pooled r=0.81 of Fig. 10, but D.2(iii) reports within-model correlations of r=0.85 (Haiku), r=0.54 (DeepSeek), and r=0.12 (GPT-5-mini), and Sec. 4.3 admits both null regions were identified post hoc. The pooled correlation is thus largely carried by the between-model separation (Python and GPT-5-mini as low-defect, low-gain axes) that motivated the mechanism, so the aggregated r is partly selection-dependent rather than an independent confirmation. The within-model statistics, or a leave-one-model-out analysis, should be presented in the main text where the compensation claim is made.","section":"Sec. D.2 / Sec. 4.3"},{"comment":"The claim in Sec. 4.5 that the textual ecosystem-specific fraction (20-40% of codeable units) and the behavioral ecosystem margin (32-52% of native gain lost by the universal harness) are 'the same quantity, measured once in text and once in held-out solve rate' conflates a content share with a gain-retention share; overlapping ranges are suggestive but do not constitute the same measurement, so the C4 triangulation wording should be softened. In addition, the distillation evaluation is run on a single rollout model (DeepSeek-V4-Flash), and the rho greater than 1 values for Go (1.08) and JavaScript (1.13) are described as statistically indistinguishable from native without confidence intervals or a paired test.","section":"Sec. 4.5 (C4 triangulation)"}],"minor_comments":[{"comment":"The text states that every evolved harness is 20-40% ecosystem-specific, but Table 5 reports 18% for the JavaScript/Haiku cell; the band or the table should be adjusted.","section":"Sec. 4.4 / Table 5"},{"comment":"The main-text statement that Python's point estimate lies within the rollout standard deviation does not match the DeepSeek-Python row (delta = -7.1 pp, z = -1.69, starred as marginally significant); the appendix wording 'null or slightly negative' is more accurate and should be used in the main text.","section":"Sec. 4.3 / Table 4"},{"comment":"The paper states 'Code is available at Github' without a URL or repository identifier, which undermines the reproducibility that the detailed instrumentation otherwise supports.","section":"Sec. 1 (Availability)"},{"comment":"Retention rho = (u - m) / (n - m) is a ratio of small differences; for the Go (1.08) and JavaScript (1.13) cells, the claim of matching native evolution should be accompanied by confidence intervals or a paired significance test given the 50-instance held-out sets.","section":"Sec. 4.5"},{"comment":"The statement that no recovered instance was found in which the evolved harness enables a repair the base policy could not otherwise express is a strong universal negative; it should be given an operational definition of 'express' or restricted to the detected defect classes.","section":"Sec. 4.3"},{"comment":"The limitations section notes that evolution runs once per cell and that cell-specificity is consequently a lower bound on agreement; this caveat should be restated where Fig. 5 and Table 5 present per-cell percentages, since the main text otherwise invites over-reading of individual values.","section":"Sec. H / Sec. 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's framing is honest: it explicitly builds on AHE and positions its novelty in the attribution analysis rather than in the evolution algorithm, and it discloses the post-hoc identification of both null regions. The main risk to the central claim is the D.1 test-edit precision issue; if the authors' re-analysis shows the causal intersection to be much smaller than 81%, the compensation account would need to be weakened to a qualitative claim. Fit with the journal's scope is appropriate; the contribution is analytical rather than algorithmic, which is a legitimate framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the first paper in this line that tries to say what self-evolving harnesses actually encode rather than just reporting that they work. That alone makes it worth reading. The design is right: hold the evolution recipe fixed across an 8 x 3 language-model grid, route every edit through typed failure signals and falsifiable contracts, and measure held-out outcomes on disjoint splits. The instrumentation is a real step up from the aggregate-gain papers, and the limitations section is unusually honest, including a planted-defect probe that comes back negative and a clear statement that no compute-matched baseline was run. Credit is due for the transfer and distillation study too; the convergent estimates from textual, distillation, and transplantation evidence are a thoughtful way to bound the ecosystem margin.\n\nThe central compensation claim is plausible and mostly holds up: the loop installs rules like \"never edit test files\" and \"verify it compiles before submitting,\" and the worked example in A.6 shows the collision mechanism concretely. But the stress-test concern about D.1 is real. The 88% statistic counts any submitted patch touching a test path, not only edits that collided with the hidden gold test patch and zeroed the grading. The mechanism the paper actually needs is the narrower patch_broke_tests bucket. Table 6 does not report the intersection of test-edit with zero-tests-graded, so the 81% figure conflates a true blocking defect with an incidental action. If many of those test edits left the target tests runnable, the recovery attribution is inflated, and the defect-mass correlation in D.2 inherits the same overcount. This is a fixable analysis, but it is the main quantitative pillar of C2, so it should be redone before publication.\n\nOther soft spots are minor or already acknowledged: the null regions were identified post hoc, each cell is evolved only once, the splits are small (20 evolution / 50 held-out), and the code repository is not verifiable from the preprint. None of these is load-bearing; the paper reads aggregate structure rather than single cells, and it says so.\n\nWho gets value from this: anyone working on coding-agent harnesses, self-evolution, or multi-language SWE benchmarks. The mechanism account and the portability numbers are practically useful even if the 88% number needs correction. I would bring it to a reading group and I would cite it. It deserves a serious referee: the question is important, the methodology is mostly sound, and the flaws are addressable rather than fatal. Send it out, but ask the authors to tighten D.1 and release the code.","headline":"A genuinely legibility-focused study of self-evolving harnesses with real instrumentation, but the headline 88% recovery statistic overcounts incidental test edits and needs a stricter causal signal before the compensation story carries the weight the paper puts on it.","tokens_in":28558,"tokens_out":2230,"would_cite":true,"duration_ms":25880,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-evolved harnesses improve coding agents by compensating recoverable execution defects, not by raising model capability.","keywords":["self-evolving harnesses","coding agents","recoverable execution defects","harness compensation","Multi-SWE-Bench","programming language ecosystems","model capability gaps","harness transfer and distillation"],"falsifier":"A concrete check: on a held-out language sample, audit all recovered instances that carry neither detector signature—if most of them were corrected by a rule-like behavioral change, the 88% attribution undercounts; conversely, if any cell combines high measured defect mass with zero harness gain, the compensation account as measured would fail.","tokens_in":27533,"feed_emoji":"🛠️","tokens_out":9538,"duration_ms":80286,"temperature":0.7,"pith_summary":"Self-evolving harnesses—loops that let a coding agent edit its own prompts, tools, and memory from its rollouts—reliably raise evaluation scores, but earlier work only reports the aggregate gains. This paper argues the gains are legible compensation: the harness installs rules that stop a frozen base model from making recoverable execution mistakes it is already capable of avoiding, such as editing a test file that collides with the hidden gold patch, shipping code that does not compile, or submitting without running the failing test. Across eight programming languages and three base models, held-out solve rates improve in most cells, and the larger a cell's automatically measured share of such mistakes, the larger the gain ($r = 0.81$). Which mistake binds is cell-specific, so evolved harnesses share an abstract playbook but express it through almost disjoint language-specific machinery, with 20–40% of each harness bound to its ecosystem. Two flat regions—Python and one of the three models—fit the same mechanism: there the covered mistakes are rare, leaving little for a harness to install.","feed_headline":"Harness gains come from fixing agents' recoverable mistakes","feed_subtitle":"Across eight languages and three models, gains track a measurable defect mass; the ecosystem-specific part resists reuse.","key_machinery":"The instrumented evolution recipe TRIAGE (Typed Routing and Instrumented Attribution for Guided Evolution) carries the argument. It holds a frozen task policy fixed, exposes four editable harness slots—system prompt, workflow hooks, lessons memory, and tool implementations—and routes every edit through a typed failure signal, recording each edit as a falsifiable contract that names the instances it should fix and those it may regress. This makes each modification attributable after evolution: a later round tests the prediction and reverts unsupported changes. The second load-bearing object is the pair of automatic defect detectors—test-file editing and build break—that measure a cell's recoverable defect mass from artifact-only evidence; these detectors define the mechanism's observable scope.","core_discovery":"The central claim is that a self-evolved harness is a compensation layer: it closes the gap between what a frozen base policy can do and what it actually does under a bare scaffold, without raising the policy's capability ceiling. The evidence is that 88% of 256 recovered held-out instances carry a detector-covered execution defect under the minimal seed harness—81% had edited a test file and a further 7% had broken the test build—and the evolved gain per cell tracks the seed's measured defect mass (pooled $r = 0.81$). The account also explains the two null regions: Python base policies already follow the covered disciplines, and the GPT-5-mini model commits few such defects at all, leaving only localization and repair failures that no rule can address. The same compensation view explains why evolved harnesses are cell-specific—the dominant defect shifts across languages—and why portability is bounded: the shared disciplinary core transfers and distills into a universal harness, while the 20–40% ecosystem-specific machinery does not and must be re-evolved natively.","pith_inferences":["A testable extension of the compensation account: if it generalizes beyond code repair, harness evolution is a cheap substitute for model-side retraining only for defects already within a policy's behavioral repertoire, so its value should shrink as base models internalize these disciplines.","The paper's detectors leave 12% of recovered instances unattributed, and its planted-defect probe shows the loop fails to discover novel environment-specific operational facts; this suggests the mechanism is better described as disciplined recall of recurring failure modes than as open-ended discovery.","The JS→TS negative transfer isolates a general boundary: a portable harness breaks exactly where an ecosystem's toolchain silently drops a discipline (TypeScript transpiles without type-checking), predicting that similar walls exist wherever a test runner or build shim hides a required verification step.","The portability results may be partly shaped by the memory cap and instance-token filter, which force distillation; an unbounded memory slot could let harnesses accumulate ecosystem trivia and worsen both transfer and the clean two-level split."],"forward_implications":["Harness gains can be anticipated cheaply: a base policy's measured defect profile on a language is a leading indicator of where harness engineering will pay off.","A static human-designed scaffold is structurally at a disadvantage because it must hard-code one guess for which defect binds in every cell, while an evolved harness installs the cell-specific discipline.","Reuse guidance follows: the disciplinary core of an evolved harness transfers to foreign languages in most cases and distills into generic guidelines, but on ecosystem-heavy targets native re-evolution remains necessary to recover the full gain.","The gain does not come from a stronger meta-agent: a weak 20B-parameter outer-loop driver recovers the same held-out improvement, isolating the diagnostic-routing and contract machinery as the active ingredient.","In the regime where a harness helps at all, an automatically evolved one can outperform a manually designed scaffold at equal inference cost, because it names the exact commands, paths, and rules the cell's dominant defect requires."],"supporting_citations":[{"why":"Supplies the eight-language Multi-SWE-Bench benchmark and its per-language subsets that define the language axis and the evolution/holdout splits.","marker":"Zan et al. 2025"},{"why":"Provides the manual mini-SWE-agent scaffold used as the baseline and as the plugin base for the four editable harness slots.","marker":"SWE-agent team 2025"},{"why":"AHE's observability-driven backbone contributes the frozen policy, layered evidence, and later-round validation that TRIAGE retains and extends.","marker":"Lin et al. 2026b"},{"why":"Establishes the repository-level repair setting and the grader convention of applying a hidden gold test patch on top of the agent's diff, which makes test-file edits fatal.","marker":"Jimenez et al. 2024"}],"fun_headline_variants":["Evolved harness gains map to recoverable defect mass","Self-evolution compensates for undone actions, not limits","Harness fixes what agents skip, not what they can't do","Defect mass drives harness gains across languages, models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that the two automatic checks for recoverable mistakes—editing a test file and breaking the build—measure those mistakes equally well across all eight languages; a check that cannot fire in Python or that misses other correctable mistakes would distort the correlation and the null-region explanations.","fun_headline_variants_meta":{"raw":{"variants":["Evolved harness gains map to recoverable defect mass","Self-evolution compensates for undone actions, not limits","Harness fixes what agents skip, not what they can't do","Defect mass drives harness gains across languages, models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1835,"prompt_tokens":1052,"completion_tokens":783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":668,"tokens_out":783,"duration_ms":8389,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:09.950873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: on a held-out language sample, audit all recovered instances that carry neither detector signature—if most of them were corrected by a rule-like behavioral change, the 88% attribution undercounts; conversely, if any cell combines high measured defect mass with zero harness gain, the compensation account as measured would fail.","supporting_citations":[],"review_version":1}