{"id":"9ae71d4a-a34a-4521-aba3-88ab5281569f","arxiv_id":"2608.07645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Mendel Gödel Machine uses comparative evidence across tasks and lineages to edit coding agents' own scaffolds, outperforming single-trajectory baselines on SWE-bench and Polyglot.","lead":"This paper presents a self-improving coding agent framework with two new edit strategies: comparing an agent's failures across tasks, and borrowing behavior from another agent lineage that solved the same task. On coding benchmarks, the method lifts a small open model from 50.8% to 93.2% accuracy under a fixed budget, suggesting scaffold evolution can rival much larger closed models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1 is conditional on unmeasured 'sound comparative evidence'; the real-benchmark gains do not actually test the diagnostic-compression mechanism.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing premise: the causal-defect structure and the filtering of non-causal loci are assumed rather than derived. My reading reinforces that this is not merely an empirical caveat but a formal one: Proposition 1 is conditional on an unmodeled property of the editor and the failure distribution, so the theoretical claim does not establish that MGM's operators are superior under the stated generative model. The simulation's ρ > 1 setting makes the expected result visible but does not measure ρ in real agents, and the ablations show only that removing the operators hurts, not that the operators work through diagnostic compression. The paper's own Limitations section partially acknowledges this, and the reader's CONDITIONAL verdict already captures the risk. I therefore do not move the verdict; I would keep it conditional pending the empirical fix-probability measurement described in the concrete test. No objection is raised about the authors' honesty or the validity of the code-release claim; the concern is entirely about whether the central mechanism is actually evidenced.","tokens_in":51935,"tokens_out":4045,"duration_ms":46521,"concrete_test":"Use the released MGM logs from the Table 1 runs. For each of the 24 Φ-expansions, record operator type, the evidence tasks used, and whether the child resolved the parent's failed task or a later task requiring the same suspected recurring defect. Estimate p̂CM, p̂RM, and p̂CH as the fraction of edits that produce such a fix. If p̂RM and p̂CH exceed p̂CM by a margin comparable to the ρ = 1.2–2.0 sweep used in the simulation, the diagnostic-compression mechanism is supported; if the ratios are near 1 while the Table 1 gains persist, the gains must be attributed to operator-induced exploration or selection effects rather than to the proposed mechanism. As a complementary check, rerun MGM and HGM on 10 randomly drawn 60-task subsets and report seed-level accuracy distributions to rule out subset and seed luck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central formal claim, Proposition 1, is not derived from the stated additive model; it is an assumption presented as a premise. In Appendix B.1, pRM > pCM requires that two failed trajectories 'share a recurring causal defect b' (Eq. 38), and pCH > pCM requires that the contrastive candidate set CCH = (Rτ ∩ M(at)) ∪ N removes enough non-causal loci (Eq. 48). Both requirements concern properties of the unobserved genotype that are not implied by the observation model in Eqs. (20)–(23): two failures of the same agent can be caused by disjoint incorrect loci, and a successful reference trajectory does not by itself identify which task-relevant loci in the target are defective. The model only tells the operator that Rτ ∩ M(a) ≠ ∅; it does not provide the intersection structure or the reference genotype. Thus Proposition 1 is a conditional statement about assumed 'sound comparative evidence,' not a theorem about the operators under the generative model. The Monte Carlo simulation cannot rescue this: it injects the conclusion by setting ρ > 1, and the ρ = 1 null case merely shows that without the assumption MGM is not better. The real-benchmark gains in Table 1 are consistent with the proposed mechanism but do not test it; they could also arise because ΦRM/ΦCH add exploration or diversity that HGM lacks, or from subset and seed luck. The paper honestly lists this premise among assumptions in B.1, but acknowledging an assumption does not supply evidence that real coding failures satisfy it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces the Mendel Gödel Machine (MGM), an archive-based self-improving coding-agent framework that extends the HGM/DGM lineage by replacing the single self-modification operator with three evidence-conditioned operators: clonal mutation (single failed trajectory), reaction-norm mutation (multiple trajectories of one agent across tasks), and cross-lineage hybridization (trajectories of different lineages on a shared task). The authors develop an additive binary-locus fitness model in which tasks probe random k-locus subsets, and state Proposition 1 asserting that, under 'sound comparative evidence,' reaction-norm mutation and cross-lineage hybridization have higher effective fix probability than clonal mutation. They then run Monte Carlo simulations in which the fix-probability advantage enters as an input parameter ρ, and they report empirical results on SWE-bench Verified-60, Polyglot-60, SWE-bench Pro, SWE-bench Multilingual, and cross-model transfer, together with a two-way ablation. The headline empirical claim is that, under a matched 200-evaluation budget, MGM reaches 78.3% on SWE-bench Verified-60 and 93.2% on Polyglot-60 versus 73.3% and 77.9% for HGM, and that removing either comparative operator degrades performance. The manuscript includes a detailed limitations section acknowledging conditional assumptions, single-run primary evolution, and residual subset/seed uncertainty.","tokens_in":52331,"tokens_out":8999,"duration_ms":94188,"significance":"If the central claims are established, the paper addresses a genuinely under-explored bottleneck in self-improving coding agents: the quality of the evidence used at each self-modification step rather than just the archive sampling policy. The operator design is clean and the idea of using archived trajectories as controlled comparisons is plausible and independent of the specific LLM backbone. The paper also ships concrete assets: code and project pages are promised; the experiments use a matched budget; the ablation includes both comparative operators; and the full Polyglot-225 evaluation provides a useful check against subset artifacts. The theoretical contribution is best read as a conditional analysis of when comparative evidence helps, and the authors are unusually candid about the assumptions (sparse defects, recurring genotype-level weaknesses, informative reference trajectories) and about the fact that primary evolution is reported on a single run. Those strengths do not, however, remove the need to test the comparative-evidence premise directly or to provide variance evidence for the empirical ordering.","major_comments":[{"comment":"Proposition 1 is not a theorem about the operators under the generative model of Eqs. (20)–(23); it is a conditional statement that assumes a recurring genotype-level defect b lies in Rτt∩Rτr and that the reference comparison isolates the target-specific causal set, CCH=(Rτ∩M(at))∪N with |CCH|<k. Two failed trajectories under the model are consistent with disjoint causal loci, and a successful reference trajectory only guarantees Rτ∩M(ar)=∅; neither observation by itself identifies the target's defective loci. The derivation inAppendix B.1 is internally valid as a conditional proof, but the abstract and Section 1 overstate the result by claiming a proof of faster convergence, since the existence of 'sound comparative evidence' is an extra empirical premise rather than a consequence of the additive landscape. I recommend restating Proposition 1 as an explicit conditional theorem and providing a direct or proxy test of the sparse-defect and recurrence assumptions.","section":"§4.2 / Appendix B.1, Eqs. (38) and (48)"},{"comment":"The Monte Carlo simulation takes the diagnostic advantage ρ=pRM_f/pCM_f=pCH_f/pCM_f as an input, so it cannot independently validate Proposition 1 or measure whether real coding failures satisfy its premise. The null case ρ=1 demonstrates only that the assumed advantage is load-bearing, and the real-benchmark gains in Table 1 are equally consistent with ΦRM and ΦCH acting as extra exploration or diversity operators rather than as diagnostic compression. To support the causal mechanism, the paper should either measure or approximate candidate-set density from actual trajectories (for example, by comparing edited loci against oracle-relevant loci in a controlled setting, or by ablating against an operator that adds diversity without comparative conditioning) or explicitly label the simulation as a consequence analysis under the assumed advantage.","section":"§4.3, Eq. (26)"},{"comment":"The headline empirical comparisons are all single evolution runs on fixed 60-task subsets, with no independent seeds, confidence intervals, or statistical tests. Given that the paper claims 'consistent improvement' across benchmarks, ablations, and transfer settings, the single-run status is load-bearing: Appendix E.2 shows that evolution trees contain high-utility nodes evaluated only a handful of times, so the procedure for selecting the 'best-belief agent' is noise-sensitive. I ask for at least a small number of independent evolution seeds, or a variance-sensitive analysis such as bootstrapping over the 60-task subset and over Thompson-sampling randomness, to establish that the HGM–MGM gap and the ablation ordering are stable rather than artifacts of one realization.","section":"§5, Tables 1–4; Limitations"},{"comment":"The formal result actually obtained is a comparison of per-edit effective fix probabilities, pRM_f>pCM_f and pCH_f>pCM_f, not a theorem about expected convergence speed under the archive-search process. The claim of 'strictly faster expected convergence' is demonstrated only by simulation with ρ as an input. The manuscript should either add a theorem linking higher per-edit fix probability to faster convergence under the HGM evaluation and expansion policies, or soften the abstract and Section 1 to state that the model and simulation provide a conditional argument rather than a proof of convergence.","section":"Abstract / §1 / §4.2"}],"minor_comments":[{"comment":"With fixed-size task subsets |Rτ|=k, the per-task success probability at edit distance d is hypergeometric, C(L−d,k)/C(L,k), rather than ((L−d)/L)^k; the current formula corresponds to independent per-locus inclusion. Please correct the expression or state the sampling convention explicitly.","section":"§4.1, Eq. (23)"},{"comment":"The experimental hyperparameter is βfail=1.0, which makes the failed-pool boost exactly neutral; the text's claim that the failed-task pool 'deliberately creates overlap across lineages' is therefore not exercised in the reported experiments. Please clarify that the pool is inert in this configuration and that overlap comes from the common task distribution.","section":"§3.2, Eq. (17) and Table 6"},{"comment":"The full Polyglot-225 evaluation reports 93.3% (210/225), while Table 1 reports 93.2% on Polyglot-60, and Figure 2 plots an asterisked HGM point from Polyglot-60 alongside full-benchmark points; the exact mapping between subset and full-benchmark numbers should be stated so the figure cannot be misread as comparing different scales.","section":"Appendix E.1 / Figure 2"},{"comment":"The 60-task subsets for SWE-bench Pro and SWE-bench Multilingual are selected via 'ChatGPT' without reporting the selection prompt, sampling temperature, or a seed; since the headlined gains inherit sensitivity to subset composition, please include the full selection protocol and a reproducibility statement for these subsets.","section":"Appendix D"},{"comment":"The notation such as '78.3+10.0' is intended to show absolute percentage-point improvement over the initial agent, but as rendered it can be misread as a value plus an error or confidence interval; a separate delta column would be clearer.","section":"Tables 1–4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually honest: the Limitations section already concedes single-run primary evolution, subset sensitivity, and the conditional nature of the theory. My main concern is that the abstract and introduction market a 'proof' and a 'validation' where the proof is conditional and the simulation inputs the conclusion. If the authors reframe the theoretical contribution as a conditional theorem, add stability evidence for the empirical ordering, and provide at least one direct or proxy probe of the comparative-evidence premise, I would be willing to accept a later version. The paper is a reasonable fit for this venue, and the single-run status, while serious, is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper probably does what it claims — two new comparative self-modification operators that improve coding-agent scaffold evolution under a matched budget — and it is honest about the gap between its theory and its evidence. The two operators, reaction-norm mutation and cross-lineage hybridization, are genuinely new relative to the DGM/HGM line, and the framing of self-modification as diagnostic compression over candidate loci is a useful way to think about the problem. The empirical section is well controlled: same ancestor, same 200-evaluation budget, same number of expansions, comparable token costs, and ablations show both operators matter. Cross-benchmark and cross-model transfer results are a strong plus.\n\nThe soft spots are real but not fatal. Proposition 1 is a conditional statement: it holds only under 'sound comparative evidence,' meaning repeated failures share a recurring defect or a successful reference filters out non-causal loci. That is acknowledged in the appendix. The surrogate simulation does not close the gap — it takes the fix-probability advantage rho as an input, so it tests the downstream dynamics of an assumed diagnostic benefit, not whether the operators actually earn that benefit in real coding failures. The benchmark gains are consistent with the diagnostic-compression story, but they could also come from added exploration or diversity that the operators introduce. The main results are single runs on fixed 60-task subsets chosen by ChatGPT, so variance is unknown. None of these issues sink the paper; they are addressable with multi-seed runs, random subset replication, and a direct measurement of comparative fix probability from archived trajectories.\n\nIf I were editing, I'd send this to peer review. The work is clear, the claims are scoped, and the contribution is useful to anyone building self-improving coding agents. I'd ask reviewers to demand multi-seed experiments and a more direct test of the mechanism before acceptance, but the paper deserves a serious referee.","headline":"MGM is an honest, well-scoped extension of archive-based self-improvement for coding agents; the benchmark gains look real but the mechanism isn't yet directly tested.","tokens_in":52806,"tokens_out":3247,"would_cite":true,"duration_ms":30270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that coding agents learn to improve themselves faster when each self-edit is informed by comparisons across tasks and lineages rather than by a single failure.","keywords":["self-improving agents","recursive self-modification","evolutionary computation","coding agents","reaction-norm mutation","cross-lineage hybridization","LLM agents","software engineering benchmarks"],"falsifier":"Run the surrogate or the real pipeline with tasks deliberately engineered so that failures never share a common underlying defect and no reference agent supplies a meaningful contrast; if MGM's convergence then matches the single-trajectory baseline, the sound-comparison premise is falsified. A more direct measurement is to compare the actual probability that a targeted locus is truly incorrect under each operator's candidate set; if \\($\\Pr[\\ell \\in M(a) \\mid \\ell \\sim C_{\\mathrm{RM}}]$\\) is not greater than the same probability for \\($C_{\\mathrm{CM}}$\\), Proposition 1's diagnostic-compression mechanism is not operating in practice.","tokens_in":2070,"feed_emoji":"🧬","tokens_out":3159,"duration_ms":126062,"temperature":0.7,"pith_summary":"The paper is trying to establish that a recursive self-improving coding agent improves faster and further when each self-modification is conditioned on comparative evidence from the archive of its own past attempts, instead of on a single failure trajectory. It adds two comparative operators to the usual clonal mutation: reaction-norm mutation, which edits an agent using its trajectories across several tasks, and cross-lineage hybridization, which edits an agent using a reference agent from another lineage on the same task. Under an additive fitness landscape, the paper proves that both operators have strictly higher effective fix probability than clonal mutation, and controlled surrogate simulations confirm faster expected convergence. Under a matched 200-evaluation budget, the method reaches 78.3% on SWE-bench Verified-60 and 93.2% on Polyglot-60, versus 73.3% and 77.9% for the single-trajectory baseline. This matters because comparative operators reuse trajectories already collected, so the gain costs no extra evaluations.","feed_headline":"Comparative self-edits lift coding agents to 93.2%","feed_subtitle":"Reusing failures and successes across tasks and lineages sharpens each self-edit without extra evaluations.","key_machinery":"The load-bearing object is the effective fix probability \\($p_\\sigma^f = s \\cdot \\Pr_{\\ell \\sim C_\\sigma(E)}[\\ell \\in M(a)]$\\), where \\($M(a)$\\) is the set of incorrect loci of agent \\($a$\\), \\($s$\\) is the probability that an edit repairs a locus it targets, and \\($C_\\sigma(E)$\\) is the candidate set of loci the operator infers from its evidence. Clonal mutation searches the full failed task's locus set \\($C_{\\mathrm{CM}} = R_{\\tau_t}$\\); reaction-norm mutation searches the intersection \\($C_{\\mathrm{RM}} = R_{\\tau_t} \\cap R_{\\tau_r}$\\), which is smaller when failures share a recurring defect; cross-lineage hybridization searches the contrastively filtered set \\($C_{\\mathrm{CH}} = (R_\\tau \\cap M(a_t)) \\cup N$\\), where a successful reference agent removes non-causal loci. The additive fitness landscape model represents each agent as a binary genotype, with utility as the Hamming distance to an oracle, and a task is solved only if all \\($k$\\) loci it examines are correct. Proposition 1 converts these candidate-set containments into strict inequalities on \\($p_\\sigma^f$\\), which is what the Monte Carlo surrogate then instantiates as the fix-probability ratio \\($\\rho$\\).","core_discovery":"The central discovery is that the bottleneck in recursive self-improvement is diagnostic: a self-modification operator only helps when it can localize a real scaffold-level defect. MGM treats the agent's source code as a genotype and its evaluation outcomes as phenotypes, then structures self-modification as Mendelian controlled comparisons. Clonal mutation edits from a single failed task; reaction-norm mutation asks the agent to compare its own trajectories across multiple tasks; cross-lineage hybridization asks a failing agent to compare itself with a reference agent that attempted the same task. The paper formalizes an additive fitness landscape in which each task requires several loci to be simultaneously correct, and shows that comparative evidence compresses the candidate set of defective loci: the intersection of two failures is smaller than one failure's task-relevant region, and a successful reference lineage filters out non-causal loci. Proposition 1 states that under sound comparative evidence, both comparative operators have strictly higher effective fix probability than clonal mutation. The agent is not a stronger editor; it is an editor given cleaner hypotheses.","pith_inferences":["The paper only claims results for coding-agent scaffolds, but the diagnostic-compression mechanism is generic: any archive of outcome traces for a self-modifying system could in principle use the same two comparisons, so a natural extension is to test MGM-style operators on non-coding agent loops that accumulate logged successes and failures.","The null case \\($\\rho=1$\\) in the simulations implies a practical diagnostic: track whether comparative edits actually outperform clonal edits; if the gap narrows, the archive lacks informative contrasts and task sampling should be reweighted toward shared, repeated failure modes.","A sharp testable extension would be to seed the archive with corrupted contrasts, tasks whose failures are deliberately independent, and verify that MGM's advantage over the baseline vanishes, as Proposition 1's sound-comparison premise predicts.","Because the paper's own appendix shows a larger coding-specialized backbone can evolve worse than a smaller general-reasoning one, the limiting resource may be the editor's ability to reason over comparative evidence; an interesting follow-up is to measure how operator gains scale with backbone reasoning ability."],"forward_implications":["Under the additive fitness model with sound comparative evidence, reaction-norm mutation and cross-lineage hybridization have strictly higher effective fix probability than clonal mutation, so each self-edit has a higher chance of correcting a real defect per unit budget.","Under a matched budget of 200 evaluations and 24 expansions, MGM improves SWE-bench Verified-60 from 68.3% to 78.3% and Polyglot-60 from 50.8% to 93.2%, against 73.3% and 77.9% for the single-trajectory baseline.","Evolved scaffolds generalize: a Polyglot-evolved MGM scaffold transfers zero-shot to SWE-bench Pro (26.7% vs. 13.3% for baseline) and SWE-bench Multilingual (55.0% vs. 43.3%), and a Qwen-evolved MGM scaffold transferred to DeepSeek backbones averages 70.8% on SWE-bench Verified-60 versus 65.0% for the baseline.","Ablations show both comparative operators matter: removing reaction-norm mutation drops Polyglot-60 from 93.2% to 79.7%, and removing cross-lineage hybridization drops it to 74.6%.","All three operators have comparable token costs, so the reported gains are attributed to diagnostic quality rather than to spending more compute on edits."],"supporting_citations":[{"why":"Supplies the archive-based evolution-tree framework that MGM extends, including the archive of agent variants and evaluated trajectories.","marker":"Zhang et al. (2026a)"},{"why":"Supplies the fixed-budget tree-search baseline, its sampling policies, and the matched-budget performance comparison against MGM.","marker":"Wang et al. (2026)"},{"why":"Supplies the Gödel Machine ideal of self-referential self-rewriting that gives MGM its name and goal.","marker":"Schmidhuber (2003; 2007)"},{"why":"Supplies the reaction-norm concept, one genotype expressing different phenotypes across environments, that motivates the reaction-norm mutation operator.","marker":"Woltereck (1909); Pigliucci (2001)"},{"why":"Demonstrates that a coding agent with file-editing tools can autonomously refactor its own code, establishing the feasibility of scaffold self-evolution.","marker":"Robeyns et al. (2025)"},{"why":"Supplies the SWE-bench Verified benchmark used for repository-level software engineering evaluation.","marker":"Jimenez et al. (2024)"},{"why":"Supplies the Polyglot benchmark and its leaderboard, used for the main multilingual coding and full-benchmark evaluations.","marker":"Gauthier (2024)"},{"why":"Supplies the Qwen3.6-35B-A3B backbone model on which all main evolution runs and ablation comparisons are executed.","marker":"Qwen Team (2026)"}],"fun_headline_variants":["Mendel-inspired self-edits make coding agents converge faster","Coding agents improve by comparing their own trajectories","Comparative self-modification beats single-trajectory fixes","New mutation rules speed up self-improving code agents"],"cache_read_input_tokens":54912,"weakest_assumption_plain":"The entire advantage depends on the assumption that when the same agent fails repeatedly, the failures share an underlying weakness in the agent's own code, and that watching a successful agent on the same task reveals which parts of the task were not responsible; if most failures are just unrelated accidents, the comparative edits give no extra information and the method reverts to the standard single-failure baseline.","fun_headline_variants_meta":{"raw":{"variants":["Mendel-inspired self-edits make coding agents converge faster","Coding agents improve by comparing their own trajectories","Comparative self-modification beats single-trajectory fixes","New mutation rules speed up self-improving code agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000716,"raw_usage":{"total_tokens":3208,"prompt_tokens":928,"completion_tokens":2280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2216}},"tokens_in":544,"tokens_out":2280,"duration_ms":14732,"temperature":1.0,"reasoning_tokens":2216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:27:15.411060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the surrogate or the real pipeline with tasks deliberately engineered so that failures never share a common underlying defect and no reference agent supplies a meaningful contrast; if MGM's convergence then matches the single-trajectory baseline, the sound-comparison premise is falsified. A more direct measurement is to compare the actual probability that a targeted locus is truly incorrect under each operator's candidate set; if \\($\\Pr[\\ell \\in M(a) \\mid \\ell \\sim C_{\\mathrm{RM}}]$\\) is not greater than the same probability for \\($C_{\\mathrm{CM}}$\\), Proposition 1's diagnostic-compression mechanism is not operating in practice.","supporting_citations":[],"review_version":1}