{"id":"327c1e4e-d254-4204-bd45-f52befb8209f","arxiv_id":"2607.19592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A protocol that keeps agents generic and disposable while improving a persistent curated knowledge base outperforms agent-centric self-improvement baselines on ARC, coding, and terminal benchmarks.","lead":"This paper proposes that AI agents can improve by building a shared, reusable knowledge base instead of by rewriting their own prompts or code. The authors test a multi-step knowledge-curation loop on reasoning, coding, and terminal tasks and report higher solve rates and lower cost than agent-focused self-improvement methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-knowledge generation control leaves the causal attribution to knowledge unproven","rationale":"The reader's weakest_assumption is exactly the concern I find most load-bearing: the main-loop causal attribution lacks a no-knowledge generation control. This control is essential because the protocol retries unsolved tasks each generation with fresh stochastic agents while removing solved tasks, so repeated attempts alone can produce cumulative improvement independent of the knowledge base. The transfer experiment in Section 4.4 does provide evidence that the knowledge asset has standalone value on held-out tasks: no-knowledge baselines are clearly lower than with-knowledge conditions in every donor–recipient pairing. This is real and should be credited. However, it is a different question: it tests zero-shot cross-task transfer, not whether the generation-over-generation gains on the training pool are driven primarily by knowledge. A no-knowledge loop on the same 50 tasks, with the same retry and removal schedule, is the minimal missing experiment. The paper is otherwise transparent—it documents forks, baseline deviations, selection procedures, and cost accounting, and it makes code available—so the concern is addressable rather than disqualifying. For this reason the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment: the central claim should not be asserted at full strength until this control is run.","tokens_in":29260,"tokens_out":7275,"duration_ms":68322,"concrete_test":"Run the full KSI protocol on the ARC-AGI-1 50-task split for 10 generations with knowledge curation disabled: each generation, spawn one fresh Haiku 4.5 agent per unsolved task with the same tools and prompt, but with no per-task bundle, no cross-task bundle, no forum, and no distillation; keep the same solved-task removal schedule, temperature 0.0, and cost accounting. Compare the final solve rate and cumulative cost against Table 1a. If the no-knowledge loop's solve rate is within the confidence interval of 86.7% ± 4.2 (or similarly close on Polyglot/SWE-bench Pro), then repeated retrying alone explains much of the gain and the central claim is overstated. If it is far below, the knowledge-attribution concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 (Table 1) presents solve rates after 10 generations, but there is no control condition that runs the same fresh-agent retry schedule without any knowledge base. Because solved tasks are removed from the pool and unsolved tasks are retried by a new stochastic agent each generation, cumulative chance success after 10 attempts can be nontrivial; the paper's own Table 4 shows single-generation no-knowledge solve rates of 13–23% on held-out ARC-AGI-1 tasks. If per-attempt success probabilities on the 50-task self-improvement pool are similar, repeated retrying alone could explain a substantial fraction of the reported 86.7% solve rate. The paper asserts in Section 3 that 'any performance gain across generations must come from the accumulated and distilled knowledge,' but this is not established by a control. The held-out transfer experiment (Section 4.4) is a valuable partial substitute: it isolates knowledge value on new tasks at zero-shot inference time. However, it does not measure how much of the main-loop generation-over-generation gain on the training tasks comes from knowledge versus retrying. This gap is load-bearing because the central claim—that progress is driven primarily by curated persistent knowledge—relies on precisely this attribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes knowledge-centric self-improvement: instead of improving the agent, the system maintains a persistent, curated knowledge base that fresh, stateless agents read from and write to. A three-stage protocol (task-level forums, cross-task forums, and distillation) converts agent attempts into typed, evidence-grounded bundles. The authors evaluate this protocol on ARC-AGI-1/2, Polyglot, SWE-bench Pro, and Terminal-Bench 2, comparing against agent-centric self-improvement baselines (DGM, HyperAgents, Meta-Harness) and prompt-optimization baselines (GEPA, OpenEvolve). They report higher or competitive solve rates at lower dollar cost, cross-LLM effectiveness (Haiku 4.5 and GPT-5.4-mini), and positive transfer of a frozen knowledge asset to held-out tasks and to a different LLM family. The central claim is that progress in self-improving agentic systems can be driven primarily by the curated persistent knowledge rather than by the agent design.","tokens_in":29633,"tokens_out":3651,"duration_ms":36568,"significance":"If the central claim is sustained, the paper makes a useful conceptual and practical contribution: it identifies a design axis (what persists) that is complementary to the usual agent-centric optimization, and it provides a cheap, inspectable, and transferable improvement artifact. The paper is unusually transparent: it ships detailed protocol schemas (Appendix E), full task maps (Appendix J), cost accounting (Appendix G), baseline provenance/deviation documentation (Appendix H), and worked examples of disagreement-driven curation (Appendix C). The held-out transfer experiment is a genuine out-of-sample test with a frozen knowledge asset and disjoint task splits, and the cross-LLM transfer is a concrete falsifiable prediction. These strengths are real. However, the evidence currently underdetermines the central attribution claim because the main self-improvement loop lacks a no-knowledge control, and the baseline comparisons are against modified forks with information-parity gates rather than the original published systems.","major_comments":[{"comment":"The paper's central attribution claim—\"any performance gain across generations must come from the accumulated and distilled knowledge\" (§3)—is not established by the reported experiments. Section 4.1 runs 10 generations on a 50-task pool, but there is no control condition that repeats the same fresh-agent retry schedule without any knowledge base. Because solved tasks are removed and unsolved tasks are retried by new stochastic agents, cumulative chance success across 10 attempts can be substantial; the paper's own Table 4 shows single-generation no-knowledge solve rates of 13–23% on held-out ARC-AGI-1 tasks. A no-knowledge generation-loop control (same number of generations, same per-task budget, no forums, no distillation, no knowledge injection) on the same 50-task pool is necessary to separate retrying/stochastic effects from knowledge-driven gains. The held-out transfer experiment (","section":"§3, §4.1, Table 1"},{"comment":"Table 1a compares against DGM and HyperAgents, but the reported baseline numbers come from forks with substantial modifications: LLM substitution, information-parity gates that remove gold answers and hidden tests from the self-improvement loop, egress isolation, and newly authored benchmark adapters. The claim that \"all baselines are rerun under our evaluation protocol\" is transparent, but the claim in the abstract and Section 4.1 that the method \"outperforms agent-centric baselines\" should be qualified as outperforming the authors' modified, information-parity-gated re-runs, not the original published systems. Whether the parity gates preserve the \"core optimization algorithm\" is an assumption, not a demonstrated fact; for example, DGM's diagnosis loop is described in Appendix H as no longer seeing gold patches and hidden tests, which may be integral to its original self-modification m","section":"§4.1, Appendix H"},{"comment":"The transfer evidence is weakened by post-hoc design choices that are disclosed but not tested. First, the task-conditioned adapter's item bounds were relaxed after observing benchmark variation (Appendix L: \"we relaxed the constraints and allowed the agent to dynamically determine how much knowledge to transfer\"), making the adapter a tuned component rather than a fixed property of the frozen knowledge asset. Second, the transfer evaluation set was selected from tasks that both recipient LLMs fail at seed 0 (Appendix J), which inflates headroom and makes the no-knowledge baseline low by construction; this is disclosed, but it means the transfer numbers are conditional on a selection procedure that depends on the recipients' no-knowledge abilities. Third, for Polyglot the transfer split is task-identifier-disjoint but not exercise-disjoint: 7 of 20 evaluation tasks are the same Exercism","section":"§4.4, Appendix L, Appendix J"}],"minor_comments":[{"comment":"Typographical: \"in thisknowledge-centric self-improvementparadigm\" is missing spaces around \"knowledge-centric\" and \"paradigm.\"","section":"§1"},{"comment":"The table header \"OURShaiku 4.5\" and \"OURSgpt 5.4−mini\" are visually cramped; please add spacing or a line break for readability.","section":"Table 1/Table 3"},{"comment":"The example labels contain apparent typos (\"Count-Dataset-T okens\", \"T ask-level Insights\"). Please proofread figure text.","section":"Figure 3b"},{"comment":"The caption for Table 4 says \"20 held-out tasks\" but the selection procedure in Appendix J says tasks were chosen from the intersection of both recipients' failure sets; please make the selection criterion explicit in the main text so readers do not interpret the held-out set as a random sample of all held-out tasks.","section":"§4.4"},{"comment":"The comparison with GEPA and OpenEvolve uses matched dollar budgets but different iteration counts; it would be helpful to state explicitly that the prompt-optimization baselines were not extended to other benchmarks for budget reasons, and to note that this comparison is therefore narrower than the agent-centric comparison.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The missing no-knowledge generation-loop control is the main substantive blocker; I would request it as a condition of acceptance. The paper is well-written and unusually transparent, and the held-out transfer experiment is a genuine partial substitute, but the central attribution claim is currently stronger than the evidence supports. The baseline-forking issue is also important for how the results are framed in the abstract and introduction. I do not see evidence of circularity or fabrication; the concerns are about control conditions and generalizability of comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, readable paper with a genuinely different mechanism — evidence-adjudicated forums and typed distillation as the persistent substrate instead of agent memory — and the cross-LLM transfer results are the strongest thing in it. But the central claim, that generation-over-generation gains come from the curated knowledge, is not actually established by the main-loop experiments.\n\nWhat's new: prior memory and experience-reuse systems store and retrieve trajectories; this protocol turns attempts into scoped, contradictory, evidence-grounded claims, exposes them to a cross-task forum where peers must take agree/disagree/synthesize stances, and then distills what survives into typed bundles. That is a real mechanism, distinct from Voyager-style skill libraries or EXPEL-style lessons. The authors also deserve credit for shipping code and for an unusually transparent appendix: they document baseline forks, cost accounting, task maps, and even admit where their transfer split overlaps (7/20 Polyglot eval tasks share an exercise in a different language).\n\nSoft spots, in proportion. The main one is what the stress test flagged: Section 4.1 has no no-knowledge control that retries unsolved tasks for the same number of generations. Solved tasks leave the pool, fresh stochastic agents retry the rest, and Table 4 shows single-generation no-knowledge solve rates of 13–23% on held-out ARC tasks. Ten generations of retrying can plausibly explain a good chunk of the reported 86.7% solve rate on the 50-task pool. The assertion in Section 3 that gains must come from knowledge is doing work it can't support. The held-out transfer experiment partially compensates: at zero-shot, with no forum or distillation, the frozen bundle genuinely improves solve rates over the N/A condition, and that is clean evidence that knowledge has standalone value. But it doesn't rescue the attribution claim for the main loop.\n\nThe baseline comparisons are the second soft spot: reruns on modified forks with information-parity gates and LLM substitution, not the original published systems. The authors are honest about this in Appendix H, and the gates make sense methodologically, but the Table 1 numbers aren't head-to-head with the papers as published. Terminal-Bench uses leaderboard scores from the Meta-Harness paper rather than reruns — weaker, but clearly labeled.\n\nMinor: the transfer adapter's item bounds were relaxed after observing benchmark variation — a post-hoc tuning choice, acknowledged, but worth flagging.\n\nThe paper deserves a serious referee. The mechanism is novel enough, the transparency is a model, and the missing control is addressable. A no-knowledge retry ablation plus one original-baseline comparison would largely resolve my concerns. If I were editing, I'd send it out.","headline":"A plausible, transparent, and genuinely different mechanism for self-improvement via curated knowledge, but the main-loop causal claim is not yet supported — the held-out transfer results are the cleanest evidence in the paper.","tokens_in":30026,"tokens_out":2892,"would_cite":true,"duration_ms":26024,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-improvement can be driven primarily by a curated knowledge base rather than by the agent itself.","keywords":["knowledge-centric self-improvement","knowledge base curation","multi-agent forums","knowledge distillation","disposable agents","zero-shot transfer","LLM generalization","evidence-grounded claims"],"falsifier":"Run the same 10-generation loop with a no-knowledge control: fresh agents with identical prompts, tools, and retry budget but an empty or sham seed knowledge base. If this control closes the gap with the knowledge-seeded solve rates on any benchmark, the claimed attribution to curated knowledge fails. On the transfer side, replacing the frozen distilled bundle with task-irrelevant text should recover the no-knowledge baseline if the bundle is the true carrier of value.","tokens_in":29224,"feed_emoji":"📚","tokens_out":9293,"duration_ms":75610,"temperature":0.7,"pith_summary":"The paper tries to establish that the object of self-improvement in an AI system can be a shared, curated knowledge base instead of the agent. Its protocol keeps agents generic, stateless, and disposable: each reads distilled guidance, attempts one task, and writes evidence-grounded claims into task-level and cross-task forums, which are then distilled back into the knowledge base. Because every agent starts from a fresh context, any gain across generations is attributed to the knowledge they received. Across abstract-reasoning, coding, and terminal benchmarks, the paper reports higher solve rates and lower dollar costs than agent-centric self-improvement baselines, and it shows the distilled knowledge transfers to held-out tasks and across LLM families. This would move the bottleneck in autonomous reasoning from agent architecture to the quality and structure of the curated information agents consume.","feed_headline":"Knowledge, not agent upgrades, drives self-improving AI","feed_subtitle":"Curated knowledge from disposable agents beats agent-centric baselines—for less money.","key_machinery":"The central object is the curated knowledge base shaped by a three-stage protocol: task-level forums, where agents post evidence-grounded claims about what worked and failed; cross-task forums, where claims must take explicit agree, disagree, or synthesize stances toward cited posts and survive only when grounded in concrete evidence; and distillation, which compresses the surviving claims into typed, actionable bundles—'when X, do Y'—that the next generation of fresh agents consumes. The mechanism isolates knowledge as the only changing variable: every agent is re-instantiated with fresh context and no private memory, so improvement across generations is attributed to the knowledge base.","core_discovery":"The paper's central claim is that a system can improve at tasks without ever improving its agents, and that the persistent improving object should be a curated knowledge base. Agents are kept generic, stateless, and disposable: each one starts with a clean context, receives a distilled knowledge bundle, attempts a task, and contributes evidence back. A three-stage protocol consisting of task-level forum discussion, cross-task forum debate, and distillation converts individual attempts into scoped, evidence-grounded guidance. On the evaluated benchmarks, the paper reports that this protocol outperforms agent-centric self-improvement baselines in solve rate while using less money, that the dis","pith_inferences":["A direct extension of the paper's design is a no-knowledge retry control that re-attempts unsolved tasks for the same number of generations with empty seed knowledge; the paper does not report one, so the size of the pure retrying component is untested.","If the knowledge-centric paradigm scales, the cost of self-improvement shifts from inference-time agent complexity to offline curation, so knowledge can be amortized across many tasks and models.","The protocol's practice of preserving unresolved disagreements as falsified-versus-untried hypotheses suggests a general anti-premature-consensus mechanism for multi-agent systems.","A natural extension, not studied in the paper, is to fine-tune a base model on the distilled bundles, turning the reusable knowledge artifact into a direct weight update."],"forward_implications":["Self-improvement no longer requires a persistent, specialized agent; generic disposable agents suffice, shifting the bottleneck from agent architecture to the curated knowledge they consume.","Solve-rate gains and lower dollar costs on the evaluated benchmarks imply self-improvement can be made cheaper to operate and easier to maintain.","Because distilled knowledge transfers across LLM families, knowledge produced by one model can be consumed by another, making the improvement portable.","Frozen knowledge improves zero-shot performance on held-out tasks, so the curated asset has standalone value after the run ends.","Disagreement in forums is preserved as evidence rather than resolved, which may prevent premature convergence on wrong hypotheses."],"fun_headline_variants":["Improve the knowledge, not the agent","AI learns by building knowledge, not upgrading agents","Disposable agents, lasting knowledge: cheaper AI improvement","Knowledge base, not agent tweaks, boosts AI solve rates","Self-improving AI? Keep agents dumb, grow the knowledge"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the assumption that every generation-over-generation gain comes from the knowledge base, because agents are re-instantiated with fresh context; the paper includes no control that retries unsolved tasks for the same number of generations without seed knowledge, so stochastic retrying alone could account for part of the improvement.","fun_headline_variants_meta":{"raw":{"variants":["Improve the knowledge, not the agent","AI learns by building knowledge, not upgrading agents","Disposable agents, lasting knowledge: cheaper AI improvement","Knowledge base, not agent tweaks, boosts AI solve rates","Self-improving AI? Keep agents dumb, grow the knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1049,"prompt_tokens":743,"completion_tokens":306,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":487,"tokens_out":306,"duration_ms":3470,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:17:33.813212+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 10-generation loop with a no-knowledge control: fresh agents with identical prompts, tools, and retry budget but an empty or sham seed knowledge base. If this control closes the gap with the knowledge-seeded solve rates on any benchmark, the claimed attribution to curated knowledge fails. On the transfer side, replacing the frozen distilled bundle with task-irrelevant text should recover the no-knowledge baseline if the bundle is the true carrier of value.","supporting_citations":[],"review_version":1}