{"id":"cd0e074b-0a4a-4ab0-ba37-5f095cf92e22","arxiv_id":"2609.31186","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces Evolutionary Safety, a taxonomy and evaluation framework for how safety properties persist, erode, and propagate as AI systems recursively self-improve.","lead":"This paper proposes a framework for thinking about AI safety when AI systems improve themselves over time, rather than checking safety only at a single moment. It organizes known failure modes and suggests how to evaluate and govern systems that remember, reuse, and pass on their own changes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Eq. (4)'s frozen-updater baseline is ill-defined when the updater is itself revised, so the central measurement claim lacks a well-defined reference in the recursive case.","rationale":"I read the paper as a conceptual and taxonomic contribution: there are no experiments, no code, and the numerical example in Eq. (5) is explicitly labeled illustrative. The taxonomy, the manifestation list, and the governance principles are broadly and honestly tied to cited empirical work, and the authors repeatedly flag open problems including changing judges, latent risks, and unresolved independence requirements. The strongest part of the paper is the organization of risk carriers and the concrete comparisons in Table 1. The load-bearing weakness is the one the reader identified: the formal state/loop equations in Eqs. (1)-(2) treat the evaluator and updater as external selectors, while Section 3.5 argues that recursive development involves changing those selectors from within. Equation (4) then compares an evolving regime with a frozen-updater baseline, but the recursive case lacks a well-defined freeze intervention unless the external authority is made structural. The paper acknowledges much of this difficulty in Sections 6 and 7, which is why this is an addressable condition rather than a fatal flaw. Because the concern is already reflected in the reader's CONDITIONAL verdict and is not a reason to reject the paper, I leave the verdict unchanged. The secondary concern about asserted rather than demonstrated novelty over existing taxonomies is real but less load-bearing, since the paper's contribution is primarily a framing and measurement scheme rather than a new empirical result.","tokens_in":26764,"tokens_out":4308,"duration_ms":31377,"concrete_test":"Instantiate a minimal recursive self-improvement loop, for example a two-step agent that can write to memory and edit its own selection rule, and compute Gamma from Eq. (4) under three distinct definitions of the frozen-updater baseline: (a) U is held at its initial value, (b) U is held at its value just before a proposed revision is committed, and (c) U is frozen by a structural external-authority block that no candidate can write to. If the three Gamma values differ materially, Eq. (4) is under-specified for recursive self-improvement and the paper must either extend the state vector and dynamics to cover updater revision or explicitly restrict Eq. (4) to non-recursive adaptation. If the three values agree, the reader's concern is resolved and the estimand has a stable reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative core of the paper is Eq. (4), which defines the evolutionary effect as the difference between an evolving regime and a frozen-updater baseline: Gamma = E[g(Z^{pi_evo}_{t+h}) - g(Z^{pi_freeze}_{t+h})]. This estimand is the only concrete handle on the paper's central claims about persistence, propagation, and accumulation. But the paper's own definition of recursive development, Section 3.5, makes the updater and evaluator part of what changes, while the state loop in Eq. (2) keeps J_t and U_t outside S_t. Those two commitments are not jointly formalized. In particular, the intervention 'freeze the updater' is not well-defined when the system can propose and commit changes to its own update procedure: the system may revise the very switch that is supposed to be frozen. Different natural readings of the baseline (U held at its initial value, U held at its pre-revision value, or U frozen by an external authority) can produce different values of Gamma. The paper acknowledges part of this in Section 7, noting that the evaluator can change and that the required degree of independence between generator, actor, and judge remains unresolved, and Section 6 introduces external authorization as a governance requirement. But Eq. (4) is not conditioned on that governance machinery, so it is not merely incomplete: as written, the causal reference for the recursive case is under-specified. This is an internal-consistency issue rather than a disagreement with current consensus, and it lands directly on the paper's central measurement claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces 'Evolutionary Safety' as a conceptual framework for studying how safety properties are preserved, weakened, or restored as AI systems persist, adapt, and recursively self-improve. It defines the object of study (Definition 1), sketches a state-loop formalism in Eqs. (1)-(2), catalogs six recurring manifestations of safety degradation (intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, risk inheritance), and organizes these across five carrier domains: persistent agent state, model state, evaluation/environmental feedback, computational substrate, and meta-level update mechanisms with co-evolution. It then proposes evaluation units (states, updates, trajectories, lineages), two quantitative instruments (the update-level comparison in Eq. (3), the causal estimand in Eq. (4), and a selection-induced risk shift in Eq. (5)), and derives governance principles (modification boundaries, pre-commit gating, independent verification, lineage traceability, recovery). The numerical illustration in Eq. (5) is explicitly marked as illustrative, and the paper closes with an honest list of open problems. The central contribution is organizational and conceptual rather than empirical or theorem-based.","tokens_in":27107,"tokens_out":5687,"duration_ms":41674,"significance":"If the framework holds, it provides a useful common language for a scattered literature: the paper connects memory poisoning, harmful fine-tuning, evaluator drift, model merging, and updater revision under a single temporal lens, and it gives concrete measurement units (persistence, propagation reach/depth/delay) that could ground future benchmarks. Strengths worth crediting: the definition is explicit; the numerical example is correctly labeled non-empirical; there are no fitted parameters; limitations are stated plainly (Sections 6-7, including the unresolved degree of independence between generator, actor, and judge); and the taxonomy is internally organized with clear carrier-to-intervention mappings. The paper's honesty about scope is a genuine asset. The main risk is that the quantitative core, Eq. (4), is not fully well-defined in the recursive case that is the paper's central subject, which requires extension rather than mere copy-editing.","major_comments":[{"comment":"The causal estimand Gamma in Eq. (4) compares an evolving regime with a 'frozen-updater' baseline, but the paper never defines what 'freeze the updater' means when the system can revise the procedure that proposes or admits its own updates, which is exactly the recursive case defined in Section 3.5. Because the state loop in Eq. (2) keeps J_t and U_t outside S_t, while recursive development is characterized by changes to those very components, at least three readings of the baseline are possible: U held at its initial value, U held at its pre-revision value, or U frozen by an external authority the system cannot rewrite. These readings produce different Gamma values and different answers to the persistence/propagation questions the paper claims to measure. Section 6 and Section 7 acknowledge that external authorization is needed and that the required degree of independence remains unresolved, but Eq. (4) is not conditioned on that governance machinery. The paper should either extend the formalism to represent updater and evaluator revision inside the state, or explicitly restrict Eq. (4) to regimes in which the frozen component is enforced externally, stating which reading is intended and why the other readings are excluded.","section":"§4.3, Eq. (4); §3.5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The state decomposition S_t = (B_t, A_t, theta_t) places the evaluator J_t and updater U_t outside the evolving state, yet Definition 1 and Section 3.5 define recursive development precisely by changes to these components. As written, the formalism cannot represent two of the paper's own domains, evaluator drift and meta-level update mechanisms, as changes within the system; they can only appear as exogenous switches. This is not a notational quibble: the evaluation units in Section 4.1 that involve evaluator or updater revision have no corresponding transition in Eq. (2). Adding transitions of the form U_{t+1} = F_U(S_t, Delta*_t) and J_{t+1} = F_J(S_t, Delta*_t) would let the taxonomy, the evaluation units, and the estimand in Eq. (4) be expressed in a single language and would make explicit which interventions are internal to the system versus externally imposed. As it stands, Section 3.5.1's claim that 'comparing a frozen updater with a mutable one... isolates this additional effect' is a commitment the formalism does not yet support.","section":"§2.1, Eqs. (1)-(2); §3.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The caption contains the typo 'and e.t.c.'; this should read 'etc.' or the phrase should be removed.","section":"Fig. 2 caption","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract says the paper 'derives governance principles for modification, selection, authorization, provenance, and recovery,' but Section 6 presents design requirements whose effectiveness 'still depends on the threat model.' The word 'derive' overstates the epistemic status; 'proposes' or 'articulates' would be more accurate.","section":"Abstract and §6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Equation (3) uses the explicit reference (J*, E*), while Eq. (4) bundles the persistent state, environment, evaluator, and updater into Z without repeating that convention. A sentence linking the two notations would prevent the reader from conflating the fixed evaluation reference in Eq. (3) with the evolving Z in Eq. (4).","section":"§4.1, Eqs. (3)-(4)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Definition 1 mixes the formal definition with reporting guidance (the paragraph on separate risk and control readouts r_t, c_t and the units of analysis). Consider moving the reporting guidance to Section 4.1 so the definition paragraph stays focused.","section":"§2.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"A few entries are inconsistently formatted, notably [110] which lists 'ZenGen Team' followed by names, and several 2026 preprints (e.g., [13], [16], [22]) whose venue or status is unclear. A pass to standardize the bibliography would be worthwhile.","section":"Reference list","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"You should read this if you care about safety for persistent, self-improving agents. It does not give you a mechanism or a proof. It gives you a clean vocabulary and a map: five carriers (agent state, model state, evaluation feedback, substrate, meta-level update mechanisms), four evaluation units (state, update, trajectory, lineage), six risk manifestations, and a set of governance principles tied to modification, selection, authorization, provenance, and recovery. The writing is unusually honest for this area—the numerical example is labeled illustrative, the authors repeatedly state that the taxonomy is organizing rather than empirical, and Section 7 openly lists what remains unresolved. That intellectual honesty is the paper's real strength.\n\nThe main thing to check is the formal core. Equation (4) defines the evolutionary effect as the difference between an evolving regime and a frozen-updater baseline, but the paper's own definition of recursive development includes revising the updater itself. If the system can change the very switch that is supposed to be frozen, 'freeze the updater' is not a well-defined intervention. The authors acknowledge nearby issues—evaluator drift, unresolved independence between generator/actor/judge, external authorization—but Eq. (4) is not conditioned on any such governance machinery. This is an internal-consistency problem, not merely a missing detail. I think the concern lands.\n\nTwo softer issues. First, the claimed novelty over existing self-evolving-agent safety taxonomies is asserted rather than compared; a reader cannot tell exactly what is new beyond the synthesis. Second, there is no empirical or formal validation, which is acceptable for a position paper, but it means the framework's value will depend on whether others can operationalize the measurement units.\n\nWho is this for? AI safety researchers and evaluators working on self-evolving and recursive systems. It is a useful shared-language contribution, and the evaluation units are concrete enough to build benchmarks around. It deserves a serious referee and probably conditional acceptance; the authors should be asked to fix Eq. (4), ideally by restricting the estimand to cases with external control over the updater or by formalizing updater revision within the state loop.","headline":"A useful and honest conceptual taxonomy for safety under self-improvement, whose main formal claim (Eq. 4) is under-specified when the updater can revise itself, but the flaw is fixable and the paper deserves refereeing.","tokens_in":27862,"tokens_out":1009,"duration_ms":12747},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"For self-improving AI, safety is not a snapshot: it is a matter of what changes get retained, selected, inherited, and propagated to later versions.","keywords":["Evolutionary Safety","Recursive Self-Improvement","Self-Improving AI","AI Safety","AI Evaluation","risk propagation","evaluator drift","persistent agent state"],"falsifier":"Run two identical self-improving agents from the same saved checkpoint on the same recorded task stream, freezing every update channel in one branch while permitting memory writes, skill updates, and evaluator or updater revision in the other, and score both with one common external reference: if the frozen branch shows the same rate of safety-property erosion and the same accumulated violations as the evolving branch, the claim that retention, selection, and recursive revision drive evolutionary risk is undercut for that setting.","tokens_in":28484,"feed_emoji":"🧬","tokens_out":16032,"duration_ms":125190,"temperature":0.7,"pith_summary":"This paper argues that as AI systems begin to improve themselves—keeping memories, revising skills and tools, updating their own parameters, or rewriting the procedures that admit new versions—safety can no longer be assessed at a single moment in time. The authors introduce Evolutionary Safety as the study of how safety properties are preserved, weakened, or restored across the history of a system's own improvement process. They identify six recurring ways safety degrades (intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk inheritance and propagation), locate these risks in five domains of the system, and propose evaluation units and governance controls that follow the history of a safety-relevant change. If the framework is right, safety analysis and regulation of self-improving AI must track where a change came from, what it affects, and how it could be reversed, rather than relying on endpoint tests alone.","feed_headline":"Track AI safety across change history, not a single test","feed_subtitle":"Retained skills, drifting evaluators, and inherited risks can turn one local failure into a persistent one.","key_machinery":"The central object is the safety-relevant change with a temporal history: a modification that persists in memory, skills, tools, parameters, evaluation criteria, or update procedures and thereby shapes later states. The machinery that carries the argument is the formal improvement loop $S_t = (B_t, A_t, \\boldsymbol{\\theta}_t)$, describing computational substrate, persistent agent artifacts, and model parameters, with $D_t \\sim Q_{U_t}(\\cdot \\mid S_t, X_t)$, $\\Delta^{\\star}_t \\sim \\mathrm{Select}_{J_t}(D_t)$, and $S_{t+1} = F_S(S_t, \\Delta^{\\star}_t)$; the equation separates generation, selection, and commitment so that a change can be attributed to the proposal process, the admission rule, or the retained state. On this loop the taxonomy places carriers in five domains—persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms with co-evolution—and the evaluation section defines the units of analysis (states, updates, trajectories, lineages) along with the selection shift $\\Delta^{\\mathrm{sel}}_j(t) = \\sum_k p_t(k) r^{\\star}_{j,k} - \\sum_k q_t(k) r^{\\star}_{j,k}$. The comparison engine is $\\Gamma_{t,h}[g]$ of Eq. (4), an estimand contrasting an evolving regime with a frozen-update-channel baseline under a common external evaluation reference.","core_discovery":"The paper's central claim is that for AI systems that persist and improve themselves, safety is not a state property but a temporal one: a safety property is preserved, weakened, or restored through the mechanisms of selection, persistent state, and inheritance. The formal core is the improvement loop $S_t = (B_t, A_t, \\boldsymbol{\\theta}_t)$, with $D_t \\sim Q_{U_t}(\\cdot \\mid S_t, X_t)$, $\\Delta^{\\star}_t \\sim \\mathrm{Select}_{J_t}(D_t)$, and $S_{t+1} = F_S(S_t, \\Delta^{\\star}_t)$, where the evaluator and the updater determine which candidates survive. Against this loop the paper defines the object of study—a safety-relevant change with a temporal history—and shows that such a change can persist in memory, skills, tools, parameters, feedback criteria, or the update process itself, which is why deleting the initiating input may leave its effects intact. It proposes evaluation units (states, updates, trajectories, and lineages), a selection-induced risk shift $\\Delta^{\\mathrm{sel}}_j(t) = \\sum_k p_t(k) r^{\\star}_{j,k} - \\sum_k q_t(k) r^{\\star}_{j,k}$ that compares proposed versus selected candidate pools, and governance requirements (modification boundaries, pre-commit gating, independent verification and authorization, lineage traceability, and recovery) so that consequential changes cannot become persistent, selected, or inherited without evidence and a path back.","pith_inferences":["If the framework is right, current endpoint-test practice systematically undercounts evolutionary risk: a system that scores safe on a benchmark may already carry a latent unsafe tendency that only a lineage record or a source-removal experiment would expose.","The framework's units could be turned into a standard protocol: benchmarks with known injected causes, checkpoints saved at every accepted update, and identical task streams replayed under frozen versus mutable update channels would give the field a shared way to measure persistence and propagation.","The framework implies that the strongest controls are needed exactly where measurement is hardest—when the evaluator or updater itself is part of what changes—because that is where the fixed external reference that makes any comparison valid disappears."],"forward_implications":["Safety evaluation of self-improving systems must report risk and control readouts across states, accepted updates, trajectories, and lineages, with thresholds and operating conditions stated, rather than endpoint scores alone.","Because an accepted update that is retained or reused changes the starting conditions of later behavior, deleting the initiating input or rolling back software is not enough: recovery must locate, quarantine, and revalidate affected descendants and test for recurrence during continued adaptation.","When a system generates its own training material or judgments, evaluator revision and system change cannot be separated from scores alone; saved states must be rescored under overlapping evaluator versions to isolate each effect.","Selection-induced risk can be measured from the admission record: comparing proposal weights with selection weights over a common candidate pool via $\\Delta^{\\mathrm{sel}}$ reveals whether the selection rule itself favors riskier changes.","Governance must sit inside the improvement loop: modification boundaries, pre-commit gating, independent verification and authorization, and lineage traceability should scale with the persistence and downstream reach of a proposed change."],"supporting_citations":[{"why":"Defines and surveys recursive self-improvement, supplying the boundary the paper uses for when a system counts as a persistent self-improver.","marker":"[17]"},{"why":"Earlier theory on whether an agent that modifies its own policy or utility can preserve its objectives, which Evolutionary Safety extends into a temporal safety property.","marker":"[34]"},{"why":"Gödel machines, the self-referential self-improver framework whose preservation question the paper carries into safety analysis.","marker":"[102]"},{"why":"A Self-Improving Coding Agent that edits its own orchestration code, the concrete demonstration of persistent change the paper builds its boundary on.","marker":"[99]"},{"why":"Sleeper Agents, showing trained backdoor behavior can survive later safety training, the key evidence for risk inheritance across updates.","marker":"[54]"},{"why":"Alignment-faking study showing expressed priorities depend on perceived training or evaluation context, grounding intent drift and selective compliance.","marker":"[44]"},{"why":"Continual-learning review supplying the stability–plasticity and catastrophic-forgetting vocabulary used for model-state safety retention.","marker":"[90]"},{"why":"Reflexion, an instance of verbal feedback carried into later attempts, illustrating persistent agent state in the taxonomy.","marker":"[106]"},{"why":"Meta-Rewarding, placing generated judgments inside the improvement loop, the paper's key example of evaluator drift.","marker":"[120]"}],"fun_headline_variants":["AI safety is a timeline, not a snapshot","How AI risks persist across self-improvement loops","Recursive AI safety depends on change history","Safety must outlast AI's own evolution","Track a safety property as AI rewrites itself"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The whole measurement scheme presupposes a stable outside reference—a fixed evaluator and update procedure—that can judge the evolving system, and although the paper flags the needed independence of generator, actor, and judge as an open question, its formalism does not cover the case where that reference itself is what evolves.","fun_headline_variants_meta":{"raw":{"variants":["AI safety is a timeline, not a snapshot","How AI risks persist across self-improvement loops","Recursive AI safety depends on change history","Safety must outlast AI's own evolution","Track a safety property as AI rewrites itself"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"cost":0.000161868,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000161868,"upstream_inference_prompt_cost":0.000129668,"upstream_inference_completions_cost":3.22e-05},"total_tokens":1246,"prompt_tokens":1131,"completion_tokens":115,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":45}},"tokens_in":1131,"tokens_out":115,"cost_source":"openrouter_usage","duration_ms":2386,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":45,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-28T07:29:36.015681+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run two identical self-improving agents from the same saved checkpoint on the same recorded task stream, freezing every update channel in one branch while permitting memory writes, skill updates, and evaluator or updater revision in the other, and score both with one common external reference: if the frozen branch shows the same rate of safety-property erosion and the same accumulated violations as the evolving branch, the claim that retention, selection, and recursive revision drive evolutionary risk is undercut for that setting.","supporting_citations":[{"cited_title":"Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops","cited_arxiv_id":"2607.07663","evidence_quote":"Defines and surveys recursive self-improvement, supplying the boundary the paper uses for when a system counts as a persistent self-improver."},{"cited_title":"Self-Modification of Policy and Utility Function in Rational Agents","cited_arxiv_id":"1611.08219","evidence_quote":"Earlier theory on whether an agent that modifies its own policy or utility can preserve its objectives, which Evolutionary Safety extends into a temporal safety property."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gödel machines, the self-referential self-improver framework whose preservation question the paper carries into safety analysis."},{"cited_title":"A Self-Improving Coding Agent","cited_arxiv_id":"2504.15228","evidence_quote":"A Self-Improving Coding Agent that edits its own orchestration code, the concrete demonstration of persistent change the paper builds its boundary on."},{"cited_title":"Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training","cited_arxiv_id":"2401.05566","evidence_quote":"Sleeper Agents, showing trained backdoor behavior can survive later safety training, the key evidence for risk inheritance across updates."},{"cited_title":"Alignment faking in large language models","cited_arxiv_id":"2412.14093","evidence_quote":"Alignment-faking study showing expressed priorities depend on perceived training or evaluation context, grounding intent drift and selective compliance."},{"cited_title":"Continual lifelong learning with neural networks: A review","cited_arxiv_id":null,"evidence_quote":"Continual-learning review supplying the stability–plasticity and catastrophic-forgetting vocabulary used for model-state safety retention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reflexion, an instance of verbal feedback carried into later attempts, illustrating persistent agent state in the taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Meta-Rewarding, placing generated judgments inside the improvement loop, the paper's key example of evaluator drift."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}