Pith.

Artificial Intelligence · arXiv:2609.31186 v2 · submitted 25 Sep 2026

Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation

Chang Gong, Jingping Bi, Di Yao, Ran Cheng, Yaodong Yang, Kaile Wang, Xinjian Liang, Chao Xiang

show 1 more author

Ruijie Guo

The paper says

“Rapid advances in AI are bringing recursive self-improvement (RSI) closer to practice, with growing progress in model self-training, self-evolving agents, and automated AI development. Yet the very mechanisms that enable continuous improvement also introduce new challenges for AI safety.”

more of the abstract

Unlike isolated failures, safety risks can persist across updates, accumulate through repeated adaptation, and propagate to successor systems. More importantly, the processes that generate and select future improvements may themselves evolve, making safety increasingly difficult to assess and maintain. This raises a fundamental question, i.e., how can safety be maintained when both AI systems and the processes that improve them continue to change? Therefore, we introduce Evolutionary Safety to study how safety properties are preserved, weakened, or restored throughout persistent and recursive self-improvement. We characterize six recurring manifestations of evolutionary risk and develop a five-domain taxonomy that captures how safety-relevant changes arise, persist, and propagate across evolving systems. Building on this foundation, we establish a framework for risk discovery and evaluation across states, updates, trajectories, and lineages, and examine governance principles for controlling modifications, verifying and authorizing updates, and ensuring traceability and recovery. We further identify open problems in maintaining safety as both AI systems and their improvement mechanisms continue to evolve.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 28 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A useful and honest conceptual taxonomy for safety under self-improvement, whose main formal claim (Eq. 4) is under-specified when the updater can revise itself, but the flaw is fixable and the paper deserves refereeing. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §4.3, Eq. (4); §3.5.1

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The causal estimand Gamma in Eq. (4) compares an evolving regime with a 'frozen-updater' baseline, but the paper never defines what 'freeze the updater' means when the system can revise the procedure that proposes or admits its own updates, which is exactly the recursive case defined in Section 3.5.

more of objection 1

Because the state loop in Eq. (2) keeps J_t and U_t outside S_t, while recursive development is characterized by changes to those very components, at least three readings of the baseline are possible: U held at its initial value, U held at its pre-revision value, or U frozen by an external authority the system cannot rewrite. These readings produce different Gamma values and different answers to the persistence/propagation questions the paper claims to measure. Section 6 and Section 7 acknowledge that external authorization is needed and that the required degree of independence remains unresolved, but Eq. (4) is not conditioned on that governance machinery. The paper should either extend the formalism to represent updater and evaluator revision inside the state, or explicitly restrict Eq. (4) to regimes in which the frozen component is enforced externally, stating which reading is intended and why the other readings are excluded.

The paper says, at §2.1, Eqs. (1)-(2); §3.5

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The state decomposition S_t = (B_t, A_t, theta_t) places the evaluator J_t and updater U_t outside the evolving state, yet Definition 1 and Section 3.5 define recursive development precisely by changes to these components. As written, the formalism cannot represent two of the paper's own domains, evaluator drift and meta-level update mechanisms, as changes within the system; they can only appear as exogenous switches.

more of objection 2

This is not a notational quibble: the evaluation units in Section 4.1 that involve evaluator or updater revision have no corresponding transition in Eq. (2). Adding transitions of the form U_{t+1} = F_U(S_t, Delta*_t) and J_{t+1} = F_J(S_t, Delta*_t) would let the taxonomy, the evaluation units, and the estimand in Eq. (4) be expressed in a single language and would make explicit which interventions are internal to the system versus externally imposed. As it stands, Section 3.5.1's claim that 'comparing a frozen updater with a mutable one... isolates this additional effect' is a commitment the formalism does not yet support.

5 minor objections

The paper says, at Fig. 2 caption

Pith says: minor objection 1 of 5

The caption contains the typo 'and e.t.c.'; this should read 'etc.' or the phrase should be removed.

The paper says, at Abstract and §6

Pith says: minor objection 2 of 5

The abstract says the paper 'derives governance principles for modification, selection, authorization, provenance, and recovery,' but Section 6 presents design requirements whose effectiveness 'still depends on the threat model.' The word 'derive' overstates the epistemic status; 'proposes' or 'articulates' would be more accurate.

show 3 more

The paper says, at §4.1, Eqs. (3)-(4)

Pith says: minor objection 3 of 5

Equation (3) uses the explicit reference (J*, E*), while Eq. (4) bundles the persistent state, environment, evaluator, and updater into Z without repeating that convention. A sentence linking the two notations would prevent the reader from conflating the fixed evaluation reference in Eq. (3) with the evolving Z in Eq. (4).

The paper says, at §2.5

Pith says: minor objection 4 of 5

Definition 1 mixes the formal definition with reporting guidance (the paragraph on separate risk and control readouts r_t, c_t and the units of analysis). Consider moving the reporting guidance to Section 4.1 so the definition paragraph stays focused.

The paper says, at Reference list

Pith says: minor objection 5 of 5

A few entries are inconsistently formatted, notably [110] which lists 'ZenGen Team' followed by names, and several 2026 preprints (e.g., [13], [16], [22]) whose venue or status is unclear. A pass to standardize the bibliography would be worthwhile.

What would settle it

Instantiate a minimal recursive self-improvement loop, for example a two-step agent that can write to memory and edit its own selection rule, and compute Gamma from Eq. (4) under three distinct definitions of the frozen-updater baseline: (a) U is held at its initial value, (b) U is held at its value just before a proposed revision is committed, and (c) U is frozen by a structural external-authority block that no candidate can write to. If the three Gamma values differ materially, Eq. (4) is under-specified for recursive self-improvement and the paper must either extend the state vector and dynamics to cover updater revision or explicitly restrict Eq. (4) to non-recursive adaptation. If the three values agree, the reader's concern is resolved and the estimand has a stable reference.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 28 Sep 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 31 s · 26,764 tokens read, 4,308 written
  3. Editor wrote the verdict above 13 s · 27,862 tokens read, 1,009 written
  4. Referee major revision: 2 major objections, 5 minor 42 s · 27,107 tokens read, 5,687 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should read this if you care about safety for persistent, self-improving agents. It does not give you a mechanism or a proof. It gives you a clean vocabulary and a map: five carriers (agent state, model state, evaluation feedback, substrate, meta-level update mechanisms), four evaluation units (state, update, trajectory, lineage), six risk manifestations, and a set of governance principles tied to modification, selection, authorization, provenance, and recovery. The writing is unusually honest for this area—the numerical example is labeled illustrative, the authors repeatedly state that the taxonomy is organizing rather than empirical, and Section 7 openly lists what remains unresolved. That intellectual honesty is the paper's real strength.

The main thing to check is the formal core. Equation (4) defines the evolutionary effect as the difference between an evolving regime and a frozen-updater baseline, but the paper's own definition of recursive development includes revising the updater itself. If the system can change the very switch that is supposed to be frozen, 'freeze the updater' is not a well-defined intervention. The authors acknowledge nearby issues—evaluator drift, unresolved independence between generator/actor/judge, external authorization—but Eq. (4) is not conditioned on any such governance machinery. This is an internal-consistency problem, not merely a missing detail. I think the concern lands.

Two softer issues. First, the claimed novelty over existing self-evolving-agent safety taxonomies is asserted rather than compared; a reader cannot tell exactly what is new beyond the synthesis. Second, there is no empirical or formal validation, which is acceptable for a position paper, but it means the framework's value will depend on whether others can operationalize the measurement units.

Who is this for? AI safety researchers and evaluators working on self-evolving and recursive systems. It is a useful shared-language contribution, and the evaluation units are concrete enough to build benchmarks around. It deserves a serious referee and probably conditional acceptance; the authors should be asked to fix Eq. (4), ideally by restricting the estimand to cases with external control over the updater or by formalizing updater revision within the state loop.

The strongest objection

The quantitative core of the paper is Eq. (4), which defines the evolutionary effect as the difference between an evolving regime and a frozen-updater baseline: Gamma = E[g(Z^{pi_evo}_{t+h}) - g(Z^{pi_freeze}_{t+h})]. This estimand is the only concrete handle on the paper's central claims about persistence, propagation, and accumulation. But the paper's own definition of recursive development, Section 3.5, makes the updater and evaluator part of what changes, while the state loop in Eq. (2) keeps J_t and U_t outside S_t. Those two commitments are not jointly formalized. In particular, the intervention 'freeze the updater' is not well-defined when the system can propose and commit changes to its own update procedure: the system may revise the very switch that is supposed to be frozen. Different natural readings of the baseline (U held at its initial value, U held at its pre-revision value, or U frozen by an external authority) can produce different values of Gamma. The paper acknowledges part of this in Section 7, noting that the evaluator can change and that the required degree of independence between generator, actor, and judge remains unresolved, and Section 6 introduces external authorization as a governance requirement. But Eq. (4) is not conditioned on that governance machinery, so it is not merely incomplete: as written, the causal reference for the recursive case is under-specified. This is an internal-consistency issue rather than a disagreement with current consensus, and it lands directly on the paper's central measurement claim.

I read the paper as a conceptual and taxonomic contribution: there are no experiments, no code, and the numerical example in Eq. (5) is explicitly labeled illustrative. The taxonomy, the manifestation list, and the governance principles are broadly and honestly tied to cited empirical work, and the authors repeatedly flag open problems including changing judges, latent risks, and unresolved independence requirements. The strongest part of the paper is the organization of risk carriers and the concrete comparisons in Table 1. The load-bearing weakness is the one the reader identified: the formal state/loop equations in Eqs. (1)-(2) treat the evaluator and updater as external selectors, while Section 3.5 argues that recursive development involves changing those selectors from within. Equation (4) then compares an evolving regime with a frozen-updater baseline, but the recursive case lacks a well-defined freeze intervention unless the external authority is made structural. The paper acknowledges much of this difficulty in Sections 6 and 7, which is why this is an addressable condition rather than a fatal flaw. Because the concern is already reflected in the reader's CONDITIONAL verdict and is not a reason to reject the paper, I leave the verdict unchanged. The secondary concern about asserted rather than demonstrated novelty over existing taxonomies is real but less load-bearing, since the paper's contribution is primarily a framing and measurement scheme rather than a new empirical result.

The referee's report

If the framework holds, it provides a useful common language for a scattered literature: the paper connects memory poisoning, harmful fine-tuning, evaluator drift, model merging, and updater revision under a single temporal lens, and it gives concrete measurement units (persistence, propagation reach/depth/delay) that could ground future benchmarks. Strengths worth crediting: the definition is explicit; the numerical example is correctly labeled non-empirical; there are no fitted parameters; limitations are stated plainly (Sections 6-7, including the unresolved degree of independence between generator, actor, and judge); and the taxonomy is internally organized with clear carrier-to-intervention mappings. The paper's honesty about scope is a genuine asset. The main risk is that the quantitative core, Eq. (4), is not fully well-defined in the recursive case that is the paper's central subject, which requires extension rather than mere copy-editing.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that as AI systems begin to improve themselves—keeping memories, revising skills and tools, updating their own parameters, or rewriting the procedures that admit new versions—safety can no longer be assessed at a single moment in time. The authors introduce Evolutionary Safety as the study of how safety properties are preserved, weakened, or restored across the history of a system's own improvement process. They identify six recurring ways safety degrades (intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk inheritance and propagation), locate these risks in five domains of the system, and propose evaluation units and governance controls that follow the history of a safety-relevant change. If the framework is right, safety analysis and regulation of self-improving AI must track where a change came from, what it affects, and how it could be reversed, rather than relying on endpoint tests alone.

What carries the argument. The central object is the safety-relevant change with a temporal history: a modification that persists in memory, skills, tools, parameters, evaluation criteria, or update procedures and thereby shapes later states. The machinery that carries the argument is the formal improvement loop $S_t = (B_t, A_t, \boldsymbol{\theta}_t)$, describing computational substrate, persistent agent artifacts, and model parameters, with $D_t \sim Q_{U_t}(\cdot \mid S_t, X_t)$, $\Delta^{\star}_t \sim \mathrm{Select}_{J_t}(D_t)$, and $S_{t+1} = F_S(S_t, \Delta^{\star}_t)$; the equation separates generation, selection, and commitment so that a change can be attributed to the proposal process, the admission rule, or the retained state. On this loop the taxonomy places carriers in five domains—persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms with co-evolution—and the evaluation section defines the units of analysis (states, updates, trajectories, lineages) along with the selection shift $\Delta^{\mathrm{sel}}_j(t) = \sum_k p_t(k) r^{\star}_{j,k} - \sum_k q_t(k) r^{\star}_{j,k}$. The comparison engine is $\Gamma_{t,h}[g]$ of Eq. (4), an estimand contrasting an evolving regime with a frozen-update-channel baseline under a common external evaluation reference.

Core claim. The paper's central claim is that for AI systems that persist and improve themselves, safety is not a state property but a temporal one: a safety property is preserved, weakened, or restored through the mechanisms of selection, persistent state, and inheritance. The formal core is the improvement loop $S_t = (B_t, A_t, \boldsymbol{\theta}_t)$, with $D_t \sim Q_{U_t}(\cdot \mid S_t, X_t)$, $\Delta^{\star}_t \sim \mathrm{Select}_{J_t}(D_t)$, and $S_{t+1} = F_S(S_t, \Delta^{\star}_t)$, where the evaluator and the updater determine which candidates survive. Against this loop the paper defines the object of study—a safety-relevant change with a temporal history—and shows that such a change can persist in memory, skills, tools, parameters, feedback criteria, or the update process itself, which is why deleting the initiating input may leave its effects intact. It proposes evaluation units (states, updates, trajectories, and lineages), a selection-induced risk shift $\Delta^{\mathrm{sel}}_j(t) = \sum_k p_t(k) r^{\star}_{j,k} - \sum_k q_t(k) r^{\star}_{j,k}$ that compares proposed versus selected candidate pools, and governance requirements (modification boundaries, pre-commit gating, independent verification and authorization, lineage traceability, and recovery) so that consequential changes cannot become persistent, selected, or inherited without evidence and a path back.

The premise it rests on. The whole measurement scheme presupposes a stable outside reference—a fixed evaluator and update procedure—that can judge the evolving system, and although the paper flags the needed independence of generator, actor, and judge as an open question, its formalism does not cover the case where that reference itself is what evolves.

If the paper is right (Pith's inferences, not the authors')

Safety evaluation of self-improving systems must report risk and control readouts across states, accepted updates, trajectories, and lineages, with thresholds and operating conditions stated, rather than endpoint scores alone.

Because an accepted update that is retained or reused changes the starting conditions of later behavior, deleting the initiating input or rolling back software is not enough: recovery must locate, quarantine, and revalidate affected descendants and test for recurrence during continued adaptation.

When a system generates its own training material or judgments, evaluator revision and system change cannot be separated from scores alone; saved states must be rescored under overlapping evaluator versions to isolate each effect.

Selection-induced risk can be measured from the admission record: comparing proposal weights with selection weights over a common candidate pool via $\Delta^{\mathrm{sel}}$ reveals whether the selection rule itself favors riskier changes.

Governance must sit inside the improvement loop: modification boundaries, pre-commit gating, independent verification and authorization, and lineage traceability should scale with the persistence and downstream reach of a proposed change.

If the framework is right, current endpoint-test practice systematically undercounts evolutionary risk: a system that scores safe on a benchmark may already carry a latent unsafe tendency that only a lineage record or a source-removal experiment would expose.

The framework's units could be turned into a standard protocol: benchmarks with known injected causes, checkpoints saved at every accepted update, and identical task streams replayed under frozen versus mutable update channels would give the field a shared way to measure persistence and propagation.

The framework implies that the strongest controls are needed exactly where measurement is hardest—when the evaluator or updater itself is part of what changes—because that is where the fixed external reference that makes any comparison valid disappears.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Rapid advances in AI are bringing recursive self-improvement (RSI) closer to practice, with growing progress in model self-training, self-evolving agents, and automated AI development. Yet the very mechanisms that enable continuous improvement also introduce new challenges for AI safety. Unlike isolated failures, safety risks can persist across updates, accumulate through repeated adaptation, and propagate to successor systems. More importantly, the processes that generate and select future improvements may themselves evolve, making safety increasingly difficult to assess and maintain. This raises a fundamental question, i.e., how can safety be maintained when both AI systems and the processes that improve them continue to change? Therefore, we introduce Evolutionary Safety to study how safety properties are preserved, weakened, or restored throughout persistent and recursive self-improvement. We characterize six recurring manifestations of evolutionary risk and develop a five-domain taxonomy that captures how safety-relevant changes arise, persist, and propagate across evolving systems. Building on this foundation, we establish a framework for risk discovery and evaluation across states, updates, trajectories, and lineages, and examine governance principles for controlling modifications, verifying and authorizing updates, and ensuring traceability and recovery. We further identify open problems in maintaining safety as both AI systems and their improvement mechanisms continue to evolve.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-28

The authors' main claim, as the classifier picked it out 0.56

“Building on this taxonomy, we examine how evolutionary risks can be discovered and evaluated across states, updates, trajectories, and lineages, and derive governance principles for modification, selection, authorization, provenance, and recovery.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.52 · human participants 0.04 · controlled comparison 0.02 · randomized assignment 0.01 · states quantitative result 0.02 · compares with prior results 0.04 · mentions code or data availability 0.88

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Artificial intelligence is advancing rapidly, with increasingly capable systems taking larger roles in reasoning, decision-making, scientific discovery, and autonomous development. background 1.00 0.04 0.02 asserted 1.00
As AI begins to participate in its own improvement, from model training and experience accumulation to agent evolution and automated AI development, the prospect of recursive self-improvement (RSI) is becoming increasingly relevant. background 1.00 0.06 0.03 asserted 0.65
This transition raises a fundamental safety question: how can safety be maintained when the system, its accumulated experience, and even the process producing its successors continue to change? background 0.80 0.04 0.03 asserted 0.71
We introduce Evolutionary Safety as a perspective for studying safety under persistent and recursive self-improvement. contribution 1.00 0.92 0.13 asserted 0.99
It concerns not only whether an AI system is safe at a particular moment, but how safety properties change, persist, accumulate, and propagate throughout evolution. contribution 0.97 0.43 0.12 asserted 1.00
We characterize recurring manifestations, including intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk propagation. contribution 0.96 0.87 0.09 asserted 1.00
We then develop a taxonomy spanning persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms. contribution 0.71 0.93 0.05 asserted 0.99
Building on this taxonomy, we examine how evolutionary risks can be discovered and evaluated across states, updates, trajectories, and lineages, and derive governance principles for modification, selection, authorization, provenance, and recovery. contribution 0.93 0.94 0.26 asserted 0.99
Finally, we outline open problems toward maintaining safety guarantees as AI systems become increasingly persistent, adaptive, and recursively self-improving. future work 1.00 0.73 0.10 unclear 0.54
Project resources and proposed evaluation systems are available at https://chaunceykung.github.io/evolutionary-safety-rsi. contribution 0.63 0.45 0.02 asserted 0.96

What kind of paper the classifier reports this is: position or commentary 0.75 · evidence from argument 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-28: 1 request, 7,581 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Taxonomy, evaluation, and governance for evolutionary safety.
Figure 2 of the paper
Figure 2. Publication trends for RSI, self-evolving AI and evolution-coupled safety work, with a conceptual map of safety issues and
Figure 3 of the paper
Figure 3. Four research questions guiding our study of Evolutionary Safety.
Figure 4 of the paper
Figure 4. Illustration of Recursive Self-Improving AI and Evolutionary Safety.
Figure 5 of the paper
Figure 5. Taxonomy of Evolutionary Safety. an executable skill: the artifact gains reach while its original context may be lost. This domain therefore centers on how experience is admitted, transformed, reused, and repaired across an agent’s working history. An agent can change its future behavior without changing its weights. It may save a reflection, add a skill to a library, revise a tool description, or reuse a workflow. Reflexion, Voyager, and Agent Workflow Memory illustrate different ways to carry experience beyond the interaction that produced it [106, 113, 119]. ToolEvo extends this…
Figure 6 of the paper
Figure 6. Illustrative mechanisms through which local safety failures become evolutionary risks across persistent agent state, evaluation

Figures are the authors', reproduced from the paper.

Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment