Self-Improving Large Language Models via Progressive Experience Evolution

Fandong Meng; Haoyang Zhou; Meng Li; Shijie Ren; Xiting Wang; Xunlong Wang; Yuetan Chen; Yujie Guo; Yunhang Yao; Yunlong Liang

SPEE claims that distilling a model's own evolving textual experience into its parameters before reinforcement learning improves math reasoning and data efficiency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 13:47 UTC pith:ECBPXMRZ

load-bearing objection SPEE is a sensible, potentially useful recipe—distill a curated text experience pool into the policy before GRPO—but the paper oversells the utility filter and lacks the statistical rigor to back up the close benchmark comparisons. the 4 major comments →

arxiv 2608.02139 v2 pith:ECBPXMRZ submitted 2026-08-03 cs.CL cs.AIcs.LG

Self-Improving Large Language Models via Progressive Experience Evolution

Shijie Ren , Xiting Wang , Meng Li , Yujie Guo , Yunhang Yao , Ziheng Peng , Xunlong Wang , Yuetan Chen

show 3 more authors

Haoyang Zhou Yunlong Liang Fandong Meng

This is my paper

classification cs.CL cs.AIcs.LG

keywords self-improvementexperience distillationon-policy self-distillationreinforcement learningmathematical reasoninglarge language modelsprogressive experience evolution

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the missing step in LLM self-improvement is an intermediate experience-distillation stage between test-time prompting and training-time reinforcement learning. SPEE maintains a global, evolving pool of abstract reasoning strategies extracted from the model's own successful and failed trajectories, filters them by measured marginal utility on held-out problems, distills them into the policy via on-policy self-distillation, and only then applies GRPO. The authors report consistent gains across three model scales and five math benchmarks, with up to 6.96 percentage points over the base model, and about 28% fewer training trajectories than GRPO at the same performance. The central bet is that prompt-level helpfulness of an experience item is a valid proxy for its value when internalized into parameters.

Core claim

The discovery is that transient interaction trajectories can be converted into persistent capability by first extracting compact, transferable experience items—reasoning strategies, constraints, and failure patterns—from both correct and incorrect solutions, then evolving a global pool of these items across rounds (merging, filtering, validating), then internalizing the pool through privilege-guided on-policy self-distillation, and finally optimizing with group-relative policy optimization. The paper reports that this sequence outperforms both test-time experience (Domain Prompt) and training-time RL (GRPO) baselines, as well as trajectory-level self-distillation (SDPO), on AIME24/25, GSM8K,

What carries the argument

The central object is the global experience pool, a continuously updated set of textual experience items extracted by the current policy from its own trajectories, consolidated by an evolution operator that merges and abstracts them, and filtered by an empirical marginal-utility score computed on a held-out probe set. The utility score measures the expected reward gain from adding an item to the prompt; only items with positive utility are retained. These items are then internalized by a teacher-student distillation: a teacher branch conditions on the problem plus experience and rescored on-policy student outputs, and the student minimizes reverse KL to match that distribution. The claim is

Load-bearing premise

The claim depends on the assumption that an experience item that helps the current policy when placed in the prompt will also help when its content is distilled into model parameters; if prompt-level helpfulness does not transfer to parameter-level value, the filter that keeps experiences could keep useless or harmful items and the stage-I mechanism loses its justification.

What would settle it

Distill only the lowest-utility items (or random items) and compare final accuracy after the same GRPO stage; if performance matches SPEE, the marginal-utility filter is not doing the work. More directly, compute the correlation between each item's prompt-level utility score and the actual gain from distilling that item alone; a zero or negative correlation would refute the transfer assumption.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

Experience distillation before RL improves final accuracy: up to +6.96 percentage points over the base model and an average 1.16% improvement over GRPO across scales.
The shared evolving pool matters: removing it lowers average accuracy (on 4B, from 36.67% to 34.13%; on 8B, from 38.71% to 35.48%).
Data efficiency improves: at the same performance level, SPEE needs roughly 28% fewer training trajectories than GRPO.
More evolution iterations monotonically improve the policy's sampling accuracy, from 18.83% at iteration 0 to 22.66% after four iterations.
Stage I reduces the share of problems with all-incorrect response groups while keeping policy entropy comparable, creating more informative GRPO gradient signals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The marginal-utility filter could be replaced by a learned predictor of experience value, since running it on a held-out probe set for each candidate is expensive; the reported correlation between prompt-level utility and final gains is indirect evidence that such a predictor is feasible.
The same progressive evolution loop may transfer to domains with sparse or noisy rewards, such as code generation or agentic tasks, where abstract strategies matter more than instance-specific answers; the paper's math experiments leave this open.
One risk is pool drift toward generic advice: the utility filter might favor vague strategies that help on the probe set but do not transfer to harder distributions, so a distribution-shift evaluation of the distilled pool would be a valuable stress test.
The monotonic improvement with evolution iterations suggests a curriculum interpretation, where each round abstracts the previous round's experience; this could be tested by ablating the merge/abstract operator and keeping only accumulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

SPEE is a sensible, potentially useful recipe—distill a curated text experience pool into the policy before GRPO—but the paper oversells the utility filter and lacks the statistical rigor to back up the close benchmark comparisons.

read the letter

Short version: this is a real integration, not a mirage. The global evolving experience pool, on-policy self-distillation before RL, and the data-efficiency claim are new as a package. The aggregate improvements over GRPO and SDPO across three scales are consistent enough on average, and the qualitative example shows the experience actually changes behavior. Code is public. That earns credit.

That said, the soft spots are real and non-trivial. The stress-test concern about the utility filter is fair. Eq 9/10 measure whether adding an experience to the prompt helps the current policy on probe problems, but the distillation objective makes the student solve without that prompt. The paper never checks that prompt-level helpfulness transfers to parameter-level distillation value. The ablation removes the entire shared pool, not just the filter, so we learn nothing about whether the threshold actually selects useful experiences. I'd want to see an ablation that keeps all candidate experiences or filters randomly. Without that, the claim that filtering low-utility experience is a driver of SPEE's gains is unsupported.

Second, the empirical reporting is thin. No error bars, no multiple seeds. On qwen3-4b AIME25, SPEE (8.75) is well below SDPO (10.83) and Domain Prompt (9.17); the average wins are partly carried by other tasks. The phrase "consistently outperforms" is doing more work than the tables justify.

Minor points: Eq 1's mutual-information framing is decorative, never connected to the loss. Key hyperparameters (epsilon, S, M, the extraction and evolution prompts) are not in the paper, which will make replication harder even with code.

The core idea—internalize explicit experience before RL—is coherent and worth pursuing. The paper is not a desk reject; it deserves a serious referee. But the referee should push for variance estimates, a filter-ablations, and a direct test of the prompt-to-parameter transfer assumption. My own verdict would be conditional too.

Referee Report

4 major / 5 minor

Summary. The paper proposes SPEE, a two-stage post-training framework for mathematical reasoning. In Stage I, the current policy samples multiple responses per problem; successful and failed trajectories are converted by an LLM-based extractor into textual 'experience' items. These items are merged into a global pool, filtered by a held-out probe utility metric (Eq. 9-10), and then internalized into the policy through On-Policy Self-Distillation (OPSD, Eq. 15). In Stage II, the distilled policy is further optimized with GRPO. Experiments on five math benchmarks with Qwen3-1.7B/4B/8B report consistent accuracy gains over the base model, GRPO, SDPO, and a domain-prompt baseline, with the largest average improvement of +6.96 pp on qwen3-4b-base. The authors argue that experience distillation provides a stronger initialization for RL, improving exploration efficiency and final accuracy.

Significance. The idea of explicitly constructing a transferable, filtered experience pool and distilling it into parameters before RL is a natural and potentially useful synthesis of test-time and training-time self-improvement. If the mechanism were validated, the framework would be a practical contribution to LLM post-training. The paper includes a public code link, and Figure 4's demonstration that sampling accuracy improves with experience-evolution iterations is a suggestive positive signal. However, the empirical claims currently outrun the evidence: the main results have no error bars or repeated seeds, and the utility filter's selection criterion is not shown to transfer from prompt-level helpfulness to parameter-level value. These issues are fixable, but they are load-bearing for the paper's central claims.

major comments (4)

[Stage I, Eqs. (9)-(10) vs. Eq. (15)] The utility filter scores an experience item by its marginal effect on the current policy's expected reward when the item is added to the prompt (Eq. 9), and Eq. 10 keeps items with w(k)(e) > epsilon. But OPSD (Eq. 15) trains the student to solve problems without access to e at inference. The selection criterion and usage condition are therefore mismatched: high-w items may be explicit recipes that help only when present in-context, while low-w items that are redundant in-context could still provide useful dense training signal when distilled. Table 2 ablates the shared pool but does not validate the Eq. 10 filter itself. To support the claim that filtering low-utility experience is what makes SPEE superior, please add a direct check: e.g., compare the Eq. 9 filter against random filtering or no filtering, or measure the correlation between prompt-level utility scores and the post-distil
[Table 1 and experimental protocol] No error bars, multiple seeds, or statistical significance tests are reported. Many of the headline comparisons are small: on qwen3-1.7b the SPEE-GRPO average gap is +0.75 pp, on qwen3-8b it is +0.79 pp, and on individual benchmarks SPEE is sometimes numerically worse than GRPO (e.g., MINA 13.97 vs. 14.07 on 1.7b; MINA 24.06 vs. 23.86 on 8b). Without replicated runs or confidence intervals, the claim that SPEE 'consistently outperforms' GRPO across scales is not established. Please report means and standard deviations over at least three seeds, or provide paired bootstrap intervals over benchmark problems. This is central to the paper's empirical thesis.
[Background, Eq. (1)] Eq. (1) is presented as the formal objective of experience extraction: min_{pΣ(E|T)} I(T;E) - λ I(E;C). However, the implemented extractor (Eq. 6) is an LLM prompted to summarize trajectories, and neither the utility filter (Eq. 9) nor the OPSD loss (Eq. 15) is derived from or shown to optimize Eq. (1). The equation is never referenced again after the Background section. Either provide a concrete argument connecting Eq. (1) to the implemented components, or clearly label it as conceptual intuition. As written, it gives an appearance of formal grounding that the method does not actually use.
[Table 2 and experimental setup] The ablation 'w/o Shared Experience Pool' is under-specified. Does it remove the global pool entirely and distill only per-problem extracted experience, or does it use a non-evolving pool? The distinction matters because the authors use this ablation to conclude that the shared pool is 'particularly important.' Please define the exact variant, including what the teacher branch conditions on in this condition. Additionally, the main experiments use a single evolution round (the text states this is for computational efficiency), yet the framework is described as 'progressive.' Please state the exact number of evolution rounds M used in Tables 1-2 and clarify whether the Figure 4 multi-round results would change the main findings if M > 1 were used.

minor comments (5)

[Eq. (9)] The notation \hat{E}^{(k+1)}_e is not defined. It presumably denotes the provisional pool with candidate experience e added, but this should be stated explicitly.
[Hyperparameters] Key hyperparameters are not reported: S (number of sampled trajectories per problem), epsilon (utility threshold), probe set size, M (number of evolution rounds), GRPO steps, rollout counts, and learning rates. Please include a full hyperparameter table.
[Table 1] The abbreviation 'MINA' is used without expansion in the main text; it denotes MinervaMath but should be spelled out at first use.
[Conclusion] The final sentence is grammatically awkward: 'The framework supports closed-loop iteration, future work may further explore multiple rounds of methods.' Please rephrase.
[Figure 5] The caption uses informal phrasing ('Winner with experience') and the divergence between the baseline and experience-conditioned outputs is described in a way that presupposes the interpretation. Minor rewording would improve clarity.

Circularity Check

0 steps flagged

No circularity: SPEE's derivation is empirical and self-contained; Eq. 9/10 filter is a prompt-level heuristic but not definitionally equivalent to the distillation objective.

full rationale

The paper does not derive a conclusion from an assumption that encodes that same conclusion. Eq. 1 (I(T;E)-lambda*I(E;C)) is motivation, not used to prove SPEE's gains. Eq. 9 defines marginal utility as improvement in expected reward when experience e is added to the prompt on a held-out probe set; Eq. 10 selects items with positive utility; Eq. 15 then distills the pool via reverse KL. The selection criterion and the distillation objective are distinct quantities--prompt-conditioned expected reward vs. parameter-level KL matching--so no equation reduces to another by construction. The fact that the extractor, teacher, and student share the same model is a design property of self-improvement, not a logical circle. The paper's central claims are tested against held-out benchmarks (AIME, GSM8K, MATH500, Minerva) with comparisons to GRPO, SDPO, and prompt baselines, and ablations validate components. The skeptic's concern that prompt-level utility may not transfer to parameter-level distillation value is a legitimate empirical risk about the validity of the filter, but it is not a circularity: the paper never defines distillation value as Eq. 9, and the claim is falsifiable. No self-citation chain or imported uniqueness theorem is load-bearing. Accordingly, no circular step can be exhibited with a specific reduction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claim rests on the model's ability to extract useful transferable experience from its own trajectories and on the assumption that prompt-level utility on a probe set predicts parameter-level value after distillation. These are domain assumptions that are plausible but not proven. The free parameters are mostly unreported hyperparameters. No new physical entities are introduced; the experience pool is an algorithmic artifact.

free parameters (6)

epsilon (utility acceptance threshold) = not reported
Eq 10 keeps experience items with marginal utility w(k)(e) > epsilon; epsilon controls pool quality and is a hand-set hyperparameter without a reported value.
S (trajectories sampled per problem) = not reported
Eq 6 randomly selects S trajectories per problem for experience extraction; this controls how much trajectory variety is distilled.
G (responses sampled per problem) = 8 in Figure 2, not specified in method
Eqs 4 and 16 use G responses; Figure 2 suggests 8 rollouts, but the exact G for Stage I and Stage II is not stated.
M (number of experience evolution rounds) = 1 in main experiments
Eq 12 defines E(0) to E(M); the main experiments use a single round while Figure 4 shows monotone gains through 4 rounds, so M is a consequential choice.
delta (GRPO advantage stability constant) = not reported
Eq 17 divides by (sigma_G + delta); the constant is standard but its value is needed for exact reproduction.
epsilon_clip (GRPO clipping coefficient) = not reported
Eq 19 clips importance ratios to [1-epsilon_clip, 1+epsilon_clip]; the value is not reported.

axioms (5)

domain assumption The same model can extract reliable transferable experience from its own trajectories using a prompt.
Eq 6 implements the extractor Sigma_theta_k as the current policy; no external verifier checks the extracted lessons.
domain assumption Prompt-level marginal utility predicts parameter-level distillation value.
Eqs 9 and 10 filter experiences by whether the current policy solves held-out probes better when the item is in context; the paper assumes this transfers to the value of training the policy to imitate the teacher.
domain assumption Binary verifier rewards are sufficient supervision for mathematical self-improvement.
Eqs 5 and 16 use only r(q,y) in {0,1}; this is standard for math but still an assumption about the training distribution and verifier.
ad hoc to paper Eq 1 is a valid formalization of experience extraction.
The mutual information objective is introduced as a reformulation but is never solved or connected to the implemented pipeline; it is motivational framing.
standard math GRPO convergence and clipped importance sampling behave as in the cited work.
Eqs 16 to 20 use GRPO without re-derivation, relying on Shao et al. 2024.

invented entities (1)

global experience pool E no independent evidence
purpose: stores evolved textual experience items that condition the teacher branch during OPSD and are filtered by marginal utility
This is an algorithmic component of the method, not a natural entity. Its effectiveness is measured only through the paper's own ablations and utility filter; no independent handle outside the paper is provided.

reviewed 2026-08-04 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Self-Improving Large Language Models via Progressive Experience Evolution." pith.science (2026). https://pith.science/paper/ECBPXMRZ

@misc{pith2026260802139,
  author       = {Pith},
  title        = {Pith review of: Self-Improving Large Language Models via Progressive Experience Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ECBPXMRZ}},
  note         = {Machine review of arXiv:2608.02139}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Large language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities. Existing self-improvement paradigms remain fragmented: test-time methods can explicitly extract experience but cannot internalize it into model parameters, whereas training-time optimization methods can update model parameters but lack an explicit mechanism for accumulating transferable experience. Bridging these two paradigms requires a critical intermediate stage that remains underexplored, namely \emph{experience distillation}. To address this gap, we propose \textbf{SPEE} (\textbf{S}elf-\textbf{P}rogressive \textbf{E}xperience \textbf{E}volution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization. During explicit experience evolution, SPEE reflects on trajectories collected from multiple interactions to extract, verify, and progressively evolve transferable experience, which is subsequently internalized into the policy through privilege-guided On-Policy Self-Distillation (OPSD). During implicit policy optimization, reward-driven reinforcement learning leverages these internalized priors to explore novel solution strategies. In the experience evolution stage, a continuously evolving global experience pool consolidates knowledge from both successful and failed trajectories, filters out low-utility experience, and mitigates post-hoc rationalization induced by individual trajectories. Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales. The source code is available at https://github.com/rrrsj/SPEE.

Figures

Figures reproduced from arXiv: 2608.02139 by Fandong Meng, Haoyang Zhou, Meng Li, Shijie Ren, Xiting Wang, Xunlong Wang, Yuetan Chen, Yujie Guo, Yunhang Yao, Yunlong Liang, Ziheng Peng.