Pith.

Computation and Language · arXiv:2609.14858 v2 · submitted 14 Sep 2026

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai

show 9 more authors

Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo

The paper says

“Recursive self-improvement is becoming essential for autonomous AI agents, whose progress depends on discovering high-value solutions across complex domains. Effective exploration drives this process, yet managing and improving exploration strategies remains a major bottleneck.”

more of the abstract

Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization must navigate vast meta-search spaces under delayed, expensive feedback from long-horizon rollouts. We introduce \textscDream-RSI, a framework for scalable, recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying base agent unchanged. Our key insight is that accumulated discovery history can act as a replay simulator over the realized search space. By dreaming within this simulator built from historical discovery trees, \textscDream-RSI obtains immediate, low-cost off-policy feedback to evaluate and refine exploration policies without repeated, expensive online evaluation. The improved policy is then redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across 9 tasks in 4 domains, \textscDream-RSI achieves competitive quality and improves discovery efficiency in several settings.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Genuinely new idea—history as replay simulator for exploration policy improvement—but the evaluation needs multi-run statistics and a cleaner test of sim-to-real transfer. This line is wrong

5 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §4, Figs. 3–4, Table 1

The review did not quote this passage.

Pith says: Objection 1 of 5 · major

No experiment reports error bars, confidence intervals, or results across multiple independent seeds. All claims of improvement (e.g., cost reduction, performance gain) rest on point estimates from what appears to be a single run per configuration. Because the discovery process involves stochastic components (LLM generation), this is insufficient to support the reported quantitative claims. The authors should run at least 3–5 independent seeds for each condition and report means with standard deviations or interquartile ranges.

The paper says, at §4.1, Table 3 (reader's take mentions SimpleTES comparison)

The review did not quote this passage.

Pith says: Objection 2 of 5 · major

The comparison with SimpleTES uses different foundation models (GPT-OSS-120B vs Gemini-3.1 Pro) and different total budgets, making the claimed 162× cost reduction uninterpretable. The observed performance differences could be driven by model capability rather than by the Dream-RSI framework. A controlled comparison using the same base model and comparable budget is needed, or the comparison should be removed and the claim qualified accordingly.

The paper says, at §3, Offline evaluation; §4, experimental design

The review did not quote this passage.

Pith says: Objection 3 of 5 · major

The replay simulator returns recorded outcomes deterministically (one child per parent node), but the real online environment is stochastic—the same parent can yield different children across runs. The paper provides no evidence that policies optimized on deterministic replay transfer to stochastic online deployment.

more of objection 3

This is a load-bearing assumption because the entire cost-savings claim rests on replay being a faithful proxy. The authors should either (a) conduct a calibration experiment that compares rankings on deterministic replay with rankings from live online runs on held-out trees, or (b) quantify the stochasticity of their environment and show that deterministic replay preserves relative policy ordering.

The paper says, at §3, Eq. (1), replay objective

The review did not quote this passage.

Pith says: Objection 4 of 5 · major

The replay objective introduces two free coefficients, β₁ and β₂, with no sensitivity analysis. The paper's experimental results may be sensitive to these values. The authors should either demonstrate that the main conclusions are robust across a range of reasonable β values, or provide a principled justification for the chosen values (e.g., via cross-validation on a calibration task).

The paper says, at §4.1, Table 3, SimpleTES row

The review did not quote this passage.

Pith says: Objection 5 of 5 · major

The table reports a 'SimpleTES†' row that is described only as 'SimpleTES†' with no explanation of what this variant is. If this is a reproduction attempt, the setup should be fully described; otherwise the row is confusing and should be removed or clarified.

8 minor objections

The paper says, at General

Pith says: minor objection 1 of 8

The claim 'up to 162× reduction' (abstract, §1) is based on the SimpleTES comparison with different models and budgets. This number is likely to mislead readers; a more cautious phrasing (e.g., 'substantial reduction in agent calls under our settings') is warranted.

The paper says, at Fig. 1 caption

Pith says: minor objection 2 of 8

The caption contains LaTeX-like '1⁄bigcircle' which does not render correctly. Replace with plain '①' or the Unicode circled digit.

show 6 more

The paper says, at Fig. 3(b)

Pith says: minor objection 3 of 8

The round numbers (1,2,3,4,5) are placed on the trajectories but the text does not explain why Dream-RSI uses fewer cumulative compute per round than fixed exploration. A note that Dream-RSI adaptively reduces per-round calls would help reader comprehension.

The paper says, at §4.3, Fig. 4

Pith says: minor objection 4 of 8

The figure shows Dream-RSI achieving '2.09× higher score' on ConvDiv under 'similar budget', but the x-axis is 'Number of Generations'—it appears that Dream-RSI used comparable or slightly fewer generations, not exactly the same. Clarify the budget equivalence in the caption or text.

The paper says, at App. C

Pith says: minor objection 5 of 8

The discovered Lasso solver code is 843 lines without a summary. A brief textual description of the algorithmic innovations (e.g., adaptive Cauchy–Schwarz pruning, lazy Gram-matrix construction, SIMD register blocking) would make the appendix more accessible.

The paper says, at §2, Fig. 2

Pith says: minor objection 6 of 8

The figure's caption says 'thousands of candidate policies can be tested', but the paper does not report how many candidate policies were actually tested per dreaming phase. Adding a typical number (e.g., M = 5) would ground the claim.

The paper says, at References

Pith says: minor objection 7 of 8

Several entries (e.g., OpenEvolve, ShinkaEvolve) appear with URLs or GitHub links but lack formal publication venues or archival versions. If preprints exist, please update references accordingly.

The paper says, at §6, Related work

Pith says: minor objection 8 of 8

The phrase 'object level' vs 'meta level' is used without definition until later in the paragraph. Consider clarifying these terms earlier, perhaps with a brief sentence.

What would settle it

For a held-out task, gather 10-20 candidate exploration policies from the dreaming loop. Compute each policy's replay score on the historical trees (deterministic). Then deploy each policy online for multiple independent runs (e.g., 5 runs each) and measure the mean and variance of real performance. Compute the Spearman rank correlation between the deterministic replay ranking and the mean real-performance ranking. If the correlation is below 0.6, the assumption that deterministic replay is a faithful proxy for stochastic online deployment is not supported, reducing the force of the cost-savings claim.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 43 s · 31,671 tokens read, 4,256 written
  3. Editor wrote the verdict above 16 s · 32,542 tokens read, 1,350 written
  4. Referee major revision: 5 major objections, 8 minor 25 s · 32,016 tokens read, 2,423 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The core idea here is solid and genuinely new. Using completed discovery histories as a replay simulator to evaluate alternative exploration policies offline is a clever way to sidestep the expensive online meta-optimization bottleneck. The framework is cleanly described, the three-domain evaluation covers algorithm engineering, math, and GPU kernels, and the reported cost savings (often >10x) are striking when the simulated policies are later deployed online.

That said, the evaluation has some rough edges that keep me from fully trusting the headline numbers. No error bars, no multiple runs—every result is a single trace. The comparison with SimpleTES uses a different foundation model (GPT-OSS-120B vs Gemini-3.1 Pro), which muddies the claimed efficiency advantage. The stress-test note about deterministic replay is real but not fatal: the paper does deploy the improved policy online and observes gains, which implicitly validates that replay-based improvements transfer, at least in these settings. Still, they should run a controlled experiment that directly compares policies selected by replay versus random selection or a simpler baseline to isolate the simulator's fidelity.

Who gets value from this: anyone working on agentic discovery, LLM-driven search, or meta-optimization for open-ended exploration. The method is practical enough to adopt, and the discovery-tree abstraction is reusable.

Verdict: Deserves peer review. The idea is strong and the experiments, while underpowered, point in a believable direction. I'd ask for multi-run statistics, error bars, and a brief sim-to-real transfer analysis. The comparison with SimpleTES should acknowledge the model mismatch. With those additions, it's a solid NeurIPS or ICML paper.

The strongest objection

The replay simulator described in Section 3 (Offline Evaluation) returns recorded children deterministically: for a given parent node on a historical tree, replay always reveals the exact same child that was recorded during the original online rollout. This means the simulator evaluates candidate policies on a single fixed outcome per decision point, while the real discovery environment is stochastic—the same parent node can produce different children when run again. A policy that looks efficient or balanced on the deterministic tree (e.g., it times branch-opening oppurtunely) might systematically fail in reality because the real outcomes are less predictable. Conversely, a policy that hedges against variability might be undervalued in replay. The paper provides no evidence that rankings on the deterministic simulator correlate with rankings on live stochastic deployments. Since the central claim of cost savings depends on off-policy evaluation being a reliable proxy for online performance, this gap is load-bearing. Section 5.1's comparison of replay vs. guidance does not address this: both arms use the same deterministic replay paradigm, so stochastic transfer is not measured.

The reader's weakest_assumption identified the general risk that the replay simulator may not be a faithful proxy for online exploration. This stress test sharpens that concern to a specific technical gap: the simulator is deterministic while the real process is stochastic. The paper's evidence (Section 5.1) shows that the replay-based dreaming loop outperforms a guidance-based alternative, but both use the same deterministic replay, so the stochastic-transfer problem remains unaddressed. This does not invalidate the central claim—the system as a whole achieves cost savings—but it does mean the savings may come from policy refinement via LLM reasoning about history rather than from reliable off-policy evaluation. The condition that transfer is validated experimentally is therefore strengthened, but the verdict of CONDITIONAL remains appropriate. Hence UNCHANGED.

The referee's report

The core insight—reusing completed discovery trees as deterministic replay simulators for policy improvement—is conceptually clean and addresses a genuine bottleneck in long-horizon exploration for AI-driven discovery. If the method is robust and the cost savings generalize, Dream-RSI could make recursive self-improvement at the meta-exploration layer practical at scale. The paper provides a detailed system description and a diverse evaluation spanning three domains, which supports the plausibility of the approach.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that the bottleneck in recursive self-improvement (RSI) is the inability to efficiently improve the exploration policy that guides discovery. It introduces Dream-RSI, a framework that converts completed discovery histories—structured trees of past decisions and outcomes—into replay simulators. Within these simulators, thousands of alternative exploration policies can be evaluated at negligible cost by simply replaying recorded outcomes, without rerunning the underlying coding agent. The best policy is then deployed online to collect new histories, closing a self-improvement loop at the meta-exploration layer. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI reduces discovery cost by 1.7x to over 50x while matching or improving discovery quality.

What carries the argument. The replay simulator constructed from historical discovery trees. Each node in the tree records the outcome of a generation–evaluation attempt, including score, diagnostics, and workspace snapshot. An alternative exploration policy navigates this pre-recorded tree by selecting which branches to open, in what order, with what parallelism, and when to stop. Because all outcomes are already saved, evaluating a new policy requires only simulating deterministically over recorded children, at zero execution cost. The simulator is the central object that enables off-policy, low-cost feedback for policy improvement.

Core claim. The central claim is that accumulated discovery history, when organized into a structured tree of explored branches, attempted refinements, and realized outcomes, can serve as a replay simulator for cheap off-policy evaluation of exploration policies. By evaluating candidate policies on this frozen simulator—which requires only reading pre-stored node outcomes rather than executing costly online generations—Dream-RSI can rapidly improve the exploration policy through a dreaming phase inside the simulator. The improved policy is then redeployed online, expanding the simulator pool for the next iteration. This turns meta-optimization of exploration from an expensive online trial-and-error process into a fast, simulation-based procedure.

The premise it rests on. The replay simulator built from historical discovery trees is a faithful enough proxy for real online exploration that policies improving on the simulator will also improve on live deployment.

If the paper is right (Pith's inferences, not the authors')

Dream-RSI could be applied to any long-horizon agent-driven discovery process where historical traces can be structured as trees, such as drug molecule search, code generation, or scientific hypothesis exploration.

If the replay simulator is a faithful proxy for real exploration, then exploration policy improvement becomes feasible within a fixed compute budget, enabling agents to autonomously become more efficient over time without manual intervention.

The framework suggests that future RSI systems should explicitly store and structure discovery history for reuse as simulators, rather than discarding or only using it as static context.

By dramatically reducing the cost of meta-policy evaluation, Dream-RSI allows thousands of candidate policies to be screened, increasing the chance of discovering a globally efficient exploration strategy.

The approach may generalize beyond exploration policies to other meta-level decisions in agent systems, such as skill selection or tool orchestration, provided a replayable trace exists.

A natural next step is to test the simulation-to-reality transfer gap: measure how often a policy that performs well on the replay simulator actually improves online outcomes. The paper does not provide this controlled experiment.

The replay simulator is inherently limited to the history already collected; policies that would open entirely new branches not present in the tree cannot be evaluated. This may bias policy improvement toward refining known paths rather than radical exploration.

The framework implicitly assumes that the discovery space is sufficiently structured and that past branches are representative of future useful directions. In highly non-stationary or open-ended discovery, the replay simulator may become stale quickly.

Since the policy-development agent uses feedforward from past replays, there is a risk of overfitting to the historical tree and losing generality. Future work could introduce regularization or diversity bonuses to counteract this.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Recursive self-improvement is becoming essential for autonomous AI agents, whose progress depends on discovering high-value solutions across complex domains. Effective exploration drives this process, yet managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization must navigate vast meta-search spaces under delayed, expensive feedback from long-horizon rollouts. We introduce \textscDream-RSI, a framework for scalable, recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying base agent unchanged. Our key insight is that accumulated discovery history can act as a replay simulator over the realized search space. By dreaming within this simulator built from historical discovery trees, \textscDream-RSI obtains immediate, low-cost off-policy feedback to evaluate and refine exploration policies without repeated, expensive online evaluation. The improved policy is then redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across 9 tasks in 4 domains, \textscDream-RSI achieves competitive quality and improves discovery efficiency in several settings.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.18 · randomized assignment 0.01 · states quantitative result 0.06 · compares with prior results 0.68 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. background 1.00 0.05 0.03 asserted 0.98
The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. background 0.69 0.08 0.03 asserted 1.00
Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. background 0.55 0.06 0.03 asserted 1.00
We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. contribution 1.00 0.85 0.10 asserted 0.99
A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. method 0.84 0.76 0.04 asserted 1.00
Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. method 0.50 0.52 0.12 asserted 0.96
By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. method 0.98 0.85 0.19 asserted 1.00
The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. method 0.99 0.76 0.10 asserted 1.00
Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings. result 1.00 0.94 0.76 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.73

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,881 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Cited by 20 Pith papers

Reviewed papers on Pith that cite this one. The full citation record

  1. Epi: Towards a Persistent, Self-Improving Agent for Autonomous Research cs.CL · Oct 2026
  2. EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality cs.AI · Nov 2026
  3. Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback cs.AI · Nov 2026
  4. ImageRSI: Verification-Driven Recursive Self-Improvement for Agentic Image Generation cs.CV · Oct 2026
  5. Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims cs.LG · Oct 2026
  6. NIOCIS: Typed-Decision Next-Hop Routing with Node-Conditioned Option Spaces for Tool-Use Agents cs.CR · Oct 2026
  7. AReaL-RSI: Adaptive Scaling and Coordination of Recursive Self-Improvement cs.AI · Oct 2026
  8. From Recursion to Convolution: Parallel Self-Improvement for Multi-Agent Systems cs.AI · Nov 2026
  9. WiSI: Wireless-in-the-Loop Self-Improvement eess.SY · Nov 2026
  10. Recursive Self-Improvement in Physical AI: Mechanisms, Evidence, and Evaluation cs.RO · Oct 2026
  11. Work Agents: Systems, Capabilities, and Optimization cs.LG · Oct 2026
  12. DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses cs.CL · Oct 2026
  13. MedEvo: Medical Reasoning Evolution through Knowledge Navigation cs.AI · Oct 2026
  14. Scaling Trajectories for Complex Tasks through Recursive Self-Rewrite cs.AI · Oct 2026
  15. Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing cs.CV · Sep 2026
  16. Experimental Experience Modeling for Autonomous Research cs.AI · Sep 2026
  17. Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification cs.CL · Sep 2026
  18. SwarmRSI: Recursive Self-Improvement through Evolving Scientific Agent Swarm cs.AI · Oct 2026
  19. Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems cs.AI · Sep 2026
  20. GeoPilot: Diagnosis-Guided Interventions and Geometric Evaluation for Aerial Reconstruction cs.CV · Nov 2026

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment