Computation and Language · arXiv:2610.23344 v2 · submitted 16 Oct 2026
Language Models Learn to Improve Language Models
The paper says
“Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges.”
more of the abstract
First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 19 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. LM2LM combines a frozen world-model reward with on-policy calibration to train a proposer that beats a frontier model at lower cost. The idea is solid and the experiments are thorough, but the single-run results and reliance on GPT-6-sol leave open questions about reproducibility and statistical robustness. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Sec. 4.2, Table 1, Fig. 3–5
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
All main results (Avg@4, Best@4, win rates, ablation studies) are reported from a single RL training run. DAPO training involves stochasticity in proposal sampling, RWM predictions, calibration proposal selection, and weight updates. Without multiple seeds (or at least confidence intervals from bootstrapping the held-out states, which the paper already does for some analysis in Appendix I but not for the headline numbers), the claimed margins over baselines—especially the headline claim that Qwen3.8-27B outperforms GPT-6-sol—cannot be assessed for statistical significance.
more of objection 1
The paper should rerun the entire LM2LM pipeline with at least 3 random seeds and report means and standard deviations or interval estimates for the key metrics in Table 1 and Fig. 3.
The paper says, at Sec. 3.2, 4.1
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The RWM is instantiated as GPT-6-sol, a proprietary model whose behavior on out-of-distribution proposals cannot be independently verified or reproduced. While the paper acknowledges this indirectly through the limitations section, the central claim depends on the RWM's predictive accuracy. The authors should either (a) include an open-source surrogate (e.g., a smaller open LLM) as an additional RWM to demonstrate that …
more of objection 2
… the method does not rely on a specific proprietary model, or (b) explicitly discuss reproducibility risks and provide a detailed plan for making the RWM component reproducible (e.g., releasing the prompt templates and the full set of in-context records used at each calibration stage).
The paper says, at Eq. (9), Sec. 3.3
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The on-policy calibration selects the top-rated proposals from the current policy. Because these proposals are also used to generate the next RWM records, this creates a potential feedback loop: the RWM's evaluation influences which proposals are executed, and those outcomes further influence the RWM.
more of objection 3
The ablation study (Fig. 4b) shows that random calibration is worse, but the paper does not analyze whether top-rated calibration could systematically under-sample proposals that are under-estimated by the RWM, potentially reducing exploration. A brief discussion of this asymmetry and its implications for the reliability of the RWM signal would strengthen the contribution.
4 minor objections
The paper says, at Sec. 4.2, Table 1
Pith says: minor objection 1 of 4
The win rates 'Row vs. Base' and 'LM2LM vs. Row' are computed from a single deterministic evaluation (four proposals per state per method). Because evaluation is deterministic given the proposals, the win rates are point estimates without variance. The paper should clarify that these are win rates over the fixed set of held-out states, not over resampling of proposals or seeds.
The paper says, at Sec. 4.4, Fig. 4
Pith says: minor objection 2 of 4
The two static variants use M0 (100 records) and M0' (M0 + 100 random records). The random records in M0' are drawn from the same pool of 300 pre-executed records. It would be helpful to state the selection procedure for these random records and whether they are stratified by task or mechanism direction.
show 2 more
The paper says, at Appendix E.3
Pith says: minor objection 3 of 4
The calibration batch sizes (B0=31, B1=38, B2=31) are reported without justification. The paper should at least note how these numbers were chosen (e.g., a fixed cost budget) or include a sensitivity analysis.
The paper says, at Sec. 4.5, Fig. 5
Pith says: minor objection 4 of 4
The advantage cosine and pairwise accuracy are reported only for AR tasks. The paper should show the corresponding metrics for diffusion tasks to confirm that the alignment holds across both modalities.
What would settle it
Run LM2LM training from scratch with at least 3 different random seeds (varying the base model's sampling seed, DAPO rollout seed, and calibration selection seed). Report mean ± std of Avg@4 and Best@4 on the held-out AR states at step 96. If the mean Avg@4 remains above 10 (compared to GPT-6-sol-ICL's 6.15) and the standard deviation is less than 30% of the mean, the concern is substantially mitigated.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 19 Oct 2026
- Reader holds up, with conditions
- Skeptic disagrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper asks how to train an LLM to propose good LM improvements without executing every proposal, and answers with a clean loop: use a frozen LLM as a research world model (RWM) to predict gains, reward the proposer with those predictions via DAPO, and periodically calibrate the RWM by running a handful of its top-rated proposals. That combination – RWM-based RL plus targeted real execution – is new, and the empirical work is what a good paper in this area should look like: real training runs on eight model families, held-out tasks, cost analysis, and ablations that isolate the calibration mechanism. The headline result – a 27B open model outperforming GPT-6-sol at 33% lower data-generation cost – is striking and, if it holds, practically important.
That said, there are two soft spots big enough to mention. First, every result in Section 4.2 comes from a single RL training run. No multiple seeds, no confidence intervals. Given the stochasticity in proposal sampling, RWM predictions, calibration selection, and DAPO updates, the observed margins could shift non-trivially with different random seeds. The paper shows no sign of this sensitivity, and the stress-test note is correct to flag it. Second, the RWM is GPT-6-sol, a proprietary model. The authors acknowledge the limitation, but it means the central claim is not independently reproducible and the method's generalizability to other LLM backbones is untested. The ablation that swaps initial records for random ones helps, but it does not address the RWM black box.
The paper is otherwise well executed. The calibration mechanism is clearly motivated and the case studies (e.g., overestimated optimizer changes) are convincing. The cost accounting is honest and puts LM2LM's savings in perspective. The writing is direct, the limitations section is fair, and the paper does not oversell.
Who should read this? Anyone working on AutoResearch, self-improving agents, or RL with learned rewards. It is a serious piece of work that makes a plausible case for a practical advance.
Recommendation for the editor: send to peer review. The single-run issue is real but fixable with a short appendix showing seed variance, and the RWM reliance is an acknowledged boundary condition, not a fatal flaw. A good referee will push on the statistical side and on whether the advantage holds with a more accessible RWM, but the paper deserves that conversation.
The strongest objection
The central claim that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even with 300 in-context records, rests on a single execution of the entire RL–calibration pipeline. Section 4.2 reports Avg@4, Best@4, and win rates from one training run (step 96); no multiple seeds or statistical intervals are provided. Given the stochasticity in proposal sampling, RWM predictions, calibration selection, and DAPO updates, the observed margins over baselines (e.g., AR Avg@4 17.06 vs. GPT-6-sol-ICL 6.15) could be inflated or deflated by chance. The paper's own ablation and analysis figures (Figs. 3–5) show only point estimates. Without quantifying seed variance, the robustness of the headline comparison is unknown.
The reader's weakest assumption focuses on the RWM's predictive accuracy as a reward signal. While that is a valid concern, it is partially addressed by on-policy calibration and the paper's own alignment analysis (Section 4.5). A more fundamental methodological concern is the absence of multiple training runs to assess variance. The paper reports a single RL run; given the complexity and stochasticity of the pipeline, the headline numerical advantage could be a lucky draw. This is not an attack on the authors' integrity but a call for standard reporting of variability. The reader's verdict of CONDITIONAL is already cautious, but this additional concern reinforces the need for the condition to include evidence of stability across seeds. Therefore, I recommend keeping CONDITIONAL, as the paper still needs this verification to fully support its central claim.
The referee's report
If the results are robust, LM2LM is a significant step toward scalable learning of research judgment for LM development. The core idea—using a frozen LLM as a world model that is periodically calibrated with a small number of real experiments—is novel and could reduce the cost of automated research. The evaluation spans multiple model families and includes held-out tasks and an unseen family generalization test. The paper provides detailed analysis of RWM alignment and the benefits of on-policy calibration.
Pith's reading of the paper (a machine's summary, not the authors')
The paper establishes that a language model can learn to generate effective proposals for improving other language models by training with reinforcement learning, where rewards come from a frozen LLM (the research world model) that predicts experimental gains without running costly training experiments. To keep the surrogate feedback reliable as the proposer evolves, the framework periodically executes the most highly rated proposals and adds their real outcomes to the world model's context. The central result is that this approach lets a 27-billion-parameter model (Qwen3.8-27B) outperform the frontier GPT-6-sol in generating improvement proposals across eight autoregressive and diffusion language model families, while requiring roughly one third less data-generation cost. The method also generalizes to a model family unseen during training, suggesting that learned research judgment can transfer beyond the original experimental context.
What carries the argument. The central machinery is the research world model (RWM), a frozen LLM (GPT-6-sol) that predicts the experimental gain of a proposal relative to a reference configuration, taking as input the task specification and a set of real experimental records provided in-context. These predictions are normalized and clipped to form rewards for the DAPO reinforcement-learning algorithm. To maintain reliability as the proposer changes, the framework introduces on-policy calibration: periodically, the proposals ranked highest by the current RWM are executed in reality, and their measured outcomes are added to the RWM's in-context records. The reward design also includes validity checks, repetition penalties, and a diversity bonus that encourages exploration of less common mechanism directions.
Core claim. The paper claims that LM2LM, a reinforcement-learning framework using a frozen research world model (RWM) as a reward function and on-policy calibration with real execution, enables a modestly sized proposer (Qwen3.8-27B) to outperform GPT-6-sol in generating language-model improvement proposals, even when GPT-6-sol is given 300 real experimental records as in-context examples. On eight held-out tasks across autoregressive and diffusion models, LM2LM achieves average gains of 17.06 × $10^{-3}$ BPB (autoregressive) and 7.10 × $10^{-2}$ nats per token (diffusion), the highest among all baselines, and does so at 33% lower data-generation cost. The framework also generalizes to the Qwen3 family, which was completely held out during training, achieving substantially higher gains than the strongest baseline. The authors argue that the RWM provides informative rankings of proposal gains and that on-policy calibration prevents reward overoptimization as the proposer's distribution shifts during training.
The premise it rests on. The frozen GPT-6-sol, used as the research world model, can predict the experimental gain of a proposal accurately enough to serve as a reliable reward signal for reinforcement learning, given a set of in-context experimental records; the paper provides evidence of alignment but relies on a proprietary model whose behavior on out-of-distribution proposals remains unknown.
If the paper is right (Pith's inferences, not the authors')
If the central claim holds, reinforcement learning with a research world model could dramatically reduce the cost of automated language-model development, making iterative model improvement accessible to more research groups.
The approach demonstrates a potential path toward recursive self-improvement, where one LLM learns to generate improvements for other LLMs without requiring thousands of expensive training runs.
The combination of surrogate reward and periodic on-policy calibration may apply to other scientific domains where experimental feedback is costly but surrogate predictions can be grounded in limited real data.
The framework suggests that learned research judgment can transfer across model families, implying that a proposer trained on cheaper models may generate useful proposals for larger or newer models.
On-policy calibration provides a concrete method to mitigate reward overoptimization in settings where the reward model's coverage is limited, with potential relevance to broader RLHF and RLVR pipelines.
The RWM's reliance on in-context records rather than fine-tuning means it can be updated without modifying model weights, but its predictive accuracy depends heavily on the quality and relevance of those records; the paper shows that random record selection can mislead the RWM, so targeted selection is critical.
The paper uses GPT-6-sol as both the RWM and as a baseline proposer, but does not test whether a smaller or open-weight world model would provide similarly informative rewards; if the RWM must itself be a frontier model, the cost and API dependency may limit broader adoption.
The evaluation focuses on proposal-level gains in a single step; extending the framework to multi-step research campaigns where proposals interact or compound could reveal additional challenges in credit assignment and reward design.
The measured gains, while statistically significant, are expressed in units of validation loss and may not directly translate to practical downstream task improvements; future work might test whether the proposed changes also improve performance on standard benchmarks.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 1.00
“Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.37 · human participants 0.03 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.96 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. | background 1.00 | 0.06 | 0.03 | asserted 0.55 |
| However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. | gap 0.71 | 0.13 | 0.02 | asserted 1.00 |
| First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. | background 0.78 | 0.15 | 0.02 | asserted 1.00 |
| Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. | background 0.93 | 0.28 | 0.02 | asserted 0.81 |
| Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. | background 0.80 | 0.11 | 0.02 | hedged possibility 0.53 |
| To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. | contribution 0.92 | 0.84 | 0.09 | asserted 0.92 |
| To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. | method 0.99 | 0.89 | 0.04 | asserted 0.99 |
| To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. | method 0.99 | 0.91 | 0.03 | asserted 0.96 |
| Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. | result 1.00 | 0.95 | 0.77 | asserted 1.00 |
| LM2LM also generalizes to a model family unseen during training. | result 1.00 | 0.91 | 0.12 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,591 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Showing 11 of the paper's 12 figures. Figures are the authors', reproduced from the paper.
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment