Pith.

Computation and Language · arXiv:2610.23344 v2 · submitted 16 Oct 2026

Language Models Learn to Improve Language Models

Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang

show 2 more authors

Suhang Wang, Hanqing Lu

The paper says

“Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges.”

more of the abstract

First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 19 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. LM2LM combines a frozen world-model reward with on-policy calibration to train a proposer that beats a frontier model at lower cost. The idea is solid and the experiments are thorough, but the single-run results and reliance on GPT-6-sol leave open questions about reproducibility and statistical robustness. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Sec. 4.2, Table 1, Fig. 3–5

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

All main results (Avg@4, Best@4, win rates, ablation studies) are reported from a single RL training run. DAPO training involves stochasticity in proposal sampling, RWM predictions, calibration proposal selection, and weight updates. Without multiple seeds (or at least confidence intervals from bootstrapping the held-out states, which the paper already does for some analysis in Appendix I but not for the headline numbers), the claimed margins over baselines—especially the headline claim that Qwen3.8-27B outperforms GPT-6-sol—cannot be assessed for statistical significance.

more of objection 1

The paper should rerun the entire LM2LM pipeline with at least 3 random seeds and report means and standard deviations or interval estimates for the key metrics in Table 1 and Fig. 3.

The paper says, at Sec. 3.2, 4.1

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The RWM is instantiated as GPT-6-sol, a proprietary model whose behavior on out-of-distribution proposals cannot be independently verified or reproduced. While the paper acknowledges this indirectly through the limitations section, the central claim depends on the RWM's predictive accuracy. The authors should either (a) include an open-source surrogate (e.g., a smaller open LLM) as an additional RWM to demonstrate that …

more of objection 2

… the method does not rely on a specific proprietary model, or (b) explicitly discuss reproducibility risks and provide a detailed plan for making the RWM component reproducible (e.g., releasing the prompt templates and the full set of in-context records used at each calibration stage).

The paper says, at Eq. (9), Sec. 3.3

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The on-policy calibration selects the top-rated proposals from the current policy. Because these proposals are also used to generate the next RWM records, this creates a potential feedback loop: the RWM's evaluation influences which proposals are executed, and those outcomes further influence the RWM.

more of objection 3

The ablation study (Fig. 4b) shows that random calibration is worse, but the paper does not analyze whether top-rated calibration could systematically under-sample proposals that are under-estimated by the RWM, potentially reducing exploration. A brief discussion of this asymmetry and its implications for the reliability of the RWM signal would strengthen the contribution.

4 minor objections

The paper says, at Sec. 4.2, Table 1

Pith says: minor objection 1 of 4

The win rates 'Row vs. Base' and 'LM2LM vs. Row' are computed from a single deterministic evaluation (four proposals per state per method). Because evaluation is deterministic given the proposals, the win rates are point estimates without variance. The paper should clarify that these are win rates over the fixed set of held-out states, not over resampling of proposals or seeds.

The paper says, at Sec. 4.4, Fig. 4

Pith says: minor objection 2 of 4

The two static variants use M0 (100 records) and M0' (M0 + 100 random records). The random records in M0' are drawn from the same pool of 300 pre-executed records. It would be helpful to state the selection procedure for these random records and whether they are stratified by task or mechanism direction.

show 2 more

The paper says, at Appendix E.3

Pith says: minor objection 3 of 4

The calibration batch sizes (B0=31, B1=38, B2=31) are reported without justification. The paper should at least note how these numbers were chosen (e.g., a fixed cost budget) or include a sensitivity analysis.

The paper says, at Sec. 4.5, Fig. 5

Pith says: minor objection 4 of 4

The advantage cosine and pairwise accuracy are reported only for AR tasks. The paper should show the corresponding metrics for diffusion tasks to confirm that the alignment holds across both modalities.

What would settle it

Run LM2LM training from scratch with at least 3 different random seeds (varying the base model's sampling seed, DAPO rollout seed, and calibration selection seed). Report mean ± std of Avg@4 and Best@4 on the held-out AR states at step 96. If the mean Avg@4 remains above 10 (compared to GPT-6-sol-ICL's 6.15) and the standard deviation is less than 30% of the mean, the concern is substantially mitigated.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 19 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic disagrees 50 s · 32,025 tokens read, 4,685 written
  3. Editor wrote the verdict above 16 s · 32,830 tokens read, 1,452 written
  4. Referee major revision: 3 major objections, 4 minor 22 s · 32,353 tokens read, 2,179 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper asks how to train an LLM to propose good LM improvements without executing every proposal, and answers with a clean loop: use a frozen LLM as a research world model (RWM) to predict gains, reward the proposer with those predictions via DAPO, and periodically calibrate the RWM by running a handful of its top-rated proposals. That combination – RWM-based RL plus targeted real execution – is new, and the empirical work is what a good paper in this area should look like: real training runs on eight model families, held-out tasks, cost analysis, and ablations that isolate the calibration mechanism. The headline result – a 27B open model outperforming GPT-6-sol at 33% lower data-generation cost – is striking and, if it holds, practically important.

That said, there are two soft spots big enough to mention. First, every result in Section 4.2 comes from a single RL training run. No multiple seeds, no confidence intervals. Given the stochasticity in proposal sampling, RWM predictions, calibration selection, and DAPO updates, the observed margins could shift non-trivially with different random seeds. The paper shows no sign of this sensitivity, and the stress-test note is correct to flag it. Second, the RWM is GPT-6-sol, a proprietary model. The authors acknowledge the limitation, but it means the central claim is not independently reproducible and the method's generalizability to other LLM backbones is untested. The ablation that swaps initial records for random ones helps, but it does not address the RWM black box.

The paper is otherwise well executed. The calibration mechanism is clearly motivated and the case studies (e.g., overestimated optimizer changes) are convincing. The cost accounting is honest and puts LM2LM's savings in perspective. The writing is direct, the limitations section is fair, and the paper does not oversell.

Who should read this? Anyone working on AutoResearch, self-improving agents, or RL with learned rewards. It is a serious piece of work that makes a plausible case for a practical advance.

Recommendation for the editor: send to peer review. The single-run issue is real but fixable with a short appendix showing seed variance, and the RWM reliance is an acknowledged boundary condition, not a fatal flaw. A good referee will push on the statistical side and on whether the advantage holds with a more accessible RWM, but the paper deserves that conversation.

The strongest objection

The central claim that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even with 300 in-context records, rests on a single execution of the entire RL–calibration pipeline. Section 4.2 reports Avg@4, Best@4, and win rates from one training run (step 96); no multiple seeds or statistical intervals are provided. Given the stochasticity in proposal sampling, RWM predictions, calibration selection, and DAPO updates, the observed margins over baselines (e.g., AR Avg@4 17.06 vs. GPT-6-sol-ICL 6.15) could be inflated or deflated by chance. The paper's own ablation and analysis figures (Figs. 3–5) show only point estimates. Without quantifying seed variance, the robustness of the headline comparison is unknown.

The reader's weakest assumption focuses on the RWM's predictive accuracy as a reward signal. While that is a valid concern, it is partially addressed by on-policy calibration and the paper's own alignment analysis (Section 4.5). A more fundamental methodological concern is the absence of multiple training runs to assess variance. The paper reports a single RL run; given the complexity and stochasticity of the pipeline, the headline numerical advantage could be a lucky draw. This is not an attack on the authors' integrity but a call for standard reporting of variability. The reader's verdict of CONDITIONAL is already cautious, but this additional concern reinforces the need for the condition to include evidence of stability across seeds. Therefore, I recommend keeping CONDITIONAL, as the paper still needs this verification to fully support its central claim.

The referee's report

If the results are robust, LM2LM is a significant step toward scalable learning of research judgment for LM development. The core idea—using a frozen LLM as a world model that is periodically calibrated with a small number of real experiments—is novel and could reduce the cost of automated research. The evaluation spans multiple model families and includes held-out tasks and an unseen family generalization test. The paper provides detailed analysis of RWM alignment and the benefits of on-policy calibration.

Pith's reading of the paper (a machine's summary, not the authors')

The paper establishes that a language model can learn to generate effective proposals for improving other language models by training with reinforcement learning, where rewards come from a frozen LLM (the research world model) that predicts experimental gains without running costly training experiments. To keep the surrogate feedback reliable as the proposer evolves, the framework periodically executes the most highly rated proposals and adds their real outcomes to the world model's context. The central result is that this approach lets a 27-billion-parameter model (Qwen3.8-27B) outperform the frontier GPT-6-sol in generating improvement proposals across eight autoregressive and diffusion language model families, while requiring roughly one third less data-generation cost. The method also generalizes to a model family unseen during training, suggesting that learned research judgment can transfer beyond the original experimental context.

What carries the argument. The central machinery is the research world model (RWM), a frozen LLM (GPT-6-sol) that predicts the experimental gain of a proposal relative to a reference configuration, taking as input the task specification and a set of real experimental records provided in-context. These predictions are normalized and clipped to form rewards for the DAPO reinforcement-learning algorithm. To maintain reliability as the proposer changes, the framework introduces on-policy calibration: periodically, the proposals ranked highest by the current RWM are executed in reality, and their measured outcomes are added to the RWM's in-context records. The reward design also includes validity checks, repetition penalties, and a diversity bonus that encourages exploration of less common mechanism directions.

Core claim. The paper claims that LM2LM, a reinforcement-learning framework using a frozen research world model (RWM) as a reward function and on-policy calibration with real execution, enables a modestly sized proposer (Qwen3.8-27B) to outperform GPT-6-sol in generating language-model improvement proposals, even when GPT-6-sol is given 300 real experimental records as in-context examples. On eight held-out tasks across autoregressive and diffusion models, LM2LM achieves average gains of 17.06 × $10^{-3}$ BPB (autoregressive) and 7.10 × $10^{-2}$ nats per token (diffusion), the highest among all baselines, and does so at 33% lower data-generation cost. The framework also generalizes to the Qwen3 family, which was completely held out during training, achieving substantially higher gains than the strongest baseline. The authors argue that the RWM provides informative rankings of proposal gains and that on-policy calibration prevents reward overoptimization as the proposer's distribution shifts during training.

The premise it rests on. The frozen GPT-6-sol, used as the research world model, can predict the experimental gain of a proposal accurately enough to serve as a reliable reward signal for reinforcement learning, given a set of in-context experimental records; the paper provides evidence of alignment but relies on a proprietary model whose behavior on out-of-distribution proposals remains unknown.

If the paper is right (Pith's inferences, not the authors')

If the central claim holds, reinforcement learning with a research world model could dramatically reduce the cost of automated language-model development, making iterative model improvement accessible to more research groups.

The approach demonstrates a potential path toward recursive self-improvement, where one LLM learns to generate improvements for other LLMs without requiring thousands of expensive training runs.

The combination of surrogate reward and periodic on-policy calibration may apply to other scientific domains where experimental feedback is costly but surrogate predictions can be grounded in limited real data.

The framework suggests that learned research judgment can transfer across model families, implying that a proposer trained on cheaper models may generate useful proposals for larger or newer models.

On-policy calibration provides a concrete method to mitigate reward overoptimization in settings where the reward model's coverage is limited, with potential relevance to broader RLHF and RLVR pipelines.

The RWM's reliance on in-context records rather than fine-tuning means it can be updated without modifying model weights, but its predictive accuracy depends heavily on the quality and relevance of those records; the paper shows that random record selection can mislead the RWM, so targeted selection is critical.

The paper uses GPT-6-sol as both the RWM and as a baseline proposer, but does not test whether a smaller or open-weight world model would provide similarly informative rewards; if the RWM must itself be a frontier model, the cost and API dependency may limit broader adoption.

The evaluation focuses on proposal-level gains in a single step; extending the framework to multi-step research campaigns where proposals interact or compound could reveal additional challenges in credit assignment and reward design.

The measured gains, while statistically significant, are expressed in units of validation loss and may not directly translate to practical downstream task improvements; future work might test whether the proposed changes also improve performance on standard benchmarks.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 1.00

“Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.37 · human participants 0.03 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.96 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. background 1.00 0.06 0.03 asserted 0.55
However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. gap 0.71 0.13 0.02 asserted 1.00
First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. background 0.78 0.15 0.02 asserted 1.00
Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. background 0.93 0.28 0.02 asserted 0.81
Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. background 0.80 0.11 0.02 hedged possibility 0.53
To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. contribution 0.92 0.84 0.09 asserted 0.92
To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. method 0.99 0.89 0.04 asserted 0.99
To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. method 0.99 0.91 0.03 asserted 0.96
Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. result 1.00 0.95 0.77 asserted 1.00
LM2LM also generalizes to a model family unseen during training. result 1.00 0.91 0.12 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,591 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 2 of the paper
Figure 2. Overview of LM2LM. 3 Methodology In this section, we present LM2LM. We first for￾malize AutoResearch for LM development and mo￾tivate RL-based proposer training (Sec. 3.1). To provide scalable feedback, we use a frozen LLM as a research world model (RWM) to predict ex￾perimental gains without executing proposals, and use its predictions as RL rewards (Sec. 3.2). To maintain reliable feedback as the proposer evolves, we periodically execute high-rated proposals from the current policy and incorporate their measured outcomes into the RWM’s context (Sec. 3.3). An overview of LM2LM is…
Figure 3 of the paper
Figure 3. Real experimental gains (Avg@4) versus data-generation cost and time on held-out research states. (a, b)
Figure 4 of the paper
Figure 4. Ablation of on-policy calibration on held￾out AR research states at step 96. performing the strongest baseline (21.63 and 25.27 for GPT-6-sol-ICL), demonstrating its ability to generate effective proposals for an unseen model family. (ii) Out-of-family in-context records pro￾vide limited benefits. Base-ICL performs similarly to Base (2.97 vs. 3.53 Avg@4), while GPT-6-sol￾ICL only slightly improves over GPT-6-sol (21.63 vs. 20.19). This suggests that directly using out-of￾family records offers limited benefits, highlighting the advantage of LM2LM’s RWM-based RL. 4.4 Ablation Studies…
Figure 5 of the paper
Figure 5. RWM feedback reliability on held-out AR research states. lection can miss them or add misleading records. Case studies are in Appendix H. 4.5 In-depth Dissection of LM2LM We further analyze why RWM feedback can guide RL and how calibration keeps it reliable. For this analysis, we extend training from 96 to 128 steps with a fourth calibration of LM2LM (236 real runs) and compare it with the static variant whose RWM keeps M0 throughout; each variant is scored by the RWM it is trained with. Why Can RWM Feedback Guide RL? We ex￾amine the alignment between RWM-predicted and…
Figure 6 of the paper
Figure 6. Real experimental gains (Best@4) versus data-generation cost and time on held-out research states. (a, b) show AR tasks in 10−3 BPB, while (c, d) show diffusion tasks in 10−2 nats per token
Figure 7 of the paper
Figure 7. M-UPD-72 under top-rated and random calibration. (a) Fraction of NanoChat training pro￾posals that apply it. (b) Mean RWM reward of these proposals in the training reward scale; the dashed line is their real reward on held-out states. C1–C3 mark the three calibrations. Misleading Records from Random Selection. Random calibration can also add records that mis￾lead the RWM. Its first batch contains an OLMo change that replaces local attention with an alterna￾tive token-mixing operator, which happens to im￾prove its training task by 1.43%. With this record, the RWM starts rewarding the…
Figure 8 of the paper
Figure 8. OLMo token-mixing change under top￾rated and random calibration. (a) Fraction of OLMo training proposals that apply it. (b) Mean RWM reward of these proposals in the training reward scale. C1–C3 mark the three calibrations. der top-rated calibration to −6.37. Under top-rated calibration, the RWM keeps penalizing the change (−1.2 to −0.5), and its share stays below 5%. I Additional In-depth Analysis I.1 Diagnosing RWM Feedback Reliability Feedback Metric. At each checkpoint, we exe￾cute the four proposals of each of the 60 held-out AR states, whose outcomes never enter the records.…
Figure 9 of the paper
Figure 9. Additional feedback diagnostics on held-out AR states through step 128. Each variant is evaluated using its own training RWM. (d) reports the mean real advantage of proposals with positive predicted advantage. Error bars indicate 95% bootstrap confidence intervals over states. 0 32 64 96 128 Training Step 0.4 0.6 0.8 1.0 Pairwise Accuracy random C1 C2 C3 C4 Initial records Calibrated records
Figure 10 of the paper
Figure 10. Pairwise accuracy of the same proposals of LM2LM scored with the initial records M0 and with the latest calibrated records (M2 up to step 64, M3 at step 96, and M4 at step 128). 23
Figure 11 of the paper
Figure 11. Predicted versus real advantages of the proposer’s own proposals on held-out AR states, at steps 16, 64, and 128. Top: static variant scored by W(M0); bottom: LM2LM scored by the RWM in force. Points on the dotted diagonal are predicted exactly; the shaded region contains wrongly reinforced proposals (positive predicted, negative real advantage). Each panel reports the pairwise accuracy (acc) and wrong-reinforcement rate (wrong). Values are clipped to ±4.5 for display. 16 32 48 64 Training Step 0 25 50 75 100 Fraction (%) C1 C2 Static LM2LM (a) Training share. 16 32 48 64 Training…
Figure 12 of the paper
Figure 12. Case study of the NanoChat change M-UPD-72 (tokens per optimizer update) for the static variant and LM2LM. (a) Fraction of NanoChat training proposals that apply it. (b) Mean reward the training RWM assigns to these proposals; the dashed line is their real reward on held-out states. (c) Fraction of held-out NanoChat proposals that apply it. C1 and C2 mark the first two calibrations of LM2LM. 24

Showing 11 of the paper's 12 figures. Figures are the authors', reproduced from the paper.

Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment