Computation and Language · arXiv:2610.25836 v1 · submitted 19 Oct 2026 · 51 references
Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining
The paper says
“Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do.”
more of the abstract
This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure n-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3–4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by 2.0 points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by 1.9 points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A clean, honest paper that does something sensible with RL failure signals—modest effects, transparent limitations, worth a serious read. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 51 references say.
The paper says, at Section 4.4, Table 3, Table 1, Table 14
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.93):
“What the repair changes most is therefore how often the model emits these phrases, so its accuracy gain cannot be attributed to the diagnosed states alone.”
Pith says: Objection 1 of 2 · major
The paper's narrative centers on repairing 'low-value states' marked by failure n-grams, but the evidence suggests the primary effect is increasing how often the model enters those states, not how often it succeeds from them. For example, in Table 1 (OLMo2-1B/GSM8K), the conditional accuracy a(g) of 'there's a mistake' moves from 6.2% to 8.1% and 'Let's recheck' from 5.6% to 6.3%, while the entry count for 'Let's recheck' triples.
more of objection 1
In Table 14, the success rate from the full failure set rises modestly (25.7→29.2) while the entry rate nearly doubles (5.7→10.6). The paper acknowledges this in a qualifying observation (Sec. 4.4, paragraph 3: 'the accuracy gain cannot be attributed to the diagnosed states alone'), but the central claim that the method 'repairs' the marked states is overstated. The evidence better supports the interpretation that CPT increases the model's propensity to emit the trigger phrases, with a secondary and smaller effect on the probability of recovery. The authors should either adjust the mechanistic claim or provide additional analysis (e.g., isolating the effect on states that are truly the bottleneck) to justify the stronger narrative.
The paper says, at Section 4.1, Appendix D, Limitations paragraph
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):
“Stronger conclusions require more seeds and completed runs.”
Pith says: Objection 2 of 2 · major
Statistical significance of the main results is weak. Only the OLMo2-1B GSM8K condition uses three RL seeds; all other comparisons are single-seed. The reported standard deviation of ~1 point on GSM8K means that most individual gains (0.1–2.6 points) are within two standard deviations.
more of objection 2
The paper acknowledges this limitation ('Stronger conclusions require more seeds and completed runs'), but the abstract and introduction state the gains as if they are established. I recommend that the authors add a summary statement of statistical reliability, e.g., marking which comparisons exceed 2σ or reporting the fraction of gains that are larger than the typical within-condition noise. This would help the reader calibrate confidence in the results.
7 minor objections
The paper says, at Abstract, line 3
Pith says: minor objection 1 of 7
The phrase 'whether a rollout fails is strongly predicted by a small set of characteristic n-grams' could be misinterpreted as causal. Consider rewording to 'is correlated with' or 'is strongly associated with' to match the associational framing in Section 3.2.
The paper says, at Section 3.2
Pith says: minor objection 2 of 7
Equation (2) defines conditional accuracy a(g) = Pr[r=1 | E_g(y)=1]. This is a conditional probability, not an expectation. The text says 'measures how strongly g marks a low-accuracy state'. It may help to explicitly note that this is a population-level association, not a causal effect, since a rollout containing g is a consequence of the state, not an intervention.
show 5 more
The paper says, at Table 2, Table 4, Table 5
Pith says: minor objection 3 of 7
The tables use numerical formatting that is sometimes ambiguous (e.g., '38.2' appears as a single column value). Consider aligning numbers on the decimal point and adding a note that bold or italic indicates the best in each row, to improve readability.
The paper says, at Appendix C, Table 7
Pith says: minor objection 4 of 7
The retrieval queries for the matched control include '1', 'first', 'step 1'. These are not truly content-neutral; they select text that might be structurally similar to reasoning traces. It would strengthen the control to use queries that are equally frequent but known to have no relevance to reasoning, such as article-specific stop words. The current choice may underestimate the baseline.
The paper says, at Section 4.5, Table 5
Pith says: minor objection 5 of 7
The finding that a 1B token budget on the General Reasoner setting lowers performance by 19 points is striking. The paper attributes this to 'why large CPT budgets hurt' but does not analyze it. A brief discussion (e.g., catastrophic forgetting, distribution shift) would be helpful.
The paper says, at Figure 2
Pith says: minor objection 6 of 7
The figure uses a 30-step moving average. For the first panel (accuracy), the starting point at step 100 is 35%, but the text says training-rollout accuracy rises from 0.2% to 47%. Clarify that the 0.2% refers to early training, not the plotted window.
The paper says, at General
Pith says: minor objection 7 of 7
Several citations are incomplete or use 'et al.' inconsistently. For example, reference 'Akter et al., 2026' should include the full author list in the bibliography. Check that all citations follow the journal style.
What would settle it
Run three additional RL seeds for the OLMo2-1B multi-task after-stage-1 condition and for the Qwen3-4B General Reasoner self-mined corpus condition. For each, compute the greedy accuracy advantage over the matched control. If the advantage is negative or below 0.3 points in two or more of these six runs, the claim of consistent directional gains would be materially weakened.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 20 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Xinyi and Danqi have written a paper worth taking seriously. The core idea is straightforward, and they execute it cleanly: use failure n-grams from RL rollouts as retrieval queries over the pretraining corpus, continue-pretrain on the retrieved 35M tokens, and re-run RL. It's a cheap, label-free way to target the base model's weak spots before RL, and it works—10 of 11 settings beat a matched control.
What's new is the combination. Prior work used failure tokens to modify the RL objective (Ko et al., 2026) or hand-picked mid-training data by task similarity. Here the failure signal itself selects the data. The paper also shows cross-family transfer (OLMo2-1B failures helping Qwen3-4B) and that insertion before mid-training gives the largest gain. The experiments are controlled (same pipeline, same budget, content-neutral queries as baseline), and the honest limitations section is a model of good practice.
The soft spots are real but acknowledged. Gains are modest (0.1–2.6 points), most results are single-seed, and the diagnosis is associational—the n-grams mark low-value states but don't cause them. The matched control often shrinks the gain, especially on released base models. And the method depends on open pretraining corpora; for Qwen3 they use Olmo's corpora, which conflates corpus transfer with the repair. These are genuine constraints, not fatal flaws.
Bottom line: the paper is a solid empirical contribution to the practical question of how to improve RL for language models. The effect sizes are too small for strong conclusions about the repair mechanism, but the pipeline is reproducible and the idea is useful. I'd bring it to a reading group and would cite it if I worked on RL data selection. A serious editor should send this to peer review—it deserves referee time.
The strongest objection
The paper honestly documents its limitations: modest effect sizes, single-seed comparisons for most settings, and an associational diagnosis that does not establish causality. The reader's weakest_assumption—that exact-match n-gram retrieval supplies repair data despite being only associational—is directly acknowledged and partially addressed by the matched control. The central claim of consistent directional gains over matched controls is supported by the reported pattern but not by strong statistical evidence in most individual comparisons. However, the paper does not overclaim; it explicitly states that stronger conclusions require more seeds. Given this transparency and the controlled experimental design, I do not identify an unaddressed load-bearing concern that undermines the core empirical contribution.
The reader's weakest_assumption correctly identifies that the retrieval signal is associational, not causal, and may not directly repair the marked failure states. The paper acknowledges this limitation and uses a matched control to isolate the effect of the chosen text from the effect of continued pretraining itself. The reader's verdict (ACCEPT, moderate confidence) is proportionate to the evidence. My stress-test does not reveal an unaddressed vulnerability that warrants a change. The statistical concerns are honestly flagged in the Limitations section, and the paper does not assert strong significance. The consistency across 10 of 11 settings, even if some are single-seed, supports the practical utility of the method. No revision to the verdict is needed.
The referee's report
If the results hold, this is a significant contribution: it introduces a cheap, label-free, and model-specific signal for selecting pretraining data to patch RL failures, operating at 3–4 orders of magnitude smaller scale than standard mid-training. The transparent experimental design—matched controls, ablation of token budget, insertion point comparisons, and explicit acknowledgement of limitations—adds credibility. However, the modest effect sizes (0.1–2.6 points), the single-seed nature of most comparisons, and the partial disconnect between the claimed mechanism (repairing low-value states) and the observed effects (increased entry rate rather than success rate) temper the immediate impact. The paper opens a promising direction for data selection that could be strengthened by future work with more seeds and deeper causal analysis.
Pith's reading of the paper (a machine's summary, not the authors')
This paper shows that when reinforcement learning with verifiable rewards (RLVR) almost works but still fails, the failures are not random. They concentrate on a small set of characteristic n-grams in the model's own rollouts—phrases like "let's recheck" or "the inverse of A". These n-grams mark low-value states from which the model rarely recovers, and RL itself does little to repair them because it learns only from the few successful continuations it samples there. The authors use these failure n-grams to retrieve a tiny 35M-token corpus from the pretraining data, continue pretraining the base model on it, and then re-run RL. This targeted corpus consistently outperforms continued pretraining on the same amount of generic pretraining text, with gains up to 2.6 percentage points, and the corpus transfers across model families. If the paper is right, a model's own failures are a cheap, label-free signal for selecting the pretraining data it most needs.
What carries the argument. The key mechanism is failure-n-gram-guided continued pretraining. First, failure n-grams are mined from RL rollouts of the trained policy; they are n-grams (length 3–6) whose presence in a rollout predicts low accuracy. Second, these n-grams are used as exact-match queries via an infini-gram index over the pretraining and mid-training corpora, and each match is extracted with a 2000-token window. A balanced budget of 35M tokens is sampled. Third, the base model is continue-pretrained on this corpus before RL is re-run with identical hyperparameters. The matched control uses content-neutral queries (e.g., 'step 1') to separate the effect of targeted selection from the effect of continued pretraining itself.
Core claim. The central discovery is that RL failures are localized in a small set of n-grams that strongly predict rollout failure, and that using these n-grams to retrieve and continue pretrain on text from the pretraining corpus consistently improves final RL accuracy over matched controls. The failure n-grams vary by model strength and task: reflection phrases for weak models on elementary math, specific operations for stronger models on broad reasoning. The retrieved corpus is tiny (35M tokens, 3–4 orders of magnitude smaller than typical mid-training) yet yields consistent gains. The paper shows that the repair works primarily by making the model enter the marked states more often and succeed from them slightly more often, though it acknowledges that the accuracy gain cannot be fully attributed to the diagnosed states alone.
The premise it rests on. The method assumes that exact-match n-gram retrieval from the pretraining corpus provides text that teaches the model how to recover from the failure states marked by those n-grams, even though the n-grams are only associational markers and the retrieved spans often lack a resolution of the failure.
If the paper is right (Pith's inferences, not the authors')
Inserted before mid-training, the targeted corpus yields larger gains (+2.0 on GSM8K) than when inserted after mid-training (+0.6), and generic text at the same budget can hurt.
The corpus mined from one model family (OLMo2) improves RL for another (Qwen3), suggesting that failure signals generalize across model architectures and pretraining corpora.
For missing skills (e.g., operations), the repair raises success rates from near 0% to 50–62%, whereas for behavioral deficits (reflection phrases) the increase is only a few percentage points, indicating skill deficits are more straightforwardly repaired.
The approach works with as little as 35M tokens; scaling to 1B tokens can degrade performance, confirming that targeted selection matters more than volume.
The repair makes the model enter the failure states more often and succeed from them at a slightly higher rate, but the primary effect is increasing exposure to the failure contexts rather than teaching recovery from them.
The diagnose-repair loop could be iterated: after the first repair, new failure n-grams may emerge, and a second round of retrieval-and-CPT might push performance further; the paper leaves this for future work.
Semantic retrieval (embedding-based) seeded by failure n-grams could capture paraphrases and alternative notations that exact match misses, potentially improving recall for skill deficits.
The technique may be applicable beyond RLVR to any setting where a model's failures are localized and a pretraining corpus is available, such as safety alignment or domain adaptation failures.
The finding that failure n-grams transfer across models suggests that there exist universal failure points in reasoning tasks that are shared across model families, which could inform curriculum design.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure n-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3–4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by 2.0 points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by 1.9 points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 1.00
“Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.”
How the main claim is stated: hedged possibility 0.98
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.27 · human participants 0.02 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. | background 0.98 | 0.07 | 0.06 | hedged possibility 0.56 |
| This raises a practical question: when RL fails, what is the base model missing? | gap 0.89 | 0.05 | 0.02 | unclear 0.71 |
| We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". | result 0.98 | 0.96 | 0.10 | asserted 1.00 |
| These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. | result 0.98 | 0.91 | 0.04 | asserted 0.99 |
| Building on this observation, we use the failure $n$-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. | method 0.98 | 0.95 | 0.07 | asserted 0.99 |
| The resulting corpus has only 35M tokens, 3--4 orders of magnitude smaller than typical mid-training. | method 0.87 | 0.93 | 0.03 | asserted 1.00 |
| We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. | method 0.92 | 0.95 | 0.03 | asserted 0.90 |
| Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by $2.0$ points on GSM8K when inserted before mid-training. | result 1.00 | 0.97 | 0.20 | asserted 1.00 |
| The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by $1.9$ points over the same baseline. | result 1.00 | 0.96 | 0.08 | asserted 1.00 |
| Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data. | interpretation 1.00 | 0.84 | 0.92 | hedged possibility 0.98 |
What kind of paper the classifier reports this is: method or tool 0.95 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,840 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Figures are the authors', reproduced from the paper.
References (51)
- [1] Istabrak Abbes, Gopeshh Subbaraj, Matthew Riemer, Nizar Islah, Tsuguchika Tabaru, Hiroaki Kingetsu, Sarath Chandar, and Irina Rish. Revisiting replay and gradient alignment for continual pre-training of large language models. In Proceedings of The 4th Conference on Lifelong Le... on Pith
- [2] Syeda Nahida Akter, Shrimai Prabhumoye, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, Yejin Choi, and Bryan Catanzaro. Front-loading reasoning: The synergy between pretraining and post-training data. In The Fourteenth International Conference on Learning Representations, 202... on Pith
- [3] Pengxiang Cai, Tianchen Fang, Xiaohan Li, Qingyuan Zeng, Guocong Li, and Jintai Chen. Curriculum reinforcement learning can incentivize reasoning capacity in LLMs beyond the base model. arXiv preprint arXiv:2606.22317, 2026 on PithPith's reviewarXiv
- [4] Mayee F Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher R \'e , Luca Soldaini, and Kyle Lo. Olmix: A framework for data mixing throughout LM development. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview... on Pith
- [5] Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, and Victor Zhong. Atomic skills are the prerequisite: When reinforcement learning synthesizes compositional reasoning, and when it only amplifies. arXiv preprint arXiv:2512.01970, 2025 on PithPith's reviewarXiv
- [6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021 on PithPith's reviewarXiv
- [7] Dylan J. Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? The computational role of the base model in exploration. In COLT, pp.\ 2026--2142, 2025. URL https://proceedings.mlr.press/v291/foster25a.html on Pith
- [8] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective STaRs . In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=QGJ9ttXLTy on Pith
- [9] Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645 0 (8081): 0 633--638, 2025. doi:10.1038/s41586-025-09422-z on Pithdoi:10.1038/s41586-025-09422-z
- [10] Kshitij Gupta, Benjamin Th \'e rien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timoth \'e e Lesort. Continual pre-training of large language models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Mo... on Pith
- [11] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ on Pith
- [12] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.01798 on PithPith's reviewarXiv
- [13] Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu, Xing Sun, and Weinan Zhang. ReMiT : RL -guided mid-training for iterative LLM evolution. arXiv preprint arXiv:2602.03075, 2026 on PitharXiv
- [14] Prateek Humane, Paolo Cudrano, Daniel Z Kaplan, Matteo Matteucci, Supriyo Chakraborty, and Irina Rish. Influence functions for efficient data selection in reasoning. In NeurIPS 2025 Workshop on Efficient Reasoning, 2025. URL https://openreview.net/forum?id=wQyK6NEpoy on Pith
- [15] Adam Ibrahim, Benjamin Th \'e rien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \'e e Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024... on Pith
- [16] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA : A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1... on Pith
- [17] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs . Transactions of the Association for Computational Linguistics (TACL), 2024. arXiv:2406.01297 on PithPith's reviewarXiv
- [18] Jaeyong Ko, Pilsung Kang, and Yukyung Lee. Cliff tokens: Identifying single-token failure triggers in LLM mathematical reasoning. arXiv preprint arXiv:2606.25524, 2026 on PithPith's reviewarXiv
- [19] Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Evander Yang, Kun Li, Haoyuan Wu, Jiajia Wu, Zihao Zheng, Chenchen Zhang, et al. Reinforcement learning on pre-training data. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo... on Pithdoi:10.18653/v1/2026.acl-long.506
- [20] Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=u2vAyMeLMm on Pith
show 31 more
- [21] Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL : Prolonged reinforcement learning expands reasoning boundaries in large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL... on Pith
- [22] Quanyu Long, Kai Jie Jiang, Jianda Chen, Xu Guo, Leilei Gan, and Wenya Wang. Self-verification dilemma: Experience-driven suppression of overused checking in LLM reasoning. arXiv preprint arXiv:2602.03485, 2026 on PithPith's reviewarXiv
- [23] Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun MA, and Wenhu Chen. General-Reasoner : Advancing LLM reasoning across all domains. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=pBFVoll8Xa on Pith
- [24] Aswin RRV, Jacob Dineen, Divij Handa, Mihir Parmar, Ben Zhou, Swaroop Mishra, and Chitta Baral. Mid-training with self-generated data improves reinforcement learning in language models. arXiv preprint arXiv:2605.08472, 2026 on PithPith's reviewarXiv
- [25] Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rockt\"aschel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth Internati... on Pith
- [26] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa : Commonsense reasoning about social interactions. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural... on Pith
- [27] Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training. arXiv preprint arXiv:2504.04022, 2025 on PithPith's reviewarXiv
- [28] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in RLVR . In Forty-th... on Pith
- [29] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. URL https://doi.org/10.48550/arXiv.2402.03300 on PithPith's reviewarXivdoi:10.48550/arxiv.2402.03300
- [30] Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, and Pavel Izmailov. Understanding reasoning from pretraining to post-training: Chess as a controlled testbed. In The Fortieth Annual Conference on Neural Information Processing Systems, 2026. URL ... on Pith
- [31] Peiyang Song, Pengrui Han, and Noah Goodman. Large language model reasoning failures. Transactions on Machine Learning Research, 2026. arXiv:2602.06176 on PitharXiv
- [32] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models. Transactions on Machine Learning Research (TMLR), 2025.... on PithPith's reviewarXiv
- [33] Team Olmo , Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025 on PithPith's reviewarXiv
- [34] Yijun Tian, Shaoyu Chen, Zhichao Xu, Yawei Wang, Jinhe Bi, Peng Han, and Wei Wang. Reinforcement mid-training. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=uJUhi3FQNa on Pith
- [35] Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, et al. 2 OLMo 2 furious. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=2ezugTT9kU on Pith
- [36] Liang Wang, Nan Yang, Shaohan Huang, Li Dong, and Furu Wei. Thinking augmented pre-training. arXiv preprint arXiv:2509.20186, 2025 a on PitharXiv
- [37] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive eff... on Pith
- [38] Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models capabilities back to pretraining data. In The Thirteenth International Conference on Learning Representa... on Pith
- [39] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. OctoThinker : Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025 d on PithPith's reviewarXiv
- [40] Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs . In The Fourteenth International Conference on Le... on Pith
- [41] Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why RLVR may or may not escape its origin. arXiv preprint arXiv:2507.14843, 2025 on PithPith's reviewarXiv
- [42] Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang. Reasoning or memorization? Unreliable results of reinforcement learning due to data contamination. In Proce... on Pithdoi:10.1609/aaai.v40i40.40687
- [43] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS : Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML), 2024. arXiv:2402.04333 on PithPith's reviewarXiv
- [44] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2302.03169 on PithPith's reviewarXiv
- [45] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 on PithPith's reviewarXiv
- [46] Zichun Yu, Spandan Das, and Chenyan Xiong. MATES : Model-aware data selection for efficient pretraining with data influence models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.06046 on PithPith's reviewarXiv
- [47] Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, and Bo An. Understanding diversity collapse in RLVR via the lens of overtraining. arXiv preprint arXiv:2606.15455, 2026 on PitharXiv
- [48] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLM s beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.... on Pith
- [49] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and RL on reasoning language models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=TBaUfO9znF on Pith
- [50] Rosie Zhao, Alexandru Meterez, Sham M. Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=dp4KWuSDzj on Pith
- [51] Todd Zhou. When RLVR shrinks the reasoning boundary: Diagnosing pass@ k inversion. arXiv preprint arXiv:2607.20543, 2026 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment