The reviewed record of science sign in
Pith

arxiv: 2504.04022 · v1 · pith:XN7DDASU · submitted 2025-04-05 · cs.CL · cs.AI

Rethinking Reflection in Pre-Training

Essential AI: Darsh J Shah , Peter Rushton , Somanshu Singla , Mohit Parmar , Kurt Smith , Yash Vanjani , Ashish Vaswani , Adarsh Chaluvaraju
show 19 more authors
Andrew Hojel Andrew Ma Anil Thomas Anthony Polloreno Ashish Tanwer Burhan Drak Sibai Divya S Mansingka Divya Shivaprasad Ishaan Shah Karl Stratos Khoi Nguyen Michael Callahan Michael Pust Mrinal Iyer Philip Monk Platon Mazarakis Ritvik Kapila Saurabh Srivastava Tim Romanski
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:XN7DDASUrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords modelabilitypre-trainingduringacrossactuallyadvantageanswer
0
0 comments X
read the original abstract

A language model's ability to reflect on its own reasoning provides a key advantage for solving complex problems. While most recent research has focused on how this ability develops during reinforcement learning, we show that it actually begins to emerge much earlier - during the model's pre-training. To study this, we introduce deliberate errors into chains-of-thought and test whether the model can still arrive at the correct answer by recognizing and correcting these mistakes. By tracking performance across different stages of pre-training, we observe that this self-correcting ability appears early and improves steadily over time. For instance, an OLMo2-7B model pre-trained on 4 trillion tokens displays self-correction on our six self-reflection tasks.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation

    cs.LG 2026-06 unverdicted novelty 7.0

    OrderGrad supplies unbiased likelihood-ratio and reparameterization gradient estimators for finite-sample L-statistics by applying a rank-based reward transformation usable in standard policy-gradient updates.

  2. Reinforcement Learning for Reasoning in Large Language Models with One Training Example

    cs.LG 2025-04 accept novelty 7.0

    One training example via RLVR boosts LLM math reasoning from 17.6% to 35.7% average across six benchmarks.

  3. On Advantage Estimates for Max@K Policy Gradients

    cs.LG 2026-06 unverdicted novelty 6.0

    Proposes MaxPO using a Leave-Two-Out baseline for centered unbiased advantages in max@K policy gradients, with a unified derivation of finite-batch estimators.

  4. How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoning

    cs.AI 2026-04 conditional novelty 6.0

    Across four frontier reasoning models, 61–93% of correct chain-of-thought steps are redundant, and this over-thinking is provably optimal under any length-agnostic outcome reward.

  5. Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards

    cs.LG 2025-09 conditional novelty 5.0

    The paper identifies confounds in RLVR evaluations that inflate apparent gains and proposes a minimum standard for budget-matched, contamination-aware assessment with calibration tracking.

  6. Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

    cs.LG 2025-08 unverdicted novelty 5.0

    DARS adaptively increases rollouts on hard problems in RLVR to improve Pass@K, and when paired with batch scaling for breadth, achieves gains in both Pass@K and Pass@1 by treating depth and breadth as complementary ex...