Pith. sign in

REVIEW 11 cited by

S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.12853 v1 pith:7KMPHQDO submitted 2025-02-18 cs.CL cs.LG

Ruotian Ma , Peisong Wang , Cheng Liu , Xingyan Liu , Jiaqi Chen , Bang Zhang , Xin Zhou , Nan Du
show 1 more author
Jia Li
This is my paper
classification cs.CLcs.LG
keywords datamodelsllmsabilitiesbaseduringeffectivenessinference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Recent studies have demonstrated the effectiveness of LLM test-time scaling. However, existing approaches to incentivize LLMs' deep thinking abilities generally require large-scale data or significant training efforts. Meanwhile, it remains unclear how to improve the thinking abilities of less powerful base models. In this work, we introduce S$^2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference. Specifically, we first initialize LLMs with iterative self-verification and self-correction behaviors through supervised fine-tuning on carefully curated data. The self-verification and self-correction skills are then further strengthened by both outcome-level and process-level reinforcement learning, with minimized resource requirements, enabling the model to adaptively refine its reasoning process during inference. Our results demonstrate that, with only 3.1k self-verifying and self-correcting behavior initialization samples, Qwen2.5-math-7B achieves an accuracy improvement from 51.0\% to 81.6\%, outperforming models trained on an equivalent amount of long-CoT distilled data. Extensive experiments and analysis based on three base models across both in-domain and out-of-domain benchmarks validate the effectiveness of S$^2$R. Our code and data are available at https://github.com/NineAbyss/S2R.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Hidden Signal of Verifier Strictness: Controlling and Improving Step-Wise Verification via Selective Latent Steering

    cs.LG 2026-05 conditional novelty 7.0 of 10

    VerifySteer selectively steers hidden states at paragraph boundaries using latent correctness signals to control verifier strictness and outperform baselines on ProcessBench and Hard2Verify with lower compute.

  2. Multi-modal Reasoning with LLMs for Visual Semantic Arithmetic

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    SAri-RFT applies GRPO-based reinforcement fine-tuning to LVLMs on novel two-term and three-term visual semantic arithmetic tasks, reaching SOTA on the new IRPD dataset and Visual7W-Telling.

  3. Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    A new RL paradigm for reasoning where models generate their own internal process supervision from outcome feedback by recycling failed trajectories.

  4. CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A role-playing LLM that reasons about the scene and its own state before responding, trained with two semantic rewards, beats stronger baselines on role-play benchmarks.

  5. Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Self-correction blind spots in residual-stream autoregressive models arise iff the product of attention Jacobians has spectral radius ≥1, with a sharp marker threshold and RL coupling condition derived from that radius.

  6. Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    A method internalizes outcome supervision into process supervision by extracting step-level learning signals from failed reasoning trajectories during reinforcement learning.

  7. SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning

    cs.AI 2026-04 conditional novelty 5.0 of 10

    A cold-start SFT plus semi-online DPO pipeline that embeds self-verification and self-rectification into VLMs, raising multimodal reasoning accuracy and implicit performance from only ~5k–20k preference samples.

  8. SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    SVSR trains multimodal models to verify and correct their own reasoning using a preference dataset, supervised fine-tuning, and semi-online DPO with a teacher model.

  9. Towards Sparse Video Understanding and Reasoning

    cs.CV 2026-02 conditional novelty 5.0 of 10

    A video-QA agent that carries only a structured text summary between rounds beats dense-frame baselines on accuracy while using a handful of frames per video.

  10. Self-Reflective Generation at Test Time

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.

  11. From System 1 to System 2: A Survey of Reasoning Large Language Models

    cs.AI 2025-02 accept novelty 3.0 of 10

    The survey organizes the shift of LLMs toward deliberate System 2 reasoning, covering model construction techniques, performance on math and coding benchmarks, and future research directions.

Pith tools