Pith. sign in

REVIEW 5 cited by

Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.06533 v1 pith:W7J5BAKA submitted 2025-02-10 cs.CL cs.LG

Jean Vassoyan , Nathanaël Beau , Roman Plaud This is my paper · ORCID
classification cs.CLcs.LG
keywords explorationcriticalllmspenaltytokensfine-tuninglanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The ability to achieve long-term goals is a key challenge in the current development of large language models (LLMs). To address this, pre-trained LLMs can be fine-tuned with reinforcement learning (RL) to explore solutions that optimize a given goal. However, exploration with LLMs is difficult, as a balance has to be struck between discovering new solutions and staying close enough to the pre-trained model, so as not to degrade basic capabilities. This is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we investigate the exploration dynamics of a small language model on a simple arithmetic task. We show how varying degrees of pre-training influence exploration and demonstrate the importance of "critical tokens" which have a dramatic impact on the final outcome. Consequently, we introduce a simple modification to the KL penalty that favors exploration on critical tokens, increasing the efficiency of the RL fine-tuning stage.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Embarrassingly Simple Self-Distillation Improves Code Generation

    cs.CL 2026-04 conditional novelty 7.0 of 10

    Simple self-distillation—fine-tuning a code model on its own temperature-sampled, truncated outputs—raises LiveCodeBench pass@1 substantially without verifiers, teachers, or RL.

  2. Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

    cs.CL 2025-10 conditional novelty 6.0 of 10

    LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.

  3. Self-Reflective Generation at Test Time

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SRGen improves LLM math reasoning by detecting high-entropy tokens and injecting a small corrected vector into the hidden state at those points during decoding, without training.

  4. Discovering Algorithms with Computational Language Processing

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A machine learning framework called CLP discovers, improves, and tailors algorithms by chaining computational tokens with MCTS and RL, with strong results on the Quadratic Assignment Problem and quantum search.

  5. CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generation

    cs.AI 2025-07 reject novelty 3.0 of 10

    A 7B model trained with GRPO and a sparse execution-correctness reward reaches 59.97% execution accuracy on BIRD dev, though the evaluation protocol and baseline numbers contain inconsistencies.

Pith tools