Privileged-context on-policy self-distillation degrades thinking models' long-budget accuracy by suppressing forking and self-correction behaviors, while helping instruction-tuned models.
Forking paths in neural text generation
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
Estimating uncertainty in Large Language Models (LLMs) is important for properly evaluating LLMs, and ensuring safety for users. However, prior approaches to uncertainty estimation focus on the final answer in generated text, ignoring intermediate steps that might dramatically impact the outcome. We hypothesize that there exist key forking tokens, such that re-sampling the system at those specific tokens, but not others, leads to very different outcomes. To test this empirically, we develop a novel approach to representing uncertainty dynamics across individual tokens of text generation, and applying statistical models to test our hypothesis. Our approach is highly flexible: it can be applied to any dataset and any LLM, without fine tuning or accessing model weights. We use our method to analyze LLM responses on 7 different tasks across 4 domains, spanning a wide range of typical use cases. We find many examples of forking tokens, including surprising ones such as punctuation marks, suggesting that LLMs are often just a single token away from saying something very different.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Resampling subsequent text in chain-of-thought traces enables causal analysis of partial reasoning steps in LLMs, showing limited causal impact from self-preservation statements and advantages of on-policy interventions over off-policy edits.
Policy entropy remains constant in flow-matching models during RLHF due to fixed noise schedules while perceptual diversity collapses from mode-seeking policy gradients, so perceptual entropy constraints are introduced to preserve diversity and improve quality.
citing papers explorer
-
Rethinking On-Policy Self-Distillation for Thinking Models
Privileged-context on-policy self-distillation degrades thinking models' long-budget accuracy by suppressing forking and self-correction behaviors, while helping instruction-tuned models.
-
Thought Branches: Interpreting LLM Reasoning Requires Resampling
Resampling subsequent text in chain-of-thought traces enables causal analysis of partial reasoning steps in LLMs, showing limited causal impact from self-preservation statements and advantages of on-policy interventions over off-policy edits.
-
When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
Policy entropy remains constant in flow-matching models during RLHF due to fixed noise schedules while perceptual diversity collapses from mode-seeking policy gradients, so perceptual entropy constraints are introduced to preserve diversity and improve quality.