REVIEW 7 cited by
Learning Dynamics of LLM Finetuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.
Forward citations
Cited by 7 Pith papers
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
-
Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion
In a linear model of LLM personalization with shared compute, SFT beats ICL above a coverage-dependent signal-to-noise threshold, congestion can reverse that ranking, and adding SFT never reduces platform profit.
-
An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
Emergent misalignment and realignment are brittle surface effects driven by dataset artifacts like response length rather than stable representational changes.
-
Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation
A text-to-3D pipeline that predicts a global semantic-geometric layout, conditions panoramic and video diffusion on it, and fuses views with 3D Gaussian Splatting into an absolute-scale navigable indoor scene.
-
Decoupling Task-Solving and Output Formatting in LLM Generation
A decoding-time method that keeps the format in a separate module improves LLM accuracy by 1–6% with guaranteed format compliance on math, judging, and extraction.
-
RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
Reinforcement fine-tuning largely recovers out-of-distribution performance lost during supervised fine-tuning by counteracting rotations of singular vector directions, with low-rank or shallow-layer restoration recove...
-
Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
SFT is cast as a special case of implicit reward learning, and a slower SFT learning rate improves downstream DPO/SimPO results by up to 25% relative win rate.
Discussion (0). Sign in to comment.