REVIEW 7 cited by
Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The surge in Large Language Models (LLMs) has revolutionized natural language processing, but fine-tuning them for specific tasks often encounters challenges in balancing performance and preserving general instruction-following abilities. In this paper, we posit that the distribution gap between task datasets and the LLMs serves as the primary underlying cause. To address the problem, we introduce Self-Distillation Fine-Tuning (SDFT), a novel approach that bridges the distribution gap by guiding fine-tuning with a distilled dataset generated by the model itself to match its original distribution. Experimental results on the Llama-2-chat model across various benchmarks demonstrate that SDFT effectively mitigates catastrophic forgetting while achieving comparable or superior performance on downstream tasks compared to the vanilla fine-tuning. Moreover, SDFT demonstrates the potential to maintain the helpfulness and safety alignment of LLMs. Our code is available at https://github.com/sail-sg/sdft.
Forward citations
Cited by 7 Pith papers
-
MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.
-
Learning to Prepare Molecular Ground States with Transformer Models
Transformers trained on ADAPT-VQE data generate imipramine ground-state circuits in seconds at roughly reference accuracy — and beat the training data after reinforcement learning — though real-hardware energies still...
-
Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Models
Table-r1 combines a layout-transformation self-supervised task and a mix-paradigm GRPO stage so 7B/8B models outperform other small-model table reasoners and approach GPT-4o-level accuracy.
-
Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation
MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.
-
Continual Learning in Transition
A tri-axial framework of When, Where, and How organizes the ongoing transition of continual learning from parameter-centric updates to system-level capability evolution.
-
SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment
Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.
-
Reliable Reasoning Path: Distilling Effective Guidance for LLM Reasoning with Knowledge Graphs
RRP generates semantic and structural reasoning paths, reranks them with a rethinking module, and reports SOTA Hits@1 of 90.0 on WebQSP and 64.5 on CWQ.
Discussion (0). Sign in to comment.