REVIEW 6 cited by
Towards the Law of Capacity Gap in Distilling Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Language model (LM) distillation aims at distilling the knowledge in a large teacher LM to a small student one. As a critical issue facing LM distillation, a superior student often arises from a teacher of a relatively small scale instead of a larger one, especially in the presence of substantial capacity gap between the teacher and student. This issue, often referred to as the \textit{curse of capacity gap}, suggests that there is likely an optimal teacher yielding the best-performing student along the scaling course of the teacher. Consequently, distillation trials on teachers of a wide range of scales are called for to determine the optimal teacher, which becomes computationally intensive in the context of large LMs (LLMs). This paper addresses this critical bottleneck by providing the \textit{law of capacity gap} inducted from a preliminary study on distilling a broad range of small-scale (<3B) LMs, where the optimal teacher consistently scales linearly with the student scale across different model and data scales. By extending the law to LLM distillation on a larger scale (7B), we succeed in obtaining versatile LLMs that outperform a wide array of competitors.
Forward citations
Cited by 6 Pith papers
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
-
ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models
ReaLM trains small language models to learn from both right and wrong reasoning chains, then fades the chains out so the model reasons independently, improving benchmark accuracy.
-
MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
Training small language models on intermediate-length reasoning chains from a merged mid-sized teacher assistant improves their math reasoning scores over direct distillation from a large teacher.
-
LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
LaCache keeps a layer-dependent diagonal slice of the KV cache and iteratively compacts old entries, improving long-context perplexity and retrieval accuracy versus StreamingLLM at fixed cache sizes.
-
ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
A layer-uncertainty-based budget allocator for KV cache compression slightly outperforms uniform-budget and pyramid-budget baselines on LongBench and Needle-in-a-Haystack.
-
FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigation
Fine-tuned small language models, coached by a GPT-4 teacher through iterative prompting, can perform multi-point robot navigation on edge devices with success rates approaching larger models.
Discussion (0). Continue with ORCID to comment.