Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.