A fully online, loss-based reweighting scheme that down-weights low-loss samples during LLM pretraining yields small average benchmark gains at 1.4B and 7B scale, together with a convergence bound under convexity and interpolation assumptions.
Skill-it! a data-driven skills framework for understanding and training language models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
A fully online, loss-based reweighting scheme that down-weights low-loss samples during LLM pretraining yields small average benchmark gains at 1.4B and 7B scale, together with a convergence bound under convexity and interpolation assumptions.