REVIEW 6 cited by
Irreducible Curriculum for Language Model Pretraining
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Automatic data selection and curriculum design for training large language models is challenging, with only a few existing methods showing improvements over standard training. Furthermore, current schemes focus on domain-level selection, overlooking the more fine-grained contributions of each individual training point. It is difficult to apply traditional datapoint selection methods on large language models: most online batch selection methods perform two-times forward or backward passes, which introduces considerable extra costs with large-scale models. To mitigate these obstacles, we propose irreducible curriculum as a curriculum learning algorithm for language model pretraining, which prioritizes samples with higher learnability. Specifically, to avoid prohibitive extra computation overhead, we simulate the sample loss along the main model's training trajectory using a small-scale proxy model. Our experiments on the RedPajama-1B dataset demonstrate a consistent improvement on validation perplexity across all 7 domains compared to random uniform baseline and the anti-curriculum strategy. Our method also reduces the sharpness of the network and illustrates a better 5-shot accuracy on MMLU benchmarks.
Forward citations
Cited by 6 Pith papers
-
Active Data Curation Effectively Distills Large-Scale Multimodal Models
Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.
-
Investigating the Zone of Proximal Development of Language Models for In-Context Learning
A framework that predicts, per query, whether an LLM can solve it directly, only with demonstrations, or not at all, and uses those predictions for selective in-context learning and curriculum fine-tuning.
-
Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
A fully online, loss-based reweighting scheme that down-weights low-loss samples during LLM pretraining yields small average benchmark gains at 1.4B and 7B scale, together with a convergence bound under convexity and ...
-
Dynamic Skill Adaptation for Large Language Models
A training pipeline that orders generated textbook and exercise data by a skill dependency graph and dynamically updates the data during fine-tuning improves LLM performance on calculus and social studies evaluations.
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
Discussion (0). Continue with ORCID to comment.