REVIEW 13 cited by
When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large volumes of text data have contributed significantly to the development of large language models (LLMs) in recent years. This data is typically acquired by scraping the internet, leading to pretraining datasets comprised of noisy web text. To date, efforts to prune these datasets down to a higher quality subset have relied on hand-crafted heuristics encoded as rule-based filters. In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data. We perform a rigorous comparison at scale of the simple data quality estimator of perplexity, as well as more sophisticated and computationally intensive estimates of the Error L2-Norm and memorization. These metrics are used to rank and prune pretraining corpora, and we subsequently compare LLMs trained on these pruned datasets. Surprisingly, we find that the simple technique of perplexity outperforms our more computationally expensive scoring methods. We improve over our no-pruning baseline while training on as little as 30% of the original training dataset. Our work sets the foundation for unexplored strategies in automatically curating high quality corpora and suggests the majority of pretraining data can be removed while retaining performance.
Forward citations
Cited by 13 Pith papers
-
OPERA: Online Data Pruning for Efficient Retrieval Model Adaptation
Dynamic hierarchical data pruning improves NDCG@10 and Recall@20 for dense retrievers while reaching full performance in half the iterations.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
Disentangling the Roles of Representation and Selection in Data Pruning
In NLP data pruning, the representation used to score examples (especially gradients) influences selected data and downstream performance more than the selection algorithm, and difficulty-oriented algorithms often do ...
-
Efficient dataset generation for machine learning perovskite alloys
A clustering and active learning workflow reduces the density functional theory data needed to train machine-learning models for perovskite alloys by up to 20% to 50% in validation tests.
-
Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.
-
GORACS: Group-level Optimal Transport-guided Coreset Selection for LLM-based Recommender Systems
GORACS selects small groups of fine-tuning examples via an optimal-transport and gradient-norm proxy objective, outperforming prior coreset methods for LLM-based recommendation.
-
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
Using Bayesian optimization over Gaussian-process surrogates, data mixtures for LLM training can be found much faster than with linear or exponential regression baselines, including across model sizes.
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
-
Efficient Data Selection at Scale via Influence Distillation
Influence Distillation selects LLM fine-tuning data by approximating each sample's gradient influence on a target task via landmarks and JVP embeddings, matching or beating RDS+ accuracy at roughly one third the selec...
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
-
Improving Chemical Understanding of LLMs via SMILES Parsing
Pretraining LLMs on deterministic SMILES parsing tasks improves molecular structural understanding and downstream chemistry performance.
-
Small-to-Large Generalization: Data Influences Models Consistently Across Scale
Training data changes influence small and large language models' losses similarly, so small proxy models can substitute for large models in data attribution and selection.
-
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.
Discussion (0). Sign in to comment.