REVIEW 8 cited by
How to Train Data-Efficient LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.
Forward citations
Cited by 8 Pith papers
-
LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
Disentangling the Roles of Representation and Selection in Data Pruning
In NLP data pruning, the representation used to score examples (especially gradients) influences selected data and downstream performance more than the selection algorithm, and difficulty-oriented algorithms often do ...
-
Ming-Omni: A Unified Multimodal Model for Perception and Generation
A single model with modality-specific routing processes image, text, audio, and video inputs and generates text, speech, and images, with public benchmarks reported across all of these abilities.
-
Assessing the Role of Data Quality in Training Bilingual Language Models
A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.
-
Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
High data redundancy and over-training decelerate LLM performance gains, and the authors fit a sub-optimal scaling law with logistic correction terms to predict the slowdown.
-
Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Need
GSTDS selects a scheduled fraction of each training batch via Fiedler-vector ranking and reference-model losses, reporting comparable or better accuracy on three small image benchmarks with up to 4x fewer training FLO...
-
Beyond Traditional Algorithms: Leveraging LLMs for Accurate Cross-Border Entity Identification
A 65-case comparison claims commercial chatbot LLMs are the most accurate for Portuguese entity matching, but the reported false-positive rates contradict the claim.
Discussion (0). Sign in to comment.