REVIEW 14 cited by
How to Train Data-Efficient LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.
Forward citations
Cited by 14 Pith papers
-
LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection
LAMDAS selects domain-relevant training data via an LLM likelihood ratio with a learned domain prefix, beating full-data training and nine baselines on code and math.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
Disentangling the Roles of Representation and Selection in Data Pruning
In NLP data pruning, the representation used to score examples (especially gradients) influences selected data and downstream performance more than the selection algorithm, and difficulty-oriented algorithms often do ...
-
Ming-Omni: A Unified Multimodal Model for Perception and Generation
A single model with modality-specific routing processes image, text, audio, and video inputs and generates text, speech, and images, with public benchmarks reported across all of these abilities.
-
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
JQL trains small multilingual quality scorers from LLM judgments and human annotations, and filtering pretraining data with them improves downstream multilingual model performance over heuristic baselines.
-
Enhancing LLMs via High-Knowledge Data Selection
A knowledge-element density and coverage scorer selects pre-training data that improves LLM performance on knowledge-intensive and general understanding benchmarks by 2 to 3 points.
-
Assessing the Role of Data Quality in Training Bilingual Language Models
A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.
-
Truly Self-Improving Agents Require Intrinsic Metacognitive Learning
The paper proposes that self-improving agents must learn to manage their own learning processes, framing this as intrinsic metacognitive learning, and argues it is necessary for sustained and generalized improvement.
-
Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Markets
AI agents in future labor markets will need metacognitive and strategic reasoning because incomplete information creates adverse selection, moral hazard, and reputation effects.
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
-
FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
A greedy token-level Fisher information data selection method that reports improved sample efficiency for GPT-2 supervised fine-tuning on Shakespeare text relative to uniform, density, and AskLLM baselines.
-
Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
High data redundancy and over-training decelerate LLM performance gains, and the authors fit a sub-optimal scaling law with logistic correction terms to predict the slowdown.
-
Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Need
GSTDS selects a scheduled fraction of each training batch via Fiedler-vector ranking and reference-model losses, reporting comparable or better accuracy on three small image benchmarks with up to 4x fewer training FLO...
-
Beyond Traditional Algorithms: Leveraging LLMs for Accurate Cross-Border Entity Identification
A 65-case comparison claims commercial chatbot LLMs are the most accurate for Portuguese entity matching, but the reported false-positive rates contradict the claim.
Discussion (0). Sign in to comment.