REVIEW 5 cited by
DsDm: Model-Aware Dataset Selection with Datamodels
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in practice the opposite can often happen: we find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. To develop better methods for selecting data, we start by framing dataset selection as an optimization problem that we can directly solve for: given target tasks, a learning algorithm, and candidate data, select the subset that maximizes model performance. This framework thus avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks. Our resulting method greatly improves language model (LM) performance on both pre-specified tasks and previously unseen tasks. Specifically, choosing target tasks representative of standard LM problems and evaluating on diverse held-out benchmarks, our selected datasets provide a 2x compute multiplier over baseline methods.
Forward citations
Cited by 5 Pith papers
-
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.
-
GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.
-
BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.
-
Approximating Language Model Training Data from Weights
A gradient-based greedy selection method (SELECT) recovers effective substitute fine-tuning data from two language model checkpoints, approaching the original model's performance on classification and SFT tasks.
-
LLM Data Selection and Utilization via Dynamic Bi-level Optimization
A bi-level-learned per-batch reweighting model improves average downstream accuracy of LLMs pre-trained on selected data by about one to two points, but without error bars or code the effect size is uncertain.
Discussion (0). Sign in to comment.