REVIEW 51 cited by
A Survey on Data Selection for Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.
Forward citations
Cited by 51 Pith papers
-
CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems
A generative model of student code responses improves early-stage adaptive-test question selection, yielding up to 4.32% higher AUC than a 1PL baseline.
-
ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models
ChemPile is an open 75-billion-token, multimodal chemical dataset spanning education, papers, property tables, code, images, and reasoning traces, released for training chemical foundation models.
-
The Leaderboard Illusion
Chatbot Arena's rankings are systematically distorted by undisclosed private testing, selective score reporting, and data access asymmetries that favor large proprietary providers.
-
SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement
A cost-aware gate that routes simple SFT procurement cases to cheap statistics and hard cases to an adjudicated LLM debate reports 0.90 accuracy on a synthetic closed-loop benchmark.
-
Online Data Selection Is Implicit Alignment
Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.
-
BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.
-
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Classifier-based quality filtering for LLM pretraining improves downstream tasks by implicitly filtering the reference high-quality set rather than by mimicking it, and its quality scores fail a data-conditioning test.
-
AdaThink-Med: Optimizing Inference-Time Compute for Medical Reasoning via Uncertainty Quantification
An entropy-guided reward that shortens answers to easy, confident questions and extends reasoning on hard ones reduces medical QA tokens by 4.7x to 6.4x with minimal accuracy loss.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning
SCIZOR filters suboptimal and redundant state-action pairs from robot demonstrations without human labels, improving imitation-learning policy success rates by about 15% on average.
-
Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?
Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.
-
Merge to Mix: Mixing Datasets via Model Merging
Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.
-
Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization
An evaluation-set selection method that adds real-time model feedback to semantic sampling improves the accuracy and stability of three prompt optimization methods on two datasets.
-
RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection
RICo scores instruction examples by their in-context perplexity effect on an assessment set, then trains a lightweight selector to pick top-scoring data, achieving better benchmark results from 5% to 15% of the original data.
-
When Incentives Backfire, Data Stops Being Human
Incentive-driven crowdwork erodes intrinsic motivation and data quality, so data collection should be redesigned around intrinsic motivation, with games as a promising template.
-
Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights
A gradient alignment score against a reference model's weights selects and reweights training samples, improving data efficiency in image classification and CLIP pretraining.
-
How to Synthesize Text Data without Model Collapse?
Token-level editing of human text with a high-confidence threshold yields modest performance gains over the original data and avoids the sharp degradation seen with purely synthetic data.
-
Bridging the Data Provenance Gap Across Text, Speech and Video
A manual audit of nearly 4,000 text, speech, and video datasets finds AI training data increasingly comes from web and social media sources, carries hidden non-commercial restrictions, and remains Western-centric with...
-
LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning
An LLM-scored easy-to-hard curriculum for training grammatical error correction models yields small but consistent F0.5 gains over one-shot and length-based training.
-
TAROT: Targeted Data Selection via Optimal Transport
Targeted data selection via whitened feature distance and optimal transport minimization consistently outperforms influence-based greedy heuristics across vision and language tasks.
-
AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
AttriBoT combines caching, hierarchical pruning, and smaller proxy models to approximate leave-one-out context attribution with a >300x speedup and little loss in faithfulness.
-
Quantifying Political Partisanship for Cross-Platform Analyses
Partisanship of individual posts can be scored on a common embedding axis anchored by AllSides news-bias labels, yielding cross-platform scores that transfer from Bluesky/Truth Social to X.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
-
Improving Translation Quality by Selecting Better Data for LLM Fine-Tuning: A Comparative Analysis
Fine-tuning on data selected by COMET-Kiwi sometimes improves Japanese-English translation, but the paper's claim that semantic selectors consistently win is contradicted by its own results.
-
ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization
Using Bayesian optimization over Gaussian-process surrogates, data mixtures for LLM training can be found much faster than with linear or exponential regression baselines, including across model sizes.
-
From Black Box to Transparency: Enhancing Automated Interpreting Assessment with Explainable AI in College Classrooms
An explainable model using BLEURT, CometKiwi, pause features, and Chinese phraseological diversity predicts human-rated quality dimensions in English-Chinese consecutive interpreting, with SHAP identifying the stronge...
-
Assessing the Role of Data Quality in Training Bilingual Language Models
A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.
-
Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
-
A Survey of LLM $\times$ DATA
A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.
-
IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment
IDEAL tunes SFT data mixture proportions per domain with influence-function gradients, claiming about 7% average benchmark improvement over uniform mixing.
-
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.
-
Less is More: Adaptive Coverage for Synthetic Training Data
A max-coverage graph algorithm with an adaptive similarity threshold selects 10-30% of synthetic data that trains classifiers as well as or better than the full dataset on sentiment, relation extraction, and NER tasks.
-
Meta-Prompt Optimization for LLM-Based Sequential Decision Making
EXPO uses adversarial bandit weighting over LLM-generated prompt variations to optimize the meta-prompt of LLM-based sequential decision-making agents, improving performance on optimization and bandit tasks.
-
Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
Simple size-diversity heuristics outperform learned data mixing, while an LLM-based utility estimator (MEDU) matches expensive ablations at roughly 200x lower compute.
-
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
EDiT trains LLMs with local SGD plus model sharding, adding a pseudo-gradient penalty to stabilize loss; it reports modest quality gains and higher throughput, though a proof of convergence contains an error.
-
VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs
A data composition method that aligns SFT data proportions with a model's detected domain knowledge distribution and dynamically reweights domains by learnable potential improves multi-domain performance versus unifor...
-
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment
BPO balances knowledge breadth and depth in preference data by compressing prompts and dynamically augmenting the number of response pairs per prompt using gradient-based clustering, achieving stronger alignment with ...
-
RAG System for Supporting Japanese Litigation Procedures: Faithful Response Generation Complying with Legal Norms
A norm-compliant RAG for Japanese medical litigation must restrict knowledge sources, stay faithful to retrieved context, and reference sources valid at the time of the disputed issue.
-
A Survey on Open Dataset Search in the LLM Era: Retrospectives and Perspectives
A structured review of open dataset search across tabular, spatial, JSON, graph, and vector data, plus the two-way relationship with LLMs.
-
Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation
A hierarchical direct preference optimization with four alignment levels plus automated data selection improves physical plausibility of text-to-video models.
-
LLM Data Selection and Utilization via Dynamic Bi-level Optimization
A bi-level-learned per-batch reweighting model improves average downstream accuracy of LLMs pre-trained on selected data by about one to two points, but without error bars or code the effect size is uncertain.
-
Data Efficacy for Language Model Training
Ordering training data by a gradient-based score, using a folding scheme that interleaves multiple curriculum passes, improves small-scale LM accuracy by roughly 1.5 to 2 points on average benchmarks.
-
Text2Cypher: Data Pruning using Hard Example Selection
Hard-example selection can halve Text2Cypher fine-tuning cost with modest accuracy loss, beating random sampling on the same budget.
-
Advancing AI-assisted Hardware Design with Hierarchical Decentralized Training and Personalized Inference-Time Optimization
A two-stage framework of hierarchical decentralized training plus personalized test-time optimization raises LLM hardware generation accuracy and speed in HLS and Qiskit benchmarks.
-
Investigating the Impact of Data Selection Strategies on Language Model Performance
Neural-embedding-based data selection underperforms DSIR on most GLUE tasks, and the paper's hybrid method is untested because only α=0 was run.
-
A Survey on Large Language Model Acceleration based on KV Cache Management
A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
Generative KI f\"ur TA
Generative AI should be used only as a checked support in technology assessment because persistent structural deficiencies make its outputs unreliable.
-
The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)
The paper catalogs the lifecycle stages and production-readiness challenges of software built around foundation models (FMware) and proposes an action plan of engineering practices and research directions.
-
The Rise of Small Language Models in Healthcare: A Comprehensive Survey
A comprehensive survey of small language models in healthcare, with a taxonomy of building, adapting, and compressing them for clinical NLP tasks.
Discussion (0). Continue with ORCID to comment.