Pith. sign in

REVIEW 29 cited by

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.16952 v2 pith:SOCYVDPU submitted 2024-03-25 cs.CL cs.AIcs.LG

Jiasheng Ye , Peiju Liu , Tianxiang Sun , Jun Zhan , Yunhua Zhou , Xipeng Qiu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords datamixtureperformancelawsmixingmixturesmodeltraining
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Using a synthetic alphabet video testbed, balanced data mixing and caption precision are shown to dominate T2V model quality, while CFG and fine-tuning only partially compensate for corrupted captions.

  2. Domain-Aware Scaling Laws Uncover Data Synergy

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.

  3. Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A forward-only controller sets multi-domain LoRA participation from label-free competence and cross-domain affinity, improving average accuracy while using half the data.

  4. ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    ChemPile is an open 75-billion-token, multimodal chemical dataset spanning education, papers, property tables, code, images, and reasoning traces, released for training chemical foundation models.

  5. The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Capability formation in small transformers is claimed to obey a driven-nucleation rate law J = Nνσ(c)e^{−βK} − D, read forward as emergence, backward as plasticity loss, and completed as circuit control.

  6. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning

    cs.LG 2026-02 reject novelty 6.0 of 10

    GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.

  7. Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Domain2Vec finds better LLM pretraining data mixtures by aligning, in a training-free way, the meta-domain distribution of the training set with the validation set's distribution.

  8. Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Chameleon uses kernel ridge leverage scores on domain embeddings to set LLM training-mixture weights, matching DoGE-level pretraining quality at roughly one fifth the compute and improving finetuning perplexity.

  9. Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.

  10. Merge to Mix: Mixing Datasets via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.

  11. Learning Dynamics in Continual Pre-Training for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A continual pre-training scaling law decomposes the loss curve into a pre-training baseline plus a distribution-shift term, enabling per-step and cross-schedule loss prediction.

  12. AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    AttentionInfluence uses retrieval-head masking loss differences in a 1.3B model to select 73B tokens that improve a 7B model's reasoning benchmarks by 1.4 to 3.5pp.

  13. QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining

    cs.CL 2025-04 conditional novelty 6.0 of 10

    QuaDMix jointly optimizes data quality and diversity for LLM pretraining, using proxy models and LightGBM to search the parameters of a per-domain sampling function.

  14. Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction

    cs.RO 2025-04 conditional novelty 6.0 of 10

    A motion-based self-reflection framework that lets multimodal language models correct fine-grained robot actions via coarse motion instructions.

  15. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Organizing web pretraining text into topic and format domains and reweighting those domains improves 1B-scale language model benchmarks, and combining the reweighting with quality filters yields further gains.

  16. PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts

    cs.LG 2025-02 conditional novelty 6.0 of 10

    PiKE adaptively re-weights pretraining data sources by gradient magnitude and variance, exploiting low gradient conflicts to speed up convergence and improve downstream accuracy in LLM pretraining.

  17. Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A fine-tuned 14B LLM judge, trained with scenario-based prompts and controlled instruction generation, approaches GPT-4's human-agreement performance, and the paper documents why scaling distillation data can fail.

  18. BTS: Harmonizing Specialized Experts into a Generalist LLM

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Branch-Train-Stitch merges frozen domain-specialist LLMs into a generalist by training only lightweight stitch layers, and reports the best average benchmark performance among merging baselines.

  19. Loss-to-Loss Prediction: Scaling Laws for All Datasets

    cs.LG 2024-11 conditional novelty 6.0 of 10

    Losses of models trained on different datasets are related by shifted power laws, enabling translation of scaling laws and prediction of downstream performance from a few runs.

  20. Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

    cs.CL 2026-08 conditional novelty 5.0 of 10

    A one-parameter coupling of model size and data in the Chinchilla loss form reduces boundary prediction error and enables cheaper L-shaped profiling grids.

  21. Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    The authors introduce and test an optimization method for fine-tuning data mixtures that matches costly grid-search results within 0.66% validation loss.

  22. Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Running multiple short annealing runs at different token scales can reveal per-source utility scaling curves that change data-source rankings compared with single point estimates.

  23. MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Multi-domain RLVR data mixing, guided by a quadratic surrogate fitted to 11 pilot runs, improves a Qwen2-VL-2B model's out-of-distribution accuracy by about 5 points over uniform mixing.

  24. Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

    cs.CL 2025-05 accept novelty 5.0 of 10

    A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.

  25. IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment

    cs.AI 2025-05 reject novelty 5.0 of 10

    IDEAL tunes SFT data mixture proportions per domain with influence-function gradients, claiming about 7% average benchmark improvement over uniform mixing.

  26. R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training

    cs.LG 2025-05 conditional novelty 5.0 of 10

    R&B repartitions training data into embedding-based semantic clusters and reweights them online via a gradient Gram matrix, matching or exceeding prior data-mixing methods at under 0.1% extra compute.

  27. Xmodel-2 Technical Report

    cs.AI 2024-12 reject novelty 5.0 of 10

    A 1.2B language model trained with tensor-program hyperparameter transfer and WSD decay with SFT data mixing; the headline SOTA claim is not supported by the paper's own benchmark tables.

  28. Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

    cs.LG 2024-12 conditional novelty 4.0 of 10

    This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...

  29. Foundations of GenIR

    cs.IR 2025-01 unverdicted novelty 1.0 of 10

    A survey chapter proposing that generative AI reshapes information access through two paradigms, information generation and information synthesis.

Pith tools