Pith. sign in

REVIEW 11 cited by

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.10429 v4 pith:CWGHMEMW submitted 2023-05-17 cs.CL cs.LG

Sang Michael Xie , Hieu Pham , Xuanyi Dong , Nan Du , Hanxiao Liu , Yifeng Lu , Percy Liang , Quoc V. Le
show 2 more authors
Tengyu Ma Adams Wei Yu
This is my paper
classification cs.CLcs.LG
keywords domainmodeldoremiweightsdownstreamdomainstasksaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The mixture proportions of pretraining data domains (e.g., Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting with Minimax Optimization (DoReMi), which first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights (mixture proportions) without knowledge of downstream tasks. We then resample a dataset with these domain weights and train a larger, full-sized model. In our experiments, we use DoReMi on a 280M-parameter proxy model to set the domain weights for training an 8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves perplexity across all domains, even when it downweights a domain. DoReMi improves average few-shot downstream accuracy by 6.5% points over a baseline model trained using The Pile's default domain weights and reaches the baseline accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has no knowledge of downstream tasks, even matches the performance of using domain weights tuned on downstream tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

    cs.LG 2026-07 unverdicted novelty 7.0 of 10

    HERMES provides a reusable hierarchical labeling substrate for pre-training data that reveals granularity-specific effects in data mixing rules during model training.

  2. Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    During pretraining, language models exhibit natural ungrokking where learned rules are forgotten based on their support frequency in the corpus, with asymmetric editability of rule survival.

  3. Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Robotics

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    Ambient Diffusion Policy enables better imitation learning from suboptimal robot data by leveraging spectral properties to restrict data usage to specific diffusion times.

  4. RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A controlled benchmark shows LLM agents can sometimes discover better training-data strategies through feedback, but their improvements are fragile and usually not sustained.

  5. Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

    cs.LG 2026-05 conditional novelty 6.0 of 10

    Mixing auxiliary high-resource language data outperforms hyperparameter tuning in data-constrained bilingual pre-training, with gains equivalent to 2-13 times more unique target data.

  6. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning

    cs.LG 2026-02 reject novelty 6.0 of 10

    GradAlign selects RL training problems whose policy gradients align with validation-set gradients, claiming more stable and better-performing LLM post-training.

  7. Multi-Task GRPO: Reliable LLM Reasoning Across Tasks

    cs.CL 2026-02 conditional novelty 6.0 of 10

    MT-GRPO reweights tasks by reward and improvement and enforces those weights after zero-gradient filtering, improving worst-task accuracy by 6–28% over GRPO/DAPO baselines on 3- and 9-task setups.

  8. Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Models

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Fin-PRM is a domain-specialized process reward model that supplies binary step-level and trajectory-level supervision signals for financial reasoning in LLMs and outperforms general PRMs on CFLUE and FinQA benchmarks.

  9. Llemma: An Open Language Model For Mathematics

    cs.CL 2023-10 unverdicted novelty 6.0 of 10

    Continued pretraining of Code Llama on Proof-Pile-2 yields Llemma, an open math-specialized LLM that beats known open base models on MATH and supports tool use plus formal proving out of the box.

  10. AgentOmnia: Scaling Agentic Models for Full-Scenario Applications

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Post-training an open Qwen3-30B agent on 52,361 verifiable tasks in 5,018 synthesized stateful environments lifts its average across four agent benchmarks from 22.9% to 41.7%.

  11. A Survey of Large Language Models

    cs.CL 2023-03 accept novelty 3.0 of 10

    This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.

Pith tools