Pith. sign in

REVIEW 5 cited by

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.07263 v1 pith:GYDXRJ6J submitted 2024-07-09 cs.CL

Jupinder Parmar , Sanjev Satheesh , Mostofa Patwary , Mohammad Shoeybi , Bryan Catanzaro This is my paper
classification cs.CL
keywords pretrainingcontinuedlanguagemodelmodelsreusecostrecipe
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  2. DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining

    cs.CV 2025-09 conditional novelty 6.0 of 10

    DIET-CP continues pretraining a vision model by classifying each image by its index on roughly 1000 unlabeled images, improving medical and astronomical classification while degrading fine-grained natural-image classi...

  3. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  4. Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Properly spaced multi-epoch reuse of high-quality data, guided by a measured memorization window, continues to improve LLM performance far beyond the common four-epoch heuristic.

  5. Large-Scale Diverse Synthesis for Mid-Training

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    BoostQA is a 100B-token synthesized QA corpus whose mid-training on a 40B-token subset improves Llama-3 8B by 12.74% on average across MMLU and CMMLU and reaches top average performance on 12 benchmarks.

Pith tools