Pith. sign in

REVIEW 16 cited by

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.07263 v1 pith:GYDXRJ6J submitted 2024-07-09 cs.CL

Jupinder Parmar , Sanjev Satheesh , Mostofa Patwary , Mohammad Shoeybi , Bryan Catanzaro This is my paper
classification cs.CL
keywords pretrainingcontinuedlanguagemodelmodelsreusecostrecipe
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.

  2. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  3. How Post-Training Shapes Biological Reasoning Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Post-training stages reshape generalization in biological reasoning models distinctly: CPT aligns with biological language, SFT boosts ID performance but causes OOD to peak early and decline, while RL on strong SFT ch...

  4. Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Optimal hyperparameters for LLM continued pre-training follow predictable scaling laws derived from proxy models, enabling a two-stage framework that predicts settings from compute budget and checkpoint state to reduc...

  5. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Expert upcycling expands MoE models by duplicating experts and continuing pre-training, matching baseline performance while saving 32% GPU hours in 7B-13B experiments.

  6. DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining

    cs.CV 2025-09 conditional novelty 6.0 of 10

    DIET-CP continues pretraining a vision model by classifying each image by its index on roughly 1000 unlabeled images, improving medical and astronomical classification while degrading fine-grained natural-image classi...

  7. Optimization Hyper-parameter Laws for Large Language Models

    cs.LG 2024-09 unverdicted novelty 6.0 of 10

    Opt-Laws predicts LLM final training loss from LR schedules via SDE-derived convergence and escape features, with 94% Top-2 hit rate on held-out schedules and F1=0.92 for divergence detection.

  8. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time decoder-only LMs to 4B parameters on 1T chronologically filtered FineWeb tokens largely closes the quality gap to unrestricted open models while preserving temporal validity.

  9. Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Properly spaced multi-epoch reuse of high-quality data, guided by a measured memorization window, continues to improve LLM performance far beyond the common four-epoch heuristic.

  10. Small LLMs: Pruning vs. Training from Scratch

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Controlled experiments show pruning a large LLM outperforms training small models from scratch with matched token budgets, but the advantage narrows with extra tokens or coarse pruning.

  11. Small LLMs: Pruning vs. Training from Scratch

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Pruned initializations from an 8B model outperform random starts with equal training tokens, but with full token budgets fine-grained pruning retains advantage while coarse structured pruning does not.

  12. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.0 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  13. Large-Scale Diverse Synthesis for Mid-Training

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    BoostQA is a 100B-token synthesized QA corpus whose mid-training on a 40B-token subset improves Llama-3 8B by 12.74% on average across MMLU and CMMLU and reaches top average performance on 12 benchmarks.

  14. WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

    cs.CL 2024-08 unverdicted novelty 5.0 of 10

    WildFeedback extracts preference pairs from in-situ user feedback in LLM conversations to fine-tune models for better alignment with real user preferences.

  15. Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights

    cs.CR 2026-05 unverdicted novelty 3.0 of 10

    Domain-adapted LLMs and SLMs do not consistently outperform general models on STRIDE threat classification for 5G, with decoding strategies and model scale affecting validity but gains remaining insufficient for reliable use.

  16. Phoenix-VL 1.5 Medium Technical Report

    cs.CL 2026-05 unverdicted novelty 3.0 of 10

    Phoenix-VL 1.5 Medium is a 123B-parameter natively multimodal model that reaches state-of-the-art results on Singapore multimodal, legal, and policy benchmarks after localized training on 1T+ tokens while staying comp...

Pith tools