Pith. sign in

REVIEW 16 cited by

Continual Pre-Training of Large Language Models: How to (re)warm your model?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.04014 v2 pith:KTJGQIU4 submitted 2023-08-08 cs.CL cs.LG

Kshitij Gupta , Benjamin Thérien , Adam Ibrahim , Mats L. Richter , Quentin Anthony , Eugene Belilovsky , Irina Rish , Timothée Lesort This is my paper · ORCID
classification cs.CLcs.LG
keywords datamodelsdownstreampre-trainingcontinuallanguagelargeperformance
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these models, i.e. updating pre-trained models with new data instead of re-training them from scratch. However, the distribution shift induced by novel data typically results in degraded performance on past data. Taking a step towards efficient continual pre-training, in this work, we examine the effect of different warm-up strategies. Our hypothesis is that the learning rate must be re-increased to improve compute efficiency when training on a new dataset. We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule. We conduct all experiments on the Pythia 410M language model architecture and evaluate performance through validation perplexity. We experiment with different pre-training checkpoints, various maximum learning rates, and various warmup lengths. Our results show that while rewarming models first increases the loss on upstream and downstream data, in the longer run it improves the downstream performance, outperforming models trained from scratch$\unicode{x2013}$even for a large downstream dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning in Transition

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Continual learning is reframed as continual capability evolution along three axes: when learning happens, where capability is stored, and how updates are driven.

  2. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  3. Universal Music Representations? Evaluating Foundation Models on World Music Corpora

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Five audio foundation models are evaluated across six Western and non-Western music corpora, showing a consistent Western-centric bias and only limited generalization to culturally distant traditions.

  4. Learning Dynamics in Continual Pre-Training for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A continual pre-training scaling law decomposes the loss curve into a pre-training baseline plus a distribution-shift term, enabling per-step and cross-schedule loss prediction.

  5. EnronQA: Towards Personalized RAG over Private Documents

    cs.IR 2025-05 conditional novelty 6.0 of 10

    A new 528,304-question benchmark over 103,638 Enron emails across 150 inboxes lets RAG pipelines be evaluated in a setting where LLMs cannot answer from memory.

  6. Domain Adaptation of Foundation LLMs for e-Commerce

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Continued pretraining Llama 3.1 on 1 trillion e-commerce tokens produces e-Llama 8B/70B models that improve in-house e-commerce benchmarks by roughly 25-30% while retaining most general-domain accuracy.

  7. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  8. MedLLM: An Open Medical Language Model at the Sub-Billion Scale

    cs.AI 2026-07 reject novelty 5.0 of 10

    A 100M-parameter model, MedLLM, is reported to beat 7B medical baselines on MedMCQA and MMLU while staying near the floor on MedQA, but the claimed 'within 2.9pp of a medically adapted 7B' result has no matching row i...

  9. ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

    cs.LG 2026-07 conditional novelty 5.0 of 10

    ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.

  10. Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Router Upcycling initializes MoE routers from the dense model's attention heads, achieving about 2% higher average zero-shot accuracy than vanilla upcycling on small Qwen 8x0.5B models.

  11. CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning

    cs.SD 2025-06 conditional novelty 5.0 of 10

    CultureMERT adapts the MERT-95M music model to Greek, Turkish and Indian traditions via two-stage continual pre-training, improving non-Western auto-tagging by 4.9% on average while keeping Western performance intact.

  12. WenyanGPT: A Large Language Model for Classical Chinese Tasks

    cs.CL 2025-04 reject novelty 5.0 of 10

    The authors train WenyanGPT, a Classical Chinese LLM, and claim it beats GPT-4o and other models on their WenyanBENCH tasks, though the benchmark shares data sources with the training set.

  13. Kuwain 1.5B: An Arabic SLM via Language Injection

    cs.CL 2025-04 conditional novelty 5.0 of 10

    Kuwain 1.5B, built by freezing TinyLlama's original layers and training eight added layers plus a 26K Arabic tokenizer, improves Arabic benchmark average from 36.95 to 44.49 while keeping English near 53.28.

  14. Sparse Upcycling: Inference Inefficient Finetuning

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Sparse upcycling beats continued pretraining on quality by up to roughly 20 percent at matched compute, but cut serving throughput by 34 to 44 percent in vLLM benchmarks.

  15. DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model

    cs.CV 2025-04 conditional novelty 4.0 of 10

    Fine-tuning an open vision-language model with multi-task learning and perceptual chain-of-thought, plus MBR decoding, achieves strong BLEU scores on end-to-end document image translation at ICDAR 2025.

  16. ChronoLLM: A Framework for Customizing Large Language Model for Digital Twins generalization based on PyChrono

    cs.SE 2025-01 conditional novelty 4.0 of 10

    Fine-tuning LLMs on PyChrono-specific data improves their success rate at generating runnable simulation code from about 40% to about 85%, compared to prompting general models.

Pith tools