REVIEW 3 cited by
Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training vision or language models on large datasets can take days, if not weeks. We show that averaging the weights of the k latest checkpoints, each collected at the end of an epoch, can speed up the training progression in terms of loss and accuracy by dozens of epochs, corresponding to time savings up to ~68 and ~30 GPU hours when training a ResNet50 on ImageNet and RoBERTa-Base model on WikiText-103, respectively. We also provide the code and model checkpoint trajectory to reproduce the results and facilitate research on reusing historical weights for faster convergence.
Forward citations
Cited by 3 Pith papers
-
EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
A bias-corrected exponential moving average (BEMA) is claimed to remove the lag of standard EMA weight averaging during LLM fine-tuning, improving convergence and final performance over EMA and vanilla training.
-
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Checkpoint merging during constant-LR training can replace LR decay and yields improved LLM benchmark scores over Warmup-Stable-Decay.
-
SeWA: Selective Weight Average via Probabilistic Masking
SeWA adaptively selects a sparse set of checkpoints for weight averaging via learned probabilistic masks, claiming better generalization with fewer averaged points.
Discussion (0). Continue with ORCID to comment.