REVIEW 7 cited by
Spike No More: Stabilizing the Pre-training of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.
Forward citations
Cited by 7 Pith papers
-
Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions
HyenaND is an input-dependent, multi-dimensional convolution that runs in near-linear time and matches attention baselines on vision, medical, PDE, and genomics benchmarks.
-
When Does Sparsity Mitigate the Curse of Depth in LLMs
Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.
-
DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers
DataStates-LLM hides LLM checkpointing overhead by lazily copying immutable weights during forward/backward passes and streaming heterogeneous shards to storage via composable state providers, cutting end-to-end train...
-
GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
GPAS scales down intermediate activations while preserving backward gradients, reducing activation variance growth in Pre-LN transformers and improving pretraining convergence and downstream performance.
-
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Weight decay on scale-invariant weights creates a norm-dependent sharpness boundary; crossing it predicts loss spikes in normalized networks.
-
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
SpanNorm—a block-level residual with PostNorm-style normalization—trains deeper transformers more stably and outperforms PreNorm and hybrid normalization on LM benchmarks.
-
Beyond Text Compression: Evaluating Tokenizers Across Scales
Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.
Discussion (0). Sign in to comment.