REVIEW 8 cited by
Understanding the Difficulty of Training Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Transformers have proved effective in many NLP tasks. However, their training requires non-trivial efforts regarding designing cutting-edge optimizers and learning rate schedulers carefully (e.g., conventional SGD fails to train Transformers effectively). Our objective here is to understand $\textit{what complicates Transformer training}$ from both empirical and theoretical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that influences training substantially -- for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output. Yet we observe that a light dependency limits the model potential and leads to inferior trained models. Inspired by our analysis, we propose Admin ($\textbf{Ad}$aptive $\textbf{m}$odel $\textbf{in}$itialization) to stabilize stabilize the early stage's training and unleash its full potential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance. Implementations are released at: https://github.com/LiyuanLucasLiu/Transforemr-Clinic.
Forward citations
Cited by 8 Pith papers
-
One task to rule them all: A closer look at traffic classification generalizability
Traffic classifiers that seem near-perfect on their own datasets fall to 30-40% accuracy on another network's same-task data, and a 1-Nearest Neighbor baseline is competitive.
-
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Controlling model complexity through smaller initialization rates and stronger weight decay improved LLM benchmark scores and made loss-versus-scale curves descend faster.
-
One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.
-
Avoiding spurious sharpness minimization broadens applicability of SAM
SAM's failure in language modeling is traced to a dominant 'logit path' that minimizes sharpness spuriously, and the proposed Functional-SAM, which removes that path, improves validation loss over AdamW and SAM.
-
On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.
-
SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.
-
An Analysis for Reasoning Bias of Language Models with Small Initialization
Initialization scale controls whether a transformer learns compositional reasoning or memorized mappings, because reasoning tokens acquire more differentiated embeddings early in training.
-
DBF-Net: A Dual-Branch Network with Feature Fusion for Ultrasound Image Segmentation
DBF-Net, a dual-branch network with body and boundary supervision plus feature fusion, reports Dice scores of 81.05%, 76.41%, and 87.75% on BUSI, UNS, and UHES ultrasound datasets.
Discussion (0). Continue with ORCID to comment.