REVIEW 4 cited by
A Survey on Efficient Training of Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Survey on Efficient Training of Transformers
read the original abstract
Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.
Forward citations
Cited by 4 Pith papers
-
Imputation Meets Clustering: Exploiting Latent Subgroup Structure for Missing Data Recovery
Alternating clustering and GAN-based imputation in a feedback loop yields more accurate missing-value recovery on heterogeneous data than single-distribution methods.
-
Agentic Physical AI toward a Domain-Specific Foundation Model for Nuclear Reactor Control
A compact language model trained on scaled synthetic nuclear reactor control data exhibits variance collapse and emergent concentration on a single actuation strategy driven by physical execution success.
-
Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation
Severity-specific fine-tuning of Wav2Vec2 with SRM, PM, FM, and VTLP augmentations yields relative WER reductions of 15-30% on dysarthric speech across severity levels.
-
Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
This survey organizes LLM optimizer literature into categories and argues the field is shifting toward rigorous, multi-factor comparisons of convergence, memory, stability, and complexity.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.