Pith. sign in

REVIEW 6 cited by

Linear attention is (maybe) all you need (to understand transformer optimization)

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01082 v2 pith:R7BWDAPI submitted 2023-10-02 cs.LG cs.AImath.OC

Kwangjun Ahn , Xiang Cheng , Minhak Song , Chulhee Yun , Ali Jadbabaie , Suvrit Sra This is my paper
classification cs.LGcs.AImath.OC
keywords transformerlinearizedtraininglinearmodeloptimizationsimpletransformers
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Convergence Behavior of Adam under Heavy-Tailed Noise

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Under heavy-tailed noise with bounded p-th moments, vector-form Adam converges to (ρ,ε)-stationary points at rate O(ε^{-(5p/(3p-4)+3/2)}) for p∈(4/3,2]; with known-radius clipping the rate is optimal O(ε^{-(p/(p-1)+3/2)}).

  2. Nonconvex Decentralized Stochastic Bilevel Optimization under Heavy-Tailed Noise

    cs.LG 2025-09 conditional novelty 6.0 of 10

    The paper introduces D-NSVRGDA, a decentralized normalized variance-reduced method for nonconvex bilevel optimization, and proves the first convergence rate under heavy-tailed noise without gradient clipping.

  3. Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A one-layer linear-attention transformer implementing noisy gradient descent gives provable bounds showing linear noise plus ensembling avoids label-noise overfitting, and majority voting beats greedy decoding in spar...

  4. Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.

  5. A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Single-layer linear self-attention can represent, train on, and length-generalize pairwise interaction functions under data-versatility and exact-realizability assumptions, and the paper introduces higher-order HyperA...

  6. Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Central-difference random gradient estimation (CD-RGE) trains RNNs up to 1.1B parameters without backpropagation through time, matching or beating BPTT on several small tasks while using less GPU memory.

Pith tools