Pith. sign in

REVIEW 11 cited by

In-Context Convergence of Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.05249 v1 pith:QPVP3C7O submitted 2023-10-08 cs.LG cs.AImath.OCstat.ML

Yu Huang , Yuan Cheng , Yingbin Liang This is my paper · ORCID
classification cs.LGcs.AImath.OCstat.ML
keywords featureslearningin-contexttransformersattentionconvergencedatadynamics
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Transformers have recently revolutionized many domains in modern machine learning and one salient discovery is their remarkable in-context learning capability, where models can solve an unseen task by utilizing task-specific prompts without further parameters fine-tuning. This also inspired recent theoretical studies aiming to understand the in-context learning mechanism of transformers, which however focused only on linear transformers. In this work, we take the first step toward studying the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent in order to in-context learn linear function classes. We consider a structured data model, where each token is randomly sampled from a set of feature vectors in either balanced or imbalanced fashion. For data with balanced features, we establish the finite-time convergence guarantee with near-zero prediction error by navigating our analysis over two phases of the training dynamics of the attention map. More notably, for data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process, where the transformer first converges to a near-zero prediction error for the query tokens of dominant features, and then converges later to a near-zero prediction error for the query tokens of under-represented features, respectively via one and four training phases. Our proof features new techniques for analyzing the competing strengths of two types of attention weights, the change of which determines different training phases.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Emergence of Implicit Curriculum in RLVR Learning Dynamics

    cs.LG 2026-02 unverdicted novelty 8.0 of 10

    RLVR training on transformers for compositional tasks follows an implicit curriculum from easy to hard problems, with difficulty spectrum smoothness determining steady relay progress or grokking phase transitions.

  2. Provable Low-Frequency Bias of In-Context Learning of Representations

    cs.LG 2025-07 conditional novelty 7.0 of 10

    In-context learning biases hidden representations toward low-frequency eigenvectors of a reweighted graph Laplacian, a phenomenon the authors prove and test.

  3. Learning Compositional Functions with Transformers from Easy-to-Hard Data

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A transformer with O(log k) layers provably learns the k-fold permutation composition task in poly(N,k) samples with curriculum or mixed easy-to-hard data, despite an SQ lower bound requiring N^{Omega(k)} samples on h...

  4. Reassessing Muon for Matrix Factorization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Muon's advantage over AdamW is problem-dependent: it loses or ties on plain low-rank factorization and completion but wins on nonnegative matrix factorization.

  5. Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Under hand-designed masks and task-specific activations, RL fine-tuning learns a k-sparse Boolean reasoning chain in one gradient update while SFT learns it one CoT step per update.

  6. How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A one-layer transformer trained on even pairs provably passes through a fast attention-growth phase into a slow max-margin phase, and with chain-of-thought the same model can solve parity checking.

  7. Transformers versus the EM Algorithm in Multi-class Clustering

    stat.ML 2025-02 conditional novelty 6.0 of 10

    A pretrained transformer can approximate Lloyd's EM algorithm for multi-class Gaussian clustering and can achieve the minimax optimal clustering error with enough pretraining data.

  8. Learning Spectral Methods by Transformers

    stat.ML 2025-01 conditional novelty 6.0 of 10

    Pre-trained Transformers can provably approximate spectral methods (PCA and two-class Gaussian clustering) when trained on many instances, with bounds that decay as sample size grows.

  9. Rethinking Associative Memory Mechanism in Induction Head

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A two-layer transformer with relative positional encoding keeps its induction head active across the whole sequence, while absolute positional encoding loses it in the second half.

  10. Minimalist Softmax Attention Provably Learns Constrained Boolean Functions

    cs.LG 2025-05 reject novelty 5.0 of 10

    With teacher forcing that reveals pairwise products of the relevant bits, one gradient step lets a single-head attention recover the support of a k-bit AND/OR; the paper's claimed end-to-end hardness lower bound is in...

  11. StaICC: Standardized Evaluation for Classification Task in In-context Learning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    StaICC standardizes in-context classification evaluation with fixed prompts and splits, then measures 29 LMs and 10 inference methods under those fixed settings.

Pith tools