Pith. sign in

REVIEW 18 cited by

Are Transformers universal approximators of sequence-to-sequence functions?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1912.10077 v2 pith:ITFBBX2Y submitted 2019-12-20 cs.LG stat.ML

Chulhee Yun , Srinadh Bhojanapalli , Ankit Singh Rawat , Sashank J. Reddi , Sanjiv Kumar This is my paper
classification cs.LGstat.ML
keywords modelsfunctionslayersself-attentionsequence-to-sequencetransformertransformersuniversal
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self-attention layers and empirically evaluate them.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Universal Approximation Theorems for Dynamical Systems with Infinite-Time Horizon Guarantees

    math.DS 2026-02 conditional novelty 7.0 of 10

    Neural ODEs can approximate Morse-Smale and continuous-attractor dynamical systems over infinite time in an ε-δ sense, provided limit-cycle periods are matched exactly.

  3. Identifiability-Aware Source Apportionment in City-Scale Advection-Diffusion Systems

    eess.SP 2026-07 conditional novelty 6.0 of 10

    IASA makes source apportionment report what a sensor network can actually tell apart: the rank and singular values of the projected response matrix set the finest defensible attribution resolution.

  4. On Transformer Dynamics

    math.CO 2026-07 conditional novelty 6.0 of 10

    A universal, finitely parametrized family of geometric interaction laws realizes any prescribed attention digraph, with cost governed by the biclique cover number and a new hub-chromatic index.

  5. Unraveling Syntax: Language Modeling and the Substructure of Grammars

    cs.CL 2025-10 conditional novelty 6.0 of 10

    Language-modeling loss decomposes linearly over the sub-grammars of a probabilistic context-free grammar, and models learn these sub-grammars in parallel rather than in stages.

  6. Decoding Consumer Preferences Using Attention-Based Language Models

    econ.EM 2025-07 conditional novelty 6.0 of 10

    A two-stage language-model method estimates private valuations and bidder counts from car auction descriptions and outperforms OLS and one-stage baselines out of sample.

  7. Time Resolution Independent Operator Learning

    cs.CE 2025-07 conditional novelty 6.0 of 10

    A DeepONet with a neural controlled differential equation branch and a trunk that takes space and time as inputs predicts transient mechanical fields from load histories at arbitrary spatiotemporal query points.

  8. Existing Large Language Model Unlearning Evaluations Are Inconclusive

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.

  9. GSA-TTS : Toward Zero-Shot Speech Synthesis based on Gradual Style Adaptor

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A zero-shot TTS method that splits reference audio into ASR word segments, encodes local styles, and merges them via self-attention improves intelligibility and speaker similarity on unseen voices.

  10. Rethinking Causal Mask Attention for Vision-Language Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Relaxing causal masking so image tokens can preview future image and text context during prefill improves several vision-language benchmarks, and pooling future attention into a single prefix token preserves most of the gain.

  11. Solving Empirical Bayes via Transformers

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A transformer pre-trained on synthetic Poisson data can beat the classical NPMLE estimator on several empirical Bayes tasks and run about 100x faster.

  12. Pre-Strings Lectures on Artificial Intelligence

    hep-th 2026-07 accept novelty 5.5 of 10

    Lecture notes define neural-network field theory and survey how it recovers known QFT/string results plus applied AI techniques for string problems.

  13. A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Single-layer linear self-attention can represent, train on, and length-generalize pairwise interaction functions under data-versatility and exact-realizability assumptions, and the paper introduces higher-order HyperA...

  14. Context-aware Rotary Position Embedding

    cs.CL 2025-07 reject novelty 4.0 of 10

    CARoPE makes rotary position embedding frequencies depend on token content, and the authors report lower perplexity and faster training on GPT-2 variants trained on FineWeb-Edu-10B.

  15. Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

    cs.LG 2025-06 reject novelty 4.0 of 10

    Wavelet-based positional encodings are claimed to improve how transformers extrapolate to longer sequences, with a toy experiment supporting the claim but with weak theory.

  16. Transformers Are Universally Consistent

    cs.LG 2025-05 reject novelty 4.0 of 10

    HyT, a hyperbolic Transformer, is claimed to be universally consistent for L2 regression, but the proof is invalidated by an algebraic error and circular reasoning.

  17. On the Mathematical Impossibility of Safe Universal Approximators

    cs.LG 2025-07 reject novelty 2.0 of 10

    This preprint argues that useful universal approximators must have dense catastrophic failures, but the proof rests on unproven measure convergence and circular use of adversarial examples.

  18. Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques

    cs.LG 2025-06 reject novelty 2.0 of 10

    The paper claims that in-context learning with finite example sets can approximate supervised fine-tuning in transformers, but the proof assumes the very approximation it sets out to establish.

Pith tools