Pith. sign in

REVIEW 2 major objections 2 minor 31 references

Fixed Universal Transformers

T0 review · 2 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fixed transformers can simulate any transformer in a class by encoding its description in the input embedding.

desk verdict Fixed transformers become universal by encoding target models in the input embedding, via explicit sparse constructions when dim is large enough plus an almost-sure result for random weights. read the letter →

arxiv 2605.31423 v1 pith:NID7CIB7 submitted 2026-05-29 cs.LG

Jingwen Liu , Alexandr Andoni , Daniel Hsu This is my paper
classification cs.LG
keywords universaltransformersfixedweightsinputembeddingexpressivepowersimulationrandominitializationparenthesisbalancingmulti-hopreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that there exist fixed transformers whose weights never change but which can still act as any member of a broad class of transformers. This happens because the input to the fixed model includes an encoding of the particular transformer one wants to simulate. A sympathetic reader would care because it relocates most of the model's capability from the learned weights to the choice of input representation. The authors give explicit constructions that work when the embedding dimension is large, prove that random fixed transformers are almost always universal, and check the idea on parenthesis balancing and multi-hop reasoning tasks. If the claim holds, training a transformer may be less about discovering the right weights and more about discovering the right way to present information to a fixed computational core.

What carries the argument

The input embedding that encodes a description of the target transformer, allowing the fixed transformer to simulate it.

What would settle it

A concrete class of transformers for which no fixed transformer succeeds in simulating every member when given only an input embedding that describes the target.

Watch

Extended reading notes

Core claim

We introduce universal transformers: fixed transformers that can simulate any transformer in a given class via a suitable input embedding. Analogous to a universal Turing machine, the input embedding encodes a description of the target model while all internal parameters remain fixed. We provide explicit sparse constructions achieving universality when the embedding dimension is sufficiently large, and further show that universality is generic: randomly initialized transformers are universal almost surely, which aligns with recent empirical results of Zhong and Andreas (2024). We empirically validate our theory on the algorithmic tasks of parenthesis balancing and multi-hop reasoning. Our re

Load-bearing premise

The embedding dimension is large enough to encode a description of the target transformer.

Editorial extensions

If this is right

  • Explicit sparse constructions achieve universality whenever the embedding dimension is large enough.
  • Randomly initialized transformers are universal almost surely.
  • Expressive power largely resides in the input representation rather than the learned weights.
  • The construction is validated on parenthesis balancing and multi-hop reasoning tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Training might shift focus toward learning effective input encodings for a fixed computational core.
  • The result offers one explanation for why prompt engineering can elicit new behaviors without weight changes.
  • One could test whether a single randomly initialized fixed transformer handles many distinct tasks when supplied with appropriate input descriptions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces universal transformers: fixed-parameter transformers that simulate any transformer from a given class by encoding a description of the target model in a suitable input embedding, analogous to a universal Turing machine. It provides explicit sparse constructions achieving this when the embedding dimension is large enough to encode the target description, and proves that universality holds generically (randomly initialized transformers are universal almost surely). The claims are supported by empirical validation on parenthesis balancing and multi-hop reasoning tasks.

Significance. If the constructions and almost-sure universality argument hold, the result is significant: it provides a theoretical basis for the idea that much of a transformer's power resides in input representations rather than learned weights, aligning with and explaining recent empirical observations. The explicit sparse constructions and generic (parameter-free) almost-sure result are notable strengths that could influence how expressivity is analyzed in sequence models.

major comments (2)
  1. [Abstract / Constructions section] The abstract states that explicit sparse constructions exist when embedding dimension suffices to encode the target; the full manuscript must include the precise construction (e.g., how the embedding encodes weights and architecture) and the proof that the fixed transformer simulates the target, as this is load-bearing for the central universality claim.
  2. [Generic universality argument] The generic universality result (random weights yield universality a.s.) is stated as aligning with Zhong and Andreas (2024); the manuscript should specify the probability space and measure under which 'almost surely' holds, and confirm it does not rely on any hidden parameter tuning.
minor comments (2)
  1. [Empirical validation] The empirical section should report quantitative metrics (accuracy, number of examples, comparison to non-universal baselines) rather than only stating that validation was performed on the two tasks.
  2. Ensure the reference to Zhong and Andreas (2024) appears in the bibliography with full citation details.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive review and constructive feedback. We address each major comment below and will incorporate clarifications in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract / Constructions section] The abstract states that explicit sparse constructions exist when embedding dimension suffices to encode the target; the full manuscript must include the precise construction (e.g., how the embedding encodes weights and architecture) and the proof that the fixed transformer simulates the target, as this is load-bearing for the central universality claim.

    Authors: We agree that the explicit construction and simulation proof are central. Section 3 of the manuscript already provides the sparse construction, including how the input embedding encodes the target transformer's weights and architecture via a fixed sparse attention pattern that routes information according to the encoded description. The proof that the fixed transformer simulates the target appears in Theorem 3.1. To address the comment, we will expand this section with a fully worked example (including explicit embedding vectors for a small target model) and a more detailed proof sketch in the revision. revision: yes

  2. Referee: [Generic universality argument] The generic universality result (random weights yield universality a.s.) is stated as aligning with Zhong and Andreas (2024); the manuscript should specify the probability space and measure under which 'almost surely' holds, and confirm it does not rely on any hidden parameter tuning.

    Authors: We will revise the relevant section (currently Section 4) to explicitly define the probability space as the product measure over independent continuous distributions (standard Gaussian) on each entry of the fixed transformer's parameter matrices. Under this measure, the set of parameter values for which universality fails has Lebesgue measure zero. The argument uses no hidden parameter tuning: the fixed parameters are drawn once from this distribution and then held constant, with universality holding for almost all such draws. This aligns with the measure-theoretic argument in Zhong and Andreas (2024) but is self-contained. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper states explicit sparse constructions for universality conditioned on embedding dimension being large enough to encode a target transformer description, plus a generic almost-sure result for random weights. These are presented as holding under the stated assumptions with no reduction of claims to fitted parameters, self-definitions, or load-bearing self-citations. The single external citation (Zhong and Andreas 2024) is not used to justify the core constructions or uniqueness. The argument chain is self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review provides no visible free parameters, axioms, or invented entities; ledger entries cannot be populated from available text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fixed Universal Transformers." pith.science (2026). https://pith.science/paper/NID7CIB7

@misc{pith2026260531423,
  author       = {Pith},
  title        = {Pith review of: Fixed Universal Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NID7CIB7}},
  note         = {Machine review of arXiv:2605.31423}
}
read the original abstract

We introduce \emph{universal transformers}: fixed transformers that can simulate any transformer in a given class via a suitable input embedding. Analogous to a universal Turing machine, the input embedding encodes a description of the target model while all internal parameters remain fixed. We provide explicit sparse constructions achieving universality when the embedding dimension is sufficiently large, and further show that universality is generic: randomly initialized transformers are universal almost surely, which aligns with recent empirical results of Zhong and Andreas (2024). We empirically validate our theory on the algorithmic tasks of parenthesis balancing and multi-hop reasoning. Our results suggest that much of a transformer's expressive power may reside in its input representation rather than its learned weights.

Figures

Figures reproduced from arXiv: 2605.31423 by the authors.

Figure 1
Figure 1. Schematic overview of universal transformers. (a) Each target transformer corresponds [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Embedding dimension scaling for 2-hop induction heads. Results are shown for 2-head, 3-layer transformers. Left: Performance under constructed parameterization from Theorem 3.1, where m = m. Right: Performance under standard parameterization, where m = Hd. Error bars indicate ±1 standard deviation across 5 random seeds. On parenthesis balancing, the sparse universal transformer is robust across all variants and achi… view at source ↗
Figure 3
Figure 3. Block structure of E and a schematic recursive expansion. Likewise, R 2,h2 Q and R 2,h2 K selects the corresponding Q, K part in each El=1,h1 , and R 2,h2 V selects El=2,h1,h2 . Do this recursively until the L-th layer, where we set each EL,h1,...,hL = W 1,h1 V W 1,h1 O · · · W L,hL V W L,hL O . We formalize it below. For convenience define Ml,h = W l,h V W l,h O ∈ R din×din , M1:t,(h1,...,ht) = Yt s=1 Ms,hs ∈ R din… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 2 canonical work pages

  1. [1]

    On scram- bling phenomena for randomly initialized recurrent networks.Advances in Neural Information Processing Systems, 35:18501–18513, 2022

    Vaggos Chatziafratis, Ioannis Panageas, Clayton Sanford, and Stelios Stavroulakis. On scram- bling phenomena for randomly initialized recurrent networks.Advances in Neural Information Processing Systems, 35:18501–18513, 2022

    2022

  2. [2]

    Model reprogramming: Resource-efficient cross-domain machine learning

    Pin-Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22584–22591, 2024

    2024

  3. [3]

    The lottery ticket hypothesis for pre-trained bert networks

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. InAdvances in Neural Information Processing Systems 33, pages 15834–15846, 2020

    2020

  4. [4]

    Schützenberger

    Noam Chomsky and Marcel P. Schützenberger. The algebraic theory of context-free languages. Studies in Logic and the Foundations of Mathematics, 35:118–161, 1963

    1963

  5. [5]

    Turing completeness of bounded-precision recurrent neural networks

    Stephen Chung and Hava Siegelmann. Turing completeness of bounded-precision recurrent neural networks. InAdvances in Neural Information Processing Systems 34, pages 28431–28441, 2021

    2021

  6. [6]

    Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

    George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

    1989

  7. [7]

    Uni- versal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. InInternational Conference on Learning Representations, 2019

    2019

  8. [8]

    arXiv preprint arXiv:2506.01115 , year=

    Yihe Dong, Lorenzo Noci, Mikhail Khodak, and Mufan Li. Is random attention sufficient for sequence modeling? disentangling trainable components in the transformer.arXiv preprint arXiv:2506.01115, 2025

    work page arXiv 2025

Show all 31 references
  1. [9]

    A mathematical framework for transformer circuits.Transformer Circuits Thread,

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

  2. [10]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

  3. [11]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. InInternational Conference on Learning Representations, 2019

    2019

  4. [12]

    On the approximate realization of continuous mappings by neural networks

    Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 2(3):183–192, 1989. 10

    1989

  5. [13]

    Looped transformers as programmable computers

    Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. InInternational Conference on Machine Learning, 2023

    2023

  6. [14]

    Multilayer feedforward networks are universal approximators.Neural Networks, 2(5):359–366, 1989

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators.Neural Networks, 2(5):359–366, 1989

    1989

  7. [15]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems 31, 2018

    2018

  8. [16]

    echo state

    Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks. GMD Report 148, GMD - German National Research Institute for Computer Science, 2001

    2001

  9. [17]

    Real-time computing without stable states: A new framework for neural computation based on perturbations.Neural compu- tation, 14(11):2531–2560, 2002

    Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations.Neural compu- tation, 14(11):2531–2560, 2002

    2002

  10. [18]

    Priors for infinite networks

    Radford M Neal. Priors for infinite networks. InBayesian learning for neural networks, pages 29–53. Springer, 1996

    1996

  11. [19]

    In-context learning and induction heads.Transformer Circuits Thread, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  12. [20]

    The strong lottery ticket hypothesis for multi-head attention mechanisms

    Hikari Otsuka, Daiki Chijiwa, Yasuyuki Okoshi, Daichi Fujiki, Susumu Takeuchi, and Masato Motomura. The strong lottery ticket hypothesis for multi-head attention mechanisms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 24639–24647, 2026

    2026

  13. [21]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAdvances in Neural Information Processing Systems 20, 2007

    2007

  14. [22]

    Transformers, parallel computation, and logarithmic depth

    Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Transformers, parallel computation, and logarithmic depth. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  15. [23]

    On the computational power of neural nets

    Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. InFifth Annual ACM Conference on Computational Learning Theory, pages 440–449, 1992

    1992

  16. [24]

    Alan M. Turing. On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, 2(42):230–265, 1936

    1936

  17. [25]

    Universal circuits (preliminary report)

    Leslie G Valiant. Universal circuits (preliminary report). InProceedings of the eighth annual ACM symposium on Theory of computing, pages 196–203, 1976

    1976

  18. [26]

    Statistically meaningful approximation: a case study on approximating turing machines with transformers.Advances in Neural Information Processing Systems, 35:12071–12083, 2022

    Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers.Advances in Neural Information Processing Systems, 35:12071–12083, 2022

    2022

  19. [27]

    V oice2series: Reprogramming acoustic models for time series classification

    Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. V oice2series: Reprogramming acoustic models for time series classification. InInternational conference on machine learning, pages 11808–11819. PMLR, 2021

    2021

  20. [28]

    Self-attention networks can process bounded hierarchical languages

    Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Chengqing Zong, Fei Xia, Wen- jie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Asso- ciation for C...

  21. [29]

    doi: 10.18653/v1/2021.acl-long.292

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.292. URL https://aclanthology.org/2021.acl-long.292/

    2021 doi

  22. [30]

    Are transformers universal approximators of sequence-to-sequence functions? InInternational Conference on Learning Representations, 2020

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? InInternational Conference on Learning Representations, 2020. 11

    2020

  23. [31]

    ?” followed by the target answer token. The answer is “)

    Ziqian Zhong and Jacob Andreas. Algorithmic capabilities of random transformers. InAdvances in Neural Information Processing Systems 37, 2024. 12 A Proof of Theorem 3.1 (universal transformer) Given an input X0, let Y :=X L denote the output of the target transformer, and Z :=...

    2024

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.