REVIEW 4 cited by
Looped Transformers as Programmable Computers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.
Forward citations
Cited by 4 Pith papers
-
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
In looped transformers, halting-gate failures come mainly from how gate training reshapes the trajectory; fixed-prior depth supervision plus simple confidence readouts yields better accuracy per unit of compute.
-
StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel
StagFormer staggers transformer layers one time step apart with delayed cross-attention, enabling depth-parallel decoding at quality comparable to a deeper baseline.
-
Intra-Layer Recurrence in Transformers for Language Modeling
Selectively reusing early transformer layers inside a forward pass improves language modeling perplexity without adding parameters, but at extra compute cost and with evidence only from small-scale runs.
-
Code Simulation as a Proxy for High-order Tasks in Large Language Models
LLM performance on naturalistic reasoning tasks tracks performance on equivalent Python code simulation, but the effect is partly driven by pattern matching and memorization rather than faithful execution.
Discussion (0). Continue with ORCID to comment.