Pith. sign in

REVIEW 4 cited by

Block-Recurrent Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.07852 v3 pith:HJRDBO4B submitted 2022-03-11 cs.LG cs.AIcs.NE

DeLesley Hutchins , Imanol Schlag , Yuhuai Wu , Ethan Dyer , Behnam Neyshabur This is my paper · ORCID
classification cs.LGcs.AIcs.NE
keywords transformercelllayerrecurrenttokensblock-recurrentcodecomputation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce the Block-Recurrent Transformer, which applies a transformer layer in a recurrent fashion along a sequence, and has linear complexity with respect to sequence length. Our recurrent cell operates on blocks of tokens rather than single tokens during training, and leverages parallel computation within a block in order to make efficient use of accelerator hardware. The cell itself is strikingly simple. It is merely a transformer layer: it uses self-attention and cross-attention to efficiently compute a recurrent function over a large set of state vectors and tokens. Our design was inspired in part by LSTM cells, and it uses LSTM-style gates, but it scales the typical LSTM cell up by several orders of magnitude. Our implementation of recurrence has the same cost in both computation time and parameter count as a conventional transformer layer, but offers dramatically improved perplexity in language modeling tasks over very long sequences. Our model out-performs a long-range Transformer XL baseline by a wide margin, while running twice as fast. We demonstrate its effectiveness on PG19 (books), arXiv papers, and GitHub source code. Our code has been released as open source.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 28 citations worldwide. Full citation record

  1. Addressable Memory for Video World Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video world models can recall revisited scenes far beyond their training horizon by storing compressed memory at fixed in-distribution positions and averaging keys in a rotation-free space.

  2. StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel

    cs.LG 2025-01 conditional novelty 6.0 of 10

    StagFormer staggers transformer layers one time step apart with delayed cross-attention, enabling depth-parallel decoding at quality comparable to a deeper baseline.

  3. Doctoral Thesis: Geometric Deep Learning For Camera Pose Prediction, Registration, Depth Estimation, and 3D Reconstruction

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A PhD thesis showing that adding geometric priors (skyline, normals, focus cues, wavelet depth) to deep networks improves pose estimation, registration, depth prediction, and reconstruction.

  4. Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures

    cs.CL 2025-02 conditional novelty 4.0 of 10

    A hybrid architecture with two QRNN layers followed by a PAR Transformer reaches 1.013 BPC on enwik8 and 20.91 PPL on Wikitext-103 with 41-60M parameters.

Pith tools