Pith. sign in

REVIEW 1 cited by

Reservoir Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.15045 v2 pith:KKUJ66P5 submitted 2020-12-30 cs.CL

Reservoir Transformers

Sheng Shen , Alexei Baevski , Ari S. Morcos , Kurt Keutzer , Michael Auli , Douwe Kiela This is my paper
classification cs.CL
keywords layersmachineperformancereservoirtransformerscomputeconvergencedemonstrate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Quantum Reservoir Computing: Recent Advances and Future Directions

    quant-ph 2026-07 accept novelty 4.0

    A comprehensive survey of quantum reservoir computing that proposes a common system model, a memory-architecture taxonomy, and resource-accounting standards, concluding that no broad quantum advantage is currently dem...