Pith. sign in

REVIEW 4 cited by

Hierarchical Transformers Are More Efficient Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.13711 v2 pith:INEYQLSG submitted 2021-10-26 cs.LG cs.CL

Piotr Nawrot , Szymon Tworkowski , Micha{l} Tyrolski , {L}ukasz Kaiser , Yuhuai Wu , Christian Szegedy , Henryk Michalewski This is my paper · ORCID
classification cs.LGcs.CL
keywords transformershierarchicallanguagemodelstransformerhourglasslongefficiently
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass - a hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Controllably Efficient Language Models

    cs.LG 2025-11 conditional novelty 6.0 of 10

    A single transformer variant can compress past context into chunk summaries and use chunk size as a test-time knob to trade quality against speed and memory, outperforming many efficient baselines on recall benchmarks.

  2. VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A pixel-space hierarchical video diffusion transformer solves PDE forward, inverse, and sparse-observation tasks by inpainting trajectories, with reported order-of-magnitude error reductions on several 2D benchmark PDEs.

  3. Synergy: End-to-end Concept Model

    cs.CL 2025-07 reject novelty 5.0 of 10

    A byte-level transformer with a learned top-k router matches a tokenized Llama3 baseline on Wikipedia bits-per-byte, and works best when positional encoding is removed from its middle layers.

  4. ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    ShapeLLM-Omni unifies text, image, and 3D generation and understanding in one autoregressive LLM using discrete 3D tokens and a new 3D-Alpaca training dataset.

Pith tools