Pith. sign in

REVIEW 3 cited by

Leveraging redundancy in attention with Reuse Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.06821 v1 pith:YP46SLVE submitted 2021-10-13 cs.LG cs.CLcs.CV

Srinadh Bhojanapalli , Ayan Chakrabarti , Andreas Veit , Michal Lukasik , Himanshu Jain , Frederick Liu , Yin-Wen Chang , Sanjiv Kumar This is my paper
classification cs.LGcs.CLcs.CV
keywords attentionlayersmultiplescorestransformersacrossheadspairwise
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision. However, a typical Transformer model computes such pairwise attention scores repeatedly for the same sequence, in multiple heads in multiple layers. We systematically analyze the empirical similarity of these scores across heads and layers and find them to be considerably redundant, especially adjacent layers showing high similarity. Motivated by these findings, we propose a novel architecture that reuses attention scores computed in one layer in multiple subsequent layers. Experiments on a number of standard benchmarks show that reusing attention delivers performance equivalent to or better than standard transformers, while reducing both compute and memory usage.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    CachePrune enables fine-grained, token-level KV cache reuse across LLM requests by masking sensitive segments, eliminating direct side-channel leakage while cutting TTFT by 4.5x and raising hit rates by 44% versus pri...

  2. ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection

    cs.LG 2026-05 conditional novelty 7.0 of 10

    ASAP amortizes Sinkhorn-based doubly-stochastic attention by learning a parametric map from 1D potentials to the Sinkhorn dual and reconstructing the plan via two-sided entropic c-transform, delivering 5.3x faster inf...

  3. From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Sparsity-guided distillation enables replacing attention layers in ViTs with simpler sequential modules, with sparser layers showing smaller performance drops.

Pith tools