Pith Number

pith:TJZ4X5Z3

pith:2020:TJZ4X5Z3IOQ6GU5FOPQ3XEQ5RN

not attested not anchored not stored refs resolved

Linformer: Self-Attention with Linear Complexity

Belinda Z. Li, Han Fang, Hao Ma, Madian Khabsa, Sinong Wang

Self-attention in transformers can be approximated by a low-rank matrix to reduce complexity to linear in sequence length.

arxiv:2006.04768 v3 · 2020-06-08 · cs.LG · stat.ML

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{TJZ4X5Z3IOQ6GU5FOPQ3XEQ5RN}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We demonstrate that the self-attention mechanism can be approximated by a low-rank matrix... The resulting linear transformer, the Linformer, performs on par with standard Transformer models, while being much more memory- and time-efficient.

C2weakest assumption

That the low-rank projection matrices learned or fixed during training preserve enough information for downstream tasks across the full range of sequence lengths and domains the model will encounter.

C3one line summary

Linformer approximates self-attention with a low-rank projection to achieve O(n) time and space complexity while matching Transformer accuracy on standard NLP tasks.

References

19 extracted · 19 resolved · 11 Pith anchors

[1] Longformer: The Long-Document Transformer 2004 · arXiv:2004.05150

[2] Language Models are Few-Shot Learners 2005 · arXiv:2005.14165

[3] Training Deep Nets with Sublinear Memory Cost · arXiv:1604.06174

[4] Generating Long Sequences with Sparse Transformers 1904 · arXiv:1904.10509

[5] Bert: Pre-training of deep bidirectional transformers for language understanding 2019

Formal links

3 machine-checked theorem links

Cited by

115 papers in Pith

DnA: Denoising Attention for Visual Tasks

Structure Before Collapse: Transient semantic geometry in next-token prediction

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

Navigating the Safety-Fidelity Trade-off: Massive-Variate Time Series Forecasting for Power Systems via Probabilistic Scenarios

Receipt and verification

First computed	2026-07-05T01:09:59.490874Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

9a73cbf73b43a1e353a573e1bb921d8b536f35c1e1325dc4d516c765a5766f55

Aliases

arxiv: 2006.04768 · arxiv_version: 2006.04768v3 · doi: 10.48550/arxiv.2006.04768 · pith_short_12: TJZ4X5Z3IOQ6 · pith_short_16: TJZ4X5Z3IOQ6GU5F · pith_short_8: TJZ4X5Z3

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/TJZ4X5Z3IOQ6GU5FOPQ3XEQ5RN \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 9a73cbf73b43a1e353a573e1bb921d8b536f35c1e1325dc4d516c765a5766f55

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "0556f2b940c7c5bbbba7a010eb8e5581eed12c0783236e58aa8dc6b3c4621650",
    "cross_cats_sorted": [
      "stat.ML"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2020-06-08T17:37:52Z",
    "title_canon_sha256": "b5efd79fc27856c2c0e08c6b6be2d77bca3e4535ab2b0f02c37293802bc2e9ef"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2006.04768",
    "kind": "arxiv",
    "version": 3
  }
}