Pith. sign in

REVIEW 34 cited by

Long Range Arena: A Benchmark for Efficient Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2011.04006 v1 pith:QVOEIEVV submitted 2020-11-08 cs.LG cs.AIcs.CLcs.CVcs.IR

Yi Tay , Mostafa Dehghani , Samira Abnar , Yikang Shen , Dara Bahri , Philip Pham , Jinfeng Rao , Liu Yang
show 2 more authors
Sebastian Ruder Donald Metzler
This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CVcs.IR
keywords transformersbenchmarkmodelsefficientmodelqualitytaskstransformer
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 34 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Effect of Depth on the Expressivity of Deep Linear State-Space Models

    cs.LG 2025-06 conditional novelty 7.0 of 10

    Deeper linear state-space models can match the expressive power of shallower wider ones, and under norm constraints they do so with exponentially smaller parameter norms.

  2. BLAB: Brutally Long Audio Bench

    cs.AI 2025-05 conditional novelty 7.0 of 10

    BLAB is the first benchmark to evaluate audio language models on hour-scale audio, and state-of-the-art models score below 22% exact match on most of its tasks.

  3. DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Delay State Space Models augment diagonal SSMs with explicit delayed feedback, stable discrete parameterization, and FFT training, improving delayed-retrieval tasks and matching or beating S4D on most standard sequenc...

  4. Foveated Probes Recover Localized Binding Information in Vision Foundation Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A foveated attention readout over frozen patch tokens recovers most of the localized binding signal that global pooling discards, so apparent spatial blindness can be readout-level rather than representational.

  5. Pretraining Recurrent Networks without Recurrence

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.

  6. Small transformer architectures for task switching

    cs.LG 2025-08 reject novelty 6.0 of 10

    On a new task-switching benchmark, a cisformer with expressive attention reaches about 95% accuracy, while standard transformers, LSTMs, and MLPs all stay near or below 60%.

  7. ContextCache: Context-Aware Semantic Cache for Multi-Turn Queries in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ContextCache adds conversation context to LLM semantic caching via self-attention over query history, improving cache precision and recall over GPTCache by roughly 11-15% and serving cached responses about 10 times fa...

  8. From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.

  9. A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks

    cs.NE 2025-06 conditional novelty 6.0 of 10

    HYPR parallelizes the online learning rule e-prop over sequence segments using associative scans, achieving constant memory, large speedups, and near-BPTT accuracy on several tasks with oscillatory spiking neurons.

  10. LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Language agents' believability and goal achievement decline over multi-episode social interactions, and curated memory summaries only partially close the gap with humans.

  11. Neuromorphic Sequential Arena: A Benchmark for Neuromorphic Temporal Processing

    cs.NE 2025-05 conditional novelty 6.0 of 10

    The authors introduce NSA, a seven-task benchmark with an STP validity probe, and benchmark spiking neuron models and architectures on accuracy and efficiency.

  12. Irrational Complex Rotations Empower Low-bit Optimizers

    cs.LG 2025-01 reject novelty 6.0 of 10

    π-Quant's core representation theorem fails: the curve e^{iθ}+e^{iπθ} is dense in the disk but does not cover it, and Lemma 3.2's angle formulas are internally inconsistent.

  13. Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing

    cs.LG 2024-12 conditional novelty 6.0 of 10

    State space models have an inherent recency bias and over-smoothing in deep stacks, and a two-channel polarization fix improves long-range associative recall.

  14. DEHYDRATOR: Enhancing Provenance Graph Storage via Hierarchical Encoding and Sequence Generation

    cs.CR 2024-12 conditional novelty 6.0 of 10

    DEHYDRATOR reduces provenance graph edge storage by 84.55% across seven datasets using field mapping, hierarchical encoding, and a transformer with an error-correction table.

  15. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  16. Reverse Browser: Vector-Image-to-Code Generator

    cs.SE 2025-09 conditional novelty 5.0 of 10

    An open-weights system that turns vector images of web designs into HTML/CSS, with new datasets and a multi-scale pixel metric, though accuracy remains below production quality.

  17. MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses

    eess.AS 2025-07 reject novelty 5.0 of 10

    MMW combines a Mamba-based Mix Block, a Frame Diarization Mamba layer, and multi-scale GRPO to reduce side-talk interference in Whisper ASR, reporting WER as low as 3.71%.

  18. Leaner Transformers: More Heads, Less Depth

    cs.LG 2025-05 conditional novelty 5.0 of 10

    The authors claim that more attention heads improve transformer conditioning enough to replace depth with width, yielding 30-50% parameter reductions at matched accuracy.

  19. Revisiting Glorot Initialization for Long-Range Linear Recurrences

    cs.LG 2025-05 reject novelty 5.0 of 10

    Glorot initialization is unstable for long-range linear RNNs because the spectral radius exceeds 1, with a proposed rescaling that helps in practice, but the theoretical claim that explosion starts at t = O(√n) is not...

  20. Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

    cs.LG 2025-05 conditional novelty 5.0 of 10

    AGF replaces softmax attention with a learned singular-value-domain graph filter that runs in O(n d^2), and reports moderate accuracy improvements on UEA and LRA benchmarks.

  21. Contextual Feedback Loops: Amplifying Deep Reasoning with Iterative Top-Down Feedback

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Adding a lightweight iterative top-down feedback loop to ViTs and Transformers is claimed to improve accuracy by up to 1.3 percentage points and reduce perplexity by about 6 percent, with little added overhead.

  22. Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A dual cross-entropy and KL-divergence loss lets recurrent networks maintain stable accuracy over very long streams without hidden-state resets, matching and sometimes slightly beating periodic reset baselines.

  23. MASV: Speaker Verification with Global and Local Context Mamba

    eess.AS 2024-12 conditional novelty 5.0 of 10

    MASV is an ECAPA-TDNN variant whose Res2 blocks are replaced by local-context and tri-directional Mamba blocks, improving speaker verification EER and lowering FLOPS on a private dataset.

  24. Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

    cs.AI 2026-07 conditional novelty 4.0 of 10

    Decoupling the forget gate from the input gate in a discrete state-space recurrence lets one fixed-size state both preserve old bindings over long horizons and overwrite stale ones.

  25. ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

    cs.LG 2026-07 conditional novelty 4.0 of 10

    ELSAA fuses sparse exact-block attention with a low-rank bucket-sketch branch using a denominator-aware scalar, achieving linear-time attention with competitive long-context accuracy.

  26. Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling

    cs.CL 2025-08 conditional novelty 4.0 of 10

    GAM combines a causal convolution and a fixed memory-bank retrieval with a learnable gate to match or beat Transformer and Mamba at small scale.

  27. SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models

    cs.CL 2025-07 reject novelty 4.0 of 10

    SCOPE estimates a model's position bias with nonsense prompts, puts correct answers in disliked slots, and spreads similar distractors apart to cap lucky guessing.

  28. QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Quantization-aware training allows S4D sequence models to run at much lower precision, cutting estimated hardware costs by up to two orders of magnitude while keeping accuracy.

  29. Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals

    cs.LG 2025-06 reject novelty 4.0 of 10

    A Hopfield energy with Gaussian time weights and a strong pull toward the next frame replays stored movie frames in order, but the frame to retrieve is already in the energy function.

  30. Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis

    cs.IR 2025-05 conditional novelty 4.0 of 10

    Fixed-size chunking trade-offs depend on answer locality and embedding model: 64-128 token chunks suit fact-based QA, 512-1024 token chunks suit long-context narrative QA.

  31. MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models

    cs.LG 2025-04 conditional novelty 4.0 of 10

    MOM reduces prefill memory by chunking MLP layers into mini-sequences and offloading KV caches, stretching Llama-3.2-8B to 455k tokens on a single 80GB A100 with identical outputs.

  32. Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts

    cs.LG 2025-09 reject novelty 3.0 of 10

    DASG-MoE combines grouped sliding-window attention, a two-tier expert structure, and adaptive routing, and claims state-of-the-art results, but the experimental support is internally inconsistent.

  33. W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling

    cs.LG 2025-06 conditional novelty 3.0 of 10

    W4S4 initializes S4 state space models with WaLRUS wavelet frames and reports better delay reconstruction and classification accuracy than HiPPO-based S4, with frozen (A,B).

  34. A Survey on Large Language Models with some Insights on their Capabilities and Limitations

    cs.CL 2025-01 unverdicted novelty 3.0 of 10

    A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.

Pith tools