REVIEW 21 cited by
Long Range Arena: A Benchmark for Efficient Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.
Forward citations
Cited by 21 Pith papers
-
DSSMs: State Space Models with Explicit Memory via Delay Differential Equations
Delay State Space Models augment diagonal SSMs with explicit delayed feedback, stable discrete parameterization, and FFT training, improving delayed-retrieval tasks and matching or beating S4D on most standard sequenc...
-
Pretraining Recurrent Networks without Recurrence
SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.
-
Small transformer architectures for task switching
On a new task-switching benchmark, a cisformer with expressive attention reaches about 95% accuracy, while standard transformers, LSTMs, and MLPs all stay near or below 60%.
-
ContextCache: Context-Aware Semantic Cache for Multi-Turn Queries in Large Language Models
ContextCache adds conversation context to LLM semantic caching via self-attention over query history, improving cache precision and recall over GPTCache by roughly 11-15% and serving cached responses about 10 times fa...
-
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
ProxyReward trains long-form generation models by rewarding how well an AI judge can answer generated yes/no questions about the response, improving open-source models on ProxyQA.
-
A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks
HYPR parallelizes the online learning rule e-prop over sequence segments using associative scans, achieving constant memory, large speedups, and near-BPTT accuracy on several tasks with oscillatory spiking neurons.
-
LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions
Language agents' believability and goal achievement decline over multi-episode social interactions, and curated memory summaries only partially close the gap with humans.
-
Neuromorphic Sequential Arena: A Benchmark for Neuromorphic Temporal Processing
The authors introduce NSA, a seven-task benchmark with an STP validity probe, and benchmark spiking neuron models and architectures on accuracy and efficiency.
-
X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment
X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.
-
Reverse Browser: Vector-Image-to-Code Generator
An open-weights system that turns vector images of web designs into HTML/CSS, with new datasets and a multi-scale pixel metric, though accuracy remains below production quality.
-
MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses
MMW combines a Mamba-based Mix Block, a Frame Diarization Mamba layer, and multi-scale GRPO to reduce side-talk interference in Whisper ASR, reporting WER as low as 3.71%.
-
Leaner Transformers: More Heads, Less Depth
The authors claim that more attention heads improve transformer conditioning enough to replace depth with width, yielding 30-50% parameter reductions at matched accuracy.
-
Revisiting Glorot Initialization for Long-Range Linear Recurrences
Glorot initialization is unstable for long-range linear RNNs because the spectral radius exceeds 1, with a proposed rescaling that helps in practice, but the theoretical claim that explosion starts at t = O(√n) is not...
-
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
Decoupling the forget gate from the input gate in a discrete state-space recurrence lets one fixed-size state both preserve old bindings over long horizons and overwrite stale ones.
-
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
ELSAA fuses sparse exact-block attention with a low-rank bucket-sketch branch using a denominator-aware scalar, achieving linear-time attention with competitive long-context accuracy.
-
Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling
GAM combines a causal convolution and a fixed memory-bank retrieval with a learnable gate to match or beat Transformer and Mamba at small scale.
-
SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models
SCOPE estimates a model's position bias with nonsense prompts, puts correct answers in disliked slots, and spreads similar distractors apart to cap lucky guessing.
-
QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models
Quantization-aware training allows S4D sequence models to run at much lower precision, cutting estimated hardware costs by up to two orders of magnitude while keeping accuracy.
-
Long-Sequence Memory with Temporal Kernels and Dense Hopfield Functionals
A Hopfield energy with Gaussian time weights and a strong pull toward the next frame replays stored movie frames in order, but the frame to retrieve is already in the energy function.
-
Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis
Fixed-size chunking trade-offs depend on answer locality and embedding model: 64-128 token chunks suit fact-based QA, 512-1024 token chunks suit long-context narrative QA.
-
W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
W4S4 initializes S4 state space models with WaLRUS wavelet frames and reports better delay reconstruction and classification accuracy than HiPPO-based S4, with frozen (A,B).
Discussion (0). Sign in to comment.