Pith. sign in

REVIEW 2 major objections 2 minor 60 cited by

The Reformer matches standard Transformer performance on long sequences while using much less memory and running faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-13 07:14 UTC pith:MZ4GFAY7

load-bearing objection Reformer shows LSH attention plus reversible residuals can match standard Transformer performance on long sequences while cutting time and memory costs, and the experiments largely support the claims. the 2 major comments →

arxiv 2001.04451 v2 pith:MZ4GFAY7 submitted 2020-01-13 cs.LG cs.CLstat.ML

Reformer: The Efficient Transformer

Nikita Kitaev , {\L}ukasz Kaiser , Anselm Levskaya This is my paper
classification cs.LG cs.CLstat.ML
keywords reformertransformerlocality-sensitive hashingreversible residual layersefficient attentionlong sequencesmemory efficiencysequence modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large Transformer models become prohibitively expensive to train on long sequences because attention costs grow quadratically with length and activations must be stored once per layer. The paper replaces dot-product attention with a locality-sensitive hashing version whose cost scales as O(L log L) instead of O(L squared). It also switches to reversible residual layers so that activations are stored only once during training rather than once for each of the N layers. The resulting Reformer achieves comparable accuracy to a standard Transformer yet requires far less memory and runs faster when sequences are long. A reader would care because this change makes it practical to apply the same powerful sequence modeling approach to inputs that were previously too large to handle.

Core claim

The Reformer achieves performance on par with Transformer models by using locality-sensitive hashing for attention, changing its complexity from O(L squared) to O(L log L), and reversible residual layers that allow storing activations only once instead of N times where N is the number of layers, resulting in a model that is much more memory-efficient and faster on long sequences.

What carries the argument

Locality-sensitive hashing attention combined with reversible residual layers, which approximate exact attention at lower cost and eliminate the need to store separate activations for every layer.

Load-bearing premise

The locality-sensitive hashing approximation to attention preserves enough accuracy that downstream task performance remains comparable to exact attention.

What would settle it

A side-by-side experiment on a long-sequence task in which the Reformer achieves noticeably lower accuracy than a standard Transformer of the same size and depth would show that the hashing approximation does not preserve sufficient accuracy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The model can handle sequences much longer than those feasible with standard Transformers without exceeding available memory.
  • Training time on long sequences decreases because attention computation drops from quadratic to linearithmic.
  • Performance on typical sequence tasks stays comparable to that of exact-attention Transformers.
  • Memory use for activations stays constant regardless of how many layers are stacked.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hashing idea could be applied to other quadratic operations inside Transformers to gain further speedups.
  • Tasks that demand very fine-grained attention might require more hash buckets or rounds to keep quality close to the exact version.
  • The reversible-layer trick could be paired with other memory-saving methods such as checkpointing to push sequence lengths even higher.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces the Reformer, a Transformer variant that replaces standard dot-product attention with locality-sensitive hashing (LSH) attention to reduce complexity from O(L²) to O(L log L) and uses reversible residual layers to store activations only once during training instead of N times. The central claim is that the resulting model achieves performance on par with standard Transformers while being substantially more memory-efficient and faster on long sequences.

Significance. If the empirical parity holds, the work is significant for enabling Transformer-scale models on longer sequences in domains such as long-document NLP, music, and genomics. The reversible-residual technique is exact and provides a clean memory reduction; the LSH approximation supplies the asymptotic speedup. Together they form a practical, reproducible recipe that has already influenced subsequent efficient-attention designs.

major comments (2)
  1. [Section 3 (LSH Attention)] LSH attention mechanism (Section 3): the headline claim of performance parity rests on the assertion that LSH with the reported bucket counts and hash rounds produces attention outputs sufficiently close to exact dot-product attention. No quantitative bound on approximation error, no analysis of collision probability under sparse or long-range attention patterns, and no ablation varying the number of rounds or buckets are provided; this leaves the weakest assumption untested and load-bearing for the central result.
  2. [Section 4 (Experiments)] Experimental evaluation (Section 4 and Tables 1-3): while parity is asserted on several benchmarks, the manuscript does not report the exact LSH hyperparameters used per task or compare against a strong exact-attention baseline with identical optimization settings. Without these controls it is impossible to isolate whether observed differences (or lack thereof) are due to the approximation or to other implementation choices.
minor comments (2)
  1. [Section 3] Notation for the number of hash rounds and buckets is introduced without a compact summary table; adding one would improve readability when readers compare configurations across experiments.
  2. [Figure 2] Figure captions for the complexity and memory plots could explicitly state the sequence lengths at which the O(L log L) and single-activation regimes become advantageous.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and positive assessment of the work's significance. We address each major comment below and commit to revisions that strengthen the empirical validation while noting the inherent challenges in providing tight theoretical bounds.

read point-by-point responses

  1. Referee: [Section 3 (LSH Attention)] LSH attention mechanism (Section 3): the headline claim of performance parity rests on the assertion that LSH with the reported bucket counts and hash rounds produces attention outputs sufficiently close to exact dot-product attention. No quantitative bound on approximation error, no analysis of collision probability under sparse or long-range attention patterns, and no ablation varying the number of rounds or buckets are provided; this leaves the weakest assumption untested and load-bearing for the central result.

    Authors: We acknowledge that the manuscript does not include a quantitative theoretical bound on the LSH approximation error or a dedicated analysis of collision probabilities for sparse/long-range patterns. Deriving tight, data-independent bounds is difficult because the error depends on the distribution of attention scores. The central results instead rest on consistent empirical parity across tasks. We will add an ablation study varying the number of hash rounds and buckets (and report the resulting performance and memory trade-offs) to the revised manuscript. We will also include a short discussion referencing standard LSH collision-probability results from the literature to clarify the probabilistic nature of the approximation. revision: partial

  2. Referee: [Section 4 (Experiments)] Experimental evaluation (Section 4 and Tables 1-3): while parity is asserted on several benchmarks, the manuscript does not report the exact LSH hyperparameters used per task or compare against a strong exact-attention baseline with identical optimization settings. Without these controls it is impossible to isolate whether observed differences (or lack thereof) are due to the approximation or to other implementation choices.

    Authors: We will add a supplementary table listing the precise LSH hyperparameters (buckets, hash rounds, chunk size, etc.) used for every reported experiment. Regarding baselines, standard Transformer comparisons were performed with matching optimizer, learning-rate schedule, and initialization wherever memory allowed. For sequences longer than a few thousand tokens, exact attention becomes infeasible on the same hardware, which is the core motivation of the work. We will add controlled side-by-side runs on shorter sequences (lengths where exact attention fits) using identical optimization settings to isolate the effect of the LSH approximation. revision: yes

Circularity Check

0 steps flagged

No circularity: Reformer introduces independent algorithmic mechanisms

full rationale

The paper's core contributions are two explicit algorithmic replacements: LSH-based attention (reducing complexity from O(L²) to O(L log L)) and reversible residual layers (storing activations once instead of N times). Neither is derived from fitted parameters, self-referential equations, or load-bearing self-citations; both are presented as new constructions whose correctness is evaluated via standard benchmark comparisons. The LSH approximation and reversibility are defined directly in the text without reducing to prior outputs of the same paper. No self-definitional loops, renamed empirical patterns, or uniqueness theorems imported from the authors' own prior work appear in the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper relies on the standard mathematical properties of locality-sensitive hashing for approximate nearest-neighbor search and on the invertibility property of reversible residual blocks; no new free parameters or invented entities are introduced in the abstract.

axioms (2)
  • domain assumption Locality-sensitive hashing can be used to approximate dot-product attention with high probability
    Invoked to justify replacing exact attention with the LSH variant
  • standard math Reversible residual layers allow exact reconstruction of activations from later layers
    Used to claim memory savings without loss of information

pith-pipeline@v0.9.0 · 5425 in / 1208 out tokens · 38996 ms · 2026-05-13T07:14:15.536184+00:00 · methodology

0 comments
read the original abstract

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

    cs.LG 2026-07 conditional novelty 7.0

    Positional schemes set the default spectral algebra of attention heads: previous-token heads are rotational under RoPE and content-like under absolute/ALiBi, as a post-function fingerprint rather than a hard constraint.

  2. LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    LongSpike integrates fractional-order state-space modeling into spiking neural networks, enabling better long-sequence performance than prior SNNs on LRA, WikiText-103, and Speech Commands benchmarks while retaining s...

  3. TENNOR: Trustworthy Execution for Neural Networks through Obliviousness and Retrievals

    cs.CR 2026-05 unverdicted novelty 7.0

    TENNOR enables efficient private training of wide neural networks in TEEs by recasting sparsification as doubly oblivious LSH retrievals and introducing MP-WTA to cut hash table memory by 50x while preserving accuracy.

  4. Stream-CQSA: Avoiding Out-of-Memory in Attention Computation via Flexible Workload Scheduling

    cs.LG 2026-04 unverdicted novelty 7.0

    Stream-CQSA uses CQS-based decomposition to stream exact attention computations for billion-token sequences on limited-memory hardware.

  5. Cross-Stage Attention Propagation for Efficient Semantic Segmentation

    cs.CV 2026-04 unverdicted novelty 7.0

    CSAP computes attention at the deepest scale and propagates the maps to shallower stages, bypassing per-scale query-key computations to cut decoder FLOPs while preserving multi-scale performance and beating SegNeXt-Ti...

  6. Fast Cross-Operator Optimization of Attention Dataflow

    cs.AR 2026-04 unverdicted novelty 7.0

    MMEE encodes dataflow decisions in matrix form for fast exhaustive search, delivering 40-69% lower latency and energy use than prior methods while running 64-343x faster.

  7. FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

    cs.CV 2026-03 unverdicted novelty 7.0

    FrameVGGT replaces token-level KV retention with frame-level segments and prototypes to bound memory while preserving geometric coherence in streaming VGGT.

  8. Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

    cs.LG 2025-12 unverdicted novelty 7.0

    Exact Flow Linear Attention derives a closed-form exact update for delta-rule linear attention from continuous-time dynamics, removing Euler discretization error while preserving linear complexity and structure.

  9. SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators

    cs.AI 2025-11 unverdicted novelty 7.0

    SnapStream deploys sparse KV attention in a production inference system on dataflow accelerators, delivering 4x on-chip memory savings for DeepSeek-671B at 128k context with up to 1832 tokens/sec and minimal accuracy ...

  10. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

    cs.CL 2024-10 unverdicted novelty 7.0

    LongMemEval benchmarks long-term memory in chat assistants, revealing 30% accuracy drops across sustained interactions and proposing indexing-retrieval-reading optimizations that boost performance.

  11. RWKV: Reinventing RNNs for the Transformer Era

    cs.CL 2023-05 unverdicted novelty 7.0

    RWKV uses a linear attention mechanism to deliver Transformer-level performance with RNN-style inference efficiency, demonstrated at up to 14 billion parameters.

  12. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    cs.LG 2021-01 accept novelty 7.0

    Switch Transformers use top-1 expert routing in a Mixture of Experts setup to scale to trillion-parameter language models with constant compute and up to 4x speedup over T5-XXL.

  13. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  14. Hierarchical Domain Generalization

    cs.LG 2026-07 conditional novelty 6.0

    Over infinite domains, hierarchy-uniform domain generalization is impossible for every nontrivial hypothesis class; a length-generalization bound is a property of the length hierarchy, not a hierarchy-free guarantee.

  15. Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

    cs.LG 2026-07 conditional novelty 6.0

    A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.

  16. ELiTeFormer: An Efficient Transformer for FPGAs

    cs.AR 2026-07 conditional novelty 6.0

    Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.

  17. TopoCast: A Topological Fidelity Framework for Evaluating Transformer-Based Time Series Forecasting

    cs.LG 2026-06 unverdicted novelty 6.0

    A topological evaluation framework for transformer time series forecasts that derives fidelity scores from persistence diagrams of delay embeddings, including a phase-aware localized metric.

  18. Building Social World Models with Large Language Models

    cs.SI 2026-06 unverdicted novelty 6.0

    SWM framework uses LLMs to model social belief dynamics from events via temporal pattern mining and ELBO optimization, outperforming time-series models on a new 12k-point benchmark from Kalshi and Polymarket predictio...

  19. Chiaroscuro Attention: Spending Compute in the Dark

    cs.CL 2026-06 unverdicted novelty 6.0

    CHIAR-Former routes tokens via spectral entropy to DCT mixing or attention, yielding 35-40% FLOP savings at 400M parameters with modest perplexity increase on WikiText-103.

  20. Do Transformers Need Three Projections? Systematic Study of QKV Variants

    cs.LG 2026-06 conditional novelty 6.0

    Q-K=V projection sharing in transformers matches standard QKV performance with 50% KV cache reduction and combines with GQA/MQA for up to 96.9% reduction across vision and language tasks.

  21. Policy-based Foveated Imaging and Perception

    cs.CV 2026-06 unverdicted novelty 6.0

    A task-aware policy learned via reinforcement learning allocates high-resolution pixels on dual-stream sensors in real time, outperforming fixed or non-predictive baselines under tight pixel budgets in both simulation...

  22. Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers

    cs.CV 2026-05 unverdicted novelty 6.0

    Tensor Memory augments Transformers with a constant-size 3D voxel grid using differentiable soft writes at predicted locations, local interaction, and gated recurrent dynamics to decouple memory capacity from sequence length.

  23. Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

    cs.LG 2026-05 unverdicted novelty 6.0

    TSCOMP is the first large-scale benchmark that deconstructs deep multivariate time series forecasters into fine-grained components, builds a corpus of over 20,000 evaluations, and shows that corpus-driven component se...

  24. How Do Electrocardiogram Models Scale?

    cs.LG 2026-05 conditional novelty 6.0

    Empirical scaling study of ECG models finds SSL scales robustly while ResNets show 1.3-2.5x better parameter efficiency and SSL up to 16x better data efficiency than supervised baselines on out-of-distribution tasks.

  25. Elastic Attention Cores for Scalable Vision Transformers

    cs.CV 2026-05 unverdicted novelty 6.0

    VECA learns effective visual representations using core-periphery attention where patches interact exclusively via a resolution-invariant set of learned core embeddings, achieving linear O(N) complexity while maintain...

  26. Spectral Transformer Neural Processes

    cs.LG 2026-05 unverdicted novelty 6.0

    STNPs extend TNPs with a spectral aggregator that estimates context spectra, forms spectral mixtures, and injects task-adaptive frequency features to better handle periodicity.

  27. The Impossibility Triangle of Long-Context Modeling

    cs.CL 2026-05 unverdicted novelty 6.0

    No model can achieve efficiency, compactness, and recall capacity scaling with sequence length at once, as any two imply a strict bound of O(poly(d)/log V) on recallable facts.

  28. HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

    cs.LG 2026-04 unverdicted novelty 6.0

    HubRouter is a sub-quadratic routing primitive using learned hubs that replaces attention layers in hybrid models while delivering competitive perplexity and large throughput gains.

  29. DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

    cs.CL 2026-04 unverdicted novelty 6.0

    DASH-KV accelerates long-context LLM inference to linear complexity via asymmetric KV cache hashing and mixed-precision retention, matching full attention performance on LongBench.

  30. GAMMA-Net: Adaptive Long-Horizon Traffic Spatio-Temporal Forecasting Model based on Interleaved Graph Attention and Multi-Axis Mamba

    cs.AI 2026-04 unverdicted novelty 6.0

    GAMMA-Net combines Graph Attention Networks and multi-axis Mamba to outperform prior models in long-horizon traffic forecasting, with up to 16.25% lower MAE on benchmarks like METR-LA and PEMS datasets.

  31. Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

    cs.LG 2026-03 conditional novelty 6.0

    Naive text+time-series fusion frequently degrades forecasting, while a low-rank controlled adapter (CFA) consistently improves over unimodal baselines across 14 backbones, 4 text encoders, and 9 datasets.

  32. HiSAC: Hierarchical Sparse Activation Compression for Ultra-long Sequence Modeling in Recommenders

    cs.IR 2026-02 conditional novelty 6.0

    Hierarchical semantic tokenization with sparse interest-agent voting and soft-routing attention compresses ultra-long user histories for recommenders, reporting a 1.65% online CTR lift at Taobao.

  33. SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm

    cs.LG 2026-02 unverdicted novelty 6.0

    SiameseNorm is a two-stream architecture that reconciles Pre-Norm and Post-Norm in Transformers by coupling streams via shared residual blocks, yielding performance gains with maintained stability on language, vision,...

  34. Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

    cs.CV 2025-12 conditional novelty 6.0

    LLSA uses hierarchical top-K selection plus coarse-token enrichment to make diffusion-transformer attention O(N log N), giving about 6x faster training at 256x256 pixel resolution with FID comparable to full attention.

  35. Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression

    cs.LG 2025-11 unverdicted novelty 6.0

    Gated KalmaNet uses exact Kalman gain computation with adaptive gating and Chebyshev iteration to improve SSM performance on long-context tasks over prior approximations like DeltaNet.

  36. Kimi Linear: An Expressive, Efficient Attention Architecture

    cs.CL 2025-10 unverdicted novelty 6.0

    Kimi Linear hybridizes linear attention with a new KDA module to beat full attention on tasks while slashing KV cache by 75% and speeding decoding up to 6x.

  37. MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training

    cs.CL 2025-10 conditional novelty 6.0

    MTraining scales LLM training to 512K-token contexts on 32 A100 GPUs by integrating dynamic sparse training patterns with balanced and hierarchical sparse ring attention, achieving up to 6x throughput gains without ac...

  38. LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

    cs.CL 2025-02 unverdicted novelty 6.0

    LIFT fine-tunes short-context LLMs on long inputs with synthetic tasks to absorb information into parameters, enabling answers without the input present at inference.

  39. MoBA: Mixture of Block Attention for Long-Context LLMs

    cs.LG 2025-02 unverdicted novelty 6.0

    MoBA routes attention over blocks via MoE-style gating to enable dynamic, bias-light long-context attention that matches full attention performance at lower cost.

  40. Gated Linear Attention Transformers with Hardware-Efficient Training

    cs.LG 2023-12 unverdicted novelty 6.0

    Gated linear attention Transformers achieve competitive language modeling results with linear-time inference, superior length generalization, and higher training throughput than Mamba.

  41. MemGPT: Towards LLMs as Operating Systems

    cs.AI 2023-10 unverdicted novelty 6.0

    MemGPT uses OS-inspired virtual context management to extend LLM context windows for large document analysis and long-term multi-session chat.

  42. H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

    cs.LG 2023-06 unverdicted novelty 6.0

    H2O evicts non-heavy-hitter tokens from the KV cache using a dynamic submodular policy, retaining recent and frequent-co-occurrence tokens to reduce memory while preserving accuracy.

  43. PaLM: Scaling Language Modeling with Pathways

    cs.CL 2022-04 accept novelty 6.0

    PaLM 540B demonstrates continued scaling benefits by setting new few-shot SOTA results on hundreds of benchmarks and outperforming humans on BIG-bench.

  44. ST-MoE: Designing Stable and Transferable Sparse Expert Models

    cs.CL 2022-02 unverdicted novelty 6.0

    ST-MoE introduces stability techniques for sparse expert models, allowing a 269B-parameter model to achieve state-of-the-art transfer learning results across reasoning, summarization, and QA tasks at the compute cost ...

  45. Aligning AI With Shared Human Values

    cs.CY 2020-08 conditional novelty 6.0

    Introduces ETHICS benchmark showing current language models have promising but incomplete ability to predict basic human ethical judgments on text scenarios.

  46. HuggingFace's Transformers: State-of-the-art Natural Language Processing

    cs.CL 2019-10 accept novelty 6.0

    Hugging Face releases an open-source Python library that supplies a unified API and pretrained weights for major Transformer architectures used in natural language processing.

  47. Structured Thoughts For Improved Reasoning And Context Pruning

    cs.CL 2026-07 conditional novelty 5.5

    Structured try/outcome SFT improves math reasoning by up to 8% over standard SFT and enables pruning ~85% of context with ~9% accuracy drop.

  48. MABLE: Masked Autoencoding with Bi-Lipschitz Decoding for Embeddings and Graph Metric Learning

    cs.LG 2026-07 conditional novelty 5.5

    MABLE learns stable node and graph embeddings on heterogeneous geospatial graphs via masked autoencoding, bi-Lipschitz decoding, and fixed cosine alignment/uniformity losses without discriminators or hard negatives.

  49. KAN-LSTM-Transformer Neural Networks, MFV and Cosmological Parameters

    astro-ph.CO 2026-07 conditional novelty 5.0

    KLT-Net reconstructs the SN Ia distance modulus non-parametrically; with MFV M_B and flat-ΛCDM Bayesian/Hessian inference it yields H0 ≈ 69.6 km s⁻¹ Mpc⁻¹ and Ωm ≈ 0.30.

  50. Zeus: Towards Tuning-Free Foundation Model for Time Series Analysis

    cs.LG 2026-07 unverdicted novelty 5.0

    Zeus proposes a multi-scale Transformer with point-wise tokenization and Multi-Objective Temporal Masking to enable tuning-free performance on forecasting, interpolation, and other time series tasks.

  51. Enhancing Layer Interaction Using Key-Correlated Layer Attention

    cs.CV 2026-06 unverdicted novelty 5.0

    KCLA is a linear-complexity layer attention mechanism that exploits high key cosine similarity to preserve dynamic updates and long-range cross-layer connections.

  52. Q-Delta: Beyond Key-Value Associative State Evolution

    cs.AI 2026-06 unverdicted novelty 5.0

    Q-Delta extends linear attention by introducing a query-conditioned delta rule that incorporates mixed key-query errors into recurrent state updates for improved stability and performance.

  53. Online Irregular Multivariate Time Series Forecasting via Uncertainty-Driven Dual-Expert Calibration

    cs.LG 2026-05 unverdicted novelty 5.0

    Under-Cali is an uncertainty-driven dual-expert calibration framework for online adaptation in irregular multivariate time series forecasting that freezes the base model.

  54. Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

    cs.CL 2026-05 unverdicted novelty 5.0

    Grammatically-Guided Sparse Attention uses POS tags to generate hard or soft masks that constrain self-attention, achieving 0.8200 and 0.8165 accuracy on SST-2 versus 0.8200 for full attention in a DistilBERT-like model.

  55. Spectral Priors vs. Attention: Investigating the Utility of Attention Mechanisms in EEG-Based Diagnosis

    cs.LG 2026-05 unverdicted novelty 5.0

    Spectral features from frequency and time-frequency domains enable traditional ML to match or exceed attention-based DL performance in EEG disease diagnosis on small datasets, with attention unable to capture stable s...

  56. Kaczmarz Linear Attention

    cs.LG 2026-05 unverdicted novelty 5.0

    Kaczmarz Linear Attention replaces the empirical coefficient in Gated DeltaNet with a key-norm-normalized step size derived from the online regression objective, yielding lower perplexity and better needle-in-haystack...

  57. StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

    cs.LG 2026-05 accept novelty 5.0

    Chunked streaming top-k enables CSA indexer execution at 1M sequence length with 6.21 GB peak memory and >=0.998 recall on synthetic V4-shaped inputs.

  58. Learning Fingerprints for Medical Time Series with Redundancy-Constrained Information Maximization

    cs.LG 2026-04 unverdicted novelty 5.0

    A self-supervised method learns a fixed set of disentangled fingerprint tokens from medical time series by combining reconstruction loss with a total coding rate diversity penalty, framed as a disentangled rate-distor...

  59. Toeplitz MLP Mixers are Low Complexity, Information-Rich Sequence Models

    cs.LG 2026-04 unverdicted novelty 5.0

    Toeplitz MLP Mixers replace attention with masked Toeplitz multiplications for sub-quadratic complexity while retaining more sequence information and outperforming on copying and in-context tasks.

  60. climt-paraformer: Stable Emulation of Convective Parameterization using a Temporal Memory-aware Transformer

    physics.ao-ph 2026-04 unverdicted novelty 5.0

    A temporal memory-aware Transformer emulator for the Emanuel convective parameterization shows lower offline errors and 10-year stability in single-column model tests compared to memory-less MLP and LSTM baselines.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 75 Pith papers · 3 internal anchors

  1. [2]

    Character-Level Language Modeling with Deeper Self-Attention

    URL http: //arxiv.org/abs/1808.04444. Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya P. Razenshteyn, and Ludwig Schmidt. Practical and optimal LSH for angular distance. CoRR, abs/1509.02897,

    work page Pith review arXiv

  2. [3]

    Practical and Optimal LSH for Angular Distance

    URL http://arxiv. org/abs/1509.02897. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450,

    work page Pith review arXiv

  3. [4]

    Layer Normalization

    URL http://arxiv.org/abs/1607.06450. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [5]

    Large-scale Simple Question Answering with Memory Networks

    URL http://arxiv.org/ abs/1506.02075. Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Ben- gio. Hierarchical memory networks. arXiv preprint arXiv:1605.07427,

    work page Pith review arXiv

  5. [7]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    URL http://arxiv.org/abs/1810.04805. Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual net- work: Backpropagation without storing activations. InAdvances in neural information processing systems, pp. 2214–2224,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [9]

    The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

    URL http://arxiv.org/abs/1511.02301. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, An- drew M Dai, Matthew D Hoffman, and Douglas Eck. Music transformer: Generating music with long-term structure. arXiv preprint arXiv:1809.04281,

    work page Pith review arXiv

  7. [11]

    URL http: //arxiv.org/abs/1907.05242. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. CoRR, abs/1801.10198,

    work page Pith review arXiv 1907

  8. [12]

    Generating Wikipedia by Summarizing Long Sequences

    URL http://arxiv.org/abs/1801.10198. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine trans- lation. In Proceedings of the Third Conference on Machine Translation: Research Papers , pp. 1–9, Brussels, Belgium, October

    work page Pith review arXiv

  9. [13]

    Scaling Neural Machine Translation

    Association for Computational Linguistics. doi: 10.18653/v1/W18-6301. URL https://www.aclweb.org/anthology/W18-6301. 10 Published as a conference paper at ICLR 2020 Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. Image transformer. CoRR, abs/1802.05751,

    work page doi:10.18653/v1/w18-6301 2020

  10. [15]

    Stand-Alone Self-Attention in Vision Models

    URL http: //arxiv.org/abs/1906.05909. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. One- shot learning with memory-augmented neural networks. CoRR, abs/1605.06065,

    work page Pith review arXiv 1906

  11. [16]

    One-shot Learning with Memory-Augmented Neural Networks

    URL http://arxiv.org/abs/1605.06065. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235,

    work page Pith review arXiv

  12. [17]

    URL http://arxiv.org/abs/1804.04235. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hecht- man. Mesh-tensorflow: Deep learning for supercomputers. CoRR, abs/1811.02084,

    work page Pith review arXiv

  13. [18]

    Mesh-TensorFlow: Deep Learning for Supercomputers

    URL http://arxiv.org/abs/1811.02084. Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christo- pher R´e. Low-memory neural network training: A technical report.CoRR, abs/1904.10631,

    work page Pith review arXiv 1904

  14. [19]

    Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin

    URL http://arxiv.org/abs/1904.10631. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive atten- tion span in transformers. CoRR, abs/1905.07799, 2019a. URL http://arxiv.org/abs/ 1905.07799. Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herv´e J´egou, and Armand Joulin. Aug- menting self-attention with persistent mem...

    work page arXiv 1904

  15. [20]

    Attention Is All You Need

    URL http: //arxiv.org/abs/1706.03762. Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks.CoRR, abs/1410.3916,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [21]

    Memory Networks

    URL http://arxiv.org/abs/1410.3916. 11 Published as a conference paper at ICLR 2020 A M ULTI-ROUND LSH A TTENTION In this section we describe in more detail the multi-hash version of our LSH attention mechanism. We first repeat Equation (3) from the main text, which describes a general formulation of attention with sparsity: oi = ∑ j∈ ˜Pi exp (qi· kj− m(j,...

    work page Pith review arXiv 2020