Pith. sign in

REVIEW 2 cited by

DeciMamba: Exploring the Length Extrapolation Potential of Mamba

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.14528 v3 pith:LLFKJLEZ submitted 2024-06-20 cs.LG cs.AI

Assaf Ben-Kish , Itamar Zimerman , Shady Abu-Hussein , Nadav Cohen , Amir Globerson , Lior Wolf , Raja Giryes This is my paper · ORCID
classification cs.LGcs.AI
keywords mambadecimambalengthtrainingcapabilitiesduringextrapolatelong-range
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    SCOUT matches full-attention transformer scores on long-context language modeling and reasoning benchmarks at 400M and 1.3B scales while attending only to strided checkpoint tokens.

  2. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

    cs.LG 2025-06 conditional novelty 5.0 of 10

    On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.

Pith tools