2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-05-21 11:45 UTCgrok-4.3pith:SXBPWOYQrecord.jsonopen to challenge →
The pith
A simplified Mamba-2 variant with A-mask and hidden-state tweaks nearly matches softmax attention accuracy while staying linear in complexity.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By simplifying Mamba-2 to its core components and then improving the A-mask together with an increase in hidden-state order, the resulting 2Mamba model reaches accuracy nearly equal to softmax attention while remaining linear in complexity and therefore far more memory-efficient at long context lengths.
What carries the argument
The A-mask and the order of the hidden state, whose specific modifications turn the simplified Mamba-2S into the higher-accuracy 2Mamba.
If this is right
- 2Mamba can handle much longer sequences than softmax attention at comparable memory cost.
- Accuracy remains competitive on tasks where standard transformers are typically evaluated.
- Certain untouched components of the original Mamba-2 can push accuracy above the softmax baseline.
- The simplification step isolates which design choices most affect final performance.
Where Pith is reading between the lines
- The same mask and state-order adjustments might transfer to other linear-attention families beyond Mamba.
- A hybrid system could switch between 2Mamba for long prefixes and softmax for short, high-precision segments.
- Direct measurement of memory usage versus sequence length on hardware would quantify the practical efficiency gain.
Load-bearing premise
The observed accuracy gains are produced by the A-mask and hidden-state-order changes themselves rather than by other unstated implementation choices or hyperparameter adjustments made during the simplification process.
What would settle it
Re-run the 2Mamba experiments using the original A-mask and the lower hidden-state order from Mamba-2S and check whether accuracy falls back to the level of the simplified model without those two changes.
read the original abstract
Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes the development of 2Mamba, obtained by first simplifying Mamba-2 to a core variant called Mamba-2S through evaluation of component importance, followed by an A-mask modification and an increase in hidden-state order. The central empirical claim is that 2Mamba achieves accuracy nearly matching softmax attention while remaining linear in complexity and more memory-efficient for long contexts; code for all experiments is released.
Significance. If the accuracy comparisons hold under controlled conditions, the work could advance practical linear-attention alternatives for long-sequence modeling. The explicit component-wise simplification process and code release are positive elements that support reproducibility and allow the community to verify the reported gains.
major comments (1)
- [Experiments] Experiments section: the transition from Mamba-2S to 2Mamba attributes accuracy gains specifically to the A-mask and increased hidden-state order, yet the reported ablations do not isolate these two changes from other implementation or hyperparameter adjustments that may have occurred during the simplification phase. Because the headline result is an empirical accuracy comparison rather than a parameter-free derivation, this attribution is load-bearing and requires clearer incremental tables or controlled re-runs.
minor comments (1)
- [Abstract] Abstract: the phrase 'nearly as accurate as softmax attention' would benefit from a parenthetical reference to the specific datasets and metric values that support the claim.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the experiments section. We agree that clearer isolation of the A-mask and hidden-state order contributions is important given the empirical nature of the headline claims, and we will strengthen this in the revision.
read point-by-point responses
-
Referee: [Experiments] Experiments section: the transition from Mamba-2S to 2Mamba attributes accuracy gains specifically to the A-mask and increased hidden-state order, yet the reported ablations do not isolate these two changes from other implementation or hyperparameter adjustments that may have occurred during the simplification phase. Because the headline result is an empirical accuracy comparison rather than a parameter-free derivation, this attribution is load-bearing and requires clearer incremental tables or controlled re-runs.
Authors: We acknowledge that the current presentation of ablations could be strengthened to more explicitly isolate the A-mask modification and hidden-state order increase. The simplification to Mamba-2S was performed first via component-wise evaluation with fixed hyperparameters, after which only the A-mask and order changes were introduced on the resulting Mamba-2S configuration. To directly address the concern, we will add new incremental ablation tables to the revised manuscript. These tables will apply the A-mask change alone, the order increase alone, and both changes together, all starting from the identical Mamba-2S baseline with no other implementation or hyperparameter adjustments. This will make the attribution of gains fully transparent and controlled. revision: yes
Circularity Check
No circularity: empirical architecture search with independent evaluation steps
full rationale
The paper describes an empirical process of simplifying Mamba-2 by component evaluation to obtain Mamba-2S, followed by targeted modifications (A-mask and hidden-state order) to produce 2Mamba. No derivation chain, first-principles equations, or predictions are presented that reduce by construction to fitted inputs, self-citations, or ansatzes. The central claims rest on accuracy comparisons and memory measurements against external softmax baselines, with code release stated, rendering the work self-contained and falsifiable outside any internal fit.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.