REVIEW 5 cited by
The Hidden Attention of Mamba Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and computer vision. Selective SSMs are viewed as dual models, in which one trains in parallel on the entire sequence via an IO-aware parallel scan, and deploys in an autoregressive manner. We add a third view and show that such models can be viewed as attention-driven models. This new perspective enables us to empirically and theoretically compare the underlying mechanisms to that of the self-attention layers in transformers and allows us to peer inside the inner workings of the Mamba model with explainability methods. Our code is publicly available.
Forward citations
Cited by 5 Pith papers
-
Beyond BEV: Optimizing Point-Level Tokens for Collaborative Perception
CoPLOT replaces BEV features with semantically ordered, frequency-enhanced point-level tokens for collaborative perception, improving 3D detection while cutting overhead.
-
How Can Mamba Learn In Context with Outliers and Generalize Provably?
A simplified one-layer Mamba provably learns in-context binary classification tolerating outlier fractions approaching 1, whereas a linear Transformer can only tolerate α < 1/2.
-
Mamba Knockout for Unraveling Factual Information Flow
Fine-grained token-to-token knockout reveals that Mamba models, like Transformers, rely on subject-token information flow in late-intermediate layers, with architecture-specific variations in relation-token and first-...
-
Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability
PA-LRP extends Layer-wise Relevance Propagation to attribute relevance to positional encodings in Transformers, improving faithfulness of explanations.
-
Change of Thought: Adaptive Test-Time Computation
A transformer layer that iteratively refines its attention matrix to a fixed point is claimed to improve accuracy with no extra parameters, but the benchmark evidence is not reproducible.
Discussion (0). Sign in to comment.