SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series
read the original abstract
Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\url{https://github.com/badripatro/Simba}.
This paper has not been read by Pith yet.
Forward citations
Cited by 14 Pith papers
-
CHASM: Cross-frequency Harmonized Axis-Separable Mixing for Spectral Token Operators
CHASM introduces a cross-frequency harmonized axis-separable spectral mixer using a shared channel eigenbasis plus per-frequency positive gains, yielding consistent gains over same-backbone baselines in medical and na...
-
A Novel Schur-Decomposition-Based Weight Projection Method for Stable State-Space Neural-Network Architectures
A real Schur decomposition projection maps the state matrix of discrete-time state-space layers onto its nearest stable counterpart, delivering accuracy comparable to prior stable identification methods with fewer weights.
-
GeoCert: Certified Geometric AI for Reliable Forecasting
GeoCert uses hyperbolic geometry to unify forecasting with physical reasoning and built-in formal certification, claiming major gains in accuracy and efficiency.
-
NAKUL-Med: Spectral-Graph State Space Models with Dynamics Kernels for Medical Signals
NAKUL achieves 91.7% accuracy on motor imagery EEG with 28% fewer parameters than EEG-Conformer by using dynamic kernel generation, spectral context modeling, and graph-guided spatial attention.
-
HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet
HAMSA achieves 85.7% ImageNet-1K top-1 accuracy as a spectral-domain SSM with 2.2x faster inference and lower memory than transformers or scanning-based SSMs.
-
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
ABMamba uses Mamba-based linear-complexity processing plus a novel Aligned Hierarchical Bidirectional Scan to deliver competitive video captioning on VATEX and MSR-VTT at roughly 3x higher throughput than typical Tran...
-
UniMamba: A Unified Spatial-Temporal Modeling Framework with State-Space and Attention Integration
UniMamba integrates Mamba state-space dynamics with attention layers and transforms like FFT-Laplace to outperform prior models on multivariate time series forecasting benchmarks.
-
Titans: Learning to Memorize at Test Time
Titans combine attention for current context with a learnable neural memory for long-term history, achieving better performance and scaling to over 2M-token contexts on language, reasoning, genomics, and time-series tasks.
-
Graph Mamba Survival Analysis Based on Topology-Aware ordering
TopoMamSurv introduces topology-aware ordering and bidirectional Mamba with GCN for efficient WSI graph survival analysis, claiming performance gains on five TCGA datasets.
-
Deep Learning Surrogates for Emulating Stochastic Climate Tipping Dynamics
A dynamics-informed Temporal Fusion Transformer surrogate emulates stochastic tipping events in global ocean transport simulations with 465x speedup and high-fidelity timing predictions.
-
Dual Mamba for Node-Specific Representation Learning: Tackling Over-Smoothing with Selective State Space Modeling
DMbaGCN combines a local state-evolution Mamba for node-specific dynamics with a global context-aware Mamba to reduce over-smoothing in deep graph neural networks.
-
Can Visual Mamba Improve AI-Generated Image Detection? An In-Depth Investigation
Benchmarks Vision Mamba variants for AI-generated image detection against CNN, ViT, and VLM detectors on diverse datasets and synthetic sources, reporting promise alongside limitations.
-
The Hyperscale Lottery: How State-Space Models Have Sacrificed Edge Efficiency
Mamba-3 architectural changes optimized for hyperscale GPUs cause 28% higher edge latency at 880M parameters and 48% at 15M parameters compared to earlier versions.
-
Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State- Space Architectures from S4 to Mamba
A survey tracing the evolution of state-space models like S4 and Mamba, their efficiency trade-offs, and applications in NLP, vision, and other domains.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.