REVIEW 13 cited by
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In recent years, the application of multimodal large language models (MLLM) in various fields has achieved remarkable success. However, as the foundation model for many downstream tasks, current MLLMs are composed of the well-known Transformer network, which has a less efficient quadratic computation complexity. To improve the efficiency of such basic models, we propose Cobra, a linear computational complexity MLLM. Specifically, Cobra integrates the efficient Mamba language model into the visual modality. Moreover, we explore and study various modal fusion schemes to create an effective multi-modal Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely competitive performance with current computationally efficient state-of-the-art methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster speed due to Cobra's linear sequential modeling. (2) Interestingly, the results of closed-set challenging prediction benchmarks show that Cobra performs well in overcoming visual illusions and spatial relationship judgments. (3) Notably, Cobra even achieves comparable performance to LLaVA with about 43% of the number of parameters. We will make all codes of Cobra open-source and hope that the proposed method can facilitate future research on complexity problems in MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.
Forward citations
Cited by 13 Pith papers
-
Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution
Auras, a perception-generation disaggregation framework with a public context buffer and asynchronous pipeline executor, raises embodied-agent throughput by 2.54x on average without losing accuracy (102.7%).
-
scMamba: A Scalable Foundation Model for Single-Cell Multi-Omics Integration Beyond Highly Variable Feature Selection
A Mamba-based model integrates paired single-cell RNA and ATAC data using all genomic features, with patch tokenization and contrastive learning, and reports better integration and downstream performance than seven ex...
-
Diffusion Transformer-to-Mamba Distillation for High-Resolution Image Generation
The authors distill a diffusion transformer into a mostly-Mamba hybrid model, reaching teacher-level GenEval scores while generating up to 4K images with linear-complexity speed.
-
MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods
A quantization framework using KLT-enhanced and smooth-fused rotations lets Mamba models run at 8-bit precision with near-full accuracy and at 4-bit weights with moderate loss.
-
MBQ: Modality-Balanced Quantization for Large Vision-Language Models
A modality-weighted quantization method improves accuracy of 3-bit and 4-bit vision-language models by protecting sensitive language tokens during calibration.
-
Efficient Self-Supervised Video Hashing with Selective State Spaces
S5VH uses bidirectional Mamba layers and a hash-center alignment loss to improve self-supervised video hashing accuracy and efficiency.
-
CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction
A coarse-to-fine autoregressive policy with multi-scale action tokenization matches or beats diffusion policies on robot manipulation benchmarks at roughly 10x lower inference cost.
-
Finding Change in Satellite Archives from Text: How to Combine Before-and-After Images Efficiently
A training-free subtraction-then-attention cascade matches or beats full fusion recall on LEVIR-CC at 10–15× lower query cost; Mamba is no faster than attention at L=196; TBF cuts parameters 2.3× for a 0.007 BLEU-1 cost.
-
HAMF: A Hybrid Attention-Mamba Framework for Joint Scene Context Understanding and Future Motion Representation Learning
HAMF feeds learnable future motion tokens into the scene encoder alongside road and agent tokens, then uses a Mamba decoder to output six diverse trajectories, achieving competitive Argoverse 2 results with 3.0M parameters.
-
PerPO: Perceptual Preference Optimization via Discriminative Rewarding
PerPO trains multimodal LLMs by ranking their candidate answers with deterministic visual rewards (IoU, edit distance) and using the reward differences as margins in listwise preference optimization.
-
Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
Modality-specific projection weights let a Mamba model match dense multimodal baselines at the same loss using 25% to 65% of the training compute.
-
QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning
Compressing 10-step action chunks into discrete latent codes lets an 8B multimodal model drive a quadruped at controller frequency and raises average task success by about 65%.
-
AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment
AlignMamba fuses audio, video, and language by matching tokens to a language anchor and enforcing distribution similarity, reporting small accuracy gains with large efficiency gains on MOSI and MOSEI.
Discussion (0). Continue with ORCID to comment.