REVIEW 3 major objections 7 minor 1 cited by
Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read By giving each modality its own projection weights inside the Mamba block, Mixture-of-Mamba matches multimodal loss with 35–65% of the FLOPs.
desk verdict The paper shows a real data-efficiency gain from modality-specific Mamba projections, but the headline FLOPs-to-match percentages rest on an undefined fitting procedure and should not be quoted without a rigorous methodology. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Mixture-of-Mamba block is the central object. All four projections that consume single-modality features, namely the input projection $\mathbf{W}_{\mathrm{in}}$, the intermediate projection $\mathbf{W}_{\mathbf{x}}$ producing $\delta, B, C$, the $\mathbf{W}_{\Delta t}$ projection for the discretization step $\Delta$, and the output projection $\mathbf{W}_{\mathrm{out}}$, are replaced by modality-indexed copies, while Conv1D and the state transition $A$ stay shared because they act across features or on aggregated hidden states. The routing function $\mathcal{M}(\mathbf{X}, W, b; M)$ applies $W_m$ to the tokens of modality $m$ and concatenates the results, preserving the output shape of the original linear layer. This one change, without learned gating or mixture-of-experts machinery, is what the paper credits for the earlier loss matching.
What would settle it
Re-run the 1.4B Transfusion training while recording actual forward-backward FLOPs for both models at a fixed sequence length and modality mix, including the fused input projections and low-rank delta factorization, then plot the resulting loss-matching FLOPs ratio as a function of the matching tolerance (for example loss differences of 0.01, 0.001, and 0.0001) and of extrapolation beyond the final 250k training steps.
Extended reading notes
Core claim
The paper's central claim is that replacing a Mamba block's dense projections with modality-conditional copies, writing $XW+b$ as $\mathcal{M}(X,W,b;M)$, produces consistently lower training loss and earlier matching of the dense baseline's loss across the Transfusion, Chameleon, and Chameleon-plus-speech settings. Concretely, Mixture-of-Mamba matches dense Mamba's image loss with 34.76% of the training FLOPs in Transfusion at 1.4B, with 42.50% of the FLOPs for Chameleon image loss, with 24.80% of the FLOPs for speech loss in the three-modality run, and with 65.40% of the FLOPs for Chameleon text loss. The efficiency is framed as reaching the same loss value earlier in training at comparable per-token FLOPs, not as a reduction in per-step cost.
Load-bearing premise
The headline FLOPs savings assume that Mixture-of-Mamba and dense Mamba consume the same FLOPs per token, so that a step-count ratio equals a FLOPs ratio, and that the steps-to-match procedure has a defined tolerance and fitting rule; neither is stated in the paper.
Editorial extensions
If this is right
- If the claim is correct, modality-aware sparsity is a general design principle for sequence models and can be applied inside an SSM block, not only in Transformers.
- The joint decoupling of all four projections beats every subset in the ablation, implying a synergy that future architectures should reproduce rather than decoupling components piecemeal.
- Since Conv1D and state transitions remain shared, Mixture-of-Mamba is orthogonal to MoE sparsification of MLP layers and could be stacked with it.
- The gains appear across continuous diffusion-loss image tokens, discrete VQ tokens, and a third speech modality, so the benefit is not tied to one representation or objective.
- Consistent across scales from 37M to 1.5B parameters, the pattern suggests the efficiency advantage holds during early-fusion multimodal pretraining in general.
Reading between the lines
- If per-token FLOPs are truly equal, the reported percentages imply roughly a threefold data-efficiency multiplier at constant compute; this should be tested outside loss matching, for example by fixing a compute budget and measuring final validation loss.
- The paper measures FLOPs, not wall-clock time; the real training-time gain could be larger if modality copies improve memory layout or smaller if they hurt tensor-core utilization, and that gap is left untested.
- The shared $A$ and Conv1D may be carrying cross-modal transfer, so ablating the sharing, or making $A$ modality-specific, would reveal whether the remaining shared parts are the right place for generalization.
- A natural extension is adding more modalities or learned routing; the rule-based modality mask is simple and stable, but the paper does not show how it scales beyond three modalities.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Mixture-of-Mamba (MoM), a state-space model that applies modality-specific projection matrices (input, intermediate, and output projections) in each Mamba block, while keeping convolutions and state transitions shared. It evaluates MoM against dense Mamba and Flex-Attention baselines in three multimodal pretraining settings—Transfusion (continuous image tokens with diffusion loss), Chameleon (discrete image tokens), and Chameleon+Speech—across model scales from 37M to 1.5B. The central claim is that MoM matches dense Mamba's training loss at a fraction of the training FLOPs (e.g., 34.76% for Transfusion image loss at 1.4B), while also improving final loss. An ablation study on decoupling the four projection components is included.
Significance. If the efficiency claim is correct, the paper makes a useful contribution by transferring the modality-aware sparsity design from Transformers (Mixture-of-Transformers) to SSMs, with consistent qualitative evidence across multiple scales and settings. The paper's strengths include the breadth of evaluation (three settings, five scales), the inclusion of validation loss curves on multiple datasets, and the availability of code. However, the quantitative FLOPs-savings claims are not backed by a described methodology, and the paper's own figures appear to contradict the table values in at least one case. The qualitative claim of faster convergence is well supported by the loss curves; the quantitative headline numbers are not.
major comments (3)
- [Section 3.1, Figures 3-14, Tables 1/4/7] The metric 'Relative Training FLOPs to Match Mamba' is never operationally defined. The paper does not specify the tolerance for a loss 'match', the interpolation or smoothing of loss curves, the range of steps used for the linear fit that produces the slope s, or the FLOPs accounting that converts a step ratio into a FLOPs ratio. This is load-bearing because the abstract's headline numbers (34.76%, 42.50%, 24.80%) are exactly these fitted slopes, yet the fitting procedure is absent. Moreover, several s values in the figures are inconsistent with the table entries: for example, Figure 3b reports s=0.983 for MoM in the 1.4B image setting, while Table 4 reports 34.76% for the same row. The authors must provide the full matching and conversion procedure and reconcile the figures and tables.
- [Sections 3.1-3.3] All results are from single training runs with no error bars or repeated seeds. Given the noise in loss curves, the reported four-significant-figure percentages (e.g., 34.76%, 42.50%) are not robust. This is especially relevant because the matching procedure is sensitive to small loss differences; the authors should report variance across seeds or at least a sensitivity analysis showing how the percentages change under reasonable perturbations of the matching tolerance.
- [Abstract and Section 3] The paper claims to 'set new benchmarks in multi-modal pretraining' but never evaluates on any standard multimodal benchmark or downstream task; the evidence is limited to training and validation loss curves. To support the significance claim, the authors should either temper the language or provide downstream evaluation (e.g., image generation quality, text-to-image retrieval, or standard pretraining evaluations such as those used in Chameleon or Transfusion).
minor comments (7)
- [Section 2.2, Eq. (2)] The diffusion loss weight λ is never specified. Please provide its value or schedule, since it affects the relative scaling of image and text losses and thus the interpretation of the average loss curves.
- [Tables 5 and 6] The training configuration tables list only architecture and token-batch sizes; the optimizer, learning rate, schedule, and other training hyperparameters are not given anywhere in the paper. These are needed for reproducibility.
- [Figure captions, Figures 3-14] The loss-matching plots do not define the y-axis or the meaning of the reported s values. The captions should explain that the points are 'steps to match dense' and that s is the slope of the fitted line, along with the fitting range and any smoothing.
- [Table 4] The caption says the table reports relative training FLOPs for both Mixture-of-Mamba and Flex-Attention Transformer, but only one set of numbers appears. Clarify which model the column refers to, or add the missing values.
- [Section 3.4, Table 3] The 'synergistic effect' is claimed because the full decoupling gain (3.80%) exceeds the sum of individual gains (0.44%). This is a trivially true comparison; a more meaningful test would compare the full model against the best pair or an additive model of expected gains.
- [References] Several references have broken URLs (e.g., 'URLhttps://' with no space) and inconsistent capitalization (e.g., 'V LAV A'). Please correct these formatting issues.
- [Impact Statement] The impact statement says 'reduces computational costs by up to 65%' but the paper's data show reductions up to roughly 75% (e.g., 24.80% relative FLOPs for speech). Please align these figures.
Circularity Check
No significant circularity: the headline efficiency results are empirical comparisons of independently trained loss curves, not conclusions forced by the architecture's definition or by self-citation.
full rationale
The paper's central quantitative claims are measurements, not derivations. Mixture-of-Mamba, Mamba Dense, and the Flex-Attention Transformer are trained from scratch under the same schedules, and the reported loss values (Tables 1, 2, 4, 7), performance gains, and 'Relative Training FLOPs to Match Mamba' percentages are read off the training and validation curves in Figures 3-14. Nothing in the definition of the architecture (Algorithm 1) or in the loss formulas (Eqs. 1-3) presupposes the outcome that MoM matches dense baselines at 34.76% or 42.50% of the FLOPs; those numbers are empirical summaries of the fitted slopes s in the 'Steps to Match Dense' plots. The only self-referential element is the design hypothesis in Section 2.1, which is attributed to the authors' own Mixture-of-Transformers paper ('Our hypothesis is that explicitly parametrizing the selection in SSMs with the modality can improve the data efficiency of multi-modality training (Liang et al., 2024)'). That citation motivates the architecture but is not load-bearing for the empirical results: the comparison to Mamba Dense is an external benchmark that would stand or fall on the loss curves regardless of the citation. The under-specified 'match' tolerance and the implicit equal-FLOPs-per-step conversion are reproducibility weaknesses, not circular steps, because no equation or fitted parameter makes the reported percentages equal to the paper's assumptions by construction. Accordingly, no circular step can be exhibited, and the honest finding is no circularity.
Assumptions & free parameters
free parameters (2)
- lambda (diffusion loss weight in Transfusion objective) =
not reported
- loss-matching fit slope s =
per-figure, e.g., s=0.983, s=0.378
assumptions (4)
- ad hoc to paper Modality-specific parameterization of SSM projections improves multimodal data efficiency.
- domain assumption Per-token FLOPs of MoM equal those of Mamba Dense, so the step ratio equals the FLOPs ratio.
- ad hoc to paper Training loss curves from different runs are comparable at equal step indices, and the match point can be estimated by a linear fit (slope s).
- domain assumption Every token carries a known, fixed modality label used for routing.
Cite this review
Pith. "Pith review of Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity." pith.science (2026). https://pith.science/paper/SG7JEMZG
@misc{pith2026250116295,
author = {Pith},
title = {Pith review of: Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity},
year = {2026},
howpublished = {\url{https://pith.science/paper/SG7JEMZG}},
note = {Machine review of arXiv:2501.16295}
}
read the original abstract
State Space Models (SSMs) have emerged as efficient alternatives to Transformers for sequential modeling, but their inability to leverage modality-specific features limits their performance in multi-modal pretraining. Here, we propose Mixture-of-Mamba, a novel SSM architecture that introduces modality-aware sparsity through modality-specific parameterization of the Mamba block. Building on Mixture-of-Transformers (W. Liang et al. arXiv:2411.04996; 2024), we extend the benefits of modality-aware sparsity to SSMs while preserving their computational efficiency. We evaluate Mixture-of-Mamba across three multi-modal pretraining settings: Transfusion (interleaved text and continuous image tokens with diffusion loss), Chameleon (interleaved text and discrete image tokens), and an extended three-modality framework incorporating speech. Mixture-of-Mamba consistently reaches the same loss values at earlier training steps with significantly reduced computational costs. In the Transfusion setting, Mixture-of-Mamba achieves equivalent image loss using only 34.76% of the training FLOPs at the 1.4B scale. In the Chameleon setting, Mixture-of-Mamba reaches similar image loss with just 42.50% of the FLOPs at the 1.4B scale, and similar text loss with just 65.40% of the FLOPs. In the three-modality setting, MoM matches speech loss at 24.80% of the FLOPs at the 1.4B scale. Our ablation study highlights the synergistic effects of decoupling projection components, where joint decoupling yields greater gains than individual modifications. These results establish modality-aware sparsity as a versatile and effective design principle, extending its impact from Transformers to SSMs and setting new benchmarks in multi-modal pretraining. Our code can be accessed at https://github.com/Weixin-Liang/Mixture-of-Mamba
Figures
Figures from the paper (11 more)
Forward citations
Cited by 1 Pith paper
-
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Scaling the number of interaction steps, trained via a curriculum over rollout horizon, improves web-agent task success and outperforms scaling per-step reasoning under fixed token budgets.
Reference graph
Works this paper leans on
-
[1]
Blackmamba: Mixture of experts for state-space models
Anthony, Q., Tokpanov, Y ., Glorioso, P., and Millidge, B. Blackmamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771,
-
[3]
URL https://arxiv. org/abs/2405.09818. Eigen, D., Ranzato, M., and Sutskever, I. Learning fac- tored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314,
-
[5]
Make-a-scene: Scene-based text- to-image generation with human priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y . Make-a-scene: Scene-based text- to-image generation with human priors. arXiv preprint arXiv:2203.13131,
-
[6]
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,
-
[9]
URLhttps://arxiv. org/abs/2401.04088. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,
arXiv 2000
-
[11]
Mixture-of-transformers: A sparse and scalable architec- ture for multi-modal foundation models
Liang, W., Yu, L., Luo, L., Iyer, S., Dong, N., Zhou, C., Ghosh, G., Lewis, M., Yih, W.-t., Zettlemoyer, L., et al. Mixture-of-transformers: A sparse and scalable architec- ture for multi-modal foundation models. arXiv preprint arXiv:2411.04996,
-
[12]
V ., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Gosh, G., Zettlemoyer, L., and Aghajanyan, A
Lin, X. V ., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Gosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint arXiv:2407.21770,
-
[13]
H., Chang, H.-J., Auli, M., Hsu, W.-N., and Glass, J
Liu, A. H., Chang, H.-J., Auli, M., Hsu, W.-N., and Glass, J. R. Dinosr: Self-distillation and online clustering for self-supervised speech representation learning, 2024a. URL https://arxiv.org/abs/2305.10005. Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, L., Shrirao, S., Lambert, C., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-t...
Show all 24 references
-
[14]
and Tian, Y
Mo, S. and Tian, Y . Scaling diffusion mamba with bidi- rectional ssms for efficient image and video generation. arXiv preprint arXiv:2405.15881,
-
[15]
Vl-mamba: Exploring state space models for multimodal learning
Qiao, Y ., Yu, Z., Guo, L., Chen, S., Zhao, Z., Sun, M., Wu, Q., and Liu, J. Vl-mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600,
-
[16]
Glu variants improve transformer
Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202,
2002 arXiv
-
[19]
M., Staten, C., Khodak, M., Neubig, G., and Talwalkar, A
Shen, J., Li, L., Dery, L. M., Staten, C., Khodak, M., Neubig, G., and Talwalkar, A. Cross-modal fine-tuning: align then refine. In Proceedings of the 40th International Conference on Machine Learning, 2023a. Shen, J., Jain, A., Xiao, Z., Amlekar, I., Hadji, M., Podolny, A., a...
-
[21]
org/abs/2403.07816
URL https://arxiv. org/abs/2403.07816. Sun, Y ., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y ., Chen, X., Wang, X., Koyejo, S., et al. Learn- ing to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620,
-
[22]
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y ., Ji, J., Yang, Z., Zhao, L., Song, X., et al
URL https://arxiv.org/abs/2208.10442. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y ., Ji, J., Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079,
-
[23]
URL https://arxiv.org/abs/2411.02796. Yan, J. N., Gu, J., and Rush, A. M. Diffusion models without attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8239– 8249,
-
[24]
Cobra: Extending mamba to multi-modal large language model for efficient inference
Zhao, H., Zhang, M., Zhao, W., Ding, P., Huang, S., and Wang, D. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403.14520,
-
[25]
Transfusion: Predict the next token and dif- fuse images with one multi-modal model
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and 11 Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity Levy, O. Transfusion: Predict the next token and dif- fuse images with one mult...
-
[2017]
URL http:// arxiv.org/abs/1701.06538. Shen, J. and Yang, L. F. Theoretically principled deep rl acceleration via nearest neighbor function approximation. Proceedings of the AAAI Conference on Artificial Intel- ligence, 35(11):9558–9566, May
-
[2020]
Liang, V
URL https://arxiv.org/abs/ 2006.16668. Liang, V . W., Zhang, Y ., Kwon, Y ., Yeung, S., and Zou, J. Y . Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems , 35: 17612–17625,
2006 arXiv
-
[2021]
Mars: Mixture of auto-regressive models for fine-grained text-to-image syn- thesis
He, W., Fu, S., Liu, M., Wang, X., Xiao, W., Shu, F., Wang, Y ., Zhang, L., Yu, Z., Li, H., et al. Mars: Mixture of auto-regressive models for fine-grained text-to-image syn- thesis. arXiv preprint arXiv:2407.07614,
-
[2022]
Fei, Z., Fan, M., Yu, C., and Huang, J
URL https://arxiv.org/ abs/2101.03961. Fei, Z., Fan, M., Yu, C., and Huang, J. Scalable diffu- sion models with state space backbone. arXiv preprint arXiv:2402.05608,
-
[2023]
Efficiently modeling long sequences with structured state spaces
Gu, A., Goel, K., and R ´e, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396,
-
[2024]
K., Aggarwal, K., Som, S., Piao, S., and Wei, F
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., and Wei, F. Vlmo: Unified vision-language pre-training with mixture-of- modality-experts. Advances in Neural Information Pro- cessing Systems, 35:32897–32912, 2022a. Bao, H., Wang, W., Dong...
-
[2025]
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., and He, Y
URL https://arxiv.org/abs/ 2501.03120. Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., and He, Y . Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023b. Shen, Y ., Guo, Z., Cai, T., and Qin, Z. Jetmoe: Reaching llama2 perfo...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.