{"id":"596a4381-6c0a-48bb-b37e-dfa40bb458fa","arxiv_id":"2501.16295","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Modality-specific projection weights let a Mamba model match dense multimodal baselines at the same loss using 25% to 65% of the training compute.","lead":"Mixture-of-Mamba gives each data type, such as text, image, or speech, its own set of weight matrices inside a Mamba model, instead of forcing every token through the same weights. Across three multimodal training recipes it reaches the baseline's final loss in far fewer steps, using as little as 25% of the compute for the strongest modality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline FLOPs-to-match percentages depend on an unspecified loss-matching and FLOPs-conversion procedure; a reproducibility check on the released curves would settle whether the numbers are robust.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I find: the headline FLOPs savings are derived from an unspecified loss-matching procedure and an unstated equality of per-token FLOPs. I agree with the CONDITIONAL verdict. The qualitative result, that modality-split projections improve data efficiency in the loss curves, is consistent across scales and settings, so the paper need not be rejected; but the abstract's precise percentages cannot be taken at face value until the matching methodology and FLOPs-counting rule are specified and shown to reproduce the reported numbers. My proposed test directly checks reproducibility, which is the appropriate way to settle whether the central claim holds. I do not see a separate internal contradiction in the architecture itself, and the parameter/memory overhead, while real, is not the load-bearing part of the paper's stated claim about FLOPs.","tokens_in":31689,"tokens_out":8198,"duration_ms":79252,"concrete_test":"Using the released code or loss logs, reconstruct the 'Steps to Match Dense' curves with a fully specified rule: for each Dense checkpoint t, find the first MoM step where the smoothed MoM loss is no greater than the Dense loss at t, using linear interpolation between logged checkpoints, with a stated tolerance and smoothing window. Fit the slope over t in [0, T] and recompute the headline values: Table 4's 1.4B Transfusion image value (34.76%), Table 7's 1.5B Chameleon image value (42.50%), and Table 2's 1.5B speech value (24.80%). Separately compute per-step FLOPs for both architectures from the stated configs, including fused Win projections and the low-rank delta path, and verify whether the per-token FLOPs ratio is exactly 1.0.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the set of 'Relative Training FLOPs to Match Mamba' values (34.76% in Transfusion, 42.50% in Chameleon, 24.80% for speech), which appear in the abstract and in Tables 1, 2, 4, and 7. These numbers are evidently the fitted slope s of 'Steps to Match Dense' versus 'Training Step for Dense' shown in Figures 3-14 (for example, Figure 7b reports s=0.259 for the 37M image case, matching the table's 25.90%). However, the paper never defines how a 'match' is determined: no tolerance on loss difference, no smoothing or interpolation rule, no statement of the range of steps used for the linear fit, and no treatment of non-monotonic loss curves. If the match point is taken as the first crossing of a noisy loss curve, the fitted slope can shift substantially with the chosen smoothing window or interpolation rule. Furthermore, converting a step ratio into a FLOPs ratio requires per-token FLOPs to be identical for MoM and Mamba Dense; this is plausible because each token applies only its modality-specific projection, but it is nowhere stated, and the accounting for fused input projections and the low-rank delta factorization is not given. Every headline efficiency number is computed from single runs with no error bars. Because the abstract's 'significantly reduced computational costs' rests entirely on these percentages, the missing methodology is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixture-of-Mamba (MoM), a state-space model that applies modality-specific projection matrices (input, intermediate, and output projections) in each Mamba block, while keeping convolutions and state transitions shared. It evaluates MoM against dense Mamba and Flex-Attention baselines in three multimodal pretraining settings—Transfusion (continuous image tokens with diffusion loss), Chameleon (discrete image tokens), and Chameleon+Speech—across model scales from 37M to 1.5B. The central claim is that MoM matches dense Mamba's training loss at a fraction of the training FLOPs (e.g., 34.76% for Transfusion image loss at 1.4B), while also improving final loss. An ablation study on decoupling the four projection components is included.","tokens_in":31860,"tokens_out":9024,"duration_ms":69922,"significance":"If the efficiency claim is correct, the paper makes a useful contribution by transferring the modality-aware sparsity design from Transformers (Mixture-of-Transformers) to SSMs, with consistent qualitative evidence across multiple scales and settings. The paper's strengths include the breadth of evaluation (three settings, five scales), the inclusion of validation loss curves on multiple datasets, and the availability of code. However, the quantitative FLOPs-savings claims are not backed by a described methodology, and the paper's own figures appear to contradict the table values in at least one case. The qualitative claim of faster convergence is well supported by the loss curves; the quantitative headline numbers are not.","major_comments":[{"comment":"The metric 'Relative Training FLOPs to Match Mamba' is never operationally defined. The paper does not specify the tolerance for a loss 'match', the interpolation or smoothing of loss curves, the range of steps used for the linear fit that produces the slope s, or the FLOPs accounting that converts a step ratio into a FLOPs ratio. This is load-bearing because the abstract's headline numbers (34.76%, 42.50%, 24.80%) are exactly these fitted slopes, yet the fitting procedure is absent. Moreover, several s values in the figures are inconsistent with the table entries: for example, Figure 3b reports s=0.983 for MoM in the 1.4B image setting, while Table 4 reports 34.76% for the same row. The authors must provide the full matching and conversion procedure and reconcile the figures and tables.","section":"Section 3.1, Figures 3-14, Tables 1/4/7"},{"comment":"All results are from single training runs with no error bars or repeated seeds. Given the noise in loss curves, the reported four-significant-figure percentages (e.g., 34.76%, 42.50%) are not robust. This is especially relevant because the matching procedure is sensitive to small loss differences; the authors should report variance across seeds or at least a sensitivity analysis showing how the percentages change under reasonable perturbations of the matching tolerance.","section":"Sections 3.1-3.3"},{"comment":"The paper claims to 'set new benchmarks in multi-modal pretraining' but never evaluates on any standard multimodal benchmark or downstream task; the evidence is limited to training and validation loss curves. To support the significance claim, the authors should either temper the language or provide downstream evaluation (e.g., image generation quality, text-to-image retrieval, or standard pretraining evaluations such as those used in Chameleon or Transfusion).","section":"Abstract and Section 3"}],"minor_comments":[{"comment":"The diffusion loss weight λ is never specified. Please provide its value or schedule, since it affects the relative scaling of image and text losses and thus the interpretation of the average loss curves.","section":"Section 2.2, Eq. (2)"},{"comment":"The training configuration tables list only architecture and token-batch sizes; the optimizer, learning rate, schedule, and other training hyperparameters are not given anywhere in the paper. These are needed for reproducibility.","section":"Tables 5 and 6"},{"comment":"The loss-matching plots do not define the y-axis or the meaning of the reported s values. The captions should explain that the points are 'steps to match dense' and that s is the slope of the fitted line, along with the fitting range and any smoothing.","section":"Figure captions, Figures 3-14"},{"comment":"The caption says the table reports relative training FLOPs for both Mixture-of-Mamba and Flex-Attention Transformer, but only one set of numbers appears. Clarify which model the column refers to, or add the missing values.","section":"Table 4"},{"comment":"The 'synergistic effect' is claimed because the full decoupling gain (3.80%) exceeds the sum of individual gains (0.44%). This is a trivially true comparison; a more meaningful test would compare the full model against the best pair or an additive model of expected gains.","section":"Section 3.4, Table 3"},{"comment":"Several references have broken URLs (e.g., 'URLhttps://' with no space) and inconsistent capitalization (e.g., 'V LAV A'). Please correct these formatting issues.","section":"References"},{"comment":"The impact statement says 'reduces computational costs by up to 65%' but the paper's data show reductions up to roughly 75% (e.g., 24.80% relative FLOPs for speech). Please align these figures.","section":"Impact Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct extension of the authors' own Mixture-of-Transformers work, and the novelty is incremental. The more serious concern is the reliability of the headline efficiency numbers given the unspecified matching procedure and the apparent mismatch between Figure 3b and Table 4. I recommend that the editors require the authors to document the fitting procedure, reconcile the figures and tables, and provide variance estimates before the paper is considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core design principle is sound - modality-specific Mamba projections improve multimodal data efficiency - and the qualitative evidence is consistent across three settings and five scales. The headline 'Relative Training FLOPs to Match Mamba' percentages (34.76%, 42.50%, 24.80%) are less solid. The loss-matching procedure is never defined: no tolerance, no smoothing, no fit range, and the step-to-FLOPs conversion assumes identical per-token FLOPs without ever saying so. Single runs, no error bars. That is the load-bearing weakness, and the stress-test concern lands. What the paper does well: it makes a clean architectural choice about which projections to split (Win, Wx, Wdt, Wout) and which to keep shared (Conv1D, A), and the ablation is genuinely informative - decoupling Wx or Wdt alone slightly hurts, but joint decoupling is synergistic. That is a nontrivial design insight. The evaluation is extensive: three settings, five scales, training and validation losses, and a sensible comparison with a causal-only caveat for Flex-Attention. This is real empirical work. Soft spots beyond the matching methodology: the abstract overclaims 'new benchmarks' for what is a loss-curve analysis with no downstream evaluation. The impact statement's 'up to 65% cost reduction' refers to steps-to-match savings, not per-step compute reduction, and the parameter/memory overhead of storing one projection copy per modality is never quantified. These claims need to be toned down or carefully scoped. Who is this for: researchers working on multimodal SSMs and efficient pretraining. The qualitative finding is likely useful and citable; the specific FLOPs numbers should be treated cautiously until the matching procedure is specified and error bars are provided. The paper deserves a serious referee - the central idea is plausible and well-tested qualitatively - but it needs major revision on the efficiency claims before publication.","headline":"The paper shows a real data-efficiency gain from modality-specific Mamba projections, but the headline FLOPs-to-match percentages rest on an undefined fitting procedure and should not be quoted without a rigorous methodology.","tokens_in":557,"tokens_out":573,"would_cite":true,"duration_ms":29261,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By giving each modality its own projection weights inside the Mamba block, Mixture-of-Mamba matches multimodal loss with 35–65% of the FLOPs.","keywords":["Mixture-of-Mamba","state space models","modality-aware sparsity","multi-modal pretraining","Mamba block","training FLOPs","loss matching","early fusion"],"falsifier":"Re-run the 1.4B Transfusion training while recording actual forward-backward FLOPs for both models at a fixed sequence length and modality mix, including the fused input projections and low-rank delta factorization, then plot the resulting loss-matching FLOPs ratio as a function of the matching tolerance (for example loss differences of 0.01, 0.001, and 0.0001) and of extrapolation beyond the final 250k training steps.","tokens_in":1663,"feed_emoji":"","tokens_out":4191,"duration_ms":64806,"temperature":0.7,"pith_summary":"Mixture-of-Mamba is a state-space model architecture that makes the Mamba block modality-aware: instead of one dense set of projections for every token, it keeps separate input, intermediate, discretization, and output projections per modality, selected by the token's modality. The paper claims this change lets the model match the image, text, and speech losses of dense Mamba baselines at a fraction of the training compute, for example 34.76% of the FLOPs for image loss in the Transfusion setting at the 1.4B scale. If true, modality-aware sparsity, previously demonstrated for Transformers, transfers to state-space models and buys data efficiency without adding per-token compute. The authors also report a synergistic effect: jointly decoupling all four projections gives a larger gain than the sum of the individual decoupling gains.","feed_headline":"Modality-split Mamba matches image loss at 35% of FLOPs","feed_subtitle":"Splitting Mamba's projections by modality buys the same loss with a fraction of the training compute.","key_machinery":"The Mixture-of-Mamba block is the central object. All four projections that consume single-modality features, namely the input projection $\\mathbf{W}_{\\mathrm{in}}$, the intermediate projection $\\mathbf{W}_{\\mathbf{x}}$ producing $\\delta, B, C$, the $\\mathbf{W}_{\\Delta t}$ projection for the discretization step $\\Delta$, and the output projection $\\mathbf{W}_{\\mathrm{out}}$, are replaced by modality-indexed copies, while Conv1D and the state transition $A$ stay shared because they act across features or on aggregated hidden states. The routing function $\\mathcal{M}(\\mathbf{X}, W, b; M)$ applies $W_m$ to the tokens of modality $m$ and concatenates the results, preserving the output shape of the original linear layer. This one change, without learned gating or mixture-of-experts machinery, is what the paper credits for the earlier loss matching.","core_discovery":"The paper's central claim is that replacing a Mamba block's dense projections with modality-conditional copies, writing $XW+b$ as $\\mathcal{M}(X,W,b;M)$, produces consistently lower training loss and earlier matching of the dense baseline's loss across the Transfusion, Chameleon, and Chameleon-plus-speech settings. Concretely, Mixture-of-Mamba matches dense Mamba's image loss with 34.76% of the training FLOPs in Transfusion at 1.4B, with 42.50% of the FLOPs for Chameleon image loss, with 24.80% of the FLOPs for speech loss in the three-modality run, and with 65.40% of the FLOPs for Chameleon text loss. The efficiency is framed as reaching the same loss value earlier in training at comparable per-token FLOPs, not as a reduction in per-step cost.","pith_inferences":["If per-token FLOPs are truly equal, the reported percentages imply roughly a threefold data-efficiency multiplier at constant compute; this should be tested outside loss matching, for example by fixing a compute budget and measuring final validation loss.","The paper measures FLOPs, not wall-clock time; the real training-time gain could be larger if modality copies improve memory layout or smaller if they hurt tensor-core utilization, and that gap is left untested.","The shared $A$ and Conv1D may be carrying cross-modal transfer, so ablating the sharing, or making $A$ modality-specific, would reveal whether the remaining shared parts are the right place for generalization.","A natural extension is adding more modalities or learned routing; the rule-based modality mask is simple and stable, but the paper does not show how it scales beyond three modalities."],"forward_implications":["If the claim is correct, modality-aware sparsity is a general design principle for sequence models and can be applied inside an SSM block, not only in Transformers.","The joint decoupling of all four projections beats every subset in the ablation, implying a synergy that future architectures should reproduce rather than decoupling components piecemeal.","Since Conv1D and state transitions remain shared, Mixture-of-Mamba is orthogonal to MoE sparsification of MLP layers and could be stacked with it.","The gains appear across continuous diffusion-loss image tokens, discrete VQ tokens, and a third speech modality, so the benefit is not tied to one representation or objective.","Consistent across scales from 37M to 1.5B parameters, the pattern suggests the efficiency advantage holds during early-fusion multimodal pretraining in general."],"supporting_citations":[{"why":"Defines the Mamba block whose projections Mixture-of-Mamba decouples; supplies the base architecture and selective state-space machinery.","marker":"(Gu & Dao, 2023)"},{"why":"Source of the modality-aware sparsity idea; Mixture-of-Transformers demonstrated it for Transformers, and this paper extends the same principle to the Mamba block.","marker":"(Liang et al., 2024)"},{"why":"Defines the Transfusion training setting with interleaved text and continuous image tokens under diffusion loss, the benchmark for the headline 34.76% claim.","marker":"(Zhou et al., 2024)"},{"why":"Defines the Chameleon discrete-token setting used for the 42.50% image-loss and 65.40% text-loss claims.","marker":"(Chameleon Team, 2024)"},{"why":"The closest prior sparsity-in-SSM work; it sparsifies only MLP layers via MoE, leaving the dense Mamba block intact, which this paper contrasts with direct projection decoupling.","marker":"(Pióro et al., 2024)"}],"fun_headline_variants":["Modality-split Mamba matches loss at 35% FLOPs","Mixture-of-Mamba: same loss, 35% of compute","Splitting Mamba projections by modality cuts training FLOPs","Mamba with modality-aware sparsity matches loss, uses 35% FLOPs","Multi-modal Mamba: modality-split projections match loss at 35% FLOPs"],"cache_read_input_tokens":34432,"weakest_assumption_plain":"The headline FLOPs savings assume that Mixture-of-Mamba and dense Mamba consume the same FLOPs per token, so that a step-count ratio equals a FLOPs ratio, and that the steps-to-match procedure has a defined tolerance and fitting rule; neither is stated in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Modality-split Mamba matches loss at 35% FLOPs","Mixture-of-Mamba: same loss, 35% of compute","Splitting Mamba projections by modality cuts training FLOPs","Mamba with modality-aware sparsity matches loss, uses 35% FLOPs","Multi-modal Mamba: modality-split projections match loss at 35% FLOPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001503,"raw_usage":{"total_tokens":6110,"prompt_tokens":1111,"completion_tokens":4999,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":727,"completion_tokens_details":{"reasoning_tokens":4896}},"tokens_in":727,"tokens_out":4999,"duration_ms":34468,"temperature":1.0,"reasoning_tokens":4896,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:35:04.541143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1.4B Transfusion training while recording actual forward-backward FLOPs for both models at a fixed sequence length and modality mix, including the fused input projections and low-rank delta factorization, then plot the resulting loss-matching FLOPs ratio as a function of the matching tolerance (for example loss differences of 0.01, 0.001, and 0.0001) and of extrapolation beyond the final 250k training steps.","supporting_citations":[],"review_version":1}