{"id":"2028e4cc-d81a-4b4c-9603-d49126e9e35a","arxiv_id":"2411.08840","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"MMJamba applies a hybrid transformer-Mamba LLM backbone to multimodal instruction tuning, claiming 4x faster inference on long visual contexts and state-of-the-art accuracy on image and video benchmarks.","lead":"A multimodal language model built on Jamba, a hybrid transformer-Mamba backbone, claims to handle over 100k input tokens from high-resolution images and videos at roughly four times the throughput of a comparable LLaVA model. The paper also reports a train-on-short-infer-on-long recipe, training on low-resolution images and few frames while inferring on high-resolution and many frames.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparisons in Tables 1/2/5 are invalidated by implausibly low LLaVA-NeXT scores at high resolution (TextVQA 65.7→19.4→0.5), so the central 'outperforming existing models' claim is not supported until the baseline protocol is reproduced.","rationale":"The reader's verdict is REJECT, and I agree with that outcome, but my primary reason differs from the stated weakest_assumption. The train-short-infer-long extrapolation is a genuine risk, and Tables 5 and 6 do show declines as context grows (EgoSchema drops from 58.7 at 8 frames to 52.45 at 64 frames; MME drops from 1655 to 1640 as resolution rises), which the paper itself acknowledges. However, as a scientific claim, 'outperforming existing models' depends on the baseline rows being correct, and those rows look broken. Published LLaVA-NeXT numbers are well established; a drop from 65.7 to 0.5 on TextVQA at 2688×2688 is not graceful degradation but a sign that the evaluation harness is not the model's intended protocol. The same 13B model scores 1575 MME at 672², 1353 at 1344², and 1251 at 2688², which is not typical behavior for LLaVA-NeXT with dynamic tiling. If the baseline protocol is flawed, all state-of-the-art comparisons in Tables 1 through 5, plus the efficiency analysis in Figure 2 where LLaVA-NeXT is the reference point, are suspect. I am not alleging misconduct; a preprocessing mismatch is the simplest explanation. This is a correctness risk, not a novelty problem. The 'first to train short and infer long' claim is also unsupported because the model trains on at most 4 tiles and 8 frames but infers on up to 64 tiles and 64 frames without demonstrated generalization, yet that is secondary. The concrete test above would settle whether the central comparative claim can be trusted.","tokens_in":17723,"tokens_out":5433,"duration_ms":50970,"concrete_test":"Reproduce Table 5's LLaVA-NeXT-7B/13B rows using the official released checkpoints, prompt templates, and image preprocessing for LLaVA-NeXT, with AnyRes maximum tiles set to match 1344×1344 and 2688×2688 as the paper specifies. Evaluate the same subsets of TextVQA, MM-Vet, MME, and VizWiz used in Table 5. If TextVQA recovers to approximately 65 rather than 19.4 or 0.50, and MME to approximately 1590 rather than 1246 or 1158, the paper's baseline protocol is not the intended LLaVA-NeXT protocol, and the 'outperforms' conclusion is unsupported. Additionally, verify whether the official LLaVA-NeXT code supports tiling beyond its designed maximum and whether Table 5's numbers were produced by upscaling images before tiling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Tables 1, 2 and 5 compare MMJAMBA against LLaVA-NeXT, but the LLaVA-NeXT rows in Table 5 are internally inconsistent with published and even same-paper numbers. In Table 2, LLaVA-NeXT-7B scores 65.7 on TextVQA, 47.3 on MM-Vet, and 1589.7 on MME under its normal protocol. In Table 5, the same model at 1344×1344 drops to 19.4 TextVQA, 30.9 MM-Vet, and 1246 MME; at 2688×2688 TextVQA collapses to 0.50, MM-Vet to 22.4, and MME to 1158. These values are far below any reported LLaVA-NeXT result and indicate an evaluation-protocol mismatch, for example upscaling low-resolution benchmark images to high resolution or applying a tiling strategy that the released checkpoints do not support. Because every comparative claim in the paper is measured against these baselines, the central assertion that MMJAMBA outperforms existing models does not currently rest on valid comparisons. This concern is more load-bearing than the train-short-infer-long extrapolation issue: even if Mamba generalizes perfectly, the yardstick used to demonstrate superiority is unreliable. The paper releases no code or weights, so the baselines cannot be independently reproduced from the manuscript alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMJamba, a multimodal large language model built on the hybrid transformer-Mamba backbone Jamba, with the stated goal of efficiently handling long visual contexts arising from high-resolution images and high-frame-rate videos. The method uses an AnyRes/AnyFrame visual encoding scheme, trains on short contexts (at most 4 image tiles and 8 video frames), and then performs inference on much longer contexts (up to 2688x2688 resolution or 64 frames). The authors report results on 18 image and video benchmarks, claim state-of-the-art performance relative to open-source and some proprietary models, and report roughly 4x throughput gains over LLaVA-NeXT at high resolution. The paper is empirical: there is no theoretical derivation, and the central claims are measurements of accuracy and efficiency.","tokens_in":18003,"tokens_out":4655,"duration_ms":44007,"significance":"If the empirical claims were sound, the train-short-infer-long recipe would be a practically interesting contribution, since it would decouple training cost from inference-time context length and could make long-context multimodal inference more affordable. The efficiency analysis in Figure 2 also addresses a real bottleneck in high-resolution multimodal processing. However, the paper's central comparative claims rest on baseline numbers in Tables 5 and 7 that are internally inconsistent and implausible, and the paper's own scaling results in Tables 5 and 6 show nontrivial degradations on several benchmarks as context length increases. Without corrected baseline evaluations and a more careful characterization of the length-scaling behavior, the stated conclusions are not supported. No code or model weights are released, which further limits the reproducibility of the empirical claims.","major_comments":[{"comment":"Table 7, which reports results 'without AnyRes', shows even more extreme baseline collapses: LLaVA-NeXT-7B scores 6 on MME, 3.3 on MMB-EN, 0.48 on TextVQA and 0.27 on GQA at 1344x1344. These numbers are not informative about the models' capabilities; they indicate that the evaluation harness is feeding inputs outside the operating range of the baseline models. If the intent is to show that LLaVA-NeXT cannot process such inputs without AnyRes, that is a known and unremarkable fact; it does not support the paper's claim that MMJamba outperforms these models under a fair protocol. The authors must clarify exactly what preprocessing was applied to baseline inputs and provide results under each baseline's supported protocol.","section":"Section 4.2, Table 7"},{"comment":"The central claim that the model 'can effectively process long context input' is undermined by the paper's own frame-scaling results. Increasing the number of inference frames from 8 to 64 causes EgoSchema to drop from 58.7 to 52.45, MVBench from 61.0 to 58.8, and VideoMME from 50.1 to 46.3. The text describes these as 'slight declines', but a 6.3-point drop on EgoSchema is substantial. At 64 frames, VideoMME is worse than at 8 frames, which contradicts the narrative that longer visual context improves effectiveness. The authors should either explain these degradations with analysis or revise the claim to state that the train-short-infer-long recipe improves performance on some benchmarks but degrades it on others.","section":"Section 5.2, Table 6"},{"comment":"The resolution scaling for MMJamba itself is also mixed. While MM-Vet, TextVQA, and MMMU improve from 672 to 1344, other benchmarks decline: MME falls from 1655 to 1640, LLaVA-Wild from 83.9 to 80.0, and VizWiz from 57.6 to 54.3 as resolution increases to 2688. The text attributes these to 'the inherent nature and characteristics of different benchmarks', but no evidence is provided for this explanation. Since the paper's headline claim is that training on short contexts and inferring on long contexts yields better performance, the authors need to quantify how often the recipe helps versus hurts and avoid blanket statements that higher inference resolution 'enhances the performance'.","section":"Section 6, Table 5"},{"comment":"The efficiency claim of 'about 4 times faster' is presented without sufficient experimental detail. Figure 2 reports first-inference latency and throughput, but the hardware, batch size, tensor-parallel configuration, tokenizer, and generation length are not specified. The text says throughput is 'calculated as average tokens processed when generating 1024 tokens', but no numerical values are given in the text or figure. To support the efficiency claim, the authors should provide a table with exact latency and throughput numbers for each model and configuration, including the 4368x4368 setting, and specify the evaluation environment.","section":"Section 6, Figure 2"}],"minor_comments":[{"comment":"There is a typo in the contributions list: 'We conducte experiments' should be 'We conduct experiments'.","section":"Section 1"},{"comment":"The evaluation section lists MathVista among the benchmarks, but no MathVista results appear in any table. Either add the results or remove the mention.","section":"Section 4.1"},{"comment":"The video training data description says 'LLaVA-665K' is used, but the same name is used for the image training set. Please clarify whether these are the same or different collections.","section":"Section 5.1"},{"comment":"The table header for Table 5 is misaligned: the column label 'LLaVA' is separated from 'Wild' and 'Vet', which makes the table hard to read. Please fix the header formatting.","section":"Section 6, Table 5"},{"comment":"The abstract and introduction state that the model is 'the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos.' This claim should be qualified: the ability to run at longer contexts does not imply that performance is maintained or improved, as Tables 5 and 6 show declines on several benchmarks.","section":"Section 1"},{"comment":"The video case studies are referenced in the main text but appear only in the appendix. Please add explicit references to Figure 4 in the appropriate section of the main text.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the validity of the baseline comparisons, not the absence of a theoretical derivation. The authors should be asked to rerun all LLaVA-NeXT and other baseline evaluations at high resolution using the official evaluation code and to report the exact preprocessing steps. If the corrected baselines no longer show a consistent advantage for MMJamba, the paper's headline claim will need to be substantially weakened. Given that no code or weights are released, the authors should also be strongly encouraged to release the evaluation scripts and model to make the corrected numbers verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one genuinely interesting thing here is the train-short/infer-long recipe: train a Jamba-based MLLM on low resolution or few frames, then at inference feed it much longer visual inputs and let the Mamba state handle the extrapolation. That is a real, practical idea, and the efficiency story (4x faster at 4368^2 than LLaVA-NeXT-13B) is mechanistically plausible given the hybrid SSM backbone. The model itself, MMJamba, is also competitive at its native 672^2 resolution on several benchmarks, which suggests the backbone swap doesn't hurt ordinary performance. Credit where due: they ran a lot of benchmarks, reported ablations over resolution and frame count, and cited LongLLaVA as concurrent work.\n\nBut the comparative claims are not supported. Table 5 is a red flag: LLaVA-NeXT-7B, which scores 65.7 on TextVQA in the paper's own Table 2, collapses to 19.4 at 1344^2 and 0.5 at 2688^2. That is not normal degradation; it looks like the baselines were evaluated with a protocol the released checkpoints don't support, maybe upscaling low-res benchmark images to high res or applying a tiling scheme they weren't designed for. Since every \"outperforms existing models\" claim is measured against these baselines, the central assertion currently rests on invalid comparisons. No code or weights are released, so the baselines can't be independently reproduced.\n\nThere's also an internal tension in the narrative. The abstract and introduction say the model maintains or improves performance as resolution and frame count rise, but Tables 5 and 6 show MME dropping from 1655 to 1640, LLaVA-Wild from 83.9 to 80.0, VizWiz from 57.6 to 54.3, EgoSchema from 58.7 to 52.45, and MVBench from 61.0 to 58.8. The text does acknowledge \"slight decline\" in some benchmarks, but the overall framing of \"effectively process long context\" is overstated. The train-short/infer-long idea works for some tasks, not others, and the paper doesn't investigate why.\n\nThe novelty is also thinner than the title suggests. Applying Jamba to multimodal instruction tuning is a direct extension of LongLLaVA, which they cite; the new bits are the training recipe and the AnyFrame mechanism. Those are worth taking seriously, but they need to be evaluated against properly run baselines, with code released.\n\nWho is this for? Researchers working on long-context multimodal models, especially those interested in hybrid SSM architectures, would benefit from seeing the recipe and the efficiency measurements. But I'd caution them to ignore the comparative tables until a revision with corrected baselines appears.\n\nRecommendation: send it to peer review, because the train-short/infer-long idea deserves referee attention and possible revival after major revision. But as it stands, I would not cite it, and I'd tell the authors to fix the baseline protocol, reconcile the scaling tables with the claims, and release code and weights.","headline":"A promising train-short/infer-long recipe for hybrid SSM multimodal LLMs, but the evaluation is not trustworthy because the LLaVA-NeXT baselines are broken and the scaling tables contradict the abstract.","tokens_in":18590,"tokens_out":1881,"would_cite":false,"duration_ms":18884,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a hybrid transformer-Mamba backbone lets a multimodal model handle over 100k visual tokens and run about four times faster than current open-source models at high resolutions and frame rates.","keywords":["multimodal large language model","hybrid state space model","Mamba","long-context vision","high-resolution images","video understanding","instruction tuning","train-short-infer-long"],"falsifier":"Feed MMJamba a single multimodal input over 100k tokens, such as 64 high-resolution frames plus a long prompt, and measure time-to-first-token together with accuracy on a question about one of the earliest frames. The central claim predicts near-linear latency growth and stable accuracy; quadratic latency scaling or a sharp accuracy drop for the early-frame question would refute it.","tokens_in":17480,"feed_emoji":"⚡","tokens_out":10882,"duration_ms":90103,"temperature":0.7,"pith_summary":"Multimodal models that read high-resolution images or many video frames pay a quadratic price because self-attention scales with the number of visual tokens. This paper argues that swapping the attention-only backbone for a hybrid transformer-Mamba model removes that quadratic penalty, allowing a model trained on short, low-cost inputs to process more than 100k tokens at inference. The two concrete promises are that the resulting model, MMJamba, outperforms existing open models across the benchmarks tested and runs about four times faster than current open-source models at very high resolutions and frame rates. If true, high-resolution and high-frame-rate multimodal understanding becomes available without the usual training cost for long contexts.","feed_headline":"Mamba-transformer hybrid handles 100k visual tokens 4x faster","feed_subtitle":"A train-short-infer-long recipe removes the quadratic attention penalty for high-resolution images and long videos.","key_machinery":"The load-bearing mechanism is the hybrid Jamba backbone: Mamba selective state-space layers, whose hidden state updates every token at linear cost, interleaved with transformer attention layers and mixture-of-experts. The recurrent state is what lets the model carry information over sequences far longer than those seen in training, and the paper's train-on-short-infer-on-long recipe is built directly on that recurrence. The recipe freezes the vision encoder, trains only the adapter and LLM on short sequences (at most 4 tiles, 672x672 images, and 8 frames), and then at inference feeds much larger tile counts and frame counts through the same recurrent layers.","core_discovery":"The paper's central claim is that a multimodal instruction-tuned model named MMJamba can train on short visual contexts and then infer on much longer ones. MMJamba uses a frozen CLIP vision encoder, an AnyRes image-tiling scheme that allows at most 4 tiles during training but up to 64 during inference, an AnyFrame video sampler that trains on 8 frames and infers on up to 64, and a Jamba-52B hybrid decoder that interleaves Mamba state-space layers with transformer layers and mixture-of-experts. The authors report top results among the open models they compare with on image benchmarks, gains over open-source video baselines with occasional matches or wins against proprietary models, and an efficiency analysis showing throughput about four times higher than the primary open-source baseline at 4368x4368 resolution, with the gap widening as resolution or frame count rises. They also claim to be the first to train on low-resolution images or low-frame-rate videos while inferring on high-resolution and high-frame-rate inputs.","pith_inferences":["I infer that the same train-short-infer-long recipe would likely transfer to other long-form modalities, such as audio or long interleaved documents, because the bottleneck the paper removes is architectural rather than visual; the paper does not test this.","I infer that the recurrent state's ability to compress visual tokens is the real precondition for the recipe, so a direct probe of hidden-state consistency across 1, 4, 16, and 64 tiles would be a sharper test than benchmark averages; the paper does not report such a probe.","I infer that the observed accuracy drops on EgoSchema and VideoMME at 64 frames, which the paper attributes to benchmark characteristics, could also signal a length ceiling in the Mamba state; distinguishing those requires a frame-order perturbation experiment.","I infer that the throughput comparison may understate the real-world speedup for long generations, since the paper measures average tokens processed over 1024 generated tokens rather than complete responses."],"forward_implications":["If the recipe generalizes, long-context visual pretraining becomes unnecessary for this architecture, cutting the training cost of high-resolution and high-frame-rate models.","The efficiency gap over attention-based models should widen as resolution or frame count grows, because attention cost is quadratic while Mamba's recurrence is linear in token count.","Visual token pruning and downsampling become less necessary, which the paper argues avoids the information loss that such compression introduces.","Deploying the same checkpoint at different resolutions or frame counts becomes a runtime choice between speed and fidelity, without retraining."],"supporting_citations":[{"why":"Supplies the Jamba hybrid transformer-Mamba/MoE backbone, the architectural base of MMJamba.","marker":"Lieber et al., 2024"},{"why":"Supplies the selective state-space Mamba layer whose linear-time recurrence is the basis for the train-short-infer-long recipe.","marker":"Gu & Dao, 2023"},{"why":"Supplies the AnyRes image-tiling method and the primary open-source baseline for quality and efficiency comparisons.","marker":"Liu et al., 2024c"},{"why":"Supplies the two-stage visual instruction tuning recipe and the adapter-pretraining data that MMJamba follows.","marker":"Liu et al., 2024d"},{"why":"Supplies the frozen CLIP-ViT-Large vision encoder used for images and video frames.","marker":"Radford et al., 2021"},{"why":"Supplies the concurrent LongLLaVA hybrid-architecture model that MMJamba compares against and claims to outperform.","marker":"Wang et al., 2024"}],"fun_headline_variants":["Train short, infer long: Mamba-MLLM cuts cost 4x","MMJamba: 100k token multimodal, 4x faster inference","Hybrid Mamba-transformer boosts visual token speed 4x","Long visual context, 4x faster: Mamba-transformer hybrid"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model trained on at most 4 image tiles and 8 video frames will keep working when asked to reason over 64 tiles or 64 frames at inference, because the Mamba layers' recurrent state is expected to generalize to sequence lengths it never saw in training.","fun_headline_variants_meta":{"raw":{"variants":["Train short, infer long: Mamba-MLLM cuts cost 4x","MMJamba: 100k token multimodal, 4x faster inference","Hybrid Mamba-transformer boosts visual token speed 4x","Long visual context, 4x faster: Mamba-transformer hybrid"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1400,"prompt_tokens":994,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":610,"tokens_out":406,"duration_ms":115951,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:15:56.594493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed MMJamba a single multimodal input over 100k tokens, such as 64 high-resolution frames plus a long prompt, and measure time-to-first-token together with accuracy on a question about one of the earliest frames. The central claim predicts near-linear latency growth and stable accuracy; quadratic latency scaling or a sharp accuracy drop for the early-frame question would refute it.","supporting_citations":[],"review_version":1}