REVIEW 4 major objections 5 minor 35 references
Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A three-phase curriculum—text problems first, captions as a bridge, pure visual questions last—unlocks state-of-the-art multimodal math reasoning in a 3-billion-parameter model.
desk verdict A plausible three-phase RL curriculum for 3B multimodal reasoning, but the central claim needs a matched-compute control before it can be believed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the three-phase curriculum itself, driven by Group Relative Policy Optimization (GRPO)—a reinforcement learning update that scores a group of sampled responses by relative advantage, removing the need for a critic model. The reward function is staged: a format reward checks for the <think>...</think> reasoning structure and a final answer, and only when the format is valid does an accuracy reward compare the extracted answer against ground truth via mathematical verification, string matching, or multiple-choice matching. The bridge object is the caption-augmented dataset: for each image category, a caption generator produces a concise description that is prepended to the question during Phase 2, letting text-trained reasoning anchor to visual content before captions are withdrawn in Phase 3. The curriculum's effect is shown by the training dynamics: text-first training initially lengthens responses, then stabilizes them, while vision-first training keeps generating longer but less accurate answers. That difference is the paper's evidence that text-only RL activates reasoning in a way that later transfers across modalities.
What would settle it
Search every test question in MathVerse, MathVision, OlympiadBench, and MathVista against the training sets using a stricter similarity cut-off (say 0.80) and public near-duplicate lists; if Infi-MMR-3B's scores drop materially, the curriculum's reported gains are inflated by leaked answers. An independent fresh multimodal math test set would settle the claim more directly.
Extended reading notes
Core claim
The core claim is that the order in which a multimodal small language model meets reasoning data determines whether it learns to reason across modalities. Starting directly with multimodal reinforcement learning produces long, unstable, and often wrong reasoning traces; starting with text-only rule-based RL builds a stable reasoning base that can then be transferred. Caption-augmented multimodal data acts as the transfer bridge, and the final caption-free phase teaches the model to trust the image rather than the caption. The paper demonstrates this with ablations: text-first training outperforms vision-first training at every step, and caption-bridge training beats going straight to caption-free data on reasoning benchmarks, even though caption-free data improves general visual question answering. Infi-MMR-3B's reported scores on four benchmarks are presented as the result of this progression.
Load-bearing premise
The reported scores are valid only if the cleaning step that removes problems nearly identical or highly similar to test questions actually removed every overlap between training and test data; the similarity cut-off is chosen by hand and no leaked examples are shown.
Editorial extensions
If this is right
- A 3-billion-parameter model trained on this curriculum surpasses larger open-source models and one proprietary model on multimodal math reasoning benchmarks.
- Text-only rule-based RL is a viable and more stable first stage for multimodal reasoning than starting with visual data; it should be preferred whenever verifiable text reasoning data is available.
- Caption-augmented data is an effective bridge for transferring reasoning to vision: it outperforms going straight to caption-free multimodal RL on reasoning-heavy benchmarks, even though caption-free data yields better general visual question answering.
- A staged reward that enforces output format before rewarding answer accuracy guides the model to produce well-structured reasoning and then improves correctness.
- The curriculum's benefits hold across benchmarks of different difficulty, from diagram-heavy problems (MathVerse) to olympiad-level scientific questions (OlympiadBench).
Reading between the lines
- The same text-first -> caption-bridge -> caption-free schedule may transfer to other verifiable reasoning domains, such as STEM diagram problems or time-series chart reasoning, provided rule-based answer checking exists.
- The response-length signature (rise, then decline, then stabilization under text-first RL) could serve as a practical training monitor: if average response length keeps climbing without accuracy gains, the RL phase is likely producing verbose but unreliable reasoning.
- The decontamination threshold is a free parameter; a public audit of samples removed at the 0.95 similarity cut-off, and a re-evaluation under a stricter threshold, would directly test whether the reported state-of-the-art numbers are leakage-free.
- The caption-augmentation stage suggests a general principle for skill transfer across modalities: provide a shared semantic scaffold, then remove it gradually; the same pattern could apply to transferring reasoning from synthetic to real imagery.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Infi-MMR, a three-phase rule-based reinforcement learning curriculum for multimodal small language models (MSLMs). Phase 1 (FRA) trains on text-only mathematical reasoning data (DeepScaleR), Phase 2 (CMRA) trains on caption-augmented multimodal data (ViRL39k with Omnicaptioner-generated captions), and Phase 3 (MRE) continues on caption-free multimodal data (ViRL39k). The authors report that the resulting Infi-MMR-3B, built on Qwen2.5-VL-3B-Instruct, achieves 43.68% on MathVerse testmini, 27.04% on MathVision test, 21.33% on OlympiadBench, and 67.2% on MathVista testmini, and they claim state-of-the-art performance among the compared models. The paper also reports ablations comparing text-first versus multimodal-first initial RL and caption-free versus caption-augmented second-stage RL.
Significance. If the reported results hold, the paper makes a practical contribution by demonstrating a simple, reproducible training recipe that improves a 3B multimodal model's mathematical and general reasoning on external benchmarks, and by releasing the model checkpoint. The proposed curriculum is clearly described, and the explicit acknowledgment of caption quality as a limitation is a good-faith step. The decontamination effort is also a positive, if incomplete, step. However, the central causal claim that the phased ordering, rather than additional RL compute, drives the gains is not yet established; the missing matched-compute control and the un-audited decontamination are the main barriers to accepting the contribution at face value.
major comments (4)
- [Section 5.3, Table 1] The monotonic improvement from Infi-MMR_FRA (40.8/23.91/19.33/62.9) to Infi-MMR_CMRA (42.84/26.34/19.33/63.5) to Infi-MMR-3B (43.68/27.04/21.33/67.2) is presented as evidence for the efficacy of the phased approach, but Infi-MMR-3B is the only model that receives both Phase 2 and Phase 3 training on ViRL39k, so it has strictly more RL updates on that dataset than the intermediate checkpoints. Without a matched-compute control (for example, a second caption-free pass, or a second caption-augmented pass, after FRA for the same total number of steps), the reported gains could be explained by additional training steps rather than by the curriculum ordering or by the caption-free transition. Table 2 compares single-phase continuations after FRA but does not compare either CapFre or CMRA against Infi-MMR-3B at the same total post-FRA budget, so it cannot resolve this confound.
- [Section 5.1, Decontamination] The two-stage decontamination uses a 32-gram exact match after removing numerical information and an embedding-similarity filter with a hand-set threshold of 0.95 using gme-Qwen2-VL-2B-Instruct. The paper does not report how many samples were removed, what fraction of the evaluation sets were flagged as near-duplicates, or any sensitivity analysis of the 0.95 threshold. Without a leaked-sample audit (for example, inspecting the nearest neighbors or showing that model accuracy on the high-similarity subset is not elevated), the reported benchmark numbers and the phase-comparison effects could be inflated by memorization of benchmark-specific content. The authors should provide such an audit or otherwise demonstrate that the filter is effective.
- [Section 5.1, Implementation Details] The manuscript reports learning rate, batch size, and rollout count for each phase but never reports the number of optimization steps or epochs per phase, nor the total wall-clock GPU time. These numbers are necessary to assess whether the three phases are comparable in scale and to construct the matched-compute baseline requested above; without them, the reader cannot determine whether the improvement from FRA to CMRA to MRE is driven by training duration rather than by the data curriculum.
- [Abstract and Section 5.3] The 'state-of-the-art' claim is made relative to a small hand-picked comparison set; within Table 1, FAST-3B, a same-base model, is only 0.68 points behind on MathVerse, 0.24 points behind on MathVision, and 1.0 point behind on MathVista. If the authors intend the claim broadly, they should either compare against all relevant models reported at submission time or restrict the claim to the specific baselines in Table 1.
minor comments (5)
- [Section 3 heading] The heading 'Priliminary' is a typo for 'Preliminary'.
- [Table 2 caption] The caption refers to 'Infi-MMR_FRP'; this should be 'Infi-MMR_FRA'.
- [Section 5.3] The statement that 'all Infi-MMR series models demonstrate distinct improvements in reasoning strength' is inconsistent with Table 1, where Infi-MMR_FRA on MathVista (62.9) is below Qwen2.5-VL-3B (63.4).
- [Section 5.4.1 and Figure 3] The analysis of average response token length does not specify how tokens are counted across failed versus successful rollouts or whether the trend is averaged over the group of 16 rollouts; please clarify.
- [Section 6] The limitation about caption quality is good, but it could be moved to the experiments section and accompanied by a human evaluation or a caption-quality metric to make the claimed bridge effect more concrete.
Circularity Check
No significant circularity: benchmark claims are evaluated on external tests; the sole self-citation (Omnicaptioner) is non-load-bearing tool use.
full rationale
The paper's central claim—that the three-phase curriculum (text-only RL, caption-augmented multimodal RL, caption-free multimodal RL) unlocks multimodal reasoning in a 3B MSLM—is supported by external benchmarks (MATH500, MathVerse, MathVision, OlympiadBench, MathVista) that are not used to set any reward weight. The reward function (Eq. 3) has hand-set coefficients wf=0.1 and wa=0.9; no parameter in the derivation is fitted to a benchmark value, and no equation reduces to a training-set statistic. The comparisons in Tables 1-2 are empirical ablations over training-data modality and captioning, not constructions: Infi-MMR_FRA, Infi-MMR_CMRA and Infi-MMR-3B are different checkpoints reported on held-out evals. The only self-citation with operational impact is Omnicaptioner [20], used to generate captions in Phase 2 (Sec. 4.2); this is a tool invocation, not an argument whose conclusion equals its premise, and the paper even reports a caption-free alternative (Infi-MMR_CapFre) and flags caption quality as a limitation (Sec. 6). A compute-matched confound between phase order and total RL updates is a validity concern, but it concerns causal attribution, not definitional circularity; the benchmark numbers are not obtained by renaming or re-fitting their inputs. No circular step is exhibited.
Assumptions & free parameters
free parameters (4)
- Reward weights wf and wa =
wf=0.1, wa=0.9
- Math reward weights wt and wp =
wt=0.2, wp=0.8
- Training hyperparameters =
lr=1e-6, batch=256, rollout batch=256, G=16
- Decontamination similarity threshold =
0.95
assumptions (4)
- domain assumption GRPO with a rule-based reward correctly optimizes reasoning behavior in a 3B multimodal policy.
- domain assumption Text-only reinforcement learning on DeepScaleR transfers to multimodal reasoning.
- domain assumption Omnicaptioner-generated captions are accurate and serve as an effective bridge.
- domain assumption Decontamination removes all test overlap.
Cite this review
Pith. "Pith review of Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models." pith.science (2026). https://pith.science/paper/TTO4PV7L
@misc{pith2026250523091,
author = {Pith},
title = {Pith review of: Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/TTO4PV7L}},
note = {Machine review of arXiv:2505.23091}
}
read the original abstract
Recent advancements in large language models (LLMs) have demonstrated substantial progress in reasoning capabilities, such as DeepSeek-R1, which leverages rule-based reinforcement learning to enhance logical reasoning significantly. However, extending these achievements to multimodal large language models (MLLMs) presents critical challenges, which are frequently more pronounced for Multimodal Small Language Models (MSLMs) given their typically weaker foundational reasoning abilities: (1) the scarcity of high-quality multimodal reasoning datasets, (2) the degradation of reasoning capabilities due to the integration of visual processing, and (3) the risk that direct application of reinforcement learning may produce complex yet incorrect reasoning processes. To address these challenges, we design a novel framework Infi-MMR to systematically unlock the reasoning potential of MSLMs through a curriculum of three carefully structured phases and propose our multimodal reasoning model Infi-MMR-3B. The first phase, Foundational Reasoning Activation, leverages high-quality textual reasoning datasets to activate and strengthen the model's logical reasoning capabilities. The second phase, Cross-Modal Reasoning Adaptation, utilizes caption-augmented multimodal data to facilitate the progressive transfer of reasoning skills to multimodal contexts. The third phase, Multimodal Reasoning Enhancement, employs curated, caption-free multimodal data to mitigate linguistic biases and promote robust cross-modal reasoning. Infi-MMR-3B achieves both state-of-the-art multimodal math reasoning ability (43.68% on MathVerse testmini, 27.04% on MathVision test, and 21.33% on OlympiadBench) and general reasoning ability (67.2% on MathVista testmini). Resources are available at https://huggingface.co/Reallm-Labs/Infi-MMR-3B.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Guo, D., D. Yang, H. Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[2]
Yang, A., B. Yang, B. Hui, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024
arXiv 2024
- [3]
-
[4]
Lai, X., Z. Tian, Y . Chen, et al. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024
arXiv 2024
-
[5]
Lightman, H., V . Kosaraju, Y . Burda, et al. Let’s verify step by step. InThe TwelfthInternational Conference on Learning Representations. 2023
work page 2023
-
[6]
Yao, S., D. Yu, J. Zhao, et al. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023
work page 2023
-
[7]
Cheng, Z., Q. Chen, J. Zhang, et al. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pages 23678–23686. 2025
work page 2025
-
[8]
Xiang, K., Z. Liu, Z. Jiang, et al. Atomthink: A slow thinking framework for multimodal mathematical reasoning. arXiv preprint arXiv:2411.11930, 2024
arXiv 2024
Show all 35 references
-
[9]
Huang, W., B. Jia, Z. Zhai, et al. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025
2025 arXiv
-
[10]
Donahue, P
Alayrac, J.-B., J. Donahue, P. Luc, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022
2022
-
[11]
Liu, H., C. Li, Q. Wu, et al. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023
2023
-
[12]
Zhang, Y
Zhang, R., B. Zhang, Y . Li, et al. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024
2024 arXiv
-
[13]
Ouyang, L., J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022
2022
-
[14]
Yang, Y ., X. He, H. Pan, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025
2025 arXiv
-
[15]
Zhang, G
Liu, Q., S. Zhang, G. Qin, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprint arXiv:2505.03981, 2025
2025 arXiv
-
[16]
Zhang, M
Peng, Y ., G. Zhang, M. Zhang, et al. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025
2025 arXiv
-
[17]
Wang, X., Y . Chen, W. Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021
2021
-
[18]
Platanios, E. A., O. Stretcu, G. Neubig, et al. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848, 2019
1903 arXiv
-
[19]
Wolski, P
Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 11
2017 arXiv
-
[20]
Lu, Y ., J. Yuan, Z. Li, et al. Omnicaptioner: One captioner to rule them all. arXiv preprint arXiv:2504.07089, 2025
2025 arXiv
-
[21]
Bai, S., K. Chen, X. Liu, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025
2025 arXiv
-
[22]
Luo, M., S. Tan, J. Wong, et al. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog
2025
-
[23]
Wang, H., C. Qu, Z. Huang, et al. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025
2025 arXiv
-
[24]
Wen, L., Y . Cai, F. Xiao, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460, 2025
2025 arXiv
-
[25]
Zhang, W
Zhang, X., Y . Zhang, W. Xie, et al. Gme: Improving universal multimodal retrieval by multimodal llms, 2024
2024
-
[26]
Jiang, Y
Zhang, R., D. Jiang, Y . Zhang, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024
2024
-
[27]
Wang, K., J. Pan, W. Shi, et al. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024
2024
-
[28]
He, C., R. Luo, Y . Bai, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), pages 3828...
2024
-
[29]
Bansal, T
Lu, P., H. Bansal, T. Xia, et al. Mathvista: Evaluating mathematical reasoning of foundation mod- els in visual contexts. In The Twelfth International Conference on Learning Representations
-
[30]
Lerer, A
Hurst, A., A. Lerer, A. P. Goucher, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[31]
Wang, P., S. Bai, S. Tan, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024
2024 arXiv
-
[32]
Huang, W
Yao, H., J. Huang, W. Wu, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024
2024 arXiv
-
[33]
Chen, Z., W. Wang, Y . Cao, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024
2024 arXiv
-
[34]
Meng, F., L. Du, Z. Liu, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025
2025 arXiv
-
[35]
Xiao, W., L. Gan, W. Dai, et al. Fast-slow thinking for large vision-language model reasoning. arXiv preprint arXiv:2504.18458, 2025. 12 A Examples of Generated Caption Figure 4: Generated Caption With aigc Type. Figure 5: Generated Caption With mathgeo Type. Figure 6: Generat...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.