Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A three-phase curriculum—text problems first, captions as a bridge, pure visual questions last—unlocks state-of-the-art multimodal math reasoning in a 3-billion-parameter model.

desk verdict A plausible three-phase RL curriculum for 3B multimodal reasoning, but the central claim needs a matched-compute control before it can be believed. read the letter →

arxiv 2505.23091 v3 pith:TTO4PV7L submitted 2025-05-29 cs.AI cs.CL

classification cs.AIcs.CL
keywords multimodalreasoningreinforcementlearningcurriculumrule-basedRLsmalllanguagemodelsmathematicalGRPOvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that small multimodal language models fail at reasoning not because they are small, but because their training mixes vision and language too early. It proposes a curriculum of three rule-based reinforcement learning phases: first, text-only math problems to rebuild a strong reasoning base; second, multimodal problems augmented with image captions to transfer that base to vision; third, caption-free multimodal problems to remove reliance on text. The resulting 3-billion-parameter model, Infi-MMR-3B, reports state-of-the-art scores on MathVerse (43.68%), MathVision (27.04%), OlympiadBench (21.33%), and MathVista (67.2%), beating models with more parameters. A sympathetic reader would take this as evidence that the training recipe, not model scale, is what unlocks multimodal reasoning.

What carries the argument

The central mechanism is the three-phase curriculum itself, driven by Group Relative Policy Optimization (GRPO)—a reinforcement learning update that scores a group of sampled responses by relative advantage, removing the need for a critic model. The reward function is staged: a format reward checks for the <think>...</think> reasoning structure and a final answer, and only when the format is valid does an accuracy reward compare the extracted answer against ground truth via mathematical verification, string matching, or multiple-choice matching. The bridge object is the caption-augmented dataset: for each image category, a caption generator produces a concise description that is prepended to the question during Phase 2, letting text-trained reasoning anchor to visual content before captions are withdrawn in Phase 3. The curriculum's effect is shown by the training dynamics: text-first training initially lengthens responses, then stabilizes them, while vision-first training keeps generating longer but less accurate answers. That difference is the paper's evidence that text-only RL activates reasoning in a way that later transfers across modalities.

What would settle it

Search every test question in MathVerse, MathVision, OlympiadBench, and MathVista against the training sets using a stricter similarity cut-off (say 0.80) and public near-duplicate lists; if Infi-MMR-3B's scores drop materially, the curriculum's reported gains are inflated by leaked answers. An independent fresh multimodal math test set would settle the claim more directly.

Watch

Extended reading notes

Core claim

The core claim is that the order in which a multimodal small language model meets reasoning data determines whether it learns to reason across modalities. Starting directly with multimodal reinforcement learning produces long, unstable, and often wrong reasoning traces; starting with text-only rule-based RL builds a stable reasoning base that can then be transferred. Caption-augmented multimodal data acts as the transfer bridge, and the final caption-free phase teaches the model to trust the image rather than the caption. The paper demonstrates this with ablations: text-first training outperforms vision-first training at every step, and caption-bridge training beats going straight to caption-free data on reasoning benchmarks, even though caption-free data improves general visual question answering. Infi-MMR-3B's reported scores on four benchmarks are presented as the result of this progression.

Load-bearing premise

The reported scores are valid only if the cleaning step that removes problems nearly identical or highly similar to test questions actually removed every overlap between training and test data; the similarity cut-off is chosen by hand and no leaked examples are shown.

Editorial extensions

If this is right

  • A 3-billion-parameter model trained on this curriculum surpasses larger open-source models and one proprietary model on multimodal math reasoning benchmarks.
  • Text-only rule-based RL is a viable and more stable first stage for multimodal reasoning than starting with visual data; it should be preferred whenever verifiable text reasoning data is available.
  • Caption-augmented data is an effective bridge for transferring reasoning to vision: it outperforms going straight to caption-free multimodal RL on reasoning-heavy benchmarks, even though caption-free data yields better general visual question answering.
  • A staged reward that enforces output format before rewarding answer accuracy guides the model to produce well-structured reasoning and then improves correctness.
  • The curriculum's benefits hold across benchmarks of different difficulty, from diagram-heavy problems (MathVerse) to olympiad-level scientific questions (OlympiadBench).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same text-first -> caption-bridge -> caption-free schedule may transfer to other verifiable reasoning domains, such as STEM diagram problems or time-series chart reasoning, provided rule-based answer checking exists.
  • The response-length signature (rise, then decline, then stabilization under text-first RL) could serve as a practical training monitor: if average response length keeps climbing without accuracy gains, the RL phase is likely producing verbose but unreliable reasoning.
  • The decontamination threshold is a free parameter; a public audit of samples removed at the 0.95 similarity cut-off, and a re-evaluation under a stricter threshold, would directly test whether the reported state-of-the-art numbers are leakage-free.
  • The caption-augmentation stage suggests a general principle for skill transfer across modalities: provide a shared semantic scaffold, then remove it gradually; the same pattern could apply to transferring reasoning from synthetic to real imagery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces Infi-MMR, a three-phase rule-based reinforcement learning curriculum for multimodal small language models (MSLMs). Phase 1 (FRA) trains on text-only mathematical reasoning data (DeepScaleR), Phase 2 (CMRA) trains on caption-augmented multimodal data (ViRL39k with Omnicaptioner-generated captions), and Phase 3 (MRE) continues on caption-free multimodal data (ViRL39k). The authors report that the resulting Infi-MMR-3B, built on Qwen2.5-VL-3B-Instruct, achieves 43.68% on MathVerse testmini, 27.04% on MathVision test, 21.33% on OlympiadBench, and 67.2% on MathVista testmini, and they claim state-of-the-art performance among the compared models. The paper also reports ablations comparing text-first versus multimodal-first initial RL and caption-free versus caption-augmented second-stage RL.

Significance. If the reported results hold, the paper makes a practical contribution by demonstrating a simple, reproducible training recipe that improves a 3B multimodal model's mathematical and general reasoning on external benchmarks, and by releasing the model checkpoint. The proposed curriculum is clearly described, and the explicit acknowledgment of caption quality as a limitation is a good-faith step. The decontamination effort is also a positive, if incomplete, step. However, the central causal claim that the phased ordering, rather than additional RL compute, drives the gains is not yet established; the missing matched-compute control and the un-audited decontamination are the main barriers to accepting the contribution at face value.

major comments (4)
  1. [Section 5.3, Table 1] The monotonic improvement from Infi-MMR_FRA (40.8/23.91/19.33/62.9) to Infi-MMR_CMRA (42.84/26.34/19.33/63.5) to Infi-MMR-3B (43.68/27.04/21.33/67.2) is presented as evidence for the efficacy of the phased approach, but Infi-MMR-3B is the only model that receives both Phase 2 and Phase 3 training on ViRL39k, so it has strictly more RL updates on that dataset than the intermediate checkpoints. Without a matched-compute control (for example, a second caption-free pass, or a second caption-augmented pass, after FRA for the same total number of steps), the reported gains could be explained by additional training steps rather than by the curriculum ordering or by the caption-free transition. Table 2 compares single-phase continuations after FRA but does not compare either CapFre or CMRA against Infi-MMR-3B at the same total post-FRA budget, so it cannot resolve this confound.
  2. [Section 5.1, Decontamination] The two-stage decontamination uses a 32-gram exact match after removing numerical information and an embedding-similarity filter with a hand-set threshold of 0.95 using gme-Qwen2-VL-2B-Instruct. The paper does not report how many samples were removed, what fraction of the evaluation sets were flagged as near-duplicates, or any sensitivity analysis of the 0.95 threshold. Without a leaked-sample audit (for example, inspecting the nearest neighbors or showing that model accuracy on the high-similarity subset is not elevated), the reported benchmark numbers and the phase-comparison effects could be inflated by memorization of benchmark-specific content. The authors should provide such an audit or otherwise demonstrate that the filter is effective.
  3. [Section 5.1, Implementation Details] The manuscript reports learning rate, batch size, and rollout count for each phase but never reports the number of optimization steps or epochs per phase, nor the total wall-clock GPU time. These numbers are necessary to assess whether the three phases are comparable in scale and to construct the matched-compute baseline requested above; without them, the reader cannot determine whether the improvement from FRA to CMRA to MRE is driven by training duration rather than by the data curriculum.
  4. [Abstract and Section 5.3] The 'state-of-the-art' claim is made relative to a small hand-picked comparison set; within Table 1, FAST-3B, a same-base model, is only 0.68 points behind on MathVerse, 0.24 points behind on MathVision, and 1.0 point behind on MathVista. If the authors intend the claim broadly, they should either compare against all relevant models reported at submission time or restrict the claim to the specific baselines in Table 1.
minor comments (5)
  1. [Section 3 heading] The heading 'Priliminary' is a typo for 'Preliminary'.
  2. [Table 2 caption] The caption refers to 'Infi-MMR_FRP'; this should be 'Infi-MMR_FRA'.
  3. [Section 5.3] The statement that 'all Infi-MMR series models demonstrate distinct improvements in reasoning strength' is inconsistent with Table 1, where Infi-MMR_FRA on MathVista (62.9) is below Qwen2.5-VL-3B (63.4).
  4. [Section 5.4.1 and Figure 3] The analysis of average response token length does not specify how tokens are counted across failed versus successful rollouts or whether the trend is averaged over the group of 16 rollouts; please clarify.
  5. [Section 6] The limitation about caption quality is good, but it could be moved to the experiments section and accompanied by a human evaluation or a caption-quality metric to make the claimed bridge effect more concrete.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: benchmark claims are evaluated on external tests; the sole self-citation (Omnicaptioner) is non-load-bearing tool use.

full rationale

The paper's central claim—that the three-phase curriculum (text-only RL, caption-augmented multimodal RL, caption-free multimodal RL) unlocks multimodal reasoning in a 3B MSLM—is supported by external benchmarks (MATH500, MathVerse, MathVision, OlympiadBench, MathVista) that are not used to set any reward weight. The reward function (Eq. 3) has hand-set coefficients wf=0.1 and wa=0.9; no parameter in the derivation is fitted to a benchmark value, and no equation reduces to a training-set statistic. The comparisons in Tables 1-2 are empirical ablations over training-data modality and captioning, not constructions: Infi-MMR_FRA, Infi-MMR_CMRA and Infi-MMR-3B are different checkpoints reported on held-out evals. The only self-citation with operational impact is Omnicaptioner [20], used to generate captions in Phase 2 (Sec. 4.2); this is a tool invocation, not an argument whose conclusion equals its premise, and the paper even reports a caption-free alternative (Infi-MMR_CapFre) and flags caption quality as a limitation (Sec. 6). A compute-matched confound between phase order and total RL updates is a validity concern, but it concerns causal attribution, not definitional circularity; the benchmark numbers are not obtained by renaming or re-fitting their inputs. No circular step is exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The reported benchmark improvements rest on hand-set hyperparameters, a hand-set decontamination threshold, and several domain assumptions about RL transfer and caption quality. The paper introduces no new physical entities; Infi-MMR-3B is a trained model, not a newly postulated object.

free parameters (4)
  • Reward weights wf and wa = wf=0.1, wa=0.9
    Hand-set weights in R_total = wf*R_format + wa*R_acc; affect the trade-off between format compliance and answer accuracy.
  • Math reward weights wt and wp = wt=0.2, wp=0.8
    Hand-set weights for type matching and exact parameter matching in the math and choice rewards.
  • Training hyperparameters = lr=1e-6, batch=256, rollout batch=256, G=16
    Hand-set; control optimization stability and exploration in GRPO; not swept or reported with sensitivity.
  • Decontamination similarity threshold = 0.95
    Embedding similarity cutoff above which samples are removed as duplicates; central to the no-leakage assumption.
assumptions (4)
  • domain assumption GRPO with a rule-based reward correctly optimizes reasoning behavior in a 3B multimodal policy.
    The paper adopts Eq. (2) from DeepSeek-R1 without proof in this setting; if RL destabilizes or overfits the reward, the benchmark gains would not transfer.
  • domain assumption Text-only reinforcement learning on DeepScaleR transfers to multimodal reasoning.
    Phase 1 (FRA) is justified by this transfer assumption, supported only by the ablation in Figure 3.
  • domain assumption Omnicaptioner-generated captions are accurate and serve as an effective bridge.
    Phase 2 depends on caption quality; the authors state in Section 6 that caption quality was not a primary research focus and its precise impact warrants further investigation.
  • domain assumption Decontamination removes all test overlap.
    The paper relies on 32-gram matching and a 0.95 embedding threshold in Section 5.1 without auditing remaining near-duplicates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models." pith.science (2026). https://pith.science/paper/TTO4PV7L

@misc{pith2026250523091,
  author       = {Pith},
  title        = {Pith review of: Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTO4PV7L}},
  note         = {Machine review of arXiv:2505.23091}
}
read the original abstract

Recent advancements in large language models (LLMs) have demonstrated substantial progress in reasoning capabilities, such as DeepSeek-R1, which leverages rule-based reinforcement learning to enhance logical reasoning significantly. However, extending these achievements to multimodal large language models (MLLMs) presents critical challenges, which are frequently more pronounced for Multimodal Small Language Models (MSLMs) given their typically weaker foundational reasoning abilities: (1) the scarcity of high-quality multimodal reasoning datasets, (2) the degradation of reasoning capabilities due to the integration of visual processing, and (3) the risk that direct application of reinforcement learning may produce complex yet incorrect reasoning processes. To address these challenges, we design a novel framework Infi-MMR to systematically unlock the reasoning potential of MSLMs through a curriculum of three carefully structured phases and propose our multimodal reasoning model Infi-MMR-3B. The first phase, Foundational Reasoning Activation, leverages high-quality textual reasoning datasets to activate and strengthen the model's logical reasoning capabilities. The second phase, Cross-Modal Reasoning Adaptation, utilizes caption-augmented multimodal data to facilitate the progressive transfer of reasoning skills to multimodal contexts. The third phase, Multimodal Reasoning Enhancement, employs curated, caption-free multimodal data to mitigate linguistic biases and promote robust cross-modal reasoning. Infi-MMR-3B achieves both state-of-the-art multimodal math reasoning ability (43.68% on MathVerse testmini, 27.04% on MathVision test, and 21.33% on OlympiadBench) and general reasoning ability (67.2% on MathVista testmini). Resources are available at https://huggingface.co/Reallm-Labs/Infi-MMR-3B.

Figures

Figures reproduced from arXiv: 2505.23091 by the authors.

Figure 1
Figure 1. Utilization of Data Types Across Different Training Stages in the Infi-MMR Framework [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Overall Framework of Infi-MMR. System Prompt for RL Training and Inference You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. • Final Answer Provision: It confirms whether a final answer is explicitly provided by the model, particularly when the instructions associated with the query q requir… view at source ↗
Figure 3
Figure 3. Analysis of Different Modality Data for Initial Training. Text RL and Vision RL represent [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Generated Caption With aigc Type [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Generated Caption With mathgeo Type [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Generated Caption With table Type. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Generated Caption With chart Type. B Instruction for Clarifying Images Instruction for Clarifying Images Classify the image into exactly one of the following categories: For structured data (can be written in Markdown or Latex): mathgeo (Euclidean geometric shapes or m…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 12 canonical work pages

  1. [1]

    Guo, D., D. Yang, H. Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

  2. [2]

    Yang, A., B. Yang, B. Hui, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

  3. [3]

    Kalai, A

    Jaech, A., A. Kalai, A. Lerer, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

  4. [4]

    Lai, X., Z. Tian, Y . Chen, et al. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024

  5. [5]

    Kosaraju, Y

    Lightman, H., V . Kosaraju, Y . Burda, et al. Let’s verify step by step. InThe TwelfthInternational Conference on Learning Representations. 2023

  6. [6]

    Yao, S., D. Yu, J. Zhao, et al. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822, 2023

  7. [7]

    Cheng, Z., Q. Chen, J. Zhang, et al. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pages 23678–23686. 2025

  8. [8]

    Xiang, K., Z. Liu, Z. Jiang, et al. Atomthink: A slow thinking framework for multimodal mathematical reasoning. arXiv preprint arXiv:2411.11930, 2024

Show all 35 references
  1. [9]

    Huang, W., B. Jia, Z. Zhai, et al. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

  2. [10]

    Donahue, P

    Alayrac, J.-B., J. Donahue, P. Luc, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

  3. [11]

    Liu, H., C. Li, Q. Wu, et al. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023

  4. [12]

    Zhang, Y

    Zhang, R., B. Zhang, Y . Li, et al. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024

  5. [13]

    Ouyang, L., J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

  6. [14]

    Yang, Y ., X. He, H. Pan, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

  7. [15]

    Zhang, G

    Liu, Q., S. Zhang, G. Qin, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprint arXiv:2505.03981, 2025

  8. [16]

    Zhang, M

    Peng, Y ., G. Zhang, M. Zhang, et al. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025

  9. [17]

    Wang, X., Y . Chen, W. Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576, 2021

  10. [18]

    Platanios, E. A., O. Stretcu, G. Neubig, et al. Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848, 2019

  11. [19]

    Wolski, P

    Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 11

  12. [20]

    Lu, Y ., J. Yuan, Z. Li, et al. Omnicaptioner: One captioner to rule them all. arXiv preprint arXiv:2504.07089, 2025

  13. [21]

    Bai, S., K. Chen, X. Liu, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

  14. [22]

    Luo, M., S. Tan, J. Wong, et al. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

  15. [23]

    Wang, H., C. Qu, Z. Huang, et al. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025

  16. [24]

    Wen, L., Y . Cai, F. Xiao, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460, 2025

  17. [25]

    Zhang, W

    Zhang, X., Y . Zhang, W. Xie, et al. Gme: Improving universal multimodal retrieval by multimodal llms, 2024

  18. [26]

    Jiang, Y

    Zhang, R., D. Jiang, Y . Zhang, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024

  19. [27]

    Wang, K., J. Pan, W. Shi, et al. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024

  20. [28]

    He, C., R. Luo, Y . Bai, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), pages 3828...

  21. [29]

    Bansal, T

    Lu, P., H. Bansal, T. Xia, et al. Mathvista: Evaluating mathematical reasoning of foundation mod- els in visual contexts. In The Twelfth International Conference on Learning Representations

  22. [30]

    Lerer, A

    Hurst, A., A. Lerer, A. P. Goucher, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

  23. [31]

    Wang, P., S. Bai, S. Tan, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

  24. [32]

    Huang, W

    Yao, H., J. Huang, W. Wu, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

  25. [33]

    Chen, Z., W. Wang, Y . Cao, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

  26. [34]

    Meng, F., L. Du, Z. Liu, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

  27. [35]

    Xiao, W., L. Gan, W. Dai, et al. Fast-slow thinking for large vision-language model reasoning. arXiv preprint arXiv:2504.18458, 2025. 12 A Examples of Generated Caption Figure 4: Generated Caption With aigc Type. Figure 5: Generated Caption With mathgeo Type. Figure 6: Generat...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.