Pith. sign in

REVIEW 4 major objections 5 minor 81 references

The MIND framework trains multimodal models to reason from many correct and deliberately wrong rationales, and reports state-of-the-art accuracy on three visual question answering benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 18:20 UTC pith:W6RXDRV2

load-bearing objection The headline SOTA numbers mostly reflect infrastructure changes, but the matched-baseline ablations show a real 2–5 point gain from training on multiple positive/negative rationales. the 4 major comments →

arxiv 2512.05530 v2 pith:W6RXDRV2 submitted 2025-12-05 cs.AI

MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models

classification cs.AI
keywords multi-rationale reasoningchain-of-thoughtrationale augmentationcontrastive alignmentself-correctionvisual question answeringScienceQAA-OKVQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that a multimodal language model can reason better when it is trained not on a single chain of thought but on many diverse correct rationales plus deliberately wrong ones, and when it is explicitly taught to spot and fix the wrong ones. To that end it introduces MIND, a two-stage training scheme: first learn the common logic across many positively rephrased rationales; then, given either a positive or a negative rationale, produce the correct answer together with a corrected rationale. A contrastive loss with hard-example mining pulls the embeddings of correct rationales together and pushes incorrect ones apart. The paper reports state-of-the-art accuracies on three visual question answering benchmarks — 92.29% on ScienceQA, 70.6% on A-OKVQA, and 57.38% (base) / 61.56% (large) on M3CoT — with the P2CL and MCA components showing a combined '1+1>2' gain. A sympathetic reader would care because this is a step from learning to imitate a single reasoning path toward learning to discriminate and self-correct, which is closer to how humans actually reason.

Core claim

On its own terms, the central claim is that a multi-rationale semantic space, built by RAD, can be shaped by P2CL and MCA so that the model learns both semantic consistency of correct reasoning and the boundary that separates it from incorrect reasoning. The training signal is: in Phase I, randomly sample from a pool of positive rationales and optimize a maximum-likelihood generation loss; in Phase II, take a positive or negative rationale as input and supervise the output [answer + corrected positive rationale]. MCA regularizes the same embeddings with a margin-based contrastive loss over hard positives and hard negatives. The reported result is that this combination outperforms single-rati

What carries the argument

The load-bearing objects are the three components of MIND. RAD (Rationale Augmentation and Discrimination) is a data-construction step: prompted large models rewrite each original rationale into several semantically equivalent positive versions and several semantically inverted negative versions, producing 'Multi-Rationale-Pos' and 'Multi-Rationale-Neg' pools. P2CL (Progressive Two-stage Correction Learning) is the training curriculum: Phase I teaches multi-rationale positive learning; Phase II teaches active logic discrimination and correction by conditioning on positive or negative rationales and supervising with the correct answer plus a positive rationale. MCA (Multi-rationale Contrastiv

Load-bearing premise

The load-bearing premise is that the reported accuracy gaps come from MIND's multi-rationale training rather than from the stronger visual features, richer captions, and 200–400 training epochs used in the MIND runs; the paper's own Table 4 baseline (90.29) is already 4.98 points above the single-rationale baseline it cites (85.31), so the comparison does not isolate the new components.

What would settle it

Run MIND and the single-rationale CoT baseline under identical conditions — same visual encoder, same image captions, same number of training epochs, same rationale scale — and compare. If the multi-rationale components add little or nothing once infrastructure is matched, the central SOTA claim fails; if the gap persists, the multi-rationale discrimination mechanism is the cause.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If MIND works as claimed, a dataset with one gold rationale can be turned into a 500–1000x larger training resource through LLM-based paraphrasing and semantic inversion, so multi-rationale supervision becomes a cheap add-on to existing CoT benchmarks.
  • The two-phase design implies that a model can learn to detect a flawed rationale and then generate a corrected one, which is a path toward verifiable reasoning rather than answer-only mimicry.
  • The reported 1+1>2 effect for P2CL and MCA implies that contrastive alignment is most useful when it regularizes an already-strong generative rationale-learning objective, not as a standalone method.
  • The gains at 223M/738M parameters suggest that small models can close part of the gap with much larger zero-shot MLLMs by learning to discriminate among rationales.
  • The saturation of gains beyond ~1000x rationale expansion suggests a practical ceiling: more rationales help, but with diminishing returns.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the paper does not audit the correctness or quality of the RAD-generated rationales, a key open question is how the method behaves when the 'positive' pool contains noisy or actually wrong rationales; the contrastive hard-mining might be sensitive to label noise in the rationale space.
  • Editorial inference: The reported gains may partly reflect the upgraded infrastructure (stronger visual features, better captions, longer training) rather than the multi-rationale signal itself; the paper does not run MIND under the baseline's exact settings. A cleaner test would control those variables.
  • Editorial inference: The RAD generation recipe is model-agnostic, so the same prompt-driven positive/negative expansion could be applied to other chain-of-thought datasets beyond VQA, e.g., text-only math or commonsense reasoning, potentially giving similar correction ability.
  • Editorial inference: If the 'understand-rethink-correct' curriculum is the real driver, one would expect stronger gains on questions where the baseline's rationale is confidently wrong and the image provides a disambiguating cue; this can be tested by stratifying results by rationale correctness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MIND, a training framework for small multimodal encoder-decoder models (T5-based, 223M/738M) that augments single-rationale supervision with multiple positive and negative rationales. RAD constructs multi-rationale datasets via LLM prompts; P2CL-I trains the model to generate answers and rationales from randomly sampled positive rationales; P2CL-II trains it to judge and correct a given positive or negative rationale; MCA adds a margin-based contrastive loss over hard positive and negative rationale embeddings. The paper reports state-of-the-art accuracy on ScienceQA (92.29%), A-OKVQA (70.6%), and M3CoT (57.38% base / 61.56% large), and ablations attributing gains to P2CL and MCA.

Significance. The core idea of training small MLLMs to learn from multiple positive rationales and to actively correct negative rationales is timely and potentially useful. The loss formulations in Eqs. (2)-(4) and (8) are clear, the ablations are structured, and the authors commit to releasing data and code. If the controlled gains (roughly 2-5 points over a matched baseline) hold up, this is a solid engineering contribution to reasoning training recipes. However, the paper's headline SOTA claims are not currently supported: the comparisons against Multimodal-CoT and other baselines change several infrastructure variables at once, and the paper's own matched baseline already accounts for most of the reported gap. The contribution is therefore a promising method with modest, partly confounded evidence, not an established SOTA result as written.

major comments (4)
  1. [§4.2, Tables 1-3 vs. Table 4/Table S1] The headline improvements over Multimodal-CoT (6.98 points on ScienceQA, 20.0 on A-OKVQA, 12.53 on M3CoT) are confounded with infrastructure changes. Section 4.2 states that MIND uses BLIP2-flan-t5-xxl visual features, Qwen2.5-VL-72B captions, and 200-400 training epochs, whereas the cited Multimodal-CoT rows in Tables 1-3 do not use these settings. The paper's own controlled baseline in Table 4 (and 'Original (21K)' in Table 6) uses the same infrastructure with single-rationale data and already reaches 90.29 on ScienceQA, 4.98 points above the cited 85.31. Similarly, Table S1 shows controlled baselines of 65.85 on A-OKVQA and 52.67 on M3CoT, already 15.25 and 7.82 points above the cited Multimodal-CoT numbers. Thus the reported improvements in the main tables mostly reflect the changed backbone, captions, and training length, not the MIND components. Please report all baselines under ma
  2. [§4.4, Table 4 and Table S1] No variance or error bars are reported, and the '1+1>2' synergy claim is not statistically supported. On ScienceQA the MCA-only gain is 0.07% (90.29 to 90.36), which is within the noise expected from a single run; the combined gain is 2.00% versus a sum of individual gains of 1.93%. On A-OKVQA and M3CoT (Table S1), the combined gains (4.72 and 4.71) are actually smaller than the sums of the individual P2CL and MCA gains (6.46 and 6.69), so superadditivity is not observed outside ScienceQA. Please report multiple seeds with standard deviations and, if the synergy claim is retained, provide a statistical test or a consistent pattern across datasets.
  3. [§3.3-§3.4 and Figure 1] The inference procedure is underspecified relative to training. P2CL-II is trained on rationales drawn from the RAD pools (gold positive and generated negative rationales), but at inference the input to the second stage is the model's own Phase I generated rationale. This train/inference distribution shift, model-generated rationales are likely noisier and more varied than RAD rationales, is not discussed. Please specify the exact inference protocol and, ideally, include model-generated rationales during training or validate that the RAD-trained correction behavior transfers to the model's own outputs.
  4. [§3.2, RAD paradigm] The quality of generated rationales is load-bearing for the method, but there is no human or automated audit of the RAD outputs. The paper states that rationales are filtered and cleaned after generation, but it does not report how many are rejected, whether negative rationales were checked to actually fail to derive the answer, or whether positive rationales preserve the original semantics. The downstream accuracy comparison in Table 5 is an indirect, partly circular proxy for rationale quality because the same model's final accuracy is used to validate the data. Please add a sample-quality evaluation (e.g., human annotation on a random subset, or an automated entailment/consistency check) and report agreement numbers.
minor comments (5)
  1. [Eq. (7)] There is a typo: the second equation should select Top-k from the negative similarities {s_j^-}, not from {s_i^+}. As written both lines use {s_i^+}, which makes the hard-negative mining step undefined.
  2. [§3.4, last paragraph] The sentence ends with 'forming a stable.' which is incomplete. Please complete it.
  3. [Table 3] Minor formatting issues: 'LLaMA-Adaper' should be 'LLaMA-Adapter'; category names are inconsistently spaced (e.g., 'M 3CoT'), and the table could be simplified for readability.
  4. [§4.2] The initialization is described as 'FLAN-Alpaca weights' but the citation [7] is to the FLAN paper. If the Alpaca variant is used, please cite the corresponding source or clarify the checkpoint.
  5. [Throughout] The notation 'MINDbase' versus 'MIND base' is used inconsistently; please unify for clarity.

Circularity Check

0 steps flagged

No significant circularity: MIND is an empirical supervised-training recipe whose reported gains are benchmark comparisons, not derivations from the method's own outputs.

full rationale

The paper contains no equation whose output is reused as its input. RAD constructs training data by prompting external LLMs with positive/negative prompts (Sec. 3.2); P2CL-I and P2CL-II losses (Eqs. 2-4) are standard supervised maximum-likelihood objectives; MCA (Eq. 8) is a contrastive regularizer whose positive/negative sets are labeled by the RAD construction. The ablations are measured against a matched 'Original (21K)' baseline (Table 4, 90.29) and report internal increments (0.07, 1.86, 2.00), so the framework's own contribution is empirically isolated rather than assumed. The headline gains over Multimodal-CoT in Tables 1-3 are confounded by infrastructure changes (BLIP2-flan-t5-xxl features, Qwen2.5-VL-72B captions, 200/400 epochs), but this is an experimental comparison concern, not a circularity: the paper does not define its improvement in terms of the baseline numbers. There are no load-bearing self-citations (the cited Multimodal-CoT and other baselines are external works), no fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. Using downstream accuracy to compare rationale generators (Table 5) is an evaluation choice, not circular reasoning. The core derivation chain is therefore self-contained with respect to circularity, even if the strength of the SOTA claim is open to benchmarking critique.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

No new physical or mathematical entities are introduced. The free parameters are standard training/hyperparameter choices, and the axioms are empirical assumptions about data quality and embedding behavior.

free parameters (5)
  • MCA margin m = 0.2
    Chosen by grid search on ScienceQA (Fig. S1); used across datasets.
  • MCA loss weight α = 1.0
    Chosen by grid search on ScienceQA; used across datasets.
  • Number of sampled rationales in MCA = 5 or 50 (dataset-dependent)
    Fig. S1: best (sampled, Top-k) = (5,1) and (50,20), respectively.
  • Training epochs = 200 (SQA), 400 (A-OKVQA, M3CoT)
    Selected by inspection of Fig. 4; affects all reported numbers.
  • RAD paraphrase range 10%-50%
    Ad hoc prompt parameter controlling positive rationale diversity.
axioms (3)
  • domain assumption LLM-generated positives preserve the gold answer's logic and negatives are semantically wrong without giving away the answer by formatting cues.
    RAD prompt design relies on Qwen/DeepSeek outputs being high-signal; no human quality audit is reported (Sec. 3.2, Tab. 5).
  • domain assumption The linear projection g_phi into a semantic contrastive space preserves rationale identity for cosine similarity.
    Eq. (5)-(8) assume embeddings of paraphrased correct rationales cluster and wrong ones separate; no analysis of embedding space is provided.
  • domain assumption Training on 21M generated rationales does not simply memorize paraphrase templates or answer leakage.
    Tab. 6 shows monotonic gain with scale; the paper does not test transfer to out-of-distribution reasoning tasks.

pith-pipeline@v1.3.0-alltime-deepseek · 30546 in / 10585 out tokens · 107327 ms · 2026-08-03T18:20:42.201795+00:00 · methodology

0 comments
read the original abstract

Recently, multimodal large language models (MLLMs) have been widely applied to reasoning tasks. However, they suffer from limited multi-rationale semantic modeling, insufficient logical robustness, and susceptibility to misleading cues. Therefore, we propose a Multi-rationale INtegrated Discriminative (MIND) reasoning framework, which is designed to endow MLLMs with human-like cognitive abilities of "Understand -> Rethink -> Correct", and achieves a paradigm evolution from passive imitation-based reasoning to active discriminative reasoning. Specifically, we introduce a Rationale Augmentation and Discrimination (RAD) paradigm, which provides a unified and extensible data foundation. Meanwhile, we design a Progressive Two-stage Correction Learning (P2CL) strategy. The first phase enhances multi-rationale positive learning, while the second phase enables active logic discrimination and correction. In addition, to mitigate representation entanglement in the multi-rationale semantic space, we propose a Multi-rationale Contrastive Alignment (MCA) optimization strategy. Extensive experiments show that our MIND achieves SOTA performance on multiple public datasets. Our data and code are available at https://github.com/YuChuang1205/MIND

Figures

Figures reproduced from arXiv: 2512.05530 by Bo Wang, Chuang Yu, Jinmiao Zhao, Mingxuan Zhao, Xiangyu Yue, Xiujun Shu, Yuanhao Feng, Yunpeng Liu.

Figure 1
Figure 1. Figure 1: Illustration of MIND’s “Understand → Rethink → Correct” paradigm. It consists of two phases: Rationale (rea￾soning chain) generation and Answer generation. In Phase I, the model focuses on understanding the essential elements of problem-solving. In Phase II, the model rethinks the generated rationales and corrects erroneous reasoning logic. Chain-of-Thought (MCoT) [74], which incorporates inter￾mediate rea… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MIND reasoning framework. The blue circle denotes the format of the supervision signal. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the RAD paradigm. the above problem. Based on the given context, please make minor edits to “{Solution}” to reverse its mean￾ing and ensure the correct answer cannot be logically de￾rived, while keeping most of the original words and struc￾ture intact. Please output {Repeat number} different so￾lutions. Each output format is “Negative Solution:”. Use “\n\n~~~\n\n” to separate them. The output m… view at source ↗
Figure 4
Figure 4. Figure 4: Performance analysis of Epochs and Caption generation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

81 extracted references · 25 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

  2. [2]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. InCVPR, pages 6077–6086,

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 5, 7

  4. [4]

    1, 3, 5, 12

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che.m 3cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought.arXiv preprint arXiv:2405.16473, 2024. 1, 3, 5, 12

  5. [5]

    See, think, confirm: In- teractive prompting between vision and language mod- els for knowledge-based visual reasoning.arXiv preprint arXiv:2301.05226, 2023

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: In- teractive prompting between vision and language mod- els for knowledge-based visual reasoning.arXiv preprint arXiv:2301.05226, 2023. 2, 5, 6

  6. [6]

    Comt: A novel benchmark for chain of multi-modal thought on large vision- language models

    Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: A novel benchmark for chain of multi-modal thought on large vision- language models. InAAAI, pages 23678–23686, 2025. 1

  7. [7]

    Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 5

  8. [8]

    Instructblip: Towards general-purpose vision- language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023. 7

  9. [9]

    Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. InCVPR, pages 9062–9072, 2025. 1, 2

  10. [10]

    Dy- namic fusion with intra-and inter-modality attention flow for visual question answering

    Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dy- namic fusion with intra-and inter-modality attention flow for visual question answering. InCVPR, pages 6639–6648,

  11. [11]

    Cantor: Inspiring multimodal chain-of-thought of mllm

    Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, et al. Cantor: Inspiring multimodal chain-of-thought of mllm. InProceedings of the ACM International Conference on Multimedia (ACM MM), pages 9096–9105, 2024. 2

  12. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 7, 8

  13. [13]

    Multi- modal latent space learning for chain-of-thought reasoning in language models

    Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi- modal latent space learning for chain-of-thought reasoning in language models. InAAAI, pages 18180–18187, 2024. 1, 2, 5, 6

  14. [14]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.Advances in Neural Informa- tion Processing Systems, 37:139348–139379, 2024

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Osten- dorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.Advances in Neural Informa- tion Processing Systems, 37:139348–139379, 2024. 2

  15. [15]

    Towards a multimodal large language model with pixel-level insight for biomedicine

    Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine. InAAAI, pages 3779–3787, 2025. 1

  16. [16]

    Words over pixels? rethinking vision in multimodal large language mod- els

    Anubhooti Jain, Mayank Vatsa, and Richa Singh. Words over pixels? rethinking vision in multimodal large language mod- els. InProceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 10481–10489, 2024. 1

  17. [17]

    Pythia v0

    Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0. 1: the win- ning entry to the vqa challenge 2018.arXiv preprint arXiv:1807.09956, 2018. 6

  18. [18]

    Webly su- pervised concept expansion for general purpose vision mod- els

    Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly su- pervised concept expansion for general purpose vision mod- els. InECCV, pages 662–681. Springer, 2022. 6

  19. [19]

    Unifiedqa: Crossing format boundaries with a single qa sys- tem.arXiv preprint arXiv:2005.00700, 2020

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabhar- wal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa sys- tem.arXiv preprint arXiv:2005.00700, 2020. 6

  20. [20]

    Bi- linear attention networks.Advances in neural information processing systems, 31, 2018

    Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bi- linear attention networks.Advances in neural information processing systems, 31, 2018. 6

  21. [21]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. InProceedings of the International Conference on Machine Learning (ICML), pages 5583–5594. PMLR, 2021. 6

  22. [22]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026,

  23. [23]

    Gen- erating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506,

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Gen- erating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506,

  24. [24]

    Multi- modal reasoning with multimodal knowledge graph.arXiv preprint arXiv:2406.02030, 2024

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multi- modal reasoning with multimodal knowledge graph.arXiv preprint arXiv:2406.02030, 2024. 2

  25. [25]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML, pages 12888–12900. PMLR, 2022. 8

  26. [26]

    Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models. InICML, pages 19730–19742. PMLR, 2023. 5, 7, 8

  27. [27]

    What does bert with vision look at? In Proceedings of the Association for Computational Linguis- tics (ACL), pages 5265–5275, 2020

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does bert with vision look at? In Proceedings of the Association for Computational Linguis- tics (ACL), pages 5265–5275, 2020. 6

  28. [28]

    V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal mod- els

    Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Xuan- Jing Huang, and Zhongyu Wei. V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal mod- els. InProceedings of the Conference of the North Ameri- can Chapter of the Association for Computational Linguis- tics: Human Language Technologies (NAACL-HLT), pages 3769–3798, 2025. 2

  29. [29]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 7, 8

  30. [30]

    Retrieval-augmented multi- modal chain-of-thoughts reasoning for large language mod- els.arXiv preprint arXiv:2312.01714, 2023

    Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, and Longyue Wang. Retrieval-augmented multi- modal chain-of-thoughts reasoning for large language mod- els.arXiv preprint arXiv:2312.01714, 2023. 2

  31. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024. 5, 6, 7

  32. [32]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019. 6

  33. [33]

    Iconqa: A new benchmark for abstract diagram under- standing and visual language reasoning.arXiv preprint arXiv:2110.13214, 2021

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram under- standing and visual language reasoning.arXiv preprint arXiv:2110.13214, 2021. 6

  34. [34]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

  35. [35]

    Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Pro- cessing Systems, 36:43447–43478, 2023

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Pro- cessing Systems, 36:43447–43478, 2023. 6, 7

  36. [36]

    Cheap and quick: Efficient vision-language instruction tuning for large language mod- els.Advances in Neural Information Processing Systems, 36:29615–29627, 2023

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xi- aoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language mod- els.Advances in Neural Information Processing Systems, 36:29615–29627, 2023. 6

  37. [37]

    Pkrd-cot: A unified chain-of-thought prompting for multi-modal large language models in au- tonomous driving

    Xuewen Luo, Fan Ding, Yinsheng Song, Xiaofeng Zhang, and Junnyong Loo. Pkrd-cot: A unified chain-of-thought prompting for multi-modal large language models in au- tonomous driving. InProceedings of the International Con- ference on Neural Information Processing (ICONIP), pages 62–76. Springer, 2024. 2

  38. [38]

    Krisp: Integrating implicit and sym- bolic knowledge for open-domain knowledge-based vqa

    Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and sym- bolic knowledge for open-domain knowledge-based vqa. In CVPR, pages 14111–14121, 2021. 6

  39. [39]

    Chain of images for intuitively reasoning.arXiv preprint arXiv:2311.09241, 2023

    Fanxu Meng, Haotong Yang, Yiding Wang, and Muhan Zhang. Chain of images for intuitively reasoning.arXiv preprint arXiv:2311.09241, 2023. 2

  40. [40]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. InCVPR, pages 14420–14431, 2024. 2

  41. [41]

    Clip- cap: Clip prefix for image captioning.arXiv preprint arXiv:2111.09734, 2021

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning.arXiv preprint arXiv:2111.09734, 2021. 6

  42. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 7, 8

  43. [43]

    Kosmos-2: Ground- ing multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023. 7

  44. [44]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, pages 8748–8763. PmLR, 2021. 7, 8

  45. [45]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 5

  46. [46]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. InECCV, pages 146–162. Springer, 2022. 3, 5, 12

  47. [47]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180,

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180,

  48. [48]

    Assessing gpt4-v on structured reasoning tasks.arXiv preprint arXiv:2312.11524, 2023

    Mukul Singh, Jos ´e Cambronero, Sumit Gulwani, Vu Le, and Gust Verbruggen. Assessing gpt4-v on structured reasoning tasks.arXiv preprint arXiv:2312.11524, 2023. 2

  49. [49]

    Boosting the power of small multimodal reasoning models to match larger models with self-consistency training

    Cheng Tan, Jingxuan Wei, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Ruifeng Guo, Bihui Yu, and Stan Z Li. Boosting the power of small multimodal reasoning models to match larger models with self-consistency training. InECCV, pages 305–322. Springer, 2024. 1, 2, 5, 6, 7

  50. [50]

    Retrieval meets reasoning: Even high-school textbook knowledge benefits multimodal reasoning.arXiv preprint arXiv:2405.20834, 2024

    Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. Retrieval meets reasoning: Even high-school textbook knowledge benefits multimodal reasoning.arXiv preprint arXiv:2405.20834, 2024. 2, 6

  51. [51]

    Lxmert: Learning cross- modality encoder representations from transformers.arXiv preprint arXiv:1908.07490, 2019

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers.arXiv preprint arXiv:1908.07490, 2019. 6

  52. [52]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, 10 Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 6, 7

  53. [53]

    Ranked from within: Ranking large multimodal models without labels

    Weijie Tu, Weijian Deng, Dylan Campbell, Yu Yao, Jiyang Zheng, Tom Gedeon, and Tongliang Liu. Ranked from within: Ranking large multimodal models without labels. In ICML, 2025. 1

  54. [54]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024. 1

  55. [55]

    T-sciq: Teaching multimodal chain- of-thought reasoning via large language model signals for science question answering

    Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal chain- of-thought reasoning via large language model signals for science question answering. InAAAI, pages 19162–19170,

  56. [56]

    Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024. 7

  57. [57]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. InECCV, pages 58–76. Springer, 2024. 2

  58. [58]

    Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025. 1, 2

  59. [59]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 6

  60. [60]

    Visual chatgpt: Talking, drawing and editing with visual foundation models.arXiv preprint arXiv:2303.04671, 2023

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models.arXiv preprint arXiv:2303.04671, 2023. 7

  61. [61]

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks.Advances in Neural Information Processing Systems, 37:69925–69975, 2024. 1

  62. [62]

    Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large lan- guage models.Advances in Neural Information Processing Systems, 37:90277–90317, 2024

    Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large lan- guage models.Advances in Neural Information Processing Systems, 37:90277–90317, 2024. 2

  63. [63]

    The role of chain-of-thought in complex vision-language reasoning task.arXiv preprint arXiv:2311.09193, 2023

    Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C Gee, and Yixin Nie. The role of chain-of-thought in complex vision-language reasoning task.arXiv preprint arXiv:2311.09193, 2023. 2

  64. [64]

    Det- toolchain: A new prompting paradigm to unleash detection ability of mllm

    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Det- toolchain: A new prompting paradigm to unleash detection ability of mllm. InECCV, pages 164–182. Springer, 2024. 2

  65. [65]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 7, 8

  66. [66]

    Thinking in space: How mul- timodal large language models see, remember, and recall spaces

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. InCVPR, pages 10632–10643, 2025. 1

  67. [67]

    Magic-vqa: Multimodal and grounded inference with com- monsense knowledge for visual question answering

    Shuo Yang, Caren Han, Siwen Luo, and Eduard Hovy. Magic-vqa: Multimodal and grounded inference with com- monsense knowledge for visual question answering. InFind- ings of the Association for Computational Linguistics (Find- ings of ACL), pages 16967–16986, 2025. 1

  68. [68]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yu- mao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. InAAAI, pages 3081–3089, 2022. 6

  69. [69]

    Study- ing and improving reasoning in humans and machines.Com- munications Psychology, 2(1):51, 2024

    Nicolas Yax, Hern ´an Anll´o, and Stefano Palminteri. Study- ing and improving reasoning in humans and machines.Com- munications Psychology, 2(1):51, 2024. 1

  70. [70]

    Idealgpt: Iteratively decomposing vision and lan- guage reasoning via large language models

    Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. Idealgpt: Iteratively decomposing vision and lan- guage reasoning via large language models. InFindings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), pages 11289–11303, 2023. 7

  71. [71]

    Deep modular co-attention networks for visual question an- swering

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question an- swering. InCVPR, pages 6281–6290, 2019. 5, 6

  72. [72]

    Cocot: Con- trastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582, 2024

    Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Con- trastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582, 2024. 2

  73. [73]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023. 6, 7

  74. [74]

    Multimodal chain-of-thought rea- soning in language models.Transactions on Machine Learn- ing Research, 2024, 2024

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought rea- soning in language models.Transactions on Machine Learn- ing Research, 2024, 2024. 1, 2, 5, 6, 7, 12

  75. [75]

    Antgpt: Can large language models help long-term ac- tion anticipation from videos?arXiv preprint arXiv:2307

    Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term ac- tion anticipation from videos?arXiv preprint arXiv:2307. 16368, 2023. 2

  76. [76]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023. 1, 2, 6

  77. [77]

    Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation

    Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In CVPR, pages 13246–13257, 2024. 2 11 MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models Supplemen...

  78. [78]

    Baseline

    More Break-Down Ablation.To more comprehen- sively analyze the independent contributions and synergis- tic gains of each core component, we conduct systematic ablation studies on the P2CL strategy and MCA optimiza- tion strategies across ScienceQA, A-OKVQA, and M 3CoT datasets. The experimental results are shown in Tab. S1. First, compared to the“Baseline...

  79. [79]

    From Fig

    Exploration of Hyperparameters in the MCA opti- mization Strategy.We conduct a systematic analysis under various parameter settings. From Fig. S1, the experimental results with different values ofmandαdemonstrate that the MCA optimization strategy has remarkable stability. In particular, whenm∈[0.1−0.4]andα∈[0.5−5.0], Table S1. Break-Down ablation experim...

  80. [80]

    (B) 6.8 km

    Comparison Between Multimodal-CoT and MIND. To more intuitively demonstrate the superior interpretabil- ity and logical consistency of MIND in multimodal reason- 12 ScienceQA #843 Question: Will these magnets attract or repel each other? Options: Gold Rationale: Will these magnets attract or repel? To find out, look at which poles are closest to each othe...

Showing first 80 references.