REVIEW 4 major objections 5 minor 81 references
The MIND framework trains multimodal models to reason from many correct and deliberately wrong rationales, and reports state-of-the-art accuracy on three visual question answering benchmarks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 18:20 UTC pith:W6RXDRV2
load-bearing objection The headline SOTA numbers mostly reflect infrastructure changes, but the matched-baseline ablations show a real 2–5 point gain from training on multiple positive/negative rationales. the 4 major comments →
MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the central claim is that a multi-rationale semantic space, built by RAD, can be shaped by P2CL and MCA so that the model learns both semantic consistency of correct reasoning and the boundary that separates it from incorrect reasoning. The training signal is: in Phase I, randomly sample from a pool of positive rationales and optimize a maximum-likelihood generation loss; in Phase II, take a positive or negative rationale as input and supervise the output [answer + corrected positive rationale]. MCA regularizes the same embeddings with a margin-based contrastive loss over hard positives and hard negatives. The reported result is that this combination outperforms single-rati
What carries the argument
The load-bearing objects are the three components of MIND. RAD (Rationale Augmentation and Discrimination) is a data-construction step: prompted large models rewrite each original rationale into several semantically equivalent positive versions and several semantically inverted negative versions, producing 'Multi-Rationale-Pos' and 'Multi-Rationale-Neg' pools. P2CL (Progressive Two-stage Correction Learning) is the training curriculum: Phase I teaches multi-rationale positive learning; Phase II teaches active logic discrimination and correction by conditioning on positive or negative rationales and supervising with the correct answer plus a positive rationale. MCA (Multi-rationale Contrastiv
Load-bearing premise
The load-bearing premise is that the reported accuracy gaps come from MIND's multi-rationale training rather than from the stronger visual features, richer captions, and 200–400 training epochs used in the MIND runs; the paper's own Table 4 baseline (90.29) is already 4.98 points above the single-rationale baseline it cites (85.31), so the comparison does not isolate the new components.
What would settle it
Run MIND and the single-rationale CoT baseline under identical conditions — same visual encoder, same image captions, same number of training epochs, same rationale scale — and compare. If the multi-rationale components add little or nothing once infrastructure is matched, the central SOTA claim fails; if the gap persists, the multi-rationale discrimination mechanism is the cause.
If this is right
- If MIND works as claimed, a dataset with one gold rationale can be turned into a 500–1000x larger training resource through LLM-based paraphrasing and semantic inversion, so multi-rationale supervision becomes a cheap add-on to existing CoT benchmarks.
- The two-phase design implies that a model can learn to detect a flawed rationale and then generate a corrected one, which is a path toward verifiable reasoning rather than answer-only mimicry.
- The reported 1+1>2 effect for P2CL and MCA implies that contrastive alignment is most useful when it regularizes an already-strong generative rationale-learning objective, not as a standalone method.
- The gains at 223M/738M parameters suggest that small models can close part of the gap with much larger zero-shot MLLMs by learning to discriminate among rationales.
- The saturation of gains beyond ~1000x rationale expansion suggests a practical ceiling: more rationales help, but with diminishing returns.
Where Pith is reading between the lines
- Editorial inference: Because the paper does not audit the correctness or quality of the RAD-generated rationales, a key open question is how the method behaves when the 'positive' pool contains noisy or actually wrong rationales; the contrastive hard-mining might be sensitive to label noise in the rationale space.
- Editorial inference: The reported gains may partly reflect the upgraded infrastructure (stronger visual features, better captions, longer training) rather than the multi-rationale signal itself; the paper does not run MIND under the baseline's exact settings. A cleaner test would control those variables.
- Editorial inference: The RAD generation recipe is model-agnostic, so the same prompt-driven positive/negative expansion could be applied to other chain-of-thought datasets beyond VQA, e.g., text-only math or commonsense reasoning, potentially giving similar correction ability.
- Editorial inference: If the 'understand-rethink-correct' curriculum is the real driver, one would expect stronger gains on questions where the baseline's rationale is confidently wrong and the image provides a disambiguating cue; this can be tested by stratifying results by rationale correctness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MIND, a training framework for small multimodal encoder-decoder models (T5-based, 223M/738M) that augments single-rationale supervision with multiple positive and negative rationales. RAD constructs multi-rationale datasets via LLM prompts; P2CL-I trains the model to generate answers and rationales from randomly sampled positive rationales; P2CL-II trains it to judge and correct a given positive or negative rationale; MCA adds a margin-based contrastive loss over hard positive and negative rationale embeddings. The paper reports state-of-the-art accuracy on ScienceQA (92.29%), A-OKVQA (70.6%), and M3CoT (57.38% base / 61.56% large), and ablations attributing gains to P2CL and MCA.
Significance. The core idea of training small MLLMs to learn from multiple positive rationales and to actively correct negative rationales is timely and potentially useful. The loss formulations in Eqs. (2)-(4) and (8) are clear, the ablations are structured, and the authors commit to releasing data and code. If the controlled gains (roughly 2-5 points over a matched baseline) hold up, this is a solid engineering contribution to reasoning training recipes. However, the paper's headline SOTA claims are not currently supported: the comparisons against Multimodal-CoT and other baselines change several infrastructure variables at once, and the paper's own matched baseline already accounts for most of the reported gap. The contribution is therefore a promising method with modest, partly confounded evidence, not an established SOTA result as written.
major comments (4)
- [§4.2, Tables 1-3 vs. Table 4/Table S1] The headline improvements over Multimodal-CoT (6.98 points on ScienceQA, 20.0 on A-OKVQA, 12.53 on M3CoT) are confounded with infrastructure changes. Section 4.2 states that MIND uses BLIP2-flan-t5-xxl visual features, Qwen2.5-VL-72B captions, and 200-400 training epochs, whereas the cited Multimodal-CoT rows in Tables 1-3 do not use these settings. The paper's own controlled baseline in Table 4 (and 'Original (21K)' in Table 6) uses the same infrastructure with single-rationale data and already reaches 90.29 on ScienceQA, 4.98 points above the cited 85.31. Similarly, Table S1 shows controlled baselines of 65.85 on A-OKVQA and 52.67 on M3CoT, already 15.25 and 7.82 points above the cited Multimodal-CoT numbers. Thus the reported improvements in the main tables mostly reflect the changed backbone, captions, and training length, not the MIND components. Please report all baselines under ma
- [§4.4, Table 4 and Table S1] No variance or error bars are reported, and the '1+1>2' synergy claim is not statistically supported. On ScienceQA the MCA-only gain is 0.07% (90.29 to 90.36), which is within the noise expected from a single run; the combined gain is 2.00% versus a sum of individual gains of 1.93%. On A-OKVQA and M3CoT (Table S1), the combined gains (4.72 and 4.71) are actually smaller than the sums of the individual P2CL and MCA gains (6.46 and 6.69), so superadditivity is not observed outside ScienceQA. Please report multiple seeds with standard deviations and, if the synergy claim is retained, provide a statistical test or a consistent pattern across datasets.
- [§3.3-§3.4 and Figure 1] The inference procedure is underspecified relative to training. P2CL-II is trained on rationales drawn from the RAD pools (gold positive and generated negative rationales), but at inference the input to the second stage is the model's own Phase I generated rationale. This train/inference distribution shift, model-generated rationales are likely noisier and more varied than RAD rationales, is not discussed. Please specify the exact inference protocol and, ideally, include model-generated rationales during training or validate that the RAD-trained correction behavior transfers to the model's own outputs.
- [§3.2, RAD paradigm] The quality of generated rationales is load-bearing for the method, but there is no human or automated audit of the RAD outputs. The paper states that rationales are filtered and cleaned after generation, but it does not report how many are rejected, whether negative rationales were checked to actually fail to derive the answer, or whether positive rationales preserve the original semantics. The downstream accuracy comparison in Table 5 is an indirect, partly circular proxy for rationale quality because the same model's final accuracy is used to validate the data. Please add a sample-quality evaluation (e.g., human annotation on a random subset, or an automated entailment/consistency check) and report agreement numbers.
minor comments (5)
- [Eq. (7)] There is a typo: the second equation should select Top-k from the negative similarities {s_j^-}, not from {s_i^+}. As written both lines use {s_i^+}, which makes the hard-negative mining step undefined.
- [§3.4, last paragraph] The sentence ends with 'forming a stable.' which is incomplete. Please complete it.
- [Table 3] Minor formatting issues: 'LLaMA-Adaper' should be 'LLaMA-Adapter'; category names are inconsistently spaced (e.g., 'M 3CoT'), and the table could be simplified for readability.
- [§4.2] The initialization is described as 'FLAN-Alpaca weights' but the citation [7] is to the FLAN paper. If the Alpaca variant is used, please cite the corresponding source or clarify the checkpoint.
- [Throughout] The notation 'MINDbase' versus 'MIND base' is used inconsistently; please unify for clarity.
Circularity Check
No significant circularity: MIND is an empirical supervised-training recipe whose reported gains are benchmark comparisons, not derivations from the method's own outputs.
full rationale
The paper contains no equation whose output is reused as its input. RAD constructs training data by prompting external LLMs with positive/negative prompts (Sec. 3.2); P2CL-I and P2CL-II losses (Eqs. 2-4) are standard supervised maximum-likelihood objectives; MCA (Eq. 8) is a contrastive regularizer whose positive/negative sets are labeled by the RAD construction. The ablations are measured against a matched 'Original (21K)' baseline (Table 4, 90.29) and report internal increments (0.07, 1.86, 2.00), so the framework's own contribution is empirically isolated rather than assumed. The headline gains over Multimodal-CoT in Tables 1-3 are confounded by infrastructure changes (BLIP2-flan-t5-xxl features, Qwen2.5-VL-72B captions, 200/400 epochs), but this is an experimental comparison concern, not a circularity: the paper does not define its improvement in terms of the baseline numbers. There are no load-bearing self-citations (the cited Multimodal-CoT and other baselines are external works), no fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via citation. Using downstream accuracy to compare rationale generators (Table 5) is an evaluation choice, not circular reasoning. The core derivation chain is therefore self-contained with respect to circularity, even if the strength of the SOTA claim is open to benchmarking critique.
Axiom & Free-Parameter Ledger
free parameters (5)
- MCA margin m =
0.2
- MCA loss weight α =
1.0
- Number of sampled rationales in MCA =
5 or 50 (dataset-dependent)
- Training epochs =
200 (SQA), 400 (A-OKVQA, M3CoT)
- RAD paraphrase range 10%-50%
axioms (3)
- domain assumption LLM-generated positives preserve the gold answer's logic and negatives are semantically wrong without giving away the answer by formatting cues.
- domain assumption The linear projection g_phi into a semantic contrastive space preserves rationale identity for cosine similarity.
- domain assumption Training on 21M generated rationales does not simply memorize paraphrase templates or answer leakage.
read the original abstract
Recently, multimodal large language models (MLLMs) have been widely applied to reasoning tasks. However, they suffer from limited multi-rationale semantic modeling, insufficient logical robustness, and susceptibility to misleading cues. Therefore, we propose a Multi-rationale INtegrated Discriminative (MIND) reasoning framework, which is designed to endow MLLMs with human-like cognitive abilities of "Understand -> Rethink -> Correct", and achieves a paradigm evolution from passive imitation-based reasoning to active discriminative reasoning. Specifically, we introduce a Rationale Augmentation and Discrimination (RAD) paradigm, which provides a unified and extensible data foundation. Meanwhile, we design a Progressive Two-stage Correction Learning (P2CL) strategy. The first phase enhances multi-rationale positive learning, while the second phase enables active logic discrimination and correction. In addition, to mitigate representation entanglement in the multi-rationale semantic space, we propose a Multi-rationale Contrastive Alignment (MCA) optimization strategy. Extensive experiments show that our MIND achieves SOTA performance on multiple public datasets. Our data and code are available at https://github.com/YuChuang1205/MIND
Figures
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[2]
Bottom-up and top-down attention for image captioning and visual question answering
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. InCVPR, pages 6077–6086,
-
[3]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 5, 7
Pith/arXiv arXiv 2025
-
[4]
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che.m 3cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought.arXiv preprint arXiv:2405.16473, 2024. 1, 3, 5, 12
Pith/arXiv arXiv 2024
-
[5]
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: In- teractive prompting between vision and language mod- els for knowledge-based visual reasoning.arXiv preprint arXiv:2301.05226, 2023. 2, 5, 6
Pith/arXiv arXiv 2023
-
[6]
Comt: A novel benchmark for chain of multi-modal thought on large vision- language models
Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: A novel benchmark for chain of multi-modal thought on large vision- language models. InAAAI, pages 23678–23686, 2025. 1
2025
-
[7]
Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 5
2024
-
[8]
Instructblip: Towards general-purpose vision- language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023. 7
2023
-
[9]
Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Ex- ploring long-chain visual reasoning with multimodal large language models. InCVPR, pages 9062–9072, 2025. 1, 2
2025
-
[10]
Dy- namic fusion with intra-and inter-modality attention flow for visual question answering
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dy- namic fusion with intra-and inter-modality attention flow for visual question answering. InCVPR, pages 6639–6648,
-
[11]
Cantor: Inspiring multimodal chain-of-thought of mllm
Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, et al. Cantor: Inspiring multimodal chain-of-thought of mllm. InProceedings of the ACM International Conference on Multimedia (ACM MM), pages 9096–9105, 2024. 2
2024
-
[12]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 7, 8
Pith/arXiv arXiv 2025
-
[13]
Multi- modal latent space learning for chain-of-thought reasoning in language models
Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi- modal latent space learning for chain-of-thought reasoning in language models. InAAAI, pages 18180–18187, 2024. 1, 2, 5, 6
2024
-
[14]
Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.Advances in Neural Informa- tion Processing Systems, 37:139348–139379, 2024
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Osten- dorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models.Advances in Neural Informa- tion Processing Systems, 37:139348–139379, 2024. 2
2024
-
[15]
Towards a multimodal large language model with pixel-level insight for biomedicine
Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine. InAAAI, pages 3779–3787, 2025. 1
2025
-
[16]
Words over pixels? rethinking vision in multimodal large language mod- els
Anubhooti Jain, Mayank Vatsa, and Richa Singh. Words over pixels? rethinking vision in multimodal large language mod- els. InProceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 10481–10489, 2024. 1
2024
-
[17]
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0. 1: the win- ning entry to the vqa challenge 2018.arXiv preprint arXiv:1807.09956, 2018. 6
Pith/arXiv arXiv 2018
-
[18]
Webly su- pervised concept expansion for general purpose vision mod- els
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly su- pervised concept expansion for general purpose vision mod- els. InECCV, pages 662–681. Springer, 2022. 6
2022
-
[19]
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabhar- wal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa sys- tem.arXiv preprint arXiv:2005.00700, 2020. 6
Pith/arXiv arXiv 2005
-
[20]
Bi- linear attention networks.Advances in neural information processing systems, 31, 2018
Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bi- linear attention networks.Advances in neural information processing systems, 31, 2018. 6
2018
-
[21]
Vilt: Vision- and-language transformer without convolution or region su- pervision
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. InProceedings of the International Conference on Machine Learning (ICML), pages 5583–5594. PMLR, 2021. 6
2021
-
[22]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026,
-
[23]
Gen- erating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506,
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Gen- erating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506,
-
[24]
Multi- modal reasoning with multimodal knowledge graph.arXiv preprint arXiv:2406.02030, 2024
Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multi- modal reasoning with multimodal knowledge graph.arXiv preprint arXiv:2406.02030, 2024. 2
Pith/arXiv arXiv 2024
-
[25]
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML, pages 12888–12900. PMLR, 2022. 8
2022
-
[26]
Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models. InICML, pages 19730–19742. PMLR, 2023. 5, 7, 8
2023
-
[27]
What does bert with vision look at? In Proceedings of the Association for Computational Linguis- tics (ACL), pages 5265–5275, 2020
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does bert with vision look at? In Proceedings of the Association for Computational Linguis- tics (ACL), pages 5265–5275, 2020. 6
2020
-
[28]
V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal mod- els
Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Xuan- Jing Huang, and Zhongyu Wei. V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal mod- els. InProceedings of the Conference of the North Ameri- can Chapter of the Association for Computational Linguis- tics: Human Language Technologies (NAACL-HLT), pages 3769–3798, 2025. 2
2025
-
[29]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 7, 8
Pith/arXiv arXiv 2024
-
[30]
Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, and Longyue Wang. Retrieval-augmented multi- modal chain-of-thoughts reasoning for large language mod- els.arXiv preprint arXiv:2312.01714, 2023. 2
Pith/arXiv arXiv 2023
-
[31]
Improved baselines with visual instruction tuning
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024. 5, 6, 7
2024
-
[32]
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019. 6
2019
-
[33]
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram under- standing and visual language reasoning.arXiv preprint arXiv:2110.13214, 2021. 6
Pith/arXiv arXiv 2021
-
[34]
Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,
-
[35]
Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Pro- cessing Systems, 36:43447–43478, 2023
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Pro- cessing Systems, 36:43447–43478, 2023. 6, 7
2023
-
[36]
Cheap and quick: Efficient vision-language instruction tuning for large language mod- els.Advances in Neural Information Processing Systems, 36:29615–29627, 2023
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xi- aoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language mod- els.Advances in Neural Information Processing Systems, 36:29615–29627, 2023. 6
2023
-
[37]
Pkrd-cot: A unified chain-of-thought prompting for multi-modal large language models in au- tonomous driving
Xuewen Luo, Fan Ding, Yinsheng Song, Xiaofeng Zhang, and Junnyong Loo. Pkrd-cot: A unified chain-of-thought prompting for multi-modal large language models in au- tonomous driving. InProceedings of the International Con- ference on Neural Information Processing (ICONIP), pages 62–76. Springer, 2024. 2
2024
-
[38]
Krisp: Integrating implicit and sym- bolic knowledge for open-domain knowledge-based vqa
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and sym- bolic knowledge for open-domain knowledge-based vqa. In CVPR, pages 14111–14121, 2021. 6
2021
-
[39]
Chain of images for intuitively reasoning.arXiv preprint arXiv:2311.09241, 2023
Fanxu Meng, Haotong Yang, Yiding Wang, and Muhan Zhang. Chain of images for intuitively reasoning.arXiv preprint arXiv:2311.09241, 2023. 2
Pith/arXiv arXiv 2023
-
[40]
Compositional chain-of-thought prompting for large multimodal models
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. InCVPR, pages 14420–14431, 2024. 2
2024
-
[41]
Clip- cap: Clip prefix for image captioning.arXiv preprint arXiv:2111.09734, 2021
Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning.arXiv preprint arXiv:2111.09734, 2021. 6
Pith/arXiv arXiv 2021
-
[42]
Dinov2: Learning robust visual features without supervision
Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 7, 8
Pith/arXiv arXiv 2023
-
[43]
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023. 7
Pith/arXiv arXiv 2023
-
[44]
Learn- ing transferable visual models from natural language super- vision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, pages 8748–8763. PmLR, 2021. 7, 8
2021
-
[45]
Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 5
2020
-
[46]
A-okvqa: A benchmark for visual question answering using world knowl- edge
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. InECCV, pages 146–162. Springer, 2022. 3, 5, 12
2022
-
[47]
Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180,
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.Advances in Neural Information Processing Systems, 36:38154–38180,
-
[48]
Assessing gpt4-v on structured reasoning tasks.arXiv preprint arXiv:2312.11524, 2023
Mukul Singh, Jos ´e Cambronero, Sumit Gulwani, Vu Le, and Gust Verbruggen. Assessing gpt4-v on structured reasoning tasks.arXiv preprint arXiv:2312.11524, 2023. 2
Pith/arXiv arXiv 2023
-
[49]
Boosting the power of small multimodal reasoning models to match larger models with self-consistency training
Cheng Tan, Jingxuan Wei, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Ruifeng Guo, Bihui Yu, and Stan Z Li. Boosting the power of small multimodal reasoning models to match larger models with self-consistency training. InECCV, pages 305–322. Springer, 2024. 1, 2, 5, 6, 7
2024
-
[50]
Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. Retrieval meets reasoning: Even high-school textbook knowledge benefits multimodal reasoning.arXiv preprint arXiv:2405.20834, 2024. 2, 6
Pith/arXiv arXiv 2024
-
[51]
Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers.arXiv preprint arXiv:1908.07490, 2019. 6
Pith/arXiv arXiv 1908
-
[52]
Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, 10 Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 6, 7
Pith/arXiv arXiv 2023
-
[53]
Ranked from within: Ranking large multimodal models without labels
Weijie Tu, Weijian Deng, Dylan Campbell, Yu Yao, Jiyang Zheng, Tom Gedeon, and Tongliang Liu. Ranked from within: Ranking large multimodal models without labels. In ICML, 2025. 1
2025
-
[54]
Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 37:75392–75421, 2024. 1
2024
-
[55]
T-sciq: Teaching multimodal chain- of-thought reasoning via large language model signals for science question answering
Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal chain- of-thought reasoning via large language model signals for science question answering. InAAAI, pages 19162–19170,
-
[56]
Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024. 7
2024
-
[57]
Videoagent: Long-form video understanding with large language model as agent
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. InECCV, pages 58–76. Springer, 2024. 2
2024
-
[58]
Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025. 1, 2
Pith/arXiv arXiv 2025
-
[59]
Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 6
2022
-
[60]
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models.arXiv preprint arXiv:2303.04671, 2023. 7
Pith/arXiv arXiv 2023
-
[61]
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end general- ist multimodal large language model for hundreds of vision- language tasks.Advances in Neural Information Processing Systems, 37:69925–69975, 2024. 1
2024
-
[62]
Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large lan- guage models.Advances in Neural Information Processing Systems, 37:90277–90317, 2024
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large lan- guage models.Advances in Neural Information Processing Systems, 37:90277–90317, 2024. 2
2024
-
[63]
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C Gee, and Yixin Nie. The role of chain-of-thought in complex vision-language reasoning task.arXiv preprint arXiv:2311.09193, 2023. 2
Pith/arXiv arXiv 2023
-
[64]
Det- toolchain: A new prompting paradigm to unleash detection ability of mllm
Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Det- toolchain: A new prompting paradigm to unleash detection ability of mllm. InECCV, pages 164–182. Springer, 2024. 2
2024
-
[65]
Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 7, 8
Pith/arXiv arXiv 2025
-
[66]
Thinking in space: How mul- timodal large language models see, remember, and recall spaces
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces. InCVPR, pages 10632–10643, 2025. 1
2025
-
[67]
Magic-vqa: Multimodal and grounded inference with com- monsense knowledge for visual question answering
Shuo Yang, Caren Han, Siwen Luo, and Eduard Hovy. Magic-vqa: Multimodal and grounded inference with com- monsense knowledge for visual question answering. InFind- ings of the Association for Computational Linguistics (Find- ings of ACL), pages 16967–16986, 2025. 1
2025
-
[68]
An empirical study of gpt-3 for few-shot knowledge-based vqa
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yu- mao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. InAAAI, pages 3081–3089, 2022. 6
2022
-
[69]
Study- ing and improving reasoning in humans and machines.Com- munications Psychology, 2(1):51, 2024
Nicolas Yax, Hern ´an Anll´o, and Stefano Palminteri. Study- ing and improving reasoning in humans and machines.Com- munications Psychology, 2(1):51, 2024. 1
2024
-
[70]
Idealgpt: Iteratively decomposing vision and lan- guage reasoning via large language models
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. Idealgpt: Iteratively decomposing vision and lan- guage reasoning via large language models. InFindings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP), pages 11289–11303, 2023. 7
2023
-
[71]
Deep modular co-attention networks for visual question an- swering
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question an- swering. InCVPR, pages 6281–6290, 2019. 5, 6
2019
-
[72]
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Con- trastive chain-of-thought prompting for large multimodal models with multiple image inputs.arXiv preprint arXiv:2401.02582, 2024. 2
Pith/arXiv arXiv 2024
-
[73]
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023. 6, 7
Pith/arXiv arXiv 2023
-
[74]
Multimodal chain-of-thought rea- soning in language models.Transactions on Machine Learn- ing Research, 2024, 2024
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought rea- soning in language models.Transactions on Machine Learn- ing Research, 2024, 2024. 1, 2, 5, 6, 7, 12
2024
-
[75]
Antgpt: Can large language models help long-term ac- tion anticipation from videos?arXiv preprint arXiv:2307
Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term ac- tion anticipation from videos?arXiv preprint arXiv:2307. 16368, 2023. 2
2023
-
[76]
Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023
Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neu- ral Information Processing Systems, 36:5168–5191, 2023. 1, 2, 6
2023
-
[77]
Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation
Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In CVPR, pages 13246–13257, 2024. 2 11 MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models Supplemen...
2024
-
[78]
Baseline
More Break-Down Ablation.To more comprehen- sively analyze the independent contributions and synergis- tic gains of each core component, we conduct systematic ablation studies on the P2CL strategy and MCA optimiza- tion strategies across ScienceQA, A-OKVQA, and M 3CoT datasets. The experimental results are shown in Tab. S1. First, compared to the“Baseline...
-
[79]
From Fig
Exploration of Hyperparameters in the MCA opti- mization Strategy.We conduct a systematic analysis under various parameter settings. From Fig. S1, the experimental results with different values ofmandαdemonstrate that the MCA optimization strategy has remarkable stability. In particular, whenm∈[0.1−0.4]andα∈[0.5−5.0], Table S1. Break-Down ablation experim...
-
[80]
(B) 6.8 km
Comparison Between Multimodal-CoT and MIND. To more intuitively demonstrate the superior interpretabil- ity and logical consistency of MIND in multimodal reason- 12 ScienceQA #843 Question: Will these magnets attract or repel each other? Options: Gold Rationale: Will these magnets attract or repel? To find out, look at which poles are closest to each othe...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.