Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central assertion is that current vision-language models do not effectively use demonstration-level information in multimodal in-context learning: few-shot accuracy rarely exceeds zero-shot performance and is flat across shot…

desk verdict Worth a referee, but the headline negative claim is scoped too broadly: every benchmark is a Case I task solvable zero-shot, no Case II experiments appear, and the largest few-shot gain is unexplained. read the letter →

arxiv 2506.07936 v1 pith:YQKWMRID submitted 2025-06-09 cs.CV cs.AIcs.CLcs.LG

Chengyue Huang , Yuchen Zhu , Sichen Zhu , Jingyun Xiao , Moises Andrade , Shivang Chopra , Zsolt Kira This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords multimodalin-contextlearningvision-languagemodelsreasoningchain-of-thoughtdistributionshiftfew-shotevaluationmethodologyrationalequality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models are widely assumed to inherit in-context learning from their language-model backbones, but this paper argues that the assumption breaks down in the multimodal setting. Its central claim is that current VLMs do not effectively learn from demonstrations: across perception and reasoning benchmarks, few-shot accuracy rarely beats zero-shot accuracy, and performance stays flat when the number of shots, the retrieval method, the source distribution, and even the quality of the supplied rationales are varied. To give demonstrations their best chance, the paper introduces a reasoning-augmented pipeline in which every support example carries a step-by-step rationale plus the answer, formatted consistently with what reasoning-tuned models are trained to produce. A reader should care because if the claim holds, many reported MM-ICL gains are actually format mimicry or answer copying, not task learning, which changes how VLM capabilities should be benchmarked and trained.

What carries the argument

The central object is the format-consistent reasoning-augmented demonstration: each support example contains an image, a question, a rationale, and an answer, where the rationale is either generated by the model itself (Pseudo Reasoning) or reformulated from ground truth (Gold Reasoning), and the whole support-query prompt matches the exact output format the reasoner was trained on. The paper's key move is to identify and fix a format inconsistency in prior MM-ICL protocols: demonstrations carried only answers while the model was expected to emit rationale plus answer, which penalized the model and masked what it actually learns. The pipeline also includes a correctness-based filter and a multimodal retriever for selecting support examples, so that the negative results can be attributed to the model's use of demonstrations rather than to a poorly constructed prompt.

What would settle it

A single model-dataset configuration in which accuracy rises steadily with shot count under out-of-distribution support sets, and the gain survives format-controlled scoring, would refute the claim that current VLMs fail to learn from demonstrations. A sharper test: build support sets whose rationales are deliberately wrong but whose answers are correct; if a model's query accuracy tracks the rationale quality, it is using demonstration content, not just copying answers.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that current vision-language models and VLM reasoners do not effectively leverage demonstration-level information in the way multimodal in-context learning is supposed to. The evidence is a broad controlled study: zero-shot accuracy is the best or near-best configuration in most model-dataset pairs; adding one, two, four, or eight demonstrations rarely helps and often hurts; out-of-distribution support sets sometimes beat in-distribution ones; multimodal retrieval helps base models on some tasks but hurts reasoning models; and injecting ground-truth rationales or filtering out incorrect support samples does not consistently improve results. The paper treats these insensitivities as a negative result about the models rather than about the demonstrations: current VLMs are largely insensitive to the information content of demonstrations, so observed few-shot improvements are attributed to answer-format alignment and shallow heuristics such as copying. The proposed MM-ICL with Reasoning pipeline is presented as a stronger and fairer testbed for this conclusion, because it removes a format mismatch that previously confounded reasoning-model evaluation.

Load-bearing premise

The load-bearing premise is that the accuracy metrics, exact-match scoring for perception tasks and LLM judges for reasoning tasks, measure task understanding rather than whether the model happened to write its answer in the expected format; if formatting dominates the scores, the negative conclusion about learning from demonstrations does not follow.

Editorial extensions

If this is right

  • Few-shot multimodal in-context learning should not be assumed to improve task performance; benchmark gains need to be checked against a format-controlled zero-shot baseline.
  • For reasoning-tuned VLMs, demonstration format must match the expected generation format; consistent rationale-plus-answer demos outperform answer-only demos, especially at higher shot counts.
  • In-distribution support relevance is not the main driver of MM-ICL performance, since out-of-distribution support sets often perform as well as or better than in-distribution sets when format confounds are removed.
  • Retrieval algorithms that help base models, such as multimodal similarity retrieval, can hurt reasoning models, so retriever choice is not a reliable lever for improving MM-ICL.
  • Demonstration quality, measured by correctness of the answer or rationale, does not reliably improve few-shot performance, suggesting that models are not extracting task methodology from demonstrations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's logic is to hold answer format constant with constrained decoding or normalized scoring; if few-shot gains disappear entirely, the remaining signal is format alignment rather than content learning.
  • The paper's observation that filtered support sets can lose diversity points to a broader design principle: support-set diversity and coverage may matter more than per-example correctness for current VLMs, which could be tested by controlling diversity independently of quality.
  • The contrast with language-only in-context learning, where demonstrations usually help, suggests the bottleneck may be multimodal fusion rather than in-context learning itself; comparing text-only and image-plus-text versions of the same task could localize where demonstrations stop being useful.
  • Because reasoning-focused reinforcement training did not change the models' sensitivity to demonstrations, a natural next step is to train or fine-tune explicitly with an objective that rewards correct query answers conditional on demonstrations, and test whether sensitivity to demonstration content increases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper revisits the assumption that vision-language models (VLMs) perform genuine multimodal in-context learning (MM-ICL). It defines Case I tasks (well-defined and solvable without demonstrations) and Case II tasks (ill-defined without demonstrations), then evaluates models under in-distribution versus out-of-distribution support sets, and proposes an MM-ICL with Reasoning pipeline that augments demonstrations with generated rationales. Experiments cover open-source VLMs from 3B to 72B plus Gemini 2.0 across five datasets, varying shot count, retrieval method, and rationale quality. The central claim is that current VLMs do not effectively leverage demonstration-level information: few-shot performance rarely exceeds zero-shot and is largely insensitive to these factors.

Significance. The paper makes a useful methodological contribution by identifying and partially fixing the format-inconsistency problem in MM-ICL evaluation of reasoning models, and by providing a broad, controlled comparison across many model families and datasets. It also transparently reports training-data overlap in Appendix Table 12, which is valuable for interpreting benchmark scores. If the central claim is restricted to Case I tasks, the evidence is substantial and suggestive. However, the paper's headline conclusion is broader than the evidence: all experiments use Case I datasets where demonstrations are redundant, and the paper never tests the Case II tasks it itself defines as the defining setting for ICL. The unexplained large few-shot gains of one model also undermine the blanket negative conclusion.

major comments (4)
  1. [Sec. 4.1 and Sec. 6] The manuscript defines Case II tasks as those where demonstrations are necessary to define the task (operator induction, synthetic object categories), yet every experiment in Sections 4-6 uses only Case I datasets: TextVQA, OK-VQA, A-OKVQA, ScienceQA, and M3CoT. On these datasets zero-shot accuracy is often high (e.g., InternVL2.5 reaches 97-98% on ScienceQA), and Appendix Table 12 shows that several evaluated models were trained on the same benchmarks. A flat or declining few-shot curve on tasks the model can already solve zero-shot does not establish that the model cannot learn from demonstrations when it needs to. The central claim 'current VLMs do not effectively leverage demonstration-level information' therefore requires either Case II experiments or an explicit restriction of the conclusion to Case I tasks.
  2. [Table 3] Llama-3.2-11B-Vision-Instruct improves from 53.87 to 74.63 on TextVQA and from 20.05 to 44.03 on OK-VQA when going from 0-shot to the best few-shot setting. These are gains of 20.76 and 23.98 points, respectively, and are neither 'rare' nor 'minimal.' The paper does not analyze whether these gains reflect genuine in-context task learning, format alignment, copying, or majority-vote heuristics. This is a direct counterexample to the blanket claim in Section 6.2 that 'in the majority of cases, MM-ICL with a few demonstrations does not exceed the performance when no demonstrations are presented,' and it needs to be resolved, for instance by isolating the mechanism or by reporting per-model heterogeneity as part of the main conclusion.
  3. [Sec. 6.1, Tab. 2] The 'inconsistent' versus 'consistent' comparison in Table 2 changes more than the demonstration format. In the consistent condition, the query prompt also instructs the model to 'Give step-by-step reasoning before you answer,' and the model is expected to produce a full rationale plus answer; in the inconsistent condition, the same query-side reasoning instruction is paired with answer-only demonstrations. The observed gains could therefore be driven by query-side chain-of-thought prompting rather than by support-query format alignment. A cleaner control would hold the query prompt fixed while varying only whether demonstrations include rationales, or would vary the demonstration format while keeping the query output format constant.
  4. [Tables 3-5 and Sec. 6.2] All accuracy numbers are reported as single-point estimates without error bars, confidence intervals, or multiple random support-set draws. The paper's 'insensitivity' conclusion is based on differences that are often below one percentage point (e.g., Table 5 deltas of -0.49 to +0.43). Without a measure of run-to-run variability, statements such as 'limited performance sensitivity across these factors' are not statistically supported. Reporting variance over random retrievers and support draws, or at least a seed sensitivity analysis, is needed for the central negative claim.
minor comments (5)
  1. [Sec. 4.1] The definition of Case I tasks as having solutions 'uniquely determined by the query' is too strong for visual question answering, where multiple valid answers are common; consider replacing 'uniquely determined' with 'well-defined without demonstrations.'
  2. [Sec. 6, first paragraph] The text says TextVQA and OK-VQA 'use their own answer matching metrics (e.g., string normalization and consensus-based accuracy)' but then states that the evaluation follows VLMEvalKit with GPT-4o mini as judge. Clarify which metric is used for each dataset in the reported tables.
  3. [Fig. 6 caption] The caption says '6 vision-language datasets' but lists five datasets (A-OKVQA, ScienceQA, M3CoT, TextVQA, OK-VQA), and the figure shows only two models. Please either include all models or state in the caption that other models are reported in the appendix.
  4. [Appendix Table 12] The table is useful but hard to read: some cells contain '✓∗', others '✓', and the VLM-R1 row has a misplaced value. Consider using a legend with distinct symbols for 'in pretraining data', 'in fine-tuning data', and 'not found'.
  5. [Sec. 5, Protocol 4] The correctness-based filtering step uses the correctness of the generated answer as a proxy for rationale quality. This is a pragmatic heuristic, but its validity is assumed rather than justified; a sentence acknowledging that the proxy may discard correct rationales paired with wrong answers would strengthen the exposition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark evaluation against external datasets, with no fitted parameters and no claim that reduces to its own inputs.

full rationale

This paper does not contain a derivation chain that reduces to its inputs. Its central claim, that current VLMs show limited sensitivity to shot count, retrieval method, and rationale quality in MM-ICL, is supported by tables of measured accuracy on external benchmarks (TextVQA, OK-VQA, A-OKVQA, ScienceQA, M3CoT) using publicly available models. No parameter is fitted to the evaluation data, and no quantity called a 'prediction' is obtained from a fitted value. The proposed MM-ICL with Reasoning pipeline is an experimental protocol, not a theoretical derivation: the pseudo-rationales are generated by the models themselves, and the paper explicitly reports ablations to show that neither better rationales nor ground-truth rationales change the conclusions, so the results are not forced by the construction of the rationales. The Case I / Case II taxonomy is a classification of tasks, and the paper's limitation that its experiments focus on Case I datasets (recognized by the skeptical reading and implicitly by the paper's own Sec. 4.1 framing) is a scope concern about external validity, not a circularity. Similarly, Appendix Table 12 documenting possible training-data overlap is transparency about contamination, not a reduction of the conclusion to its own assumptions. Even if the few-shot-versus-zero-shot comparison is challenged on metric sensitivity or benchmark difficulty, that is a correctness or robustness critique, not a circularity critique. Since no derivation is self-referential and no load-bearing step is justified only by a self-citation, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper has no fitted parameters and no invented physical entities. The assumptions above are evaluation-design choices that the central negative claim depends on.

assumptions (4)
  • domain assumption Support sets drawn from a different dataset but with the same task type constitute a meaningful distribution shift for MM-ICL.
    Sec 4.2 defines OOD this way and uses it to argue that format and copying, not task understanding, drive ID/OOD gaps.
  • domain assumption Exact-match scoring and GPT-4o mini LLM judges faithfully measure the target capability.
    Sec 6 relies on these for accuracy; if judges are insensitive to reasoning quality, the invariance results are artifacts.
  • domain assumption Ground-truth rationales reformatted by the target model remain informative.
    Sec 5 Gold Reasoning uses the model to reformat rationales; information loss during reformatting would explain insensitivity.
  • ad hoc to paper Correctness-based filtering of support samples is a valid reliability control.
    Sec 5 introduces filtering to remove misleading rationales; the paper notes filtering can reduce diversity, which confounds quality comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models." pith.science (2026). https://pith.science/paper/YQKWMRID

@misc{pith2026250607936,
  author       = {Pith},
  title        = {Pith review of: Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQKWMRID}},
  note         = {Machine review of arXiv:2506.07936}
}
read the original abstract

Vision-language models (VLMs) are widely assumed to exhibit in-context learning (ICL), a property similar to that of their language-only counterparts. While recent work suggests VLMs can perform multimodal ICL (MM-ICL), studies show they often rely on shallow heuristics -- such as copying or majority voting -- rather than true task understanding. We revisit this assumption by evaluating VLMs under distribution shifts, where support examples come from a dataset different from the query. Surprisingly, performance often degrades with more demonstrations, and models tend to copy answers rather than learn from them. To investigate further, we propose a new MM-ICL with Reasoning pipeline that augments each demonstration with a generated rationale alongside the answer. We conduct extensive and comprehensive experiments on both perception- and reasoning-required datasets with open-source VLMs ranging from 3B to 72B and proprietary models such as Gemini 2.0. We conduct controlled studies varying shot count, retrieval method, rationale quality, and distribution. Our results show limited performance sensitivity across these factors, suggesting that current VLMs do not effectively utilize demonstration-level information as intended in MM-ICL.

Figures

Figures reproduced from arXiv: 2506.07936 by the authors.

Figure 1
Figure 1. Left: Performance difference between ID and OOD using random retriever. Middle: Performance of different retrieval methods on OK-VQA. ID: OK-VQA as support set. OOD: TextVQA as support set. We include the unimodal retriever to highlight that the multimodal retriever achieves the best performance in the ID setting, consistent with Qin et al. [23]. Right: Wrong answer format directly increases error rate. where psys i… view at source ↗
Figure 2
Figure 2. Demonstration of examples from Case I/II tasks. Case II problems are ill-defined if no [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Success and failure of MM-ICL with IDEFICS2. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of Full Pipeline for ICL with VLM Reasoner [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Prompt format for each MM-ICL protocol. GT stands for ground truth. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Multimodal Retriever vs. Random Selection on 6 vision-language datasets. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Performance difference between OOD and ID on ScienceQA and A-OKVQA [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Gemini 2.0 Flash w/ Pseudo Reasoning Demos on M 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: LLaVA-CoT w/ Pseudo Reasoning Demos on ScienceQA (2-shot & random selection) [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. True Multimodal In-Context Learning Needs Attention to the Visual Context

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 160-parameter attention-scaling method, DARA, improves true multimodal in-context learning on a new dataset, TrueMICL, that forces models to use demo images rather than copy text patterns.

  2. In-context Learning of Vision Language Models for Detection of Physical and Digital Attacks against Face Recognition Systems

    cs.CV 2025-07 conditional novelty 6.0 of 10

    An in-context learning framework with open-source vision-language models detects face presentation and morphing attacks without training, beating CLIP-based zero-shot baselines on PAD but with performance highly sensi...

Reference graph

Works this paper leans on

55 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  2. [2]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, October 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, October 2023. URL http://arxiv.org/abs/2308. 12966. arXiv:2308.12966 [cs]

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    What makes multimodal in-context learning work? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539–1550, 2024

    Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Pi- wowarski. What makes multimodal in-context learning work? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539–1550, 2024

    2024

  5. [5]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024. URL https://arxiv.org/abs/2405.16473

    arXiv 2024

  7. [7]

    Can Multimodal Large Language Models Truly Perform Multi- modal In-Context Learning?, December 2024

    Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, V olker Tresp, and Jindong Gu. Can Multimodal Large Language Models Truly Perform Multi- modal In-Context Learning?, December 2024. URL http://arxiv.org/abs/2311.18021. arXiv:2311.18021 [cs]

    arXiv 2024

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

Show all 55 references
  1. [9]

    A survey on in-context learning.arXiv preprint arXiv:2301.00234, 2022

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  2. [10]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia,...

    2024

  3. [11]

    Interleaved-Modal Chain-of-Thought, November 2024

    Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li. Interleaved-Modal Chain-of-Thought, November 2024. URLhttp://arxiv.org/abs/2411.19488. arXiv:2411.19488 [cs]

    2024 arXiv

  4. [12]

    Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking, March 2025

    Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi, and Xueqi Cheng. Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking, March 2025. URL http://arxiv.org/abs/2503.19602. arXiv:2503.19602 [cs]

    2025 arXiv

  5. [13]

    Can MLLMs Reason in Multimodality? EMMA: An Enhanced Multi- Modal ReAsoning Benchmark, January 2025

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can MLLMs Reason in Multimodality? EMMA: An Enhanced Multi- Modal ReAsoning Benchmark, January 2025. URL http://arxiv.org/abs/2501.05444. arXiv:2501.05444 [cs] version: 1. 11

    2025 arXiv

  6. [14]

    MME- CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency, February 2025

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, and Hongsheng Li. MME- CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Eff...

    2025 arXiv

  7. [15]

    Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP, January 2023

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP, January 2023. URL http://arxiv.org/abs/2212.14024. arXiv:2212.14024 [cs]

    2023 arXiv

  8. [16]

    What matters when building vision-language models?arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  9. [17]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. URLhttps://arxiv.org/abs/2209.09513

    2022 arXiv

  10. [18]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts, January 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts, January 2024. URL http://arxiv. org/abs/2310.02255....

    2024 arXiv

  11. [19]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.arXiv preprint arXiv:2104.08786, 2021

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.arXiv preprint arXiv:2104.08786, 2021

    2021 arXiv

  12. [20]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.https: //ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024

    MetaAI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.https: //ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024

    2024

  13. [21]

    Few- shot fine-tuning vs

    Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context learning: A fair comparison and evaluation, 2023. URL https: //arxiv.org/abs/2305.16938

    2023 arXiv

  14. [22]

    Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.arXiv preprint arXiv:2310.14799, 2023

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages.arXiv preprint arXiv:2310.14799, 2023

    2023 arXiv

  15. [23]

    What factors affect multi-modal in-context learning? an in-depth exploration.arXiv preprint arXiv:2410.20482, 2024

    Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration.arXiv preprint arXiv:2410.20482, 2024

    2024 arXiv

  16. [24]

    Benjamin Z. Reichman, Anirudh Sundar, Christopher Richardson, Tamara Zubatiy, Prithwijit Chowdhury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside Knowledge Visual Question Answering Version 2.0. InICASSP ...

    2023

  17. [25]

    A-okvqa: A benchmark for visual question answering using world knowledge, 2022

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022. URLhttps://arxiv.org/abs/2206.01718

    2022 arXiv

  18. [26]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  19. [27]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 12

    2019

  20. [28]

    Vl- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025

    2025 arXiv

  21. [29]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  22. [30]

    Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, March 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, March 2025. URLhttp://arxiv.org/abs/2503.12605. arXiv:2503.12605 [cs]

    2025 arXiv

  23. [31]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  24. [32]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URLhttps://arxiv.org/abs/2201.11903

    2023 arXiv

  25. [33]

    Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. arXiv preprint arXiv:2212.10375, 2022

    2022 arXiv

  26. [34]

    Addressing order sensitivity of in-context demonstration examples in causal language models.arXiv preprint arXiv:2402.15637, 2024

    Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in-context demonstration examples in causal language models.arXiv preprint arXiv:2402.15637, 2024

    2024 arXiv

  27. [35]

    Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  28. [36]

    From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning.arXiv preprint arXiv:2407.00902, 2024

    Nan Xu, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning.arXiv preprint arXiv:2407.00902, 2024

    2024 arXiv

  29. [37]

    Formal Mathematical Reasoning: A New Frontier in AI, December 2024

    Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. Formal Mathematical Reasoning: A New Frontier in AI, December 2024. URL http://arxiv.org/abs/2412.16075. arXiv:2412.16075 [cs]

    2024 arXiv

  30. [38]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 3081–3089, 2022

    2022

  31. [39]

    Automatic chain of thought prompting in large language models, 2022

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models, 2022. URLhttps://arxiv.org/abs/2210.03493

    2022 arXiv

  32. [40]

    Wong, and Simon See

    Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y . Wong, and Simon See. The curse of cot: On the limitations of chain-of-thought in in-context learning, 2025. URL https://arxiv.org/abs/2504.05081

    2025

  33. [41]

    Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, April 2023

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, April 2023. URL http://arxiv. org/abs/2205.10625. arXiv...

    2023 arXiv

  34. [42]

    Vl-icl bench: The devil in the details of benchmarking multimodal in-context learning.arXiv preprint arXiv:2403.13164, 2024

    Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. Vl-icl bench: The devil in the details of benchmarking multimodal in-context learning.arXiv preprint arXiv:2403.13164, 2024

    2024 arXiv

  35. [43]

    Final answer:

    Michal Štefánik and Marek Kadlˇcík. Can In-context Learners Learn a Reasoning Concept from Demonstrations?, July 2023. URL http://arxiv.org/abs/2212.01692. arXiv:2212.01692 [cs]. 13 A Additional results on ID v.s. OOD for IDEFICS2 and Qwen2.5-VL-3B-Instruct In Sec. 4.2, we pre...

    2023 arXiv

  36. [44]

    Density is calculated as mass/volume

    **Density Calculation:** An object will sink if its density is greater than the density of water (approximately 1000 kg/m³). Density is calculated as mass/volume

  37. [45]

    **Calculate the density of each object:** * Object A: 5108.68 kg / 8.92 m³ = 572.72 kg/m³ * Object B: 5521.41 kg / 14.11 m³ = 391.31 kg/m³ * Object C: 4973.41 kg / 10.2 m³ = 487.59 kg/m³ * Object D: 2472.27 kg / 16.97 m³ = 145.68 kg/m³ * Object E: 7085.55 kg / 19.08 m³ = 371.3...

  38. [46]

    **Compare to Water Density:** None of the objects have a density greater than 1000 kg/m³

  39. [47]

    Final answer:

    **Conclusion:** Since none of the objects have a density greater than water, none of them will sink. Final answer: A User:[Support Sample 2] Image: Question: How many countries have achieved fiscal surpluses in 2020? Options: A. 2 B. 5 C. 6 D. 7 E. 4 Please select the correct ...

    2020

  40. [48]

    In the graphs, we need to compare the export value (top graph) with the import value (bottom graph) for each country in the year 2020

    **Understanding Fiscal Surplus:** A fiscal surplus occurs when export value exceeds import value. In the graphs, we need to compare the export value (top graph) with the import value (bottom graph) for each country in the year 2020

    2020

  41. [49]

    Export > Import, so Country 3 has a surplus

    **Analyzing Each Country:** * **Country 3:** Export value in 2020 is approximately 4.03, and import value is approximately 3.23. Export > Import, so Country 3 has a surplus. * **Country 4:** Export value in 2020 is approximately 2.41, and import value is approximately 2.85. Ex...

    2020

  42. [50]

    **Counting Surpluses:** Countries 3 and 5 have fiscal surpluses in 2020

    2020

  43. [51]

    Final answer:

    **Total:** Therefore, 2 countries have fiscal surpluses in 2020. Final answer: A User:[Query] Image: Question: Based on the information in the picture, can you determine the correct solution for the Algebra problem? Options: A. option D in the image is the answer. B. the corre...

    2020

  44. [52]

    a² varies inversely with b³

    **Inverse Variation:** The statement "a² varies inversely with b³" means that a² * b³ = k, where k is a constant

  45. [53]

    Therefore, a² = 7² = 49

    **Find the Constant (k):** We are given that a = 7 when b = 3. Therefore, a² = 7² = 49. So, k = a² * b³ = 49 * 3³ = 49 * 27 = 1323

  46. [54]

    When b = 6, a² = 1323 / 6³ = 1323 / 216 = 6.125

    **Find a² when b = 6:** We know that a² * b³ = k, so a² = k / b³. When b = 6, a² = 1323 / 6³ = 1323 / 216 = 6.125

  47. [55]

    **Match with the Choices:** The value of a² when b = 6 is 6.125, which corresponds to option D in the image. Final answer: A Figure 8: Gemini 2.0 Flash w/ Pseudo Reasoning Demos on M3CoT (2-shot & random selection) 17 LLaV A-CoT w/ Pseudo Reasoning Demos on ScienceQA (2-shot &...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.