CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

Hao Fei; Jin Zhang; Libo Qin; Min Li; Qiguang Chen; Wanxiang Che; Xiaocheng Feng; Zihui Cheng

REVIEW 4 major objections 7 minor 1 cited by

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces CoMT, the first benchmark that forces large vision-language models to reason in images as well as text, and shows that current models score near random chance on it.

desk verdict Useful new multimodal reasoning benchmark and a striking negative result, but the Visual Creation task is likely solvable from text alone and the accuracy metric never enforces multi-modal output, so the blanket claim that LVLMs lack multi-modal thought needs an ablation and softer wording. read the letter →

arxiv 2412.12932 v3 pith:QVQWTDYN submitted 2024-12-17 cs.CV cs.AI

Zihui Cheng , Qiguang Chen , Jin Zhang , Hao Fei , Xiaocheng Feng , Wanxiang Che , Min Li , Libo Qin This is my paper · ORCID

classification cs.CVcs.AI

keywords chainofmulti-modalthoughtbenchmarklargevision-languagemodelsvisualreasoningchain-of-thoughtin-contextlearningoperationsCLIPScorealignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current benchmarks for multimodal reasoning are too easy because they accept text-only rationales, letting models describe visual operations without ever performing them. To close that gap, it introduces CoMT, a benchmark of 3,853 questions with 14,801 images that demands the model's reasoning chain itself be multimodal—interleaving generated or edited images with text across four visual operations: creation, deletion, update, and selection. The paper's central finding is that essentially all current LVLMs score at or near random chance on CoMT (best accuracy 33.44% versus a 25.48% random baseline), with only Gemini-Pro beating chance by a small margin. It also finds that text-only chain-of-thought prompting fails to help, that in-context learning with multimodal demonstrations helps up to about four shots, and that accuracy tracks both rationale quality and a CLIPScore measure of how well generated images align with the text. If the benchmark is a fair instrument, then today's LVLMs lack a core human reasoning ability: manipulating an image as part of thinking.

What carries the argument

The load-bearing object is the CoMT benchmark itself, together with its formal definition of multi-modal thought. In the paper's formulation (Equation 2), a chain of multi-modal thought is a sequence of rationale steps in which each step is chosen as either a visual generation or a text token, with the visual step taken when the model's probability of producing a useful visual rationale exceeds that of a textual one. The benchmark operationalizes this through a template with five keys—question, options, image, rationale, and answer—and four task pipelines that convert existing datasets into samples whose rationales interleave images with text. The paper also introduces an evaluation lens: CLIPScore between generated images and the reference images in the ideal rationale serves as a multi-modal alignment score, ROSCOE measures rationale quality, and human scoring (relevance plus logical correctness) on a sampled subset quantifies how often generated images are useless or inconsistent with the text.

What would settle it

Give a text-only LLM the Visual Creation questions with their augmented text descriptions but without the original diagram; if it matches or exceeds the accuracy of image-generating LVLMs on those items, then low Visual Creation scores measure image-generation failure rather than the absence of multimodal reasoning.

Watch

Extended reading notes

Core claim

The central claim is that CoMT is the first benchmark to require chain of multi-modal thought in LVLMs, meaning both the input and the reasoning output are multimodal. The paper constructs four task families from existing datasets: Visual Creation (from GeoQA+ geometry problems, moving the diagram into the rationale so the model must generate it), Visual Deletion (from JHU-CROWD++, masking faces stepwise so the model must remove visual clutter), Visual Update (from KILOGRAM tangrams, progressively coloring and annotating pieces), and Visual Selection (from spot-the-difference pairs, cropping and highlighting differences). Under this benchmark, the paper reports that almost all tested LVLMs—including image-capable models such as GILL, NExT-GPT and AnyGPT—perform near random chance, and that explicit attempts to make models visualize (VoT) or describe (Desp-CoT) before answering generally reduce accuracy. The paper therefore concludes that current models fail to execute visual operations inside the reasoning chain and that progress will require integrating multi-modal generation, textual logic, and visual logic.

Load-bearing premise

The benchmark's Visual Creation task assumes that a model cannot answer a geometry question from the written description alone; if the added text fully specifies the diagram, then the task tests image generation, not multi-modal reasoning.

Editorial extensions

If this is right

Existing MCoT benchmarks, which accept text-only rationales, likely overstate LVLMs' reasoning ability because they never force the model to perform a visual operation mid-chain.
Text-only prompting methods—CoT, Desp-CoT, and VoT—are not a path to multimodal reasoning; they can even push models below random accuracy.
Multi-modal in-context learning is the most promising lever identified, but it saturates at about four demonstrations and then degrades, so context-length effects matter.
Accuracy on CoMT correlates with the quality of the rationale and with visual-text alignment, suggesting that improving either component should directly improve performance.
Models trained on image generation are not automatically good at CoMT; the gap between generation and reasoning means the two capabilities must be co-designed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A text-only language model given the augmented textual descriptions used in Visual Creation might solve many questions without generating any image; if so, low scores on that task would partly reflect image-generation failure rather than missing multimodal reasoning, and the benchmark should be re-validated with text-only baselines on the full task text.
The positive CLIPScore-accuracy correlation suggests CoMT could be turned into a training objective: models could be rewarded for emitting rationales whose images align with their text, effectively learning visual logic by imitation.
The four visual operations are general enough to inspire analogous benchmarks in other modalities—audio editing or video manipulation—and in embodied settings where an agent must modify its environment to reason.
A testable extension is whether fine-tuning on CoMT's multimodal rationales transfers to existing benchmarks like ScienceQA or M3CoT; if transfer is positive, CoMT is not just a diagnostic but a curriculum.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

CoMT is worth knowing about: it's the first benchmark that asks for interleaved image-and-text rationales, across four visual operation types (create, delete, update, select). The construction is transparent—built on GeoQA+, JHU-CROWD++, KILOGRAM, and spotdiff—and the annotation pipeline has real quality controls (onboarding, double recheck, kappa 0.93). The main empirical finding, that almost every tested LVLM hovers near random while only Gemini edges slightly above, is clearly displayed and easy to believe. That alone makes the benchmark a useful stress test for multimodal reasoning research.

The soft spots are real, though they don't sink the whole thing. First, Visual Creation removes the image from the input and augments the text to fully describe the diagram (Section 2.1, Appendix C.2). That means the task is solvable from text alone; a model can do the geometry without ever generating an image. Low accuracy there could reflect text-based geometry failure or image-generation failure, not missing multi-modal thought. The authors should report human accuracy on this task and run an ablation that withholds the generated image to show it is actually needed. Second, the automatic evaluation extracts only the final answer option; it never checks whether the model produced a multi-modal rationale. So the accuracy numbers measure question-answering, not the presence of multi-modal reasoning. The paper's language—'requires both multi-modal input and multi-modal reasoning output'—describes the dataset format, not what the evaluation enforces. That weakens the causal reading that low scores are due to a lack of multi-modal thought. Third, human performance is claimed in the abstract and intro but no human numbers appear anywhere in the paper. That's an easy fix and should be mandatory before this goes out for real.

The ICL and CLIPScore correlations are suggestive but lack error bars or significance tests; treat them as qualitative. The paper also cites the relevant prior work (M3CoT, ScienceQA, VoT) and doesn't overclaim novelty about the underlying datasets.

Bottom line: this is a solid benchmark contribution with a clear negative result, but the central interpretation needs tightening. I'd send it to peer review and ask for the Visual Creation ablation, human numbers, and a softer claim about what the accuracy scores demonstrate. It's a useful resource for anyone evaluating LVLMs on visual reasoning.

Referee Report

4 major / 7 minor

Summary. The paper introduces CoMT, a multiple-choice benchmark of 3,853 samples across four task families—Visual Creation, Visual Deletion, Visual Update, and Visual Selection—in which the model is expected to produce a rationale that interleaves generated or edited images with text. The authors construct the benchmark from four existing datasets, add human quality assurance, evaluate eight LVLMs under Direct, CoT, Desp-CoT, and VoT prompting, and report that most models score near or below the random baseline while Gemini achieves the best accuracy. They also report in-context-learning experiments, rationale-quality correlations, and error analyses, and they argue that current LVLMs lack the ability to perform chain of multi-modal thought.

Significance. If the benchmark's defining requirement is actually enforced, CoMT addresses a real gap: existing MCoT benchmarks require multimodal input but only text output, whereas CoMT aims to require visual operations as part of the reasoning chain. The paper has concrete strengths: transparent construction from existing datasets, detailed annotation guidelines, human quality control with reported agreement (kappa 0.93), a random baseline, and evaluation across a broad set of LVLMs and prompting schemes. The central negative finding—that current LVLMs are at or below chance on CoMT—is directly visible in the main tables and is not dependent on fitted parameters. However, the benchmark's validity rests on two assumptions that are not tested: that the automatic scoring enforces the production of a multimodal rationale, and that each task genuinely requires a visual thought. Both assumptions are load-bearing for the paper's main claim, so the significance can only be realized after those points are resolved.

major comments (4)

[§4.1 / Appendix D.1] The evaluation protocol does not enforce the benchmark's defining property. Final answers are extracted with regular expressions, and the reported accuracy and Macro-F1 scores do not check whether the model's rationale contains an image, whether that image is relevant to the question, or whether the answer was reached through the required visual operation. A model that outputs a text-only rationale, or no rationale at all, receives full credit for a correct final letter. Because CoMT is explicitly defined as requiring multimodal reasoning output, the current metric measures multiple-choice answer accuracy, not chain of multi-modal thought. Please add an evaluation mode that conditions credit on the presence of a valid visual thought (for example, answer accuracy restricted to samples where the model emitted a relevant image in the rationale), or otherwise demonstrate that the final-answer metric cannot be achieved by text-only reasoning.
[§2.1 / Figure 2a / Appendix C.2] Visual Creation is likely solvable from the augmented text alone, which undermines the benchmark-wide negative conclusion for this 1,020-sample task. The construction removes the image from the model input and supplements the question with a text description that fully specifies the geometry, with the image moved into the rationale as the supposed visual thought. If a model can derive the correct angle from the text alone, low accuracy on Visual Creation reflects failure in text-based geometry or in image generation, not absence of multi-modal thought. No human accuracy and no ablation withholding the generated image are reported for this task. Please provide (i) a text-only variant that does not require or allow image generation and (ii) human accuracy on Visual Creation, to establish that generating an image is actually necessary for above-random performance.
[Abstract / §1 / §6] The paper repeatedly claims a 'huge performance gap between LVLMs and humans,' but no human performance is reported anywhere in the main text, tables, or appendix. Table 3 contains only a Random baseline; there is no Human row. Without a human accuracy measurement on the same samples and with the same answer-extraction procedures, the magnitude of the claimed gap is unsupported. This is directly load-bearing for the conclusion that current LVLMs are far below human-level CoMT ability. Please report human accuracy on all four tasks, with the same evaluation protocol as the models.
[§4.4 / Figure 6] The in-context-learning claim is stronger than the experimental setting supports. The ICL experiments are run only on Gemini-Pro, and Appendix D and the main text state that Gemini is limited to producing rationales in the textual modality alone. Therefore the observed improvement from providing multi-modal demonstrations shows that multi-modal input demonstrations help text-only reasoning; it does not demonstrate that models can learn to produce multi-modal output. The conclusions in §4.4 and §6 that ICL with 'multi-modal input and output' effectively promotes CoMT performance should be rephrased, or an ICL evaluation should be run on models that can actually emit images.

minor comments (7)

[Table 3] In the GILL Direct row, Visual Creation accuracy is printed as '16.9315.75', which should be two separate numbers (for example, 16.93 and 15.75).
[§2.2] The term 'boxing' should be 'bounding boxes' or 'boxing annotations'; as written it reads as a grammatical error introduced by the annotation pipeline.
[§4.2] The statement that VoT results in an 'average accuracy decrease of 12.28%' needs a defined baseline: it is not directly computable from Table 3, and for Gemini Direct versus VoT the accuracy actually increases (28.91 to 33.27). Please specify whether this is an average over models, which prompting baseline is used, and whether the decrease is absolute or relative.
[§4.3 / Figure 5] The text claims a 'significant positive correlation' between CLIPScore and accuracy, but no correlation coefficient, confidence interval, or significance test is reported. With the small number of points in each panel, please provide the correlation statistic or soften the claim.
[Appendix C.3] The image-quality scoring procedure states that only scores agreed upon by at least three annotators are considered valid, but it does not say how disagreements are resolved when fewer than three annotators agree. Please clarify whether such samples are discarded and report inter-annotator agreement for this scoring step.
[§2.4] The source dataset is referred to as 'spot-diff3' and 'spot-diff'; use one consistent name and provide a stable reference or URL for the dataset.
[Table 4 / Appendix F] The NExT-GPT Direct row lists the citation as 'Direct(?)'; this placeholder should be replaced with the proper reference.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CoMT is an external benchmark; the self-citations are procedural conventions, and the Visual Creation leakage concern is a validity risk, not a circular reduction.

full rationale

This paper introduces a benchmark and evaluates existing LVLMs on it. The central claim—that CoMT is the first benchmark requiring multi-modal reasoning output and that current LVLMs perform near random—is an empirical claim about an external evaluation artifact, not a derivation from fitted parameters. The benchmark construction is transparent: each task is built from existing datasets (GeoQA+, JHU-CROWD++, KILOGRAM, spot-diff) with template-based modification and human recheck, and the evaluation compares models against an independently computed random baseline by sampling options. No equation in the paper defines a predicted quantity in terms of the paper's own fitted values; the reported accuracies are direct model outputs on fixed multiple-choice questions. The self-citations (e.g., Chen et al. 2024b for answer extraction and the ICL prompt template, and Qin et al. 2023 for answer extraction) are procedural conventions and are not load-bearing for the benchmark's validity or for the empirical conclusion that current LVLMs perform poorly. The most substantive concern—Visual Creation removes the input image and augments the question with a full geometric text description, so the task may be solvable from text alone—is a validity or leakage question, not a circularity: it does not make the reported evaluation scores equal to the construction choices by definition, and the paper does not use this task as a fitted predictor of its own conclusions. Under the stated circularity criteria, no specific reduction can be exhibited, so the appropriate finding is no significant circularity. The score of 1 reflects only mild self-referentiality in adopting the authors' own M3CoT conventions for answer extraction and in-context-learning formatting, which does not affect the independent content of the benchmark evaluation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the paper is empirical benchmark construction. It relies on domain assumptions about what multi-modal thought should be, on the validity of its evaluation proxies, and on the completeness of its four operation categories.

assumptions (3)

domain assumption Human reasoning integrates visual operations at intermediate reasoning steps, and this is the correct target for LVLMs.
Section 1 and Appendix A.2 motivate CoMT from human visual-concrete versus abstract thought (Lehmann et al. 2010), but no formal argument or measurement establishes that LVLMs should imitate this or that it improves answer accuracy.
domain assumption The four visual operations (creation, deletion, update, selection) are fundamental and comprehensive categories for multi-modal thought.
Section 1 and Table 1 assert these four categories without a systematic derivation from a task taxonomy.
domain assumption CLIPScore and ROSCOE are valid proxies for multi-modal alignment and rationale quality, and the observed correlations imply a causal benefit.
Section 4.3 and Appendix E use CLIPScore and ROSCOE without reporting correlation coefficients or significance tests; the visual-quality human evaluation uses criteria defined by the authors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models." pith.science (2026). https://pith.science/paper/QVQWTDYN

@misc{pith2026241212932,
  author       = {Pith},
  title        = {Pith review of: CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVQWTDYN}},
  note         = {Machine review of arXiv:2412.12932}
}

read the original abstract

Large Vision-Language Models (LVLMs) have recently demonstrated amazing success in multi-modal tasks, including advancements in Multi-modal Chain-of-Thought (MCoT) reasoning. Despite these successes, current benchmarks still follow a traditional paradigm with multi-modal input and text-modal output, which leads to significant drawbacks such as missing visual operations and vague expressions. Motivated by this, we introduce a novel Chain of Multi-modal Thought (CoMT) benchmark to address these limitations. Different from the traditional MCoT benchmark, CoMT requires both multi-modal input and multi-modal reasoning output, aiming to mimic human-like reasoning that inherently integrates visual operation. Specifically, CoMT consists of four categories: (1) Visual Creation, (2) Visual Deletion, (3) Visual Update, and (4) Visual Selection to comprehensively explore complex visual operations and concise expression in real scenarios. We evaluate various LVLMs and strategies on CoMT, revealing some key insights into the capabilities and limitations of the current approaches. We hope that CoMT can inspire more research on introducing multi-modal generation into the reasoning process.

Figures

Figures reproduced from arXiv: 2412.12932 by the authors.

**Figure 2.** The overall annotation process for four tasks of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distribution of CoMT tasks across four types of image processing. plete sample by following the CoMT template, involving inquiries about the people count in the image (question) and clarifications of the identified count (rationale), etc. The prepared images serve as the visual thought within the rationale. 2.3 Visual Update Marking can help sort out the logic. LVLMs often make mistakes in reasoning due to forgetti… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Analysis of the correlation between the model per [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: CLIPScore of LVLMs on 4 tasks within CoMT. The x-axis represents the CLIPScore, and the y-axis represents the accuracy. 4.3 Analysis This section will conduct a further analysis on CoMT. See Technical Appendix E for more implementation details. Improving the quality o…

**Figure 7.** Figure 7: Image generation frequency during reasoning. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 9.** Figure 9: Comparison of paradigm between (a) traditional [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

**Figure 10.** Figure 10: Logical errors in textual statements for [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
cs.CV 2025-01 conditional novelty 6.0 of 10

A new multimodal reasoning benchmark shows that state-of-the-art AI models lag human experts by more than 30 percentage points, with visual reasoning errors as the main bottleneck.

Reference graph

Works this paper leans on

55 extracted references · 23 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966

arXiv 2023
[4]

Cao, J.; and Xiao, J. 2022. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, 1511--1520

work page 2022
[5]

Chen, Q.; Qin, L.; Jiaqi, W.; Jinxuan, Z.; and Che, W. 2024 a . Unlocking the Boundaries of Thought: A Reasoning Granularity Framework to Quantify and Optimize Chain-of-Thought. In Proc. of NeurIPS

work page 2024
[6]

Chen, Q.; Qin, L.; Zhang, J.; Chen, Z.; Xu, X.; and Che, W. 2024 b . M ^ 3 CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. arXiv preprint arXiv:2405.16473

arXiv 2024
[7]

Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 c . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24185--24198

2024
[8]

H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S

Dai, W.; Li, J.; Li, D.; Tiong, A. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500

arXiv 2023

Show all 55 references

[9]

Fei, H.; Li, B.; Liu, Q.; Bing, L.; Li, F.; and Chua, T.-S. 2023. Reasoning Implicit Sentiment with Chain-of-Thought Prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 1171--1182

2023
[10]

Fei, H.; Wu, S.; Ji, W.; Zhang, H.; Zhang, M.; Lee, M.-L.; and Hsu, W. 2024 a . Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning

2024
[11]

Fei, H.; Wu, S.; Zhang, H.; Chua, T.-S.; and Shuicheng, Y. 2024 b . Vitron: A unified pixel-level vision llm for understanding, generating, segmenting, editing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[12]

Fei, H.; Wu, S.; Zhang, M.; Zhang, M.; Chua, T.-S.; and Yan, S. 2024 c . Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence

2024
[13]

P.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A

Golovneva, O.; Chen, M. P.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A. 2023. ROSCOE : A Suite of Metrics for Scoring Step-by-Step Reasoning. In The Eleventh International Conference on Learning Representations

2023
[14]

He, L.; Li, Z.; Cai, X.; and Wang, P. 2024. Multi-modal latent space learning for chain-of-thought reasoning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18180--18187

2024
[15]

Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7514--7528

2021
[16]

K.; Hawkins, R

Ji, A.; Kojima, N.; Rush, N.; Suhr, A.; Vong, W. K.; Hawkins, R. D.; and Artzi, Y. 2022. Abstract visual reasoning with tangram shapes. arXiv preprint arXiv:2211.16492

2022 arXiv
[17]

Y.; Fried, D.; and Salakhutdinov, R

Koh, J. Y.; Fried, D.; and Salakhutdinov, R. 2023. Generating Images with Multimodal Language Models. NeurIPS

2023
[18]

S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Proc. of NeurIPS, 35: 22199--22213

2022
[19]

R.; and Koch, G

Landis, J. R.; and Koch, G. G. 1977. The measurement of observer agreement for categorical data. biometrics, 159--174

1977
[20]

Laurencon, H.; Tronchon, L.; Cord, M.; and Sanh, V. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246

2024 arXiv
[21]

Lee, J.; Wang, Y.; Li, J.; and Zhang, M. 2024. Multimodal Reasoning with Multimodal Knowledge Graph. arXiv preprint arXiv:2406.02030

2024 arXiv
[22]

D.; Strik, W

Lehmann, D.; Pascual-Marqui, R. D.; Strik, W. K.; and Koenig, T. 2010. Core networks for visual-concrete and abstract thought content: a brain electric microstate analysis. Neuroimage, 49(1): 1073--1079

2010
[23]

Li, X.; Lv, K.; Yan, H.; Lin, T.; Zhu, W.; Ni, Y.; Xie, G.; Wang, X.; and Qiu, X. 2023 a . Unified Demonstration Retriever for In-Context Learning. arXiv:2305.04320

2023 arXiv
[24]

Li, Y.; Wang, L.; Hu, B.; Chen, X.; Zhong, W.; Lyu, C.; and Zhang, M. 2023 b . A comprehensive evaluation of gpt-4v on knowledge-intensive visual question answering. arXiv preprint arXiv:2311.07536

2023 arXiv
[25]

Lin, W.; Wei, X.; An, R.; Gao, P.; Zou, B.; Luo, Y.; Huang, S.; Zhang, S.; and Li, H. 2024. Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want. arXiv preprint arXiv:2403.20271

2024 arXiv
[26]

Liu, B.; Lyu, C.; Min, Z.; Wang, Z.; Su, J.; and Wang, L. 2023. Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models. arXiv preprint arXiv:2312.01714

2023 arXiv
[27]

Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024 a . LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

2024
[28]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024 b . Visual instruction tuning. Proc. of NeurIPS, 36

2024
[29]

Lu, H.; Liu, W.; Zhang, B.; Wang, B.; Dong, K.; Liu, B.; Sun, J.; Ren, T.; Li, Z.; Yang, H.; Sun, Y.; Deng, C.; Xu, H.; Xie, Z.; and Ruan, C. 2024. DeepSeek-VL: Towards Real-World Vision-Language Understanding. arXiv:2403.05525

2024 arXiv
[30]

Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Proc. of NeurIPS, 35: 2507--2521

2022
[31]

Meng, F.; Yang, H.; Wang, Y.; and Zhang, M. 2023. Chain of Images for Intuitively Reasoning. arXiv preprint arXiv:2311.09241

2023 arXiv
[32]

Mondal, D.; Modi, S.; Panda, S.; Singh, R.; and Rao, G. S. 2024. KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning. arXiv preprint arXiv:2401.12863

2024 arXiv
[33]

Qin, L.; Chen, Q.; Fei, H.; Chen, Z.; Li, M.; and Che, W. 2024 a . What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration. arXiv preprint arXiv:2410.20482

2024 arXiv
[34]

Qin, L.; Chen, Q.; Feng, X.; Wu, Y.; Zhang, Y.; Li, Y.; Li, M.; Che, W.; and Yu, P. S. 2024 b . Large Language Models Meet NLP: A Survey. arXiv preprint arXiv:2405.12819

2024 arXiv
[35]

Qin, L.; Chen, Q.; Wei, F.; Huang, S.; and Che, W. 2023. Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2695--2709

2023
[36]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

2021
[37]

Schwenk, D.; Khandelwal, A.; Clark, C.; Marino, K.; and Mottaghi, R. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, 146--162. Springer

2022
[38]

A.; Yasarla, R.; and Patel, V

Sindagi, V. A.; Yasarla, R.; and Patel, V. M. 2020. Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5): 2594--2609

2020
[39]

Tan, C.; Wei, J.; Sun, L.; Gao, Z.; Li, S.; Yu, B.; Guo, R.; and Li, S. Z. 2024. Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning. arXiv preprint arXiv:2405.20834

2024 arXiv
[40]

M.; Hauth, A.; et al

Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

2023 arXiv
[41]

Wang, L.; Hu, Y.; He, J.; Xu, X.; Liu, N.; Liu, H.; and Shen, H. T. 2024 a . T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 191...

2024
[42]

Wang, W.; Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Zhu, J.; Zhu, X.; Lu, L.; Qiao, Y.; et al. 2024 b . Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442

2024 arXiv
[43]

Wang, Z.; Han, Z.; Chen, S.; Xue, F.; Ding, Z.; Xiao, X.; Tresp, V.; Torr, P.; and Gu, J. 2024 c . Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image. In Proc. of CoLM

2024
[44]

Wu, S.; Fei, H.; Qu, L.; Ji, W.; and Chua, T.-S. 2024 a . NE x T - GPT : Any-to-Any Multimodal LLM . In Proceedings of the International Conference on Machine Learning, 53366--53397

2024
[45]

Wu, W.; Mao, S.; Zhang, Y.; Xia, Y.; Dong, L.; Cui, L.; and Wei, F. 2024 b . Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models. arXiv preprint arXiv:2404.03622

2024 arXiv
[46]

C.; and Nie, Y

Wu, Y.; Zhang, P.; Xiong, W.; Oguz, B.; Gee, J. C.; and Nie, Y. 2023. The role of chain-of-thought in complex vision-language reasoning task. arXiv preprint arXiv:2311.09193

2023 arXiv
[47]

Xu, J.; Fei, H.; Pan, L.; Liu, Q.; Lee, M.-L.; and Hsu, W. 2024. Faithful Logical Reasoning via Symbolic Chain-of-Thought. arXiv preprint arXiv:2405.18357

2024 arXiv
[48]

Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; et al. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502

2023 arXiv
[49]

Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6720--6731

2019
[50]

Zhan, J.; Dai, J.; Ye, J.; Zhou, Y.; Zhang, D.; Liu, Z.; Zhang, X.; Yuan, R.; Zhang, G.; Li, L.; et al. 2024. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. arXiv preprint arXiv:2402.12226

2024 arXiv
[51]

Zhang, D.; Yang, J.; Lyu, H.; Jin, Z.; Yao, Y.; Chen, M.; and Luo, J. 2024 a . Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401.02582

2024 arXiv
[52]

Zhang, J.; Huang, J.; Jin, S.; and Lu, S. 2024 b . Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

2024
[53]

Zhang, Z.; Zhang, A.; Li, M.; Zhao, H.; Karypis, G.; and Smola, A. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923

2023 arXiv
[54]

Zheng, G.; Yang, B.; Tang, J.; Zhou, H.-Y.; and Yang, S. 2024. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Proc. of NeurIPS, 36

2024
[55]

Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966

arXiv 2023

[4] [4]

Cao, J.; and Xiao, J. 2022. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, 1511--1520

work page 2022

[5] [5]

Chen, Q.; Qin, L.; Jiaqi, W.; Jinxuan, Z.; and Che, W. 2024 a . Unlocking the Boundaries of Thought: A Reasoning Granularity Framework to Quantify and Optimize Chain-of-Thought. In Proc. of NeurIPS

work page 2024

[6] [6]

Chen, Q.; Qin, L.; Zhang, J.; Chen, Z.; Xu, X.; and Che, W. 2024 b . M ^ 3 CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. arXiv preprint arXiv:2405.16473

arXiv 2024

[7] [7]

Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 c . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24185--24198

2024

[8] [8]

H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S

Dai, W.; Li, J.; Li, D.; Tiong, A. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500

arXiv 2023