{"id":"e1f47c11-a9cc-4954-ab50-a138571e1ccf","arxiv_id":"2412.12932","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CoMT is the first benchmark to ask LVLMs to produce interleaved image and text rationales, and current models perform near random on it.","lead":"CoMT is a new benchmark that requires AI models to include visual outputs (drawing, deleting, updating, or selecting images) in their step-by-step reasoning, not just text. Most current vision-language models score near random on it, suggesting multi-modal reasoning is largely unsolved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual Creation is likely solvable from the augmented text alone; low accuracy may reflect image-generation failure, not absence of multi-modal thought, so the benchmark-wide negative claim needs an ablation.","rationale":"The paper's central value is a benchmark that requires multi-modal reasoning outputs. The Visual Creation task is the only one where the input image is removed and replaced by a text description (Appendix C.2). A strong text-only model can likely solve these geometry questions without generating an image, so the task does not isolate multi-modal thought. The paper's headline negative result (LVLMs near random) includes this task; if the task is solvable by text alone, the result conflates failures in image generation, text reasoning, and multi-modal integration. The proposed ablation would settle this. This does not invalidate the other three tasks, which keep images in the input and require visual operations, so conditional acceptance with a required ablation and softened claim is appropriate. The paper's other strengths (quality assurance, two-stage annotation, human evaluation of generated images) remain.","tokens_in":20493,"tokens_out":4526,"duration_ms":44378,"concrete_test":"Ablation on Visual Creation: take the 1,020 samples and run a strong LVLM (e.g., Gemini-Pro) in three conditions: (a) text-only input with no image-generation requirement, (b) text input plus the original GeoQA+ image and no image in the rationale, and (c) the CoMT format requiring image generation in the rationale. Compare accuracy across conditions. If (b) >> (c) and (a) ≈ (b), then image generation is not needed for solving, so Visual Creation does not test multi-modal reasoning; if (a) ≈ (c) << (b), then the low CoMT score comes from withholding the image, not from absence of multi-modal thought. Report human accuracy on (a) and (b) as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CoMT requires multi-modal reasoning output is most insecure for Visual Creation (§2.1, Figure 2a, Appendix C.2). The construction removes the image from the input and then augments the question with a text description that 'fully specifies' the geometry. This makes the task solvable from text alone: a model can reason about angles without ever generating an image. Low accuracy on this 1,020-sample task may therefore reflect failure at text-based geometry or at image generation, not failure of multi-modal thought. Since Visual Creation is one of the four 'fundamental operations', the benchmark-wide conclusion that current LVLMs lack multi-modal reasoning is not established unless it is shown that image generation is actually necessary for above-random performance. The paper reports no human accuracy on Visual Creation and no ablation withholding the generated image, so the validity of this task as a test of multi-modal reasoning remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoMT, a multiple-choice benchmark of 3,853 samples across four task families—Visual Creation, Visual Deletion, Visual Update, and Visual Selection—in which the model is expected to produce a rationale that interleaves generated or edited images with text. The authors construct the benchmark from four existing datasets, add human quality assurance, evaluate eight LVLMs under Direct, CoT, Desp-CoT, and VoT prompting, and report that most models score near or below the random baseline while Gemini achieves the best accuracy. They also report in-context-learning experiments, rationale-quality correlations, and error analyses, and they argue that current LVLMs lack the ability to perform chain of multi-modal thought.","tokens_in":20644,"tokens_out":4605,"duration_ms":46563,"significance":"If the benchmark's defining requirement is actually enforced, CoMT addresses a real gap: existing MCoT benchmarks require multimodal input but only text output, whereas CoMT aims to require visual operations as part of the reasoning chain. The paper has concrete strengths: transparent construction from existing datasets, detailed annotation guidelines, human quality control with reported agreement (kappa 0.93), a random baseline, and evaluation across a broad set of LVLMs and prompting schemes. The central negative finding—that current LVLMs are at or below chance on CoMT—is directly visible in the main tables and is not dependent on fitted parameters. However, the benchmark's validity rests on two assumptions that are not tested: that the automatic scoring enforces the production of a multimodal rationale, and that each task genuinely requires a visual thought. Both assumptions are load-bearing for the paper's main claim, so the significance can only be realized after those points are resolved.","major_comments":[{"comment":"The evaluation protocol does not enforce the benchmark's defining property. Final answers are extracted with regular expressions, and the reported accuracy and Macro-F1 scores do not check whether the model's rationale contains an image, whether that image is relevant to the question, or whether the answer was reached through the required visual operation. A model that outputs a text-only rationale, or no rationale at all, receives full credit for a correct final letter. Because CoMT is explicitly defined as requiring multimodal reasoning output, the current metric measures multiple-choice answer accuracy, not chain of multi-modal thought. Please add an evaluation mode that conditions credit on the presence of a valid visual thought (for example, answer accuracy restricted to samples where the model emitted a relevant image in the rationale), or otherwise demonstrate that the final-answer metric cannot be achieved by text-only reasoning.","section":"§4.1 / Appendix D.1"},{"comment":"Visual Creation is likely solvable from the augmented text alone, which undermines the benchmark-wide negative conclusion for this 1,020-sample task. The construction removes the image from the model input and supplements the question with a text description that fully specifies the geometry, with the image moved into the rationale as the supposed visual thought. If a model can derive the correct angle from the text alone, low accuracy on Visual Creation reflects failure in text-based geometry or in image generation, not absence of multi-modal thought. No human accuracy and no ablation withholding the generated image are reported for this task. Please provide (i) a text-only variant that does not require or allow image generation and (ii) human accuracy on Visual Creation, to establish that generating an image is actually necessary for above-random performance.","section":"§2.1 / Figure 2a / Appendix C.2"},{"comment":"The paper repeatedly claims a 'huge performance gap between LVLMs and humans,' but no human performance is reported anywhere in the main text, tables, or appendix. Table 3 contains only a Random baseline; there is no Human row. Without a human accuracy measurement on the same samples and with the same answer-extraction procedures, the magnitude of the claimed gap is unsupported. This is directly load-bearing for the conclusion that current LVLMs are far below human-level CoMT ability. Please report human accuracy on all four tasks, with the same evaluation protocol as the models.","section":"Abstract / §1 / §6"},{"comment":"The in-context-learning claim is stronger than the experimental setting supports. The ICL experiments are run only on Gemini-Pro, and Appendix D and the main text state that Gemini is limited to producing rationales in the textual modality alone. Therefore the observed improvement from providing multi-modal demonstrations shows that multi-modal input demonstrations help text-only reasoning; it does not demonstrate that models can learn to produce multi-modal output. The conclusions in §4.4 and §6 that ICL with 'multi-modal input and output' effectively promotes CoMT performance should be rephrased, or an ICL evaluation should be run on models that can actually emit images.","section":"§4.4 / Figure 6"}],"minor_comments":[{"comment":"In the GILL Direct row, Visual Creation accuracy is printed as '16.9315.75', which should be two separate numbers (for example, 16.93 and 15.75).","section":"Table 3"},{"comment":"The term 'boxing' should be 'bounding boxes' or 'boxing annotations'; as written it reads as a grammatical error introduced by the annotation pipeline.","section":"§2.2"},{"comment":"The statement that VoT results in an 'average accuracy decrease of 12.28%' needs a defined baseline: it is not directly computable from Table 3, and for Gemini Direct versus VoT the accuracy actually increases (28.91 to 33.27). Please specify whether this is an average over models, which prompting baseline is used, and whether the decrease is absolute or relative.","section":"§4.2"},{"comment":"The text claims a 'significant positive correlation' between CLIPScore and accuracy, but no correlation coefficient, confidence interval, or significance test is reported. With the small number of points in each panel, please provide the correlation statistic or soften the claim.","section":"§4.3 / Figure 5"},{"comment":"The image-quality scoring procedure states that only scores agreed upon by at least three annotators are considered valid, but it does not say how disagreements are resolved when fewer than three annotators agree. Please clarify whether such samples are discarded and report inter-annotator agreement for this scoring step.","section":"Appendix C.3"},{"comment":"The source dataset is referred to as 'spot-diff3' and 'spot-diff'; use one consistent name and provide a stable reference or URL for the dataset.","section":"§2.4"},{"comment":"The NExT-GPT Direct row lists the citation as 'Direct(?)'; this placeholder should be replaced with the proper reference.","section":"Table 4 / Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially useful and the random-baseline comparison is transparent, but the central claim that CoMT requires and measures chain of multi-modal thought is currently not operationalized in the metric, and the Visual Creation task is at risk of being solvable without any visual thought. These issues are fixable with an ablation, a metric that conditions on the presence of a valid visual rationale, and a human baseline; they do not require discarding the benchmark. The paper also overclaims the ICL result given that the only ICL model cannot emit images. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoMT is worth knowing about: it's the first benchmark that asks for interleaved image-and-text rationales, across four visual operation types (create, delete, update, select). The construction is transparent—built on GeoQA+, JHU-CROWD++, KILOGRAM, and spotdiff—and the annotation pipeline has real quality controls (onboarding, double recheck, kappa 0.93). The main empirical finding, that almost every tested LVLM hovers near random while only Gemini edges slightly above, is clearly displayed and easy to believe. That alone makes the benchmark a useful stress test for multimodal reasoning research.\n\nThe soft spots are real, though they don't sink the whole thing. First, Visual Creation removes the image from the input and augments the text to fully describe the diagram (Section 2.1, Appendix C.2). That means the task is solvable from text alone; a model can do the geometry without ever generating an image. Low accuracy there could reflect text-based geometry failure or image-generation failure, not missing multi-modal thought. The authors should report human accuracy on this task and run an ablation that withholds the generated image to show it is actually needed. Second, the automatic evaluation extracts only the final answer option; it never checks whether the model produced a multi-modal rationale. So the accuracy numbers measure question-answering, not the presence of multi-modal reasoning. The paper's language—'requires both multi-modal input and multi-modal reasoning output'—describes the dataset format, not what the evaluation enforces. That weakens the causal reading that low scores are due to a lack of multi-modal thought. Third, human performance is claimed in the abstract and intro but no human numbers appear anywhere in the paper. That's an easy fix and should be mandatory before this goes out for real.\n\nThe ICL and CLIPScore correlations are suggestive but lack error bars or significance tests; treat them as qualitative. The paper also cites the relevant prior work (M3CoT, ScienceQA, VoT) and doesn't overclaim novelty about the underlying datasets.\n\nBottom line: this is a solid benchmark contribution with a clear negative result, but the central interpretation needs tightening. I'd send it to peer review and ask for the Visual Creation ablation, human numbers, and a softer claim about what the accuracy scores demonstrate. It's a useful resource for anyone evaluating LVLMs on visual reasoning.","headline":"Useful new multimodal reasoning benchmark and a striking negative result, but the Visual Creation task is likely solvable from text alone and the accuracy metric never enforces multi-modal output, so the blanket claim that LVLMs lack multi-modal thought needs an ablation and softer wording.","tokens_in":21192,"tokens_out":3637,"would_cite":true,"duration_ms":34572,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces CoMT, the first benchmark that forces large vision-language models to reason in images as well as text, and shows that current models score near random chance on it.","keywords":["chain of multi-modal thought","benchmark","large vision-language models","visual reasoning","multi-modal chain-of-thought","in-context learning","visual operations","CLIPScore alignment"],"falsifier":"Give a text-only LLM the Visual Creation questions with their augmented text descriptions but without the original diagram; if it matches or exceeds the accuracy of image-generating LVLMs on those items, then low Visual Creation scores measure image-generation failure rather than the absence of multimodal reasoning.","tokens_in":20322,"feed_emoji":"🖼️","tokens_out":6457,"duration_ms":54884,"temperature":0.7,"pith_summary":"This paper argues that current benchmarks for multimodal reasoning are too easy because they accept text-only rationales, letting models describe visual operations without ever performing them. To close that gap, it introduces CoMT, a benchmark of 3,853 questions with 14,801 images that demands the model's reasoning chain itself be multimodal—interleaving generated or edited images with text across four visual operations: creation, deletion, update, and selection. The paper's central finding is that essentially all current LVLMs score at or near random chance on CoMT (best accuracy 33.44% versus a 25.48% random baseline), with only Gemini-Pro beating chance by a small margin. It also finds that text-only chain-of-thought prompting fails to help, that in-context learning with multimodal demonstrations helps up to about four shots, and that accuracy tracks both rationale quality and a CLIPScore measure of how well generated images align with the text. If the benchmark is a fair instrument, then today's LVLMs lack a core human reasoning ability: manipulating an image as part of thinking.","feed_headline":"AI models score near random when reasoning must include images","feed_subtitle":"On a new benchmark, CoMT, models must generate images as reasoning steps; today's best barely beat chance.","key_machinery":"The load-bearing object is the CoMT benchmark itself, together with its formal definition of multi-modal thought. In the paper's formulation (Equation 2), a chain of multi-modal thought is a sequence of rationale steps in which each step is chosen as either a visual generation or a text token, with the visual step taken when the model's probability of producing a useful visual rationale exceeds that of a textual one. The benchmark operationalizes this through a template with five keys—question, options, image, rationale, and answer—and four task pipelines that convert existing datasets into samples whose rationales interleave images with text. The paper also introduces an evaluation lens: CLIPScore between generated images and the reference images in the ideal rationale serves as a multi-modal alignment score, ROSCOE measures rationale quality, and human scoring (relevance plus logical correctness) on a sampled subset quantifies how often generated images are useless or inconsistent with the text.","core_discovery":"The central claim is that CoMT is the first benchmark to require chain of multi-modal thought in LVLMs, meaning both the input and the reasoning output are multimodal. The paper constructs four task families from existing datasets: Visual Creation (from GeoQA+ geometry problems, moving the diagram into the rationale so the model must generate it), Visual Deletion (from JHU-CROWD++, masking faces stepwise so the model must remove visual clutter), Visual Update (from KILOGRAM tangrams, progressively coloring and annotating pieces), and Visual Selection (from spot-the-difference pairs, cropping and highlighting differences). Under this benchmark, the paper reports that almost all tested LVLMs—including image-capable models such as GILL, NExT-GPT and AnyGPT—perform near random chance, and that explicit attempts to make models visualize (VoT) or describe (Desp-CoT) before answering generally reduce accuracy. The paper therefore concludes that current models fail to execute visual operations inside the reasoning chain and that progress will require integrating multi-modal generation, textual logic, and visual logic.","pith_inferences":["A text-only language model given the augmented textual descriptions used in Visual Creation might solve many questions without generating any image; if so, low scores on that task would partly reflect image-generation failure rather than missing multimodal reasoning, and the benchmark should be re-validated with text-only baselines on the full task text.","The positive CLIPScore-accuracy correlation suggests CoMT could be turned into a training objective: models could be rewarded for emitting rationales whose images align with their text, effectively learning visual logic by imitation.","The four visual operations are general enough to inspire analogous benchmarks in other modalities—audio editing or video manipulation—and in embodied settings where an agent must modify its environment to reason.","A testable extension is whether fine-tuning on CoMT's multimodal rationales transfers to existing benchmarks like ScienceQA or M3CoT; if transfer is positive, CoMT is not just a diagnostic but a curriculum."],"forward_implications":["Existing MCoT benchmarks, which accept text-only rationales, likely overstate LVLMs' reasoning ability because they never force the model to perform a visual operation mid-chain.","Text-only prompting methods—CoT, Desp-CoT, and VoT—are not a path to multimodal reasoning; they can even push models below random accuracy.","Multi-modal in-context learning is the most promising lever identified, but it saturates at about four demonstrations and then degrades, so context-length effects matter.","Accuracy on CoMT correlates with the quality of the rationale and with visual-text alignment, suggesting that improving either component should directly improve performance.","Models trained on image generation are not automatically good at CoMT; the gap between generation and reasoning means the two capabilities must be co-designed."],"supporting_citations":[{"why":"provides the GeoQA+ geometry questions and diagrams that become the Visual Creation task","marker":"Cao and Xiao 2022"},{"why":"supplies the crowd images and face boxes used to build Visual Deletion's stepwise masking","marker":"Sindagi, Yasarla, and Patel 2020"},{"why":"supplies tangram images and piece/whole labels used for Visual Update's progressive annotation","marker":"Ji et al. 2022"},{"why":"M3CoT is the multi-step multimodal benchmark CoMT compares against and whose domains it extends","marker":"Chen et al. 2024b"},{"why":"ScienceQA is the standard MCoT benchmark whose paradigm CoMT argues is insufficient","marker":"Lu et al. 2022"},{"why":"introduces multimodal chain-of-thought reasoning, the paradigm CoMT redefines to include visual output","marker":"Zhang et al. 2023"},{"why":"CLIPScore is the metric used to measure alignment between generated images and reference rationales","marker":"Hessel et al. 2021"},{"why":"ROSCOE provides the rationale-quality metrics used to correlate reasoning quality with accuracy","marker":"Golovneva et al. 2023"},{"why":"provides the zero-shot CoT prompt ('Let's think step by step') used as a baseline strategy","marker":"Kojima et al. 2022"},{"why":"VoT is the visualize-your-reasoning baseline that CoMT shows fails because it stays in text","marker":"Wu et al. 2024b"}],"fun_headline_variants":["AI fails new benchmark that forces image-based reasoning","When thinking must include images, top AI drops to chance","CoMT benchmark: models can't generate reasoning images","Forcing AI to draw its reasoning: near random scores","New benchmark exposes AI's missing visual thinking step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's Visual Creation task assumes that a model cannot answer a geometry question from the written description alone; if the added text fully specifies the diagram, then the task tests image generation, not multi-modal reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI fails new benchmark that forces image-based reasoning","When thinking must include images, top AI drops to chance","CoMT benchmark: models can't generate reasoning images","Forcing AI to draw its reasoning: near random scores","New benchmark exposes AI's missing visual thinking step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000563,"raw_usage":{"total_tokens":2678,"prompt_tokens":957,"completion_tokens":1721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1645}},"tokens_in":573,"tokens_out":1721,"duration_ms":9732,"temperature":1.0,"reasoning_tokens":1645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:34:39.890718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a text-only LLM the Visual Creation questions with their augmented text descriptions but without the original diagram; if it matches or exceeds the accuracy of image-generating LVLMs on those items, then low Visual Creation scores measure image-generation failure rather than the absence of multimodal reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the GeoQA+ geometry questions and diagrams that become the Visual Creation task"},{"cited_title":"A.; Yasarla, R.; and Patel, V","cited_arxiv_id":null,"evidence_quote":"supplies the crowd images and face boxes used to build Visual Deletion's stepwise masking"},{"cited_title":"P.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A","cited_arxiv_id":null,"evidence_quote":"ROSCOE provides the rationale-quality metrics used to correlate reasoning quality with accuracy"},{"cited_title":"S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y","cited_arxiv_id":null,"evidence_quote":"provides the zero-shot CoT prompt ('Let's think step by step') used as a baseline strategy"}],"review_version":1}