{"id":"bd8e52cb-73bc-428e-9aad-43b66f289fbd","arxiv_id":"2502.09621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new multimodal benchmark evaluates chain-of-thought reasoning quality, robustness, and efficiency in large vision-language models, finding CoT often hurts perception tasks and that many reflection steps are unhelpful.","lead":"MME-CoT is a new benchmark for measuring how well multimodal AI models reason step by step, testing quality, robustness and efficiency across six question domains. It finds that some reasoning-focused models produce high-quality steps but waste effort on irrelevant reflections, and that asking models to 'think step by step' often hurts simple perception tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Kimi k1.5's top F1 (64.2) is computed on a 150-question subset, while GPT-4o's (64.0) is on all 1,130; without proof the subset is representative, the abstract's headline ranking is unsupported.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already lists 'a subset-evaluated closed model' as one of several concerns. I agree that this is a real problem, but I believe it is the single most load-bearing issue: it directly undermines the paper's most prominent conclusion (Kimi k1.5 > GPT-4o in CoT quality), and it is independently verifiable without needing to resolve the deeper question of GPT-4o judge reliability. The judge-validity concern raised by the reader is important but applies to all models roughly uniformly; the subset mismatch is a structural confound specific to the headline comparison. A 0.2 F1 difference on different question sets is not interpretable, and the paper provides no evidence of subset representativeness. The requested check would settle this cleanly. Since the issue can be fixed by releasing the subset and re-running both models on it, the CONDITIONAL verdict remains appropriate rather than moving to REJECT.","tokens_in":35485,"tokens_out":5707,"duration_ms":57976,"concrete_test":"Request the authors release the exact 150-question Kimi evaluation subset. Run GPT-4o (and QVQ-72B) with the same CoT prompt on that subset and compute the paper's F1 quality metric using the identical GPT-4o judging pipeline. Also bootstrap-resample the 115 reasoning questions to obtain a confidence interval for the F1 difference. If GPT-4o's F1 on the subset is ≥ Kimi's 64.2, or if the 95% CI for the difference includes 0, the headline 'Kimi k1.5 outperforms GPT-4o' should be withdrawn or qualified. Additionally, compare the domain distribution of the subset to the full benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that Kimi k1.5 was evaluated on 150 sampled questions (115 reasoning, 35 perception) due to access limits, yet Table 2 reports its CoT-quality F1 (64.2) against GPT-4o's F1 (64.0) computed on the full 1,130-question set. CoT quality is measured only on reasoning questions, so the comparison is Kimi on ~115 reasoning items versus GPT-4o on 837. The paper does not describe the sampling procedure, does not report which domains or subcategories the 150 questions cover, and does not provide per-domain results for Kimi in Table 3. The 0.2-point F1 gap is also far smaller than the sampling error one would expect for 115 scoring items (roughly ±3-5 points at 95% confidence). Consequently the central claim that 'Kimi k1.5 outperforming GPT-4o' (abstract, Section 4.2) is not established; the ranking could simply reflect an easier subset. The same issue also affects the reported reflection-quality and relevance scores for Kimi.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MME-CoT introduces a 1,130-question benchmark for evaluating chain-of-thought (CoT) reasoning in large multimodal models, spanning six domains (math, science, OCR, logic, space-time, general scenes) with human-verified key-step annotations and reference image captions. The paper proposes three evaluation axes: CoT quality (recall and precision of predicted steps against annotated key steps, aggregated as F1), CoT robustness (stability on perception tasks and efficacy on reasoning tasks, computed as accuracy differences between direct and CoT prompting), and CoT efficiency (relevance rate, rescaled by Eq. (10) with alpha=0.8, and reflection quality). Thirteen LMMs are evaluated, leading to three headline findings: reflection-equipped models achieve the highest CoT quality with Kimi k1.5 slightly above GPT-4o, CoT prompting degrades perception-heavy tasks for most models, and reflection models are inefficient with a large fraction of invalid reflection steps.","tokens_in":35873,"tokens_out":3619,"duration_ms":39666,"significance":"If the evaluation pipeline is trustworthy, MME-CoT would be a valuable community resource: it is the first multimodal CoT benchmark to move beyond final-answer accuracy and to measure process-level quality, robustness, and efficiency at a fine-grained step resolution. The dataset curation is a genuine strength: key steps and reference captions were human-verified, statistics are reported, and the full evaluation prompts are provided in Appendix E, which supports reproducibility. The error analysis of reflection failure modes (ineffective reflection, incompleteness, repetition, interference) is also useful and actionable. However, the paper's central ranking claim and several secondary comparisons rest on an unvalidated GPT-4o judge pipeline and on an unevaluated 150-question subset for Kimi k1.5, so the quantitative conclusions are currently not established at the claimed precision.","major_comments":[{"comment":"The headline claim that Kimi k1.5 outperforms GPT-4o in CoT quality is not supported by the reported evidence. Section 4.1 states that Kimi k1.5 was evaluated on 150 sampled questions (115 reasoning, 35 perception) due to access limits, yet Table 2 reports Kimi's F1 of 64.2 against GPT-4o's 64.0 computed on the full benchmark. Since CoT quality is computed only on reasoning questions, this is effectively Kimi on about 115 reasoning items versus GPT-4o on 837. The sampling procedure is not described, no per-category or per-domain results for Kimi are given (Kimi is absent from Table 3), and no confidence intervals are reported. A gap of 0.2 F1 points is far smaller than the sampling error expected for roughly 115 items, so the ranking could easily be an artifact of subset selection. Please report the sampling method, per-category breakdown for Kimi, and uncertainty estimates, or remove the superiority claim from the abstract and Section 4.2.","section":"Section 4.1, Table 2, abstract"},{"comment":"Every process-level metric depends on GPT-4o's ability to partition model outputs into atomic steps, match predicted steps to human key steps, judge step correctness, judge relevance, and identify and validate reflection steps. The authors validate the dataset annotations but do not report any human validation or agreement study for these GPT-4o judgments. Because the key ranking differences are small (Kimi k1.5 F1 64.2 vs GPT-4o 64.0; reflection quality 72.2 vs 61.7 for QVQ), systematic judge bias could reorder the conclusions. Please add a human agreement study on a sample of the step-matching, relevance, and reflection judgments, or at least report the judge's self-consistency and show that the qualitative findings are robust to a noisy judge.","section":"Sections 3.1, 3.3, Appendix E"},{"comment":"The relevance rate is rescaled via (r - alpha)/(1 - alpha) with a hand-picked alpha=0.8, but no justification or sensitivity analysis is provided. This transform nonlinearly amplifies small differences near the top of the raw distribution and can change model rankings. Please report the raw relevance rates alongside the rescaled ones and show that the conclusions (e.g., InternVL2.5-8B having the highest relevance, long-CoT models being more distractible) are stable across a range of alpha values.","section":"Eq. (10), Section 3.3"},{"comment":"Reflection quality is set to 100 for all models that do not reflect, as stated in Section 4.2 ('we define the reflection quality as 100 on models incapable of reflection'). This makes 'no reflection' numerically indistinguishable from 'perfect reflection' and biases the efficiency comparison in favor of non-reflective models. The paper's third claim, that reflection models are inefficient, is thus partially built into the metric definition. Please either exclude non-reflective models from the reflection-quality analysis, report the number of reflection steps separately, or clearly relabel the 100 entries as 'not applicable' rather than a score of 100.","section":"Section 4.2, Table 2"},{"comment":"The robustness scores for Mulberry, LLaVA-CoT, Virgo, QVQ, and Kimi k1.5 are flagged with asterisks as unreliable because these models refuse to answer directly and instead generate extended rationales. Nevertheless, the paper still reports their stability and efficacy values as if comparable, and the text states that 'GPT-4o achieves the highest robustness score' after acknowledging that several other scores may be misleading. Mixing unreliable and reliable numbers in the same table and in the summary statistics can mislead readers. Please either exclude these rows from the quantitative robustness comparison and discuss them only qualitatively, or provide a corrected scoring protocol for models that ignore the direct prompt.","section":"Section 4.2, Table 2 footnote"}],"minor_comments":[{"comment":"The paper contains several typographical issues, including 'MME-C OT' in Table 1 and Table 2 captions, 'Rea.' in Section 4.2, and 'Nan-Multiple-Choice' in Appendix E.3. These should be corrected before publication.","section":"Throughout"},{"comment":"Kimi k1.5 is missing from the per-category results in Table 3, even though the overall quality and efficiency numbers for Kimi are central to the paper's claims. Adding per-category results (or a clear explanation of why they cannot be provided) is necessary to assess the 150-question subset.","section":"Table 3"},{"comment":"The two-stage perception/reasoning classification uses GPT-4o and Qwen2-VL-7B accuracy differences to guide the preliminary categorization, and GPT-4o later serves as the judge for the process-level metrics. This is not circular in a formal sense because final labels are human-verified, but the potential dependence of the task split on the same model family used for evaluation should be acknowledged explicitly.","section":"Section 2.1"},{"comment":"Figure 1 contains many numbers and arrows in a small space and is difficult to parse; consider simplifying it or moving the detailed scores to a table.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and dataset are potentially useful to the community, and the qualitative findings are interesting. The main risk is overclaiming: the Kimi-versus-GPT-4o ranking is not supported without sampling-aware statistics, and the GPT-4o judge is not validated. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I've read MME-CoT. Short version: the evaluation suite is genuinely useful and the dataset looks carefully made, but the headline claim that Kimi k1.5 beats GPT-4o on CoT quality is not supported by the numbers as reported.\n\nWhat's new: the paper gives the LMM community a process-level benchmark with three families of metrics—recall/precision/F1 for step quality, stability/efficacy for robustness, relevance rate/reflection quality for efficiency—and applies them across 1,130 human-validated questions in six domains. That's a real step beyond final-answer benchmarks like MathVerse or M3CoT. The findings that CoT hurts perception-heavy tasks and that reflection models waste a lot of steps are important and well-documented, and the paper is admirably honest about cases where models refuse to follow the direct prompt.\n\nThe big soft spot is the Kimi k1.5 comparison. Kimi was evaluated on a 150-question subset (115 reasoning, 35 perception), yet Table 2 reports its F1 of 64.2 against GPT-4o's 64.0 on the full set of 837 reasoning questions. A 0.2-point gap on ~115 items is well within sampling error—you'd expect ±3-5 points. The paper doesn't describe the sampling procedure or give per-domain results for Kimi, so the abstract's 'Kimi k1.5 outperforming GPT-4o' is essentially unsupported. This is a load-bearing flaw for the first of the three key insights. The other two insights rest on more complete data and look solid.\n\nSecondary issues: all the step matching, relevance, and reflection judgments come from GPT-4o, and there's no human validation of those judgments. The prompts are in the appendix and the qualitative examples look reasonable, but I'd want to see agreement numbers on a sample. The alpha=0.8 rescaling of relevance rate is hand-chosen, though it's monotone so it only rescales, not reorders. Setting reflection quality to 100 for non-reflecting models is a bit odd but explicit.\n\nWho's this for: anyone evaluating or building LMMs with long CoT, especially o1-style reflection models. The dataset and suite are a real resource. It deserves peer review, but the revision needs to address the subset issue (evaluate Kimi on the full set or report its results with confidence intervals) and validate a sample of the GPT-4o judgments.\n\nMy recommendation: send it to review as a major revision, not a desk reject.","headline":"Genuinely useful multimodal CoT evaluation suite, but the Kimi-vs-GPT-4o headline is undercut by a 150-question subset comparison.","tokens_in":36316,"tokens_out":3184,"would_cite":true,"duration_ms":31143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MME-CoT is a benchmark that evaluates chain-of-thought in large multimodal models, claiming that reflection-equipped models reason best, CoT often degrades perception tasks, and reflective models are inefficient.","keywords":["chain-of-thought","multimodal reasoning","benchmark","large multimodal models","reflection quality","perception robustness","reasoning evaluation"],"falsifier":"Take a random sample of the judged outputs (say 200 model responses), have human annotators redo the step matching, relevance, and reflection-validity judgments, and measure agreement with GPT-4o's judgments; if agreement is below the paper's annotation reliability level, or if the re-judged F1 changes the Kimi k1.5 vs GPT-4o order, the central ranking claim is not settled.","tokens_in":35296,"feed_emoji":"🧠","tokens_out":4235,"duration_ms":38440,"temperature":0.7,"pith_summary":"MME-CoT is a benchmark for how large multimodal models reason step by step. It measures not just whether final answers are right but whether each intermediate step is informative, faithful, relevant, and whether reflecting on previous steps helps. Across six visual domains, the authors find that reflection-equipped models such as Kimi k1.5 produce the highest-quality reasoning, that forcing step-by-step thinking often degrades pure perception tasks, and that reflective models waste large fractions of their output. The paper's value is a process-level view of where chain-of-thought helps and where it backfires.","feed_headline":"Chain-of-thought hurts perception in most vision-language models","feed_subtitle":"Reflection models like Kimi k1.5 lead MME-CoT on quality, yet 25–40% of their reflection steps are useless.","key_machinery":"The evaluation suite is built on human-annotated key steps—inference conclusions plus image captions that a correct solution must contain—and on a GPT-4o-based step partitioner that splits each model output into atomic steps typed as logical inference, image caption, or background. Quality is recall (fraction of key steps matched) and precision (fraction of generated key steps correct), combined as F1. Robustness compares direct-answer versus step-by-step prompting on perception tasks (stability) and reasoning tasks (efficacy). Efficiency is relevance rate (share of steps that work toward the solution, rescaled with a floor of 0.8) and reflection quality (share of identified reflection steps judged valid). These three pairs of metrics are the mechanism that turns raw model outputs into the paper's findings.","core_discovery":"On its own terms, the paper establishes that process-level evaluation changes the ranking of vision-language models relative to answer-only benchmarks. Kimi k1.5 edges GPT-4o on CoT quality (F1 64.2 vs 64.0), mostly through precision rather than recall; GPT-4o retains the highest recall. Most tested models lose accuracy on perception tasks when asked to reason step by step (e.g., InternVL2.5-8B drops 6.8%), while only some gain on reasoning tasks. On efficiency, even the best reflective models earn low relevance rates and reflection quality around 60-72%, meaning roughly a quarter to two-fifths of reflection steps do not advance the solution. The authors interpret these as actionable failure modes: overthinking, distraction by irrelevant image detail, and reflection that repeats, degrades, or aborts.","pith_inferences":["One testable extension is to rerun the quality metrics with a second judge (e.g., a different model or human raters) and measure judge-induced rank shifts; the close Kimi/GPT-4o gap makes this meaningful.","The stability result suggests a potential training signal: penalizing CoT-generated steps that are irrelevant to perception questions could recover the lost accuracy; the authors do not propose this.","The relevance-rate rescaling with alpha=0.8 amplifies small differences; a sensitivity analysis over alpha would show whether the efficiency rankings are an artifact of that choice.","Because the benchmark samples 150 questions for Kimi k1.5, comparing its scores to models evaluated on all 1,130 questions assumes subset representativeness; re-running on a matched subset would test this."],"forward_implications":["If step quality is the criterion, reflection-based training is a promising direction: Kimi k1.5 and QVQ outscore their non-reflective counterparts on F1.","Deploying CoT universally is unsafe for perception-heavy workloads until stability is fixed; the paper's stability metric can be used as a regression test.","Efficiency budgets in long-CoT systems can be set by relevance and reflection-quality scores rather than by output length alone.","Answer-only benchmarks may overstate reasoning ability when models reach correct answers by skipping key steps."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting, the capability the benchmark systematically evaluates.","marker":"(Wei et al., 2022)"},{"why":"Defines o1-style long thinking and reflection, motivating the efficiency and reflection-quality metrics.","marker":"(OpenAI, 2024a)"},{"why":"DeepSeek-R1 represents the reflection-equipped reasoning paradigm whose multimodal extensions are under test.","marker":"(Guo et al., 2025a)"},{"why":"MathVerse supplies data and the two-step answer-extraction protocol reused for direct evaluation.","marker":"(Zhang et al., 2024c)"},{"why":"Virgo is one of the reflection-capable open-source models evaluated and analyzed.","marker":"(Du et al., 2025)"},{"why":"Qwen2-VL is the base model whose reasoning-tuned variant QVQ is compared against.","marker":"(Qwen Team, 2024)"},{"why":"QVQ is the main reflection-capable open-source model whose error patterns are studied in detail.","marker":"(Team, 2024)"},{"why":"Kimi k1.5 is the top-scoring closed-source model on CoT quality and a central comparison point.","marker":"(Team et al., 2025)"},{"why":"ReCEval frames process-level reasoning evaluation through correctness and informativeness, which the paper extends to multimodal settings.","marker":"(Prasad et al., 2023)"}],"fun_headline_variants":["Step-by-step thinking degrades image perception in LMMs","Reflection models waste up to 40% of reasoning steps","Kimi k1.5 tops quality, not speed, on multimodal CoT","Overthinking harms perception-heavy multimodal tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every trust in the rankings rests on GPT-4o correctly partitioning outputs into steps and faithfully matching and judging those steps; if it systematically misjudges, the model order can change, especially where scores are close.","fun_headline_variants_meta":{"raw":{"variants":["Step-by-step thinking degrades image perception in LMMs","Reflection models waste up to 40% of reasoning steps","Kimi k1.5 tops quality, not speed, on multimodal CoT","Overthinking harms perception-heavy multimodal tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3837,"prompt_tokens":992,"completion_tokens":2845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2775}},"tokens_in":608,"tokens_out":2845,"duration_ms":22113,"temperature":1.0,"reasoning_tokens":2775,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:47:57.488176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the judged outputs (say 200 model responses), have human annotators redo the step matching, relevance, and reflection-validity judgments, and measure agreement with GPT-4o's judgments; if agreement is below the paper's annotation reliability level, or if the re-judged F1 changes the Kimi k1.5 vs GPT-4o order, the central ranking claim is not settled.","supporting_citations":[],"review_version":1}