{"id":"adde4872-2029-4347-9fa0-6c05b7e8dc89","arxiv_id":"2505.21523","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reasoning models trade visual grounding for language-based inference, and this paper measures that trade-off with a new metric and benchmark.","lead":"Multimodal reasoning models that think before answering become less accurate at seeing: they hallucinate more on visual perception questions than their non-reasoning base models, and extra thinking makes this worse. The paper introduces a metric (RH-AUC) and a benchmark (RH-Bench) to measure the trade-off between reasoning quality and visual grounding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Latent state steering is fit on the same benchmark used for evaluation, so the non-monotonic length curves and attention decline may reflect content confounds, not reasoning length; validation with held-out steering directions or alternative length controls is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing point, and I agree with it. The paper's first descriptive claim—trained multimodal reasoning models hallucinate more than their Qwen2.5-VL base models on perception benchmarks—rests on direct comparisons across MMVP, MMEval-Pro, VMCBench, Bingo, and MMHAL, so it does not depend on the steering method; it is credible independent support for part of the central claim. The length-related claims, however, are the part that makes the paper's message about test-time compute distinctive, and they collapse to a single intervention: latent state steering. Because d_l is fit on the evaluation benchmark itself, and because the paper does not validate that alpha is a pure length controller or show that budget forcing and test-time scaling reproduce the same curves, the observed non-monotonicity and attention shift could be caused by whatever content distinguishes the long and short traces in the steering pool. This is a circularity in the evaluation of the length effect, not an internal inconsistency in equations, and it can be fixed with more careful validation. The hallucination-amplification finding for released models and the RH-Bench/RH-AUC contribution remain useful even if the steering-based length analysis is set aside, which is why I keep the conditional verdict rather than moving to reject. The condition should be: hold out the steering-development split, match traces on correctness and content, and verify that at least one non-steering length-control method reproduces the non-monotonic curves and attention decline.","tokens_in":13317,"tokens_out":7280,"duration_ms":79168,"concrete_test":"Recompute the Figure 7 length sweep using d_l derived only from a held-out steering-development split of RH-Bench, with long and short traces matched on final-answer correctness, and compare the resulting accuracy-versus-length curves and attention maps with those obtained under budget forcing and test-time scaling on the same models and items. If the non-monotonic shape or the visual-attention decline changes materially, or fails to replicate under either non-steering length-control method, then the reported effects are artifacts of the steering direction rather than genuine consequences of reasoning length.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reasoning-length conclusions in Section 3.2 and Section 4.2 rest on Latent State Steering (Eq. 1-2), but the steering direction d_l is computed from long and short reasoning traces drawn from the same RH-Bench/test benchmark on which accuracy is later measured. Nothing in the paper shows that d_l encodes reasoning length rather than answer correctness, topic, or hallucination content of the traces in the development pool. If long traces differ systematically in content, adding alpha times d_l changes what the model generates, not just how long it reasons, making the non-monotonic accuracy curves in Figure 7 and the visual-attention decline in Figure 6 potential artifacts of the steering direction. The text asserts that increasing alpha extends reasoning length but reports no validation of monotonic control on held-out items, and no comparison with budget forcing or test-time scaling, even though Section 4.1 introduces them as alternative controls. The causal claim that longer reasoning chains reduce visual attention and increase hallucination is therefore not yet established; what is established is a correlation under a specific, unvalidated steering intervention.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies hallucination in multimodal reasoning models that generate extended reasoning chains. It reports that reasoning models hallucinate more on visual perception tasks than their non-reasoning base models, attributes this to reduced attention on visual tokens, and investigates how reasoning-chain length affects the reasoning-hallucination balance. The authors introduce RH-AUC, a metric that summarizes the trade-off between reasoning accuracy and hallucination across reasoning lengths, and RH-Bench, a 1,000-sample diagnostic benchmark. Using latent state steering to control reasoning length, they find a non-monotonic relationship between length and performance, with task-dependent optimal lengths, and they report that larger models and RL-only training generally yield better reasoning-perception balance.","tokens_in":13541,"tokens_out":3940,"duration_ms":46982,"significance":"If the causal claims hold, the paper provides a valuable warning that test-time compute in multimodal LLMs may systematically trade perceptual accuracy for language-prior fluency, with direct implications for how multimodal reasoning models should be evaluated. The observational core—that reasoning models hallucinate more than their base models across multiple benchmarks—is supported by consistent results on five established hallucination datasets and is a useful empirical contribution. The release of RH-Bench and RH-AUC constitutes a practical evaluation resource, and the attention analyses offer a plausible mechanism. However, the reasoning-length and attention-decline claims rest on a steering method whose validation is incomplete, so the stronger causal conclusions are not yet established.","major_comments":[{"comment":"The steering direction d_l is computed from long and short reasoning traces drawn from the same test benchmark on which accuracy is subsequently measured. The paper does not show that d_l encodes reasoning length rather than content differences (e.g., answer correctness, topic, or hallucination patterns) between the long and short trace pools. Adding α·d_l to hidden states could change what the model generates in ways unrelated to length, making the non-monotonic accuracy curves in Figure 7 and the attention decline in Figure 6 potential artifacts of the steering direction. Please validate the steering on held-out items, report monotonic control of length on a separate set, and compare with the budget-forcing and test-time-scaling controls introduced in Section 4.1.","section":"Section 4.1, Eq. (2)"},{"comment":"The claim that longer reasoning chains reduce visual attention is based on comparing 'normal thinking' with 'overthinking' produced by the same latent state steering. Without evidence that steering isolates length, the observed shift in attention from visual to instruction tokens may be caused by the content of the steering direction rather than by chain length per se. Please provide attention analyses under budget forcing or test-time scaling as complementary controls, or demonstrate that d_l is orthogonal to answer content and other semantic features of the traces.","section":"Section 3.2, Figure 6"},{"comment":"The central conclusion that reasoning length has a non-monotonic effect on performance, and that the optimum is task-dependent, is supported only by curves generated with latent state steering. The other two control strategies from Section 4.1 are not shown in the main text. If the steering intervention is confounded with content, the claimed non-monotonicity and task-specific optimal intervals may not reflect natural reasoning-length effects. Please include results for budget forcing and test-time scaling, or explicitly restrict the claims to the steering intervention and acknowledge the confound.","section":"Section 4.2, Figure 7 and Section 5.1, Eq. (3)"},{"comment":"The conclusions about the influence of training data type, domain, and volume are drawn from comparisons among existing models with different training pipelines, data sizes, and data sources, without controlled retraining experiments. The Limitation section appropriately acknowledges this observational nature, but the abstract and Section 5.2 state these as findings. Please soften the causal language in the abstract and key claims, or add a clear caveat that these are correlational observations based on public technical reports.","section":"Section 5.2, Training Dataset"}],"minor_comments":[{"comment":"The heading contains a typo: 'Straregies' should be 'Strategies'.","section":"Section 4.1 heading"},{"comment":"In the text describing Figure 6, 'the model correctly identifies the gray well' should read 'gray wall'.","section":"Section 3.2, Figure 6 caption"},{"comment":"There are several typos: 'traing data size' should be 'training data size', and 'multimodel reasoning models' should be 'multimodal reasoning models'.","section":"Section 5.2"},{"comment":"The 'Zerothink' condition is mentioned without a definition in the main text; please clarify how it is implemented and how it differs from a simple length reduction.","section":"Section 4.2"},{"comment":"References [8] and [9] both cite the same Insight-V paper, and references [49] and [50] both cite the same Seg-Zero paper; please deduplicate.","section":"References"},{"comment":"The RH-AUC computation uses min-max normalization and a specific set of reasoning lengths; please discuss how sensitive the metric is to the chosen length set and whether the normalization preserves comparability across models with different accuracy ranges.","section":"Section 5.1, RH-AUC"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the core observation (reasoning models hallucinate more on perception tasks) is likely robust. The main risk is the latent-state steering confound: because the steering direction is derived from the same test benchmark and its content selectivity is unvalidated, the paper's strongest causal claims about reasoning length and visual attention are not yet supported. I would urge the editor to require either held-out steering validation or results with alternative length-control methods before acceptance. The duplicated references and several typos suggest the manuscript would also benefit from a careful proofreading pass."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the bottom line: the paper documents a real and fairly general phenomenon—multimodal reasoning models, across eight model variants and five perception benchmarks, hallucinate more than their non-reasoning backbones. That alone is worth publishing. The RH-AUC metric and RH-Bench are useful practical tools, even if RH-AUC's normalization is a bit underspecified. The RL-only vs SFT+RL observation is interesting and supported by the RH-AUC numbers, though it's observational.\n\nThe soft spot is the causal story about reasoning length. Section 4.1 computes the steering direction d_l from long and short traces drawn from the same RH-Bench/test benchmark on which accuracy is later measured. So the non-monotonic curves in Figure 7 and the attention decline in Figure 6 could be driven by content differences in those traces—answer correctness, topic, hallucination content—rather than by length per se. The paper introduces budget forcing and test-time scaling as alternative length controls but doesn't use them to validate the steering result. That's a load-bearing gap for the claim that longer reasoning chains cause reduced visual attention. The claim should be re-run with held-out steering directions or with budget forcing as the primary control.\n\nThe training-data conclusions in Section 5.2 are acknowledged as observational, so they're fine as hypotheses, not findings. The RH-AUC formula sorts by reasoning performance, which makes it sensitive to the number of evaluation points; the paper should specify the exact length sweep and normalization. The citation pattern looks fine—related work covers the major benchmarks and recent reasoning models.\n\nOverall, the core descriptive finding holds up. The length-causality part needs more work. This deserves serious peer review—the empirical contribution is strong enough that referees should engage with it and push for validation of the steering method.","headline":"The paper's core finding—reasoning models hallucinate more on perception—is solid; the length-causality claim is undermined by a steering method fit on the same test benchmark.","tokens_in":14051,"tokens_out":2250,"would_cite":true,"duration_ms":17343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal reasoning models hallucinate more than their non-reasoning base models on perception tasks, and longer reasoning chains make it worse by shifting attention away from the image.","keywords":["multimodal reasoning","visual hallucination","reasoning length","attention analysis","RH-AUC","RH-Bench","perception benchmarks","reinforcement learning"],"falsifier":"Force one reasoning model to produce longer chains with token-budget forcing or test-time scaling only, never latent state steering, and measure per-token attention to image tokens and hallucination rates on MMVP and MMHalu; if longer chains do not reduce visual attention or do not increase hallucination, the claimed length-attention-hallucination chain is falsified.","tokens_in":13140,"feed_emoji":"👁️","tokens_out":13342,"duration_ms":85922,"temperature":0.7,"pith_summary":"This paper sets out to show that test-time compute, the extra thinking that multimodal language models are trained to do, systematically weakens their visual grounding. Comparing eight reasoning-augmented models with their non-reasoning base models on five perception benchmarks, the paper finds that the reasoning versions hallucinate more even when the underlying backbone and scale are the same. Attention analysis attributes the effect to a shift away from image tokens toward instruction tokens, a shift that grows as reasoning chains get longer. To make the trade-off measurable, the paper introduces RH-AUC, a metric that summarizes how perception accuracy changes as reasoning length varies, and RH-Bench, a 1,000-sample benchmark spanning reasoning and perception tasks. The central message is that reasoning length has a non-monotonic, task-dependent effect on performance, so evaluations of multimodal reasoning must measure perceptual fidelity and reasoning quality jointly rather than accuracy alone.","feed_headline":"Thinking longer makes multimodal models hallucinate more","feed_subtitle":"A new metric and benchmark show why accuracy alone misses the reasoning-versus-perception trade-off.","key_machinery":"The mechanism that carries the length experiments is latent state steering. The paper collects long and short reasoning traces on the test benchmark, extracts the average hidden-state vector after the attention layer from the reasoning span at each layer, and defines a direction vector $d_\\ell = S^{\\ell}_{\\text{long}} - S^{\\ell}_{\\text{short}}$; shifting hidden states by $\\alpha d_\\ell$ with $\\alpha \\in [-0.15, 0.15]$ stretches or compresses the reasoning chain while keeping the query and model fixed. The measurement object is RH-AUC, the area under the curve formed by points $(R_T, H_T)$, where $R_T$ is reasoning accuracy and $H_T$ is hallucination performance at reasoning length $T$, min-max normalized and integrated with the trapezoidal rule; higher values mean a better balance of reasoning and perception. RH-Bench provides the 1,000 samples for this measurement, with 500 reasoning questions from four math and science benchmarks and 500 perception questions from four hallucination benchmarks, each offered in multiple-choice and open-ended form.","core_discovery":"The central claim is that the reasoning paradigm itself, not model scale, degrades visual grounding. Reasoning-augmented models consistently allocate less attention to visual tokens and more attention to instruction tokens than their non-reasoning counterparts, and extending the reasoning chain further reduces visual attention and increases reliance on language priors, which raises hallucination rates on perception tasks. The relationship between reasoning length and performance is non-monotonic: moderate lengths are best, reasoning-heavy tasks such as MathVista benefit from longer chains, and perception-heavy tasks such as MMHalu peak at shorter lengths. The paper captures this balance with RH-AUC, computed from min-max normalized pairs of reasoning accuracy and hallucination performance at different lengths via the trapezoidal rule, and with RH-Bench, a 1,000-sample diagnostic benchmark with multiple-choice and open-ended reasoning and perception questions. On that benchmark, larger models, RL-only training rather than SFT-plus-RL, and domain-matched training data all improve the reasoning-hallucination balance, while the sheer volume of training data does not.","pith_inferences":["If the causal story holds, inference-time systems could treat reasoning length as a tunable hyperparameter and stop a chain once visual evidence stops changing the answer, reducing hallucination while preserving reasoning gains; the paper does not implement such a controller.","The attention result suggests an untested intervention: explicitly re-anchoring attention to image tokens at intervals during long chains should reduce hallucination if the mechanism is causal.","The RH-AUC formulation is not tied to vision and language; it could be applied to any model family where output length trades off against a fidelity signal, such as factual consistency in long-form text generation.","Because the training-data findings are observational, a controlled study that trains one backbone on matched data with different domain mixes is the natural next step; the paper itself notes this limitation."],"forward_implications":["Evaluation of multimodal reasoning models should report perception accuracy and reasoning accuracy jointly; single-point metrics can rank models misleadingly because the two peaks occur at different reasoning lengths.","The optimal reasoning length is task-dependent, so a fixed inference budget or a universal think-longer policy will hurt perception-heavy tasks even as it helps math-style tasks.","Training data should be balanced by domain and type rather than by volume alone; adding more visual data does not guarantee better visual grounding.","RL-only training appears to produce more concise and adaptive reasoning, so supervised fine-tuning that forces imitation of long reasoning formats may be a cause of hallucination amplification rather than just a correlate.","Scaling model size improves the reasoning-hallucination balance, so the trade-off is not fixed by the reasoning paradigm itself."],"supporting_citations":[{"why":"Supplies the MMVP visual perception benchmark used to compare hallucination rates between reasoning and non-reasoning models.","marker":"[35]"},{"why":"Supplies the MMEval-Pro benchmark used in the hallucination comparisons and in the RH-Bench perception split.","marker":"[11]"},{"why":"Supplies the VMCBench benchmark used in the hallucination comparisons and in RH-Bench.","marker":"[54]"},{"why":"Supplies the Bingo benchmark used to measure hallucination in the base-versus-reasoning comparison.","marker":"[5]"},{"why":"Supplies the MMHAL benchmark used both as a hallucination benchmark and as a perception task in the reasoning-length experiments.","marker":"[32]"},{"why":"Prior work linking insufficient attention allocation to hallucination, used to motivate the attention analysis.","marker":"[12]"},{"why":"Prior work connecting attention patterns to hallucination, used to interpret the visual-attention findings.","marker":"[13]"},{"why":"Source of the budget forcing and test-time scaling strategies used to control reasoning length.","marker":"[25]"},{"why":"Basis for the latent state steering method adapted here to control reasoning length.","marker":"[18]"},{"why":"Source of the latent-space steering direction extraction technique used to build the length directions.","marker":"[17]"}],"fun_headline_variants":["Longer reasoning, weaker vision: multimodal models hallucinate more","Thinking hurts seeing: reasoning chains boost hallucination","More reasoning, less grounding: the hallucination cost","Reasoning length trades off against visual truth in AI models","RH-AUC: measuring how reasoning erodes visual grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that adjusting the steering-strength parameter changes only how long the model thinks, not what it attends to or how it reasons, so the rise-and-fall accuracy curves are caused by reasoning length itself rather than by the steering direction.","fun_headline_variants_meta":{"raw":{"variants":["Longer reasoning, weaker vision: multimodal models hallucinate more","Thinking hurts seeing: reasoning chains boost hallucination","More reasoning, less grounding: the hallucination cost","Reasoning length trades off against visual truth in AI models","RH-AUC: measuring how reasoning erodes visual grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1216,"prompt_tokens":941,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":557,"tokens_out":275,"duration_ms":2505,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:45:50.638004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Force one reasoning model to produce longer chains with token-budget forcing or test-time scaling only, never latent state steering, and measure per-token attention to image tokens and hallucination rates on MMVP and MMHalu; if longer chains do not reduce visual attention or do not increase hallucination, the claimed length-attention-hallucination chain is falsified.","supporting_citations":[{"cited_title":"Reducing hallucinations in large vision-language models via latent space steering","cited_arxiv_id":null,"evidence_quote":"Basis for the latent state steering method adapted here to control reasoning length."}],"review_version":1}