REVIEW 3 major objections 5 minor 43 references
The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Most visual tool calls don't change the answer.
desk verdict A careful empirical audit of visual tool-use with a genuinely new step-level estimand and a stable ATE decomposition, but the headline 'no causal effect' language overstates what the local VEG probe can establish. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a causal graph of a visual tool-use rollout with three routes from input to answer: the direct path, the observation-mediated path $T \to O \to Y$, and the action-induced shortcut $T \to Y$. The paper's step-level estimand, Visual Evidence Gain (VEG), is the difference between the change in the model's probability gap under the real returned crop and the change under a same-shape counterfactual crop, with prefix and tool action held fixed; the probability gap is the model's next-token preference for the correct option minus its strongest distractor, so VEG cancels the shortcut and isolates the visual contribution. A deterministic trajectory-level diagnostic then sorts each rollout into one of four groups (No-call, Calling Without Looking, Looking Without Planning, Calibrated) using the pre-call probability gap, the peak per-call VEG, whether the tool-call limit was hit, and whether calls continued after saturation. An additive decomposition of the policy-level accuracy gain into per-group contributions shows that the Calibrated group is the only consistently positive contributor.
What would settle it
Re-run the four-group diagnostic and ATE decomposition on a held-out fine-grained benchmark with the same thresholds; the central claim fails if the Calibrated subset is not the dominant positive contributor, or if trajectories labeled Calling Without Looking show mean |VEG| comparable to Calibrated trajectories when the counterfactual crop is drawn from a different image rather than a different location in the same image.
Extended reading notes
Core claim
The central discovery is a structural decoupling between the tool-use policy and the utility of the observations it elicits, which the paper calls policy miscalibration. Using a causal graph of a rollout, the paper separates the intended observation-mediated path (action to observation to answer) from an action-induced shortcut (action to answer) that can shift the answer without any visual evidence. The step-level estimand Visual Evidence Gain (VEG) measures, for each call, the change in the model's preference for the correct option under the real returned crop minus the same change under a counterfactual random crop, holding prefix and action fixed; this cancels the shortcut and isolates the visual contribution. On the V* benchmark, a deterministic four-group diagnostic (No-call, Calling Without Looking, Looking Without Planning, Calibrated) and an additive decomposition of the policy-level accuracy gain show that the Calibrated group is the only consistently positive contributor, while the majority of tool calls are either causally inert or scheduled incoherently. The paper concludes that aggregate gains are real but give an illusion of broad causal effectiveness.
Load-bearing premise
The audit assumes that Visual Evidence Gain — the difference in next-token probability-gap change between a real and a counterfactual crop under a fixed prefix and tool action — captures whether the returned observation causally affects the model's answer; the authors themselves note it is a local probe under a fixed readout, not the full causal effect on an unconstrained final-answer rollout.
Editorial extensions
If this is right
- A positive policy-level accuracy gain from visual tool-use is compatible with most tool calls being causally inert, so aggregate accuracy alone is not evidence that the model is using the returned images.
- Mode 1 (Calling Without Looking) rollouts can skip tool calls without losing accuracy, since their observations carry no causal evidence; Mode 2 (Looking Without Planning) rollouts call for an early-stopping rule that commits before late harmful calls.
- Step-level VEG provides per-step credit that outcome-only reinforcement learning cannot supply, because rewarding only final correctness treats a Calibrated trajectory and a wasted trajectory that both reach the correct answer as identical.
- An intervention on tool-use should be graded by whether it increases the Calibrated fraction and decreases Mode 1 plus Mode 2 mass, regardless of whether the aggregate ATE moves.
Reading between the lines
- A testable extension of the audit to other tools (OCR, segmentation, video frame selection) would replace the returned text, box, or frame with a plausible alternative; the paper gives this recipe in its limitations but does not run it, so whether CWL and LWP recur beyond crop-and-zoom is open.
- If the illusion is widespread, leaderboards that reward tool-calling behavior may be selecting for performative tool use, since a model can gain aggregate accuracy by calling tools without using their content; a calibration-style diagnostic would then be a useful public evaluation metric.
- Because step-level VEG needs token-level access, a black-box analogue that samples multiple free-form answers under real versus counterfactual crops and compares answer distributions could extend the audit to closed models the paper lists as out of scope.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formulates visual tool-use as a structural causal model over image, query, tool actions, returned observations, and final answer, and audits it with three interventions: a policy-level toggle between tool-use and direct inference, a trajectory-level corruption of all returned observations, and a step-level counterfactual replacement of a single observation. The step-level estimand, Visual Evidence Gain (VEG, Eq. 4), is the difference between next-token probability-gap changes under a real and a counterfactual crop with the same prefix and action. Across six open models and five benchmarks, the authors report that aggregate accuracy gains from tool-use are real but concentrated in a small 'Calibrated' subset, while most tool-using rollouts are either 'Calling Without Looking' (returned observations have no causal effect on the answer) or 'Looking Without Planning' (observations are informative but the call schedule is incoherent).
Significance. If the claims held, this would be a valuable causal audit protocol for the thinking-with-images paradigm, with a falsifiable trajectory-level diagnostic (Algorithm 1) and an operational decomposition of the policy-level ATE (Eq. 6). The empirical care is real: dynamic corruption avoids leakage from clean observations, the force-answer ablation separates truncation from evidence loss, repeated seeds and threshold sweeps appear in appendices, and code is released. The central reservation is that VEG is an immediate next-token readout, not an estimate of the effect of an observation on the final free-form answer; the manuscript itself concedes this in App. A.3, yet the headline CWL claim is phrased as a causal claim about the answer. This gap does not invalidate the diagnostic as a local probe, but it does mean that the paper's strongest conclusion currently outruns the evidence.
major comments (3)
- [§3.3.3, Eq. (4); Abstract; §4.1] The headline claim that in CWL trajectories 'returned observations have no causal effect on the answer' rests on VEG, which is a controlled contrast in next-token option-restricted probability gaps after appending '<answer>' to a fixed prefix. Appendix A.3 explicitly states that VEG is 'a local counterfactual estimate ... not a claim that we recover the full effect of the tool call on unconstrained final-answer formation.' The manuscript's own trajectory-level results (Table 2 and the force-answer ablation in App. B.4, Table 11) show that corrupting all observations produces large accuracy drops even when truncation is controlled (Mini-o3: 87.8 to 36.6; Qwen3-VL-8B: 91.1 to 73.0). If most individual calls have near-zero VEG while trajectory-level corruption strongly affects answers, the observation effect must either accumulate across calls or propagate through subsequent reasoning; either way, per-call VEG is not validated as a measure of causal effect on the final answer. Please either validate VEG against an answer-level counterfactual (e.g., ablating a single real observation in a full rollout and measuring final-answer changes, where feasible) or reframe the CWL and Calibrated claims as statements about the local next-token readout, not about causal effect on the answer.
- [§4.2, Algorithm 1; §4.3, Eq. (6)] The diagnostic defines 'Calibrated' as trajectories with V_max >= epsilon and no saturation, and Eq. (6) then shows that this Calibrated group carries most of the positive ATE. Because VEG is the very quantity used to define the group, part of the association between 'Calibrated' and positive evidence contribution is definitional. The independent part is the per-group accuracy gap, which is genuinely empirical, but its causal interpretation inherits the unvalidated VEG-to-answer link from the previous comment. The paper should state this explicitly and either supply an external validation of the group labels (the behavioral audit in App. E.3, such as pre-call commitment rates, is a useful start) or weaken the causal wording to 'groups defined by the local VEG readout.'
- [§4.3, Table 5] The central quantitative claim that the Calibrated subset contributes the majority of the positive ATE is reported without uncertainty quantification. V* has only 191 items, and some group sizes are small (e.g., DeepEyes Calibrated is 7.9% of trajectories), so the per-group contributions f_b * Delta_policy,b in Table 5 may be noisy. Since the authors already report repeated-seed results for VEG (App. C.1) and trajectory corruption (App. B.5), please add a similar repeated-seed or bootstrap analysis for the ATE decomposition, or at least report per-group confidence intervals, before treating the decomposition as a quantitative finding.
minor comments (5)
- [Abstract; §3.3.3] The paper says 'six representative models' and 'five benchmarks,' but the step-level VEG analysis (Table 3) and the trajectory diagnostic (Tables 4–5) cover only a subset; please clarify early which models and benchmarks are used at each intervention level.
- [Table 2] The row for DeepEyes in Table 2 is visually corrupted (accuracy values and Delta run together); please reformat the table.
- [Eq. (5); App. C.3] Notation is inconsistent between POER in Eq. (5), which is defined for k>=2 and g0 <= tau_sat, and the 'Post-saturation Over-Extension Rate' description in §3.3.3; please align the definition with the per-call breakdown in App. C.2.
- [Figure 4] The caption reads 'sat = 0.95' rather than 'tau_sat = 0.95'; please fix for consistency.
- [§3.3.3] The sentence 'By saturation conditions VEG on pre-call confidence' is ungrammatical; it should read 'By saturation status, VEG is conditioned on pre-call confidence.'
Circularity Check
The 'no causal effect' wording for Calling Without Looking is built into the VEG threshold that defines the label, while the underlying frequency measurements and ATE localization are empirical.
-
self definitional
[Abstract; §3.3.3 Eq. (4); §4.2 Algorithm 1]
"In Calling Without Looking, returned observations have no causal effect on the answer. ... if g0 > τsat or Vmax < ϵ then return MODE1 (CWL) ▷ saturated prior or inactive call"
Algorithm 1 assigns the CWL label precisely when the pre-first-call gap exceeds τsat or the peak per-call VEG is below ϵ = 0.01, and VEG (Eq. 4) is the paper's own operational measure of whether the observation's visual content contributes to the answer. The abstract's statement that in CWL 'returned observations have no causal effect on the answer' is therefore entailed by the labeling rule rather than derived from it. The genuinely empirical discoveries are the frequencies of the label across models (e.g., 72.8% for DeepEyes) and the independent behavioral signatures, which the paper does provide. The causal wording is a definitional consequence of thresholding VEG, not an independently established property of the so-labeled trajectories.
full rationale
The paper's quantitative core is not fitted or self-referential: VEG is a measured counterfactual contrast between real and counterfactual crops under a fixed prefix and action, and the policy-level ATE is an independent accuracy comparison. The trajectory-level corruption and force-answer experiments are additional external checks, and the threshold sensitivity sweeps show that the Calibrated-concentration result is not an artifact of one cutoff. There is no load-bearing self-citation: the Wei et al. (2026) marginal-improvement claim is corroborated by the paper's own Table 1, and the other co-authored references are contextual. The main circularity is semantic: the CWL failure mode is defined by a near-zero VEG threshold and then reported as 'no causal effect on the answer,' so that headline phrasing reduces to the definition. The paper partially mitigates this by validating the labels against independent think-block commitment and by explicitly conceding in App. A.3 that VEG is a local counterfactual estimate under a fixed readout, 'not a claim that we recover the full effect of the tool call on unconstrained final-answer formation.' That concession limits the reach of the causal wording but does not repair the definitional circularity in the abstract. Overall, the empirical prevalence numbers and the ATE decomposition carry independent content, so the circularity is partial rather than total.
Assumptions & free parameters
free parameters (2)
- saturation threshold tau_sat =
0.95
- near-zero VEG cutoff epsilon =
0.01
assumptions (5)
- standard math Pearl's structural causal model framework and its path-effect semantics are assumed as the formal foundation.
- domain assumption The tool-use rollout is generated by the SCM Ti <- f_pi(I,Q,T<i,O<i), Oi <- E_tool(I,Ti), Y <- f_pi(I,Q,T1:n,O1:n), with no unobserved confounders between observations and the answer.
- domain assumption The option-restricted next-token readout after appending '<answer>' tracks the model's answer preference, and VEG under a fixed prefix is a valid proxy for causal effect on final answers.
- domain assumption Counterfactual same-shape random crops are exchangeable with real crops except for visual content, so the real-versus-counterfactual difference in probability gaps isolates visual evidence.
- domain assumption Direct inference obtained by replacing the model's system prompt with a generic 'reason step by step and boxed answer' prompt is a valid do(pi=pi_direct) intervention that isolates tool availability.
Cite this review
Pith. "Pith review of The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images." pith.science (2026). https://pith.science/paper/FS5UTUET
@misc{pith2026260806270,
author = {Pith},
title = {Pith review of: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/FS5UTUET}},
note = {Machine review of arXiv:2608.06270}
}
read the original abstract
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Thinking with images , year =
-
[2]
2025 , howpublished =
Introducing. 2025 , howpublished =
2025
-
[3]
Deepeyes: Incentivizing" thinking with images" via reinforcement learning , author=. arXiv preprint arXiv:2505.14362 , year=
-
[4]
arXiv preprint arXiv:2505.15966 , year=
Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning , author=. arXiv preprint arXiv:2505.15966 , year=
-
[5]
arXiv preprint arXiv:2509.07969 , year=
Mini-o3: Scaling up reasoning patterns and interaction turns for visual search , author=. arXiv preprint arXiv:2509.07969 , year=
-
[6]
arXiv preprint arXiv:2602.01334 , year=
What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom , author=. arXiv preprint arXiv:2602.01334 , year=
-
[7]
arXiv preprint arXiv:2509.20912 , year=
DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning , author=. arXiv preprint arXiv:2509.20912 , year=
-
[8]
arXiv preprint arXiv:2511.19661 , year=
CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization , author=. arXiv preprint arXiv:2511.19661 , year=
Show all 43 references
-
[9]
2009 , publisher=
Causality , author=. 2009 , publisher=
2009
-
[10]
arXiv preprint arXiv:2602.11858 , year=
Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception , author=. arXiv preprint arXiv:2602.11858 , year=
-
[11]
arXiv preprint arXiv:2506.23918 , year=
Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers , author=. arXiv preprint arXiv:2506.23918 , year=
-
[12]
Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI) , pages=
Direct and indirect effects , author=. Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI) , pages=
-
[13]
Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI) , pages=
Identifiability of path-specific effects , author=. Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI) , pages=
-
[14]
Epidemiology , volume=
Identifiability and exchangeability for direct and indirect effects , author=. Epidemiology , volume=
-
[15]
The Fourteenth International Conference on Learning Representations , year=
Thyme: Think Beyond Images , author=. The Fourteenth International Conference on Learning Representations , year=
-
[16]
arXiv:2602.20739 , year=
PyVision-RL: Forging Open Agentic Vision Models via RL , author=. arXiv:2602.20739 , year=
-
[17]
Wu, Penghao and Xie, Saining , booktitle=. V ^
-
[18]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[19]
2025 , url=
YiFan Zhang and Huanyu Zhang and Haochen Tian and Chaoyou Fu and Shuangqing Zhang and Junfei Wu and Feng Li and Kun Wang and Qingsong Wen and Zhang Zhang and Liang Wang and Rong Jin , booktitle=. 2025 , url=
2025
-
[20]
arXiv preprint arXiv:2511.21631 , year=
Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=
-
[21]
Advances in Neural Information Processing Systems , year=
Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting , author=. Advances in Neural Information Processing Systems , year=
-
[22]
arXiv preprint arXiv:2307.13702 , year=
Measuring faithfulness in chain-of-thought reasoning , author=. arXiv preprint arXiv:2307.13702 , year=
-
[23]
International Conference on Machine Learning , pages=
On the Relationship Between Explanation and Prediction: A Causal View , author=. International Conference on Machine Learning , pages=. 2023 , organization=
2023
-
[24]
arXiv preprint arXiv:2402.14897 , year=
Chain-of-thought unfaithfulness as disguised accuracy , author=. arXiv preprint arXiv:2402.14897 , year=
-
[25]
arXiv preprint arXiv:2305.20050 , year=
Let's verify step by step , author=. arXiv preprint arXiv:2305.20050 , year=
-
[26]
arXiv preprint arXiv:2211.14275 , year=
Solving math word problems with process- and outcome-based feedback , author=. arXiv preprint arXiv:2211.14275 , year=
-
[27]
ACL , year=
Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations , author=. ACL , year=
-
[28]
arXiv preprint arXiv:2410.08146 , year=
Rewarding progress: Scaling automated process verifiers for LLM reasoning , author=. arXiv preprint arXiv:2410.08146 , year=
-
[29]
arXiv preprint arXiv:2606.30251 , year=
TACO: Tool-Augmented Credit Optimization for Agentic Tool Use , author=. arXiv preprint arXiv:2606.30251 , year=
-
[30]
arXiv preprint arXiv:2511.23031 , year=
From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning , author=. arXiv preprint arXiv:2511.23031 , year=
-
[31]
The Fourteenth International Conference on Learning Representations , year=
Information gain-based policy optimization: A simple and effective approach for multi-turn search agents , author=. The Fourteenth International Conference on Learning Representations , year=
-
[32]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[33]
Publications Manual , year = "1983", publisher =
1983
-
[34]
Chandra and Dexter C
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
1981
-
[35]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[36]
Dan Gusfield , title =. 1997
1997
-
[37]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[38]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[39]
arXiv preprint arXiv:2512.01827 , year=
CauSight: Learning to Supersense for Visual Causal Discovery , author=. arXiv preprint arXiv:2512.01827 , year=
-
[40]
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=
Cello: Causal evaluation of large vision-language models , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=
2024
-
[41]
International Conference on Learning Representations , volume=
Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability , author=. International Conference on Learning Representations , volume=
-
[42]
Findings of the Association for Computational Linguistics: ACL 2025 , pages=
From imitation to introspection: Probing self-consciousness in language models , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=
2025
-
[43]
Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.