{"id":"164c8b0b-0681-4169-82ac-0908f5e28763","arxiv_id":"2608.00076","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CMA uses diffusion-generated counterfactuals and two-player Shapley values to measure whether an image or the text drives a multimodal LLM's prediction, hitting 98% on synthetic biased benchmarks.","lead":"Multimodal AI models can be correct yet rely on the wrong input, for example diagnosing pneumonia from chest X-rays while ignoring a patient's severe allergy documented in text. This paper proposes Counterfactual Modality Attribution (CMA), a method that swaps out only the image, only the text, or both, then uses game theory to score how much each modality drove the prediction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Text-only counterfactual updates are under-specified (undefined `1 d0`) and DDPM/RDLM synchronization is not defined; the 98% attribution result could be an artifact of counterfactual generation rather than modality reliance.","rationale":"I read the paper as proposing CMA as a principled modality-attribution method whose central quantitative claim is the 98% controlled accuracy. For that claim to hold, the coalition values must be computed from counterfactuals that isolate each modality. The reader's weakest assumption correctly targets the under-specified coupling of the two diffusion processes and the text update rule. I agree with that diagnosis. In particular, the undefined `1 d0` is not a cosmetic typo: it is an operator in the main update equation for text-only counterfactuals, and without it the text counterfactual cannot be reproduced or even interpreted precisely. Similarly, 'synchronized at the same diffusion timestep' is the key mechanism enabling joint counterfactuals, but no formal correspondence between discrete DDPM steps and continuous RDLM time is provided. Since the Shapley values depend directly on v(∅) from the joint counterfactual, an ambiguous synchronization leaves the attribution scores not fully defined. The paper has genuine strengths: the two-player Shapley formulation is a natural and elegant way to quantify modality-level contributions, the synthetic benchmark with known ground-truth bias is a sensible evaluation design, and the OpenI qualitative example illustrates a real failure mode. The corruption study offers evidence that CMA tracks available evidence monotonically. However, none of these strengths resolves the core dependency on counterfactual validity. Because the concern is specific and addressable—code release and a synchronization-robustness experiment would settle it—the appropriate verdict is CONDITIONAL rather than REJECT or UNCHANGED. The reader's conditional verdict is justified, and the proposed test would directly probe the load-bearing assumption.","tokens_in":12558,"tokens_out":2630,"duration_ms":32680,"concrete_test":"Release code with an explicit definition of `1 d0` and the time-alignment map between DDPM step s and RDLM time t. Then run a stability check on the Synthetic MNIST Image-Biased and Text-Biased classifiers: vary the alignment scheme (e.g., linear s/T <-> t/tau, noise-level matching, and a deliberately misaligned schedule) while keeping all other hyperparameters fixed, and recompute the 98% hit rate and Image%/Text% scores. If the dominant-modality decision flips across reasonable alignments, the attribution is not robust and the headline claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that CMA correctly identifies the decision-driving modality rests on the validity of the generated counterfactuals: if image-only, text-only, and joint counterfactuals are not faithful minimal interventions, the Shapley values computed from them do not measure modality reliance. Two under-specified steps make this insecure. First, Methodology: Text-only counterfactuals updates the text distribution with `p_theta <- softmax(log p_theta - (lambda^T_c g_hat^T + lambda^T_1 (p_theta - 1_d0)))`, where `1_d0` is never defined. If it is not precisely the original token representation (embedding, one-hot, or predicted distribution), the proximal penalty is not well-defined and the text counterfactual may be neither minimal nor on-manifold. Second, 'synchronized at the same diffusion timestep' is asserted for DDPM (discrete time s) and RDLM (continuous time t) but no mapping between the two time coordinates is given. Different alignments change which image and text states are jointly evaluated by the classifier at each reverse step, so the joint counterfactual and hence v(∅) are not uniquely defined. Without specifying these choices, the reported 98% accuracy could be an artifact of one particular implementation rather than evidence about modality reliance. The paper's own Limitations section concedes that 'imperfect counterfactual generation may affect attribution quality,' but no sensitivity analysis addresses this dependency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Counterfactual Modality Attribution (CMA), a post-hoc explainability method for multimodal LLMs that quantifies the contribution of image and text modalities to a classifier's prediction. CMA generates image-only, text-only, and joint counterfactuals by running synchronized DDPM and RDLM diffusion priors under classifier guidance, then computes two-player Shapley values using the logit of the original predicted class on the four coalitions. The method is evaluated on controlled synthetic MNIST variants with known modality biases and on the OpenI chest X-ray dataset. The authors report 98% attribution accuracy on the biased classifiers and demonstrate a qualitative clinical shortcut (ECG leads) on OpenI.","tokens_in":12925,"tokens_out":6966,"duration_ms":78022,"significance":"CMA addresses a real gap: existing feature-attribution methods do not answer which modality drives an MLLM prediction. The formulation is principled, and the use of counterfactual interventions rather than gradients or static perturbations is well motivated. The paper's strengths include the controlled synthetic benchmarks with known ground truth, the inclusion of source code in the supplement, the closed-form two-player Shapley solution, and a corruption-based sanity check. However, as presented, the central 98% accuracy claim rests on two biased synthetic classifiers and on a counterfactual generation pipeline whose synchronization and text update rule are under-specified. If these issues are resolved, CMA would be a valuable contribution to multimodal explainability.","major_comments":[{"comment":"The claimed synchronization between DDPM (discrete step s) and RDLM (continuous time t) is not defined. The text says the reverse processes are 'synchronized at the same diffusion timestep' and Fig. 2 says 'both arms same step t', but no mapping is given between s and t, nor is the number of reverse steps or the alignment schedule specified. Because the joint counterfactual is generated by conditioning each modality on the other's evolving state, different alignments change the joint trajectory and therefore v(∅) and the resulting Shapley values. The 98% accuracy claim depends on this unspecified choice. Please provide the exact coupling mechanism (e.g., time reparameterization, step-count matching) and a sensitivity analysis over alignment choices.","section":"Methodology: Coupled Diffusion"},{"comment":"The update pθ ← softmax(log pθ − (λ_c^T ĝ^T + λ_1^T (pθ − 1_{d0}))) contains the undefined symbol `1 d0`. If it is not the exact original token representation (one-hot, embedding, or predictive distribution), the proximal penalty is not a well-defined function and the text counterfactual may be neither minimal nor on-manifold. Also, ĝ^T = ∇_{ℓθ} L_CW is computed w.r.t. per-digit logits while the update acts on probabilities; the chain rule is not shown. Please define all symbols and provide the exact update used in the experiments.","section":"Methodology: Text-only counterfactuals"},{"comment":"The headline 98% accuracy is reported only for the Image-Biased and Text-Biased classifiers, where the ground-truth decision rule is degenerate. In these settings any method that directly tests whether replacing one modality changes the prediction is predisposed to succeed; the result does not establish that CMA resolves the Balanced variant, which requires both modalities and is the case where modality attribution is nontrivial. The Balanced variant appears only in the corruption ablation (Fig. 5b), not in the attribution-accuracy comparison (Fig. 4). Please report attribution accuracy and interaction measures on the Balanced variant, or justify why biased classifiers suffice.","section":"Results: Synthetic Multimodal MNIST"},{"comment":"The paper concedes that 'imperfect counterfactual generation may affect attribution quality', but provides no experimental support for the adequacy of the generated counterfactuals. There is no report of flip success rates per generation mode, no distance-to-original statistics, no human or automated realism evaluation, and no sensitivity analysis over the guidance weights λ_c, λ_1, the inpainting threshold τ_img, or the token-change threshold τ_txt. Since every attribution score in the paper is computed from these counterfactuals, the absence of such validation is load-bearing for the central claim.","section":"Limitations"}],"minor_comments":[{"comment":"The OpenI evaluation is qualitative only. The paper reports one illustrative counterfactual (ECG leads removed) but no quantitative metrics such as flip rates, attribution distributions, or agreement with clinical labels. Please consider adding such numbers to support the claim that CMA reveals clinically relevant shortcuts.","section":"Results: Multiclass OpenI"},{"comment":"The notation `sign(·)` and the inpainting threshold τ_img are not numerically specified. Please provide values or point to the supplementary material for all hyperparameters used in the experiments.","section":"Methodology: Image-only counterfactuals"},{"comment":"The RDLM notation uses X_t^T for both token representation and time, and the drift expression is dense. Please clarify the variable roles and define all symbols (e.g., e_k, c(t), β(t)) in one place.","section":"Background: Text Diffusion"},{"comment":"The text reports exact hit rates only for MM-SHAP and MultiViz. Please report numerical values with confidence intervals for all baselines in the figure or caption, so the reader can verify the 'consistently outperforms' claim.","section":"Results: Figure 4"},{"comment":"The phrase 'first framework' is used in the abstract and contributions. Given the rapid growth of multimodal explainability work, please temper this claim or add a more thorough comparison to recent modality-attribution methods, especially those using causal or counterfactual interventions.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is contingent on implementation details that are not in the main text. If the supplementary material contains the missing specifications (synchronization mapping, definition of `1 d0`, hyperparameter values, sensitivity analyses), the authors should be asked to integrate them into the main text and to add an evaluation on the Balanced variant. The 98% claim as stated is too strong for the biased-only experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new way to ask which modality drives an MLLM prediction, and the two-player Shapley setup is clean. But the headline 98% comes from two synthetic classifiers that are biased by construction, and the method description has two under-specified steps that need to be nailed down before I'd believe the claim.\n\nWhat's good: the problem is well motivated. Existing methods give you pixels or tokens, not a modality-level answer. CMA's target is sensible. The Shapley formulation for two players is straightforward and correctly separates individual contributions from the interaction term. The OpenI example—where removing ECG leads flips a Nodule prediction to Normal—is a vivid demonstration of a clinically dangerous shortcut. The corruption experiment is a nice sanity check: attribution shifts to the untouched modality as one is progressively degraded, so the method isn't just outputting a fixed preference. The related work is placed honestly; they don't overclaim against MM-SHAP and MultiViz.\n\nThe soft spots are real. First, the 98% hit rate is measured on Image-Biased and Text-Biased models. On a model that only uses the digit, any method that toggles the image and sees a prediction flip will look great. The Balanced variant, which actually requires both modalities, is only used for the corruption plot, not for attribution accuracy. That's the experiment that would convince me, and it's missing. Second, the method section leaves two things undefined: the `1_d0` in the text update rule (presumably an indicator of the original token, but it's not stated) and the synchronization between discrete DDPM steps and continuous RDLM time. Without a defined alignment, the joint counterfactual and hence v(∅) are not uniquely specified. The paper's own Limitations section admits that imperfect counterfactual generation may affect attribution quality, but there's no sensitivity analysis to guidance weights, thresholds, or the alignment choice. The OpenI evidence is a single anecdote; no quantitative accuracy or counterfactual realism measure is reported there. Code is promised but not actually available.\n\nWho it's for: researchers working on multimodal explainability and anyone auditing MLLMs in high-stakes settings. It's a solid contribution to discuss, and the flaws are addressable. I'd send it to peer review, not desk reject, but the revision needs to be substantial: release the code, specify the diffusion coupling, add the Balanced attribution results, and include a sensitivity analysis.","headline":"Promising framework for modality-level attribution, but the headline accuracy rests on easy biased classifiers and the method has two under-specified steps; worth reviewing, needs major revision.","tokens_in":13409,"tokens_out":3545,"would_cite":true,"duration_ms":39515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CMA quantifies, per prediction, how much the image, the text, and their joint interaction contribute to a multimodal LLM's decision; on controlled tests it names the decision-driving modality in 98% of cases.","keywords":["modality attribution","counterfactual explanations","multimodal large language models","Shapley values","diffusion models","explainability","cross-modal reasoning","medical imaging"],"falsifier":"Take the chest X-ray case where CMA attributes the prediction to an image shortcut: manually remove the ECG leads from the image while leaving the radiology report identical, and run the classifier. If the model still predicts the nodule class, CMA's attribution to the image shortcut is wrong.","tokens_in":12472,"feed_emoji":"🩻","tokens_out":13256,"duration_ms":128722,"temperature":0.7,"pith_summary":"The paper argues that existing explainability tools answer where a model looks—which pixels or tokens matter—but not which whole channel, image or text, actually decides a prediction. It introduces CMA, a counterfactual modality-attribution framework: the method generates plausible edited versions of the image alone, the text alone, and both together, then evaluates the classifier on the four resulting input coalitions and converts the four logit values into Shapley-based Image% and Text% scores. On synthetic benchmarks with known ground truth, CMA attributes the correct modality 98% of the time and beats feature-, perturbation-, and surrogate-based baselines. On a real chest X-ray benchmark, it exposes a model that diagnoses a nodule from ECG leads in the image rather than from radiographic evidence—an unsafe reliance that prediction accuracy alone cannot reveal. The paper's broader claim is that modality attribution should be treated as a separate, complementary explainability objective for multimodal foundation models.","feed_headline":"New score names the deciding modality in 98% of controlled tests","feed_subtitle":"A counterfactual game-theoretic method exposes shortcut reasoning that accuracy alone hides.","key_machinery":"The central object is a two-player counterfactual coalition grid: for an input pair (image, text), CMA creates an image-only counterfactual, a text-only counterfactual, and a joint counterfactual, so that the classifier logit for the original predicted class can be evaluated on all four coalitions {both original, image edited, text edited, both edited}. These values feed the closed-form two-player Shapley formula, which averages marginal contributions and isolates cross-modal interaction. The generating machinery is synchronized reverse diffusion: a Denoising Diffusion Probabilistic Model (DDPM) for images and a Riemannian Diffusion Language Model (RDLM) for text move through reverse time at","core_discovery":"The central claim is that a multimodal classifier's prediction can be decomposed, at the level of whole input channels, into an image contribution, a text contribution, and their interaction, by measuring how the logit of the predicted class changes under realistic counterfactual edits of each modality. CMA computes this decomposition by generating three counterfactual inputs—image-only, text-only, and joint—through synchronized reverse diffusion guided by classifier gradients, then evaluates the classifier at the four combinations of original and edited modalities. The closed-form two-player Shapley values of this game give the modality attribution scores, and the normalized absolute scores","pith_inferences":["Going beyond the paper, modality attribution could be used as a training signal: downweighting samples where the attributed dominant modality contradicts the task's intended evidence might reduce shortcut reliance without extra labels.","The two-player game naturally extends to three or more input channels (image, structured records, free text), although that would require synchronized counterfactual priors for every new channel—an untested generalization.","Because CMA is reference-dependent, a fair model-to-model comparison would need to fix the counterfactual reference distribution; otherwise attribution gaps may reflect generator differences rather than behavioral differences.","A hierarchical extension is plausible: once the dominant modality is identified, running the same game inside that modality could locate the specific image regions or text spans that drive the effect, unifying modality attribution with feature attribution."],"forward_implications":["Audits of high-stakes multimodal systems can report a modality split (Image% vs Text%) alongside accuracy, so a correct-but-shortcutting model becomes visible before deployment.","Modality attribution accuracy could become a benchmark metric: two models with equal task accuracy can have opposite modality reliance, so accuracy alone is not a sufficient evaluation target.","Joint counterfactuals provide a measurable test of genuine multimodal fusion, since their effect on the prediction is not the sum of the two unimodal edits.","In clinical settings, CMA-style attribution can flag cases where a diagnosis is supported by spurious image cues, prompting re-labeling or re-training on the real pathological evidence.","Because CMA is model-agnostic and only requires differentiability, the same four-coalition Shapley grid applies to any multimodal architecture, from document understanding to robotics."],"supporting_citations":[{"why":"Supplies the DDPM image prior and the reverse-process posterior mean that CMA guides to generate image counterfactuals.","marker":"Ho, Jain, and Abbeel 2020"},{"why":"Supplies the RDLM hyperspherical text diffusion prior whose reverse drift CMA guides for text counterfactuals.","marker":"Jo and Hwang 2025"},{"why":"Defines the margin loss that is back-propagated from the classifier to steer both diffusion processes until the prediction flips.","marker":"Carlini and Wagner 2017"},{"why":"Establishes classifier guidance, the mechanism CMA uses to inject gradient information into reverse diffusion.","marker":"Dhariwal and Nichol 2021"},{"why":"Provides the cooperative-game value whose marginal-contribution averaging and uniqueness axioms justify the attribution scores.","marker":"Shapley 1953"},{"why":"Supplies the public chest X-ray and radiology report data used for the real-world clinical evaluation.","marker":"Demner-Fushman et al. 2016"},{"why":"Supplies the MNIST digit images used to construct the controlled synthetic benchmarks with known modality bias.","marker":"LeCun et al. 1998"},{"why":"Defines the MM-SHAP token- and patch-level Shapley aggregation baseline that CMA is compared against.","marker":"Parcalabescu and Frank 2023"},{"why":"Supplies the MultiViz LIME-based per-modality surrogate baseline that CMA is compared against.","marker":"Liang et al. 2022"}],"fun_headline_variants":["Score names the deciding modality in 98% of controlled tests","Counterfactual metric reveals which modality drives a model's call","Game-theoretic score spots when a multimodal model trusts the wrong cue","Which input wins? New method attributes decisions to image or text","Shapley-based attribution exposes shortcut learning in multimodal AI"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The accuracy claim rests on the untested premise that the synchronized image and text diffusion processes produce realistic, comparable counterfactuals; if the guiding prior drifts off the data manifold, the classifier logits used in the Shapley game are out-of-distribution and the attribution scores become unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Score names the deciding modality in 98% of controlled tests","Counterfactual metric reveals which modality drives a model's call","Game-theoretic score spots when a multimodal model trusts the wrong cue","Which input wins? New method attributes decisions to image or text","Shapley-based attribution exposes shortcut learning in multimodal AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001122,"raw_usage":{"total_tokens":4496,"prompt_tokens":728,"completion_tokens":3768,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":3682}},"tokens_in":472,"tokens_out":3768,"duration_ms":33077,"temperature":1.0,"reasoning_tokens":3682,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:20:55.896032+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the chest X-ray case where CMA attributes the prediction to an image shortcut: manually remove the ECG leads from the image while leaving the radiology report identical, and run the classifier. If the model still predicts the nodule class, CMA's attribution to the image shortcut is wrong.","supporting_citations":[],"review_version":2}