{"id":"879b6fbd-6721-4ea2-86c2-91d21200c617","arxiv_id":"2412.07292","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A counterfactual debiasing framework for image-text sentiment analysis that subtracts learned modality-direct effects from fused logits, reporting small accuracy gains on MVSA datasets.","lead":"This paper applies counterfactual causal inference to multimodal sentiment analysis, subtracting estimated direct text and image effects from a fused prediction to reduce modality bias. It reports modest accuracy gains over older baselines on two Twitter sentiment datasets, but the central causal formula double-counts the total effect and the method closely follows prior counterfactual VQA work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 24's TIE_joint is not TE − NDE; it is the sum of two indirect effects from inconsistent baselines, so the causal debiasing claim is formally unsupported.","rationale":"The reader's verdict is REJECT, and I agree that the central causal claim is not established. The reader's weakest_assumption focuses on the learnable constants c and the NDE-sharpness assumption; that is a real concern, but it is secondary. Even with perfectly estimated c values, Eq. 24's arithmetic identity is wrong: the formula labeled TE−NDE is actually TIE_text + TIE_image, and the two are not equal because of the missing +Z_{t*,i*,k*} term. This is the most load-bearing issue because it invalidates the core claim that the method 'mitigates the direct effect from unimodal bias' through causal counterfactual inference. The ablation in Table 5 and the improvements in Tables 2–4 are empirical observations that could still be meaningful as a heuristic, but they cannot be explained by the causal decomposition as written. The paper also lacks released code and error bars, but those are supporting concerns; the formal error is decisive. The reader mentioned the arithmetic problem in the rationale but selected the c-parameter assumption as the weakest point, so my agreement is partial. My recommendation is unchanged: the paper should be rejected as stated, or substantially revised to provide a correct derivation of Eq. 24 and a valid causal interpretation.","tokens_in":148,"tokens_out":5035,"duration_ms":66540,"concrete_test":"Re-derive Eq. 24 from Eqs. 5–10 in a minimal numerical instantiation. For a fixed label, take Z_{t,i,k}=0.8, Z_{t,i*,k*}=0.2, Z_{t*,i,k*}=0.6, Z_{t*,i*,k*}=0.1. Eq. 24 yields 2(0.8)−0.2−0.6=0.8, while TE−NDE_text−NDE_image = 0.8−0.2−0.6+0.1=0.1. The values differ, so Eq. 24 is not the TE−NDE quantity it claims to be. If the authors can supply a derivation showing why Z_{t*,i*,k*} is zero or cancels, the concern is resolved; otherwise the causal debiasing claim is unproven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. 24, which is labeled as TE−NDE but is obtained by adding Eqs. 8 and 9. Algebraically, TIE_text + TIE_image = 2Z_{t,i,k} − Z_{t,i*,k*} − Z_{t*,i,k*}, exactly the right-hand side of Eq. 24. However, the paper's own definitions give TE − NDE_text − NDE_image = Z_{t,i,k} − Z_{t,i*,k*} − Z_{t*,i,k*} + Z_{t*,i*,k*}. The two expressions differ by the counterfactual baseline term Z_{t*,i*,k*}, which is never set to zero or otherwise constrained. Therefore Eq. 24 does not follow from the TE/NDE decomposition in Eqs. 5–10, and the inference rule 'select the label maximizing TIE_joint' is not a validated procedure for removing unimodal direct effects. This is an internal inconsistency, not a disagreement with a particular causal convention: the stated identity is false unless a hidden assumption (Z_{t*,i*,k*}=0) is added. Since the causal claim is the paper's central contribution, the experimental gains cannot be attributed to the proposed debiasing mechanism as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CF-MSA, a counterfactual causal inference framework for multimodal sentiment analysis. It models text and image as treatments, the multimodal fusion representation K as a mediator, and the sentiment label Y as the outcome. The method defines total effect (TE), natural direct effect (NDE), and total indirect effect (TIE), and performs debiased inference by selecting the label that maximizes a joint indirect effect TIE_joint, which is computed by subtracting learned counterfactual baselines from the fused prediction. The framework is instantiated with three branches (text, image, text-image fusion) and a fusion function, plus a KL-divergence-based loss L_ti that aligns the text and image counterfactual distributions. Experiments on MVSA-Single and MVSA-Multiple report accuracy and F1 improvements over several baselines, with ablations on the loss, the counterfactual conditions, and the learnable constants c.","tokens_in":13941,"tokens_out":6586,"duration_ms":64076,"significance":"If the causal derivation were correct, the paper would offer a principled way to remove unimodal bias in multimodal sentiment analysis by leveraging mediation analysis, and the proposed framework could potentially transfer to other multimodal tasks. The paper contains extensive experiments, ablations, and a promise of code release, which are strengths. However, the central theoretical claim is internally inconsistent: Eq. (24) does not follow from the TE/NDE decomposition defined in Eqs. (5)-(10). Moreover, the learnable constants c that define the counterfactual baselines are fit to match the observed joint distribution, so the 'direct effects' subtracted are not identified as natural direct effects from a causal model. These issues undermine the claim that the experimental gains are due to the proposed causal debiasing mechanism. The paper may still contain a useful heuristic debiasing method, but the causal interpretation as presented is not established.","major_comments":[{"comment":"The joint indirect effect is defined as TIE_joint = TIE_text + TIE_image = 2Z_{t,i,k} - Z_{t,i*,k*} - Z_{t*,i,k*}. However, subtracting the two NDEs from the TE defined in Eq. (5) yields TE - NDE_text - NDE_image = Z_{t,i,k} - Z_{t,i*,k*} - Z_{t*,i,k*} + Z_{t*,i*,k*}. The reference term Z_{t*,i*,k*} is nonzero in general and is never set to zero or otherwise constrained in the paper, so Eq. (24) is not the TE - NDE decomposition claimed in the text. Consequently, the inference rule 'select the label maximizing TIE_joint' has no demonstrated connection to removing direct modality effects; the causal claim that motivates the method is formally unsupported.","section":"Section 3.1, Eqs. (8)-(10) and Eq. (24)"},{"comment":"The learnable parameters c1..c4 define the counterfactual baselines Z_{t,i*,k*} and Z_{t*,i,k*}. These parameters are fit by minimizing the KL divergence between the counterfactual distributions and the total-effect distribution (Lkl) and between the text and image counterfactual distributions (Lti). This means the 'direct effects' subtracted in Eq. (24) are not estimated as natural direct effects from a structural causal model; they are calibrated to match the observed joint distribution. The assumption that the sharpness of the NDE should be similar to that of the TE, stated before Eq. (17), is asserted without any test or justification. As a result, the ablation in Table 5 cannot distinguish a genuine causal debiasing effect from a learned reweighting that happens to improve accuracy on these particular datasets.","section":"Section 3.2, Eqs. (16)-(19)"},{"comment":"The abstract and Section 4.3 claim new state-of-the-art performance, but the only condition that clearly exceeds all baselines on MVSA-Single is 'Removing Text Bias' (76.17 ACC); the full 'Removing Text-Image Bias' condition achieves 74.80, which is lower, and 'Removing Image Bias' achieves 73.24. In Table 3, several counterfactual variants degrade performance relative to the biased baselines (e.g., CF-CMAC removing image bias drops by 2.39 ACC on MVSA-Multiple; CF-CMAC removing text-image bias drops by 0.36 ACC on MVSA-Multiple). The paper does not specify which condition corresponds to the model referred to as 'CF-MSA' in the headline claim, and the inconsistent improvements make it difficult to attribute the observed gains to the proposed debiasing mechanism.","section":"Section 4.3, Tables 2 and 3"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, e.g., 'Multimoal data' and 'attracts increasing research and industrial attention, as Therefore' in the first two paragraphs; these should be corrected.","section":"Section 1"},{"comment":"The subscript in TIE_image is written as Z_i,t*,k*, which is inconsistent with the notation used elsewhere; it should be Z_{t*,i,k*} to match the counterfactual where text is masked and image is present.","section":"Eq. (9)"},{"comment":"The dataset statistics in Table 1 (4,511 and 16,779) do not match the original dataset sizes mentioned in the text (5,129 and 19,600); the preprocessing steps that lead to the reduced counts should be described in more detail.","section":"Section 4.1 and Table 1"},{"comment":"The mediator K is described as 'multimodal fusion knowledge', but its causal role is not formalized with a structural equation or explicit graph; the current presentation leaves the relation between K and the branching models ambiguous.","section":"Section 3.1"},{"comment":"The claim of being 'the first to introduce the causal effects into multimodal sentiment analysis' is too strong given the existing causal and counterfactual methods cited later (e.g., Niu et al. 2021); the novelty should be framed more precisely.","section":"Section 1"},{"comment":"The four c-parameter hypotheses (Random, CF-MSA Prior, Uniform, Non-uniform) are not precisely defined; in particular, 'prior knowledge from dataset statistic' and 'non-uniformly distributed Niu et al. (2021)' need explicit mathematical descriptions for the experiments to be reproducible.","section":"Table 5"}],"recommendation":"reject","confidential_remarks":"The manuscript is marked as 'Published as a conference paper at ICLR 2025'. The editor may wish to check the journal's policy on prior publication and whether this submission is a substantially extended version of that paper. My rejection is based on the formal error in the core causal derivation, which cannot be corrected without changing the method and re-running the experiments, and on the circularity in the estimation of the debiasing parameters."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper takes Niu et al.'s Counterfactual VQA recipe and applies it to text-image sentiment analysis. That's a reasonable idea to try, but the central inference score in Eq. 24 is not the total indirect effect it claims to be. The authors define TIE_text = Z_tik - Z_ti*k* and TIE_image = Z_tik - Z_t*ik*, then add them to get TIE_joint = 2Z_tik - Z_ti*k* - Z_t*ik*. A proper TE - NDE_text - NDE_image would be Z_tik - Z_ti*k* - Z_t*ik* + Z_t*i*k*. The baseline term Z_t*i*k* is missing. So the label-maximization rule is not justified by their own decomposition. This isn't a matter of choosing a different causal convention; it's an internal inconsistency.\n\nWhat's new and worth crediting: the two-treatment extension and the symmetric KL loss L_ti are new to sentiment analysis, and applying this framework to MVSA is a legitimate extension. The paper reports modest gains (1-3 accuracy points) on two public datasets, and the qualitative example is illustrative.\n\nThe soft spots beyond the equation: the c1-c4 parameters are fit on the training set by forcing the NDE distributions to match the TE, which makes the debiased score a learned combination rather than an externally grounded causal estimate. The comparison set is dated—no recent large multimodal models, no error bars, and F1 sometimes drops while accuracy ticks up. The novelty claim of being 'first to introduce causal effects into multimodal sentiment analysis' is overstated given the prior causal work in multimodal emotion and VQA.\n\nMy recommendation: this is not acceptable as is. The equation bug undermines the core argument, and the empirical evidence is too thin to carry it. If the authors can fix the derivation and provide stronger baselines and variance estimates, the idea might be worth a resubmission. For now, I'd reject.","headline":"A sensible adaptation of Counterfactual VQA to multimodal sentiment, but the central debiasing formula is arithmetically wrong and the empirical gains are too thin to rescue it.","tokens_in":14489,"tokens_out":3841,"would_cite":false,"duration_ms":38459,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A counterfactual score that subtracts text-only and image-only effects from the fused prediction removes modality bias and yields new best results on MVSA-Single and MVSA-Multiple.","keywords":["multimodal sentiment analysis","counterfactual inference","causal debiasing","modality bias","total indirect effect","text-image fusion","MVSA-Single","MVSA-Multiple"],"falsifier":"Construct a held-out set of image-text pairs in which the text contains explicit sentiment words opposite to the sentiment of the image, and compare CF-MSA against the biased baseline on those pairs. If the method truly removes text bias, its predictions should track the image content substantially more often than the baseline does; a failure to flip those predictions would show that the subtracted term is not capturing the text direct effect.","tokens_in":13440,"feed_emoji":"💭","tokens_out":8333,"duration_ms":143641,"temperature":0.7,"pith_summary":"This paper tries to establish that modality bias in multimodal sentiment analysis can be modeled as a direct causal effect and then removed by counterfactual reasoning. The proposed CF-MSA framework trains three branches--text-only, image-only, and text-image fusion--and at inference scores each label by subtracting the estimated text-only and image-only contributions from the fused score. On the MVSA-Single and MVSA-Multiple datasets the authors report that this subtraction improves accuracy and F1 over biased baselines and over earlier fusion methods, with the largest gains when text bias is removed. If the causal interpretation holds, the same recipe provides a general debiasing layer that can be attached to existing multimodal classifiers.","feed_headline":"Counterfactual reasoning strips unimodal bias from sentiment models","feed_subtitle":"A fused score minus learned text-only and image-only effects corrects misleading emotional cues in image-text pairs.","key_machinery":"The load-bearing object is the counterfactual decomposition of a model score into total effect, natural direct effect, and total indirect effect on the graph where text $T$ and image $I$ influence the label $Y$ both through direct paths and through the joint multimodal knowledge $K$. The implementation uses three branches $Z_t$, $Z_i$, $Z_k$, a SUM fusion $h(Z_t,Z_i,Z_k)=\\log\\sigma(Z_t+Z_i+Z_k)$, and two auxiliary losses: $L_{ti}$ symmetrizes the text-only and image-only distributions, while $L_{kl}$ fits the missing-input constants so the estimated natural direct effect has the same sharpness as the total effect. This machinery converts the debiasing intuition into a concrete label-scoring rule.","core_discovery":"The central claim is that the score $TIE_{joint}=2Z_{t,i,k}-Z_{t,i^*,k^*}-Z_{t^*,i,k^*}$ is a causal total indirect effect: it keeps the part of the fused multimodal prediction that flows through the joint multimodal mediator and subtracts the parts contributed directly by text alone and by image alone. The authors assert that choosing the sentiment label that maximizes this counterfactual score mitigates the direct effect of unimodal bias and, on MVSA-Single and MVSA-Multiple, gives the best accuracies and F1 scores among the methods compared. The argument rests on estimating the natural direct effects of the missing modalities with learnable constants $c_1,\\dots,c_4$, fitted by KL divergence so that the sharpness of the direct-effect distributions matches the sharpness of the total-effect distribution.","pith_inferences":["Beyond the paper, the identity $TIE_{joint}=Z_{t,i,k}+(Z_{t,i,k}-Z_{t,i^*,k^*})+(Z_{t,i,k}-Z_{t^*,i,k^*})$ suggests a general ensemble-debiasing rule for any multimodal classifier with unimodal branches, not only sentiment models.","A natural testable extension is to apply the same three-branch counterfactual subtraction to other tasks with known modality bias, such as visual question answering or hate-speech detection, where a dominant modality is known to drive spurious predictions.","Because the constants $c_1,\\dots,c_4$ are fitted on the training distribution, the debiasing may be effective in-distribution but could weaken under distribution shift; evaluating on deliberately perturbed modality distributions would reveal how much of the gain is causal rather than distributional."],"forward_implications":["If the total indirect effect score is a faithful causal quantity, replacing softmax inference with the argued TIE scoring should reduce misclassifications on samples where explicit sentiment words in text contradict the image.","The counterfactual layer should transfer across fusion architectures, since the paper reports gains when the same inference strategy is applied to OTE, CMAC, HSTEC, and NaiveCat.","The intermodal bias mutual elimination loss $L_{ti}$ should make text-only and image-only prediction distributions more alike, reducing the fused model's sensitivity to whichever modality dominates.","Non-uniform learnable constants should beat uniform, random, or prior-statistic settings because they allow per-modality calibration of the direct-effect estimate."],"supporting_citations":[{"why":"Supplies the counterfactual inference formulation and the non-uniform learnable-parameter assumption that CF-MSA adapts for its direct-effect estimates.","marker":"Niu et al. (2021)"},{"why":"Provides the OTE transfer-learning method and the NaiveCat and NaiveCombine baselines that CF-MSA is compared against.","marker":"de Toledo & Marcacini (2022)"},{"why":"Provides the CMCA and HSTEC cross-attention and self-attention baselines used in the comparison tables.","marker":"Rajan et al. (2022)"},{"why":"Provides the MultiSentiNet approach and the dataset preprocessing protocol of removing inconsistently labelled image-text pairs.","marker":"Xu & Mao (2017)"},{"why":"Supplies the prior use of counterfactual reasoning to remove bias from a trained model, the template for subtracting a direct effect.","marker":"Tang et al. (2020)"},{"why":"Supplies the repository implementations of the baseline fusion models used in the experiments.","marker":"Zheng (2022)"}],"fun_headline_variants":["Counterfactual score strips text and image bias from sentiment","Causal reasoning debiases multimodal sentiment classification","New formula subtracts direct effects in sentiment AI","Total indirect effect removes unimodal bias in sentiment","Counterfactual inference sharpens multimodal sentiment accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learnable constants $c_1,\\dots,c_4$, fitted by matching the sharpness of the direct-effect distributions to the total-effect distribution, correctly estimate the real direct effects of the missing modalities; if that sharpness match fails, the subtraction removes genuine signal rather than bias.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual score strips text and image bias from sentiment","Causal reasoning debiases multimodal sentiment classification","New formula subtracts direct effects in sentiment AI","Total indirect effect removes unimodal bias in sentiment","Counterfactual inference sharpens multimodal sentiment accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2599,"prompt_tokens":898,"completion_tokens":1701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1628}},"tokens_in":514,"tokens_out":1701,"duration_ms":12441,"temperature":1.0,"reasoning_tokens":1628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:57.700279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a held-out set of image-text pairs in which the text contains explicit sentiment words opposite to the sentiment of the image, and compare CF-MSA against the biased baseline on those pairs. If the method truly removes text bias, its predictions should track the image content substantially more often than the baseline does; a failure to flip those predictions would show that the subtracted term is not capturing the text direct effect.","supporting_citations":[{"cited_title":"Multimodal sentiment analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the repository implementations of the baseline fusion models used in the experiments."}],"review_version":1}