{"id":"f15cefd7-e9d5-4551-86f8-c59d57897476","arxiv_id":"2412.00756","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new model, MICL, learns sarcasm from contradictions across words, objects, and emotions, and reports top accuracy on the MMSD benchmark and on a new spurious-correlation test set.","lead":"This paper introduces MICL, a model that detects sarcasm in text and image pairs by learning mismatches between words, objects, and emotions. It also builds a new test set to see whether models rely on misleading shortcuts instead of real sarcasm cues.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SPMSD is generated by LLM-based transformations with no human label validation, and the same generator families are used in MICL's training augmentation; the 68.7% SPMSD advantage may measure LLM-artifact familiarity rather than robustness to natural spurious correlations.","rationale":"The reader's weakest assumption and my stress-test converge on SPMSD validity. I checked the architecture and ablations for a more internal problem: the entity-object module (§3.2.2) pools text and image graph nodes rather than computing an explicit incongruity, and the sentiment module (§3.2.3) concatenates polarities and hidden states without a contrasting operation. This weakens the 'incongruity' narrative, but it does not by itself falsify the empirical claim that the added modules help; the ablations in Table 3 show consistent gains. The load-bearing issue is that the only evidence for the spurious-correlation claim is an LLM-generated, unvalidated test set, and the training augmentation uses the same generator families. The paper's own limitation section concedes accuracy on SPMSD is only 68%, so the claim is modest, but modest claims still need a valid measurement instrument. Release and human validation of SPMSD, plus a generator-independent shift benchmark, would settle it. Verdict remains conditional; no change from the reader.","tokens_in":15508,"tokens_out":4810,"duration_ms":44146,"concrete_test":"Release SPMSD with per-sample transformation metadata and have at least two independent annotators label all 1,000 samples (sarcastic/non-sarcastic) and flag invalid transformations. Recompute Table 2 on the subset with unanimous labels and valid transformations, and also report per-transformation-type accuracy. If MICL's accuracy advantage over DMSD-CL shrinks from the reported 8.1 points to below ~3 points on the validated subset, the robust-generalization claim is not supported. A complementary check: evaluate all models on a human-curated distribution-shift set that does not use LLM-generated transformations; if MICL no longer leads, the SPMSD result reflects training/test generator alignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central robustness claim—§4.4: MICL 'can effectively mitigate reliance on spurious correlations, showing better generalization ability on new data'—is supported only by the SPMSD test set. Appendix A constructs SPMSD by randomly selecting 1,000 MMSD samples and applying LLM-driven transformations (regenerating sarcastic text, swapping text-image pairs, using only image/text), yet it reports no human validation of the resulting labels, no inter-annotator agreement, and no release of the dataset. Meanwhile, §3.4 trains MICL on augmented data generated with the same tool families (ChatGPT for text, GLM-4V/Stable Diffusion for images). This creates a confound: MICL could be better at recognizing the artifacts of these generators—or at exploiting label-distribution shifts introduced by the undocumented transformation process—rather than being genuinely less reliant on spurious correlations. Since SPMSD is neither released nor validated, the 8.1-point accuracy gap over DMSD-CL in Table 2 does not establish the paper's generalizability claim. The MMSD result remains a useful empirical finding, but the robustness claim is unsubstantiated as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MICL, a multimodal sarcasm detection method that learns incongruities from three views (token-patch, entity-object, and sentiment), combines them via evidence-based credibility fusion, and augments both text and image training data. The authors also construct a new test set, SPMSD, to probe reliance on spurious correlations. On the public MMSD benchmark, MICL reports the best accuracy (92.08%) among compared methods, and on SPMSD it reports a large accuracy gain over baselines (68.7% vs. 60.6% for DMSD-CL). The central claims are that multi-view incongruity learning plus dual augmentation reduces spurious-correlation reliance and improves generalization.","tokens_in":15815,"tokens_out":2760,"duration_ms":27876,"significance":"If the central claims are substantiated, the paper makes a useful contribution: it draws attention to two concrete failure modes in multimodal sarcasm detection, proposes a multi-view architecture that goes beyond token-patch incongruity, and attempts to construct an evaluation set for spurious correlations. The paper also includes a reasonably thorough ablation study, a comparison with LVLMs, and an analysis of different text encoders. However, the significance is currently limited by verification gaps: the SPMSD set is neither released nor validated by human annotators, the reported gains lack statistical testing, and the \"incongruity\" modules are not formalized as incongruity computations. These issues are load-bearing for the paper's robustness claim.","major_comments":[{"comment":"The central robustness claim that MICL \"can effectively mitigate reliance on spurious correlations, showing better generalization ability on new data\" is supported only by SPMSD, a test set constructed by the authors using LLM-driven transformations (sentiment reversal, entity replacement, sarcastic-text regeneration, text-image swapping, and unimodal inputs). Appendix A reports no human validation of the transformed labels, no inter-annotator agreement, and no release of the dataset. Moreover, §3.4 trains MICL on augmented data generated with the same tool families (ChatGPT for text, GLM-4V/Stable Diffusion for images). This creates a confound: MICL's 8.1-point advantage over DMSD-CL on SPMSD may reflect familiarity with generator artifacts or label-distribution shifts rather than genuine robustness to natural spurious correlations. The authors should release SPMSD, report human validation of label preservation, and ideally evaluate on an independently constructed out-of-distribution benchmark before the robustness claim can be accepted.","section":"§3.2.2 and §3.2.3, Eqs. (10) and (12)"},{"comment":"The modules named \"entity-object incongruity learning\" and \"sentiment incongruity learning\" do not compute an explicit incongruity. Eq. (10) is a softmax-weighted sum over the concatenated text and image graph node features, and Eq. (12) is an MLP over sentiment polarities and text hidden states. Neither equation defines a contrast, difference, or mismatch between modalities or between expected and observed sentiment. If incongruity is meant to be captured implicitly through the downstream classifier, the paper should state this and justify why the weighted-sum pooling in Eq. (10) specifically encodes an incongruity rather than a generic multimodal feature. Without a formal definition, the central architectural claim that MICL \"learns incongruities from three views\" is not actually demonstrated.","section":"Table 1 and §4.3"},{"comment":"The reported MMSD improvement over the reproduced G2SAM* is 1.01 points in accuracy (92.08 vs. 91.07) and 1.03 points in macro-F1. No code, data, random seeds, or significance tests are provided, and the SPMSD results in Table 2 are likewise single-run numbers. Given the small margin over the strongest baseline, the authors should report the mean and standard deviation over multiple runs and perform a significance test (e.g., paired bootstrap or approximate randomization). This is necessary to determine whether the observed differences are stable rather than due to seed or initialization variance.","section":"§4.3"},{"comment":"The data augmentation description does not specify how the generated samples are verified or filtered. For example, the text augmentation prompt asks the LLM to reverse or preserve sarcasm, but no human or automatic check is described for whether the generated label matches the intended semantics. Since the contrastive loss in Eq. (15) treats same-label augmented samples as positives and different-label samples as negatives, label errors in augmented data directly corrupt the training signal. The authors should report a validation step for augmented samples or at least provide a random sample of augmented data with human judgments.","section":"§4.5, Table 4"}],"minor_comments":[{"comment":"There is a typo in the abstract: \"evaluate the the model's generalizability\" should be \"evaluate the model's generalizability.\" Similarly, Section 1 contains \"to a greater extend\" which should be \"to a greater extent.\"","section":"§3.2.2"},{"comment":"The text says \"If there is a dependency between two entites,\" where \"entites\" should be \"entities.\" Also, Eq. (9) writes α_{i,j}W^l g_j^l but the definition of α in Eq. (8) already includes LeakyReLU and an exponential; the notation is understandable but should be consistently indexed across Eq. (8) and Eq. (9).","section":"Figure 5"},{"comment":"The credibility study in Figure 5 is difficult to interpret because the subplot labels appear as unicode fragments (e.g., \"/uni00000013/uni00000011/...\") rather than readable scenario names. The figure should be regenerated with clear labels for the sarcastic, non-sarcastic, and SPMSD scenarios, and the y-axis should be defined.","section":"Appendix C"},{"comment":"In the LVLM comparison, the model name \"LLaV A 1.6\" is likely a typo for \"LLaVA 1.6.\" The paper should also clarify whether the LVLMs receive the same OCR-text as MICL; otherwise the comparison is not apples-to-apples, since MICL uses OCR-text generated by GLM-4V with manual proofreading.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible method and a useful MMSD result, but the robustness claim rests on an unreleased, unvalidated dataset whose construction is confounded with the training augmentation. I would view the paper favorably if the authors release SPMSD and the code, add human validation for the SPMSD labels, and report significance tests for the main comparisons. The formalization of the incongruity modules also needs to be strengthened; as written, the named modules do not compute an explicit incongruity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The MMSD result is real and the multi-view decomposition is sensible. The robustness claim, however, is not backed by the evidence as reported.\n\nWhat the paper does well: the token-patch, entity-object, and sentiment incongruity views are a reasonable extension of MILNet and DMSD-CL; the hybrid attention mechanism is a genuine attempt to reduce text bias; and the GLM-4V-refined OCR text plus dual text/image augmentation is a concrete, testable addition. The ablation study is systematic, and the authors include a limitation section that admits the 68% SPMSD accuracy is far from solved. The 92.08% accuracy on MMSD, 1.01 points over the reproduced G2SAM and more over DMSD-CL, is a solid empirical improvement for this niche.\n\nThe soft spot is the SPMSD test set and the central claim built on it. Appendix A says SPMSD is 1,000 randomly selected MMSD samples transformed by sentiment flipping, text regeneration, text-image swapping, and unimodal inputs. No human validation of labels, no inter-annotator agreement, and no release are reported. Meanwhile, Section 3.4 trains MICL on augmented data produced with the same tool families: ChatGPT for text, GLM-4V and Stable Diffusion for images. So the 68.7% SPMSD accuracy may reflect familiarity with generator artifacts rather than robustness to naturally occurring spurious correlations. The paper's own Table 2 shows DMSD-CL at 60.6%, but without a validated probe the 8.1-point gap does not establish the generalizability claim.\n\nThere are also two smaller issues. First, the entity-object module in Eq. (10) is a weighted sum over graph nodes, not an explicit incongruity between entities and objects; the sentiment module in Eq. (12) concatenates sentiment polarities with hidden states, again without a clear incongruity computation. This weakens the ``incongruity learning'' framing, though it does not by itself invalidate the empirical gains. Second, no code, data, seeds, or significance tests are provided, so the 1.01-point MMSD gain over G2SAM could be run-to-run variance.\n\nWho should read this: people working on multimodal sarcasm detection or on spurious-correlation robustness in multimodal classification. The MMSD result and the three-view framing are worth a serious referee. The robustness claim needs SPMSD released with validated labels and significance testing before it can be trusted. Recommendation: send to peer review, but make release and validation of SPMSD a condition of acceptance.","headline":"A useful MMSD result and a sensible three-view incongruity design, but the robustness-to-spurious-correlations claim rests on an unvalidated, unreleased LLM-generated test set and does not hold up as reported.","tokens_in":16335,"tokens_out":1750,"would_cite":false,"duration_ms":18299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that learning incongruity at token-patch, entity-object, and sentiment levels, fusing the views by estimated credibility, and training with dual text-image augmentation reduces reliance on spurious correlations in…","keywords":["multimodal sarcasm detection","spurious correlations","incongruity learning","multi-view learning","contrastive learning","data augmentation","OCR text","out-of-distribution test set"],"falsifier":"Have independent annotators verify the labels of all 1,000 SPMSD samples and rerun the comparison on the verified subset; if MICL's accuracy lead over DMSD-CL shrinks to the size of its MMSD lead or disappears, the spurious-correlation story is not supported.","tokens_in":15304,"feed_emoji":"😏","tokens_out":10409,"duration_ms":85137,"temperature":0.7,"pith_summary":"Existing multimodal sarcasm detectors often get the right answer for the wrong reason, latching onto textual patterns while ignoring images or fixating on non-critical words instead of emotional ones. This paper argues that the remedy is to learn incongruity explicitly in three views—token-patch, entity-object, and sentiment—then fuse the views by estimated credibility, and to train with balanced text and image augmentation plus contrastive learning. To demonstrate generalizability, the authors build SPMSD, a 1,000-sample test set of transformed examples designed to expose spurious correlations. They report that MICL outperforms all compared baselines on the standard MMSD benchmark and records a larger lead on SPMSD, which they take as evidence of reduced spurious-correlation reliance. If the claim is right, it gives a concrete, modular recipe for making multimodal sarcasm detection depend on task-relevant incongruities rather than surface cues.","feed_headline":"Learning three kinds of text-image mismatch improves sarcasm detection","feed_subtitle":"MICL scores 92% on the standard test and 68.7% on a set built to expose fake cues.","key_machinery":"The engine is a three-view incongruity learner. Token-patch incongruity uses hybrid cross-attention, letting text and image attend to each other symmetrically instead of always treating text as the query. Entity-object incongruity builds a text graph from dependency-parsed entities and an image graph from object regions, then applies graph attention to expose cross-modal mismatches at the object level. Sentiment incongruity compares sentiment polarity extracted from the source text and from OCR text, feeding those scores through a small network. A credibility module estimates how much to trust each view from pre-softmax evidence, and a contrastive loss over augmented text-image pairs pulls same-label examples together while pushing opposite-label examples apart. Refined, manually proofread OCR text is added as a supplementary input so the model can use text embedded in images.","core_discovery":"The paper's central claim is that a method called MICL (Multimodal Incongruities via Contrastive Learning) can make multimodal sarcasm detection both more accurate and less dependent on spurious cues. The method learns three complementary kinds of text-image incongruity: token-patch mismatches through a hybrid attention mechanism, entity-object mismatches through graph attention over semantic graphs, and sentiment mismatches between the text and OCR text read from the image. It then fuses the three views by estimated credibility rather than simple concatenation, enriches the training data with balanced text and image augmentations, and trains the representations with a contrastive loss. On the standard MMSD dataset the paper reports the highest accuracy among compared models, and on its new SPMSD set it reports a larger accuracy lead, which it interprets as evidence that the model has shifted away from spurious correlations.","pith_inferences":["If the robustness gain transfers, the same three-view incongruity recipe could apply to other incongruity-driven multimodal tasks, such as fake news or hate speech detection, where models also lean on surface cues.","A decisive isolation test would remove the sentiment view and flip emotional words in the test set; the paper's own ablations suggest sentiment drives much of the SPMSD gain, so this would reveal whether the mechanism is true incongruity modeling or a sentiment shortcut.","The SPMSD construction protocol—language-model-based regeneration, text-image swapping, and unimodal variants—could be reused as a cheap robustness audit for any multimodal classifier before building a human-labeled distribution-shift set."],"forward_implications":["MICL reports 92.08% accuracy and 91.81% macro-F1 on MMSD, outperforming all baselines including the strongest compared model, G2SAM, at 91.07% accuracy.","On SPMSD, MICL reports 68.7% accuracy versus 60.6% for DMSD-CL, a larger relative advantage than on MMSD, which the paper interprets as stronger generalization.","Ablation results show each added incongruity view raises both MMSD and SPMSD accuracy; sentiment incongruity contributes most on SPMSD, while entity-object incongruity contributes most on MMSD.","Credibility-weighted fusion improves results on both datasets, and the dual text-image augmentation raises SPMSD accuracy while leaving MMSD accuracy essentially unchanged.","Refined OCR-text from images helps all model variants on both datasets, whereas noisier OCR from prior work does not consistently help."],"supporting_citations":[{"why":"Provides the MMSD benchmark dataset that all main experiments and the SPMSD construction use.","marker":"Cai et al., 2019"},{"why":"Supplies the paper's premise that incongruity is a pivotal cue for sarcasm detection.","marker":"Joshi et al., 2015"},{"why":"DMSD-CL is the contrastive-learning debiasing baseline MICL is most directly compared with and outperforms.","marker":"Jia et al., 2024"},{"why":"MILNet is the token-patch incongruity baseline whose single-view scope MICL extends.","marker":"Qiao et al., 2023"},{"why":"Provides the earlier OCR-text and intra/inter-modality incongruity baselines; its noisy OCR is the reference point for the refined OCR.","marker":"Pan et al., 2020"},{"why":"Supplies the sentiment polarity resource used in the sentiment incongruity view.","marker":"Cambria et al., 2024"},{"why":"Establishes Dirichlet-based credibility estimation for multi-view fusion, which MICL adapts.","marker":"Han et al., 2021"},{"why":"Supplies the Beta-distribution evidence formulation for per-view credibility in binary classification.","marker":"Ma et al., 2024"},{"why":"Provides the bottom-up object-region features used to build the image graph in the entity-object view.","marker":"Anderson et al., 2018"}],"fun_headline_variants":["Three text-image mismatches teach AI to spot sarcasm better","Contrastive learning of text-image gaps boosts sarcasm spotting","New method reads text-image gaps to catch sarcasm","Sarcasm detection that ignores spurious cues via three incongruities","MICL: catching sarcasm by learning real text-image mismatches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The robustness claim rests on SPMSD being a faithful probe of natural spurious correlations, but its transformed samples are generated by language models without reported human validation of the new labels.","fun_headline_variants_meta":{"raw":{"variants":["Three text-image mismatches teach AI to spot sarcasm better","Contrastive learning of text-image gaps boosts sarcasm spotting","New method reads text-image gaps to catch sarcasm","Sarcasm detection that ignores spurious cues via three incongruities","MICL: catching sarcasm by learning real text-image mismatches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001869,"raw_usage":{"total_tokens":7315,"prompt_tokens":905,"completion_tokens":6410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":6324}},"tokens_in":521,"tokens_out":6410,"duration_ms":39626,"temperature":1.0,"reasoning_tokens":6324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:01:45.080276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators verify the labels of all 1,000 SPMSD samples and rerun the comparison on the verified subset; if MICL's accuracy lead over DMSD-CL shrinks to the size of its MMSD lead or disappears, the spurious-correlation story is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MMSD benchmark dataset that all main experiments and the SPMSD construction use."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the paper's premise that incongruity is a pivotal cue for sarcasm detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DMSD-CL is the contrastive-learning debiasing baseline MICL is most directly compared with and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MILNet is the token-patch incongruity baseline whose single-view scope MICL extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sentiment polarity resource used in the sentiment incongruity view."}],"review_version":1}