{"id":"fa6012b6-9a28-429f-8405-0c3fb88bd467","arxiv_id":"2506.06987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MultiMM provides the first Chinese-English multimodal metaphor benchmark, and the SEMD model shows that adding sentiment embeddings improves metaphor detection and sentiment analysis across both cultures.","lead":"This paper introduces MultiMM, a new bilingual English-Chinese dataset of 8,461 advertisement text-image pairs annotated for multimodal metaphors and sentiment. It also proposes SEMD, a sentiment-enriched baseline model, and argues that cultural bias significantly changes metaphor processing performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-cultural comparisons are confounded by asymmetric data collection in Section 3.1; the cultural-bias claim needs a matched-sample or same-pipeline check.","rationale":"The reader's weakest assumption—that collection procedure differences rather than culture explain the EN/CN contrasts—is exactly the most load-bearing issue for the paper's central claim. The dataset is the primary contribution, and every cultural-bias analysis (source-domain distributions, sentiment distributions, cross-lingual model comparisons) passes through the Section 3.1 pipelines. Because the Chinese data were keyword-searched for metaphor-laden material while the English data were cleaned from a pre-existing ad collection, the two subsets are not exchangeable; any observed asymmetry is underdetermined between 'culture' and 'sampling'. A matched-category or same-pipeline replication is a concrete way to adjudicate. I agree with the reader's conditional verdict: the paper is promising but not yet conclusive. I also note a secondary risk—the SEMD sentiment feature for the metaphor-detection task is underspecified and could use gold sentiment labels at test time, which would inflate the reported 80.16/77.79 F1 scores; this should be checked in the released code. Even if the model claim were repaired, the dataset-as-cultural-benchmark claim still requires the matched-collection check.","tokens_in":18138,"tokens_out":3860,"duration_ms":44410,"concrete_test":"Restrict both subsets to overlapping advertisement categories (e.g., smoking, animal protection, public service, automotive) and rerun the Section 4 distribution analyses and Table 2 model comparisons; alternatively, re-collect English samples using the same keyword-translated Baidu-style pipeline with Master Metaphor List terms and the same cleaning criteria as the Chinese subset. If the EN/CN source-domain and sentiment differences shrink to noise or reverse, the cultural-bias conclusion is an artifact of Section 3.1's asymmetric collection procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 describes two different pipelines that are treated as directly comparable. The 4,397 Chinese samples were obtained by native researchers searching Baidu with metaphor-related keywords, product/service categories, and Master Metaphor List terms. The 4,064 English samples were drawn from an existing advertisement dataset and then cleaned by removing duplicates, non-advertisements, blurry images, and small images, followed by OCR text extraction. These pipelines differ in sampling source, selection criteria, and cleaning steps, so any distributional difference reported in Section 4 (source-domain vocabulary, sentiment category balance) and any cross-lingual performance gap in Section 6.3 may reflect how the corpora were assembled rather than cultural differences in multimodal metaphor. For example, keyword-driven search for metaphors will naturally surface more symbolic or metaphorically rich content, and the English source dataset from Ye et al. (2021) may contain a different genre and topic mix, including images below the Chinese pipeline's selection threshold. The central conclusion that 'cultural bias matters' depends on interpreting EN/CN differences as cultural, but this confound is not acknowledged in the Limitations section, which only notes domain and language coverage. If a matched collection or matched-category analysis does not reproduce the reported asymmetries, the paper's central claim would be substantially weakened.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiMM, a new dataset of 8,461 Chinese and English text-image advertisement pairs with annotations for metaphor occurrence, source/target domains, and sentiment; and SEMD, a model that fuses text, image, and sentiment features. Experiments compare SEMD with 18 baselines on metaphor detection and sentiment analysis, claiming that SEMD outperforms baselines and that observed distributional and performance differences between Chinese and English subsets demonstrate cultural bias in multimodal metaphor processing. The dataset and code are released.","tokens_in":18546,"tokens_out":4342,"duration_ms":45116,"significance":"If the cross-cultural comparisons are valid, MultiMM would be a valuable resource: it is, to my knowledge, the first bilingual/bicultural multimodal metaphor benchmark for advertising; the annotation procedure is documented and inter-annotator agreement (Fleiss kappa 0.66-0.82) is reasonable; and the release of data and code supports reproducibility. The SEMD results are suggestive but their interpretation hinges on the data-collection comparability and the provenance of the sentiment feature.","major_comments":[{"comment":"The two subsets are assembled through different pipelines, so the paper's central cross-cultural comparisons are confounded. The Chinese subset (4,397 samples) is collected by native researchers searching Baidu with metaphor-related keywords and Master Metaphor List terms, while the English subset (4,064 samples) is drawn from an existing advertisement dataset (Ye et al., 2021) and then cleaned by removing duplicates, non-advertisements, blurry images, and small images. Consequently, the distributional differences in source-domain vocabulary (Figure 3), sentiment balance (Figure 4), and the performance gaps in Section 6.3 could reflect collection artifacts (e.g., keyword-driven retrieval over-sampling metaphorically rich Chinese content, and the English source having a different genre/topic mix) rather than cultural differences. The Limitations section only notes domain and language coverage and does not address this asymmetry. A matched-collection design (same search/selection procedure in both languages) or a matched-category analysis (e.g., comparing only paired product/service and sentiment categories) is needed to support the claim that cultural bias, rather than pipeline differences, drives the observed asymmetries.","section":"§3.1"},{"comment":"The provenance of the sentiment feature Si used in metaphor detection is not specified. In Section 5, metaphor detection concatenates image, text, and sentiment features (PMeta = Sigmoid(Fusion(concat(I_i, T_i, S_i)))), while sentiment analysis uses only I_i and T_i. If S_i is derived from the gold sentiment annotations, which were produced in the same annotation session as the metaphor-occurrence labels, then SEMD has access to label information at test time and is not comparable to the 18 baselines; the ablation in Table 3 would then measure label leakage rather than the value of sentiment information. The paper should state explicitly whether S_i comes from an independent sentiment classifier (and if so, how it is trained and whether it is frozen) or from gold labels. If it comes from gold labels, the metaphor-detection experiments should be rerun with predicted sentiment features or without sentiment features.","section":"§5, Figure 6, Table 3"},{"comment":"Several interpretive claims go beyond what the data can support. For example, Section 4.2 states that 'the complete absence of negative sentiment in English metaphorical advertisements may reflect cultural taboos,' but Figure 4 reports 0.69% negative in English metaphorical advertisements, not complete absence, and the near-zero count could be a byproduct of the English pipeline's genre mix or selection. Similarly, Section 6.6 attributes the lower Chinese metaphor-detection accuracy to Chinese metaphors being 'more subtle' and English metaphors 'more straightforward,' without controlling for annotator language, text length (average 33 vs. 15 words in Table 1), or collection differences. These claims should be rephrased as hypotheses or supported by a matched analysis.","section":"§4.2, §6.6, §7"}],"minor_comments":[{"comment":"In the SEMD row of the Chinese sentiment analysis results, '73.4070.66 70.51' appears to be missing a space between the accuracy and precision values; please fix the formatting.","section":"Table 6"},{"comment":"The sentence 'we also provides a case study' should be 'we also provide a case study'; please correct this grammatical error.","section":"§6.6"},{"comment":"The cleaning step 'removing images that are blurry or smaller than 350 × 350 pixels' is applied only to the English subset; state whether an analogous size/quality filter was applied to the Chinese subset and, if not, why.","section":"§3.1"},{"comment":"The claim that sentiment information is a universally recognized feature for multimodal metaphor understanding would benefit from a few more explicit citations to prior work on sentiment and metaphor beyond Mohammad et al. (2016).","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The central result is promising but rests on the comparability of the two collection pipelines. The editor may wish to ask the authors for a matched-sample analysis or explicit acknowledgment of this confound before further consideration. The self-citation pattern (MultiMET, MultiCMET, MET-MEME from the same group) is not by itself problematic given the topic, but the annotation criteria should be shown to be consistent across the new dataset and those prior datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution. MultiMM is the first bilingual (Chinese–English) multimodal metaphor dataset I know of, and the authors have released the data and code with detailed annotations. That alone makes the paper worth considering. The annotation process is careful (kappa 0.66–0.82), the guidelines follow established work like MultiMET, and the 18-baseline comparison is thorough.\n\nBut the central claim—that measured differences reflect cultural bias—does not stand on the current evidence. The Chinese and English subsets are collected through completely different pipelines. Chinese ads are found via Baidu keyword searches with metaphor-related terms; English ads are curated from an existing advertisement dataset and then filtered. Keyword-based searching will naturally surface metaphor-rich content, and the English source corpus has its own genre and topic mix. So the distributional differences in Section 4 and the performance gaps in Section 6 could easily reflect sampling artifacts, not culture. The Limitations section does not acknowledge this, which is a missed opportunity.\n\nA second, smaller soft spot is the sentiment feature in SEMD. For metaphor detection, the model concatenates a sentiment vector with text and image features, but the paper never says where that vector comes from at prediction time. If it is the annotated sentiment label, that's label leakage and the reported F1 gains (1–3 points) are meaningless. If it is a separately predicted sentiment, then the model needs to be described. As written, it's underspecified.\n\nAlso, the improvements over the best baseline are small, and no significance testing is provided. That is a minor concern for a benchmark paper, but worth noting.\n\nWho is this for? Researchers working on multimodal metaphor or cultural variation in figurative language will want to know about MultiMM and should probably get the dataset. But the broader conclusion that 'cultural bias matters' needs a matched-sample or same-pipeline validation before it can be accepted.\n\nRecommendation: send to peer review. It is a novel dataset with real potential, and the flaws are fixable with careful revision. A serious referee should push on the collection confound and the sentiment feature; the authors ought to be able to respond.","headline":"Valuable new bilingual multimodal metaphor dataset, but the cultural-bias claim is undermined by asymmetric data collection and an underspecified sentiment feature.","tokens_in":18910,"tokens_out":2485,"would_cite":false,"duration_ms":24928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 8,461-pair bilingual dataset exposes measurable cultural bias in multimodal metaphor processing, and a sentiment-enriched model beats all 18 baselines on both languages.","keywords":["multimodal metaphor","cross-cultural bias","benchmark dataset","sentiment embeddings","metaphor detection","sentiment analysis","Chinese-English advertising","text-image understanding"],"falsifier":"Take a matched-sample test: collect English ads with the same Baidu keyword procedure used for Chinese and Chinese ads from the same public advertisement source used for English, then rerun SEMD and the baselines; if the English-Chinese F1 differences disappear or reverse, the study's cultural-bias conclusion is not supported.","tokens_in":17988,"feed_emoji":"🎭","tokens_out":5040,"duration_ms":54662,"temperature":0.7,"pith_summary":"Metaphors vary across cultures, yet most automatic metaphor processing is trained on English data. This paper introduces MultiMM, a benchmark of 8,461 Chinese and English text-image advertisement pairs annotated for metaphor occurrence, source and target domains, and sentiment, making cross-cultural multimodal metaphor understanding measurable. It also proposes SEMD, a model that adds sentiment embeddings to text and image features and fuses them in a cascade. SEMD outperforms all 18 textual, visual, and multimodal baselines, reaching F1 scores of 80.16% and 77.79% for metaphor detection and 75.69% and 70.51% for sentiment analysis in English and Chinese. If the results hold, sentiment information is a transferable signal for metaphor understanding across cultures, and culturally skewed training data can mislead NLP progress.","feed_headline":"8,461 ads expose cultural bias in metaphor AI","feed_subtitle":"A Chinese-English benchmark plus a sentiment-aware model that beats all 18 baselines on both tasks.","key_machinery":"The load-bearing mechanism is the annotation model $(Occurrence, Target, Source, SentimentCategory)$ applied to each text-image pair, which turns cultural metaphor variation into labeled training and evaluation data. On the modeling side, the engine is SEMD's three-branch fusion: BERT encodes text and sentiment, ViT encodes the image, and a cascading fusion layer concatenates the 768-dimensional vectors $I_i$, $T_i$, and $S_i$ before a feed-forward network and sigmoid classifier. The sentiment branch is the novel ingredient, and the ablation study shows that including it with concatenation gives the largest gains.","core_discovery":"The paper's central claim is that cultural background measurably changes how multimodal metaphors work, and that a model can exploit this by adding sentiment as a universal auxiliary channel. The evidence is MultiMM: 4,397 Chinese and 4,064 English text-image advertisement pairs with annotations for metaphor occurrence, target and source domain vocabulary (including verbalized visual domains), and sentiment category, with moderate to near-perfect inter-annotator agreement. On this dataset, SEMD, which concatenates text, image, and sentiment features and fuses them in a cascade, beats every baseline on both tasks in both languages. The paper further argues that the data show culture-specific source domains and sentiment distributions, and that direct Chinese-English translation degrades metaphor detection, consistent with metaphors carrying culture-bound meaning.","pith_inferences":["The same sentiment-enrichment trick could be tested on text-only metaphor detection, where sentiment embeddings might also improve cross-lingual transfer without images.","A human-pairing study could use MultiMM to ask whether bicultural annotators, not just models, also diverge in metaphor detection, which would separate cultural cognition from model bias.","The dataset could serve as pretraining or evaluation data for multimodal LLMs aimed at Chinese-English advertising, where current large models struggle on both tasks.","Matched-sampling extensions of MultiMM to other genres such as social media or news would reveal whether the observed cultural patterns are specific to advertising or general to multimodal metaphor."],"forward_implications":["English-only metaphor benchmarks likely overstate how well models understand metaphor in other cultures; MultiMM provides a way to measure and close that gap.","Sentiment embeddings act as a cross-cultural bridge: adding them improves metaphor detection even though sentiment is not annotated for that purpose.","Direct translation between Chinese and English degrades metaphor detection, so cross-lingual transfer for figurative language needs culturally aware representations, not just machine translation.","The strongest baselines are multimodal graph or caption-augmented models, positioning visual grounding as necessary for metaphor understanding.","MultiMM supplies a shared testbed for future cross-cultural multilingual metaphor systems to compare against.","The dataset and code are public, so the benchmark can be reused directly by other researchers."],"supporting_citations":[{"why":"Supplies the English advertisement corpus (4,064 samples) that MultiMM cleans and re-annotates; removing it removes the English half of the benchmark.","marker":"Ye et al., 2021"},{"why":"MultiMET provides the annotation convention for metaphorical/literal labels and source and target domains that MultiMM follows, and contributes the mBERT-Res baseline model.","marker":"Zhang et al., 2021"},{"why":"The Master Metaphor List guides Chinese keyword selection for source and target domains, shaping which metaphors enter the dataset.","marker":"Lakoff, 1994"},{"why":"Evidence that metaphors carry stronger emotion than literal language; motivates the sentiment annotation in MultiMM and the sentiment branch in SEMD.","marker":"Mohammad et al., 2016"},{"why":"Supplies the visual metaphor identification method based on incongruity and a non-reversible 'A is B' relation that the annotation procedure adopts.","marker":"Šorm and Steen, 2018"},{"why":"BERT and multilingual BERT are the text encoders used in SEMD and among the strongest text-only baselines.","marker":"Devlin et al., 2019"},{"why":"ViT is the image encoder used in SEMD and the best visual-only baseline, establishing the visual branch of the model.","marker":"Dosovitskiy et al., 2020"}],"fun_headline_variants":["8,461 ads expose how culture skews metaphor AI","Sentiment-aware model learns metaphor across Chinese and English","Cross-cultural metaphor benchmark: sentiment beats all 18 baselines","Why metaphors aren't universal: 8,461 ads show culture matters","New dataset reveals cross-cultural metaphor patterns in ads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that differences between the Chinese and English subsets measure cultural bias, even though the two subsets were collected by different procedures—Chinese ads through keyword search on Baidu and English ads from an existing cleaned advertisement dataset—so sampling or selection differences could also explain part of the gap.","fun_headline_variants_meta":{"raw":{"variants":["8,461 ads expose how culture skews metaphor AI","Sentiment-aware model learns metaphor across Chinese and English","Cross-cultural metaphor benchmark: sentiment beats all 18 baselines","Why metaphors aren't universal: 8,461 ads show culture matters","New dataset reveals cross-cultural metaphor patterns in ads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000695,"raw_usage":{"total_tokens":3129,"prompt_tokens":920,"completion_tokens":2209,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2127}},"tokens_in":536,"tokens_out":2209,"duration_ms":17396,"temperature":1.0,"reasoning_tokens":2127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:43:53.526373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a matched-sample test: collect English ads with the same Baidu keyword procedure used for Chinese and Chinese ads from the same public advertisement source used for English, then rerun SEMD and the baselines; if the English-Chinese F1 differences disappear or reverse, the study's cultural-bias conclusion is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Master Metaphor List guides Chinese keyword selection for source and target domains, shaping which metaphors enter the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the visual metaphor identification method based on incongruity and a non-reversible 'A is B' relation that the annotation procedure adopts."}],"review_version":1}