EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors

Dongyu Zhang; Feng Xia; Jing Ren; Xingyuan Lu; Yuxi Liu; Zhiyao Wu

REVIEW 3 major objections 7 minor 26 references

EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new Chinese-language dataset of 5,000 metaphorical advertisements, each annotated for metaphor occurrence, source and target domains, and one of ten fine-grained emotion categories, claims to be the first resource enabling fine-grained…

desk verdict A genuinely new Chinese multimodal metaphor-emotion dataset, but the emotion labels are the load-bearing part and their reliability (κ=0.58, arbitrary conflict resolution) needs real work before this is trustworthy as fine-grained ground truth. read the letter →

arxiv 2505.13483 v1 pith:KLDEU5W5 submitted 2025-05-12 cs.CL cs.AIcs.CV

Xingyuan Lu , Yuxi Liu , Dongyu Zhang , Zhiyao Wu , Jing Ren , Feng Xia This is my paper · ORCID

classification cs.CLcs.AIcs.CV

keywords multimodalmetaphorfine-grainedemotionclassificationChinesedatasetannotationtext-imagepairsadvertisingtaxonomypublicserviceadvertisement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EmoMeta, a Chinese multimodal dataset of 5,000 text–image advertisement pairs, and claims it is the first resource of its kind: every pair is manually annotated for whether a metaphor occurs, what the source and target domains are, and which of ten fine-grained emotion categories the metaphor conveys. The motivation is that metaphors carry much of the emotional load in advertising, yet existing emotion resources are mostly English, mostly coarse-grained (positive/negative), and usually text-only or image-only. A sympathetic reader takes the central claim to be that this dataset makes fine-grained multimodal metaphorical emotion classification in Chinese possible and that its annotation model can be reused. The paper also reports distributional regularities, such as fear and anticipation dominating public-service advertisements while surprise dominates commercial advertisements.

What carries the argument

The central object is the annotation model (Occurrence, source domain, target domain, emotion category) applied to multimodal advertisement pairs. The metaphor component relies on the source-to-target domain mapping, identified by judging an irreversible 'A is B' relationship across the text and image; the emotion component uses a ten-way taxonomy built from the six basic emotions plus trust and anticipation and a neutral class. The annotation process itself is the mechanism that carries the dataset: three groups of annotators independently label, disagreements are re-evaluated, and majority vote settles remaining conflicts, with inter-annotator agreement coefficients used as quality checks.

What would settle it

Re-annotate a random sample of the image–text conflict cases using a protocol that permits multiple emotion labels per pair; if more than a fifth of those pairs receive a label different from the published single label, or if a classifier trained on the published labels performs at chance against the multi-label annotations, the claim that the dataset supports fine-grained ground-truth emotion classification is undercut.

Watch

Extended reading notes

Core claim

The paper's central claim is that EmoMeta is the first Chinese multimodal dataset for fine-grained emotion classification in metaphorical advertisements. Each of its 5,000 text–image pairs carries an annotation of the form (Occurrence, source domain, target domain, emotion category), where the emotion categories are joy, love, trust, fear, sadness, disgust, anger, surprise, anticipation, and neutral. The emotion taxonomy combines the standard six basic emotions with two additional categories, trust and anticipation, plus a neutral class, and the metaphor annotation operationalizes the source-domain-to-target-domain mapping by detecting an irreversible 'A is B' relation across text, image, or both. When image and text convey conflicting emotions, the annotation takes the more intense emotion. The paper reports inter-annotator agreement coefficients from 0.58 for emotion categories to 0.68 for metaphor identification, and it documents a distributional pattern in which public-service advertisements mostly express fear and anticipation while commercial advertisements mostly express surprise, with the two types complementing each other across the whole dataset.

Load-bearing premise

The dataset's emotion labels are treated as reliable fine-grained ground truth even though annotators agreed only moderately on emotion categories ($\kappa=0.58$), and the rule that the stronger emotion wins image–text conflicts can systematically erase genuinely mixed emotions.

Editorial extensions

If this is right

If EmoMeta is a reliable resource, fine-grained emotion classification for Chinese multimodal metaphors becomes a concrete benchmark task rather than a data-scarce aspiration.
The documented emotion distributions give a testable expectation: public-service advertisements cluster around fear and anticipation, commercial advertisements around surprise, and the two types complement each other.
The annotation model (Occurrence, source domain, target domain, emotion category) can be transferred to other multimodal metaphor corpora, giving a common template for future annotation efforts.
The public release of the dataset provides a shared evaluation ground for models that must jointly reason about metaphor and emotion across text and image in Chinese.
Because 87% of the samples are public-service advertisements, models trained on EmoMeta will be especially sensitive to warning-driven fear rhetoric; applying them to other genres would be a separate evaluation.
An alternative annotation protocol that allows multiple emotion labels per pair, rather than forcing a single stronger emotion, could directly test whether the current labels erase mixed-emotion cases that fine-grained classification is supposed to capture.
The annotated source and target domains make EmoMeta usable for generation tasks, such as producing advertisement text or images conditioned on a chosen metaphor and emotion.
Chinese-specific metaphors, like the paper's example of 'dinosaur' meaning unattractive, suggest that cross-lingual transfer of metaphor-emotion models will be limited; EmoMeta can serve as a measurement ground for that gap.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this is a resource paper, not a methods paper. The contribution is a public dataset of 5,000 Chinese text-image metaphorical advertisements annotated for metaphor presence, source/target domains, and ten fine-grained emotion categories. That combination is new as far as I can tell, and the release is real. The annotation taxonomy mixes Ekman plus Plutchik with a neutral class, which is sensible for advertising. The paper is short but honest about data sources, including reuse of the authors' own earlier datasets.

What is good: the dataset fills a genuine gap. Existing multimodal metaphor datasets are mostly English or coarse-grained; EmoMeta adds fine-grained emotions in Chinese. The annotation process is described with three groups, majority voting, and Fleiss kappa. The distribution analysis of fear/anticipation in public service ads vs. surprise in commercial ads is plausible and useful. The GitHub release means the field can actually use it.

Now the soft spots. The label that carries the whole point is the emotion category, and that is the weakest annotation. κ = 0.58 is moderate by any standard, and the paper calls it “reliable.” It does not report per-category kappa or a confusion matrix, which matters a lot for a 10-way scheme with close neighbors like joy/love/trust and fear/sadness/disgust. Worse, Section 3.2 says image-text conflicts are resolved by taking “the more intense emotion” from either source. That rule is arbitrary and can silently erase genuine mixed emotions. The paper gives no statistics on how often conflicts occur, so we cannot tell how much damage the rule does. There are also no baseline experiments, no task validation, no attempt to show the labels correlate with anything external. For a dataset paper, that is a real omission.

The 87/13 public/commercial split is worth noting but not fatal; the complementarity argument is reasonable.

Who is this for? Researchers working on Chinese metaphor, multimodal emotion recognition, or advertising NLP. They will want the resource, but they should treat the emotion labels as noisy until the authors supply per-category agreement and a conflict-resolution analysis. The paper deserves a serious referee, but it should be sent back for major revision. I would not desk-reject it, and I would probably cite it if I worked in that niche, but I would not trust the fine-grained labels without independent inspection.

Recommendation: accept for peer review with a request for validation experiments and a more careful treatment of label reliability.

Referee Report

3 major / 7 minor

Summary. The manuscript introduces EmoMeta, a dataset of 5,000 Chinese text-image advertisement pairs annotated for metaphor occurrence, source/target domains, and fine-grained emotion categories (joy, love, trust, fear, sadness, disgust, anger, surprise, anticipation, neutral). The authors describe collection from four sources, an annotation protocol with three groups and majority voting, and report Fleiss kappa values for metaphor identification, source domain, target domain, and emotion. The paper presents descriptive statistics of emotion distributions in public-service versus commercial advertisements and argues the resource fills a gap in Chinese multimodal metaphor emotion research.

Significance. If the emotion labels are reliable, EmoMeta is a useful first resource for Chinese multimodal metaphor emotion research and is the first of its kind in Chinese. The authors release the data publicly, describe manual annotation, and provide separate analyses for advertisement types. The main risk is label reliability, because the reported agreement for the emotion dimension is moderate and the conflict-resolution rule is not validated.

major comments (3)

[Section 3.4] The reported Fleiss kappa of 0.58 for emotion categories is moderate by standard benchmarks (e.g., Landis and Koch), not the 'reliable' level the text claims. The paper provides no per-category kappa values, no pairwise confusion matrix, and no discussion of which emotion pairs are confused. Because the central contribution is fine-grained emotion labels, this evidence is insufficient to establish that the labels are stable ground truth.
[Section 3.2] The rule for resolving image-text emotion conflicts by selecting 'the more intense emotion from either source' is arbitrary and unquantified. The paper gives no statistics on how often image and text convey conflicting emotions, nor any analysis of whether the intensity rule preserves or distorts mixed-emotion cases. This rule can systematically collapse genuine mixed emotions into a single label, undermining the fine-grained classification claim. The authors should report conflict frequency, show examples of resolved conflicts, and evaluate whether the rule introduces systematic bias.
[Section 4] The dataset analysis is purely descriptive; no baseline experiments are reported. The claim that EmoMeta will 'facilitating further advancements' is unsupported without at least a simple text-only, image-only, and multimodal fusion baseline to show that the emotion labels are learnable and to calibrate expected performance. Adding such experiments would also provide an external check on label quality beyond inter-annotator agreement.

minor comments (7)

[Section 3.3] The subsection heading contains a typo: 'Metaphorcial or literal' should be 'Metaphorical or literal'.
[Section 2] The sentence 'Current research on metaphorical emotions in typically utilizes broad emotion classifications' has a grammatical error; 'in' should be removed or the sentence restructured.
[Section 3.4] The Kappa score is denoted with the symbol α, but the standard symbol for Fleiss' kappa is κ; using κ consistently would avoid confusion with Cronbach's alpha or Krippendorff's alpha.
[Section 3.3] There is a stray apostrophe in "It 's important to note" that should be corrected.
[Section 3.2] The phrase 'In cases where all data is metaphorical, we specifically note this condition' is vague; it is unclear what 'all data' refers to and how this condition is recorded in the annotation model.
[Section 3.1] The sentence 'However, this part of the data is not accompanied by a URL in the dataset of the data source' is confusing; please clarify whether the images are redistributed as files and how this complies with the original dataset's terms.
[Section 4 / Figure 3] Figure 3 is referenced but its axes and units are not described in the text; please specify whether values are counts or percentages and define the emotion labels used in the figure.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the emotion labels are newly produced manual annotations grounded in external emotion theories, and the authors' prior datasets are reused only as source material, not as the target labels.

full rationale

The paper's contribution is a new dataset, not a derived numerical prediction, and there is no equation or fitted parameter that is renamed as an output. The fine-grained emotion labels are produced by a manual annotation pipeline described in Sections 3.2 and 3.4, with inter-annotator agreement reported independently (Fleiss' alpha = 0.58 for emotion categories). The emotional taxonomy is explicitly taken from Ekman's six basic emotions and Plutchik's trust and anticipation categories, so the label scheme is externally grounded rather than defined in terms of the dataset's own outcome. The authors' prior work is cited for two legitimate reasons: as one of several data sources ('the primary source involves extracting advertisements with both images and text from a previously established dataset [22]') and as methodological precedent for metaphor annotation ('Building on the methodologies of Zhang et al. [23] and Zhang et al. [22]'). Neither citation supplies the fine-grained emotion labels, which are the paper's central claimed novelty. The self-citations are therefore present but not load-bearing: the dataset's emotion annotations are not inputs to themselves. The annotation rule that image-text conflicts are resolved by choosing 'the more intense emotion from either source' is an annotation convention that may raise reliability or validity concerns, and the moderate kappa is a correctness risk, but it does not reduce the claimed contribution to its own inputs. For these reasons, the paper shows no circular derivation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The paper relies on established emotion theories and metaphor definitions as axioms.

assumptions (3)

domain assumption The combination of Ekman's six basic emotions and Plutchik's trust and anticipation provides a complete taxonomy for emotion classification in advertisements.
The paper bases its emotion categories on these theories without empirical validation for the advertising domain.
domain assumption Metaphor identification relies on the existence of an irreversible 'A is B' mapping between source and target domains, following Lakoff and Johnson.
This definition is assumed without discussion of borderline cases or non-verbal metaphors.
domain assumption Annotators' subjective judgments after majority voting reflect true emotion categories.
The paper treats coder agreement as ground truth, with no external validation or gold standard.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors." pith.science (2026). https://pith.science/paper/KLDEU5W5

@misc{pith2026250513483,
  author       = {Pith},
  title        = {Pith review of: EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLDEU5W5}},
  note         = {Machine review of arXiv:2505.13483}
}

read the original abstract

Metaphors play a pivotal role in expressing emotions, making them crucial for emotional intelligence. The advent of multimodal data and widespread communication has led to a proliferation of multimodal metaphors, amplifying the complexity of emotion classification compared to single-mode scenarios. However, the scarcity of research on constructing multimodal metaphorical fine-grained emotion datasets hampers progress in this domain. Moreover, existing studies predominantly focus on English, overlooking potential variations in emotional nuances across languages. To address these gaps, we introduce a multimodal dataset in Chinese comprising 5,000 text-image pairs of metaphorical advertisements. Each entry is meticulously annotated for metaphor occurrence, domain relations and fine-grained emotion classification encompassing joy, love, trust, fear, sadness, disgust, anger, surprise, anticipation, and neutral. Our dataset is publicly accessible (https://github.com/DUTIR-YSQ/EmoMeta), facilitating further advancements in this burgeoning field.

Figures

Figures reproduced from arXiv: 2505.13483 by the authors.

**Figure 1.** Examples of multimodal metaphors The interaction between emotion and metaphor has been extensively studied by scholars across various disciplines, including psychology [4, 8], neuroscience [13], science learning [20], and natural language processing [14, 25]. Notably, with the rapid expansion of mass communication, there has been a considerable increase in multimodal emotional information that includes metaphors. … view at source ↗

**Figure 2.** An example of a metaphorical annotation 3.3 Annotation Metaphorcial or literal. Building on the methodologies of Zhang et al. [23] and Zhang et al. [22], our approach to determining if a sample is metaphorical focuses on the relational level, identifying metaphorical links between the source and target domains. This judgment involves both verbal and visual aspects, as detailed by Phillips and McQuarrie [17]. For tex… view at source ↗

**Figure 3.** Emotional distribution of EmoMeta 4 Dataset Analysis The proportion of public service advertisements and commercial advertisements is 87% and 13% respectively. The detailed statistical analysis of emotion classifications is shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 21 canonical work pages

[1]

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020)

arXiv 2020
[2]

Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. 2020. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941 (2020)

arXiv 2020
[3]

Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6, 3-4 (1992), 169–200

1992
[4]

Adam K Fetterman, Jessica L Bair, Marc Werth, Florian Landkammer, and Michael D Robinson. 2016. The scope and consequences of metaphoric thinking: Using individual differences in metaphor usage to understand how metaphor functions. Journal of Personality and Social Psychology 110, 3 (2016), 458

work page 2016
[5]

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378

1971
[6]

Charles Forceville. 2017. Visual and multimodal metaphor in advertising: Cultural perspectives. Styles of communication 9, 2 (2017)

work page 2017
[7]

Charles Forceville, Eduardo Urios-Aparisi, et al . 2009. Multimodal metaphor . Vol. 11. Mouton de Gruyter Berlin

work page 2009
[8]

Baihui Huangfu and Wenjuan Cheng. 2025. Cognitive computing method based on decoding psychological emotional states. International Journal of Cognitive Computing in Engineering 6 (2025), 32–43

work page 2025

Show all 26 references

[9]

Zoltán Kövecses. 2010. Metaphor and culture. Acta Universitatis Sapientiae, Philologica 2, 2 (2010), 197–220

2010
[10]

George Lakoff and Mark Johnson. 1980. The metaphorical structure of the human conceptual system. Cognitive science 4, 2 (1980), 195–208

1980
[11]

Yucheng Li, Frank Guerin, and Chenghua Lin. 2023. The secret of metaphor on expressing stronger emotion. arXiv preprint arXiv:2301.13042 (2023)

2023 arXiv
[12]

Dongjun Lim and Yun-Gyung Cheong. 2024. Integrating Plutchik’s Theory with Mixture of Experts for Enhancing Emotion Classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . 857–867

2024
[13]

Josie E Malinowski and Caroline L Horton. 2015. Metaphor and hyperassocia- tivity: the imagination mechanisms behind emotion assimilation in sleep and dreaming. Frontiers in psychology 6 (2015), 150080

2015
[14]

Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. InProceedings of the fifth joint conference on lexical and computational semantics . 23–33

2016
[15]

Saif M Mohammad. 2021. Sentiment analysis: Automatically detecting valence, emotions, and other affectual states from text. In Emotion measurement. Elsevier, 323–379

2021
[16]

Sudarshan Pant, Hyung-Jeong Yang, Eunchae Lim, Soo-Hyung Kim, and Seok- Bong Yoo. 2023. PhyMER: Physiological Dataset for Multimodal Emotion Recog- nition with Personality as a Context. IEEE Access (2023)

2023
[17]

Barbara J Phillips and Edward F McQuarrie. 2004. Beyond visual metaphor: A new typology of visual rhetoric in advertising. Marketing theory 4, 1-2 (2004), 113–136

2004
[18]

Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion. Elsevier, 3–33

1980
[19]

Ronald Rivest. 1992. The MD5 message-digest algorithm . Technical Report

1992
[20]

Powered by emotions

Ella Yonai and Ron Blonder. 2025. “Powered by emotions”: Exploring emotion induction in out-of-school authentic science learning. Journal of Research in Science Teaching 62, 2 (2025), 553–575

2025
[21]

Sumaia Mohammed Zaid, Fonny Dameaty Hutagalung, Harris Shah Bin Abd Hamid, and Sahar Mohammed Taresh. 2025. The power of emotion regula- tion: how managing sadness influences depression and anxiety? BMC psychology 13, 1 (2025), 1–12

2025
[22]

Dongyu Zhang, Jingwei Yu, Senyuan Jin, Liang Yang, and Hongfei Lin. 2023. Mul- tiCMET: A Novel Chinese Benchmark for Understanding Multimodal Metaphor. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 6141– 6154

2023
[23]

Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin
[24]

Enshi Zhang, Rafael Trujillo, and Christian Poellabauer. 2024. The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 13960–13970

2024
[25]

Linhao Zhang, Li Jin, Guangluan Xu, Xiaoyu Li, Cai Xu, Kaiwen Wei, Nayu Liu, and Haonan Liu. 2024. CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. ...

2024
[2021]

Multimet: A multimodal dataset for metaphor understanding. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 3214–3225

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020)

arXiv 2020

[2] [2]

Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. 2020. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941 (2020)

arXiv 2020

[3] [3]

Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6, 3-4 (1992), 169–200

1992

[4] [4]

Adam K Fetterman, Jessica L Bair, Marc Werth, Florian Landkammer, and Michael D Robinson. 2016. The scope and consequences of metaphoric thinking: Using individual differences in metaphor usage to understand how metaphor functions. Journal of Personality and Social Psychology 110, 3 (2016), 458

work page 2016

[5] [5]

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378

1971

[6] [6]

Charles Forceville. 2017. Visual and multimodal metaphor in advertising: Cultural perspectives. Styles of communication 9, 2 (2017)

work page 2017

[7] [7]

Charles Forceville, Eduardo Urios-Aparisi, et al . 2009. Multimodal metaphor . Vol. 11. Mouton de Gruyter Berlin

work page 2009

[8] [8]

Baihui Huangfu and Wenjuan Cheng. 2025. Cognitive computing method based on decoding psychological emotional states. International Journal of Cognitive Computing in Engineering 6 (2025), 32–43

work page 2025