Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chumor 2.0 claims current LLMs cannot reliably judge whether an explanation captures a Chinese joke: the best model scores 60.3% versus 78.3% for humans.

desk verdict A genuinely new Chinese humor explanation dataset with a solid multi-model evaluation, but the label reliability and a few overclaims need fixing before the headline results can be fully trusted. read the letter →

arxiv 2412.17729 v1 pith:CYBM7XMV submitted 2024-12-23 cs.CL cs.AI

Ruiqi He , Yushu He , Longju Bai , Jiarui Liu , Zhenjie Sun , Zenghao Tang , He Wang , Hanchen Xia
show 2 more authors
Rada Mihalcea Naihao Deng
This is my paper · ORCID
classification cs.CLcs.AI
keywords ChinesehumorunderstandingLLMevaluationexplanationculturallyspecificchain-of-thoughtpromptingbenchmarkdatasetRuoZhiBa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chumor 2.0 constructs the largest Chinese humor explanation dataset to date, 3,339 joke–explanation pairs sourced from the Chinese forum Ruo Zhi Ba, and uses it to ask whether LLMs can decide if an explanation fully accounts for a joke's humor. The paper's central claim is that current LLMs largely fail at this culturally grounded reasoning task: the best tested models reach 60.3% accuracy (Matthew correlation 0.29), barely above the random baseline, while native-speaker annotators reach 78.3% accuracy (MCC 0.60). It further claims that chain-of-thought prompting does not reliably help and often hurts, and that human-written joke explanations are strongly preferred over those produced by GPT-4o and ERNIE4-turbo in pairwise A/B comparisons. A sympathetic reader would take away that Chinese humor comprehension, not just humor detection, remains an open problem for LLMs.

What carries the argument

The organizing instrument is the humor explanation classification task: each instance pairs a joke with an explanation written by GPT-4o or ERNIE4-turbo, and the label is a majority vote by five native Chinese speakers over 'fully explains' versus 'partially/does not explain.' Jokes are grouped into six named mechanisms—cultural, situational, pun-based, homophonic, glyph-based, and cross-lingual—so that errors can be traced to the kind of knowledge the model lacks. The prompting variants (direct vs. chain-of-thought) and the pairwise human-vs-LLM explanation preference test are the other load-bearing pieces.

What would settle it

Take a random sample of Chumor instances, recruit independent native Chinese speakers who have no stake in the project, and have them re-apply the same good/bad label. If their agreement with the published labels is near chance (e.g., Cohen's kappa below 0.2), then the 60.3%-versus-78.3% gap would be an artifact of one annotation team's subjective judgments rather than a stable measure of humor understanding.

Watch

Extended reading notes

Core claim

The discovery is a negative result stated as a benchmark: when asked to classify whether a given explanation 'fully explains' a joke, ten LLMs perform poorly on Chumor, with accuracy 44.6–60.3% and MCC up to 0.29, versus human accuracy 78.3% and MCC 0.60. The paper treats this as evidence that culturally specific Chinese humor understanding is not solved by current LLMs. Error analysis attributes failures to missing cultural knowledge, missed puns, missed homophones, missed glyph-based visual wordplay, parsing ambiguity, hallucinations, and cross-lingual phonetic connections; the authors also find that CoT prompting often makes models over-justify incorrect explanations, raising false-positive rates.

Load-bearing premise

The load-bearing premise is that the majority-vote labels of five native-speaker authors—deciding whether an LLM-generated explanation 'fully explains' a joke—are reliable ground truth for humor understanding; the paper does not report inter-annotator agreement for these labels or validate that explanation adequacy tracks comprehension, so if those labels are noisy the reported human–LLM gap is not a clean measure.

Editorial extensions

If this is right

  • Any claim of human-level humor understanding in LLMs should be tested on explanation adequacy, not only on detecting or generating jokes.
  • Chain-of-thought prompting should not be assumed to improve humor reasoning; in this benchmark it often lowers accuracy and inflates false positives by making models rationalize weak explanations.
  • The six joke-type breakdown gives a diagnostic target: models fail differently on cultural, homophonic, and glyph-based humor, so progress can be tracked per mechanism.
  • Human explanation quality is measurably higher than LLM explanation quality, setting a concrete bar for generation systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Explanation adequacy is a proxy for humor understanding; a model might pass by matching explanation style rather than by 'getting' the joke, so a stricter test would have models produce an explanation from scratch and have humans judge it.
  • The label distribution is imbalanced (1,887 bad vs. 1,454 good) and the human A/B preference agreement is only 61.4%, so part of the reported gap may reflect subjectivity in what counts as a full explanation rather than pure comprehension failure.
  • A natural extension is to use human-written explanations as the candidate texts and see whether LLMs still underperform; this would separate label noise from genuine reasoning limits.
  • The cultural jokes require encyclopedic Chinese knowledge; a model trained on more Chinese-centric data might close part of the gap, as the paper hints with ERNIE4-turbo's lower cultural-error rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Chumor, a Chinese humor explanation dataset of 3,339 (joke, explanation, label) instances sourced from the Ruo Zhi Ba forum. Five native-speaker authors label each LLM-generated explanation (from GPT-4o or ERNIE4-turbo) as fully explaining the joke or not. The authors evaluate ten LLMs under direct and chain-of-thought prompting, reporting that the best model reaches 60.3% accuracy and MCC 0.29, well below a human study estimate of 78.3% accuracy and MCC 0.60. They also report an A/B preference study in which six annotators prefer human-written explanations over LLM-generated ones, and an error analysis of failure types. The dataset, code, and leaderboard are released.

Significance. If the labels are reliable, Chumor is a valuable resource for studying culturally specific Chinese humor understanding and provides a challenging benchmark for LLMs. The paper's main empirical result—that all ten tested LLMs perform near chance while humans reach 78.3% accuracy—would be an important finding for the field. The release of the dataset, prompts, and code is a concrete contribution. The error taxonomy (cultural unawareness, pun-based, homophonic, glyph-based, parsing, cross-lingual) is a useful qualitative analysis. However, the central comparison depends on ground-truth label reliability, which is not yet sufficiently established.

major comments (5)
  1. [Section 3, 'Humor Explanation Classification'; Section 5.2] The gold labels in Chumor are produced by a majority vote of five authors, but no inter-annotator agreement (e.g., Fleiss' kappa) is reported for these votes. The human study in Section 5.2 reports an average accuracy of 78.3% for three independent annotators against the gold labels, but this is agreement with the gold set and does not measure agreement among the annotators themselves; no per-annotator scores or IAA are given. Because the central claim of an 18-point gap between LLMs (60.3%) and humans (78.3%) is measured against these gold labels, the paper should provide reliability evidence for the labels, at least on a sample, and for the human study annotators.
  2. [Abstract and Section 1] The abstract and introduction claim that Chumor 'exceeds the size of existing humor datasets,' but Table 2 lists several larger datasets (e.g., rJokes: 57k, #HashtagWars: 13k, TCHD: 26k). This claim is contradicted by the paper's own table and should be corrected to, for example, 'the first Chinese humor explanation dataset.'
  3. [Section 5.1 and Figures 2/16] The error analysis by joke type is based on only 200 jokes, but the paper does not report the number of jokes per type. For types such as Glyph-based and Cross-lingual, reported accuracies of 0% or 100% for several models suggest very small per-type samples. Without per-type sample sizes and confidence intervals, the claim that 'model performance varies significantly across different joke types' is not statistically supported.
  4. [Section 5.4 and Appendix B] The A/B preference study uses human explanations written by one of the authors as the gold standard, but the quality of these explanations is not independently validated before comparison with LLM explanations. The six preference annotators reach only 61.4% agreement, which is reported as 'decent' without a chance-corrected measure such as Cohen's kappa. Please report kappa and consider having the author-written explanations evaluated by independent annotators before drawing conclusions about human-vs-LLM explanation quality.
  5. [Section 5, Figure 1] The 'random' baseline in Figure 1 is 50%, but the dataset is imbalanced: 1,454 good vs 1,887 bad explanations (56.5% bad). A trivial classifier that always predicts 'bad' achieves 56.5% accuracy, so the best LLM accuracy of 60.3% is only 3.8 points above the majority-class baseline. The paper should report the majority-class baseline and statistical significance tests to support the claim that LLMs are 'slightly above random.'
minor comments (5)
  1. [Section 1] The phrase 'three folds' should be 'threefold' or 'three-fold.'
  2. [Section 3] The word 'generae' should be 'generate.'
  3. [Section 1, Contribution 3] The phrase 'better than those produced by GPT-4o and ERNIE4-turborurging., encouraging future research' is garbled and should be reworded.
  4. [Figure 4 caption] The caption contains typos: 'wining rate' should be 'winning rate' and 'preffered' should be 'preferred.'
  5. [Section 5.2] The annotation instructions given to the three human study participants are not reported; please provide them (e.g., whether they used the same DP prompt) for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark results are direct evaluations against externally human-labeled gold data, with no fitted parameter or definitional reduction.

full rationale

Chumor is an externally sourced benchmark whose labels are produced by a majority vote of five native-speaker authors, independent of the evaluated models' predictions. The central comparison (LLM accuracy 60.3% vs. human accuracy 78.3%, and MCC 0.29 vs. 0.60) is a direct evaluation of held-out model outputs against a fixed gold-label set; there are no fitted parameters, no equations, and no derivation that could reduce to its own inputs. The A/B preference study compares author-written human explanations with LLM explanations, and although the human side is curated in-house rather than independently elicited, the preference judgments come from six outside college-student annotators, so the observed preference is not forced by construction. The only self-citation used as support is Deng et al. (2023) in Appendix B, cited to justify that a 61.4% agreement rate is acceptable given the inherent subjectivity of humor; this citation is not load-bearing for any headline result. Concerns about the absence of inter-annotator agreement for the five-author label votes, and about whether 'fully explains the joke' is a valid proxy for humor understanding, are validity and correctness concerns rather than circularity: the paper does not derive its conclusions from the very claims it is trying to establish. Therefore, no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central benchmark rests on domain assumptions about data representativeness and label reliability, not on fitted parameters or invented entities. These assumptions are standard for subjective NLP benchmarks but should be validated with agreement metrics and external checks.

assumptions (3)
  • domain assumption Jokes sourced from Ruo Zhi Ba are representative of Chinese humor and appropriate for benchmarking humor understanding.
    The dataset is built entirely from this platform, and the paper generalizes from these jokes to Chinese humor understanding in Section 3.
  • domain assumption A binary judgment of whether an explanation 'fully explains' a joke is a valid proxy for humor understanding.
    This is the task definition in Section 3 and 4; no validation against other humor comprehension measures is provided.
  • domain assumption The majority vote of five native Chinese-speaking authors is a reliable gold standard for explanation adequacy.
    The labels are produced in Section 3 without inter-annotator agreement statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chumor 2.0: Towards Benchmarking Chinese Humor Understanding." pith.science (2026). https://pith.science/paper/CYBM7XMV

@misc{pith2026241217729,
  author       = {Pith},
  title        = {Pith review of: Chumor 2.0: Towards Benchmarking Chinese Humor Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYBM7XMV}},
  note         = {Machine review of arXiv:2412.17729}
}
read the original abstract

Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes. We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo. We release Chumor at https://huggingface.co/datasets/dnaihao/Chumor, our project page is at https://dnaihao.github.io/Chumor-dataset/, our leaderboard is at https://huggingface.co/spaces/dnaihao/Chumor, and our codebase is at https://github.com/dnaihao/Chumor-dataset.

Figures

Figures reproduced from arXiv: 2412.17729 by the authors.

Figure 1
Figure 1. The accuracy of different models’ test re [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. DP accuracy on different joke types (%). We [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Over-analyzing example by GPT-4o. The GPT-4o model chooses the correct answer in the DP prompting, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Annotated preference for whether human ex [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Distribution of error types for GPT-4o and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: Homophonic error example. video game, but guanine, thymine, and adenine, which are nucleotides involved in the structure of DNA. Such jokes require LLMs to identify puns and the reason for the association of the multiple meanings. Furthermore, the process requires LLMs…
Figure 7
Figure 7. Figure 7: Pun-based error example. when they have acquired such cultural knowledge, they may fail to relate to it as we humans do during the reasoning process. Error Type II: Pun-based Error. LLMs may fail to identify multiple meanings of a single word within a joke, causing the…
Figure 9
Figure 9. Figure 9: Insufficient contextual understanding exam [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Glyph-based error example. Glyph-based Error. LLMs may fail to interpret the visual puns based on the shape or form of Chi￾nese characters. Glyph-based humor in Chinese leverages its logographic writing system, where characters integrate both semantic and visual ele￾m…
Figure 12
Figure 12. Figure 12: Hallucinations example [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Cross-lingual error example. Cross-lingual Error. LLMs may fail to recog￾nize elements or information across different lan￾guages. In the explanation in [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Overcritical example by Nemotron70B. The Nemotron70B model selects the correct answer in the DP prompting, but selects the incorrect answer due to being overly critical in the CoT prompting. to provide an explanation. During our evalua￾tion, we observe this excessive …
Figure 16
Figure 16. Figure 16: CoT accuracy on different joke types(%). [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

    cs.CL 2026-03 unverdicted novelty 7.0 of 10

    HumorRank ranks nine LLMs on textual humor using GTVH-grounded pairwise tournaments and Adaptive Swiss aggregation on the SemEval-2026 MWAHAHA dataset, finding that comedic mechanism mastery matters more than scale.

Reference graph

Works this paper leans on

58 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    01.ai. 2024. Yi-34b model card. https://huggingface.co/01-ai/Yi-34B. Accessed: 2024-12-10

    work page 2024

  5. [5]

    Mistral AI. 2024. Mistral-large-instruct-2407 model card. https://huggingface.co/mistralai/Mistral-Large-Instruct-2407. Accessed: 2024-12-10

    work page 2024

  6. [6]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. https://doi.org/10.18653/v1/2024.acl-long.671 Investigating cultural alignment of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404--12422, Bangkok, Thailand. Association for Compu...

    doi:10.18653/v1/2024.acl-long.671 2024

  7. [7]

    Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. 2024. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. arXiv preprint arXiv:2403.18058

    arXiv 2024

  8. [8]

    Baidu. 2024. Ernie-4.0-turbo. https://cloud.baidu.com/doc/WENXINWORKSHOP/s/7lxwwtafj. Accessed: 2024-12-10

    work page 2024

Show all 58 references
  1. [9]

    Dario Bertero and Pascale Fung. 2016. https://aclanthology.org/L16-1079 Deep learning of audio and language features for humor prediction . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , pages 496--501, Portoro z , Slove...

    2016

  2. [10]

    BigModel. 2024. Glm-4 model documentation. https://bigmodel.cn/dev/howuse/glm-4. Accessed: 2024-12-10

    2024

  3. [11]

    Lei Chen and Chong Min Lee. 2017. https://doi.org/10.18653/v1/W17-5009 Predicting audience ' s laughter during presentations using convolutional neural network . In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications , pages 86--90, ...

    2017 doi

  4. [12]

    Yuyan Chen, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Bang Liu, and Yunwen Chen. 2023. https://doi.org/10.1145/3539597.3570431 Can pre-trained language models understand chinese humor? In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSD...

    2023

  5. [13]

    Yuyan Chen, Yichen Yuan, Panjun Liu, Dayiheng Liu, Qinghao Guan, Mengfei Guo, Haiming Peng, Bang Liu, Zhixu Li, and Yanghua Xiao. 2024. https://doi.org/10.1609/aaai.v38i16.29736 Talk funny! a large-scale humor response dataset with chain-of-humor interpretation . Proceedings o...

    2024 doi

  6. [14]

    Daniels and William Bright

    Peter T. Daniels and William Bright. 1996. The world’s writing systems. Oxford University Press

    1996

  7. [15]

    Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. 2024. Tables as images? exploring the strengths and limitations of llms on multimodal representations of tabular data. arXiv preprint arXiv:2402.12424

    2024 arXiv

  8. [16]

    Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.832 You are what you annotate: Towards better models through annotator representations . In Findings of the Association for Computational Ling...

    2023 doi

  9. [17]

    Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, et al. 2024. Chinese tiny llm: Pretraining a chinese-centric large language model. arXiv preprint arXiv:2404.04167

    2024 arXiv

  10. [18]

    Tomas Engelthaler and Thomas T Hills. 2018. Humor norms for 4,997 english words. Behavior research methods, 50:1116--1124

    2018

  11. [19]

    Marc Gelkopf et al. 2011. The use of humor in serious mental illness: A review. Evidence-Based Complementary and Alternative Medicine, 2011

    2011

  12. [20]

    Hamideh Ghanadian, Isar Nejadgholi, and Hussein Al Osman. 2023. https://doi.org/10.18653/v1/2023.wassa-1.16 C hat GPT for suicide risk assessment on social media: Quantitative evaluation of model performance, potentials and limitations . In Proceedings of the 13th Workshop on ...

    2023 doi

  13. [21]

    Google. 2024. Gemini 1.5 pro model documentation. https://ai.google.dev/gemini-api/docs/models/gemini#gemini-1.5-pro. Accessed: 2024-12-10

    2024

  14. [22]

    Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. https://doi.org/10.18653/v1/D19-1211 UR - FUNNY : A multimodal language dataset for understanding humor . In Proceedings of the...

    2019 doi

  15. [23]

    Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi

    Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.acl-long.41 Do androids laugh at electric sheep? humor `` understanding '' benchmarks from the new yorker caption contest . In...

    2023 doi

  16. [24]

    Nabil Hossain, John Krumm, and Michael Gamon. 2019. https://doi.org/10.18653/v1/N19-1012 `` president vows to cut taxes hair '' : Dataset and analysis of creative text editing for humorous headlines . In Proceedings of the 2019 Conference of the North A merican Chapter of the ...

    2019 doi

  17. [25]

    Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, and Rada Mihalcea

    Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang,...

    2024

  18. [26]

    Herbert M Lefcourt. 2001. Humor: The psychology of living buoyantly. Springer Science & Business Media

    2001

  19. [27]

    Zefeng Li, Hongfei Lin, Liang Yang, Bo Xu, and Shaowu Zhang. 2022. Memeplate: A chinese multimodal dataset for humor understanding in meme templates. In Natural Language Processing and Chinese Computing, pages 527--538, Cham. Springer International Publishing

    2022

  20. [28]

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023 a . Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439

    2023 arXiv

  21. [29]

    Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihalcea. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.944 Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond . In Proceedings of the 2023 Confer...

    2023 doi

  22. [30]

    Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. https://doi.org/10.18653/v1/2021.acl-long.468 Bridging subword gaps in pretrain-finetune paradigm for natural language generation . In Proceedings of the 59th Annual M...

    2021 doi

  23. [31]

    Rada Mihalcea, Oana Ignat, Longju Bai, Angana Borah, Luis Chiruzzo, Zhijing Jin, Claude Kwizera, Joan Nwatu, Soujanya Poria, and Thamar Solorio. 2024. Why ai is weird and should not be this way: Towards ai for everyone, with everyone, by everyone. arXiv preprint arXiv:2410.16315

    2024 arXiv

  24. [32]

    Rada Mihalcea and Carlo Strapparava. 2005. https://aclanthology.org/H05-1067 Making computers laugh: Investigations in automatic humor recognition . In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages...

    2005

  25. [33]

    Nexusflow. 2024. Athene-70b model card. https://huggingface.co/Nexusflow/Athene-70B. Accessed: 2024-12-10

    2024

  26. [34]

    NVIDIA. 2024. Llama-3.1-nemotron-70b-instruct-hf model card. https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF. Accessed: 2024-12-10

    2024

  27. [35]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  28. [36]

    OpenAI. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . arXiv preprint arXiv:2410.21276

    2024 arXiv

  29. [37]

    Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2024. Survey of cultural awareness in language models: Text and beyond. arXiv preprint arXiv:2411.00860

    2024 arXiv

  30. [38]

    Peter Potash, Alexey Romanov, and Anna Rumshisky. 2017. https://doi.org/10.18653/v1/S17-2004 S em E val-2017 task 6: \# H ashtag W ars: Learning a sense of humor . In Proceedings of the 11th International Workshop on Semantic Evaluation ( S em E val-2017) , pages 49--57, Vanco...

    2017 doi

  31. [39]

    Qwen. 2024. Qwen2.5-72b-instruct model card. https://huggingface.co/Qwen/Qwen2.5-72B-Instruct. Accessed: 2024-12-10

    2024

  32. [40]

    Dragomir Radev, Amanda Stent, Joel Tetreault, Aasish Pappu, Aikaterini Iliakopoulou, Agustin Chanfreau, Paloma de Juan, Jordi Vallmitjana, Alejandro Jaimes, Rahul Jha, and Robert Mankoff. 2016. https://aclanthology.org/L16-1076 Humor in collective discourse: Unsupervised funni...

    2016

  33. [41]

    o rn Gamb \

    Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Bj \"o rn Gamb \"a ck. 2020. https://doi.org/10.18653/v1/2020.semeval-1.99 S em E val-2020 task 8: Memotion analysis- the visuo-lingual metaphor! In Proce...

    2020 doi

  34. [42]

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. https://doi.org/10.18653/v1/2024.naacl-long.316 Understanding the capabilities and limitations of large language models for cultural commonsense . In Proceedings of the 2024 Con...

    2024 doi

  35. [43]

    Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T....

    2024 arXiv

  36. [44]

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2024. http://arxiv.org/abs/2409.12183 To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning

    2024 arXiv

  37. [45]

    Honglin Sun and Daniel Jurafsky. 2004. https://aclanthology.org/N04-1032 Shallow semantic parsing of C hinese . In Proceedings of the Human Language Technology Conference of the North A merican Chapter of the Association for Computational Linguistics: HLT - NAACL 2004 , pages ...

    2004

  38. [46]

    Weiwei Sun, Zhifang Sui, Meng Wang, and Xin Wang. 2009. https://aclanthology.org/D09-1153 C hinese semantic role labeling with shallow parsing . In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1475--1483, Singapore. Association ...

    2009

  39. [47]

    Yuen-Hsien Tseng, Wun-Syuan Wu, Chia-Yueh Chang, Hsueh-Chih Chen, and Wei-Lun Hsu. 2020. https://aclanthology.org/2020.lrec-1.168 Development and validation of a corpus for machine humor comprehension . In Proceedings of the Twelfth Language Resources and Evaluation Conference...

    2020

  40. [48]

    Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, et al. 2024. All languages matter: Evaluating lmms on culturally diverse 100 languages. arXiv preprint ...

    2024 arXiv

  41. [49]

    Benyou Wang, Xiang Wu, Xiaokang Liu, Jianquan Li, Prayag Tiwari, and Qianqian Xie. 2022. https://api.semanticscholar.org/CorpusID:250264242 Can language models make fun? a case study in chinese comical crosstalk . In Annual Meeting of the Association for Computational Linguistics

    2022

  42. [50]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  43. [51]

    Orion Weller and Kevin Seppi. 2020. https://aclanthology.org/2020.lrec-1.753 The r J okes dataset: a large scale humor collection . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6136--6141, Marseille, France. European Language Resources Association

    2020

  44. [52]

    Jiaming Wu, Hongfei Lin, Liang Yang, and Bo Xu. 2021. https://doi.org/10.1007/978-3-030-88480-2_49 Mumor: A multimodal dataset for humor detection in conversations . In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, C...

    2021 doi

  45. [53]

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.717 Hi- T o M : A benchmark for evaluating higher-order theory of mind reasoning in large language models . In Findings of the Association for Co...

    2023 doi

  46. [54]

    Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. 2015. https://doi.org/10.18653/v1/D15-1284 Humor recognition and humor anchor extraction . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2367--2376, Lisbon, Portugal. Associa...

    2015 doi

  47. [55]

    Dongyu Zhang, Heting Zhang, Xikai Liu, Hongfei Lin, and Feng Xia. 2019. https://api.semanticscholar.org/CorpusID:202767593 Telling the whole story: A manually annotated chinese dataset for the analysis of humor in jokes . In Conference on Empirical Methods in Natural Language ...

    2019

  48. [56]

    Min Zhang, Jianfeng He, Taoran Ji, and Chang-Tien Lu. 2024 a . http://arxiv.org/abs/2402.11406 Don't go to extremes: Revealing the excessive sensitivity and calibration limitations of llms in implicit hate speech detection

    2024 arXiv

  49. [57]

    Hashimoto

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024 b . https://doi.org/10.1162/tacl_a_00632 Benchmarking Large Language Models for News Summarization . Transactions of the Association for Computational Linguistics, 12:39--57

    2024 doi

  50. [58]

    Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.