{"id":"32f4273f-639f-4985-bdb2-ead500b2127d","arxiv_id":"2412.17729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Chumor is a new Chinese humor explanation benchmark where LLMs perform near chance (best accuracy 60.3%) and well below human accuracy (78.3%).","lead":"This paper builds Chumor, a Chinese humor dataset of 3,339 jokes with explanations, and tests ten large language models on judging whether each explanation fully explains the joke. The models score near random (best 60.3%) while humans reach 78.3%, suggesting culturally specific Chinese humor remains hard for LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison rests on unvalidated ground-truth labels; the reported lack of inter-annotator agreement makes the LLM-vs-human gap uninterpretable without additional reliability evidence.","rationale":"The paper's central claim is plausible: the multi-model evaluation is broad, the independent human study is genuine evidence, and the reported MCC values align with the accuracy gap. I read the paper in good faith and considered alternative concerns. The dataset-size overclaim and the internal numerical inconsistency flagged by the reader, while real, do not threaten the substantive result that LLMs underperform humans on this task. The random-baseline issue (majority class at 56.5% accuracy) would, if anything, strengthen the 'near chance' characterization, so it is not load-bearing. The decisive quantity—the 18-point gap between LLMs and humans—depends entirely on the ground-truth labels. Those labels are a majority vote of the five authors, with no inter-annotator agreement reported. The separate human study provides some support, but agreement with a gold set is not the same as reliability: it does not quantify the variance among the original labelers, nor does it rule out that the gold labels are idiosyncratic or noisy. Because the benchmark's validity hinges on this, and because the paper itself does not supply the needed reliability evidence, the concern is load-bearing. The reader's CONDITIONAL verdict is appropriate: the claim stands if, and only if, the labels prove reliable under direct measurement. Hence no verdict change; the requested check would settle the issue.","tokens_in":20281,"tokens_out":7990,"duration_ms":77636,"concrete_test":"Obtain the five authors' individual votes (if stored) for a random 200-instance subset and compute Fleiss' kappa among them. Independently, recruit 5 fresh native Chinese speakers (not authors, not the previous 3) to label the same subset with the same binary question; compute kappa among the new annotators and their agreement with the gold labels. If either kappa < 0.4 or new-human agreement with gold < 70%, the labels are too noisy to support the claimed LLM-vs-human gap; if kappa > 0.6 and agreement > 75%, the concern is largely resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—LLMs at 60.3% accuracy / 0.29 MCC vs. humans at 78.3% / 0.60 on Chumor—assumes the dataset labels are reliable. Labels in Section 3 ('Humor Explanation Classification') are produced by majority vote among the five authors on whether an LLM-generated explanation 'fully explains' the joke. No inter-annotator agreement (e.g., Fleiss' kappa) is reported for these votes. The only validation is the Section 5.2 human study, where three independent native speakers label 200 examples and match the gold labels 78.3% of the time on average. But machine-human agreement with a gold label set is not a substitute for IAA: it can be inflated by easy items and by shared cultural priors, and it does not reveal whether the five authors' votes were consistent. If the gold labels contain substantial noise—e.g., if items near the decision boundary are effectively arbitrary—then both the LLM and human scores are measured against a moving target, and the 18-point accuracy gap is not a clean measure of humor understanding. The task also conflates 'fully explains the joke' with humor comprehension; no independent evidence links explanation adequacy to humor understanding. Therefore, the reliability of the ground-truth labels is the load-bearing assumption of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Chumor, a Chinese humor explanation dataset of 3,339 (joke, explanation, label) instances sourced from the Ruo Zhi Ba forum. Five native-speaker authors label each LLM-generated explanation (from GPT-4o or ERNIE4-turbo) as fully explaining the joke or not. The authors evaluate ten LLMs under direct and chain-of-thought prompting, reporting that the best model reaches 60.3% accuracy and MCC 0.29, well below a human study estimate of 78.3% accuracy and MCC 0.60. They also report an A/B preference study in which six annotators prefer human-written explanations over LLM-generated ones, and an error analysis of failure types. The dataset, code, and leaderboard are released.","tokens_in":20512,"tokens_out":7042,"duration_ms":59464,"significance":"If the labels are reliable, Chumor is a valuable resource for studying culturally specific Chinese humor understanding and provides a challenging benchmark for LLMs. The paper's main empirical result—that all ten tested LLMs perform near chance while humans reach 78.3% accuracy—would be an important finding for the field. The release of the dataset, prompts, and code is a concrete contribution. The error taxonomy (cultural unawareness, pun-based, homophonic, glyph-based, parsing, cross-lingual) is a useful qualitative analysis. However, the central comparison depends on ground-truth label reliability, which is not yet sufficiently established.","major_comments":[{"comment":"The gold labels in Chumor are produced by a majority vote of five authors, but no inter-annotator agreement (e.g., Fleiss' kappa) is reported for these votes. The human study in Section 5.2 reports an average accuracy of 78.3% for three independent annotators against the gold labels, but this is agreement with the gold set and does not measure agreement among the annotators themselves; no per-annotator scores or IAA are given. Because the central claim of an 18-point gap between LLMs (60.3%) and humans (78.3%) is measured against these gold labels, the paper should provide reliability evidence for the labels, at least on a sample, and for the human study annotators.","section":"Section 3, 'Humor Explanation Classification'; Section 5.2"},{"comment":"The abstract and introduction claim that Chumor 'exceeds the size of existing humor datasets,' but Table 2 lists several larger datasets (e.g., rJokes: 57k, #HashtagWars: 13k, TCHD: 26k). This claim is contradicted by the paper's own table and should be corrected to, for example, 'the first Chinese humor explanation dataset.'","section":"Abstract and Section 1"},{"comment":"The error analysis by joke type is based on only 200 jokes, but the paper does not report the number of jokes per type. For types such as Glyph-based and Cross-lingual, reported accuracies of 0% or 100% for several models suggest very small per-type samples. Without per-type sample sizes and confidence intervals, the claim that 'model performance varies significantly across different joke types' is not statistically supported.","section":"Section 5.1 and Figures 2/16"},{"comment":"The A/B preference study uses human explanations written by one of the authors as the gold standard, but the quality of these explanations is not independently validated before comparison with LLM explanations. The six preference annotators reach only 61.4% agreement, which is reported as 'decent' without a chance-corrected measure such as Cohen's kappa. Please report kappa and consider having the author-written explanations evaluated by independent annotators before drawing conclusions about human-vs-LLM explanation quality.","section":"Section 5.4 and Appendix B"},{"comment":"The 'random' baseline in Figure 1 is 50%, but the dataset is imbalanced: 1,454 good vs 1,887 bad explanations (56.5% bad). A trivial classifier that always predicts 'bad' achieves 56.5% accuracy, so the best LLM accuracy of 60.3% is only 3.8 points above the majority-class baseline. The paper should report the majority-class baseline and statistical significance tests to support the claim that LLMs are 'slightly above random.'","section":"Section 5, Figure 1"}],"minor_comments":[{"comment":"The phrase 'three folds' should be 'threefold' or 'three-fold.'","section":"Section 1"},{"comment":"The word 'generae' should be 'generate.'","section":"Section 3"},{"comment":"The phrase 'better than those produced by GPT-4o and ERNIE4-turborurging., encouraging future research' is garbled and should be reworded.","section":"Section 1, Contribution 3"},{"comment":"The caption contains typos: 'wining rate' should be 'winning rate' and 'preffered' should be 'preferred.'","section":"Figure 4 caption"},{"comment":"The annotation instructions given to the three human study participants are not reported; please provide them (e.g., whether they used the same DP prompt) for reproducibility.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely gap and the resource is potentially useful. The main issue is the lack of inter-annotator agreement for the gold labels, which is common in humor research but must be addressed given the benchmark's purpose. The incorrect size claim in the abstract should be corrected. I believe the paper can be revised to address these concerns within its scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chumor is worth knowing about: it is the first Chinese humor explanation dataset with a binary adequacy task, sourced from Ruo Zhi Ba, and released openly with code, leaderboard, and a human study. The evaluation of ten LLMs is broad, and the finding that all models hover near random while humans score much higher is consistent across accuracy and MCC. The CoT result—often hurting, sometimes severely (ERNIE4-turbo drops from 0.29 to 0.11 MCC)—is a useful counterpoint to the usual \"CoT helps\" narrative.\n\nThe soft spots are real but fixable. The abstract overclaims: \"exceeds the size of existing humor datasets\" is false, since rJokes has 57k and Big Bang 44k; Chumor has 3.3k instances. The defensible claim is \"first Chinese humor explanation dataset.\" More importantly, the main labels come from a majority vote of five authors, with no inter-annotator agreement reported. The 78.3% agreement of three independent annotators on 200 examples is encouraging but not a substitute for IAA; if the authors disagreed on borderline cases, the gold labels contain noise and the LLM-human gap is less clean than presented. This is easily fixed by reporting per-vote counts or Fleiss' kappa on a sample.\n\nThe type-level analysis (Section 5.1) uses only 200 jokes, and some categories have tiny counts—the 0%, 25%, 50% figures suggest cells with 1–4 jokes. The paper doesn't report per-category counts, making those comparisons underpowered. Also, the task itself conflates \"fully explains\" with humor understanding; the A/B preference study helps, but the human explanations come from a single author, so that baseline is not strong.\n\nThese issues are in reporting and validation, not the core idea. The dataset fills a genuine gap, and the central result—LLMs underperform humans on culturally specific Chinese humor—is likely robust even with some label noise. I'd send this to a serious referee; it deserves revision, not rejection. With the abstract fixed, IAA or per-vote data, and per-category counts, this becomes a solid resource paper.","headline":"A genuinely new Chinese humor explanation dataset with a solid multi-model evaluation, but the label reliability and a few overclaims need fixing before the headline results can be fully trusted.","tokens_in":21057,"tokens_out":4261,"would_cite":true,"duration_ms":38975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chumor 2.0 claims current LLMs cannot reliably judge whether an explanation captures a Chinese joke: the best model scores 60.3% versus 78.3% for humans.","keywords":["Chinese humor","humor understanding","LLM evaluation","humor explanation","culturally specific humor","chain-of-thought prompting","benchmark dataset","Ruo Zhi Ba"],"falsifier":"Take a random sample of Chumor instances, recruit independent native Chinese speakers who have no stake in the project, and have them re-apply the same good/bad label. If their agreement with the published labels is near chance (e.g., Cohen's kappa below 0.2), then the 60.3%-versus-78.3% gap would be an artifact of one annotation team's subjective judgments rather than a stable measure of humor understanding.","tokens_in":20094,"feed_emoji":"😂","tokens_out":5356,"duration_ms":46520,"temperature":0.7,"pith_summary":"Chumor 2.0 constructs the largest Chinese humor explanation dataset to date, 3,339 joke–explanation pairs sourced from the Chinese forum Ruo Zhi Ba, and uses it to ask whether LLMs can decide if an explanation fully accounts for a joke's humor. The paper's central claim is that current LLMs largely fail at this culturally grounded reasoning task: the best tested models reach 60.3% accuracy (Matthew correlation 0.29), barely above the random baseline, while native-speaker annotators reach 78.3% accuracy (MCC 0.60). It further claims that chain-of-thought prompting does not reliably help and often hurts, and that human-written joke explanations are strongly preferred over those produced by GPT-4o and ERNIE4-turbo in pairwise A/B comparisons. A sympathetic reader would take away that Chinese humor comprehension, not just humor detection, remains an open problem for LLMs.","feed_headline":"Chinese jokes stump AI: best model hits 60.3%, humans 78.3%","feed_subtitle":"New 3,339-item benchmark: LLMs judge joke explanations near random while native speakers reach 78.3% accuracy.","key_machinery":"The organizing instrument is the humor explanation classification task: each instance pairs a joke with an explanation written by GPT-4o or ERNIE4-turbo, and the label is a majority vote by five native Chinese speakers over 'fully explains' versus 'partially/does not explain.' Jokes are grouped into six named mechanisms—cultural, situational, pun-based, homophonic, glyph-based, and cross-lingual—so that errors can be traced to the kind of knowledge the model lacks. The prompting variants (direct vs. chain-of-thought) and the pairwise human-vs-LLM explanation preference test are the other load-bearing pieces.","core_discovery":"The discovery is a negative result stated as a benchmark: when asked to classify whether a given explanation 'fully explains' a joke, ten LLMs perform poorly on Chumor, with accuracy 44.6–60.3% and MCC up to 0.29, versus human accuracy 78.3% and MCC 0.60. The paper treats this as evidence that culturally specific Chinese humor understanding is not solved by current LLMs. Error analysis attributes failures to missing cultural knowledge, missed puns, missed homophones, missed glyph-based visual wordplay, parsing ambiguity, hallucinations, and cross-lingual phonetic connections; the authors also find that CoT prompting often makes models over-justify incorrect explanations, raising false-positive rates.","pith_inferences":["Explanation adequacy is a proxy for humor understanding; a model might pass by matching explanation style rather than by 'getting' the joke, so a stricter test would have models produce an explanation from scratch and have humans judge it.","The label distribution is imbalanced (1,887 bad vs. 1,454 good) and the human A/B preference agreement is only 61.4%, so part of the reported gap may reflect subjectivity in what counts as a full explanation rather than pure comprehension failure.","A natural extension is to use human-written explanations as the candidate texts and see whether LLMs still underperform; this would separate label noise from genuine reasoning limits.","The cultural jokes require encyclopedic Chinese knowledge; a model trained on more Chinese-centric data might close part of the gap, as the paper hints with ERNIE4-turbo's lower cultural-error rate."],"forward_implications":["Any claim of human-level humor understanding in LLMs should be tested on explanation adequacy, not only on detecting or generating jokes.","Chain-of-thought prompting should not be assumed to improve humor reasoning; in this benchmark it often lowers accuracy and inflates false positives by making models rationalize weak explanations.","The six joke-type breakdown gives a diagnostic target: models fail differently on cultural, homophonic, and glyph-based humor, so progress can be tracked per mechanism.","Human explanation quality is measurably higher than LLM explanation quality, setting a concrete bar for generation systems."],"supporting_citations":[{"why":"Supplies the precedent that crowd-sourcing produces low-quality humor explanations, motivating the single-author annotation of the 1,951 explanation corpus.","marker":"Hessel et al., 2023"},{"why":"Provides the chain-of-thought prompting method that the paper tests and finds can degrade humor reasoning performance.","marker":"Wei et al., 2022"},{"why":"Reports that tuning LLMs on Ruo Zhi Ba data yields strong Chinese reasoning performance, which motivates Chumor's source platform.","marker":"Bai et al., 2024"},{"why":"Supports the paper's interpretation that chain-of-thought can hurt on tasks requiring subtle comprehension, aligning with the observed CoT declines.","marker":"Sprague et al., 2024"}],"fun_headline_variants":["AI flunks Chinese humor, best model hits 60.3% vs 78.3% human","Chain-of-thought backfires on Chinese jokes, new benchmark shows","Chumor benchmark: LLMs edge past random, humans score 78.3%","AI struggles with Chinese puns and homophones, dataset reveals","Best AI scores 60.3% on Chinese humor, humans 78.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the majority-vote labels of five native-speaker authors—deciding whether an LLM-generated explanation 'fully explains' a joke—are reliable ground truth for humor understanding; the paper does not report inter-annotator agreement for these labels or validate that explanation adequacy tracks comprehension, so if those labels are noisy the reported human–LLM gap is not a clean measure.","fun_headline_variants_meta":{"raw":{"variants":["AI flunks Chinese humor, best model hits 60.3% vs 78.3% human","Chain-of-thought backfires on Chinese jokes, new benchmark shows","Chumor benchmark: LLMs edge past random, humans score 78.3%","AI struggles with Chinese puns and homophones, dataset reveals","Best AI scores 60.3% on Chinese humor, humans 78.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1936,"prompt_tokens":898,"completion_tokens":1038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":931}},"tokens_in":514,"tokens_out":1038,"duration_ms":8173,"temperature":1.0,"reasoning_tokens":931,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:13:37.735755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Chumor instances, recruit independent native Chinese speakers who have no stake in the project, and have them re-apply the same good/bad label. If their agreement with the published labels is near chance (e.g., Cohen's kappa below 0.2), then the 60.3%-versus-78.3% gap would be an artifact of one annotation team's subjective judgments rather than a stable measure of humor understanding.","supporting_citations":[],"review_version":1}