{"id":"9b33f892-1385-44d3-a744-810ecfad54f6","arxiv_id":"2607.19011","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic survey and cross-benchmark evaluation showing that multimodal LLMs can recognize humor artifacts but still struggle to interpret the intended meaning and mechanisms of visual humor.","lead":"This survey maps the field of AI understanding of visual humor in memes, cartoons, and comics, organizing it into recognition, reasoning, and generation. It evaluates eleven recent multimodal AI models across seven humor benchmarks and finds they still lag far behind humans at explaining why something is funny.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-benchmark support for the interpretation bottleneck is confounded: the §6.1 capability ranking compares uncontrolled tasks, and Appendix A's single-sample decoding makes the headline gaps point estimates rather than established differences.","rationale":"The reader's weakest assumption is the same one I consider most load-bearing: the cross-benchmark comparison treats benchmark labels and difficulty as if they were controlled, when they are not. My reading strengthens this concern in two ways. First, the paper itself disclaims statistical significance and controlled comparison, so the exact gap figures in §6.1 should not carry the weight of a capability ranking. Second, the recognition category includes MangaUB perceptual subtasks that do not test humor understanding, so the apparent recognition strength is partly an artifact of task selection. I do not find a reason to reject the paper. The survey's qualitative synthesis and the broader literature consistently show that interpreting intended meaning in visual humor is harder for current models than recognizing depicted content. The conditional verdict remains appropriate: the paper should either release a reproducible, multi-seed evaluation with confidence intervals or reframe the empirical claims as descriptive, while retaining the qualitative bottleneck conclusion. No change to the existing verdict is needed.","tokens_in":24455,"tokens_out":6741,"duration_ms":69283,"concrete_test":"On a fixed set of YesBut-v2 items, administer two tasks to the same models: (a) a recognition-level subquestion about whether or where a contradiction occurs, and (b) the original Moral/Title multiple-choice question. If accuracy on (a) is high while (b) remains low on the same items, the interpretation bottleneck is supported within a matched artifact and prompt family. If both are low, the observed gap is better explained by item difficulty or format than by capability level.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §6.1 is that current MLLMs have made considerable progress on visual and categorical recognition but still fall short in recovering intended meaning, communicative function, and incongruity structure. The quantitative support for this claim is Table 3, where recognition-level tasks generally score higher than interpretation-level tasks, with human gaps of 6.58, 14.22, 11.70, and 21.49 points on YesBut-v2 Moral, YesBut-v2 Title, NYCC, and MemeQA. This evidence is not controlled. Section 6 states that prompts differ across benchmarks by design and that the results support 'descriptive cross-benchmark analysis, rather than a strictly controlled comparison of task difficulty.' Appendix A adds that each question is evaluated with a single sampled generation using do_sample=true and that 'small score differences may be affected by decoding stochasticity and should not be interpreted as statistically significant.' The recognition group also contains MangaUB perceptual subtasks (RecBg, CharCnt, PanelLoc) that do not require humor understanding at all, so high scores there may reflect general visual perception rather than humor recognition. DarkHumor, a humor-specific recognition task, reaches only 66.43%, below the best YesBut-v2 Moral score of 84.72%. Thus the recognition-versus-interpretation ordering in §6.1 is not robust to re-grouping or to benchmark difficulty. The qualitative claim is plausible and independently supported by the surveyed literature, but the empirical numbers as presented do not establish it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a survey of multimodal visual humor understanding for large multimodal models (MLLMs). It organizes the literature with a capability hierarchy spanning recognition, interpretation and reasoning, and generation; a two-way distinction between static visual-textual artifacts (StaVT) and sequential visual narratives (SeqVN); a synthesis of modeling paradigms (cross-modal alignment, grounded reasoning, controllable generation); and a catalogue of benchmarks and evaluation protocols. Beyond the qualitative synthesis, Section 6 contributes a cross-benchmark empirical evaluation of eleven MLLMs on twelve task settings (Table 3), and Section 7 identifies the main barriers to progress: shortcut-prone evaluation, limited cultural and narrative coverage, weak evidence grounding, and unresolved safety and ownership concerns. The central claim, stated in Section 6.1, is that current MLLMs have made considerable progress on visual and categorical recognition but still fall short in recovering the intended meaning, communicative function, and incongruity structure that make visual humor understandable to humans.","tokens_in":24731,"tokens_out":12498,"duration_ms":101803,"significance":"If the central claim holds, the survey provides a useful organizing framework and an up-to-date snapshot of the field; the capability hierarchy and the distinction between descriptive explanation and mechanism-grounded reasoning are genuinely clarifying, and the critique of MCQ and binary evaluation formats is well supported by the reviewed literature. The empirical section is a value-added contribution over a conventional survey: it evaluates a broad and recent model set under a disclosed protocol, and the paper deserves explicit credit for publishing its caveats (prompts differ by design, single-sample decoding, descriptive rather than controlled comparison, stated in Section 6 and Appendix A) and for giving reproducibility details such as the fixed GPT-4o API version and completion date. The bottleneck conclusion is not merely a self-citation artifact: it is independently supported by non-author benchmarks (NYCC, MemeQA) and by the surveyed literature. The main weakness is that the empirical support for the recognition-versus-interpretation contrast is weaker than the Section 6.1 prose suggests, for reasons detailed in the major comments.","major_comments":[{"comment":"The recognition-versus-interpretation contrast is confounded by the composition of the recognition group. Three of the five MangaUB subtasks (RecBg, CharCnt, PanelLoc) are general visual-perception tasks with no humor component, so GPT-4o's 97.60-99.21% scores on them demonstrate perception rather than humor recognition, and including them inflates the recognition column. When the comparison is restricted to humor-specific tasks, the ordering is no longer clean: DarkHumor, the only humor-specific recognition task, tops out at 66.43%, below the best YesBut-v2 Moral score of 84.72%, and HumorDB, also a recognition task, shows a 10.29-point human gap that is larger than the 6.58-point gap on YesBut-v2 Moral. The conclusion that models show considerable progress on visual and categorical recognition should be re-derived on a humor-specific subset, or reframed as a claim about perception rather than about humor recognition.","section":"§6.1, Table 3"},{"comment":"The headline gap figures (6.58, 14.22, 11.70, and 21.49 points) are point estimates from one sampled generation per question with do_sample=true, with no variance reporting or significance testing. The disclaimer in Appendix A is correct, but Section 6.1 does not carry its uncertainty into the prose: statements such as 'Interpretation and reasoning remain the central bottleneck' and 'recognition remains challenging when it depends on implicit social norms' are categorical. This matters because several adjacent results in Table 3 differ by amounts larger than the smallest headline gap (the Qwen3-VL-8B versus Qwen3-VL-8B-Thinking comparison alone swings by 4 to 16 points across tasks), so the reader cannot tell which of the reported differences are stable. The paper should report multiple samples or bootstrap confidence intervals for the key within-benchmark contrasts, or explicitly downgrade the Section 6.1 conclusions to descriptive observations.","section":"§6, Appendix A"}],"minor_comments":[{"comment":"As rendered, the Human row is ambiguous: the values 91.30, 97.50, 94.00, and 81.90 are used in Section 6.1 for YesBut-v2 Moral, YesBut-v2 Title, NYCC, and MemeQA, but in the table layout they appear under the MangaUB subtask columns, and the MangaUB human scores are then missing or double-counted; please align the row or add a per-benchmark footnote. Also, the column header 'Y esBut-v2' contains a spurious space.","section":"Table 3"},{"comment":"The opening sentence claims that 'our cross-benchmark analysis (Section 6) shows that sequential, multi-panel humor exposes the widest model-human gap on panel-sequencing and temporal reordering', but Table 3 contains no panel-sequencing or temporal-reordering task (MangaUB NextInf is next-panel inference and is grouped under recognition), so the sentence should cite the surveyed sequential-humor literature instead of the paper's own analysis.","section":"§7.3"},{"comment":"The annotation line 'qCaption q Text in Image q Questions q Object bounding boxes' renders as a garbled notation; if this is intended as a bulleted list, the formatting should be fixed in the figure source.","section":"Figure 2"},{"comment":"The paper does not explicitly flag that YesBut-v2 is one of the authors' own benchmarks in the same paragraph where it is used as evidence for the interpretation bottleneck; a one-sentence disclosure would aid the reader, even though the qualitative conclusion is independently supported by NYCC and MemeQA.","section":"§6"},{"comment":"Some classic works are cited with reprint years rather than original publication years (for example, the 2024 citation for 'Metaphors We Live By'); please verify that the in-text citations match the intended editions.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The survey's novelty claims rest substantially on its own benchmarks: YesBut, YesBut-v2, and several related papers are the authors' own, and these benchmarks feature prominently in the empirical analysis (Section 6) and in support of the central bottleneck claim. I do not regard this as disqualifying circularity because NYCC, MemeQA, and HumorDB independently support the qualitative conclusion, and the paper's caveats are honest. However, the editor may wish to ask the authors to add a disclosure in Section 6 about the self-relationship, and to check that the benchmark inventories in Appendix Tables 4 and 5 do not over-represent their own work relative to the field. The empirical section's framing as an extension of the original evaluation protocols is appropriate, but the Section 6.1 prose should be proportioned to the strength of the evidence regardless."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a solid, well-organized survey of multimodal humor understanding, but the headline empirical claim—that interpretation is the central bottleneck—is more persuasive as a qualitative synthesis than as a conclusion from its own Table 3. The taxonomy (recognition, interpretation & reasoning, generation) is a reasonable re-organization, and the paper does a good job mapping datasets, modeling paradigms, and evaluation protocols to those levels. The dataset inventory is thorough, and the cross-benchmark numbers on newer models (Qwen3.5, InternVL3.5, Gemma-4) are a useful snapshot, even if some of those models are not yet widely available.\n\nThe main soft spot is the empirical analysis. The recognition bucket mixes humor-specific tasks like DarkHumor with purely perceptual MangaUB subtasks (background recognition, character counting, panel localization). Those perceptual subtasks don't require humor understanding at all, so high scores there don't support the 'recognition is relatively solved' narrative. In fact, the best DarkHumor score is 66.43%, below the best YesBut-v2 Moral score of 84.72%. The critique that this comparison is confounded is fair: the recognition-versus-interpretation ordering in §6.1 is not robust to regrouping. Also, Appendix A admits each question is sampled once with do_sample=true, prompts differ across benchmarks by design, and no significance tests are run; the paper itself calls the results descriptive. So the specific rankings in Table 3 should be treated as point estimates, not established differences.\n\nThat said, the qualitative conclusion is plausible and independently supported. NYCC and MemeQA are not author-created benchmarks, and they show the same pattern. The authors' use of their own YesBut-v2 is a minor circularity, but the bottleneck claim doesn't rest on it alone. The paper's limitations section is honest about cultural coverage and generalizability.\n\nThis is a useful reference for anyone entering the area, and it deserves serious peer review. I'd accept it with minor revision: soften the cross-benchmark claims, separate perceptual subtasks from humor recognition, and release the evaluation code and exact prompts if feasible.","headline":"A useful survey of multimodal humor with a principled taxonomy, but the headline empirical claim about the interpretation bottleneck rests on an uncontrolled comparison and should be read as a qualitative synthesis rather than a quantitative finding.","tokens_in":25302,"tokens_out":4547,"would_cite":true,"duration_ms":37659,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal AIs recognize what a meme shows but not why it is funny.","keywords":["multimodal humor understanding","visual humor","multimodal large language models","incongruity","benchmark evaluation","meme understanding","interpretation and reasoning","humor generation"],"falsifier":"Take one set of memes or cartoons and pose recognition-style and interpretation-style questions that are matched for input, answer options, and human-rated difficulty, then run each model multiple times with sampling; if the recognition-to-interpretation gap narrows or disappears under matched formats, the paper's central bottleneck conclusion would be a benchmark artifact rather than a model limitation, and if it persists, the claim stands.","tokens_in":24232,"feed_emoji":"😂","tokens_out":7013,"duration_ms":59647,"temperature":0.7,"pith_summary":"This paper tries to establish that the hard part of computational humor is not seeing but understanding: current multimodal large language models identify the objects, roles, and genres in memes, cartoons, and comics with near-human accuracy, yet systematically fail to recover the intended meaning, communicative function, and incongruity structure that make those artifacts funny. To support this, the survey organizes the field into three capabilities—recognition, interpretation and reasoning, and generation—and reports a cross-benchmark evaluation of recent models on seven humor benchmarks with human scores. The pattern is consistent: best models trail humans by 6.58 points on moral selection, 14.22 on title selection, 11.70 on caption selection, and 21.49 on meme question answering, while recognition tasks are largely solved. If the claim is right, progress depends less on bigger multimodal encoders and more on explicit reasoning, cultural grounding, and evaluation that tests meaning rather than labels.","feed_headline":"Multimodal AIs recognize jokes but miss their meaning","feed_subtitle":"Across seven humor benchmarks, best models trail humans by up to 21 points on interpretation tasks like selecting a meme's moral or caption.","key_machinery":"The organizing device is the capability-centric task hierarchy, which separates recognition (detecting or classifying humorous phenomena), interpretation and reasoning (explaining the mechanism, target, or implicit meaning), and generation (producing humor-consistent outputs conditioned on an artifact). The load-bearing evidence is the cross-benchmark evaluation protocol of Section 6, which reuses each benchmark's original prompts, splits, and scoring rules to compare models across twelve task settings and against human baselines taken from the source papers. This design is what lets the survey attribute score differences to capability level rather than to model family or task name.","core_discovery":"The central claim, stated in Section 6.1, is that current MLLMs have made considerable progress on visual and categorical recognition but still fall short in recovering the intended meaning, communicative function, and incongruity structure that make visual humor understandable to humans. The evidence is a capability gradient across benchmarks: recognition-oriented tasks (role classification, humor detection, manga element recognition) reach high accuracy, with the strongest model scoring 97.60% on background recognition and 99.21% on panel localization, whereas interpretation-oriented tasks (YesBut-v2, NYCC, MemeQA) show large and consistent gaps from human performance. The paper also reports that no single model dominates: rankings differ by task, and a reasoning-enhanced variant improves dark-humor detection but hurts several interpretation tasks, so humor understanding is not one monolithic ability. The conclusion the authors draw is that alignment captures what co-occurs but not why, making interpretation and reasoning the central unsolved bottleneck.","pith_inferences":["If the paper's bottleneck claim holds, then deployed systems using humor understanding—content moderation, social-media analytics, creative assistance—should treat model outputs as candidate interpretations, not decisions, and require human or rubric verification when the stakes are high.","A testable extension would be to use generation as a diagnostic: on the same artifacts, compare a model's explanation faithfulness with its multiple-choice accuracy; models with similar accuracy but different explanation quality would reveal whether the gap is benchmark difficulty or genuine interpretive ability.","The large gaps on MemeQA and YesBut-v2 title selection may partly reflect training-data skew toward Western, English-centric humor; a per-culture benchmark with annotations of required background knowledge could separate missing knowledge from missing reasoning, and would sharpen the paper's cultural-grounding challenge into a measurable quantity."],"forward_implications":["If the interpretation bottleneck is real, then recognition accuracy is a misleading success metric: a model that labels a meme funny may still not know what makes it funny or whom it targets.","Interpretation-focused models will need mechanism-grounded reasoning (setup, conflict, resolution), external knowledge about cultural and social norms, and evidence-grounded explanation rather than more aggressive multimodal alignment.","Benchmark suites should be built from multiple interpretation tasks spanning different formats and cultures, because model rankings fragment across capabilities; an aggregate accuracy score hides which capability is missing.","Sequential, multi-panel humor is where the widest gaps appear, so panel-aware architectures with explicit entity co-reference and setup-punchline decomposition are needed, not single-image processing.","Evaluation should move toward rubric-based generative assessment and self-consistency checks, for example penalizing a model that rates something funny yet explains it blandly, because discriminative accuracy inflates apparent understanding."],"supporting_citations":[{"why":"Supplies the YesBut-v2 benchmark, including the moral and title selection tasks whose human baselines show the largest interpretation gaps.","marker":"Liang et al. (2025)"},{"why":"Supplies the NYCC New Yorker cartoon caption-selection benchmark and the human accuracy baseline used in the cross-benchmark comparison.","marker":"Hessel et al. (2023b)"},{"why":"Supplies the MemeQA multiple-choice fill-in-the-blank meme benchmark with the human score of 81.90%, the widest model-human gap reported.","marker":"Nguyen et al. (2025)"},{"why":"Supplies the HumorDB humor detection benchmark with the human baseline of 85.00%, showing recognition is not uniformly solved.","marker":"Jain et al. (2025)"},{"why":"Supplies the ExHVV visual semantic role labeling task used as a recognition-level benchmark in the evaluation.","marker":"Sharma et al. (2023a)"},{"why":"Supplies the D-Humor dark humor detection benchmark where even the best model reaches 66.43%, showing implicit social norms also strain recognition.","marker":"Kasu et al. (2025)"},{"why":"Supplies the MangaUB manga understanding benchmark whose subtasks illustrate strong recognition performance in the analysis.","marker":"Ikuta et al. (2025)"}],"fun_headline_variants":["AI sees humor, fails to get why it's funny","Machines detect jokes, can't explain them","Humor AI: good at spotting, bad at understanding","LLMs recognize humor, struggle with meaning","AI nails joke recognition, loses on interpretation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the benchmarks assigned to the recognition and interpretation levels are comparable in difficulty, so the score gaps reflect capability differences rather than differences in prompt format, answer format, or single-sample decoding noise.","fun_headline_variants_meta":{"raw":{"variants":["AI sees humor, fails to get why it's funny","Machines detect jokes, can't explain them","Humor AI: good at spotting, bad at understanding","LLMs recognize humor, struggle with meaning","AI nails joke recognition, loses on interpretation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2899,"prompt_tokens":894,"completion_tokens":2005,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1932}},"tokens_in":510,"tokens_out":2005,"duration_ms":12207,"temperature":1.0,"reasoning_tokens":1932,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:32:12.410442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one set of memes or cartoons and pose recognition-style and interpretation-style questions that are matched for input, answer options, and human-rated difficulty, then run each model multiple times with sampling; if the recognition-to-interpretation gap narrows or disappears under matched formats, the paper's central bottleneck conclusion would be a benchmark artifact rather than a model limitation, and if it persists, the claim stands.","supporting_citations":[],"review_version":1}