{"id":"b7761298-8fe0-4199-a18c-add00ebfc301","arxiv_id":"2412.11906","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PunchBench, a 54,000-question benchmark spanning cartoons, posts, comments, and memes, shows that multimodal LLMs lag humans in punchline comprehension, and its Simple-to-Complex Chain-of-Question prompt yields small accuracy gains.","lead":"PunchBench introduces a benchmark that tests whether AI models can understand jokes and sarcasm in image-caption pairs. It shows that top multimodal models still lag far behind humans, and proposes a simple question-ordering prompt that improves their scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The shortcut-free claim rests on unvalidated GPT-3.5 caption transformations: labels for synonymous and antonymous captions are assigned by fiat, and the quality checks do not verify semantic preservation or inversion, so the reported human-model gap may be an artifact of noisy transformed captions.","rationale":"The paper makes a real contribution: a large, multi-source, multi-format benchmark with public data and code, plus human quality checks. The consistent ordering of model results across formats and the reported human evaluations provide some independent support. However, the central innovation — synonymous and antonymous captions that remove shortcuts — is exactly where the benchmark's validity hinges. Section 3.2 describes generation by gpt-3.5-turbo with no verification step, and Section 3.3 assigns labels by rule. The quality check in Section 3.4 and the human baseline in Section 5.1 are too small and are not targeted at semantic preservation or inversion. A drop in accuracy on transformed captions (Section 5.4) is predicted both by successful shortcut removal and by noisy rewrites, so it cannot validate the construction. This is not an objection to the authors' integrity; it is a request for the one measurement that would make the benchmark's central claim trustworthy. The reader's weakest_assumption identifies the same point, so my read does not change the conditional verdict.","tokens_in":29379,"tokens_out":6236,"duration_ms":57465,"concrete_test":"Run a focused human adjudication study: sample 300 synonymous and 300 antonymous caption-image pairs stratified by domain and caption type, plus 100 original pairs as controls. For each item, three independent annotators, blind to the assigned labels, judge whether the synonymous caption retains the same punchline and whether the antonymous caption removes it, with a 'cannot determine' option. Compute agreement and Cohen's kappa with the fiat labels. If agreement is below about 90% or kappa below about 0.8, recompute the main results on only the confirmed-label items; if GPT-4o's Yes/No and Multi-option scores shift materially, or the human-model gap narrows substantially, the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 and 3.3 assign ground-truth labels to GPT-3.5-generated captions by fiat: P_s inherits the original label, P_a receives the opposite. The paper never reports a human evaluation of whether synonymous captions preserve the punchline or antonymous captions eliminate it. The quality check in Section 3.4 samples 100 instructions per format and asks annotators to answer them; this tests whether instructions are answerable, not whether the transformed captions have the intended semantics, and it covers only a small fraction of the transformed caption-image pairs. The indirect evidence in Section 5.4 — model accuracy drops on synonymous and antonymous captions — cannot distinguish successful shortcut removal from noise introduced by awkward or semantically drifted rewrites. Moreover, no caption-only baseline is reported on the transformed subsets, so text-only shortcuts (e.g., sentiment words or internal contradictions that signal humor without the image) may persist. If fiat labels are wrong for a nontrivial fraction of items, the benchmark's ground truth is corrupted for exactly the items designed to make evaluation accurate, and the reported GPT-4o vs. human gap (80.7% vs. 90.7% Yes/No; 53.1% vs. 98.3% Multi-option) could be an artifact of label noise rather than a genuine deficiency in punchline comprehension.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PunchBench, a benchmark for multimodal punchline comprehension in image-caption pairs. The dataset contains 6,000 image-caption pairs collected from prior datasets and social media, annotated by crowd voting and human-written reasoning sentences. The authors generate synonymous and antonymous captions with GPT-3.5 to mitigate text-only shortcuts, construct perception and reasoning tasks with four question formats, and evaluate ten MLLMs, reporting a large performance gap between MLLMs and humans. They also propose SC-CoQ, a simple-to-complex chain-of-question prompting strategy that improves performance over in-context learning and chain-of-thought. The paper claims that PunchBench provides accurate and comprehensive evaluation and that state-of-the-art MLLMs substantially lag human-level punchline comprehension.","tokens_in":29665,"tokens_out":5464,"duration_ms":45477,"significance":"If the benchmark construction is valid, PunchBench is a meaningful resource: it broadens domain coverage to cartoons, posts, comments, and memes; introduces multiple question formats for both perception and reasoning; and attempts to reduce shortcuts by creating synonymous and antonymous captions. The dataset and code are publicly released, and the human annotation process adds credibility. The consistent gains from SC-CoQ across ten models are also of interest. However, the central claim of accurate, shortcut-free evaluation rests on the validity of the GPT-generated caption transformations and on a measured human baseline. Both currently require strengthening before the headline human-MLLM gap and the shortcut-free claim can be accepted.","major_comments":[{"comment":"The labels for the GPT-3.5-generated synonymous and antonymous captions are assigned by fiat rather than verified by human judgment: the synonymous caption P_s inherits the original label and the antonymous caption P_a receives the opposite label. The quality check in Section 3.4 samples only 100 instructions per format and tests whether the instructions are answerable; it does not test whether synonymous captions preserve the punchline or antonymous captions eliminate it. The indirect evidence in Section 5.4, namely that model accuracy drops on the transformed captions, cannot distinguish successful shortcut removal from noise introduced by semantically drifted rewrites. Please add a human validation study on a representative sample of transformed caption-image pairs, report per-caption-type accuracy and inter-annotator agreement, and include a caption-only baseline on the transformed subsets to confirm that text-only shortcuts are actually removed.","section":"Sections 3.2-3.4, Figure 15"},{"comment":"The Generation QA human baseline is set to 100% by construction because the manually annotated reasoning sentences serve as the reference answer for the automatic evaluation. This is not a measured human performance; it trivially defines the ceiling and inflates the reported human-MLLM gap in punchline reasoning. Please provide a measured human baseline for generation, for example by having annotators write free-form reasoning sentences and scoring those with the same evaluation protocol used for MLLM outputs. In addition, the 100-instruction human baselines for the other question formats should be reported separately for original, synonymous, and antonymous captions, so the reader can see whether the transformed captions were validated by the human annotators.","section":"Section 5.1, Human Baseline; Table 1"},{"comment":"The claim that 'the P-value between SC-CoQ performance and other prompting method results is consistently less than 0.01' is not supported by a described statistical test, by error bars, or by any multiple-comparison correction across the twelve models and six question formats. Please specify the test used (e.g., paired bootstrap or Wilcoxon signed-rank), report the number of replicates and the test statistic or exact P-values, and provide confidence intervals or standard errors. Without this, the central claim that SC-CoQ outperforms in-context learning and chain-of-thought cannot be properly evaluated.","section":"Table 1 caption, Section 5.3"}],"minor_comments":[{"comment":"The benchmark name is misspelled as 'PunchBech' in the table header, and the HUB row contains 'Matching,Ranking and ExplanationSingle' with missing spaces.","section":"Table 4"},{"comment":"The prompt text contains the typo 'sunmmarize'; it should be 'summarize'.","section":"Figure 21"},{"comment":"The word 'corss' should be 'cross' in the sentence describing SC-CoQ performance across question formats.","section":"Section 5.3"},{"comment":"The description of the human baseline is ambiguous: it states that 100 instructions were selected for punchline perception, but Table 1 also reports human numbers for punchline reasoning. Please clarify that the same procedure was applied to both tasks.","section":"Section 5.1"},{"comment":"The reference for gpt-3.5-turbo-0125 is given as 'https://chatgpt.com/', which points to the ChatGPT web interface rather than the model API documentation; please cite the appropriate API reference.","section":"Section 5.2, footnote 2"},{"comment":"The text contains the typo 'multimeda platforms' in the sentence about Appendix F; it should be 'multimedia platforms'.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important gap and the released dataset is potentially valuable. The main risk is that the headline claims—shortcut-free evaluation and a large human-model gap—depend on validation that is currently missing. The fiat labeling of transformed captions and the constructed 100% generation baseline are both empirically addressable, so I would not reject on these grounds alone. A targeted human validation study of the transformed captions, a measured generation baseline, and a properly described significance test would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PunchBench is a serious benchmark and worth a referee's time, but the paper's strongest claim—that its synonymous/antonymous captions eliminate text shortcuts—is not actually supported by the evidence presented.\n\nWhat's new: a 6,000-pair, 54,000-question benchmark spanning cartoons, posts, comments, and memes, with four question formats, human labels on the original pairs, and hand-written reasoning sentences for the positive pairs. The evaluation of ten MLLMs is thorough, and the proposed SC-CoQ prompting strategy shows consistent small gains over 3-shot and CoT. The main result—that even GPT-4o is far below human on all formats—holds up without relying on the transformed captions, since it is already visible on the original captions (77.5 vs 98.3 on perception Yes/No; 75.1 vs 96.0 on reasoning Yes/No).\n\nThe soft spots are real but not fatal. The synonymous and antonymous captions are generated by GPT-3.5, and their labels are assigned by fiat: the synonymous caption inherits the original label, the antonymous caption gets the opposite. The quality check samples 100 instructions per format and asks annotators whether the instruction is answerable; it does not verify that a synonymous rewrite preserved the punchline or that an antonymous rewrite killed it. So the 'shortcut-free' evaluation is, at present, an assumption. The fact that model accuracy drops on these rewrites is consistent with shortcut removal but also with noisy or semantically drifted text. A caption-only baseline on the transformed subsets would have helped. This is the one part of the paper that needs real revision, not just a tweak.\n\nTwo smaller issues: the human baseline for Generation QA is set to 100% by construction, since the gold reasoning sentences are used as the reference; that's not a measured human performance and it inflates the gap. And the P-value claim in Table 1 has no described test or error bars.\n\nBottom line: PunchBench is a usable resource and the evaluation is informative, especially for relative model comparisons. But the shortcut-elimination claim should be toned down until the caption transformations are validated. I'd send it to review with a request for a human validation study on a sample of transformed pairs, plus caption-only baselines. The authors clearly know the area and the paper is honest about its scope; it just overreaches in one place. For anyone working on MLLM evaluation or humor understanding, this is worth engaging with.","headline":"PunchBench is a serious benchmark, but its shortcut-elimination claim rests on unvalidated caption transformations.","tokens_in":30180,"tokens_out":2973,"would_cite":true,"duration_ms":26863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PunchBench: AI scores 53% on image-caption jokes where humans score 91%","keywords":["multimodal punchline comprehension","benchmark","humor understanding","sarcasm detection","multimodal large language models","synonymous and antonymous captions","chain-of-question prompting","image-caption pairs"],"falsifier":"Ask fresh human annotators to label a random sample of the generated antonymous captions paired with their images; if a substantial share of antonymous pairs are still judged to contain punchlines, the antonym flip does not hold and the benchmark's shortcut-free claim loses its basis.","tokens_in":29178,"feed_emoji":"😂","tokens_out":8290,"duration_ms":64964,"temperature":0.7,"pith_summary":"PunchBench is a benchmark for testing whether multimodal large language models truly understand the humor or sarcasm in image-caption pairs, or are only using wording cues. The paper's central claim is that its design blocks those language shortcuts and, once blocked, exposes a large gap between models and people. On the benchmark's most demanding multiple-choice format, the strongest tested model, GPT-4o, scores 53.1 percent while humans score 90.7 percent; on plain yes/no perception the gap is 80.7 versus 98.3 percent. The paper also introduces a prompting strategy, Simple-to-Complex Chain-of-Question, that improves model performance across every question format.","feed_headline":"Best AI scores 53% on joke comprehension where humans get 91%","feed_subtitle":"PunchBench swaps captions for synonyms and antonyms to expose text-only shortcuts in multimodal humor comprehension.","key_machinery":"The central mechanism is the generation of synonymous and antonymous captions, carried out by gpt-3.5-turbo-0125 through word substitution and inversion, with context-consistency adaptation for captions that contain internal contradictions. By forcing the model to face rephrased and semantically flipped versions of the same caption, the benchmark removes the biased words and text-only inconsistencies that allow shortcuts. The second mechanism is the Simple-to-Complex Chain-of-Question prompting strategy, which orders questions from easy formats (Yes/No) to hard ones (Multi-option or Generation) so that earlier responses feed into the final answer.","core_discovery":"The paper's central claim is that PunchBench gives an accurate and comprehensive measure of multimodal punchline comprehension, and that evaluations with it show current MLLMs fall far short of human ability. The benchmark holds 6,000 image-caption pairs, each expanded into original, synonymous, and antonymous captions, and 54,000 question-answer pairs covering punchline perception and reasoning, four question formats, and domains from cartoons to memes. The labels for the rewritten captions follow a rule: synonymous captions keep the original pair's label, antonymous captions receive the opposite. Across ten MLLMs, the strongest, GPT-4o, reaches 80.7 percent on Yes/No perception but only 53.1 percent on multi-option QA, against human scores of 98.3 and 90.7 percent respectively, which the paper reads as evidence that these models lean on shallow caption cues instead of understanding the image-caption interplay.","pith_inferences":["The synonymous/antonymous caption perturbation could be adapted to other caption-dependent vision-language benchmarks as a cheap probe for how much of a model's score comes from text priors rather than image understanding.","The large drop on antonymous captions suggests a testable hypothesis beyond humor: current MLLMs may not reliably track semantic negation and contrast in multimodal settings.","Because the antonymous labels are assigned by rule, a natural next step is human verification of whether each generated caption actually flips the punchline, or reformulating the task as predicting whether a rewrite preserves meaning.","The near-ceiling human scores imply that the bottleneck is joint visual-textual reasoning rather than joke difficulty, so extending the design to video punchlines would be a demanding follow-up, as the paper itself notes."],"forward_implications":["A single question format overstates MLLM punchline ability, since rankings and raw scores shift substantially across the Yes/No, Matching, Multi-option, and Generation formats.","The performance drop when original captions are replaced with synonymous or antonymous ones indicates that current MLLMs rely on surface wording rather than the image-caption incongruity that creates the punchline.","Explaining why a pair is funny is consistently harder for MLLMs than detecting that it is funny, so evaluation should separate perception from reasoning.","SC-CoQ improves accuracy on all question formats and caption versions, and beats both 3-shot in-context learning and chain-of-thought prompting.","PunchBench's 54,000 question-answer pairs provide a reusable testbed for developing and auditing multimodal humor systems."],"supporting_citations":[{"why":"Provides the MTSD social-media post dataset that PunchBench draws image-caption pairs and punchline labels from.","marker":"(Cai et al., 2019)"},{"why":"Provides the MORE multimodal sarcasm dataset used as another source of punchline pairs and explanations.","marker":"(Kumar et al., 2022)"},{"why":"Provides the HUB New Yorker cartoon caption contest dataset, the source of cartoon-domain pairs.","marker":"(Hessel et al., 2023)"},{"why":"Supplies the ChatGPT model used to generate synonymous and antonymous captions and distractor options.","marker":"(OpenAI, 2022)"},{"why":"Supplies GPT-4o, the strongest closed-source MLLM baseline, and the model used to generate distractor captions.","marker":"(OpenAI, 2024)"},{"why":"Supplies CogVLM2, the model used in the qualitative shortcut analysis and in the synonymous/antonymous caption comparisons.","marker":"(Hong et al., 2024)"},{"why":"Defines in-context learning, the 3-shot prompting baseline that SC-CoQ is compared against.","marker":"(Brown et al., 2020)"},{"why":"Defines chain-of-thought prompting, the reasoning baseline that SC-CoQ is compared against.","marker":"(Wei et al., 2022)"}],"fun_headline_variants":["PunchBench: AI scores 53% on jokes, humans 91%","Synonym swap exposes AI's punchline deficit: 53% vs 91%","MLLMs lag 38 points in humor comprehension, PunchBench finds","AI fails multimodal punchlines: 53% accuracy, humans hit 91%","PunchBench traps text-only shortcuts, reveals AI humor gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The shortcut-free evaluation depends on the generated synonymous and antonymous captions actually preserving or inverting the punchline's meaning, since their labels are assigned by rule rather than verified by human judgment.","fun_headline_variants_meta":{"raw":{"variants":["PunchBench: AI scores 53% on jokes, humans 91%","Synonym swap exposes AI's punchline deficit: 53% vs 91%","MLLMs lag 38 points in humor comprehension, PunchBench finds","AI fails multimodal punchlines: 53% accuracy, humans hit 91%","PunchBench traps text-only shortcuts, reveals AI humor gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1526,"prompt_tokens":1016,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":632,"tokens_out":510,"duration_ms":16092,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:27:18.225030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask fresh human annotators to label a random sample of the generated antonymous captions paired with their images; if a substantial share of antonymous pairs are still judged to contain punchlines, the antonym flip does not hold and the benchmark's shortcut-free claim loses its basis.","supporting_citations":[],"review_version":1}