Pith. sign in

REVIEW 3 major objections 6 minor 66 references

PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PunchBench: AI scores 53% on image-caption jokes where humans score 91%

desk verdict PunchBench is a serious benchmark, but its shortcut-elimination claim rests on unvalidated caption transformations. read the letter →

arxiv 2412.11906 v2 pith:DVX7O6MH submitted 2024-12-16 cs.CV cs.AI

Kun Ouyang , Yuanxin Liu , Shicheng Li , Yi Liu , Hao Zhou , Fandong Meng , Jie Zhou , Xu Sun This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodalpunchlinecomprehensionbenchmarkhumorunderstandingsarcasmdetectionlargelanguagemodelssynonymousandantonymouscaptionschain-of-questionpromptingimage-captionpairs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PunchBench is a benchmark for testing whether multimodal large language models truly understand the humor or sarcasm in image-caption pairs, or are only using wording cues. The paper's central claim is that its design blocks those language shortcuts and, once blocked, exposes a large gap between models and people. On the benchmark's most demanding multiple-choice format, the strongest tested model, GPT-4o, scores 53.1 percent while humans score 90.7 percent; on plain yes/no perception the gap is 80.7 versus 98.3 percent. The paper also introduces a prompting strategy, Simple-to-Complex Chain-of-Question, that improves model performance across every question format.

What carries the argument

The central mechanism is the generation of synonymous and antonymous captions, carried out by gpt-3.5-turbo-0125 through word substitution and inversion, with context-consistency adaptation for captions that contain internal contradictions. By forcing the model to face rephrased and semantically flipped versions of the same caption, the benchmark removes the biased words and text-only inconsistencies that allow shortcuts. The second mechanism is the Simple-to-Complex Chain-of-Question prompting strategy, which orders questions from easy formats (Yes/No) to hard ones (Multi-option or Generation) so that earlier responses feed into the final answer.

What would settle it

Ask fresh human annotators to label a random sample of the generated antonymous captions paired with their images; if a substantial share of antonymous pairs are still judged to contain punchlines, the antonym flip does not hold and the benchmark's shortcut-free claim loses its basis.

Watch

Extended reading notes

Core claim

The paper's central claim is that PunchBench gives an accurate and comprehensive measure of multimodal punchline comprehension, and that evaluations with it show current MLLMs fall far short of human ability. The benchmark holds 6,000 image-caption pairs, each expanded into original, synonymous, and antonymous captions, and 54,000 question-answer pairs covering punchline perception and reasoning, four question formats, and domains from cartoons to memes. The labels for the rewritten captions follow a rule: synonymous captions keep the original pair's label, antonymous captions receive the opposite. Across ten MLLMs, the strongest, GPT-4o, reaches 80.7 percent on Yes/No perception but only 53.1 percent on multi-option QA, against human scores of 98.3 and 90.7 percent respectively, which the paper reads as evidence that these models lean on shallow caption cues instead of understanding the image-caption interplay.

Load-bearing premise

The shortcut-free evaluation depends on the generated synonymous and antonymous captions actually preserving or inverting the punchline's meaning, since their labels are assigned by rule rather than verified by human judgment.

Editorial extensions

If this is right

  • A single question format overstates MLLM punchline ability, since rankings and raw scores shift substantially across the Yes/No, Matching, Multi-option, and Generation formats.
  • The performance drop when original captions are replaced with synonymous or antonymous ones indicates that current MLLMs rely on surface wording rather than the image-caption incongruity that creates the punchline.
  • Explaining why a pair is funny is consistently harder for MLLMs than detecting that it is funny, so evaluation should separate perception from reasoning.
  • SC-CoQ improves accuracy on all question formats and caption versions, and beats both 3-shot in-context learning and chain-of-thought prompting.
  • PunchBench's 54,000 question-answer pairs provide a reusable testbed for developing and auditing multimodal humor systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The synonymous/antonymous caption perturbation could be adapted to other caption-dependent vision-language benchmarks as a cheap probe for how much of a model's score comes from text priors rather than image understanding.
  • The large drop on antonymous captions suggests a testable hypothesis beyond humor: current MLLMs may not reliably track semantic negation and contrast in multimodal settings.
  • Because the antonymous labels are assigned by rule, a natural next step is human verification of whether each generated caption actually flips the punchline, or reformulating the task as predicting whether a rewrite preserves meaning.
  • The near-ceiling human scores imply that the bottleneck is joint visual-textual reasoning rather than joke difficulty, so extending the design to video punchlines would be a demanding follow-up, as the paper itself notes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PunchBench, a benchmark for multimodal punchline comprehension in image-caption pairs. The dataset contains 6,000 image-caption pairs collected from prior datasets and social media, annotated by crowd voting and human-written reasoning sentences. The authors generate synonymous and antonymous captions with GPT-3.5 to mitigate text-only shortcuts, construct perception and reasoning tasks with four question formats, and evaluate ten MLLMs, reporting a large performance gap between MLLMs and humans. They also propose SC-CoQ, a simple-to-complex chain-of-question prompting strategy that improves performance over in-context learning and chain-of-thought. The paper claims that PunchBench provides accurate and comprehensive evaluation and that state-of-the-art MLLMs substantially lag human-level punchline comprehension.

Significance. If the benchmark construction is valid, PunchBench is a meaningful resource: it broadens domain coverage to cartoons, posts, comments, and memes; introduces multiple question formats for both perception and reasoning; and attempts to reduce shortcuts by creating synonymous and antonymous captions. The dataset and code are publicly released, and the human annotation process adds credibility. The consistent gains from SC-CoQ across ten models are also of interest. However, the central claim of accurate, shortcut-free evaluation rests on the validity of the GPT-generated caption transformations and on a measured human baseline. Both currently require strengthening before the headline human-MLLM gap and the shortcut-free claim can be accepted.

major comments (3)
  1. [Sections 3.2-3.4, Figure 15] The labels for the GPT-3.5-generated synonymous and antonymous captions are assigned by fiat rather than verified by human judgment: the synonymous caption P_s inherits the original label and the antonymous caption P_a receives the opposite label. The quality check in Section 3.4 samples only 100 instructions per format and tests whether the instructions are answerable; it does not test whether synonymous captions preserve the punchline or antonymous captions eliminate it. The indirect evidence in Section 5.4, namely that model accuracy drops on the transformed captions, cannot distinguish successful shortcut removal from noise introduced by semantically drifted rewrites. Please add a human validation study on a representative sample of transformed caption-image pairs, report per-caption-type accuracy and inter-annotator agreement, and include a caption-only baseline on the transformed subsets to confirm that text-only shortcuts are actually removed.
  2. [Section 5.1, Human Baseline; Table 1] The Generation QA human baseline is set to 100% by construction because the manually annotated reasoning sentences serve as the reference answer for the automatic evaluation. This is not a measured human performance; it trivially defines the ceiling and inflates the reported human-MLLM gap in punchline reasoning. Please provide a measured human baseline for generation, for example by having annotators write free-form reasoning sentences and scoring those with the same evaluation protocol used for MLLM outputs. In addition, the 100-instruction human baselines for the other question formats should be reported separately for original, synonymous, and antonymous captions, so the reader can see whether the transformed captions were validated by the human annotators.
  3. [Table 1 caption, Section 5.3] The claim that 'the P-value between SC-CoQ performance and other prompting method results is consistently less than 0.01' is not supported by a described statistical test, by error bars, or by any multiple-comparison correction across the twelve models and six question formats. Please specify the test used (e.g., paired bootstrap or Wilcoxon signed-rank), report the number of replicates and the test statistic or exact P-values, and provide confidence intervals or standard errors. Without this, the central claim that SC-CoQ outperforms in-context learning and chain-of-thought cannot be properly evaluated.
minor comments (6)
  1. [Table 4] The benchmark name is misspelled as 'PunchBech' in the table header, and the HUB row contains 'Matching,Ranking and ExplanationSingle' with missing spaces.
  2. [Figure 21] The prompt text contains the typo 'sunmmarize'; it should be 'summarize'.
  3. [Section 5.3] The word 'corss' should be 'cross' in the sentence describing SC-CoQ performance across question formats.
  4. [Section 5.1] The description of the human baseline is ambiguous: it states that 100 instructions were selected for punchline perception, but Table 1 also reports human numbers for punchline reasoning. Please clarify that the same procedure was applied to both tasks.
  5. [Section 5.2, footnote 2] The reference for gpt-3.5-turbo-0125 is given as 'https://chatgpt.com/', which points to the ChatGPT web interface rather than the model API documentation; please cite the appropriate API reference.
  6. [Section 3.1] The text contains the typo 'multimeda platforms' in the sentence about Appendix F; it should be 'multimedia platforms'.

Circularity Check

1 steps flagged · score 4.0 of 10

One reported human score is definitional: the Generation QA human baseline is set to the gold answer itself, making part of the MLLM-vs-human gap an input-as-output artifact.

  1. self definitional [Section 5.1 (Human Baseline), Table 1(b) Generation QA]
    "Notably, the manually annotated reasoning sentences serve as the performance of human baseline for the Generation QA."

    The human baseline for Generation QA is not measured from independent human subjects; it is defined as the gold reference answer itself, i.e., the manually annotated reasoning sentence. Therefore the reported Human score of 100.0 for Generation QA in Table 1(b) is correct by construction. The headline gap in that row (e.g., GPT-4o SC-CoQ 50.1 vs Human 100.0) is partly an artifact of setting the human output equal to the evaluation target rather than an empirical measurement of human generation ability. The perception tasks and other reasoning formats have independently measured human scores, so the overall gap claim does not collapse entirely, but this specific comparison is an input-as-output result.

full rationale

The central benchmark labels are anchored to human crowd voting and human-written reasoning annotations, so the MLLM accuracy numbers are genuinely measured against an external standard rather than fitted. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation; the self-citations present are contextual only. The one clear reduction-by-construction is the Generation QA human baseline, where the 'human' score is defined as the gold reference sentence, inflating the human side of the gap by definition. The synonymous and antonymous caption labels are assigned by fiat (P_s inherits the original label and P_a gets the opposite), and the 100-instruction-per-format quality check does not directly verify semantic preservation or inversion; this is a real construct-validity threat to the 'shortcut-free' claim, but it is not itself a circular derivation because the models' accuracy values are not determined by that label assignment. Similarly, using GPT-4o/GPT-3.5 to generate distractors and options raises contamination concerns without making the measured scores equivalent to the inputs. Overall, the paper's central claim of a large MLLM-human gap has substantial independent support, but the Generation QA human baseline makes part of the reported gap definitional rather than empirical.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The benchmark is an empirical resource; its central claims rest on the validity of GPT-generated caption perturbations, crowd-voted labels, and an LLM-based judge for open-ended answers. The 80% voting threshold is the only hand-chosen number that shapes the data.

free parameters (1)
  • Crowd voting agreement threshold = 80%
    Chosen by hand to assign punchline labels; affects which pairs are kept in the benchmark and therefore the difficulty and composition of the data.
assumptions (3)
  • ad hoc to paper GPT-3.5-generated synonymous and antonymous captions preserve or invert the punchline semantics as labeled
    Section 3.2 relies on this without direct verification; only indirect model performance drops and a small human check support it.
  • domain assumption Crowd voting with greater than 80% agreement yields correct punchline labels
    Appendix A.1; the benchmark's ground truth depends on this threshold, and no inter-annotator agreement metric is reported for the voting.
  • domain assumption GPT-3.5 automatic judge for Generation QA aligns with human judgment
    Section 5.2 and D.3 provide correlation evidence, but the judge remains an unverified LLM proxy for semantic equivalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension." pith.science (2026). https://pith.science/paper/DVX7O6MH

@misc{pith2026241211906,
  author       = {Pith},
  title        = {Pith review of: PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DVX7O6MH}},
  note         = {Machine review of arXiv:2412.11906}
}
read the original abstract

Multimodal punchlines, which involve humor or sarcasm conveyed in image-caption pairs, are a popular way of communication on online multimedia platforms. With the rapid development of multimodal large language models (MLLMs), it is essential to assess their ability to effectively comprehend these punchlines. However, existing benchmarks on punchline comprehension suffer from three major limitations: 1) language shortcuts that allow models to solely rely on text, 2) lack of question diversity, and 3) narrow focus on a specific domain of multimodal content (e.g., cartoon). To address these limitations, we introduce a multimodal \textbf{Punch}line comprehension \textbf{Bench}mark, named \textbf{PunchBench}, which is tailored for accurate and comprehensive evaluation of punchline comprehension. To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions, which mitigates the impact of shortcuts in the captions. To provide a comprehensive evaluation, PunchBench incorporates diverse question formats and image-captions from various domains. On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension. To improve punchline comprehension, we propose Simple-to-Complex Chain-of-Question (SC-CoQ) strategy, enabling the models to incrementally address complicated questions by first mastering simple ones. SC-CoQ effectively enhances the performance of various MLLMs on PunchBench, surpassing in-context learning and chain-of-thought.

Figures

Figures reproduced from arXiv: 2412.11906 by the authors.

Figure 1
Figure 1. An example of multimodal punchline compre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Upper: Data collection workflow for PunchBench. We first collect image-caption pairs from prior datasets and multimedia platforms with meticulous filtering, conduct human annotation to obtain the corresponding labels and reasoning sentences for the pairs. And we then utilize gpt-3.5-turbo-0125 to generate synonymous and antonymous captions corresponding to the original captions. Based on these image-caption pairs, w… view at source ↗
Figure 3
Figure 3. The overall data statistics of our PunchBench. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (20 more)
Figure 5
Figure 5. Figure 5: Example responses from CogVLM2 and GPT-4o to the Yes/No QA with zero-shot prompts. Re￾sponses from CogVLM2 to Multi-option QA with dif￾ferent prompting methods are also presented. ity to effectively capture the semantics of image￾caption pairs and hence achieve better …
Figure 4
Figure 4. Figure 4: Performance comparison cross original, syn [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: Performance comparison for LLaVA across original, synonymous and antonymous captions in zero￾shot, 3-shot, CoT and our SC-CoQ. Zero-shot CoT 3-shot SC-CoQ 55.0 57.5 60.0 62.5 65.0 67.5 70.0 64.0 66.6 65.0 66.8 62.1 59.5 62.6 64.8 58.1 59.3 59.0 59.5 Yes/No QA Zero-shot…
Figure 8
Figure 8. Figure 8: Performance comparison for GLM-4V across [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison for Qwen2-VL across original, synonymous and antonymous captions in zero-shot, 3-shot, CoT and our SC-CoQ. Zero-shot CoT 3-shot SC-CoQ 65.0 70.0 75.0 80.0 85.0 79.1 79.2 79.9 81.4 75.9 73.2 74.0 76.1 70.0 70.2 74.7 76.7 Yes/No QA Zero-shot CoT 3-…
Figure 11
Figure 11. Figure 11: Performance comparison for GPT-4o across [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: An example for qualitative analysis, where [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: An example for qualitative analysis, where [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Two random samples of explanations generated by CogVLM2, GPT-4o, and human-written reasoning [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Prompts used to guide gpt-3.5-turbo-0125, where Prompt1 guides the model to generate synonymous caption, Prompt2 guides it to derive antonymous caption, and Prompt3 guides it to identify the context inconsistency. Yes/No QA Yes/No QA Whether the image-caption pair con…
Figure 16
Figure 16. Figure 16: Prompt examples for Yes/No QA of punchline perception and reasoning using SC-CoQ [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Prompt examples for Matching QA of punchline perception using SC-CoQ. Matching QA Whether the image-caption pair contains punchline? Caption: {caption}. Please answer with Yes or No. The answer is: {response}. Whether the reasoning sentence properly explain why the im…
Figure 18
Figure 18. Figure 18: Prompt examples for Matching QA of punchline reasoning using SC-CoQ [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Prompt examples for Multi-option QA and Generation QA using SC-CoQ. Instruction Templates for Yes/No QA of Punchline Perception Template 1: Whether the image-caption pair contains punchline? Caption: {caption}. Please answer with Yes or No. The answer is: Template 2: …
Figure 20
Figure 20. Figure 20: Instruction templates for Yes/No QA of punchline perception. Instruction Templates for Matching QA of Punchline Perception Template 1: Which caption paired with the image conveys punchline? Caption A: {caption 1}. Caption B: {caption 2}. Please answer with Caption A o…
Figure 21
Figure 21. Figure 21: Prompts used to guide GPT-4o to generate distractor caption and instruction templates for [PITH_FULL_IMAGE:figures/full_fig_p022_21.png]
Figure 22
Figure 22. Figure 22: Prompts used to guide GPT-4o to generate distractor options and instruction templates for [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: Prompts used to guide ChatGPT to generate distractor reasoning sentence and instruction templates for [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: Instruction templates for Matching QA of punchline reasoning. Instruction Templates for Generation QA of Punchline Reasoning Template 1: Please generate a reasoning sentence to explain why the image-caption pair contains punchline? Caption: {caption}. The reasoning se…
Figure 25
Figure 25. Figure 25: Instruction templates for Generation QA of punchline reasoning [PITH_FULL_IMAGE:figures/full_fig_p024_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 14 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In ICCV , pages 2425--2433. IEEE Computer Society

    work page 2015

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 a . https://doi.org/10.48550/ARXIV.2308.12966 Qwen-vl: A frontier large vision-language model with versatile abilities . CoRR, abs/2308.12966

    arXiv doi:10.48550/arxiv.2308.12966 2023

  6. [6]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 b . Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  8. [8]

    Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. https://doi.org/10.18653/V1/P19-1239 Multi-modal sarcasm detection in twitter with hierarchical fusion model . In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 2506--2515. Association for Com...

    doi:10.18653/v1/p19-1239 2019

Show all 66 references
  1. [9]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    2024 arXiv

  2. [10]

    Santiago Castro, Devamanyu Hazarika, Ver \'o nica P \'e rez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. https://doi.org/10.18653/v1/P19-1455 Towards multimodal sarcasm detection (an \_ O bviously \_ perfect paper) . In Proceedings of the 57th Annual Meeti...

    2019 doi

  3. [11]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 a . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  4. [12]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on com...

    2024

  5. [13]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, and Lidong Bing. 2024. https://arxiv.org/abs/2406.07476 Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms . arXiv preprint arXiv...

    2024 arXiv

  6. [14]

    Shad Akhtar

    Poorav Desai, Tanmoy Chakraborty, and Md. Shad Akhtar. 2022. Nice perfume. how long did you marinate in it? multimodal sarcasm explanation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI , pages 10563--10571. AAAI

    2022

  7. [15]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://openreview.net/forum?id=YicbFdNTTy An image is worth 1...

    2021

  8. [16]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024 arXiv

  9. [17]

    Kilem L. Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. In 4th edition edition, pages 1--38. Advanced Analytics, LLC

    2014

  10. [18]

    Hempelmann and Max Petrenko

    Christian F. Hempelmann and Max Petrenko. 2015. https://doi.org/10.1007/978-3-319-20804-6\_59 An AI for humorously reframing interaction narratives with human users . In Distributed, Ambient, and Pervasive Interactions - Third International Conference, DAPI 2015, Held as Part ...

    2015 doi

  11. [19]

    understanding

    Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.41 Do androids laugh at electric sheep? humor "understanding" benchmarks from the new yorker caption contest . In Pro...

    2023 doi

  12. [20]

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500

    2024 arXiv

  13. [21]

    Noman Islam, Zeeshan Islam, and Nazia Noor. 2017. https://api.semanticscholar.org/CorpusID:1040908 A survey on optical character recognition system . ArXiv, abs/1710.05703

    2017 arXiv

  14. [22]

    Jentzsch and Kristian Kersting

    Sophie F. Jentzsch and Kristian Kersting. 2023. https://doi.org/10.18653/V1/2023.WASSA-1.29 Chatgpt is fun, but it is not funny! humor is still challenging large language models . In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Soc...

    2023 doi

  15. [23]

    Pu Jian, Donglei Yu, and Jiajun Zhang. 2024. https://aclanthology.org/2024.emnlp-main.613 Large language models know what is key visual entity: An llm-assisted multimodal retrieval for VQA . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024

  16. [24]

    Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, and Liqiang Nie. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.635 Multi-source semantic graph-based multimodal sarcasm explanation generation . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023 doi

  17. [25]

    Justin Johnson, Andrej Karpathy, and Li Fei - Fei. 2016. https://doi.org/10.1109/CVPR.2016.494 Densecap: Fully convolutional localization networks for dense captioning . In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-...

    2016 doi

  18. [26]

    Dayoon Ko, Sangho Lee, and Gunhee Kim. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.176 Can language models laugh at youtube short-form videos? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10,...

    2023 doi

  19. [27]

    Shad Akhtar, and Tanmoy Chakraborty

    Shivani Kumar, Atharva Kulkarni, Md. Shad Akhtar, and Tanmoy Chakraborty. 2022. https://doi.org/10.18653/V1/2022.ACL-LONG.411 When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dialogues . In Proceedings of the 60th Annual Meeting of the...

    2022 doi

  20. [28]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  21. [29]

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. 2024 b . Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993

    2024 arXiv

  22. [30]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 a . https://proceedings.mlr.press/v202/li23q.html BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning, ICML 20...

    2023

  23. [31]

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023 b . Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  24. [32]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  25. [33]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81. Association for Computational Linguistics

    2004

  26. [34]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . https://doi.org/10.1109/CVPR52733.2024.02484 Improved baselines with visual instruction tuning . In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 , p...

    2024

  27. [35]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 b . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  28. [36]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 a . http://papers.nips.cc/paper\_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html Visual instruction tuning . In Advances in Neural Information Processing Systems 36: Annual Conference...

    2023

  29. [37]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . Visual instruction tuning

    2023

  30. [38]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 c . https://doi.org/10.18653/V1/2024.FINDINGS-ACL.517 Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics, A...

    2024 doi

  31. [39]

    Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, and Xiang Bai. 2023 c . On the hidden mystery of OCR in large multimodal models. CoRR, abs/2305.07895

    2023 arXiv

  32. [40]

    Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang. 2023. https://doi.org/10.1109/CVPR52729.2023.01462 Capdet: Unifying dense captioning and open-world detection pretraining . In IEEE/CVF Conference on Computer Vision and Patt...

    2023

  33. [41]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai - Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://openreview.net/forum?id=KUNzEQMWU7 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts ....

    2024

  34. [42]

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. 2023. http://arxiv.org/abs/2306.07207 Valley: Video assistant with large language model enhanced ability

    2023 arXiv

  35. [43]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  36. [44]

    Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. https://www.aclweb.org/anthology/2021.wanlp-1.42/ Deep multi-task model for sarcasm detection and sentiment analysis in arabic language . In Proceedings of the S...

    2021

  37. [45]

    Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. 2021. https://doi.org/10.1109/WACV48630.2021.00225 Docvqa: A dataset for VQA on document images . In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021 , pages 2199-...

    2021

  38. [46]

    OpenAI. 2022. https://openai.com/blog/chatgpt Introducing chatgpt . CoRR

    2022

  39. [47]

    OpenAI. 2023a. https://cdn.openai.com/papers/GPTV_System_Card.pdf Gpt-4v(ision) system card

  40. [48]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Gpt-4o

    2024

  41. [49]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  42. [50]

    Kun Ouyang, Liqiang Jing, Xuemeng Song, Meng Liu, Yupeng Hu, and Liqiang Nie. 2024. https://doi.org/10.48550/ARXIV.2402.03658 Sentiment-enhanced graph-based sarcasm explanation in dialogue . CoRR, abs/2402.03658

    2024 doi arXiv

  43. [51]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei - Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, ...

    2002

  44. [52]

    Yang Qiao, Liqiang Jing, Xuemeng Song, Xiaolin Chen, Lei Zhu, and Liqiang Nie. 2023. https://doi.org/10.1609/AAAI.V37I8.26138 Mutual-enhanced incongruity learning network for multi-modal sarcasm detection . In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 202...

    2023 doi

  45. [53]

    Noam Shazeer. 2020. http://arxiv.org/abs/2002.05202 GLU variants improve transformer . CoRR, abs/2002.05202

    2020 arXiv

  46. [54]

    The Mistral AI Team. 2023. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 Mistral-7b-instruct-v0.2

    2023

  47. [55]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://doi.org/10....

    2023 doi arXiv

  48. [56]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  49. [57]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

    2024 arXiv

  50. [58]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. http://arxiv.org/abs/2311.03079 Cogvlm: Visual expert for pretrained language models

    2023 arXiv

  51. [59]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html Chain-of-thought prompting elicits reasoning...

    2022

  52. [60]

    Yubo Xie, Junze Li, and Pearl Pu. 2021. https://doi.org/10.18653/V1/2021.ACL-SHORT.6 Uncertainty and surprisal jointly deliver the punchline: Exploiting incongruity-based features for humor recognition . In Proceedings of the 59th Annual Meeting of the Association for Computat...

    2021 doi

  53. [61]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  54. [62]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 doi arXiv

  55. [63]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 c . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  56. [64]

    Yixin Yang, Zheng Li, Qingxiu Dong, Heming Xia, and Zhifang Sui. 2024 d . https://doi.org/10.48550/ARXIV.2402.11281 Can large multimodal models uncover deep semantics behind images? CoRR, abs/2402.11281

    2024 doi arXiv

  57. [65]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    2024 arXiv

  58. [66]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975--11986

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.