Pith. sign in

REVIEW 4 major objections 5 minor 52 references

VAGUE: Visual Contexts Clarify Ambiguous Expressions

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces VAGUE, a benchmark claiming that vision-language models can perceive visual details yet fail to reason from them to infer a speaker's true intent.

desk verdict A genuinely new benchmark with a plausible central finding, but the human–model gap and the 'perceive but not reason' claim rest on a single annotator and unvalidated distractors — fixable, and worth refereeing. read the letter →

arxiv 2411.14137 v3 pith:IAZDJKZK submitted 2024-11-21 cs.CV cs.CL

Heejeong Nam , Jinwoo Ahn , Keummin Ka , Jiwan Chung , Youngjae Yu This is my paper · ORCID
classification cs.CVcs.CL
keywords multimodalintentiondisambiguationvisualcontextambiguousexpressionsbenchmarkvision-languagemodelstheoryofmindcounterfactualchoiceshumanevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VAGUE, a benchmark of 1,677 images paired with indirect, ambiguous utterances and four-choice interpretations, to test whether multimodal AI can infer a speaker's true intent from visual context alone. The central claim is that current vision-language models can use visual information but cannot reason with it: accuracy rises steadily as more visual detail is supplied, yet the best model still lands near 72 percent while humans reach 94 percent. The paper further claims that the dominant failure mode is 'superficial understanding'—models pick the literal reading of the utterance instead of the intention the image disambiguates. A sympathetic reader would take this as quantitative evidence that perceiving a scene and understanding what a person wants done in it are distinct capabilities, with the latter still largely missing.

What carries the argument

The load-bearing mechanism is the benchmark's construction pipeline rather than a mathematical identity. Each item starts with a direct, solvable request grounded in a physical object present in the image, from which an indirect expression is generated that hides both the action and the object. Three interpretable counterfactual choices are engineered per item: a Fake Scene interpretation from an imagined different image, a Superficial Understanding from literal reading, and a Nonexistent Entity that swaps in an object absent from the scene. This taxonomy turns every wrong answer into a diagnostic, letting the paper attribute model errors to perception failures versus reasoning failures. Human filtering based on explicit criteria of relevance, solvability, consistency, and ambiguity is what makes the answers visually dependent, so the task cannot be solved from text priors alone.

What would settle it

Have three or more independent raters answer the same 400-item subset and compute inter-annotator agreement; if human accuracy drops toward the model range or agreement is low, then the reported 20-point gap and the 'perception without reasoning' interpretation would not survive.

Watch

Extended reading notes

Core claim

The paper's core discovery is a measurable gap between visual perception and multimodal reasoning. On VAGUE, each item presents an image from the speaker's viewpoint, an indirect expression (for example, 'Hey person1, spot the difference, this parking's a bit too special, isn't it?'), and four interpretations designed so that only the visual context makes one correct. The authors report that all twelve evaluated models improve when given captions and improve further with raw images, showing they do extract visual cues; nevertheless, on a filtered 400-item subset humans answer correctly 94 percent of the time while the strongest model, Qwen2.5-VL-Instruct (72B), reaches 72.3 percent. Error analysis attributes most mistakes to Superficial Understanding choices, which follow the literal wording, rather than to misreading the scene (Fake Scene) or hallucinating an object (Nonexistent Entity). The paper concludes that models perceive image content but fail to integrate it into intent inference.

Load-bearing premise

The claim that humans solve this task at 94 percent and that the correct answers are visually dependent rests on a single student annotator scoring a filtered 400-item subset, with no second annotator measuring agreement.

Editorial extensions

If this is right

  • Adding visual cues—from no image, to a short caption, to the raw image—consistently raises accuracy across nearly all evaluated models, confirming that the task is genuinely multimodal.
  • The roughly 20-point gap between the best model and human performance on the same 400 items implies that intent disambiguation is not yet solved by scaling or instruction tuning alone.
  • Because Superficial Understanding is the most frequent error, progress on VAGUE would come less from better object recognition than from deeper pragmatic reasoning over what the speaker is asking for.
  • The counterfactual design lets each wrong answer be classified as Fake Scene, Superficial Understanding, or Nonexistent Entity, which is how the paper identifies the dominant failure mode.
  • Chain-of-thought prompting helps proprietary models only when they see the raw image, suggesting that explicit reasoning can partially compensate for missing visual grounding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the single-annotator human ceiling of 94 percent holds up under multi-rater testing, then the VAGUE-style format could be adapted as a live probe for embodied assistants: a system that fails here would likely also mishandle indirect user requests in real scenes.
  • The paper's own observation that proprietary models do better with captions than with raw images suggests a testable extension: supplying models with more detailed, structured scene descriptions might shrink the gap more than any reasoning prompt alone.
  • Because all utterances were drafted by a single generative model and filtered by English-speaking annotators, the ambiguity types are likely skewed toward Western sarcasm and idioms; a cross-lingual version could reveal whether the perception-reasoning gap is language-dependent.
  • A direct probe of the paper's interpretation would be to feed models the correct answer's reasoning but with the image removed; if performance then collapses, the visual dependency claim is confirmed, whereas if it stays high, some items may be solvable from text priors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VAGUE, a 1,677-item benchmark for multimodal intention disambiguation (MID). Each item consists of an image, an ambiguous indirect utterance, and four multiple-choice interpretations, with distractors categorized as Fake Scene Understanding (FS), Superficial Understanding (SU), and Nonexistent Entity (NE). The dataset is built from VCR and Ego4D images with GPT-4o-generated text that is then human-rated and filtered. The authors evaluate LMs, Socratic Models, and VLMs, reporting that visual cues improve accuracy but that all models fall well below a human accuracy of 94% on a 400-item subset. Failure analysis indicates that SU errors are the most common error type, leading the authors to conclude that current models perceive image content but do not effectively reason with it. The paper also reports Chain-of-Thought experiments on proprietary and open models.

Significance. If the benchmark's validity assumptions hold, VAGUE is a useful resource for evaluating multimodal pragmatic reasoning and theory-of-mind-adjacent inference, and the explicit distractor taxonomy provides a diagnosable error analysis that goes beyond simple accuracy reporting. The paper evaluates a broad set of open and proprietary models, reports both multiple-choice and free-form results, and releases code and data, which are concrete strengths. However, the headline quantitative claims rest on two load-bearing validation legs: a human performance estimate from a single annotator, and the semantic validity of the generated distractors, particularly the SU category. Both legs are currently under-supported, and the internal inconsistency in the human row of Table 2 compounds the concern. The core idea is valuable, but the evidence as presented does not yet fully establish the strength of the stated human-model gap or the 'perceive but not reason' interpretation.

major comments (4)
  1. [Sec. 5.3, Appendix E, Table 2] The human upper bound of 94% is based on a single student researcher annotating a 400-item subset, with no inter-annotator agreement and no independent second rater. Because the headline result is the large human-model gap, this is load-bearing: a broader rater pool could yield different answers and change the gap. In addition, the human row in Table 2 is internally inconsistent: the reported Correct count is 374, the incorrect counts sum to 12+4+8=24, giving 398 items rather than 400, and 374/400 is 93.5%, not 94.0%. The authors should report the exact denominator, reconcile the counts, and provide multi-annotator agreement statistics on at least a representative subsample.
  2. [Sec. 4.2.2, Sec. 4.2.3, Fig. J10, Sec. 5.2] The central failure-mode claim that SU is the dominant error and indicates a reasoning deficit presupposes that SU distractors are semantically invalid and roughly as plausible as the correct answer. However, SU options are generated by GPT-4o in the same prompt as the indirect expression (Fig. J10) and are selected together with that expression during human filtering (Sec. 4.2.2). No human validation is reported for whether SU options are actually wrong or merely less preferred, and the same holds for FS and NE distractors. The authors should run an annotator study that rates each distractor as valid/invalid and compares perceived plausibility against the correct option; without this, the SU-error analysis cannot distinguish a reasoning deficit from a distractor-difficulty artifact.
  3. [Sec. 5.2, Fig. 4] The claim that 'Superficial Understanding is the most common error type' is made from raw error counts without comparison to chance or to option-level baselines. Because there are three incorrect options, a random chooser would select each incorrect type one-third of the time conditional on being wrong; some models' SU rates are close to or below that baseline in the full tables, e.g., Qwen2.5-VL-72B VLM on VCR has 159 SU errors out of 295 total errors (53.9%), but this is not compared with a chance baseline or with the relative prevalence of each distractor type in the dataset. The authors should report per-option choice rates, chance-adjusted error proportions, and, ideally, position- and length-controlled analyses to support the failure-mode interpretation.
  4. [Appendix B.4, Sec. 4.1.3] The OCR validation for person indicator tags is not sufficient to support the claim that the task only requires 'basic OCR' that models handle reliably. The test uses only three COCO images, all with the same correct answer (red shirts), and does not use any VAGUE images, person tags in varied fonts/overlays, or distractor questions. If person indicators are sometimes unreadable or ambiguous in the actual benchmark, model errors could be attributed to grounding failures rather than intent reasoning. A small but representative OCR evaluation on the actual VAGUE images, or an explicit analysis of how many errors involve the wrong person, would address this concern.
minor comments (5)
  1. [Table 1 vs Appendix J5] InternVL-3 (38B) SM accuracy on VAGUE-VCR is reported as 47.2 in Table 1 but 47.6 in Table J5; the numbers should be reconciled.
  2. [Appendix D title] The appendix heading reads 'Free-From Answering' and should be 'Free-Form Answering'.
  3. [References] Reference [5] lists the author name as 'Zhawnen Chen'; this appears to be a typo and should be corrected.
  4. [Fig. J25 / dataset structure] The dataset includes an 'ordering' field for MCQ options, but the paper does not state whether option order was randomized per model or per human participant; position bias could affect the failure-mode analysis, so the protocol should be described.
  5. [Sec. 5.1] The increase values in Table 1 are reported as differences from LM accuracy, but no confidence intervals or significance tests are provided; given that some differences are small (e.g., Ovis2 VLM vs SM), a statement about run-to-run variability or deterministic decoding would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: model accuracies and the human upper bound are held-out measurements, and GPT-4o's role in generating the benchmark does not force any model's score, since GPT-4o itself underperforms several open models.

full rationale

The paper's claimed chain is benchmark construction, human filtering, frozen-model evaluation, and error classification. None of these steps is a derivation in which an output quantity is equivalent to an input by construction. Model accuracies in Tables 1, 2, J5, and J6 are empirical counts on a fixed test set; no parameter is fitted to the test set and no accuracy is predicted from a fitted quantity. The fact that GPT-4o generated the indirect expressions and answer options does not force GPT-4o's accuracy: GPT-4o scores 65.1% on VAGUE-VCR and 63.6% on VAGUE-Ego4D, while Qwen2.5-VL-Instruct (72B) reaches 74.2% and 69.8%, respectively, so the benchmark outcome is not an artifact of generator-model advantage. The human upper bound of 94% on the 400-item subset is also a measurement, not an assumed constant. The single-annotator protocol and lack of inter-annotator agreement described in Appendix E are real reliability concerns about whether the human figure generalizes, but they do not make the human score a circular input to the model evaluation. The error taxonomy is defined before evaluation in Section 4.2.3, with the Superficial Understanding (SU) distractor deliberately constructed as the literal reading of the indirect expression. Observing that models select SU more often than FS or NE is an empirical finding reported in Figure 4 and Section 5.2, not a quantity forced by the SU definition; the paper does not claim to predict SU dominance from the distractor construction alone. One can question whether SU distractors are as valid or as plausible as the gold labels, but that is a benchmark-validity issue, not a circular reduction. There are no load-bearing self-citations or imported uniqueness theorems; all citations are to external datasets, models, or standard prior work. The central claim that current models perceive visual content but fail to reason about speaker intent is therefore supported by independent measurements, even though the benchmark's construct validity could be strengthened by multi-annotator validation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

VAGUE is an empirical artifact rather than a theory-driven derivation. Its validity rests on the quality of parent datasets, automated taggers, GPT-4o text generation with human filtering, and a fragile single-annotator human baseline.

free parameters (4)
  • direct_expression_rating_threshold = 4 or 5 out of 5
    Only direct expressions rated 4 or 5 by human raters are retained; this hand-chosen threshold affects the difficulty and quality of prompts in the benchmark.
  • indirect_expression_rating_threshold = 3 or higher out of 5
    Only indirect expressions rated 3 or higher are retained; this hand-chosen threshold affects how indirect and natural the kept prompts are.
  • VCR_image_object_count_cutoff = top 4,000 of 10,000 sampled images
    VCR candidate images are sorted by RAM-detected object count and the top 4,000 are kept; this hand-chosen cutoff biases the benchmark toward visually dense scenes.
  • human_evaluation_subset_size = 400 samples
    Human performance is measured on 400 high-quality items rather than the full 1,677; this selection rule shapes the reported 94% human baseline.
assumptions (5)
  • domain assumption VCR and Ego4D images are representative of complex, natural human interactions.
    Sections 4.1.1 and Appendix B.2 rely on these parent datasets for scene diversity; if they are not representative, VAGUE may not generalize to other interaction contexts.
  • domain assumption Automated object tagging (RAM) and person detection (YOLOv11) are accurate enough for the benchmark after human filtering.
    Section 4.1.2 and the Limitations section acknowledge RAM misidentifications and duplicated YOLO boxes; the quality of grounding depends on these annotations being mostly correct.
  • domain assumption GPT-4o-generated indirect expressions and multiple-choice options contain exactly one correct answer and natural distractors.
    Section 4.2 describes generation and human rating for direct and indirect expressions, but no independent human validation of the four answer options is reported.
  • ad hoc to paper A single fluent English-speaking annotator provides a reliable estimate of human performance.
    Appendix E states one student researcher annotated all 400 items with no inter-annotator agreement, making the 94% human score a fragile upper bound.
  • domain assumption Person indicator tags are readable by the evaluated models.
    The benchmark overlays person1, person2, etc. tags; the OCR sanity check in Appendix B.4 is degenerate because the correct answer is red in every test image.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VAGUE: Visual Contexts Clarify Ambiguous Expressions." pith.science (2026). https://pith.science/paper/IAZDJKZK

@misc{pith2026241114137,
  author       = {Pith},
  title        = {Pith review of: VAGUE: Visual Contexts Clarify Ambiguous Expressions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IAZDJKZK}},
  note         = {Machine review of arXiv:2411.14137}
}
read the original abstract

Human communication often relies on visual cues to resolve ambiguity. While humans can intuitively integrate these cues, AI systems often find it challenging to engage in sophisticated multimodal reasoning. We introduce VAGUE, a benchmark evaluating multimodal AI systems' ability to integrate visual context for intent disambiguation. VAGUE consists of 1.6K ambiguous textual expressions, each paired with an image and multiple-choice interpretations, where the correct answer is only apparent with visual context. The dataset spans both staged, complex (Visual Commonsense Reasoning) and natural, personal (Ego4D) scenes, ensuring diversity. Our experiments reveal that existing multimodal AI models struggle to infer the speaker's true intent. While performance consistently improves from the introduction of more visual cues, the overall accuracy remains far below human performance, highlighting a critical gap in multimodal reasoning. Analysis of failure cases demonstrates that current models fail to distinguish true intent from superficial correlations in the visual scene, indicating that they perceive images but do not effectively reason with them. We release our code and data at https://hazel-heejeong-nam.github.io/vague/.

Figures

Figures reproduced from arXiv: 2411.14137 by the authors.

Figure 1
Figure 1. A motivating example demonstrating the importance of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Description of the Multimodal Intention Disambiguation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the data generation process. Based on human-defined criteria and instructions, GPT-4o [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: We present a bar plot to analyze the distribution of incorrect answer choices selected by each model. Each number represents [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 42 canonical work pages

  1. [1]

    Text Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor

    Ashwin Baluja. Text is not all you need: Multimodal prompting helps llms understand humor. arXiv preprint arXiv:2412.05315, 2024. 3

    work page Pith review arXiv 2024

  2. [2]

    Improv- ing image generation with better captions

    James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jian- feng Wang, Linjie Li, † LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † PrafullaDhari- wal, † CaseyChu, † YunxinJiao, and Aditya Ramesh. Improv- ing image generation with better captions. 1

    work page

  3. [3]

    Explicit modelling of theory of mind for belief prediction in nonverbal social interactions

    Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and An- dreas Bulling. Explicit modelling of theory of mind for belief prediction in nonverbal social interactions. In Proc. 27th Eu- ropean Conference on Artificial Intelligence (ECAI), pages 1–8, 2024. 2

    work page 2024

  4. [4]

    FLUTE: Figurative language understand- ing through textual explanations

    Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. FLUTE: Figurative language understand- ing through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pages 7139–7159, Abu Dhabi, United Arab Emirates,

    work page 2022

  5. [5]

    Through the theory of mind’s eye: Reading minds with multimodal video large language models

    Zhawnen Chen, Tianchun Wang, Yizhou Wang, Michal Kosin- ski, Xiang Zhang, Yun Fu, and Sheng Li. Through the theory of mind’s eye: Reading minds with multimodal video large language models. arXiv preprint arXiv:2406.13763, 2024. 2

    arXiv 2024

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6, 3

    arXiv 2024

  7. [7]

    The limitations of large language models for understanding human language and cognition

    Christine Cuskley, Rebecca Woods, and Molly Flaherty. The limitations of large language models for understanding human language and cognition. Open Mind, 8:1058–1083, 2024. 2

    work page 2024

  8. [8]

    Docmsu: A comprehensive benchmark for document- level multimodal sarcasm understanding

    Hang Du, Guoshun Nan, Sicheng Zhang, Binzhu Xie, Jun- rui Xu, Hehe Fan, Qimei Cui, Xiaofeng Tao, and Xudong Jiang. Docmsu: A comprehensive benchmark for document- level multimodal sarcasm understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 17933– 17941, 2024. 2

    work page 2024

Show all 52 references
  1. [9]

    Boss: A benchmark for human belief prediction in object-context scenarios

    Jiafei Duan, Samson Yu, Nicholas Tan, Li Yi, and Cheston Tan. Boss: A benchmark for human belief prediction in object-context scenarios. arXiv preprint arXiv:2206.10665,

    arXiv

  2. [10]

    Multi-modal hallucination control by visual information grounding

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. CVPR 2024, 2024. 4

    2024

  3. [11]

    Kanishk Gandhi, Jan-Philipp Fraenkel, Tobias Gerstenberg, and Noah D. Goodman. Understanding social reasoning in language models with language models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, 2023. Dataset and Bench...

    2023

  4. [12]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 6, 3

    2024 arXiv

  5. [13]

    UR-FUNNY: A multimodal language dataset for understanding humor

    Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natura...

    2019

  6. [14]

    understanding

    Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest. In Pro- ceedings of the 61st Annual Meeting of the Associa...

    2023

  7. [15]

    Memecap: A dataset for captioning and interpreting memes

    EunJeong Hwang and Vered Shwartz. Memecap: A dataset for captioning and interpreting memes. arXiv preprint arXiv:2305.13703, 2023. 3

    2023 arXiv

  8. [16]

    Meme-ingful analysis: Enhanced understanding of cyberbullying in memes through multimodal explanations

    Prince Jha, Krishanu Maity, Raghav Jain, Apoorv Verma, Sriparna Saha, and Pushpak Bhattacharyya. Meme-ingful analysis: Enhanced understanding of cyberbullying in memes through multimodal explanations. arXiv preprint arXiv:2401.09899, 2024. 3

    2024 arXiv

  9. [17]

    MMToM-QA: Mul- timodal theory of mind question answering

    Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen- Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. MMToM-QA: Mul- timodal theory of mind question answering. In Proceedings of the 62nd Annual Meeting of the Association for Compu- ta...

    2024

  10. [18]

    Yolov11: An overview of the key architectural enhancements, 2024

    Rahima Khanam and Muhammad Hussain. Yolov11: An overview of the key architectural enhancements, 2024. 4, 8

    2024

  11. [19]

    When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dia- logues

    Shivani Kumar, Atharva Kulkarni, Md Shad Akhtar, and Tan- moy Chakraborty. When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dia- logues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume...

    2022

  12. [20]

    Explaining (sarcastic) utterances to enhance affect understanding in multimodal dialogues

    Shivani Kumar, Ishani Mondai, Md Shad Akhtar, and Tanmoy Chakraborty. Explaining (sarcastic) utterances to enhance affect understanding in multimodal dialogues. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelli- gence and Thirty-Fifth Conference on Inn...

    2023

  13. [21]

    SBU figures it out: Models explain figurative language

    Yash Kumar Lal and Mohaddeseh Bastan. SBU figures it out: Models explain figurative language. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 143–149, Abu Dhabi, United Arab Emirates (Hybrid), 2022. Association for Computational Linguistics. 2

    2022

  14. [22]

    What matters when building vision-language mod- els? Advances in Neural Information Processing Systems, 37: 87874–87907, 2025

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language mod- els? Advances in Neural Information Processing Systems, 37: 87874–87907, 2025. 6, 3

    2025

  15. [23]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6, 3

    2024 arXiv

  16. [24]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024. 6, 3

    2024

  17. [25]

    Li, Sachin Goyal, Joao D

    Kevin Y . Li, Sachin Goyal, Joao D. Semedo, and J. Zico Kolter. Inference optimal vlms need fewer visual tokens and more parameters, 2025. 4

    2025

  18. [26]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing. 1, 3

    2014

  19. [27]

    I’d rather just go to bed

    Annie Louis, Dan Roth, and Filip Radlinski. “I’d rather just go to bed”: Understanding indirect answers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7411–7425, Online,

    2020

  20. [28]

    Ovis: Structural embed- ding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 6, 3

    2024 arXiv

  21. [29]

    Phi-3 technical report: A highly capable language model locally on your phone, 2024

    Microsoft. Phi-3 technical report: A highly capable language model locally on your phone, 2024. 6, 3

    2024

  22. [30]

    What is the real intention behind this question? dataset collec- tion and intention classification

    Maryam Sadat Mirzaei, Kourosh Meshgi, and Satoshi Sekine. What is the real intention behind this question? dataset collec- tion and intention classification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), p...

    2023

  23. [31]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 5, 6, 8, 3

    2023 arXiv

  24. [32]

    Sentiment-enhanced graph-based sarcasm explanation in dialogue

    Kun Ouyang, Liqiang Jing, Xuemeng Song, Meng Liu, Yu- peng Hu, and Liqiang Nie. Sentiment-enhanced graph-based sarcasm explanation in dialogue. arXiv:2402.11414, 2024. 2

    2024

  25. [33]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311– 318, Philadelphia, Pennsylvania, USA, 2002. Asso...

    2002

  26. [34]

    Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4): 515–526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4): 515–526, 1978. 2

    1978

  27. [35]

    Movie description

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 2017. 4

    2017

  28. [36]

    Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker

    Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ...

    2023

  29. [37]

    Muma-tom: Multi-modal multi-agent theory of mind, 2024

    Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. Muma-tom: Multi-modal multi-agent theory of mind, 2024. 2

    2024

  30. [38]

    PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities

    Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12075–12097,...

    2024

  31. [39]

    DeepMet: A reading com- prehension paradigm for token-level metaphor detection

    Chuandong Su, Fumiyo Fukumoto, Xiaoxi Huang, Jiyi Li, Rongbo Wang, and Zhiqun Chen. DeepMet: A reading com- prehension paradigm for token-level metaphor detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 30–39, Online, 2020. Association f...

    2020

  32. [40]

    DI- RECT: Direct and indirect responses in conversational text corpus

    Junya Takayama, Tomoyuki Kajiwara, and Yuki Arase. DI- RECT: Direct and indirect responses in conversational text corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 1980–1989, Punta Cana, Dominican Republic, 2021. Association for Computati...

    2021

  33. [41]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Ego4d Team. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012, 2022. 2, 4

    2022

  34. [42]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    OpenGVLab Team. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

  35. [43]

    Qwen2.5-vl technical report, 2025

    Qwen Team. Qwen2.5-vl technical report, 2025. 6, 3

    2025

  36. [44]

    Chain-of- thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837, 2022. 8

    2022

  37. [45]

    Social-iq 2.0 challenge: Benchmarking multimodal so- cial understanding

    Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Yous- souf Kebe, Paul Pu Liang, and Louis-Philippe Morency. Social-iq 2.0 challenge: Benchmarking multimodal so- cial understanding. https://github.com/abwilf/ Social-IQ-2.0-Challenge , 2023. 2

    2023

  38. [46]

    From recognition to cognition: Visual commonsense reason- ing

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reason- ing. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6713–6724, 2018. 2, 4, 1

    2019

  39. [47]

    Socratic models: Composing zero-shot multimodal reasoning with language

    Andy Zeng, Maria Attarian, Krzysztof Marcin Choroman- ski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al. Socratic models: Composing zero-shot multimodal reasoning with language. In The Eleventh International Co...

    2023

  40. [48]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Wein- berger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Represen- tations, 2020. 4

    2020

  41. [49]

    Bad” column while prioritizing those in the “Good

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, and Lei Zhang. Recognize anything: A strong image tagging model. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Work...

    2024

  42. [52]

    Hey person2

    frequently generates tags for entities that are not strictly physical objects, such as places, emotions, and colors. To address this, we manually curate a list of 2,403 physical objects from the full set of 4,585 items detectable by RAM [49]. Using this refined list, we filter...

  43. [53]

    open the window

    employs an improved vision encoder and refined instruc- tion tuning for enhanced commonsense reasoning. Ovis2 (16B) [28] excels in image captioning and inference, driven by robust textual grounding and visual alignment. Among proprietary models, GPT-4o [ 31] demonstrates advan...

  44. [2020]

    Association for Computational Linguistics. 2

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.