Pith. sign in

REVIEW 3 major objections 7 minor 56 references

Acknowledging Focus Ambiguity in Visual Questions

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces VQ-FocusAmbiguity, the first VQA dataset that visually grounds every plausible region a question could refer to, and shows modern vision-language models struggle to recognize or locate this focus ambiguity.

desk verdict New benchmark for focus ambiguity in VQA, useful but with a real caveat: the ground truth is bounded by candidate segmentations, so 'all plausible regions' is more like 'all plausible segments from a fixed set.' read the letter →

arxiv 2501.02201 v2 pith:TEV332PW submitted 2025-01-04 cs.CV

Chongyan Chen , Yu-Yun Tseng , Zhuoheng Li , Anush Venkatesh , Danna Gurari This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringfocusambiguitygroundinganswerinstancesegmentationvision-languagemodelsVQAdatasetbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual questions often use under-specified language that can point at several different image regions—"What is this?" or "What is next to the mirror?"—and this paper argues that VQA systems should acknowledge that ambiguity. To make the problem measurable, it introduces VQ-FocusAmbiguity, a dataset of 5,500 visual questions, each paired with segmentations for every region that the question's focus could plausibly refer to, split nearly evenly between ambiguous and unambiguous cases. The paper defines and benchmarks two tasks: deciding whether a question has focus ambiguity, and locating all plausible focus regions. Modern vision-language models perform poorly on both tasks, with the best recognition accuracy around 70% and localization scores below 25 mAP, and analysis shows that question groundings often differ from answer groundings. A sympathetic reader would conclude that the field now has a reusable benchmark for a previously missing capability.

What carries the argument

The central object is the focus of a question: the specific image region or regions that a phrase in the question must be grounded to in order to answer it. The machinery that carries the argument is the annotation protocol that turns "focus" into data—annotators first isolate the to-be-grounded phrase, then record every region that could satisfy the question's constraints, with ambiguity defined as having more than one such region. This yields a ground-truth format that supports both a binary recognition task and an exhaustive localization task, and it is what lets the paper separate question ambiguity from answer ambiguity. The dataset's analysis and the two benchmark tasks both operate directly on these focus segmentations.

What would settle it

Have independent annotators redraw the focus regions for a random sample of questions without being offered a preset catalog of segmentations; if a substantial fraction of questions gains a new plausible region that VQ-FocusAmbiguity does not list, the claim that the dataset captures all plausible focus regions is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that where a question points in an image is a distinct and currently unmodeled source of ambiguity, distinct from answer ambiguity. The authors built VQ-FocusAmbiguity by extending four sources: PACO images with exhaustive object and part segmentations, MSRA-B single-salient-object images, and questions from VQAv2 and VizWiz-VQA linked to AnswerTherapy answer groundings. Each question gets one or more focus segmentations, and the binary ambiguity label is the count of those segmentations. The paper's analysis shows that for 79% of ambiguous AnswerTherapy-derived questions the question groundings differ from the answer groundings, establishing that question focus cannot be read off from where answers are grounded. Benchmarking four foundation models on the two new tasks, the paper finds the best ambiguity-recognition accuracy is about 70% and that no tested localization approach exceeds roughly 24 mAP, and it releases the dataset and an evaluation server for community use.

Load-bearing premise

The ground truth can list a region as plausible only if that region belongs to the candidate segmentations the annotators were shown, so a plausible focus that falls outside those candidates is missing from the dataset.

Editorial extensions

If this is right

  • A VQA service could flag a question as ambiguous instead of silently choosing one interpretation, then let the user select among the presented focus regions.
  • Because question groundings and answer groundings differ in most ambiguous cases, question grounding becomes its own supervision target rather than something derivable from answer grounding.
  • The benchmark's fine-grained breakdowns by data source, number of focus regions, and object-versus-part groundings give model developers specific failure modes to target.
  • With a shared test split and evaluation server, future work can report comparable numbers on the same two tasks, making progress on ambiguity acknowledgment measurable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's recognition analysis hints that region-level counting and pointing training helps models notice ambiguity; a natural next experiment is a two-stage model that first enumerates candidate regions with a segmentation model and then classifies the question as ambiguous if the count exceeds one.
  • The fixed candidate sets used for PACO-derived examples mean the ground truth may undercount truly plausible regions; an open-set annotation study could estimate the size of that gap.
  • The authors' own qualitative examples suggest that models often pick the most salient region and ignore alternatives; a targeted evaluation could measure how often correct ambiguity recognition depends on the model attending to non-salient regions.
  • The same focus-grounding protocol could be applied to other vision-language tasks such as captioning or image editing, and to non-visual modalities, which the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces VQ-FocusAmbiguity, a dataset of 5,500 visual questions with segmentations intended to mark all visually plausible regions a question could refer to, along with binary labels indicating whether the question has focus ambiguity. The dataset is assembled from four sources: PACO, MSRA-B, and the AnswerTherapy subsets of VQAv2 and VizWiz-VQA. The authors analyze the dataset's linguistic and segmentation properties, compare question groundings with answer groundings, and benchmark several vision-language models on two tasks: recognizing focus ambiguity and locating all plausible focus regions. They report that modern models perform poorly on both tasks and publicly release the dataset with an evaluation server.

Significance. If the central claim holds, this is a valuable and timely resource: it is the first benchmark specifically designed for focus ambiguity in visual questions, it disentangles question grounding from answer grounding, and it provides evidence that current vision-language models do not reliably track what a question points at in an image. The paper's strengths include a documented multi-source construction protocol, quality-control steps, a public evaluation server and code release, a broad set of prompting strategies, and a fine-grained analysis of model failures by data source, object/part type, and number of focus regions. The main reservation is that the 'all plausible regions' claim is limited by the candidate segmentation sets and by the MSRA-B single-object assumption; the benchmark is still useful under a weaker interpretation, but the abstract and conclusions currently overstate what the ground truth can certify.

major comments (3)
  1. [§3.1.1, §3.1.2, Supp. §7.1] The central claim that VQ-FocusAmbiguity 'visually grounds each plausible image region a question could refer to' is stronger than the annotation procedure supports. PACO-derived examples are built by having annotators select among PACO's existing segmentations (75 object and 200 part categories), and images are pre-filtered to those with more than three segmented instances (Supp. §7.1); any plausible focus outside that taxonomy—text, shadows, unlabeled objects, regions defined by color or texture, or the whole image—cannot be recorded. For MSRA-B, the automatically generated 'What is this?' questions are assumed to have the single salient segmentation as their only focus (§3.1.1, Supp. §7.2). Since the binary ambiguity label is inferred from the number of segmentations (§3.1.2), an incomplete candidate set directly affects both the recognition labels and the localization ground truth: the reported distribution (median 3, mean 4 focus regions) is a lower bound on true ambiguity, and model 'failure' on localization is measured against a candidate-restricted ground truth. Please either annotate from scratch, allowing new masks beyond the provided candidates, or explicitly qualify the 'all plausible regions' claim throughout the abstract, introduction, and Section 4.2, and report which conclusions survive under the weaker interpretation.
  2. [§3.1.2, §7.4, §7.5] No inter-annotator agreement is reported for the main annotations. The PACO portion is completed by individual crowd workers with spotchecks and automated outlier monitoring (Supp. §7.5); the AnswerTherapy ambiguity labels are produced by one author, and the 330 segmented examples were verified by two authors. The only quantitative agreement statistic is an exact match on 10 of 10 pilot examples for the two UI variants (Supp. §7.4), which is too small to establish reliability for a subjective ambiguity judgment. Please report agreement statistics (e.g., Cohen's kappa or pairwise IoU) on a held-out set for both the binary label and the region-set annotations, and state how disagreements were resolved.
  3. [§4.1, §4.2] The conclusion that models 'perform poorly' should be read in light of the candidate-restricted ground truth described above, and the reported metrics are also highly sensitive to prediction bias. For example, Molmo's 99.5% positive rate under the zero-shot prompt in Table 2 means its recognition accuracy is driven almost entirely by a tendency to predict ambiguity, while its near-100% positive rate in the zero-shot localization setting likely contributes to its higher mAP in Table 3. The paper should provide a baseline comparison (e.g., majority-class or always-ambiguous predictions) and should discuss how the main benchmarking conclusions might change if the evaluation were restricted to examples whose ground-truth region sets are more likely to be exhaustive.
minor comments (7)
  1. [§3.2] In the 'Reasons for Focus Ambiguity' paragraph, the percentage for VQAv2 multiple-instances-of-different-categories is printed as '0.1% (i.e., 9)' but 9/91 is approximately 9.9%; please correct the typo.
  2. [§3.1.1, Table 1] The PACO sample is described as 'randomly sampled' in Section 3.1.1 but as filtered by a complexity requirement in Supp. §7.1; please reconcile the two descriptions.
  3. [§3.1.2, Table 1] The example counts are inconsistent as printed: Section 3.1.1 says 2,272 PACO examples were sampled, yet the dataset totals 5,500 examples only if PACO contributes 4,544 examples (2,272 ambiguous and 2,272 unambiguous). Please clarify whether the 2,272 figure refers to images or examples, and make the column heading in Table 1 unambiguous about whether the parenthetical counts are total examples, unambiguous examples, or images.
  4. [§4.1, Figure 6] The 'answer length' analysis in Figure 6 refers to the length of the model's generated response for the binary classification task, not to dataset answers; consider renaming it to 'output length' to avoid confusion.
  5. [§4.2, Table 3] The missing FS-ECoT row for Molmo+SAM is explained in the text, but the table should also carry an explicit footnote or marker so that readers do not mistake it for an omitted result.
  6. [§4.2] mAP is invoked as the standard instance-segmentation metric, but the matching protocol (IoU threshold, COCO-style averaging) is not defined; please specify the evaluation details.
  7. [Figure 4] Figure 4(a) excludes outliers from the box plot, but the text reports means of 4 and medians of 3 for the number of focus regions; please state whether the reported mean is computed before or after outlier removal.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark conclusions are measured against human annotations and do not reduce to the dataset's construction rules.

full rationale

This paper is a dataset-and-benchmark contribution rather than a derivation, and its central empirical claim (that modern models perform poorly at recognizing focus ambiguity and localizing all plausible focus regions) is evaluated against human-created ground truth that was collected independently of the benchmarked models. The binary ambiguity label is indeed inferred from the number of segmentations in Section 3.1.2, and for PACO/MSRA-B the candidate region sets are inherited from existing taxonomies (Table 1 marks PACO annotations as 'select between candidate segmentations,' and Section 3.1.1 notes PACO 'provides exhaustive segmentations for all instances belonging to 75 object and 200 part categories'), so the dataset's 'all plausible regions' claim is bounded by the annotation protocol. That is a data-coverage limitation, not circular reasoning: the paper does not derive model failure from the annotation rule, and the benchmarked models never observe the labels during evaluation. The prompt-selection pilot in Supplement Section 9.2 tunes the evaluation instrument on model-side pilots, but it selects the definition that helps models most, so it cannot make the reported difficulty result true by construction. Self-citations to VizWiz, AnswerTherapy, and foreground-object ambiguity work are data sources and prior context rather than load-bearing proofs of the new claim. No equation or fitted parameter is renamed as a prediction, and no uniqueness or ansatz result is imported from the authors' prior work. The derivation chain is therefore self-contained with respect to circularity concerns.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No free parameters in the derivational sense; dataset-construction thresholds are listed. The central claims rest on domain assumptions about exhaustive candidate segmentations and single-annotator label reliability.

free parameters (2)
  • PACO complexity filter threshold = more than 3 segmented object or part instances
    Hand-chosen in Supp. Sec. 7.1; defines which PACO images are considered complex enough to yield focus ambiguity, shaping the ambiguous distribution of the dataset.
  • MSRA-B selection criterion = single, noncontroversial foreground object
    Selection rule in Sec. 3.1.1; forces all 626 MSRA-B examples to be unambiguous, a construction choice rather than a measured property of real questions.
assumptions (3)
  • domain assumption PACO-provided segmentations are exhaustive over all plausible focus regions for a question
    Sec. 3.1.1 and 7.4: annotators select from existing PACO segmentations; if a question points to something not in the taxonomy, no region is recorded.
  • domain assumption A single author's ambiguity judgments on the 4,440 VQA-derived examples are reliable
    Sec. 7.5: one author labeled all ambiguity over about 3 weeks; only 330 segmented examples were cross-verified by two other authors, and no inter-annotator agreement is reported.
  • domain assumption The operational definition that ambiguity means under-specified language groundable to multiple regions is the right notion
    Supp. Sec. 9.2: the definition was selected partly by model pilot performance, and the dataset labels this operationalized concept rather than all possible notions of ambiguity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Acknowledging Focus Ambiguity in Visual Questions." pith.science (2026). https://pith.science/paper/TEV332PW

@misc{pith2026250102201,
  author       = {Pith},
  title        = {Pith review of: Acknowledging Focus Ambiguity in Visual Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEV332PW}},
  note         = {Machine review of arXiv:2501.02201}
}
read the original abstract

No published work on visual question answering (VQA) accounts for ambiguity regarding where the content described in the question is located in the image. To fill this gap, we introduce VQ-FocusAmbiguity, the first VQA dataset that visually grounds each plausible image region a question could refer to when arriving at valid answers. We next analyze and compare our dataset to existing datasets to reveal its unique properties. Finally, we benchmark modern models for two novel tasks related to acknowledging focus ambiguity: recognizing whether a visual question has focus ambiguity and locating all plausible focus regions within the image. Results show that the dataset is challenging for modern models. To facilitate future progress on these tasks, we publicly share the dataset with an evaluation server at https://vizwiz.org/tasks-and-datasets/focus-ambiguity-in-visual-questions.

Figures

Figures reproduced from arXiv: 2501.02201 by the authors.

Figure 1
Figure 1. Visual questions with focus ambiguity, meaning lan [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Location of each instance segmentation using normal [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Analysis of questions in the dataset. (a) Box plot show [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (22 more)
Figure 5
Figure 5. Figure 5: Examples of groundings for the question and answers [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Fine-grained analysis of four models in recognizing fo [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Fine-grained analysis on the performance of our three benchmarked approaches for question grounding using the five prompts, [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Example of a possible user interaction interface that [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: UI interface for collecting annotations for PACO-LVIS dataset with segmentations initialized unselected. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: UI interface for collecting annotations for PACO-LVIS dataset with segmentations initialized selected. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: UI interface for collecting the question ambiguity labels for the VQA-AnswerTherapy dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: UI interface for collecting segmentation of regions described by the question for the VQA-AnswerTherapy dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Examples of visual questions with focus ambiguity from three data sources. [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Examples of visual questions without focus ambiguity from four data sources. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Examples of visual questions when the question [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Examples of visual questions are shown with question groundings and answer groundings overlaid on images from VQAv2, [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 20
Figure 20. Figure 20: The prompt for ZS-ECOT setting. age models to differentiate clearly between question and an￾swer ambiguity, similar to how contrastive learning and hu￾mans benefit from hard samples. However, with only two shots, it turned out to only confuse models and led to over￾fi…
Figure 18
Figure 18. Figure 18: The prompt for Zero-Shot setting. “(omitted, same as zero-shot setting...) Please think step by step first, and then predict whether it is ambiguous or not. Please only output “ambiguous” or “unambigu￾ous” [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: The prompt for ZS-COT setting. ECoT consistently lead to better performance. This is be￾cause, CoT is better than without CoT (not surprising); and surprisingly, ZS is better than FS across all models, except for Molmo (7B), in our setting. We attribute this unexpecte…
Figure 21
Figure 21. Figure 21: The prompt for FS setting. Model size ZS ZS-CoT ZS-ECoT FS FS-ECoT 72B 58.7 60.5 64.7 57.7 64.7 7B 57.2 63.8 65.5 53.6 59.0 3B 44.6 44.6 45.0 44.3 44.3 [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: The prompt for FS-ECoT setting. and 20 from PACO. Most of the time both GPT-4o and In￾ternVL2 focus on one valid region from multiple options. Most answers don’t include clear specifications of the ques￾tion region (example 1), while a few do (example 2). In rare case…
Figure 23
Figure 23. Figure 23: Examples of visual question answering for ambiguous questions in preliminary experiments. The coded colors highlight the [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: Qualitative results in GLaMM and ChatGPT-4o+GLaMM, illustrating various challenges in the localization task. [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Qualitative results for examples where GPT-4o suc [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]
Figure 26
Figure 26. Figure 26: Qualitative results for GPT-4o, where (a) presents am [PITH_FULL_IMAGE:figures/full_fig_p021_26.png]
Figure 27
Figure 27. Figure 27: Analysis on the results of the prediction from GPT-4o in the GPT-4o + GLaMM approach for locating question focus regions. [PITH_FULL_IMAGE:figures/full_fig_p022_27.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 37 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 2

    2015

  3. [3]

    Analysis on the results of the prediction from GPT-4o in the GPT-4o + GLaMM approach for locating question focus regions

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- 21 Figure 27. Analysis on the result...

    arXiv 2025

  4. [4]

    Why does a visual question have different answers? In Proceedings of the IEEE International Conference on Computer Vision , pages 4271–4280, 2019

    Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE International Conference on Computer Vision , pages 4271–4280, 2019. 2

    work page 2019

  5. [5]

    Grounding answers for visual questions asked by visually impaired people

    Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 19098–19107, 2022. 2

    work page 2022

  6. [6]

    Vqa therapy: Exploring answer differences by visually ground- ing answers

    Chongyan Chen, Samreen Anjum, and Danna Gurari. Vqa therapy: Exploring answer differences by visually ground- ing answers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15315–15325, 2023. 2, 3

    work page 2023

  7. [7]

    Fully authentic visual question answering dataset from online communities

    Chongyan Chen, Mengchen Liu, Noel Codella, Yunsheng Li, Lu Yuan, and Danna Gurari. Fully authentic visual question answering dataset from online communities. In European Conference on Computer Vision , pages 252–269. Springer,

    work page

  8. [8]

    Cops-ref: A new dataset and task on composi- tional referring expression comprehension

    Zhenfang Chen, Peng Wang, Lin Ma, Kwan-Yee K Wong, and Qi Wu. Cops-ref: A new dataset and task on composi- tional referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10086–10095, 2020. 2

    work page 2020

Show all 56 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 5

    2024 arXiv

  2. [10]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anasta- sios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024. 5

    2024 arXiv

  3. [11]

    Resolving language and vision ambiguities together: Joint segmentation & prepositional attachment resolution in captioned scenes

    Gordon Christie, Ankit Laddha, Aishwarya Agrawal, Stanis- law Antol, Yash Goyal, Kevin Kochersberger, and Dhruv Batra. Resolving language and vision ambiguities together: Joint segmentation & prepositional attachment resolution in captioned scenes. arXiv preprint arXiv:1604.02...

    2016 arXiv

  4. [12]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2...

    2024 arXiv

  5. [13]

    Zero-shot and few-shot video question answering with multi-modal prompts

    Deniz Engin and Yannis Avrithis. Zero-shot and few-shot video question answering with multi-modal prompts. In 22 Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2804–2810, 2023. 3

    2023

  6. [14]

    Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation

    Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Bo- qing Gong. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In Proceedings of the IEEE inter- national conference on computer vision , pages 1811–1820,

  7. [15]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2

    2017

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  9. [17]

    Abg-coqa: Clarifying ambiguity in conversa- tional question answering

    Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. Abg-coqa: Clarifying ambiguity in conversa- tional question answering. In 3rd Conference on Automated Knowledge Base Construction, 2021. 2

    2021

  10. [18]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019. 2, 3

    2019

  11. [19]

    Crowdverge: Predicting if people will agree on the answer to a visual question

    Danna Gurari and Kristen Grauman. Crowdverge: Predicting if people will agree on the answer to a visual question. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3511–3522, 2017. 2

    2017

  12. [20]

    Gurari, K

    D. Gurari, K. He, B. Xiong, J. Zhang, M. Sameki, S. D. Jain, S. Sclaroff M. Betke, and K. Grauman. Predicting fore- ground object ambiguity and efficiently crowdsourcing the segmentation(s). International Journal of Computer Vision,

  13. [21]

    Predicting foreground object ambiguity and efficiently crowdsourcing the segmentation (s)

    Danna Gurari, Kun He, Bo Xiong, Jianming Zhang, Mehrnoosh Sameki, Suyog Dutt Jain, Stan Sclaroff, Margrit Betke, and Kristen Grauman. Predicting foreground object ambiguity and efficiently crowdsourcing the segmentation (s). International Journal of Computer Vision, 126:714–730,

  14. [22]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  15. [23]

    A survey on instance segmentation: state of the art.International jour- nal of multimedia information retrieval, 9(3):171–189, 2020

    Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art.International jour- nal of multimedia information retrieval, 9(3):171–189, 2020. 2

    2020

  16. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6700– 6709, 2019. 2

    2019

  17. [25]

    Long-form answers to visual questions from blind and low vision people

    Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, and Amy Pavel. Long-form answers to visual questions from blind and low vision people. arXiv preprint arXiv:2408.06303, 2024. 3

    2024 arXiv

  18. [26]

    Salient object detection: A discriminative regional feature integration approach

    Huaizu Jiang, Jingdong Wang, Zejian Yuan, Yang Wu, Nan- ning Zheng, and Shipeng Li. Salient object detection: A discriminative regional feature integration approach. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 2083–2090, 2013. 2

    2013

  19. [27]

    Tree of clarifications: Answer- ing ambiguous questions with retrieval-augmented large lan- guage models

    Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answer- ing ambiguous questions with retrieval-augmented large lan- guage models. arXiv preprint arXiv:2310.14696, 2023. 2

    2023 arXiv

  20. [28]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 7

    2023

  21. [29]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  22. [30]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 3

    2023 arXiv

  23. [31]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  24. [32]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, January 2024. 3

    2024

  25. [33]

    Learning to detect a salient object

    Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum. Learning to detect a salient object. IEEE Transactions on Pattern analy- sis and machine intelligence, 33(2):353–367, 2010. 2, 3

    2010

  26. [34]

    Storytelling with image data: a systematic review and comparative anal- ysis of methods and tools

    Fariba Lotfi, Amin Beheshti, Helia Farhood, Matineh Pooshideh, Mansour Jamzad, and Hamid Beigy. Storytelling with image data: a systematic review and comparative anal- ysis of methods and tools. Algorithms, 16(3):135, 2023. 2

    2023

  27. [35]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 3

    arXiv

  28. [36]

    Minesh Mathew, Dimosthenis Karatzas, and C.V . Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, January 2021. 2

    2021

  29. [37]

    Resolving ambi- guities in text-to-image generative models

    Ninareh Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, and Rahul Gupta. Resolving ambi- guities in text-to-image generative models. In Proceedings of the 61st Annual Meeting of the Association for Computa- ...

    2023

  30. [38]

    Ambigqa: Answering ambiguous open-domain questions

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645, 2020. 2

    2004 arXiv

  31. [39]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. Accessed: 2024-09-01. 3, 5

    2024

  32. [40]

    Rephrase, augment, reason: Visual grounding of ques- tions for vision-language models

    Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. Rephrase, augment, reason: Visual grounding of ques- tions for vision-language models. arXiv preprint arXiv:2310.05861, 2023. 2

    2023 arXiv

  33. [41]

    Referring ex- pression comprehension: A survey of methods and datasets

    Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring ex- pression comprehension: A survey of methods and datasets. IEEE Transactions on Multimedia, 23:4426–4440, 2020. 2

    2020

  34. [42]

    PACO: Parts and attributes of common objects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Mar- quez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yi- wen Song, Abhimanyu Dubey, and Dhruv Mahajan. PACO: Parts and attributes of common objects. In arXiv preprint arXiv:...

    2023 arXiv

  35. [43]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  36. [44]

    Omnilabel: A challenging benchmark for language-based object detec- tion

    Samuel Schulter, Yumin Suh, Konstantinos M Dafnis, Zhix- ing Zhang, Shiyu Zhao, Dimitris Metaxas, et al. Omnilabel: A challenging benchmark for language-based object detec- tion. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 11953–11962, 2023. 2

    2023

  37. [45]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 2

    2019

  38. [46]

    Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in vqa

    Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in vqa. arXiv preprint arXiv:2211.07516, 2022. 2

    2022 arXiv

  39. [47]

    Vizwiz- fewshot: Locating objects in images taken by people with visual impairments

    Yu-Yun Tseng, Alexander Bell, and Danna Gurari. Vizwiz- fewshot: Locating objects in images taken by people with visual impairments. In European Conference on Computer Vision, pages 575–591. Springer, 2022. 3

    2022

  40. [48]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212, 2021. 3

    2021

  41. [49]

    Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems

    Mengting Wan and Julian McAuley. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In Data Mining (ICDM), 2016 IEEE 16th International Conference on , pages 489–498. IEEE,

    2016

  42. [50]

    Twice opportunity knocks syn- tactic ambiguity: A visual question answering model with yes/no feedback

    Jianming Wang, Wei Deng, Yukuan Sun, Yuanyuan Li, Kai Wang, and Guanghao Jin. Twice opportunity knocks syn- tactic ambiguity: A visual question answering model with yes/no feedback. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 736–741. IEEE, 2019. 2

    2019

  43. [51]

    Phrasecut: Language-based image segmen- tation in the wild

    Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language-based image segmen- tation in the wild. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020. 2

    2020

  44. [52]

    Described object detection: Liberating ob- ject detection with flexible expressions

    Chi Xie, Zhao Zhang, Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating ob- ject detection with flexible expressions. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  45. [53]

    Visual Question Answer Diversity

    Chun-Ju Yang, Kristen Grauman, and Danna Gurari. Visual Question Answer Diversity. In HCOMP, pages 184–192,

  46. [54]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  47. [55]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In The IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2016. 2

    2016

  48. [56]

    Object detection in 20 years: A survey.Proceed- ings of the IEEE, 111(3):257–276, 2023

    Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey.Proceed- ings of the IEEE, 111(3):257–276, 2023. 2 24

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.