{"id":"f4c6b4b5-825d-4eb7-ab28-09d361ac3cd2","arxiv_id":"2501.02201","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VQ-FocusAmbiguity is a 5,500-example dataset annotating all plausible focus regions for visual questions, and state-of-the-art models perform poorly at recognizing and locating focus ambiguity.","lead":"The authors built a new visual-question dataset, VQ-FocusAmbiguity, that marks every image region a question could plausibly be asking about, and showed current vision-language models often cannot detect or locate this ambiguity. The dataset gives the VQA community a benchmark for building assistants that can ask users which object they meant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's 'all plausible regions' ground truth is bounded by the PACO category taxonomy and the MSRA-B single-object assumption, so ambiguity labels and region counts may understate true referential ambiguity.","rationale":"The reader's weakest assumption—candidate-set exhaustiveness—is exactly the load-bearing point. The paper's headline contribution is a dataset that grounds 'each plausible image region,' and both benchmark tasks inherit this assumption: recognition labels are derived from the number of groundings, and localization is scored against those groundings. The PACO pipeline restricts annotators to a fixed taxonomy and the MSRA-B pipeline assumes a unique salient object, so the ground truth is systematically constrained. This is not a disagreement with field consensus but an internal gap between the claim and the construction. The paper deserves credit for publicly releasing the data, an evaluation server, and a detailed annotation protocol; however, those assets do not by themselves establish exhaustiveness. The proposed re-annotation test directly measures whether the constraint bites. If it does, the central claim should be softened from 'all plausible' to 'all plausible within the provided taxonomy/salient-object assumption,' and the benchmark numbers should be interpreted accordingly. This supports keeping the reader's CONDITIONAL verdict rather than upgrading it.","tokens_in":21783,"tokens_out":2781,"duration_ms":25236,"concrete_test":"Take a random sample of 100 PACO-derived ambiguous questions from the test split. Independently re-annotate each image by asking new annotators to segment, without any category restriction, every region to which the question's focus phrase could plausibly refer, using an open-vocabulary interactive segmentation tool such as SAM 2 with human verification. Then measure the fraction of examples in which at least one plausible region lies outside PACO's 75-object/200-part taxonomy or outside the set of PACO masks shown. Also re-annotate 100 MSRA-B examples with the same open-vocabulary protocol to check whether 'What is this?' has any plausible alternative focus beyond the single salient object.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VQ-FocusAmbiguity 'visually grounds each plausible image region a question could refer to' is only as strong as the exhaustiveness of the candidate region sets. In §3.1.1 and Supp. §7.1, PACO-derived examples are grounded by having annotators select among PACO's existing segmentations (75 object and 200 part categories), and images are pre-filtered to those with more than three segmented instances. Any plausible referent outside this taxonomy—e.g., a text label, a shadow, an unlabeled object, a region defined by color or texture, or the whole image—cannot be recorded. Likewise, MSRA-B-derived questions are assumed unambiguous by construction (§3.1.1, Supp. §7.2) with a single salient object as the only focus, so a question like 'What is this?' can never be labeled ambiguous even when the background or another region is a plausible referent. Because the binary ambiguity label is inferred from the number of segmentations (§3.1.2), an incomplete candidate set directly corrupts both the recognition labels and the localization ground truth. Table 1 itself marks PACO segmentations as 'select between candidate segmentations' rather than created from scratch, acknowledging the constraint. The reported distribution (median 3, mean 4 focus regions) is therefore a lower bound on true ambiguity, and model 'failure' on the localization task may be partially an artifact of evaluating against a restricted, taxonomy-bound ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VQ-FocusAmbiguity, a dataset of 5,500 visual questions with segmentations intended to mark all visually plausible regions a question could refer to, along with binary labels indicating whether the question has focus ambiguity. The dataset is assembled from four sources: PACO, MSRA-B, and the AnswerTherapy subsets of VQAv2 and VizWiz-VQA. The authors analyze the dataset's linguistic and segmentation properties, compare question groundings with answer groundings, and benchmark several vision-language models on two tasks: recognizing focus ambiguity and locating all plausible focus regions. They report that modern models perform poorly on both tasks and publicly release the dataset with an evaluation server.","tokens_in":22053,"tokens_out":9496,"duration_ms":86435,"significance":"If the central claim holds, this is a valuable and timely resource: it is the first benchmark specifically designed for focus ambiguity in visual questions, it disentangles question grounding from answer grounding, and it provides evidence that current vision-language models do not reliably track what a question points at in an image. The paper's strengths include a documented multi-source construction protocol, quality-control steps, a public evaluation server and code release, a broad set of prompting strategies, and a fine-grained analysis of model failures by data source, object/part type, and number of focus regions. The main reservation is that the 'all plausible regions' claim is limited by the candidate segmentation sets and by the MSRA-B single-object assumption; the benchmark is still useful under a weaker interpretation, but the abstract and conclusions currently overstate what the ground truth can certify.","major_comments":[{"comment":"The central claim that VQ-FocusAmbiguity 'visually grounds each plausible image region a question could refer to' is stronger than the annotation procedure supports. PACO-derived examples are built by having annotators select among PACO's existing segmentations (75 object and 200 part categories), and images are pre-filtered to those with more than three segmented instances (Supp. §7.1); any plausible focus outside that taxonomy—text, shadows, unlabeled objects, regions defined by color or texture, or the whole image—cannot be recorded. For MSRA-B, the automatically generated 'What is this?' questions are assumed to have the single salient segmentation as their only focus (§3.1.1, Supp. §7.2). Since the binary ambiguity label is inferred from the number of segmentations (§3.1.2), an incomplete candidate set directly affects both the recognition labels and the localization ground truth: the reported distribution (median 3, mean 4 focus regions) is a lower bound on true ambiguity, and model 'failure' on localization is measured against a candidate-restricted ground truth. Please either annotate from scratch, allowing new masks beyond the provided candidates, or explicitly qualify the 'all plausible regions' claim throughout the abstract, introduction, and Section 4.2, and report which conclusions survive under the weaker interpretation.","section":"§3.1.1, §3.1.2, Supp. §7.1"},{"comment":"No inter-annotator agreement is reported for the main annotations. The PACO portion is completed by individual crowd workers with spotchecks and automated outlier monitoring (Supp. §7.5); the AnswerTherapy ambiguity labels are produced by one author, and the 330 segmented examples were verified by two authors. The only quantitative agreement statistic is an exact match on 10 of 10 pilot examples for the two UI variants (Supp. §7.4), which is too small to establish reliability for a subjective ambiguity judgment. Please report agreement statistics (e.g., Cohen's kappa or pairwise IoU) on a held-out set for both the binary label and the region-set annotations, and state how disagreements were resolved.","section":"§3.1.2, §7.4, §7.5"},{"comment":"The conclusion that models 'perform poorly' should be read in light of the candidate-restricted ground truth described above, and the reported metrics are also highly sensitive to prediction bias. For example, Molmo's 99.5% positive rate under the zero-shot prompt in Table 2 means its recognition accuracy is driven almost entirely by a tendency to predict ambiguity, while its near-100% positive rate in the zero-shot localization setting likely contributes to its higher mAP in Table 3. The paper should provide a baseline comparison (e.g., majority-class or always-ambiguous predictions) and should discuss how the main benchmarking conclusions might change if the evaluation were restricted to examples whose ground-truth region sets are more likely to be exhaustive.","section":"§4.1, §4.2"}],"minor_comments":[{"comment":"In the 'Reasons for Focus Ambiguity' paragraph, the percentage for VQAv2 multiple-instances-of-different-categories is printed as '0.1% (i.e., 9)' but 9/91 is approximately 9.9%; please correct the typo.","section":"§3.2"},{"comment":"The PACO sample is described as 'randomly sampled' in Section 3.1.1 but as filtered by a complexity requirement in Supp. §7.1; please reconcile the two descriptions.","section":"§3.1.1, Table 1"},{"comment":"The example counts are inconsistent as printed: Section 3.1.1 says 2,272 PACO examples were sampled, yet the dataset totals 5,500 examples only if PACO contributes 4,544 examples (2,272 ambiguous and 2,272 unambiguous). Please clarify whether the 2,272 figure refers to images or examples, and make the column heading in Table 1 unambiguous about whether the parenthetical counts are total examples, unambiguous examples, or images.","section":"§3.1.2, Table 1"},{"comment":"The 'answer length' analysis in Figure 6 refers to the length of the model's generated response for the binary classification task, not to dataset answers; consider renaming it to 'output length' to avoid confusion.","section":"§4.1, Figure 6"},{"comment":"The missing FS-ECoT row for Molmo+SAM is explained in the text, but the table should also carry an explicit footnote or marker so that readers do not mistake it for an omitted result.","section":"§4.2, Table 3"},{"comment":"mAP is invoked as the standard instance-segmentation metric, but the matching protocol (IoU threshold, COCO-style averaging) is not defined; please specify the evaluation details.","section":"§4.2"},{"comment":"Figure 4(a) excludes outliers from the box plot, but the text reports means of 4 and medians of 3 for the number of focus regions; please state whether the reported mean is computed before or after outlier removal.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the dataset is likely to be useful to the VQA and grounding communities. The main blocking issue is the gap between the 'all plausible regions' claim and the annotation procedure's candidate-set restrictions; this is fixable by reframing the claims and adding a careful discussion, or by collecting additional free-form annotations. The absence of inter-annotator agreement statistics is also fixable with a supplementary agreement study. I would not reject the paper, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a legitimate new dataset and two benchmark tasks for a gap nobody had benchmarked before. The dataset is real, the documentation is thorough, and they share the data and an eval server. The main thing to know before citing it as 'all plausible focus regions' is that the ground truth is only as complete as the segmentations available to annotators—for PACO-derived examples that's the 75+200 category taxonomy, for MSRA-B it's a single salient object. So ambiguity labels are lower bounds on true ambiguity.\n\nWhat's actually new: Visual7W grounds question phrases and AnswerTherapy grounds answers, but nobody had annotated multiple plausible focus regions per question. The two tasks—recognizing whether a question is ambiguous and locating all focus regions—are well-defined. The analysis of how often question groundings differ from answer groundings (79% for ambiguous questions) is a nice contribution on its own. The paper also does the right thing by releasing the dataset and evaluation server.\n\nSoft spots, in rough order of importance:\n\n1. Exhaustiveness. The stress-test concern is correct. PACO annotators selected among existing segmentations, MSRA-B questions are unambiguous by construction, and the binary label is inferred from the number of selected segments. So the dataset's ambiguity rate is a lower bound, and model failure on localization could be partly an artifact of evaluating against a restricted ground truth. The paper is honest about this in Table 1 and the supplement, but the abstract's 'all plausible regions' phrasing overstates it.\n\n2. No inter-annotator agreement. The labels for 5,500 examples come from crowd workers and one author, with quality control but no measured agreement. For a dataset built on human judgment of ambiguity, this needs reporting.\n\n3. No human baseline. Models score 38–70% accuracy on the recognition task, but without human performance it's hard to say whether the task is hard for models or just oddly labeled.\n\n4. Prompt pilot overlap. The pilot study for selecting the definition ran on all of VizWiz-VQA and VQAv2, which include the source of the test examples. The effect is probably small, but the separation should be documented.\n\nNone of this is fatal. The dataset is useful as-is, with the caveat understood. The central argument—that models don't reliably track what a question points at—holds up.\n\nWho it's for: VQA researchers, especially those working on accessibility and clarification. It deserves a serious referee. Send it to peer review, but ask the authors to add agreement statistics, a human baseline, and a careful statement about the exhaustiveness limitation.\n\nThat's my take. I'd bring it to reading group.","headline":"New benchmark for focus ambiguity in VQA, useful but with a real caveat: the ground truth is bounded by candidate segmentations, so 'all plausible regions' is more like 'all plausible segments from a fixed set.'","tokens_in":22589,"tokens_out":3843,"would_cite":true,"duration_ms":34592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces VQ-FocusAmbiguity, the first VQA dataset that visually grounds every plausible region a question could refer to, and shows modern vision-language models struggle to recognize or locate this focus ambiguity.","keywords":["visual question answering","focus ambiguity","question grounding","answer grounding","instance segmentation","vision-language models","VQA dataset","benchmark"],"falsifier":"Have independent annotators redraw the focus regions for a random sample of questions without being offered a preset catalog of segmentations; if a substantial fraction of questions gains a new plausible region that VQ-FocusAmbiguity does not list, the claim that the dataset captures all plausible focus regions is falsified.","tokens_in":21586,"feed_emoji":"❓","tokens_out":9153,"duration_ms":83967,"temperature":0.7,"pith_summary":"Visual questions often use under-specified language that can point at several different image regions—\"What is this?\" or \"What is next to the mirror?\"—and this paper argues that VQA systems should acknowledge that ambiguity. To make the problem measurable, it introduces VQ-FocusAmbiguity, a dataset of 5,500 visual questions, each paired with segmentations for every region that the question's focus could plausibly refer to, split nearly evenly between ambiguous and unambiguous cases. The paper defines and benchmarks two tasks: deciding whether a question has focus ambiguity, and locating all plausible focus regions. Modern vision-language models perform poorly on both tasks, with the best recognition accuracy around 70% and localization scores below 25 mAP, and analysis shows that question groundings often differ from answer groundings. A sympathetic reader would conclude that the field now has a reusable benchmark for a previously missing capability.","feed_headline":"First dataset maps every region a visual question could mean","feed_subtitle":"Modern VQA models rarely flag ambiguous questions or find all the regions they point to.","key_machinery":"The central object is the focus of a question: the specific image region or regions that a phrase in the question must be grounded to in order to answer it. The machinery that carries the argument is the annotation protocol that turns \"focus\" into data—annotators first isolate the to-be-grounded phrase, then record every region that could satisfy the question's constraints, with ambiguity defined as having more than one such region. This yields a ground-truth format that supports both a binary recognition task and an exhaustive localization task, and it is what lets the paper separate question ambiguity from answer ambiguity. The dataset's analysis and the two benchmark tasks both operate directly on these focus segmentations.","core_discovery":"On its own terms, the paper claims that where a question points in an image is a distinct and currently unmodeled source of ambiguity, distinct from answer ambiguity. The authors built VQ-FocusAmbiguity by extending four sources: PACO images with exhaustive object and part segmentations, MSRA-B single-salient-object images, and questions from VQAv2 and VizWiz-VQA linked to AnswerTherapy answer groundings. Each question gets one or more focus segmentations, and the binary ambiguity label is the count of those segmentations. The paper's analysis shows that for 79% of ambiguous AnswerTherapy-derived questions the question groundings differ from the answer groundings, establishing that question focus cannot be read off from where answers are grounded. Benchmarking four foundation models on the two new tasks, the paper finds the best ambiguity-recognition accuracy is about 70% and that no tested localization approach exceeds roughly 24 mAP, and it releases the dataset and an evaluation server for community use.","pith_inferences":["The paper's recognition analysis hints that region-level counting and pointing training helps models notice ambiguity; a natural next experiment is a two-stage model that first enumerates candidate regions with a segmentation model and then classifies the question as ambiguous if the count exceeds one.","The fixed candidate sets used for PACO-derived examples mean the ground truth may undercount truly plausible regions; an open-set annotation study could estimate the size of that gap.","The authors' own qualitative examples suggest that models often pick the most salient region and ignore alternatives; a targeted evaluation could measure how often correct ambiguity recognition depends on the model attending to non-salient regions.","The same focus-grounding protocol could be applied to other vision-language tasks such as captioning or image editing, and to non-visual modalities, which the paper names as future work."],"forward_implications":["A VQA service could flag a question as ambiguous instead of silently choosing one interpretation, then let the user select among the presented focus regions.","Because question groundings and answer groundings differ in most ambiguous cases, question grounding becomes its own supervision target rather than something derivable from answer grounding.","The benchmark's fine-grained breakdowns by data source, number of focus regions, and object-versus-part groundings give model developers specific failure modes to target.","With a shared test split and evaluation server, future work can report comparable numbers on the same two tasks, making progress on ambiguity acknowledgment measurable."],"supporting_citations":[{"why":"supplies most ambiguous-example images and the object/part segmentation catalog annotators selected focus regions from.","marker":"[42]"},{"why":"supplies single-salient-object images whose one segmentation anchors the unambiguous examples.","marker":"[33]"},{"why":"provides authentic questions asked by blind users, one of the two VQA source sets and the origin of the 'What is this?' question pool.","marker":"[22]"},{"why":"provides the VQAv2 questions used as the other VQA source for ambiguous and unambiguous examples.","marker":"[16]"},{"why":"provides the answer groundings used to compare question focus groundings with answer groundings.","marker":"[6]"},{"why":"serves as the state-of-the-art pixel-grounding baseline for the focus localization task.","marker":"[43]"},{"why":"serves as the open-weights multimodal baseline that outputs point locations, and its region-level training data is used to explain recognition differences.","marker":"[12]"},{"why":"converts point prompts into segmentation masks in the Molmo+SAM localization pipeline.","marker":"[28]"},{"why":"generates candidate questions for PACO examples and serves as the best-performing ambiguity recognition baseline.","marker":"[39]"}],"fun_headline_variants":["New VQA dataset reveals focus ambiguity in questions","Where's the focus? Dataset challenges VQA models","First dataset for focus ambiguity in visual questions","Models fail at ambiguous focus: new benchmark exposes gap","In ambiguous VQA, focus often differs from answer location"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground truth can list a region as plausible only if that region belongs to the candidate segmentations the annotators were shown, so a plausible focus that falls outside those candidates is missing from the dataset.","fun_headline_variants_meta":{"raw":{"variants":["New VQA dataset reveals focus ambiguity in questions","Where's the focus? Dataset challenges VQA models","First dataset for focus ambiguity in visual questions","Models fail at ambiguous focus: new benchmark exposes gap","In ambiguous VQA, focus often differs from answer location"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001103,"raw_usage":{"total_tokens":4565,"prompt_tokens":875,"completion_tokens":3690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":3616}},"tokens_in":491,"tokens_out":3690,"duration_ms":28420,"temperature":1.0,"reasoning_tokens":3616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:14:13.078980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators redraw the focus regions for a random sample of questions without being offered a preset catalog of segmentations; if a substantial fraction of questions gains a new plausible region that VQ-FocusAmbiguity does not list, the claim that the dataset captures all plausible focus regions is falsified.","supporting_citations":[{"cited_title":"Learning to detect a salient object","cited_arxiv_id":null,"evidence_quote":"supplies single-salient-object images whose one segmentation anchors the unambiguous examples."},{"cited_title":"Vqa therapy: Exploring answer differences by visually ground- ing answers","cited_arxiv_id":null,"evidence_quote":"provides the answer groundings used to compare question focus groundings with answer groundings."},{"cited_title":"Glamm: Pixel grounding large multimodal model","cited_arxiv_id":null,"evidence_quote":"serves as the state-of-the-art pixel-grounding baseline for the focus localization task."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"converts point prompts into segmentation masks in the Molmo+SAM localization pipeline."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"generates candidate questions for PACO examples and serves as the best-performing ambiguity recognition baseline."}],"review_version":1}