{"id":"f8b7921f-c52f-4eeb-80e1-46f45093d1ca","arxiv_id":"1908.04342","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors introduce a nine-reason taxonomy and two labeled datasets explaining why visual questions get different answers, and a multi-label predictor of those reasons.","lead":"Different people often give different answers to the same visual question. This paper creates a taxonomy of nine reasons why, labels about 45,000 questions accordingly, and trains a model to predict the reasons from the question and image.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern: the 'why' labels are noisy subjective annotations; the paper's own Appendix B.1 reports worker-worker similarity peaking at 0.5, and without a chance-corrected reliability check the dataset and model evaluation rest on unstable ground truth.","rationale":"The reader's weakest assumption is that the nine-reason taxonomy is complete and consistently applied by crowd workers. My stress-test pass converges on the same point, sharpened by the paper's own Appendix B.1 figures: worker-worker similarity peaks at 0.5 even on Cohen's kappa, and the model's ground truth is a 2-of-5 aggregation of these noisy labels. Because the paper's descriptive statistics and the model evaluation both use this ground truth, any instability in the labels propagates to the central claims. I considered whether a stronger concern is the weakness of the baselines, since QI-Relevance and Unanswerable only predict three labels and assign random values to the others. That is a real limitation, but the paper's own ablations (Q, Q+I, Q+I+A) provide a more meaningful comparison, and the claimed advantage over re-purposed baselines is not the only evidence offered. The most load-bearing issue remains label validity: if the reasons themselves are not reliably identifiable by humans, then no amount of model engineering can produce a trustworthy predictor of 'why.' The proposed concrete test—a fresh annotation pass on a sample with per-reason kappa and cross-label evaluation—would directly settle whether the taxonomy is consistently applicable and whether the model's gains survive independent labels. The reader's conditional verdict already reflects this risk, so I recommend no change to the verdict.","tokens_in":16489,"tokens_out":4004,"duration_ms":47546,"concrete_test":"Sample 1,000 VQAs from the released dataset, recruit a fresh set of five workers per VQA using the same interface, and compute per-reason Cohen's kappa and macro-F1 between the original 2-of-5 labels and the fresh 2-of-5 labels. In addition, re-evaluate the Q+I+A versus Q+I model by training on the original labels and testing on the fresh labels. If the kappa falls below 0.4 or if the Q+I+A advantage over Q+I reverses on fresh labels, the central 'why' claim and the model's reported advantage are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that it identifies why visual questions elicit different answers and that its model predicts those reasons. Both halves depend on the reliability of the nine-reason labels. Section 5.1 converts five crowd labels per VQA into ground truth using a 2-of-5 threshold, but the authors' own Appendix B.1 reports worker-worker similarity distributions peaking around 0.5 across three metrics, including Cohen's kappa. That means a typical worker agrees with only about half of their co-workers on which reasons apply. The paper reports no per-reason chance-corrected agreement, no confidence intervals, and no sensitivity analysis for the 2-of-5 threshold in the model evaluation. Figure 10 shows that shifting from a 1-person to a 5-person validity threshold changes the proportion of VQAs attributed to 'both QI and answers' from 93% to 31% for VizWiz, indicating that the aggregated labels are highly threshold-dependent. Because the taxonomy's completeness is also supported only by a 100-VQA pilot with an optional 'OTHER' option, the risk is not merely mild noise: the model may be learning to predict which labels crowd workers tend to assign, not which reasons actually cause answer differences. This concern is load-bearing because it undermines both the descriptive statistics in Section 4 and the relative model gains in Section 5.2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of explaining why different people provide different answers to the same visual question. It proposes a nine-category taxonomy (plus an 'Other' category), releases two annotated datasets of about 45,000 visual questions with five crowd-sourced reason labels each, and trains a multi-label classifier that predicts the reasons from the image, question, and predicted answers. The main empirical claims are that answer differences most often arise from ambiguity, synonyms, and granularity, and that the proposed Q+I+A model outperforms three related baselines on VizWiz and VQA 2.0.","tokens_in":16789,"tokens_out":4711,"duration_ms":45644,"significance":"If the annotation ground truth is trustworthy, the paper makes a useful contribution: it is the first large-scale attempt to characterize and predict the causes of answer disagreement in VQA, it ships public datasets and code, and its model design avoids label leakage by pretraining the answer-prediction module on a different dataset. The descriptive findings (AMB/SYN/GRN prevalence and co-occurrence structure) could inform answer aggregation and question-rewriting systems. The evaluation, however, rests on subjective labels whose reliability is not demonstrated, and the reported baselines are too weak to establish the model's advantage.","major_comments":[{"comment":"The ground-truth labels are the load-bearing component of both the descriptive analysis and the model evaluation, but the reported inter-annotator agreement does not establish reliability. The WWS distributions in Figures 7-9 peak at 0.5 on all three metrics, and the paper reports no chance-corrected per-label agreement (e.g., Cohen's kappa per reason or Krippendorff's alpha). Since Section 5.1 binarizes each label with a 2-of-5 threshold, a typical worker agreeing with only half of co-workers means many labels are determined by near-tie votes. The threshold sensitivity is visible in Appendix Figure 10: the proportion of VizWiz VQAs attributed to 'QI & A' drops from 93% (1-person) to 31% (5-person), and the analogous proportion for VQA 2.0 drops from 80% to 8%. The paper should report per-reason chance-corrected agreement, the distribution of vote counts, and a sensitivity analysis showing whether the conclusions in Section 4 and the relative gains in Section 5.2 survive alternative validity thresholds (e.g., 3-of-5).","section":"Appendix B.1 and Section 3"},{"comment":"The claim that the proposed model outperforms baselines is not yet fully supported because the baselines are weak and no uncertainty is reported. QI-Relevance and Unanswerable predict only three of the ten labels (LQI, IVE, AMB) and set the remaining seven labels randomly, so the comparison measures little more than whether the model can predict those three labels plus chance. In addition, the table reports a single average-precision number per label with no error bars, significance tests, or multiple seeds, although some differences between model variants are below one point (e.g., VizWiz overall 50.02 vs. 50.01 for Q+I+A vs. Q+I+A FT). The authors should add (i) stronger baselines trained on the same task with the same input modalities (e.g., a multi-label classifier on Q+I, or predicted answers without the answer-difference head), (ii) repeated-run means with standard deviations, and (iii) a statement of how many random-restart seeds were used.","section":"Section 5.2, Table 2"}],"minor_comments":[{"comment":"The sentence about difficult questions reverses the dataset order: it says the percentage is four times higher in VQA 2.0 than VizWiz, then gives '~3% versus ~12%', which should be 'VizWiz ~3% versus VQA 2.0 ~12%'.","section":"Section 4.1"},{"comment":"The abbreviation for the 'Difficult' reason is 'DFF' in Table 2 but 'DEF' in Table 3; please use a consistent abbreviation.","section":"Table 3"},{"comment":"The histograms have axis labels and percentage values that are too small to read; consider adding a numeric table or using larger fonts and clearer legends.","section":"Figures 3 and 11"},{"comment":"References [39] and [40] are duplicate entries for the same Teney et al. CVPR 2018 paper; one should be removed and the citations merged.","section":"References"},{"comment":"The 'Random' baseline should specify whether it samples uniformly from the ten labels or uses the training-label prior; the very high AP for AMB (95.19 on VQA 2.0) suggests the latter, and this should be stated explicitly.","section":"Section 5.2"},{"comment":"The 'Other (OTH)' category is described in the labeling interface and is predicted as a tenth label in Section 5, but it is missing from Table 1; include it in the taxonomy table for consistency.","section":"Section 3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the reliability of the reason labels; the paper's own Appendix B.1 is candid about the low agreement, and the threshold analysis in Figure 10 shows how much the conclusions move with the validity threshold. I would like to see per-label chance-corrected agreement and a threshold sensitivity analysis for the model evaluation before accepting. The 'first work' claim is plausible but should be checked against prior work on answer diversity and annotation disagreement, such as [18] and [45]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. The paper does something genuinely useful: it enumerates nine reasons why people give different answers to the same visual question, labels ~45k examples from VizWiz and VQA 2.0 with which reasons apply, and defines a new prediction task for anticipating those reasons. The taxonomy is well grounded in prior crowdsourcing and VQA literature, and the datasets are a real resource for anyone studying answer disagreement or building aggregation methods.\n\nThe paper is also transparent about its annotation quality. Appendix B.1 shows worker-worker similarity distributions for three metrics, peaks around 0.5, and the main text shows how prevalence numbers shift as you raise the validity threshold from one to five workers. That honesty is a strength. But it also exposes the soft spot: 0.5 agreement is moderate, and the paper does not report chance-corrected per-reason agreement. For a multi-label task with ten categories and strong base-rate differences, kappa is the natural check. The authors chose a 2-of-5 threshold for ground truth, which is defensible, but the evaluation would be stronger with per-label kappa and a sensitivity analysis for that threshold in the model results.\n\nThe model ablations are the most informative evidence. Adding predicted answers to question+image features improves mAP by 1.4% and 4.3% on VQA 2.0 and VizWiz. Those are modest gains, and the paper gives no error bars or significance tests, so I can't tell if the answer cue helps reliably. The baselines are weak by necessity — there is no prior work on this exact task — but comparing to random and re-purposed relevance detectors doesn't make the model look strong. Still, the model is not the main contribution; the taxonomy and datasets are.\n\nOn the stress-test note: the worry that noisy labels undermine the model is overstated. The labels are subjective, but that is exactly the phenomenon under study. With a clear protocol and a fixed threshold, predicting those labels is a legitimate target. The real gaps are statistical, not conceptual.\n\nWho should read this: VQA researchers, people working on annotation disagreement, and anyone designing systems that aggregate multiple answers. It deserves a serious peer review; a good referee would push for per-label agreement numbers, confidence intervals, and stronger baselines, but the work is a solid, citable contribution.","headline":"A first useful taxonomy and dataset for why VQA answers differ; the model evaluation is honest but needs more statistical rigor.","tokens_in":17277,"tokens_out":3897,"would_cite":true,"duration_ms":39595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the reasons people give different answers to the same visual question can be captured by a nine-category taxonomy and predicted ahead of time from the image and the question.","keywords":["visual question answering","answer disagreement","annotation diversity","taxonomy","multi-label classification","crowdsourcing","VizWiz","VQA 2.0"],"falsifier":"Re-label a random sample of the same 44,955 questions with a fresh crowd using the identical interface; if the new workers' labels agree with the published majority labels no better than chance, or if raising the agreement threshold to four or five workers erases most positives, the taxonomy and its labels are too noisy to support the claimed predictions.","tokens_in":16324,"feed_emoji":"🤔","tokens_out":5819,"duration_ms":56589,"temperature":0.7,"pith_summary":"This paper tries to establish that answer diversity in visual question answering is not random noise: it can be characterized by a nine-reason taxonomy, measured at scale, and predicted from the visual question itself. The authors build two labeled datasets covering nearly 45,000 questions and train a multi-label classifier that, given an image, a question, and a model's predicted answers, outputs which reasons will drive disagreement. If the claim is right, VQA systems could warn users when a question is ambiguous or unanswerable, suggest how to rephrase or retake, and inform how to aggregate multiple answers. The paper reports that ambiguity, synonyms, and answer granularity are the dominant reasons, and that the model beats repurposed relevance and answerability baselines.","feed_headline":"Nine reasons explain why visual questions get different answers","feed_subtitle":"New labeled data and a model predict whether a question will trigger conflicting answers before anyone answers.","key_machinery":"The load-bearing machinery is the taxonomy plus a two-module model. The taxonomy is a fixed set of ten labels (nine reasons plus “other”) into which every disagreement is classified. The model's “Answer Prediction” module is a pretrained VQA network that produces a soft probability distribution over candidate answers; this distribution is concatenated with image features from a Faster R-CNN and question features from a GRU, and an “Answer Difference” module maps them through a fully connected layer to per-label sigmoid probabilities. The predicted-answer signal does the crucial work of letting the model exploit answer-level causes (synonyms, granularity, spam) without knowing the true answers at test time.","core_discovery":"The authors propose that every observed disagreement over a visual question can be assigned to one or more of nine reasons: low-quality image, answer not present, invalid question, difficult question, ambiguous question, subjective question, synonymous answers, granularity differences, and spam. They label 44,955 questions from two existing VQA datasets with these reasons and show that most questions receive several reasons, with ambiguity, synonyms, and granularity the most common. They then pose the new task of predicting the reasons from the visual question alone, and introduce a model that adds a pretrained answer-prediction module so that predicted answers, not just image and question, serve as cues. On both datasets the model improves mean average precision over random, relevance, and answerability baselines, with predicted answers contributing a consistent gain over image-plus-question alone.","pith_inferences":["A likely refinement the paper leaves implicit: because ambiguity, synonyms, and granularity co-occur so strongly, a structured model that treats them as a shared “imprecision” factor would probably be more robust than ten independent binary labels.","The reported worker-agreement distributions peaking near 0.5 suggest the ground-truth labels contain substantial noise; an evaluation using only 4- or 5-worker consensus labels would show how much of the predictive gain survives stricter supervision.","The same taxonomy and prediction strategy could transfer to other tasks where annotators disagree, such as image captioning and visual dialog, turning disagreement from a nuisance into a prediction target.","A natural follow-up is to use the predicted reasons to weight or reconcile answers during aggregation, not just to warn users."],"forward_implications":["VQA services can decide before answering whether a user should rephrase the question, retake the image, or expect conflicting answers.","The dominant role of ambiguity, synonyms, and granularity means most disagreements could be headed off by disambiguating questions and aligning answer detail level.","Directly predicting the reason labels outperforms adapting existing relevance or answerability models, so the new task needs its own modeling.","Feeding a VQA model's predicted answer distribution into the predictor improves accuracy beyond image-plus-question, especially for low-quality images and missing evidence."],"supporting_citations":[{"why":"Shows that different answers are common across hundreds of thousands of visual questions, motivating the need to explain them.","marker":"[18]"},{"why":"Supplies the VizWiz dataset and the “unanswerable question” baseline reused in evaluation.","marker":"[20]"},{"why":"Supplies the VQA 2.0 sample used as the second dataset.","marker":"[17]"},{"why":"Is the question-image relevance baseline that the new model must outperform.","marker":"[30]"},{"why":"Provides the pretrained VQA network used inside the Answer Prediction module.","marker":"[4]"},{"why":"Motivates using soft-target answer scores as a signal for uncertainty around a single answer.","marker":"[39]"},{"why":"Provides the GloVe word embeddings that initialize the question encoder.","marker":"[34]"},{"why":"Faster R-CNN features provide the image representation.","marker":"[35]"}],"fun_headline_variants":["Why do visual questions get multiple answers? Nine reasons","Predicting why visual questions spark conflicting answers","Nine-way taxonomy predicts visual question answer conflicts","First study: nine reasons behind VQA answer disagreements"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nine-reason taxonomy is complete enough and crowd workers apply the labels consistently enough that the ground-truth labels reflect real causes rather than arbitrary annotator choices.","fun_headline_variants_meta":{"raw":{"variants":["Why do visual questions get multiple answers? Nine reasons","Predicting why visual questions spark conflicting answers","Nine-way taxonomy predicts visual question answer conflicts","First study: nine reasons behind VQA answer disagreements"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2550,"prompt_tokens":806,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":1685}},"tokens_in":422,"tokens_out":1744,"duration_ms":13394,"temperature":1.0,"reasoning_tokens":1685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:44:03.558086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label a random sample of the same 44,955 questions with a fresh crowd using the identical interface; if the new workers' labels agree with the published majority labels no better than chance, or if raising the agreement threshold to four or five workers erases most positives, the taxonomy and its labels are too noisy to support the claimed predictions.","supporting_citations":[{"cited_title":"CrowdVerge: Predict- ing If People Will Agree on the Answer to a Visual Question","cited_arxiv_id":null,"evidence_quote":"Shows that different answers are common across hundreds of thousands of visual questions, motivating the need to explain them."},{"cited_title":"Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P","cited_arxiv_id":null,"evidence_quote":"Supplies the VizWiz dataset and the “unanswerable question” baseline reused in evaluation."},{"cited_title":"Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering","cited_arxiv_id":null,"evidence_quote":"Supplies the VQA 2.0 sample used as the second dataset."},{"cited_title":"The promise of premise: Harnessing question premises in visual question answering","cited_arxiv_id":null,"evidence_quote":"Is the question-image relevance baseline that the new model must outperform."},{"cited_title":"Bottom-up and top-down attention for image captioning and visual question answering","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained VQA network used inside the Answer Prediction module."},{"cited_title":"Tips and tricks for visual question an- swering: Learnings from the 2017 challenge","cited_arxiv_id":null,"evidence_quote":"Motivates using soft-target answer scores as a signal for uncertainty around a single answer."},{"cited_title":"Glove: Global vectors for word representation","cited_arxiv_id":null,"evidence_quote":"Provides the GloVe word embeddings that initialize the question encoder."},{"cited_title":"Faster r-cnn: Towards real-time object detection with region proposal networks","cited_arxiv_id":null,"evidence_quote":"Faster R-CNN features provide the image representation."}],"review_version":1}