{"id":"1f41da86-1d10-473a-9102-5445084a0e64","arxiv_id":"2411.11150","paper_version":1,"verdict":"UNVERDICTED","confidence":"MODERATE","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A broad but dated survey of VQA datasets and algorithms that organizes the pre-2021 literature into four dataset categories and six model paradigms.","lead":"This paper surveys visual question answering, grouping datasets into four types and models into six algorithmic families. It is a map for newcomers but contains little new analysis and mostly stops at 2020, before the current generation of large multimodal models.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy tables mix incompatible evaluation protocols and contain internal errors, so the survey's empirical map is not verifiable as presented.","rationale":"The reader's weakest-assumption analysis is on target: the reproduced numbers are not mutually comparable. My stress-test adds a sharper, internal-failure observation: even the dataset statistics table contains a demonstrable error (SHAPES row in Table 2), which means the survey's empirical layer cannot be taken as carefully curated. This matters because the paper's own purpose is to give newcomers 'a map and a starting point' (Conclusion), and a map whose coordinate labels are wrong or unverifiable does not serve that purpose. The algorithmic sections are largely sound: the bilinear-pooling equations (2)-(7) are standard and correctly described, and the taxonomy of attention schemes in Tables 6-8 is a reasonable organization. However, the accuracy tables are the only quantified evidence that the field has been 'meticulously analyze[d]', and they do not meet the standard of reproducibility implied by that claim. I would not adjust the reader's UNVERDICTED verdict: a survey with no falsifiable research claim is not 'accepted' or 'rejected' on correctness grounds, but the quality concerns are real. The proposed audit is a small, decisive check: if the per-row protocols disagree, the comparative columns should be revised or removed.","tokens_in":32227,"tokens_out":3896,"duration_ms":37132,"concrete_test":"Perform a per-row reproducibility audit of Table 10: for each of the twelve PGE models, fetch the original paper or official repository and record (1) which VQA-v2 split and evaluation mode (open-ended vs. multiple-choice) produced the cited accuracy, (2) which CLEVR split was used, and (3) the exact metric. If any row's score comes from a different split, answer vocabulary, or metric than the rows it is compared with, the table's ranking is not meaningful and the survey should either add per-entry protocol columns or remove the comparative accuracy cells. Run the same audit on at least one row of Tables 5, 9, 12, and 14 to confirm whether the problem is systematic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central offering is a reliable map of VQA datasets and models, and the reproduced accuracy tables (Tables 5, 9, 10, 12, 14) are the empirical backbone of that map. The load-bearing assumption is that the rows are representative and mutually comparable. That assumption fails in two concrete ways. First, the tables do not state, for each entry, which test split (train/val/test-dev/test-standard), answer vocabulary (open-ended vs. multiple-choice), or evaluation metric (VQA accuracy, exact match, WUPS) produced the number. Since VQA-v2 scores can differ by several points between open-ended and multiple-choice settings and between test-dev and test-standard, the implied ordering across rows is not reliable. For example, Table 10 compares PGE models on VQA-v2 (N2NMN 63.3, Stack-NMN 64.0, XNM 67.5) with no protocol column; if any of these numbers comes from a different split or answer mode, the comparison is invalid. Second, the tables contain internal errors that undermine their authority: Table 2 lists SHAPES as having 15.5K images, but SHAPES has 64 images; the QA-pair count is also misprinted as 15.5K instead of 15,616. The reference list also has duplicates ([4]=[5], [30]=[31]) and effectively stops at 2020, which contradicts the abstract's 'current state' claim. A survey can be a valuable organization even with these issues, but the empirical claims cannot be verified from the paper as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript surveys visual question answering (VQA) datasets and algorithms. It proposes a four-way dataset taxonomy (general, synthetic, diagnostic, knowledge-based) and a six-way model taxonomy (fusion, attention, external knowledge, compositional reasoning, explanation, graph), with additional sections on transformer-based vision-language pre-training, scene text understanding, counting, and bias reduction. The paper reproduces accuracy tables for representative models, describes evaluation metrics, and discusses dataset biases. The intended contribution is an entry-level map of the VQA field.","tokens_in":32499,"tokens_out":5781,"duration_ms":52531,"significance":"If its taxonomy and summaries are reliable, the survey would give newcomers a useful orientation to pre-2021 VQA research. The breadth is genuine: it covers datasets from COCO-QA to GQA and models from bilinear pooling to early transformer pre-training, and it correctly identifies key cross-cutting issues such as language priors and evaluation-metric heterogeneity. The survey does not present formal derivations or code, which is appropriate for its genre. Its value is organizational. However, that value is currently undercut by three fixable problems: the reference list effectively stops in 2020 despite the 'current state' claim, the accuracy tables lack evaluation-protocol metadata and contain an internal error (SHAPES image count), and the bibliography contains several duplicates. These issues make the empirical map hard to verify as written.","major_comments":[{"comment":"The abstract and conclusion describe the survey as covering the 'current state' of VQA, but the reference list contains almost nothing after 2020 (the latest entries are 2021 and 2020). Given the arXiv submission date of November 2024, this omits major subsequent developments in vision-language models (e.g., CLIP-based zero-shot VQA, ViLT, ALBEF, BLIP, Flamingo, LLaVA) and newer benchmarks. The survey should either explicitly scope itself to work published through approximately 2020 or substantially update the coverage. As written, the comprehensiveness claim is not met.","section":"Abstract, Section XII, References"},{"comment":"In the SHAPES row of Table 2, the number of images is listed as '15.5K' and the QA-pair count as '15.5K'. The body text (Section II.B.3) correctly states that SHAPES has 64 images and 15,616 questions. This is an internal contradiction in a table that is central to the survey's empirical summary. Please correct the table and check all other rows against their sources.","section":"II.B.3, Table 2"},{"comment":"The accuracy tables report single numbers per model without stating the evaluation protocol used in the source paper. For VQA-v2, scores can differ by several points between open-ended and multiple-choice settings and between test-dev and test-standard; for CLEVR, exact-match accuracy is sensitive to the question split. Consequently, the tables imply cross-model comparisons (e.g., Table 10's VQA-v2 column) that cannot be verified from the information given. Add a protocol column (split, answer mode, metric) or state explicitly that the numbers are aggregated from heterogeneous sources and warn readers against direct ranking.","section":"III.D, Tables 5, 9, 10, 12, 14"},{"comment":"The reference list contains duplicates: [4] and [5] are identical entries for Agrawal et al., 'Don't just assume; look and answer'; [30] and [31] are identical for Das et al., 'Human attention in visual question answering'; and [56] and [57] are identical for Park et al., 'Multimodal explanations'. These duplicates should be collapsed and the in-text citations renumbered accordingly.","section":"References"}],"minor_comments":[{"comment":"The text states that VQA-Rephrasings provides '3 human-provided rephrasings for 40k questions', while Table 3 lists 40K images and 160K QA pairs; please clarify whether the 160K includes the original questions or whether the rephrasings number 120K.","section":"II.C.3, Table 3"},{"comment":"The section title contains a typo: 'Tranformer' should be 'Transformer'.","section":"X"},{"comment":"The column header 'binay' should be 'binary'.","section":"Table 2"},{"comment":"In the TDIUC paragraph, 'In previous models, doing well on some classes of questions was rewarded more than others' should refer to 'In previous datasets' rather than models.","section":"II.A.6"},{"comment":"The dataset is called 'C-VQA' in the text and Table 3, but the Figure 6 caption reads 'An example from CVQA'; please standardize the spelling.","section":"II.C.1, Figure 6"},{"comment":"Equation (1) does not define the variable n; please state that n is the number of annotators who provided the predicted answer.","section":"II.E.3"},{"comment":"The spelling 'Y AGO' should be corrected to 'YAGO'.","section":"VI"}],"recommendation":"major_revision","confidential_remarks":"The survey has a clear organizational value for newcomers, but the empirical backbone needs careful curation before it can be trusted. The duplicate references and the SHAPES row error suggest that the data tables and bibliography were not systematically checked against their primary sources. The journal should also consider whether the lack of post-2020 coverage, despite the 'current state' framing, is acceptable for a survey published in 2024 or whether a scope statement or update is required."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a survey, so judge it as one: no new datasets, models, or results, and the four-category dataset split and six model paradigms are an editorial reorganization of distinctions the field already makes - CLEVR as a diagnostic compositional dataset, VQA-CP as a bias test, fusion versus attention versus knowledge. That acknowledged, the organizational work is mostly done well. The prose descriptions of individual datasets and models are faithful to the sources. The fusion section gets MCB, MLB, MFB, MUTAN, and BLOCK right, including the equations. The evaluation-metrics section, covering consensus scoring, WUPS, TDIUC's per-type metrics, and GQA's consistency/validity/plausibility checks, is a genuinely useful collection that is hard to find in one place. The conclusion honestly frames the paper as a bird's-eye view rather than a deep dive. A newcomer would get an accurate map of roughly the 2015-2020 VQA literature.\n\nThe soft spots are real, and one is structural. The reference list stops almost entirely at 2020, yet the paper was posted in November 2024 and the abstract claims to analyze \"the current state\" of VQA. No CLIP, no BLIP, no LLaVA, no ViLT, no OFA. That gap is not minor for a survey with a currency claim. Second, the accuracy tables (Tables 5, 9, 10, 12, 14) mix numbers from different test splits, answer vocabularies, and evaluation metrics, without stating which protocol produced each row. The implied cross-model ordering is not verifiable from the paper as written; the stress-test note is correct on this point. Third, there are internal inconsistencies that should have been caught: Table 2 lists SHAPES as having 15.5K images when the text of the same paper says 64 images, and references [4]=[5] plus [30]=[31] are exact duplicates. None of this breaks the taxonomy, but it undercuts the paper's claim of careful curation.\n\nWho this is for: a graduate student or a researcher from a neighboring field who wants a first map of pre-2021 VQA. Not for anyone tracking the current frontier. The descriptions are honest, there is no circular reasoning or invented content, and the flaws are sloppiness rather than misleading argument.\n\nRecommendation: send to peer review rather than desk-reject. The descriptive core and the taxonomy are sound enough to become a serviceable introductory survey after major revision - bring the bibliography up to date, add protocol columns to the tables or cut them, and fix the internal inconsistencies. A referee should be told to check the tables and the reference list carefully.","headline":"A serviceable but dated survey: accurate dataset and model descriptions and a sound taxonomy for pre-2021 VQA, undermined by a stale bibliography and accuracy tables that mix incompatible evaluation protocols.","tokens_in":32981,"tokens_out":5365,"would_cite":false,"duration_ms":43687,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims that the entire pre-2021 VQA field can be organized into four dataset families and six model paradigms, and that this map is enough for a newcomer to understand the field's main techniques and biases.","keywords":["visual question answering","dataset taxonomy","multimodal fusion","attention mechanisms","compositional reasoning","external knowledge","evaluation metrics","survey"],"falsifier":"A reader could re-run the transformer models listed in Table 14 on the same VQA-v2 split under identical training conditions; if the reported ordering does not reproduce, the survey's comparative scoreboard is not trustworthy. A broader test would apply the four-dataset/six-model taxonomy to a random sample of VQA papers published between 2015 and 2020 and count how many fit exactly one cell.","tokens_in":32046,"feed_emoji":"🗺️","tokens_out":6445,"duration_ms":57417,"temperature":0.7,"pith_summary":"The paper is a survey of visual question answering (VQA), the task of answering a natural-language question about an image. It argues that, despite the field's rapid growth, VQA research can be cleanly divided into four kinds of datasets (general, synthetic, diagnostic, and knowledge-based) and six model paradigms (fusion, attention, external knowledge, composition and reasoning, explanation, and graph models). The paper reproduces accuracy tables for representative models in each paradigm and uses them to show which techniques advanced the field and where biases still distort conclusions. A sympathetic reader would take away a structured map of pre-2021 VQA work, including the recurring lesson that raw accuracy numbers are misleading when datasets carry language priors and when evaluation protocols differ.","feed_headline":"Survey maps visual question answering into four datasets, six models","feed_subtitle":"A newcomer's map of pre-2021 VQA: which benchmarks test what, and which model tricks actually matter.","key_machinery":"The load-bearing object is the organizing taxonomy: four dataset categories (general, synthetic, diagnostic, knowledge-based) crossed with six model paradigms (fusion, attention, external knowledge, composition and reasoning, explanation, graph). This classification does the argument's work by assigning each dataset and model a cell, and the reproduced accuracy tables give each paradigm a comparative scoreboard. The paper's own treatment of bilinear pooling's count-sketch identity and of attention variants is secondary; those technical details illustrate cells rather than carry the survey's claim.","core_discovery":"The central claim is that the state of VQA research can be described by a two-axis taxonomy: every commonly used benchmark falls into one of four dataset categories, and every notable model falls into one of six algorithmic paradigms. The paper's contribution is the taxonomy itself, applied consistently across the field, plus the reproduced accuracy tables that let a reader compare approaches and see which techniques advanced accuracy under which conditions. The paper also emphasizes that accuracy alone is misleading, because datasets such as VQA-v1 and VQA-v2 contain language priors and skewed answer distributions, and because evaluation metrics differ across datasets.","pith_inferences":["As an editorial extension, the taxonomy's 2020 boundary means it would need revision for today's large vision-language models, many of which combine fusion, attention, external knowledge, and explanation in a single transformer; applying the taxonomy to them would likely assign one system to several paradigms at once.","As an editorial extension, the paper's own caveat about mixed evaluation protocols suggests a concrete test: re-running the models in Tables 5, 9, 10, 12, and 14 under one shared protocol would show whether the reported orderings are reproducible.","As an editorial extension, the absence of an agreed evaluation metric points to standardization as the field's likely next bottleneck: a unified score combining exact match, consistency, and plausibility would make future cross-paper comparisons meaningful."],"forward_implications":["A newcomer can locate any pre-2021 VQA dataset or model within the taxonomy and see which family it belongs to and which techniques it exemplifies.","The accuracy tables show a progression: transformer-based models top the VQA-v2 leaderboard, while program-generator models approach perfect accuracy on CLEVR, implying that the field's best results came from pre-training and from explicit reasoning.","The survey's bias analysis implies that models scoring high on VQA-v1 and VQA-v2 may be exploiting language priors, so evaluations should report per-question-type and normalized scores whenever possible.","The survey's coverage of explanation, counting, and scene-text tasks indicates that these are distinct subproblems requiring specialized training signals, such as OCR modules, counting graphs, and explanation annotations.","Because the survey stresses that no widely agreed evaluation metric exists, it implies that cross-paper accuracy comparisons should be read with caution and that the field would benefit from a shared protocol."],"supporting_citations":[{"why":"Defines the standard VQA task with real images and human annotators, and documents the language-prior biases the survey analyzes.","marker":"[10]"},{"why":"The balanced follow-up that pairs complementary images with different answers to force image grounding.","marker":"[42]"},{"why":"The synthetic benchmark with functional program annotations that anchors the compositional reasoning section.","marker":"[60]"},{"why":"A real-image dataset combining scene graphs with functional programs and the five evaluation metrics used to assess reasoning.","marker":"[54]"},{"why":"Introduces per-question-type evaluation metrics and the absurd-question protocol that later diagnostic work builds on.","marker":"[62]"},{"why":"Establishes bottom-up/top-down attention with object-detector features, a baseline for many attention and graph models.","marker":"[7]"},{"why":"The knowledge-based benchmark used to define the KB category and to motivate external-knowledge models.","marker":"[82]"},{"why":"Representative of the transformer paradigm, contributing the masked-region and image-text matching pre-training tasks.","marker":"[117]"},{"why":"Provides the blackbox compositional reasoning baseline on CLEVR that the survey tables use for comparison.","marker":"[53]"},{"why":"The rearrangement of VQA splits into compositional train and test sets, used as a diagnostic for bias and generalization.","marker":"[4]"}],"fun_headline_variants":["VQA field mapped into 4 dataset types and 6 model paradigms","Four dataset families, six algorithm paradigms: VQA survey","Survey taxonomizes VQA: 4 benchmark categories, 6 model designs","Visual question answering: 4 dataset kinds, 6 model approaches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey assumes that the cited papers and the accuracy numbers copied from them are representative of the field and directly comparable, even though the tables mix different backbones, answer vocabularies, and evaluation protocols.","fun_headline_variants_meta":{"raw":{"variants":["VQA field mapped into 4 dataset types and 6 model paradigms","Four dataset families, six algorithm paradigms: VQA survey","Survey taxonomizes VQA: 4 benchmark categories, 6 model designs","Visual question answering: 4 dataset kinds, 6 model approaches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1472,"prompt_tokens":963,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":579,"tokens_out":509,"duration_ms":29057,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:50:56.961090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could re-run the transformer models listed in Table 14 on the same VQA-v2 split under identical training conditions; if the reported ordering does not reproduce, the survey's comparative scoreboard is not trustworthy. A broader test would apply the four-dataset/six-model taxonomy to a random sample of VQA papers published between 2015 and 2020 and count how many fit exactly one cell.","supporting_citations":[],"review_version":1}