Pith. sign in

REVIEW 4 major objections 7 minor 3 cited by

A Comprehensive Survey on Visual Question Answering Datasets and Algorithms

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey claims that the entire pre-2021 VQA field can be organized into four dataset families and six model paradigms, and that this map is enough for a newcomer to understand the field's main techniques and biases.

desk verdict A serviceable but dated survey: accurate dataset and model descriptions and a sound taxonomy for pre-2021 VQA, undermined by a stale bibliography and accuracy tables that mix incompatible evaluation protocols. read the letter →

arxiv 2411.11150 v1 pith:H2BWFATF submitted 2024-11-17 cs.CV

Raihan Kabir , Naznin Haque , Md Saiful Islam , Marium-E-Jannat This is my paper · ORCID
classification cs.CV
keywords visualquestionansweringdatasettaxonomymultimodalfusionattentionmechanismscompositionalreasoningexternalknowledgeevaluationmetricssurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a survey of visual question answering (VQA), the task of answering a natural-language question about an image. It argues that, despite the field's rapid growth, VQA research can be cleanly divided into four kinds of datasets (general, synthetic, diagnostic, and knowledge-based) and six model paradigms (fusion, attention, external knowledge, composition and reasoning, explanation, and graph models). The paper reproduces accuracy tables for representative models in each paradigm and uses them to show which techniques advanced the field and where biases still distort conclusions. A sympathetic reader would take away a structured map of pre-2021 VQA work, including the recurring lesson that raw accuracy numbers are misleading when datasets carry language priors and when evaluation protocols differ.

What carries the argument

The load-bearing object is the organizing taxonomy: four dataset categories (general, synthetic, diagnostic, knowledge-based) crossed with six model paradigms (fusion, attention, external knowledge, composition and reasoning, explanation, graph). This classification does the argument's work by assigning each dataset and model a cell, and the reproduced accuracy tables give each paradigm a comparative scoreboard. The paper's own treatment of bilinear pooling's count-sketch identity and of attention variants is secondary; those technical details illustrate cells rather than carry the survey's claim.

What would settle it

A reader could re-run the transformer models listed in Table 14 on the same VQA-v2 split under identical training conditions; if the reported ordering does not reproduce, the survey's comparative scoreboard is not trustworthy. A broader test would apply the four-dataset/six-model taxonomy to a random sample of VQA papers published between 2015 and 2020 and count how many fit exactly one cell.

Watch

Extended reading notes

Core claim

The central claim is that the state of VQA research can be described by a two-axis taxonomy: every commonly used benchmark falls into one of four dataset categories, and every notable model falls into one of six algorithmic paradigms. The paper's contribution is the taxonomy itself, applied consistently across the field, plus the reproduced accuracy tables that let a reader compare approaches and see which techniques advanced accuracy under which conditions. The paper also emphasizes that accuracy alone is misleading, because datasets such as VQA-v1 and VQA-v2 contain language priors and skewed answer distributions, and because evaluation metrics differ across datasets.

Load-bearing premise

The survey assumes that the cited papers and the accuracy numbers copied from them are representative of the field and directly comparable, even though the tables mix different backbones, answer vocabularies, and evaluation protocols.

Editorial extensions

If this is right

  • A newcomer can locate any pre-2021 VQA dataset or model within the taxonomy and see which family it belongs to and which techniques it exemplifies.
  • The accuracy tables show a progression: transformer-based models top the VQA-v2 leaderboard, while program-generator models approach perfect accuracy on CLEVR, implying that the field's best results came from pre-training and from explicit reasoning.
  • The survey's bias analysis implies that models scoring high on VQA-v1 and VQA-v2 may be exploiting language priors, so evaluations should report per-question-type and normalized scores whenever possible.
  • The survey's coverage of explanation, counting, and scene-text tasks indicates that these are distinct subproblems requiring specialized training signals, such as OCR modules, counting graphs, and explanation annotations.
  • Because the survey stresses that no widely agreed evaluation metric exists, it implies that cross-paper accuracy comparisons should be read with caution and that the field would benefit from a shared protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an editorial extension, the taxonomy's 2020 boundary means it would need revision for today's large vision-language models, many of which combine fusion, attention, external knowledge, and explanation in a single transformer; applying the taxonomy to them would likely assign one system to several paradigms at once.
  • As an editorial extension, the paper's own caveat about mixed evaluation protocols suggests a concrete test: re-running the models in Tables 5, 9, 10, 12, and 14 under one shared protocol would show whether the reported orderings are reproducible.
  • As an editorial extension, the absence of an agreed evaluation metric points to standardization as the field's likely next bottleneck: a unified score combining exact match, consistency, and plausibility would make future cross-paper comparisons meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This manuscript surveys visual question answering (VQA) datasets and algorithms. It proposes a four-way dataset taxonomy (general, synthetic, diagnostic, knowledge-based) and a six-way model taxonomy (fusion, attention, external knowledge, compositional reasoning, explanation, graph), with additional sections on transformer-based vision-language pre-training, scene text understanding, counting, and bias reduction. The paper reproduces accuracy tables for representative models, describes evaluation metrics, and discusses dataset biases. The intended contribution is an entry-level map of the VQA field.

Significance. If its taxonomy and summaries are reliable, the survey would give newcomers a useful orientation to pre-2021 VQA research. The breadth is genuine: it covers datasets from COCO-QA to GQA and models from bilinear pooling to early transformer pre-training, and it correctly identifies key cross-cutting issues such as language priors and evaluation-metric heterogeneity. The survey does not present formal derivations or code, which is appropriate for its genre. Its value is organizational. However, that value is currently undercut by three fixable problems: the reference list effectively stops in 2020 despite the 'current state' claim, the accuracy tables lack evaluation-protocol metadata and contain an internal error (SHAPES image count), and the bibliography contains several duplicates. These issues make the empirical map hard to verify as written.

major comments (4)
  1. [Abstract, Section XII, References] The abstract and conclusion describe the survey as covering the 'current state' of VQA, but the reference list contains almost nothing after 2020 (the latest entries are 2021 and 2020). Given the arXiv submission date of November 2024, this omits major subsequent developments in vision-language models (e.g., CLIP-based zero-shot VQA, ViLT, ALBEF, BLIP, Flamingo, LLaVA) and newer benchmarks. The survey should either explicitly scope itself to work published through approximately 2020 or substantially update the coverage. As written, the comprehensiveness claim is not met.
  2. [II.B.3, Table 2] In the SHAPES row of Table 2, the number of images is listed as '15.5K' and the QA-pair count as '15.5K'. The body text (Section II.B.3) correctly states that SHAPES has 64 images and 15,616 questions. This is an internal contradiction in a table that is central to the survey's empirical summary. Please correct the table and check all other rows against their sources.
  3. [III.D, Tables 5, 9, 10, 12, 14] The accuracy tables report single numbers per model without stating the evaluation protocol used in the source paper. For VQA-v2, scores can differ by several points between open-ended and multiple-choice settings and between test-dev and test-standard; for CLEVR, exact-match accuracy is sensitive to the question split. Consequently, the tables imply cross-model comparisons (e.g., Table 10's VQA-v2 column) that cannot be verified from the information given. Add a protocol column (split, answer mode, metric) or state explicitly that the numbers are aggregated from heterogeneous sources and warn readers against direct ranking.
  4. [References] The reference list contains duplicates: [4] and [5] are identical entries for Agrawal et al., 'Don't just assume; look and answer'; [30] and [31] are identical for Das et al., 'Human attention in visual question answering'; and [56] and [57] are identical for Park et al., 'Multimodal explanations'. These duplicates should be collapsed and the in-text citations renumbered accordingly.
minor comments (7)
  1. [II.C.3, Table 3] The text states that VQA-Rephrasings provides '3 human-provided rephrasings for 40k questions', while Table 3 lists 40K images and 160K QA pairs; please clarify whether the 160K includes the original questions or whether the rephrasings number 120K.
  2. [X] The section title contains a typo: 'Tranformer' should be 'Transformer'.
  3. [Table 2] The column header 'binay' should be 'binary'.
  4. [II.A.6] In the TDIUC paragraph, 'In previous models, doing well on some classes of questions was rewarded more than others' should refer to 'In previous datasets' rather than models.
  5. [II.C.1, Figure 6] The dataset is called 'C-VQA' in the text and Table 3, but the Figure 6 caption reads 'An example from CVQA'; please standardize the spelling.
  6. [II.E.3] Equation (1) does not define the variable n; please state that n is the number of annotators who provided the predicted answer.
  7. [VI] The spelling 'Y AGO' should be corrected to 'YAGO'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey makes no predictive claims, and its organizational taxonomy is not derived from a fitted parameter or a self-citation chain.

full rationale

This is a literature survey, not a derivation. Its load-bearing content is a proposed taxonomy of VQA datasets (general, synthetic, diagnostic, knowledge-based) and model paradigms (fusion, attention, external knowledge, composition, explanation, graph), plus reproduced accuracy tables. None of these claims is obtained by fitting a parameter to a subset of data and then predicting a closely related quantity; there are no equations whose output is defined by the paper's own inputs. The reproduced accuracy numbers are attributed to external papers (e.g., Tables 5, 9, 10, 12, and 14 cite [64], [142], [50], [91], etc.), so any comparability or error concern is a correctness or verifiability issue, not circularity. The paper cites no prior work by its own authors; the duplicated references [4]=[5] and [30]=[31] are bibliographic slips that do not carry an argument. The internal error in Table 2 (SHAPES image count) and the mixed evaluation protocols in the accuracy tables are empirical quality problems, not circular reasoning. The survey is self-contained as an organizational summary and does not reduce to its inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities appear; the paper contributes a taxonomy rather than a model or fitted quantities. The main epistemic load is carried by assumptions about taxonomy completeness, comparability of copied accuracy numbers, and representativeness of the reference list.

assumptions (3)
  • domain assumption The proposed four-category dataset taxonomy and six-paradigm model taxonomy are exhaustive and disjoint.
    The survey's organizing structure assumes every dataset and model fits one category, yet overlap exists, for example VQA-abstract is both synthetic and part of VQA-v1.
  • domain assumption The accuracy numbers copied from different source papers are comparable without adjustment.
    Tables 5, 9, 10, 11, 12, and 14 list scores from models with different backbones, hyperparameters, and evaluation metrics, with no note about protocol differences.
  • domain assumption The reference set, which ends almost entirely in 2020, represents the current state of VQA as of submission in 2024.
    The paper claims to analyze the current state of VQA but does not discuss dominant modern large multimodal models such as LLaVA, BLIP-2, InstructBLIP, or GPT-4V.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comprehensive Survey on Visual Question Answering Datasets and Algorithms." pith.science (2026). https://pith.science/paper/H2BWFATF

@misc{pith2026241111150,
  author       = {Pith},
  title        = {Pith review of: A Comprehensive Survey on Visual Question Answering Datasets and Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2BWFATF}},
  note         = {Machine review of arXiv:2411.11150}
}
read the original abstract

Visual question answering (VQA) refers to the problem where, given an image and a natural language question about the image, a correct natural language answer has to be generated. A VQA model has to demonstrate both the visual understanding of the image and the semantic understanding of the question, demonstrating reasoning capability. Since the inception of this field, a plethora of VQA datasets and models have been published. In this article, we meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We divide VQA datasets into four categories: (1) available datasets that contain a rich collection of authentic images, (2) synthetic datasets that contain only synthetic images produced through artificial means, (3) diagnostic datasets that are specially designed to test model performance in a particular area, e.g., understanding the scene text, and (4) KB (Knowledge-Based) datasets that are designed to measure a model's ability to utilize outside knowledge. Concurrently, we explore six main paradigms of VQA models: fusion, where we discuss different methods of fusing information between visual and textual modalities; attention, the technique of using information from one modality to filter information from another; external knowledge base, where we discuss different models utilizing outside information; composition or reasoning, where we analyze techniques to answer advanced questions that require complex reasoning steps; explanation, which is the process of generating visual and textual descriptions to verify sound reasoning; and graph models, which encode and manipulate relationships through nodes in a graph. We also discuss some miscellaneous topics, such as scene text understanding, counting, and bias reduction.

Figures

Figures reproduced from arXiv: 2411.11150 by the authors.

Figure 1
Figure 1. FIGURE 1: Examples of VQA from the VQA-v2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2: An example from VQA-v1 [10] [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. FIGURE 4: An example image(a) from visual genome [66] [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: FIGURE 5: An example from CLEVR. Figure from [60] [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 8
Figure 8. Figure 8: FIGURE 8: An example from ST-VQA. Figure from [15] [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 7
Figure 7. Figure 7: FIGURE 7: An example from ConVQA. Figure from [100] [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: FIGURE 9: An example from FVQA. Figure from [129] [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: FIGURE 10: Basic approach of VQA algorithms [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: FIGURE 12: Mask R-CNN. Figure from [45] [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 11
Figure 11. Figure 11: FIGURE 11: CNN(left) and Faster R-CNN(right). Figure [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 13
Figure 13. Figure 13: FIGURE 13: First row presents the original images. Second row presents grid-based soft attention using CNNs. Third row [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: FIGURE 14: Multi-step attention through multiple attention [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: FIGURE 15: Co-attention where both image and question guide attention on each-other. The rows present co-attention for [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 17
Figure 17. Figure 17: FIGURE 17: An example structure of an EKB. Figure from [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: FIGURE 18: An example of a tree parsed from a question. [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: FIGURE 19: Example of both visual explanation and textual [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.

  2. Explain Before You Answer: A Survey on Compositional Visual Reasoning

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A survey that classifies compositional visual reasoning methods into five stages, from prompt-based pipelines to unified agentic vision-language models, and catalogs associated benchmarks.

  3. Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling

    cs.CV 2025-07 conditional novelty 4.0 of 10

    On the SciVQA 2025 benchmark, an InternVL3 model with optimized prompts and chain-of-thought instructions reaches ROUGE-1 and ROUGE-L F1 of 0.740, and a figure-type-aware ensemble ranks 5th.

Reference graph

Works this paper leans on

150 extracted references · 41 canonical work pages · cited by 3 Pith papers

  1. [5]

    Don’t just assume; look and answer: Overcoming priors for visual question answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kemb- havi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971–4980, 2018

    2018

  2. [31]

    Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90–100, 2017

    Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90–100, 2017

    2017

  3. [57]

    Multimodal expla- nations: Justifying decisions and pointing to the evidence

    Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal expla- nations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779–8788, 2018

    2018

  4. [1]

    Tallyqa: Answer- ing complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answer- ing complex counting questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8076–8084, 2019

    2019

  5. [2]

    Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing

    Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9690–9698, 2020

    2020

  6. [3]

    Analyzing the behavior of visual question answering models

    Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv:1606.07356, 2016

    arXiv 2016

  7. [6]

    C-vqa: A compositional split of the visual question answering (vqa) v1

    Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset. arXiv preprint arXiv:1704.08243, 2017

    arXiv 2017

  8. [7]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark John- son, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018

    2018

Show all 150 references
  1. [8]

    Learn- ing to compose neural networks for question answering

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learn- ing to compose neural networks for question answering. arXiv preprint arXiv:1601.01705, 2016

    2016 arXiv

  2. [9]

    Neural module networks

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  3. [10]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  4. [11]

    Dbpedia: A nucleus for a web of open data

    Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722–735. Springer, 2007

    2007

  5. [12]

    Open information extraction from the web

    Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broad- head, and Oren Etzioni. Open information extraction from the web. In Ijcai, volume 7, pages 2670–2676, 2007

    2007

  6. [13]

    Mutan: Multimodal tucker fusion for visual question answering

    Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2612–2620, 2017

    2017

  7. [14]

    Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection

    Hedi Ben-Younes, Rémi Cadene, Nicolas Thome, and Matthieu Cord. Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection. arXiv preprint arXiv:1902.00038, 2019

    1902 arXiv

  8. [15]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. arXiv preprint arXiv:1905.13648, 2019

    1905 arXiv

  9. [16]

    Freebase: a collaboratively created graph database for structuring human knowledge

    Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD interna- tional conference on Management of data, pages 1247–1250. AcM, 2008

    2008

  10. [17]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  11. [18]

    Top-down versus bottom- up control of attention in the prefrontal and posterior parietal cortices

    Timothy J Buschman and Earl K Miller. Top-down versus bottom- up control of attention in the prefrontal and posterior parietal cortices. science, 315(5820):1860–1862, 2007

    2007

  12. [19]

    Murel: Multimodal relational reasoning for visual question answering

    Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1989–1998, 2019

    1989

  13. [20]

    Rubi: Reducing unimodal biases in visual question answering

    Remi Cadene, Corentin Dancette, Hedi Ben-Younes, Matthieu Cord, and Devi Parikh. Rubi: Reducing unimodal biases in visual question answering. arXiv preprint arXiv:1906.10169, 2019

    1906 arXiv

  14. [21]

    Interpretable visual question answering by reasoning on dependency trees

    Qingxing Cao, Xiaodan Liang, Bailin Li, and Liang Lin. Interpretable visual question answering by reasoning on dependency trees. arXiv preprint arXiv:1809.01810, 2018

    2018 arXiv

  15. [22]

    Toward an architecture for never-ending language learning

    Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. Toward an architecture for never-ending language learning. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010

    2010

  16. [23]

    Abc-cnn: An attention based convolutional neural network for visual question answering

    Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960, 2015

    2015 arXiv

  17. [24]

    Counterfactual samples synthesizing for robust visual question answering

    Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10800–10809, 2020

    2020

  18. [25]

    Neil: Extracting visual knowledge from web data

    Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. Neil: Extracting visual knowledge from web data. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 1409–1416, 2013

    2013

  19. [26]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer, 2020

    2020

  20. [27]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bah- danau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014

    2014 arXiv

  21. [28]

    Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases

    Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683, 2019. 24 VOLUME 4, 2016

    1909 arXiv

  22. [29]

    Control of goal-directed and stimulus-driven attention in the brain

    Maurizio Corbetta and Gordon L Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience, 3(3):201, 2002

    2002

  23. [32]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  24. [33]

    Bert: Pre-training of deep bidirectional transformers for language under- standing

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language under- standing. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  25. [34]

    Open information extraction: The second generation

    Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, et al. Open information extraction: The second generation. In Twenty- Second International Joint Conference on Artificial Intelligence, 2011

    2011

  26. [35]

    Identifying rela- tions for open information extraction

    Anthony Fader, Stephen Soderland, and Oren Etzioni. Identifying rela- tions for open information extraction. In Proceedings of the conference on empirical methods in natural language processing, pages 1535–1545. Association for Computational Linguistics, 2011

    2011

  27. [36]

    Reciprocal attention fusion for visual question answering

    Moshiur R Farazi and Salman H Khan. Reciprocal attention fusion for visual question answering. arXiv preprint arXiv:1805.04247, 2018

    2018 arXiv

  28. [37]

    Multimodal compact bilinear pooling for visual question answering and visual grounding

    Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847, 2016

    2016 arXiv

  29. [38]

    Dynamic fusion with intra-and inter- modality attention flow for visual question answering

    Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter- modality attention flow for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 66...

    2019

  30. [39]

    Generat- ing natural language explanations for visual question answering using scene graphs and visual attention

    Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. Generat- ing natural language explanations for visual question answering using scene graphs and visual attention. arXiv preprint arXiv:1902.05715, 2019

    1902 arXiv

  31. [40]

    Mu- tant: A training paradigm for out-of-distribution generalization in visual question answering

    Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Mu- tant: A training paradigm for out-of-distribution generalization in visual question answering. arXiv preprint arXiv:2009.08566, 2020

    2009 arXiv

  32. [41]

    Single shot scene text retrieval

    Lluís Gómez, Andrés Mafla, Marçal Rusinol, and Dimosthenis Karatzas. Single shot scene text retrieval. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 700–715, 2018

    2018

  33. [42]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904– 6913, 2017

    2017

  34. [43]

    Bilinear graph networks for visual question answering

    Dalu Guo, Chang Xu, and Dacheng Tao. Bilinear graph networks for visual question answering. arXiv preprint arXiv:1907.09815, 2019

    1907 arXiv

  35. [44]

    Graph reasoning networks for visual question answering

    Dalu Guo, Chang Xu, and Dacheng Tao. Graph reasoning networks for visual question answering. 2019

    2019

  36. [45]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017

    2017

  37. [46]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  38. [47]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  39. [48]

    Yago2: A spatially and temporally enhanced knowledge base from wikipedia

    Johannes Hoffart, Fabian M Suchanek, Klaus Berberich, and Gerhard Weikum. Yago2: A spatially and temporally enhanced knowledge base from wikipedia. Artificial Intelligence, 194:28–61, 2013

    2013

  40. [49]

    Ex- plainable neural computation via stack neural module networks

    Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Ex- plainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pages 53–69, 2018

    2018

  41. [50]

    Learning to reason: End-to-end module networks for visual question answering

    Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804–813, 2017

    2017

  42. [51]

    Language-conditioned graph networks for relational reasoning

    Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-conditioned graph networks for relational reasoning. arXiv preprint arXiv:1905.04405, 2019

    1905 arXiv

  43. [52]

    Pixel-bert: Aligning image pixels with text by deep multi-modal transformers

    Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020

    2004 arXiv

  44. [53]

    Compositional attention networks for machine reasoning

    Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. arXiv preprint arXiv:1803.03067, 2018

    2018 arXiv

  45. [54]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019

    2019

  46. [55]

    Learning by abstraction: The neural state machine

    Drew A Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. arXiv preprint arXiv:1907.03950, 2019

    1907 arXiv

  47. [58]

    A focused dy- namic attention model for visual question answering

    Ilija Ilievski, Shuicheng Yan, and Jiashi Feng. A focused dy- namic attention model for visual question answering. arXiv preprint arXiv:1604.01485, 2016

    2016 arXiv

  48. [59]

    In defense of grid features for visual question answering

    Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10267–10276, 2020

    2020

  49. [60]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitio...

    2017

  50. [61]

    Inferring and executing programs for visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoff- man, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2989–2998, 2017

    2017

  51. [62]

    An analysis of visual question an- swering algorithms

    Kushal Kafle and Christopher Kanan. An analysis of visual question an- swering algorithms. In Proceedings of the IEEE International Conference on Computer Vision, pages 1965–1973, 2017

    1965

  52. [63]

    Multimodal residual learning for visual qa

    Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for visual qa. In Advances in neural information processing systems, pages 361–369, 2016

    2016

  53. [64]

    Hadamard product for low-rank bilinear pooling

    Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325, 2016

    2016 arXiv

  54. [65]

    Skip-thought vectors

    Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302, 2015

    2015

  55. [66]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Com- pute...

    2017

  56. [67]

    Incorporating external knowledge to answer open-domain visual questions with dynamic memory networks

    Guohao Li, Hang Su, and Wenwu Zhu. Incorporating external knowledge to answer open-domain visual questions with dynamic memory networks. arXiv preprint arXiv:1712.00733, 2017

    2017 arXiv

  57. [68]

    Relation-aware graph attention network for visual question answering

    Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. arXiv preprint arXiv:1903.12314, 2019

    1903 arXiv

  58. [69]

    Visualbert: A simple and performant baseline for vision and language

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  59. [70]

    Tell-and- answer: Towards explainable visual question answering using attributes and captions

    Qing Li, Jianlong Fu, Dongfei Yu, Tao Mei, and Jiebo Luo. Tell-and- answer: Towards explainable visual question answering using attributes and captions. arXiv preprint arXiv:1801.09041, 2018

    2018 arXiv

  60. [71]

    Vqa-e: Ex- plaining, elaborating, and enhancing your answers for visual questions

    Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Ex- plaining, elaborating, and enhancing your answers for visual questions. In VOLUME 4, 2016 25 Proceedings of the European Conference on Computer Vision (ECCV), pages 552–567, 2018

    2016

  61. [72]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121–137. Springer, 2020

    2020

  62. [73]

    Learning to contrast the counterfactual samples for robust visual question answering

    Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3285–3292, 2020

    2020

  63. [74]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per- ona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014

    2014

  64. [75]

    Feature enhancement in attention for visual question answering

    Yuetan Lin, Zhangyang Pang, Donghui Wang, and Yueting Zhuang. Feature enhancement in attention for visual question answering. In IJCAI, pages 4216–4222, 2018

    2018

  65. [76]

    Conceptnet—a practical commonsense reasoning tool-kit

    Hugo Liu and Push Singh. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226, 2004

    2004

  66. [77]

    Vilbert: Pretrain- ing task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretrain- ing task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019

    1908 arXiv

  67. [78]

    Hierarchical question-image co-attention for visual question answering

    Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pages 289–297, 2016

    2016

  68. [79]

    Co-attending free-form regions and detections with multi-modal multi- plicative feature embedding for visual question answering

    Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multi- plicative feature embedding for visual question answering. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018

    2018

  69. [80]

    Yago3: A knowledge base from multilingual wikipedias

    Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. Yago3: A knowledge base from multilingual wikipedias. 2013

    2013

  70. [81]

    The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision

    Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584, 2019

    1904 arXiv

  71. [82]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mot- taghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3195–3204, 2019

    2019

  72. [83]

    Transparency by design: Closing the gap between performance and in- terpretability in visual reasoning

    David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and in- terpretability in visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4942–4950, 2018

    2018

  73. [84]

    Distributed representations of words and phrases and their compo- sitionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compo- sitionality. In Advances in neural information processing systems, pages 3111–3119, 2013

    2013

  74. [85]

    Did the model understand the question? arXiv preprint arXiv:1805.05492, 2018

    Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. Did the model understand the question? arXiv preprint arXiv:1805.05492, 2018

    2018 arXiv

  75. [86]

    Dual attention networks for multimodal reasoning and matching

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 299–307, 2017

    2017

  76. [87]

    Out of the box: Reasoning with graph convolution nets for factual visual question answering

    Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. In Advances in Neural Information Processing Systems, pages 2654–2665, 2018

    2018

  77. [88]

    Straight to the facts: Learning knowledge base retrieval for factual visual question answer- ing

    Medhini Narasimhan and Alexander G Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question answer- ing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 451–468, 2018

    2018

  78. [89]

    Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering

    Duy-Kien Nguyen and Takayuki Okatani. Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6087–6096, 2018

    2018

  79. [90]

    Training recurrent answering units with joint loss minimization for vqa

    Hyeonwoo Noh and Bohyung Han. Training recurrent answering units with joint loss minimization for vqa. arXiv preprint arXiv:1606.03647, 2016

    2016 arXiv

  80. [91]

    Learning conditioned graph structures for interpretable visual question answering

    Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In Advances in Neural Information Processing Systems, pages 8334– 8343, 2018

    2018

  81. [92]

    Drau: Dual recurrent attention units for visual question answering

    Ahmed Osmana and Wojciech Sameka. Drau: Dual recurrent attention units for visual question answering

  82. [93]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014

    2014

  83. [94]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018

    2018

  84. [95]

    Deep contextualized word representations

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christo- pher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018

    2018 arXiv

  85. [96]

    Fast and scalable polynomial kernels via explicit feature maps

    Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 239–247. ACM, 2013

    2013

  86. [97]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018

    2018

  87. [98]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  88. [99]

    Over- coming language priors in visual question answering with adversarial regularization

    Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Over- coming language priors in visual question answering with adversarial regularization. arXiv preprint arXiv:1810.03649, 2018

    2018 arXiv

  89. [100]

    Make up your mind: Towards consistent answer predictions in vqa models

    Arijit Ray, Giedrius T Burachas, Karan Sikka, Anirban Roy, Avi Ziskind, Yi Yao, and Ajay Divakaran. Make up your mind: Towards consistent answer predictions in vqa models. In European Conference on Computer Vision (ECCV), Workshops, 2018

    2018

  90. [101]

    Exploring models and data for image question answering

    Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28:2953–2961, 2015

    2015

  91. [102]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015

    2015

  92. [103]

    High-order attention models for visual question answering

    Idan Schwartz, Alexander Schwing, and Tamir Hazan. High-order attention models for visual question answering. In Advances in Neural Information Processing Systems, pages 3664–3674, 2017

    2017

  93. [104]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakr- ishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017

    2017

  94. [105]

    Taking a hint: Leveraging explanations to make vision and language models more grounded

    Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF International Conference on Computer V...

    2019

  95. [106]

    Cycle- consistency for robust visual question answering

    Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle- consistency for robust visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6649–6658, 2019

    2019

  96. [107]

    Explainable and explicit visual reasoning over scene graphs

    Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and explicit visual reasoning over scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8376– 8384, 2019

    2019

  97. [108]

    Question type guided attention in visual question answering

    Yang Shi, Tommaso Furlanello, Sheng Zha, and Animashree Anandku- mar. Question type guided attention in visual question answering. In Proceedings of the European Conference on Computer Vision (ECCV), pages 151–166, 2018

    2018

  98. [109]

    Where to look: Focus regions for visual question answering

    Kevin J Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4613– 4621, 2016

    2016

  99. [110]

    Very deep convolutional net- works for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional net- works for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  100. [111]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019

    2019

  101. [112]

    Attention on atten- tion: Architectures for visual question answering (vqa)

    Jasdeep Singh, Vincent Ying, and Alex Nutkiewicz. Attention on atten- tion: Architectures for visual question answering (vqa). 03 2018. 26 VOLUME 4, 2016

    2018

  102. [113]

    From pixels to objects: Cubic visual attention for visual question answering

    Jingkuan Song, Pengpeng Zeng, Lianli Gao, and Heng Tao Shen. From pixels to objects: Cubic visual attention for visual question answering

  103. [114]

    Vl-bert: Pre-training of generic visual-linguistic representations

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019

    1908 arXiv

  104. [115]

    Ddrprog: A clevr differen- tiable dynamic reasoning programmer

    Joseph Suarez, Justin Johnson, and Fei-Fei Li. Ddrprog: A clevr differen- tiable dynamic reasoning programmer. arXiv preprint arXiv:1803.11361, 2018

    2018 arXiv

  105. [116]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

    2015

  106. [117]

    Lxmert: Learning cross-modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  107. [118]

    Acquiring comparative commonsense knowledge from the web

    Niket Tandon, Gerard De Melo, and Gerhard Weikum. Acquiring comparative commonsense knowledge from the web. In Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014

    2014

  108. [119]

    Separating style and content with bilinear models

    Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models. Neural computation, 12(6):1247–1283, 2000

    2000

  109. [120]

    Learning what makes a difference from counterfactual examples and gradient supervision

    Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. Learning what makes a difference from counterfactual examples and gradient supervision. arXiv preprint arXiv:2004.09034, 2020

    2004 arXiv

  110. [121]

    Tips and tricks for visual question answering: Learnings from the 2017 challenge

    Damien Teney, Peter Anderson, Xiaodong He, and Anton van den Hen- gel. Tips and tricks for visual question answering: Learnings from the 2017 challenge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4223–4232, 2018

    2017

  111. [122]

    Graph- structured representations for visual question answering

    Damien Teney, Lingqiao Liu, and Anton van den Hengel. Graph- structured representations for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2017

    2017

  112. [123]

    Interpretable counting for visual question answering

    Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable counting for visual question answering. ArXiv, abs/1712.08697, 2017

    2017 arXiv

  113. [124]

    Some mathematical notes on three-mode factor analysis

    Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):279–311, 1966

    1966

  114. [125]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017

    2017

  115. [126]

    Vqa with no questions-answers training

    Ben-Zion Vatashsky and Shimon Ullman. Vqa with no questions-answers training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10376–10386, 2020

    2020

  116. [127]

    Wikidata: a free collaborative knowledge base

    Denny Vrande ˇci´c and Markus Krötzsch. Wikidata: a free collaborative knowledge base. 2014

    2014

  117. [128]

    Explicit knowledge-based reasoning for visual question answer- ing

    Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Explicit knowledge-based reasoning for visual question answer- ing. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 1290–1296, 2017

    2017

  118. [129]

    Fvqa: Fact-based visual question answering

    Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2018

    2018

  119. [130]

    Explicit knowledge-based reasoning for visual question answering

    Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge-based reasoning for visual question answering. arXiv preprint arXiv:1511.02570, 2015

    2015 arXiv

  120. [131]

    The vqa- machine: Learning how to use existing vision algorithms to answer new questions

    Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. The vqa- machine: Learning how to use existing vision algorithms to answer new questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1173–1182, 2017

    2017

  121. [132]

    Learning convolutional text represen- tations for visual question answering

    Zhengyang Wang and Shuiwang Ji. Learning convolutional text represen- tations for visual question answering. In Proceedings of the 2018 SIAM International Conference on Data Mining, pages 594–602. SIAM, 2018

    2018

  122. [133]

    Interpretable visual reasoning via induced symbolic space

    Zhonghao Wang, Kai Wang, Mo Yu, Jinjun Xiong, Wen-mei Hwu, Mark Hasegawa-Johnson, and Humphrey Shi. Interpretable visual reasoning via induced symbolic space. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1878–1887, 2021

    2021

  123. [134]

    Learning from lexical perturbations for consistent visual question answering

    Spencer Whitehead, Hui Wu, Yi Ren Fung, Heng Ji, Rogerio Feris, and Kate Saenko. Learning from lexical perturbations for consistent visual question answering. arXiv preprint arXiv:2011.13406, 2020

    2011 arXiv

  124. [135]

    Faithful multimodal explanation for visual question answering

    Jialin Wu and Raymond J Mooney. Faithful multimodal explanation for visual question answering. arXiv preprint arXiv:1809.02805, 2018

    2018 arXiv

  125. [136]

    Self-critical reasoning for robust visual question answering

    Jialin Wu and Raymond J Mooney. Self-critical reasoning for robust visual question answering. arXiv preprint arXiv:1905.09998, 2019

    1905 arXiv

  126. [137]

    Image captioning and visual question answering based on attributes and external knowledge

    Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, and Anton van den Hengel. Image captioning and visual question answering based on attributes and external knowledge. IEEE transactions on pattern analysis and machine intelligence, 40(6):1367–1381, 2017

    2017

  127. [138]

    Ask me anything: Free-form visual question answering based on knowledge from external sources

    Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4622–4630, 2016

    2016

  128. [139]

    Verbs semantics and lexical selection

    Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Com- putational Linguistics, pages 133–138. Association for Computational Linguistics, 1994

    1994

  129. [140]

    Ask, attend and answer: Exploring question-guided spatial attention for visual question answering

    Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In European Conference on Computer Vision, pages 451–466. Springer, 2016

    2016

  130. [141]

    Trrnet: Tiered relation reasoning for compositional visual question answering

    Xiaofeng Yang, Guosheng Lin, Fengmao Lv, and Fayao Liu. Trrnet: Tiered relation reasoning for compositional visual question answering. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 414–430. Springer, 2020

    2020

  131. [142]

    Stacked attention networks for image question answering

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21–29, 2016

    2016

  132. [143]

    Cascaded mutual modulation for visual reasoning

    Yiqun Yao, Jiaming Xu, Feng Wang, and Bo Xu. Cascaded mutual modulation for visual reasoning. arXiv preprint arXiv:1809.01943, 2018

    2018 arXiv

  133. [144]

    Neural-symbolic vqa: Disentangling reasoning from vision and language understanding

    Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Infor- mation Processing Systems, pages 1031–1042, 2018

    2018

  134. [145]

    Multi-level attention networks for visual question answering

    Dongfei Yu, Jianlong Fu, Tao Mei, and Yong Rui. Multi-level attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4709– 4717, 2017

    2017

  135. [146]

    Ernie-vil: Knowledge enhanced vision-language representations through scene graph

    Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934, 1:12, 2020

    2006 arXiv

  136. [147]

    Deep modular co-attention networks for visual question answering

    Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6281–6290, 2019

    2019

  137. [148]

    Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering

    Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE transactions on neural networks and learning systems, 29(12):5947–5959, 2018

    2018

  138. [149]

    Yin and yang: Balancing and answering binary visual questions

    Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5014–5022, 2016

    2016

  139. [150]

    Learning to count objects in natural images for visual question answering

    Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Learning to count objects in natural images for visual question answering. arXiv preprint arXiv:1802.05766, 2018

    2018 arXiv

  140. [151]

    Interpretable visual question answering by visual grounding from attention supervision mining

    Yundong Zhang, Juan Carlos Niebles, and Alvaro Soto. Interpretable visual question answering by visual grounding from attention supervision mining. In 2019 IEEE Winter Conference on Applications of Computer Vision (W ACV), pages 349–357. IEEE, 2019

    2019

  141. [152]

    Unified vision-language pre-training for image captioning and vqa

    Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 13041–13049, 2020

    2020

  142. [153]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995– 5004, 2016. VOLUME 4, 2016 27

    2016

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.