Pith. sign in

REVIEW 4 major objections 4 minor 135 references

Augmented Vision-Language Models: A Systematic Review

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Augmenting vision-language models with external symbolic systems and tools consistently improves their performance on knowledge-intensive and reasoning-heavy tasks, the authors conclude from 264 papers.

desk verdict A useful taxonomy of inference-time VLM augmentation; the performance consensus it claims is asserted rather than synthesized. read the letter →

arxiv 2507.22933 v1 pith:63P5SY45 submitted 2025-07-24 cs.CL cs.AI

Anthony C Davis , Burhan Sadiq , Tianmin Shu , Chien-Ming Huang This is my paper
classification cs.CLcs.AI
keywords augmentedvision-languagemodelsneural-symbolicsystemsretrieval-augmentedgenerationtoolusevisualquestionansweringknowledgegraphsinference-timeaugmentationsystematicreview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This systematic review asks whether connecting vision-language models to external symbolic systems and tools during inference makes them better, and it answers yes. Surveying 264 papers, it reports a clear consensus that augmentation significantly boosts performance on knowledge-intensive and reasoning-heavy tasks, from knowledge-based visual question answering to robotics planning. The review defines the object of study narrowly: an Augmented Vision-Language Model is a VLM that retrieves facts, runs calculations, executes code, or calls APIs during inference, rather than a model trained on extra data. It organizes the field by fusion timing and by the type of external component, and it singles out tool use as a flexible abstraction that lets one VLM orchestrate many external capabilities. If this consensus is right, the practical route to more reliable vision-language systems does not have to wait for bigger monolithic models.

What carries the argument

The organizing device is the temporal fusion axis. Early fusion adds external information to the input before the VLM's forward pass; middle fusion lets the VLM query external systems during its internal processing and feeds results back in; late fusion uses the VLM's initial output to trigger retrieval, validation, or symbolic refinement. The second axis distinguishes retrieval of existing facts from symbolic computation that generates new information, with a combined category for systems doing both. The named object under study is the Augmented Vision-Language Model, defined as a VLM integrated with external symbolic systems, APIs, databases, or computational tools during inference, and the paper's central abstraction is tool use, which treats every external capability as a callable function the VLM selects and interprets.

What would settle it

Find a substantial cluster of relevant augmentation methods that were excluded by the review's filter or language restriction and that do not fit the early/middle/late fusion taxonomy, or show by quantitative re-analysis of the surveyed papers that augmentation produces no average accuracy gain over matched non-augmented baselines on knowledge-based visual question answering; either observation would break the claimed consensus.

Watch

Extended reading notes

Core claim

The paper's central claim is that augmenting a pretrained vision-language model with external symbolic information systems and computational tools, at inference time, reliably improves performance on exactly the tasks where standalone VLMs are weak: factual recall, precise arithmetic, logical deduction, and complex spatial reasoning. The authors arrive at this claim by systematically categorizing 264 studies into a taxonomy of early, middle, and late fusion, crossed with retrieval versus symbolic computation, and they find that hybrid systems consistently outperform purely neural methods of similar computational cost on knowledge-intensive and reasoning-heavy benchmarks. They further argue that tool use is emerging as the unifying design pattern, with the VLM acting as an orchestrator that learns when to retrieve, calculate, execute code, or call a specialized module, and that this pattern is modular and scalable even though current interaction protocols are cumbersome.

Load-bearing premise

The entire review rests on the assumption that the 264 papers that passed its AI-assisted relevance filter, with an 8-out-of-10 score threshold, followed by manual screening in an English-only two-database search, fairly represent the whole field of augmented vision-language models.

Editorial extensions

If this is right

  • Augmented vision-language models will outperform purely neural models of similar compute on knowledge-based visual question answering, object-relation analysis, and robotics planning.
  • Tool use becomes the default design pattern: new capabilities can be added by defining new tools, and the VLM's core job shifts to selecting, invoking, and interpreting them.
  • Fusion stage becomes a design trade-off: early fusion is simple but noise-prone, middle fusion is dynamic and iterative, and late fusion returns exact, traceable outputs from the VLM's first guess.
  • Benchmarking must expand beyond accuracy to include interpretability, reasoning transparency, and efficiency; the current fragmented benchmark landscape cannot fairly compare AVLMs.
  • Making tool invocation native to the VLM, for instance through specialized tokens or architectural changes, could remove the parsing and latency overhead of today's conversational tool-calling protocols.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, its consensus implies a practical sizing rule: training a moderately sized VLM to call a calculator, a knowledge base, and a code executor may be a cheaper way to reach reliable arithmetic and factual performance than scaling one monolithic model.
  • The paper's taxonomy suggests a testable prediction it does not run: on the same benchmark, retrieval-augmented early fusion should gain less and add more noise than middle or late fusion, which can be checked directly on a dataset like OK-VQA or CLEVR.
  • Because the consensus is inferred from a filtered corpus, the strongest version of the claim, that augmentation generally helps, is itself a hypothesis a broader multilingual replication could confirm or overturn.
  • As tool-calling interfaces become native, the early/middle/late boundary may blur, with verification and refinement moving from a separate post-hoc stage into the generation loop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper defines Augmented Vision-Language Models (AVLMs) as pre-trained VLMs that interact with external symbolic information systems, APIs, databases, or computational tools during inference, and presents a systematic literature review of 264 papers following PRISMA-style search and screening. The review organizes the surveyed methods into early, middle, and late fusion stages, further subdivided by whether the augmentation is retrieval-based, symbolic-computation-based, or combined; it also catalogs relevant datasets and discusses tool use as a unifying abstraction. The conclusion claims a 'clear consensus' that augmenting VLMs significantly boosts performance on knowledge-intensive and reasoning-heavy tasks.

Significance. If its claims are taken at face value, the paper provides a useful and broad organizational map of a rapidly growing area, and the early/middle/late fusion taxonomy plus the tool-use framing could help future researchers position new work. The systematic search protocol, explicit inclusion/exclusion criteria, and the large appendix tables of categorized papers are valuable resources. The paper's central empirical conclusion, however, is currently asserted rather than demonstrated, and the screening methodology has reproducibility gaps. With appropriate revisions to the evidence base and the strength of the conclusion, the survey could be a genuinely useful reference for the community.

major comments (4)
  1. [Section 8 and Section 7.2] The paper's central claim of a 'clear consensus: augmenting VLMs significantly boosts performance on knowledge-intensive and reasoning-heavy tasks' is not supported by the evidence presented. No quantitative synthesis is provided: there are no effect sizes, no aggregation of reported accuracy gains, no vote-counting of positive versus null or negative results, and no comparison against non-augmented baselines. The appendix tables list papers by category but record no per-paper outcomes, so the conclusion rests on informal qualitative generalization rather than the systematic synthesis the methodology promises. The authors should either add a synthesis of reported results (even a simple vote count or sign table over papers that report direct comparisons) or explicitly weaken the conclusion to a statement about the motivations and reported benefits found in the literature.
  2. [Appendix A.3.3, Appendix A.1, Appendix A.6] The GPT-4o relevance filter with a threshold of 8/10 is an opaque and potentially biasing selection step: the prompt given to the model is not provided, no validation of the model's scores against manual screening is described, and no agreement or sensitivity analysis is reported. Combined with the English-only restriction and the use of only two databases, the representativeness of the final 264 papers is not established. The authors should report the exact screening prompt, provide a manual audit of a random sample of excluded papers, and discuss how threshold choice affects the final set.
  3. [Section 1.2 and Appendix A.2.2] There is a scope inconsistency between the stated definition of AVLMs as inference-time augmentation only and some included papers. For example, REVEAL (Hu et al., 2022) and KB-VLP (Chen et al., 2021a) are described in the literature as retrieval-augmented pretraining methods, which appears to fall under the excluded category 'Training with Structured Knowledge' in A.2.2. The paper does not explain how such training-time methods satisfy the inference-only criterion, and this affects the coherence of the corpus selection.
  4. [Section 2 and Appendix B] The early/middle/late fusion taxonomy is the review's main organizational contribution, but the criteria are defined informally and no per-paper justification is given for category assignments. Several placements are contestable from the definitions provided (for example, what distinguishes a 'Retrieval Encoder' from 'Prompt Augmentation' in Table 1, and when a retrieval step counts as 'during the forward pass' in Table 2). Without an inter-coder reliability check, a documented coding manual, or at least a set of worked examples, the validity and reproducibility of the taxonomy as a systematic classification are not substantiated.
minor comments (4)
  1. [Section 1.2] There are minor language errors: 'A AVLM is an VLM' should be 'An AVLM is a VLM,' and 'these integrations aims' should be 'these integrations aim.'
  2. [Table 4 and References] Table 4 has the header 'CLEVER' where the dataset is CLEVR, and the reference list includes 'Krishna et al. (2016a)' and 'Krishna et al. (2016b)' for what appears to be the same Visual Genome paper.
  3. [Appendix A.3.4] The text references 'Figure 4' as a PRISMA flowchart, but no Figure 4 appears in the manuscript; the figure should be included or the reference removed.
  4. [References] Several reference entries are corrupted or contain large irrelevant run-on passages; for example, the Hayashi et al. (2024) entry and the passage beginning 'Dustin Schwenk, Apoorv Khandelwal...' include extensive unrelated author lists. This makes the bibliography unreliable for verification and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the review categorizes existing literature and its consensus claim, though not quantitatively demonstrated, is not equivalent to its own inputs.

full rationale

This paper is a systematic literature review, not a derivation or prediction exercise. Its core definition of an Augmented Vision-Language Model (Section 1.2) is a stipulated scope condition, independent of the survey's conclusions, and the Early/Middle/Late fusion taxonomy is an organizational framework rather than a consequence of the included papers. The Section 8 statement that 'augmenting VLMs significantly boosts performance' is an interpretive generalization, not a quantity fitted to or implied by the inclusion criteria; the review does not compute any effect size, but the absence of quantitative synthesis is an evidentiary weakness, not circularity. The paper's search and selection methodology (Appendix A) is transparently reported, including the GPT-4o relevance filter and language/database limitations, and these limitations are acknowledged in Appendix A.6. There are no self-citation chains, no imported uniqueness theorems, and no claimed predictions that reduce to fitted inputs. The central claim, while potentially overstated, is an independent empirical assertion about the surveyed literature rather than a restatement of the review's own definitions or selection criteria.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The review relies on the representativeness of its curated paper set and on the validity of its fusion-stage taxonomy. No free parameters or invented entities are introduced, since the paper is a survey rather than a derivation or empirical study.

assumptions (3)
  • domain assumption The selected 264 papers are representative of the AVLM literature covered by the search.
    The review's taxonomy and conclusions generalize from this curated set; stated in the PRISMA methodology and inclusion criteria (Appendix A).
  • domain assumption GPT-4o relevance scoring with a threshold of 8/10 preserves all relevant papers.
    If the AI filter systematically dropped relevant papers, the landscape description would be incomplete; Appendix A.3.3.
  • ad hoc to paper Early/middle/late fusion categories are mutually meaningful distinctions that reflect underlying architectural differences.
    This is the paper's organizing framework, defined in Section 2, and not validated against independent criteria.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Augmented Vision-Language Models: A Systematic Review." pith.science (2026). https://pith.science/paper/63P5SY45

@misc{pith2026250722933,
  author       = {Pith},
  title        = {Pith review of: Augmented Vision-Language Models: A Systematic Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63P5SY45}},
  note         = {Machine review of arXiv:2507.22933}
}
read the original abstract

Recent advances in visual-language machine learning models have demonstrated exceptional ability to use natural language and understand visual scenes by training on large, unstructured datasets. However, this training paradigm cannot produce interpretable explanations for its outputs, requires retraining to integrate new information, is highly resource-intensive, and struggles with certain forms of logical reasoning. One promising solution involves integrating neural networks with external symbolic information systems, forming neural symbolic systems that can enhance reasoning and memory abilities. These neural symbolic systems provide more interpretable explanations to their outputs and the capacity to assimilate new information without extensive retraining. Utilizing powerful pre-trained Vision-Language Models (VLMs) as the core neural component, augmented by external systems, offers a pragmatic approach to realizing the benefits of neural-symbolic integration. This systematic literature review aims to categorize techniques through which visual-language understanding can be improved by interacting with external symbolic information systems.

Figures

Figures reproduced from arXiv: 2507.22933 by the authors.

Figure 1
Figure 1. Architecture for Retrieval Augmented Chest X-Ray Report Generation by Ranjit et al. (2023). [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview of the ViLaIn approach for VLM planning of robotic actions Shirai et al. (2023). The [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The architecture of the AQuA framework Basu et al. (2020). It consists of five main modules: [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: PRISMA Flowchart A.4 Data Extraction and Synthesis From the selected studies, we extracted pertinent information to facilitate a comprehensive understanding of the methods: • Integration Techniques: Description of how external systems were integrated with vision-langua…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

135 extracted references · 37 canonical work pages

  1. [2]

    Spatial knowledge distillation to aid visual reasoning

    Somak Aditya, Rudra Saha, Yezhou Yang, and Chitta Baral. Spatial knowledge distillation to aid visual reasoning. arXiv preprint, arXiv:1812.03631v2,

    arXiv

  2. [7]

    AQuA: ASP-Based Visual Question Answering

    Kuntal Basu, Farhad Shakerin, and Gopal Gupta. AQuA: ASP-Based Visual Question Answering. In Ekaterina Komendantskaya and Yanhong A. Liu (eds.), Practical Aspects of Declarative Languages (PADL 2020), volume 12007 of Lecture Notes in Computer Science. Springer, Cham,

    2020

  3. [10]

    Visual riddles: a commonsense and world knowledge challenge for large vision and language models.arXiv preprint, arXiv:2407.19474v2,

    21 Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: a commonsense and world knowledge challenge for large vision and language models.arXiv preprint, arXiv:2407.19474v2,

    arXiv

  4. [11]

    Thinking fast and slow in ai

    Grady Booch, Francesco Fabiano, Lior Horesh, Kiran Kate, Jon Lenchner, Nick Linck, Andrea Loreggia, Keerthiram Murugesan, Nicholas Mattei, Francesca Rossi, and Biplav Srivastava. Thinking fast and slow in ai. Proceedings of the AAAI Conference on Artificial Intelligence 2021, 35(17), 15042-15046,

    2021

  5. [13]

    Linguistically routing capsule network for out-of-distribution visual question answering.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Qingxing Cao, Wentao Wan, Keze Wang, Xiaodan Liang, and Liang Lin. Linguistically routing capsule network for out-of-distribution visual question answering.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1594–1603,

    2021

  6. [15]

    Hammr: Hierarchical multimodal react agents for generic vqa.arXiv preprint, arXiv:2404.05465v2,

    Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierarchical multimodal react agents for generic vqa.arXiv preprint, arXiv:2404.05465v2,

    arXiv

  7. [16]

    Multimodal structured generation: Cvpr’s 2nd mmfm challenge technical report.arXiv preprint, arXiv:2406.11403v2,

    Franz Louis Cesista. Multimodal structured generation: Cvpr’s 2nd mmfm challenge technical report.arXiv preprint, arXiv:2406.11403v2,

    arXiv

  8. [17]

    Retrieval augmented structured generation: Business document information extraction as tool use.arXiv preprint, arXiv:2405.20245v1,

    Franz Louis Cesista, Rui Aguiar, Jason Kim, and Paolo Acilo. Retrieval augmented structured generation: Business document information extraction as tool use.arXiv preprint, arXiv:2405.20245v1,

    arXiv

Show all 135 references
  1. [18]

    Uncertainty-based visual question answering: Estimating semantic incon- sistency between image and knowledge base

    Jinyeong Chae and Jihie Kim. Uncertainty-based visual question answering: Estimating semantic incon- sistency between image and knowledge base. 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–9,

    2022

  2. [19]

    org/CorpusID:212814759

    URL https://api.semanticscholar. org/CorpusID:212814759. 22 Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? ArXiv, abs/2302.11713,

    arXiv

  3. [20]

    Pan, Zonggang Yuan, and Huajun Chen

    Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, and Huajun Chen. Zero-shot visual question answering using knowledge graph.ArXiv, abs/2107.05348, 2021b. Zhuo Chen, Yufen Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang, Jeff Z. Pan, Ningyu Zhang, and Wen Zhang. Lako:...

    arXiv

  4. [21]

    Guo, and Xueqi Cheng

    Wanqing Cui, Keping Bi, J. Guo, and Xueqi Cheng. More: Multi-modal retrieval augmented generative commonsense reasoning. ArXiv, abs/2402.13625,

    arXiv

  5. [22]

    The role of foundation models in neuro- symbolic learning and reasoning.ArXiv, abs/2402.01889,

    Daniel Cunnington, Mark Law, Jorge Lobo, and Alessandra Russo. The role of foundation models in neuro- symbolic learning and reasoning.ArXiv, abs/2402.01889,

    arXiv

  6. [23]

    External knowledge augmented text visual question answering

    Arka Ujjal Dey, Ernest Valveny, and Gaurav Harit. External knowledge augmented text visual question answering. ArXiv, abs/2108.09717,

    arXiv

  7. [24]

    Escucha, Laura Siabatto, Andr’es Link, Pablo Arbel’aez, R

    ZalanFabian, ZhongqiMiao, ChunyuanLi, YuanhanZhang, ZiweiLiu, A.Hern’andez, AndrésMontes-Rojas, Rafael S. Escucha, Laura Siabatto, Andr’es Link, Pablo Arbel’aez, R. Dodhia, and J. Ferres. Multimodal foundation models for zero-shot animal species recognition in camera trap imag...

    arXiv

  8. [25]

    Videoagent: A memory- augmented multimodal agent for video understanding.ArXiv, abs/2403.11481,

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding.ArXiv, abs/2403.11481,

    arXiv

  9. [26]

    Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, D

    Xingyu Fu, Shenmin Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, A. Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, D. Roth, and Bing Xiang. Generate then select: Open-ended visual question answering guided by world knowledge.ArXiv, abs/...

    arXiv

  10. [27]

    URLhttps://proceedings.mlr

    PMLR. URLhttps://proceedings.mlr. press/v48/gal16.html. Jingru Gan, Xinzhe Han, Shuhui Wang, and Qingming Huang. Open-set knowledge-based visual question answering with inference paths.ArXiv, abs/2310.08148,

    arXiv

  11. [28]

    Room-object entity prompting and reasoning for embodied referring expression.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:994–1010, 2023a

    23 Chen Gao, Si Liu, Jinyu Chen, Luting Wang, Qi Wu, Bo Li, and Qi Tian. Room-object entity prompting and reasoning for embodied referring expression.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:994–1010, 2023a. Difei Gao, Ruiping Wang, Shiguang Shan, and...

    1908 arXiv

  12. [29]

    Feng Gao, Q. Ping, G. Thattai, Aishwarya N. Reganti, Yingting Wu, and Premkumar Natarajan. Transform- retrieve-generate: Natural language-centric outside-knowledge visual question answering.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5057–50...

    2022

  13. [30]

    Cross-modal object detection based on a knowledge update.Sensors (Basel, Switzerland), 22, 2022b

    Yueqing Gao, Huachun Zhou, Lulu Chen, Yuting Shen, Ce Guo, and Xinyu Zhang. Cross-modal object detection based on a knowledge update.Sensors (Basel, Switzerland), 22, 2022b. Diego Garcia-Olano, Yasumasa Onoe, and J. Ghosh. Improving and diagnosing knowledge-based visual questi...

    2022

  14. [31]

    Scene graph generation with external knowledge and image reconstruction.arXiv preprint, arXiv:1904.00560v1,

    Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction.arXiv preprint, arXiv:1904.00560v1,

    1904 arXiv

  15. [32]

    Hauptmann, Yonatan Bisk, and Jianfeng Gao

    Liangke Gui, Borui Wang, Qiuyuan Huang, A. Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision-and-language.ArXiv, abs/2112.08614,

    arXiv

  16. [33]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint, arXiv:2211.11559v1,

    arXiv

  17. [34]

    Neverova, C

    Shir Gur, N. Neverova, C. Stauffer, S. Lim, Douwe Kiela, and A. Reiter. Cross-modal retrieval augmentation for multi-modal classification.ArXiv, abs/2104.08108,

    arXiv

  18. [35]

    Knowledge condensation and reasoning for knowledge-based vqa.ArXiv, abs/2403.10037, 2024a

    Dongze Hao, Jian Jia, Longteng Guo, Qunbo Wang, Te Yang, Yan Li, Yanhua Cheng, Bo Wang, Quan Chen, Han Li, and Jing Liu. Knowledge condensation and reasoning for knowledge-based vqa.ArXiv, abs/2403.10037, 2024a. Dongze Hao, Qunbo Wang, Longteng Guo, Jie Jiang, and Jing Liu. Se...

    2022 arXiv

  19. [36]

    Heo, Eun-Sol Kim, Woo Suk Choi, and Byoung-Tak Zhang

    Y. Heo, Eun-Sol Kim, Woo Suk Choi, and Byoung-Tak Zhang. Hypergraph transformer: Weakly-supervised multi-hop reasoning for knowledge-based visual question answering.ArXiv, abs/2204.10448,

    arXiv

  20. [37]

    Jingyi Hou, Xinxiao Wu, Xiaoxun Zhang, Yayun Qi, Yunde Jia, and Jiebo Luo

    URL https://arxiv.org/abs/2203.15556. Jingyi Hou, Xinxiao Wu, Xiaoxun Zhang, Yayun Qi, Yunde Jia, and Jiebo Luo. Joint commonsense and relation reasoning for image and video captioning. pp. 10973–10980,

    arXiv

  21. [38]

    Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister

    Cheng-Yu Hsieh, Sibei Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander J. Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. ArXiv, abs/2308.00675,

    arXiv

  22. [39]

    An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, R

    Xinyue Hu, Lin Gu, Qi A. An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, R. M. Summers, and Yingying Zhu. Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering.Proceedings of the 29th ...

    2024 arXiv

  23. [40]

    Schmid, David A

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, C. Schmid, David A. Ross, and A. Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory.2023 IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2023

  24. [41]

    Avis: Autonomous visual information seeking with large language model agent.arXiv preprint, arXiv:2306.08129v3, 2023c

    Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language model agent.arXiv preprint, arXiv:2306.08129v3, 2023c. Feicheng Huang, Zhixin Li, Haiyang Wei, Canlong ...

    2011 arXiv

  25. [42]

    Hudson and Christopher D

    26 Drew A. Hudson and Christopher D. Manning. Learning by abstraction: The neural state machine.arXiv preprint, arXiv:1907.03950v4,

    1907 arXiv

  26. [43]

    Shahzad, and M

    Afzaal Hussain, Ifrah Maqsood, M. Shahzad, and M. Fraz. Multimodal knowledge reasoning for enhanced visual question answering. 2022 16th International Conference on Signal-Image Technology & Internet- Based Systems (SITIS), pp. 224–230,

    2022

  27. [44]

    Fathi, and C

    Ahmet Iscen, Mathilde Caron, A. Fathi, and C. Schmid. Retrieval-enhanced contrastive vision-text models. ArXiv, abs/2306.07196,

    arXiv

  28. [45]

    Jamshed and M

    A. Jamshed and M. Fraz. Nlp meets vision for visual interpretation - a retrospective insight and future direc- tions. In 2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2), pp. 1–8,

    2021

  29. [46]

    Precision empowers, excess distracts: Visual question answering with dynamically infused knowledge in language models

    Manas Jhalani, Annervaz K M, and Pushpak Bhattacharyya. Precision empowers, excess distracts: Visual question answering with dynamically infused knowledge in language models. arXiv preprint, arXiv:2406.09994v1,

    arXiv

  30. [47]

    Narayana, Arjun Reddy Akula, G

    Zhiwei Jia, P. Narayana, Arjun Reddy Akula, G. Pruthi, Haoran Su, Sugato Basu, and Varun Jampani. Kafa: Rethinkingimageadunderstandingwithknowledge-augmentedfeatureadaptationofvision-languagemod- els. ArXiv, abs/2305.18373,

    arXiv

  31. [48]

    Chen Jiang, Masood Dehghan, and Martin Jägersand. Understanding contexts inside robot and human manipulation tasks through vision-language model and ontology system in video streams.2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8366–8372,...

    2020

  32. [49]

    Xu, Jun Araki, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know?, 2020b. URLhttps://arxiv.org/abs/1911.12543. Ruihan Jin, Ruibo Fu, Zhengqi Wen, Shuai Zhang, Yukun Liu, and Jianhua Tao. Fake news detection and manipulation reasoning via lar...

    1911 arXiv

  33. [50]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.arXiv preprint, arXiv:1612.06890v1,

    arXiv

  34. [51]

    Knowledge-aware prompt tuning for generalizable vision-language models.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15624–15634,

    2023

  35. [52]

    Mahmoud Khademi, Ziyi Yang, F

    URL https://arxiv.org/abs/2001.08361. Mahmoud Khademi, Ziyi Yang, F. Frujeri, and Chenguang Zhu. Mm-reasoner: A multi-modal knowledge- aware framework for knowledge-based visual question answering. pp. 6571–6581,

    2001 arXiv

  36. [53]

    M. A. Khaliq, P. Chang, M. Ma, B. Pflugfelder, and F. Mileti’c. Ragar, your falsehood radar: Rag-augmented reasoning for political fact-checking using multimodal large language models.ArXiv, abs/2404.12065,

    arXiv

  37. [54]

    Breslin, and E

    Muhammad Jaleed Khan, J. Breslin, and E. Curry. Expressive scene graph generation using commonsense knowledge infusion for visual understanding and reasoning. pp. 93–112, 2022a. Muhammad Jaleed Khan, J. Breslin, and E. Curry. Neusire: Neural-symbolic image representation and e...

    arXiv

  38. [55]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint, arXiv:2401.13649v2,

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.arXiv preprint, arXiv:2401.13649v2,

    arXiv

  39. [56]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yan- nis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connect- ing language and vision using crowdsourced dense image annotations...

    arXiv

  40. [57]

    Jaeyun Lee and Incheol Kim

    URLhttps://arxiv.org/abs/2205.03854. Jaeyun Lee and Incheol Kim. Vision–language–knowledge co-embedding for visual commonsense reasoning. Sensors (Basel, Switzerland), 21,

    arXiv

  41. [58]

    Multimodal reasoning with multimodal knowledge graph

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multimodal reasoning with multimodal knowledge graph. arXiv preprint, arXiv:2406.02030v2,

    arXiv

  42. [59]

    URL https://arxiv.org/abs/2005. 11401. 28 Bojin Li, Yan Sun, Xue Chen, and Xiangfeng Luo. Hkfnet: Fine-grained external knowledge fusion for fact-based visual question answering.2024 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2024a. Guohao Li, Hang Su,...

    2005 arXiv

  43. [60]

    Visual question answering as reading comprehension

    Hui Li, Peng Wang, Chunhua Shen, and Anton van den Hengel. Visual question answering as reading comprehension. arXiv preprint, arXiv:1811.11903v1,

    arXiv

  44. [61]

    Supporting vision-language model inference with causality-pruning knowledge prompt.ArXiv, abs/2205.11100, 2022a

    Jiangmeng Li, Wenyi Mo, Wenwen Qiang, Bing Su, and Changwen Zheng. Supporting vision-language model inference with causality-pruning knowledge prompt.ArXiv, abs/2205.11100, 2022a. Meng Li, Tianbao Wang, Jiahe Xu, Kairong Han, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang...

    arXiv

  45. [62]

    Bhanu, Biyun Sheng, and Richang Hong

    Qun Li, Fu Xiao, B. Bhanu, Biyun Sheng, and Richang Hong. Inner knowledge-based img2doc scheme for visual question answering.ACM Transactions on Multimedia Computing, Communications, and Applica- tions (TOMM), 18:1 – 21, 2022c. Xin Li, Yu Zhang, Weilin Yuan, and Junren Luo. In...

    arXiv

  46. [63]

    Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning.arXiv preprint, arXiv:2212.00259v2, 2022e

    Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning.arXiv preprint, arXiv:2212.00259v2, 2022e. Ziyang Li, Jiani Huang, and Mayur Naik....

    2011 arXiv

  47. [64]

    Maria: A visual experience powered conversational agent.arXiv preprint, arXiv:2105.13073v2,

    Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xiubo Geng, Yining Chen, Fan Liang, and Daxin Jiang. Maria: A visual experience powered conversational agent.arXiv preprint, arXiv:2105.13073v2,

    arXiv

  48. [65]

    Xing, and Xiaodan Liang

    Bingqian Lin, Zicong Chen, Mingjie Li, Haokun Lin, Hang Xu, Yi Zhu, Jian zhuo Liu, Wenjia Cai, Lei Yang, Shen Zhao, Chenfei Wu, Ling Chen, Xiaojun Chang, Yi Yang, L. Xing, and Xiaodan Liang. Towards med- ical artificial general intelligence via knowledge-enhanced multimodal pr...

    arXiv

  49. [66]

    Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering.ArXiv, abs/2309.17133, 2023b

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering.ArXiv, abs/2309.17133, 2023b. Weizhe Lin, Zhilin Wang, and B. Byrne. Fvqa 2.0: Introducing adversarial...

    arXiv

  50. [67]

    Knowledge-enhanced hierarchical information correlation learning for multi-modal rumor detection.ArXiv, abs/2306.15946, 2023a

    Jiawei Liu, Jingyi Xie, Fanrui Zhang, Qiang Zhang, and Zhengjun Zha. Knowledge-enhanced hierarchical information correlation learning for multi-modal rumor detection.ArXiv, abs/2306.15946, 2023a. Luping Liu, Meiling Wang, Xiaohai He, L. Qing, and Honggang Chen. Fact-based visu...

    arXiv

  51. [68]

    Llava-plus: Learning to use tools for creating multimodal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint, arXiv:2311.05437v1, 2023b. Ziyu Liu, Zey...

    arXiv

  52. [69]

    Wu, Song-Chun Zhu, and Jianfeng Gao

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Y. Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models.ArXiv, abs/2304.09842,

    arXiv

  53. [70]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi

    URLhttps: //arxiv.org/abs/2002.06177. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge.arXiv preprint, arXiv:1906.00067v2,

    2002 arXiv

  54. [71]

    Gupta, and Marcus Rohrbach

    Kenneth Marino, Xinlei Chen, Devi Parikh, A. Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14106–14116,

    2021

  55. [72]

    Uijlings, Lluís Castrejón, A

    30 Thomas Mensink, J. Uijlings, Lluís Castrejón, A. Goel, Felipe Cadar, Howard Zhou, Fei Sha, A. Araújo, and V. Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp...

    2023

  56. [73]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint, arXiv:2410.05229v1,

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint, arXiv:2410.05229v1,

    arXiv

  57. [74]

    Aakansha Mishra, S. S. Miriyala, and V. N. Rajendiran. Learning representations from explainable and con- nectionist approaches for visual question answering.ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6420–6424,

    2024

  58. [75]

    Vqa training sets are self-play environments for generating few-shot pools.ArXiv, abs/2405.19773,

    Tautvydas Misiunas, Hassan Mansoor, Jasper Uijlings, Oriana Riva, and Victor Carbune. Vqa training sets are self-play environments for generating few-shot pools.ArXiv, abs/2405.19773,

    arXiv

  59. [76]

    LudovicoMitchener, DavidTuckey, MatthewCrosby, andA.Russo

    URLhttps://arxiv.org/abs/2110.11309. LudovicoMitchener, DavidTuckey, MatthewCrosby, andA.Russo. Detect, understand, actaneuro-symbolic hierarchical reinforcement learning framework

    arXiv

  60. [77]

    O’Regan, and A

    Guglielmo Montone, J. O’Regan, and A. Terekhov. Hyper-dimensional computing for a visual question- answering system that is trainable end-to-end.ArXiv, abs/1711.10185,

    arXiv

  61. [78]

    Medhini Narasimhan and Alexander G. Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question answering.arXiv preprint, arXiv:1809.01124v1,

    arXiv

  62. [79]

    Medhini Narasimhan, Svetlana Lazebnik, and Alexander G. Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering.arXiv preprint, arXiv:1811.00538v1,

    arXiv

  63. [80]

    External commonsense knowledge as a modality for social intelligence question-answering

    Sanika Natu, Shounak Sural, and Sulagna Sarkar. External commonsense knowledge as a modality for social intelligence question-answering. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 3036–3042,

    2023

  64. [81]

    Screenagent: A vision language model-driven computer control agent.ArXiv, abs/2402.07945,

    Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent.ArXiv, abs/2402.07945,

    arXiv

  65. [82]

    Prediction of actions and places by the time series recognition from images with multimodal llm.2024 IEEE 18th International Conference on Semantic Computing (ICSC), pp

    Tomohiro Ogawa, Kango Yoshioka, Ken Fukuda, and Takeshi Morita. Prediction of actions and places by the time series recognition from images with multimodal llm.2024 IEEE 18th International Conference on Semantic Computing (ICSC), pp. 294–300,

    2024

  66. [83]

    Modal-adaptive knowledge- enhanced graph-based financial prediction from monetary policy conference calls with llm

    31 Kun Ouyang, Yi Liu, Shicheng Li, Ruihan Bao, Keiko Harimoto, and Xu Sun. Modal-adaptive knowledge- enhanced graph-based financial prediction from monetary policy conference calls with llm. ArXiv, abs/2403.16055,

    arXiv

  67. [84]

    Shalin, and Lane Peterson Fronczek

    Trilok Padhi, Ugur Kursuncu, Yaman Kumar, V. Shalin, and Lane Peterson Fronczek. Improving contextual congruence across modalities for effective multimodal marketing using knowledge-infused learning.ArXiv, abs/2402.03607,

    arXiv

  68. [85]

    Shishir G

    URLhttps://arxiv.org/ abs/2304.03442. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis,

    arXiv

  69. [86]

    URLhttps://arxiv.org/abs/2305.15334. A. Potapov, A. Belikov, V. Bogdanov, and Alexander Scherbatiy. Cognitive module networks for grounded reasoning. pp. 148–158,

    arXiv

  70. [87]

    Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A

    URL https://arxiv.org/abs/2307.16789. Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A. Crook, Y. Xu, Xin Luna Dong, Christos Faloutsos, Lei Li, Babak Damavandi, and Seungwhan Moon. Snapntell: Enhancing entity-centric visual question answering with retrieval augmented multimo...

    arXiv

  71. [88]

    Alleviating hallucination in large vision-language models with active retrieval augmentation.ArXiv, abs/2408.00555,

    Xiaoye Qu, Qiyuan Chen, Wei Wei, Jiashuo Sun, and Jianfeng Dong. Alleviating hallucination in large vision-language models with active retrieval augmentation.ArXiv, abs/2408.00555,

    arXiv

  72. [89]

    Ksf-st: Video captioning based on key semantic frames extraction and spatio-temporal attention mechanism.2020 International Wireless Communications and Mobile Computing (IWCMC), pp

    Zhaowei Qu, Luhan Zhang, Xiaoru Wang, Bingyu Cao, Yueli Li, and Fu Li. Ksf-st: Video captioning based on key semantic frames extraction and spatio-temporal attention mechanism.2020 International Wireless Communications and Mobile Computing (IWCMC), pp. 1388–1393,

    2020

  73. [90]

    Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, and Tanuja Ganu

    URLhttps://arxiv.org/abs/2103.00020. Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, and Tanuja Ganu. Retrieval augmented chest x-ray report generation using openai gpt models.arXiv preprint, arXiv:2305.03660v1,

    arXiv

  74. [91]

    Vlc-bert: Visual question answering with contextualized commonsense knowledge.arXiv preprint, arXiv:2210.13626v1,

    Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. Vlc-bert: Visual question answering with contextualized commonsense knowledge.arXiv preprint, arXiv:2210.13626v1,

    arXiv

  75. [92]

    Outside knowledge visual question answering version 2.0.ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    BenjaminZ.Reichman, AnirudhS.Sundar, ChristopherRichardson, TamaraZubatiy, PrithwijitChowdhury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside knowledge visual question answering version 2.0.ICASSP 2023 - ...

    2023

  76. [93]

    Heather Riley and M

    URLhttps://arxiv.org/abs/1602.04938. Heather Riley and M. Sridharan. Integrating non-monotonic logical reasoning and inductive learning with deep learning for explainable visual question answering.Frontiers in Robotics and AI, 6,

    arXiv

  77. [94]

    Mrinmaya Sachan

    URLhttps://arxiv.org/abs/ 2103.11251. Mrinmaya Sachan. Towards Literate Artificial Intelligence. Phd thesis, Carnegie Mellon University,

    arXiv

  78. [95]

    Alireza Salemi, Juan Altmayer Pizzorno, and Hamed Zamani

    URL https://doi.org/10.1184/R1/11898378.v1. Alireza Salemi, Juan Altmayer Pizzorno, and Hamed Zamani. A symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering.Proceedings of the 46th International ACM SIGIR Conference on Research an...

    doi

  79. [96]

    DustinSchwenk, ApoorvKhandelwal, ChristopherClark, KennethMarino, andRoozbehMottaghi

    URL https://arxiv.org/abs/2302.04761. DustinSchwenk, ApoorvKhandelwal, ChristopherClark, KennethMarino, andRoozbehMottaghi. A-okvqa: A benchmark for visual question answering using world knowledge.arXiv preprint, arXiv:2206.01718v1,

    arXiv

  80. [97]

    Slabaugh, Shanxin Yuan, and L

    Henry Senior, G. Slabaugh, Shanxin Yuan, and L. Rossi. Graph neural networks in vision-language image understanding: A survey.ArXiv, abs/2303.03761,

    arXiv

  81. [98]

    Muralikrishnna G

    URLhttps://arxiv.org/abs/1606.04422. Muralikrishnna G. Sethuraman, Ali Payani, Faramarz Fekri, and J. Clayton Kerce. Visual question answering based on formal logic.arXiv preprint, arXiv:2111.04785v1,

    arXiv

  82. [99]

    Unirag: Universal retrieval augmentation for multi-modal large language models.ArXiv, abs/2405.10311,

    Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal retrieval augmentation for multi-modal large language models.ArXiv, abs/2405.10311,

    arXiv

  83. [100]

    Vcd: Knowledge base guided visual commonsense discovery in images.arXiv preprint, arXiv:2402.17213v1,

    Xiangqing Shen, Yurun Song, Siwei Wu, and Rui Xia. Vcd: Knowledge base guided visual commonsense discovery in images.arXiv preprint, arXiv:2402.17213v1,

    arXiv

  84. [101]

    Reasoning over vision and language: Exploring the benefits of supplemental knowledge.arXiv preprint, arXiv:2101.06013v1,

    Violetta Shevchenko, Damien Teney, Anthony Dick, and Anton van den Hengel. Reasoning over vision and language: Exploring the benefits of supplemental knowledge.arXiv preprint, arXiv:2101.06013v1,

    arXiv

  85. [102]

    Keisuke Shirai, C. C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision-language interpreter for robot task planning.2024 IEEE International Conference on Robotics and Automati...

    2024

  86. [103]

    Singh, Anand Mishra, Shashank Shekhar, and Anirban Chakraborty

    33 A. Singh, Anand Mishra, Shashank Shekhar, and Anirban Chakraborty. From strings to things: Knowledge- enabled vqa model that can read and reason.2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4601–4611,

    2019

  87. [104]

    Liu, Yang Yang, Xuequn Shang, and Mingxuan Sun

    Lingyun Song, Jianao Li, J. Liu, Yang Yang, Xuequn Shang, and Mingxuan Sun. Answering knowledge-based visual questions via the exploration of question purpose.Pattern Recognit., 133:109015, 2022a. Zijie Song, Zhenzhen Hu, and Richang Hong. Efficient and self-adaptive rationale...

    2023

  88. [105]

    Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting

    URLhttps://arxiv.org/abs/1612.03975. Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder.arXiv preprint, arXiv:2406.09949v2,

    arXiv

  89. [106]

    Sk-vqa: Synthetic knowledge generation at scale for training context-augmented multimodal llms.ArXiv, abs/2406.19593,

    Xin Su, Man Luo, Kris W Pan, Tien Pei Chou, Vasudev Lal, and Phillip Howard. Sk-vqa: Synthetic knowledge generation at scale for training context-augmented multimodal llms.ArXiv, abs/2406.19593,

    arXiv

  90. [107]

    Learning visual knowledge memory networks for visual question answering.arXiv preprint, arXiv:1806.04860v1,

    Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, and Jianguo Li. Learning visual knowledge memory networks for visual question answering.arXiv preprint, arXiv:1806.04860v1,

    arXiv

  91. [108]

    Modular visual question answering via code genera- tion

    Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code genera- tion. arXiv preprint, arXiv:2306.05392v1,

    arXiv

  92. [109]

    Vipergpt: Visualinferenceviapythonexecutionforreasoning

    DídacSurís, SachitMenon, andCarlVondrick. Vipergpt: Visualinferenceviapythonexecutionforreasoning. arXiv preprint, arXiv:2303.08128v1,

    arXiv

  93. [110]

    URLhttps://arxiv.org/abs/1803.05355. P. Torino, Elena Baralis, and Dott. Andrea Pasini. Semantics-aware vqa a scene-graph-based approach to enable commonsense reasoning

    arXiv

  94. [111]

    Understand, compose and respond - answering visual questions by a composition of abstract procedures.arXiv preprint, arXiv:1810.10656v1,

    Ben Zion Vatashsky and Shimon Ullman. Understand, compose and respond - answering visual questions by a composition of abstract procedures.arXiv preprint, arXiv:1810.10656v1,

    arXiv

  95. [112]

    Noc-rek: Novel object captioning with retrieved vocabulary from external knowledge.arXiv preprint, arXiv:2203.14499v1,

    Duc Minh Vo, Hong Chen, Akihiro Sugimoto, and Hideki Nakayama. Noc-rek: Novel object captioning with retrieved vocabulary from external knowledge.arXiv preprint, arXiv:2203.14499v1,

    arXiv

  96. [113]

    Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, J

    URLhttps://arxiv.org/abs/1908.07125. Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, J. B. Tenenbaum, and Chuang Gan. Sok-bench: A situated video reasoning benchmark with aligned open- world knowledge. 2024 IEEE/CVF Conference on Compute...

    1908 arXiv

  97. [114]

    Visually-augmented language modeling.ArXiv, abs/2205.10178, 2022b

    Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Visually-augmented language modeling.ArXiv, abs/2205.10178, 2022b. Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Composi- tional 4d dynamic ...

    arXiv

  98. [115]

    Jiahui Wei, Zhixin Li, Jianwei Zhu, and Huifang Ma

    URL https://arxiv.org/abs/2201.11903. Jiahui Wei, Zhixin Li, Jianwei Zhu, and Huifang Ma. Enhance understanding and reasoning ability for image captioning. Applied Intelligence, 53:2706–2722,

    arXiv

  99. [116]

    Multimodal reranking for knowledge-intensive visual question answering.ArXiv, abs/2407.12277,

    35 HaoyangWen, HongleiZhuang, HamedZamani, AlexanderHauptmann, andMichaelBendersky. Multimodal reranking for knowledge-intensive visual question answering.ArXiv, abs/2407.12277,

    arXiv

  100. [117]

    Image captioning and visual question answering based on attributes and external knowledge.arXiv preprint, arXiv:1603.02814v2,

    Qi Wu, Chunhua Shen, Anton van den Hengel, Peng Wang, and Anthony Dick. Image captioning and visual question answering based on attributes and external knowledge.arXiv preprint, arXiv:1603.02814v2,

    arXiv

  101. [118]

    Resolving zero-shot and fact-based visual question answering via enhanced fact retrieval.IEEE Transactions on Multimedia, 26:1790–1800, 2024a

    Sen Wu, Guoshuai Zhao, and Xueming Qian. Resolving zero-shot and fact-based visual question answering via enhanced fact retrieval.IEEE Transactions on Multimedia, 26:1790–1800, 2024a. Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, V. Ioannidis, Karthik Su...

    arXiv

  102. [119]

    Reverse image retrieval cues parametric memory in multi- modal llms

    Jialiang Xu, Michael Moor, and Jure Leskovec. Reverse image retrieval cues parametric memory in multi- modal llms. arXiv preprint, arXiv:2405.18740v1, 2024a. Ruinian Xu, Hongyi Chen, Yunzhi Lin, and Patricio A. Vela. Sgl: Symbolic goal learning in a hybrid, modular framework f...

    arXiv

  103. [120]

    Rs-agent: Automating remote sensing tasks through intelligent agents.arXiv preprint, arXiv:2406.07089v1, 2024b

    Wenjia Xu, Zijian Yu, Yixu Wang, Jiuniu Wang, and Mugen Peng. Rs-agent: Automating remote sensing tasks through intelligent agents.arXiv preprint, arXiv:2406.07089v1, 2024b. Keyang Xuan, Li Yi, Fan Yang, Ruochen Wu, Y. Fung, and Heng Ji. Lemma: Towards lvlm-enhanced multimodal...

    arXiv

  104. [121]

    Echosight: Advancing visual-language models with wiki knowledge

    Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowledge. ArXiv, abs/2407.12735,

    arXiv

  105. [122]

    Tenenbaum, and Chuang Gan

    Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David Cox, J. Tenenbaum, and Chuang Gan. Object- centric diagnosis of visual reasoning.ArXiv, abs/2012.11587,

    2012 arXiv

  106. [123]

    Doraemongpt: Toward under- standing dynamic scenes with large language models (exemplified as a video agent)

    36 Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward under- standing dynamic scenes with large language models (exemplified as a video agent). arXiv preprint, arXiv:2401.08392v4,

    arXiv

  107. [124]

    Breaking shortcuts by masking for robust visual reasoning

    Keren Ye, Mingda Zhang, and Adriana Kovashka. Breaking shortcuts by masking for robust visual reasoning. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3519–3529,

    2021

  108. [125]

    Tenenbaum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint, arXiv:1910.01442v2,

    1910 arXiv

  109. [126]

    Multi-clue reasoning with memory augmentation for knowledge-based visual question answering.ArXiv, abs/2312.12723,

    Chengxiang Yin, Zhengping Che, Kun Wu, Zhiyuan Xu, and Jian Tang. Multi-clue reasoning with memory augmentation for knowledge-based visual question answering.ArXiv, abs/2312.12723,

    arXiv

  110. [127]

    Cross-modalknowledgereasoning for knowledge-based visual question answering.ArXiv, abs/2009.00145,

    J.Yu, ZihaoZhu, YujingWang, WeifengZhang, YueHu, andJianlongTan. Cross-modalknowledgereasoning for knowledge-based visual question answering.ArXiv, abs/2009.00145,

    2009 arXiv

  111. [128]

    Fung, Hao Peng, and Heng Ji

    Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. Craft: Customizing llms by creating and retrieving from specialized toolsets.arXiv preprint, arXiv:2309.17428v2, 2023a. ZhengYuan, QiaoJin, ChuanqiTan, ZhengyunZhao, HongyiYuan, FeiHuang, andSongfangHuan...

    arXiv

  112. [129]

    Vikser: Visual knowledge-driven self-reinforcing reasoning framework

    Chunbai Zhang, Chao Wang, Yang Zhou, and Yan Peng. Vikser: Visual knowledge-driven self-reinforcing reasoning framework. arXiv preprint, arXiv:2502.00711v1, 2025a. Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. Can vision-language models be a good guesser? explori...

    2024 arXiv

  113. [130]

    Jiang, Zequn Yang, T

    Shunyu Zhang, X. Jiang, Zequn Yang, T. Wan, and Zengchang Qin. Reasoning with multi-structure com- monsense knowledge in visual dialog. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4599–4608, 2022a. 37 Xiaohan Zhang, Yan Ding, S. A...

    2022 arXiv

  114. [131]

    Query and attention augmentation for knowledge-based explainable reasoning

    Yifeng Zhang, Ming Jiang, and Qi Zhao. Query and attention augmentation for knowledge-based explainable reasoning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15555– 15564, 2022c. Yifeng Zhang, Shi Chen, and Qi Zhao. Toward multi-granularity...

    2022

  115. [132]

    org/CorpusID:272430309

    URL https://api.semanticscholar. org/CorpusID:272430309. Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Y. Liu. Interpretable detection of out-of-context misinformation with neural-symbolic-enhanced large multimodal model. 2023f. Zefan Zhang, Yi Ji, and Chunping Liu. Knowle...

    2023 arXiv

  116. [133]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents.arXiv preprint, arXiv:2307.13854v4,

    arXiv

  117. [134]

    Configurable graph reasoning for visual relationship detection.IEEE Transactions on Neural Networks and Learning Systems, 33:117–129, 2020a

    38 Yi Zhu, Xiwen Liang, Bingqian Lin, Qixiang Ye, Jianbin Jiao, Liang Lin, and Xiaodan Liang. Configurable graph reasoning for visual relationship detection.IEEE Transactions on Neural Networks and Learning Systems, 33:117–129, 2020a. Zihao Zhu. From shallow to deep: Compositi...

    arXiv

  118. [135]

    Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu

    Zihao Zhu, J. Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering.ArXiv, abs/2006.09073, 2020b. M. Ziaeefard and F. Lécué. Towards knowledge-augmented visual question answering. pp. 1863–1873,

    2006 arXiv

  119. [2016]

    Answering image riddles using vision and reasoning through probabilistic soft logic.arXiv preprint, arXiv:1611.05896v1,

    Somak Aditya, Yezhou Yang, Chitta Baral, and Yiannis Aloimonos. Answering image riddles using vision and reasoning through probabilistic soft logic.arXiv preprint, arXiv:1611.05896v1,

    arXiv

  120. [2017]

    Shabbirhussain Bhaisaheb, Shubham Paliwal, Rajaswa Patil, Manasi S

    URLhttps://arxiv.org/abs/1711.03902. Shabbirhussain Bhaisaheb, Shubham Paliwal, Rajaswa Patil, Manasi S. Patwardhan, L. Vig, and Gautam M. Shroff. Program synthesis for complex qa on charts via probabilistic grammar based filtered iterative back- translation. pp. 2456–2470,

    arXiv

  121. [2018]

    Integratingknowledgeandreasoninginimageunderstanding

    SomakAditya, YezhouYang, andChittaBaral. Integratingknowledgeandreasoninginimageunderstanding. IJCAI 2019,

    2019

  122. [2019]

    Qingxing Cao, Bailin Li, Xiaodan Liang, and Liang Lin

    URLhttps://arxiv.org/abs/ 1902.02960. Qingxing Cao, Bailin Li, Xiaodan Liang, and Liang Lin. Explainable high-order visual question reasoning: A new benchmark and knowledge-routed network.ArXiv, abs/1909.10128,

    1902 arXiv

  123. [2020]

    URL https://doi.org/10.1007/978-3-030-39197-3_4

    doi: 10.1007/978-3-030-39197-3_4. URL https://doi.org/10.1007/978-3-030-39197-3_4. Ghada Besbes, H. B. Zghal, and H. Ghézala. An ontology-driven visual question-answering framework.2015 19th International Conference on Information Visualisation, pp. 127–132,

    2015 doi

  124. [2021]

    Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint, arXiv:2407.10956v1,

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu...

    arXiv

  125. [2022]

    Wenbin An, Feng Tian, Jiahao Nie, Wenkai Shi, Haonan Lin, Yan Chen, Qianying Wang, Y

    URL https://arxiv.org/abs/2204.14198. Wenbin An, Feng Tian, Jiahao Nie, Wenkai Shi, Haonan Lin, Yan Chen, Qianying Wang, Y. Wu, Guang Dai, and Ping Chen. Knowledge acquisition disentanglement for knowledge-based visual question answering with large language models.ArXiv, abs/2...

    arXiv

  126. [2023]

    Barezi and Parisa Kordjamshidi

    Elham J. Barezi and Parisa Kordjamshidi. Find the gap: Knowledge base reasoning for visual question answering. arXiv preprint, arXiv:2404.10226v1,

    arXiv

  127. [2024]

    Lawrence Zitnick, Dhruv Batra, and Devi Parikh

    Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering.arXiv preprint, arXiv:1505.00468v7,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.