Pith. sign in

REVIEW 4 major objections 6 minor 41 references

GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a four-stage zero-shot pipeline outperforms larger end-to-end VQA models on three standard benchmarks without task-specific training.

desk verdict A competent modular zero-shot KB-VQA recipe with mostly plausible numbers, but the paper's own Table 5 contradicts its 'top three captions' claim and the tuning-free framing hides test-set hyperparameter selection. read the letter →

arxiv 2505.19354 v1 pith:SJR6WOMJ submitted 2025-05-25 cs.CL cs.CV

Mohammad Mahdi Moradi , Sudhir Mudur This is my paper · ORCID
classification cs.CLcs.CV
keywords knowledge-basedvisualquestionansweringKB-VQAzero-shotVQAgroundingcaptionfilteringlargelanguagemodelspromptengineeringOK-VQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that knowledge-based visual question answering can be done without end-to-end multimodal training: a four-stage pipeline of frozen pretrained components converts the image into question-relevant captions and example QA pairs, then lets a general-purpose LLM produce the answer. The central claim is that this modular grounding-and-filtering approach outperforms prior zero-shot systems and even some much larger models, with reported scores of 54.57 on OK-VQA, 53.87 on A-OKVQA, and 67.96 on VQAv2 using a 47B framework. The authors argue that grounding question keywords to image regions, generating multiple captions with two vision-language models, filtering by semantic relevance to the question, and adding synthetic QA pairs all contribute to the gain. If true, the result matters because it suggests the quality of the auxiliary text given to the LLM, rather than model scale or task-specific training, is the main lever for zero-shot KB-VQA.

What carries the argument

The central mechanism is the four-stage grounding-and-filtering pipeline that turns an image-and-question pair into a compact, question-aware prompt. Stage one uses KeyBERT-selected keywords as the prompt for Grounding DINO, an open-set object detector, to extract question-relevant regions; counting questions are answered from the detected object count. Stage two gathers captions from two complementary vision-language models (LLaVA and InstructBLIP), ranks each caption against a distilled version of the question using all-MiniLM-L6-v2 embeddings, and keeps the three most relevant. Stage three generates two synthetic (question, answer) pairs from those captions with Llama-3-8B-Instruct. Stage four sends the original question, the top three captions, and the two QA pairs to the same LLM as a single structured prompt. The load-bearing novelty is the grounding step, which gives the captions a precise spatial referent, and the question-content-driven filtering, which strips away irrelevant caption text.

What would settle it

Run the fixed pipeline, with all thresholds frozen, on a new split of OK-VQA or on a different knowledge-based VQA benchmark; if the accuracy falls toward the no-caption baseline of about 18.8 on OK-VQA, the reported gains would look like in-sample tuning. A cheaper check is to repeat Tables 3 to 6 on a held-out validation set and see whether the same design choices survive.

Watch

Extended reading notes

Core claim

GC-KBVQA's core claim is that the quality of the auxiliary text presented to the LLM, not the size of the model, is the main lever in zero-shot knowledge-based VQA. The paper proposes to extract question keywords with KeyBERT, localize the regions they refer to with Grounding DINO, generate multiple captions for those regions with two vision-language models (LLaVA and InstructBLIP), filter the captions by semantic similarity to a distilled version of the question, derive two caption-driven QA pairs with Llama-3-8B-Instruct, and finally prompt the same LLM with the question, the top three captions, and the two QA pairs. The authors report that this four-stage pipeline outperforms previous zero-shot methods on OK-VQA, A-OKVQA, and VQAv2, and that routing counting questions directly to the object detector improves counting as well. Their ablations indicate that each stage, grounding, dual captioning, filtering, and QA-pair prompting, contributes independently to the final score.

Load-bearing premise

The fragile premise is that the reported zero-shot scores are not artifacts of having chosen every design decision on the same test sets: the thresholds and module configurations (top-3 captions, two QA pairs, KeyBERT score 0.4, Grounding DINO confidence 0.25, overlap 0.9, dual captioner) were selected by comparing accuracy on OK-VQA, A-OKVQA, and VQAv2 themselves, with no held-out validation set.

Editorial extensions

If this is right

  • Zero-shot KB-VQA can improve without end-to-end multimodal training or in-context examples, lowering training cost and deployment complexity.
  • A modular pipeline of frozen components can beat larger end-to-end models such as Flamingo-80B and BLIP-2 on OK-VQA and VQAv2.
  • The answer-generation LLM can be replaced by a different model of similar scale, such as Mistral-7B or TinyLlama-1B, with only a small accuracy drop, showing the pipeline itself drives the gains.
  • Counting questions are best handled by the grounded detector rather than by LLM reasoning over captions.
  • Caption relevance beats caption volume: using more than three captions can dilute accuracy, especially on A-OKVQA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The absence of a held-out validation set means the reported margins may not survive on unseen data; the paper's own ablations select all design choices directly on the test sets that produce the final scores.
  • The synthetic QA pairs function as self-generated few-shot exemplars, so the same distill-ground-generate-prompt pattern could transfer to other open-ended multimodal tasks such as diagram reasoning or screenshot understanding.
  • The method's ceiling is tied to open-vocabulary detection: questions that cannot be expressed as detectable objects, including temporal, relational, and numerical-estimation questions, are the paper's acknowledged failure cases in its Section 6.
  • A testable extension is to make the fixed thresholds adaptive per question rather than global constants, which could improve robustness on out-of-distribution questions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GC-KBVQA, a modular, zero-shot visual question answering framework that combines four frozen pretrained components: (1) keyword-guided visual grounding with Grounding DINO, where KeyBERT extracts question keywords and a LLaMA-3-8B-Instruct classifier routes counting questions to a direct detection branch; (2) question-aware caption generation using LLaVA and InstructBLIP, followed by a MiniLM-based relevance filter that keeps the top three captions; (3) generation of two caption-driven QA pairs by Llama-3-8B; and (4) final answer generation by Llama-3-8B from a prompt containing captions and QA pairs. The authors report state-of-the-art zero-shot accuracies of 54.57 on OK-VQA, 53.87 on A-OKVQA, and 67.96 on VQAv2 with a 47B parameter framework, and they provide ablations in Tables 2–6 over LLM choice, grounding prompt design, caption generator combination, caption count, and prompt composition. The central claim is that the pipeline outperforms larger zero- and few-shot systems without end-to-end training or in-context examples.

Significance. If the reported results are robust, the paper would make a useful empirical contribution: it demonstrates that a composition of off-the-shelf frozen components—Grounding DINO, LLaVA, InstructBLIP, and Llama-3—can achieve competitive KB-VQA accuracy with fewer parameters than end-to-end models, and the ablation suite is a valuable map of design choices. The claim is conditional, however, because the headline numbers are produced by a configuration whose hyperparameters were selected on the same test sets used for the comparison, and because one of the key ablations (Table 5) internally contradicts the text's justification. The counting-module claim, a stated contribution, has no quantitative support. The paper is transparent about its limitations and reports machine-checkable benchmark numbers, but the absence of a held-out validation set and the unresolved contradiction prevent the results from being accepted as a clean zero-shot evaluation.

major comments (4)
  1. [§4.2, §4.3–§4.7, Tables 1 and 3–6] The headline claim of zero-shot evaluation without task-specific training is weakened by test-set-driven hyperparameter selection. The manuscript reports no held-out validation split. The KeyBERT threshold (0.4), Grounding DINO confidence (0.25), bounding-box overlap threshold (0.9), region expansion factor, number of captions (3), number of QA pairs (2), and the choice of dual caption generators are all motivated by ablation results computed on the same OK-VQA, A-OKVQA, and VQAv2 test sets. Selecting a configuration that maximizes accuracy on the test set and then reporting those numbers as the zero-shot performance overstates generalization. Please either re-run the selection on a validation split and report the resulting test-set numbers, or explicitly characterize the results as test-set-tuned and provide an independent evaluation (e.g., a fourth benchmark or cross-validation) to demonstrate that the choices generalize.
  2. [§4.6, Table 5] The text states that 'The use of the top three captions yielded the best results in all datasets,' but Table 5 reports OK-VQA accuracy of 54.57 for 3 captions, 55.40 for 4 captions, and 56.47 for 5 captions. On OK-VQA, 3 captions is not the best; it is best only on A-OKVQA and VQAv2. The choice of 3 captions sacrifices 0.83–1.90 accuracy points on OK-VQA relative to 4–5 captions, gaining larger improvements on the other two benchmarks. This is a direct instance of selecting a hyperparameter on the same test sets used for the main comparison, and it contradicts the stated rationale that 3 captions best balances redundancy and utility everywhere. Please correct the text, disclose the trade-off, and justify the choice using a validation set rather than aggregate test-set performance.
  3. [§3.1, §6, Appendix A] The counting contribution is a stated major innovation ('tackles counting challenges better than all previous works'), but the paper provides no quantitative evaluation of the counting module. There is no ablation isolating the LLaMA-based counting classification branch or the use of Grounding DINO object counts versus the standard pipeline, no counting-specific accuracy on OK-VQA or another benchmark, and no error analysis of the classifier. The appendix demonstrates only a small set of manually selected counting successes, which is not evidence of a systematic improvement. Please add a quantitative comparison that turns the counting module on and off, and report counting-disjoint accuracy or a counting-specific subset.
  4. [Abstract and Conclusion] The manuscript states 'Our code will be made public,' but no code or executable configuration is provided, and the appendix does not include the full prompt strings used for the QA-pair generator and answer generator beyond the short templates in §3.3–§3.4. Without the exact prompts, generation hyperparameters (e.g., temperature, max tokens), or code, the reported numbers cannot be reproduced or verified. Please release the code at the time of revision or provide a complete specification in an appendix.
minor comments (6)
  1. [Table 4] The table header contains a typo: 'InstrcutBLIP' should be 'InstructBLIP'; the same typo appears in Section 4.5 as 'InstrcutBLIP'.
  2. [§3.1] The 'region expansion factor' is described as proportionately enlarging bounding boxes, but no default value is given and no ablation is provided for this parameter. Please specify the factor and include an ablation or justify the choice.
  3. [Table 2] The 'framework size' column is not defined. Please explain how the sizes are computed (e.g., sum of all component parameters) and whether the LLM size is included in every row, since the caption generators and Grounding DINO presumably remain fixed.
  4. [§2] There is a typo in the Related Works section: 'KB-.VQA' should be 'KB-VQA'.
  5. [Appendix A] Several appendix examples list a 'Question Concept' that differs from the KeyBERT-extracted keywords; the method section only describes the 'Question Concept' as a distilled question via Llama-3-8B, but the appendix does not clearly connect the two. Please clarify the relationship between the 'Question Concept' in the examples and the content extraction step in §3.2.
  6. [§4.1] The paper does not report whether the LLM and caption generators are run deterministically or with sampling; if sampling is used, please report variance or multiple runs.

Circularity Check

2 steps flagged · score 6.0 of 10

Benchmark circularity: the final zero-shot pipeline configuration is selected on the same OK-VQA/A-OKVQA/VQAv2 test sets whose headline numbers are then reported as state-of-the-art; Table 5 contradicts the paper's 'top-3 best in all datasets' claim.

  1. fitted input called prediction [Section 3.2 (caption count); Section 4.6/Table 5; Section 4.2/Table 1]
    "The use of the top three captions yielded the best results in all datasets. This configuration effectively balances reducing redundancy and maximizing contextual utility for the LLM."

    The number of captions is treated as a discovered optimum ('the top 3 work best'), but Table 5 shows this is not true on OK-VQA: 4 captions score 55.40 and 5 captions score 56.47, both above the 54.57 of 3 captions. The ablation is run on the same OK-VQA, A-OKVQA, and VQAv2 test sets later used for the headline zero-shot comparison in Table 1, and the reported 'ours 47B 54.57 53.87 67.96' row is exactly the 3-caption row from Table 5. The configuration is therefore a test-set compromise selected by comparing accuracy on those exact benchmarks, and the subsequent SOTA claim is a selection result rather than an independent zero-shot prediction.

  2. fitted input called prediction [Section 4.7/Table 6; Section 4.2/Table 1]
    "The integration of instructions, captions, and QA pairs achieved the best results in all datasets."

    The answer-generator prompt design is chosen by evaluating four variants on the same three test sets in Table 6, and the winning variant ('Instruction+Captions+QA Pairs', 54.57/53.87/67.96) is the identical configuration reported as zero-shot SOTA in Table 1. Because the prompt composition was selected by maximizing accuracy on those very test sets, the final reported comparison is not an out-of-sample zero-shot evaluation; the 'best' prompt is a fitted pipeline choice renamed as an independent result.

full rationale

The paper contains no equation-level derivation that reduces to itself, and there are no load-bearing self-citations: the method is a modular composition of externally pretrained components (Grounding DINO, LLaVA/InstructBLIP, all-MiniLM-L6-v2, LLaMA-3-8B-Instruct). The central claim is empirical: state-of-the-art zero-shot VQA numbers on OK-VQA, A-OKVQA, and VQAv2. The circularity is evaluative rather than formal. Several pipeline choices—Grounding DINO prompt type, caption-generator set, number of captions, and answer-prompt composition—are selected by comparing accuracy on the very test sets that are later reported as zero-shot SOTA. Section 4.6 contains an internal contradiction: it claims the top three captions 'yielded the best results in all datasets,' but Table 5 shows 4 and 5 captions beat 3 captions on OK-VQA. The final Table 1 row is exactly the 3-caption row of Table 5, so the headline configuration is a test-set compromise, not a universal optimum. This is the fitted-input-called-prediction pattern: a pipeline hyperparameter is fit to the test benchmarks, and the resulting benchmark score is presented as a zero-shot prediction. The framework's individual components remain externally grounded, so the circularity is partial rather than definitional; score 6 reflects that one or more reported predictions are substantially determined by the same test-set values used to select the configuration.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The headline result depends on a chain of external pretrained models plus several hand-set thresholds. The most consequential free parameters are the thresholds and counts selected on the test benchmarks, because they directly influence the reported accuracy. The counting branch adds an unvalidated assumption that the Llama-3-8B classifier and Grounding DINO counts are reliable. No invented entities are introduced.

free parameters (7)
  • KeyBERT relevance threshold = 0.4
    Words and phrases with cosine similarity above 0.4 are kept as Grounding DINO prompts; no sensitivity analysis is provided.
  • Grounding DINO confidence threshold = 0.25
    Bounding boxes below confidence 0.25 are discarded; threshold is chosen by hand without sensitivity analysis.
  • Bounding box overlap threshold = 0.9
    Overlapping detections with intersection above 0.9 are merged to the larger box; hand-set value.
  • Region expansion factor = not specified
    Detected regions are proportionately enlarged in Section 3.1, but the enlargement factor is never quantified, making it an implicit free parameter.
  • Number of top captions = 3
    Selected by ablation on the test benchmarks (Table 5); top 3 is called best, though OK-VQA improves to 55.40 with 4 and 56.47 with 5 captions.
  • Number of QA pairs = 2
    Chosen in Section 3.3 because two QA pairs are sufficient; no ablation table for this choice is shown.
  • Caption generator combination = LLaVA + InstructBLIP
    Dual captioners were selected by Table 4 test-set accuracy; the combination improves over each single generator.
assumptions (5)
  • domain assumption Semantic similarity from all-MiniLM-L6-v2 captures question-caption relevance.
    The filtering step relies entirely on cosine similarity in embedding space to rank captions; no qualitative validation that top-3 are the most relevant is provided.
  • ad hoc to paper Llama-3-8B-Instruct can correctly classify counting versus non-counting questions and generate useful QA pairs.
    No accuracy or failure analysis of the classifier or QA-pair generator is reported; this assumption underpins the counting branch and the QA-pair augmentation.
  • domain assumption Off-the-shelf captioners LLaVA and InstructBLIP produce descriptions faithful to image content.
    The text-only answer generator cannot verify captions against the image; hallucinated captions would propagate directly to answers.
  • domain assumption The pretrained Llama-3-8B answer generator uses the provided captions and QA pairs rather than relying only on parametric knowledge.
    Table 6 suggests captions and QA pairs matter, but there is no mechanism analysis or counterfactual evaluation to confirm the prompts drive the answer.
  • domain assumption Standard benchmark test sets are representative and correctly labeled.
    The framework uses OK-VQA, A-OKVQA, and VQAv2 test sets as ground truth for both final evaluation and hyperparameter selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance." pith.science (2026). https://pith.science/paper/SJR6WOMJ

@misc{pith2026250519354,
  author       = {Pith},
  title        = {Pith review of: GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJR6WOMJ}},
  note         = {Machine review of arXiv:2505.19354}
}
read the original abstract

Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image. Early methods relied on explicit knowledge bases to provide this auxiliary information. Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources. While KB-VQA methods have demonstrated promising results, their potential remains constrained as the auxiliary text provided may not be relevant to the question context, and may also include irrelevant information that could misguide the answer predictor. We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA), which enables LLMs to effectively perform zero-shot VQA tasks without the need for end-to-end multimodal training. Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information. This is combined with knowledge from external sources to create highly informative prompts for the LLM. GC-KBVQA can address a variety of VQA tasks, and does not require task-specific fine-tuning, thus reducing both costs and deployment complexity by leveraging general-purpose, pre-trained LLMs. Comparison with competing KB-VQA methods shows significantly improved performance. Our code will be made public.

Figures

Figures reproduced from arXiv: 2505.19354 by the authors.

Figure 1
Figure 1. The architecture of GC-KBVQA leverages four pre-trained modules: (1) Question-relevant region [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 19 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716--23736

    2022

  4. [4]

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077--6086

    2018

  5. [5]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

    2015

  6. [6]

    Elham J Barezi and Parisa Kordjamshidi. 2024. Find the gap: Knowledge base reasoning for visual question answering. arXiv preprint arXiv:2404.10226

    work page Pith review arXiv 2024

  7. [7]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  8. [8]

    Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. 2019. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1989--1998

    work page 2019

Show all 41 references
  1. [9]

    Rui Cao and Jing Jiang. 2024. Knowledge generation for zero-shot knowledge-based vqa. arXiv preprint arXiv:2402.02541

    2024 arXiv

  2. [10]

    Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Enabling multimodal generation on clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.06386

    2022 arXiv

  3. [11]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    2023 arXiv

  4. [12]

    Ana Cl \'a udia Akemi Matsuki de Faria, Felype de Castro Bastos, Jos \'e Victor Nogueira Alves da Silva, Vitor Lopes Fabris, Valeska de Sousa Uchoa, D \'e cio Gon c alves de Aguiar Neto, and Claudio Filipi Goncalves dos Santos. 2023. Visual question answering: A survey on tech...

    2023 arXiv

  5. [13]

    Y Du, J Li, T Tang, WX Zhao, and JR Wen. 2023. Zero-shot visual question answering with language model feedback. arxiv preprint. arXiv preprint arXiv:2305.17006

    2023 arXiv

  6. [14]

    Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847

    2016 arXiv

  7. [15]

    Carlo Galli, Nikolaos Donos, and Elena Calciolari. 2024. Performance of 4 pre-trained sentence transformer models in the semantic query of a systematic review dataset on peri-implantitis. Information, 15(2):68

    2024

  8. [16]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  9. [17]

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pat...

    2023

  10. [18]

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. 2022. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846

    2022 arXiv

  11. [19]

    Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699

    2022 arXiv

  12. [20]

    Kushal Kafle and Christopher Kanan. 2017. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3--20

    2017

  13. [21]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota

    2019

  14. [22]

    Zaid Khan, Vijay Kumar BG, Samuel Schulter, Manmohan Chandraker, and Yun Fu. 2024. Exploring question decomposition for zero-shot vqa. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Yunshi Lan, Xiang Li, Xin Liu, Yang Li, Wei Qin, and Weining Qian. 2023. Improving zero-shot visual question answering via large language models with reasoning question prompts. In Proceedings of the 31st ACM International Conference on Multimedia, pages 4389--4400

    2023

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  17. [25]

    Miaoyu Li, Haoxin Li, Zilin Du, and Boyang Li. 2024. Diversify, rationalize, and combine: Ensembling multiple qa strategies for zero-shot knowledge-based vqa. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1552--1566

    2024

  18. [26]

    Cheng Liu, Chao Wang, Yan Peng, and Zhixu Li. 2024 a . Zvqaf: Zero-shot visual question answering with feedback from large language models. Neurocomputing, 580:127505

    2024

  19. [27]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  20. [28]

    Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211--226

    2004

  21. [29]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and 1 others. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499

    2023 arXiv

  22. [30]

    Lin Ma, Zhengdong Lu, and Hang Li. 2016. Learning to answer questions from image using convolutional neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30

    2016

  23. [31]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

    2019

  24. [32]

    Pengpeng Qiang, Hongye Tan, Xiaoli Li, Dian Wang, Ru Li, Xinyi Sun, Hu Zhang, and Jiye Liang. 2025. Enhancing few-shot kb-vqa with panoramic image captions guided by large language models. Neurocomputing, 623:129373

    2025

  25. [33]

    Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023. Vlc-bert: Visual question answering with contextualized commonsense knowledge. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1155--1165

    2023

  26. [34]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146--162. Springer

    2022

  27. [35]

    Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 14974--14983

    2023

  28. [36]

    Prafull Sharma and Yingbo Li. 2019. Self-supervised contextual keyword and keyphrase retrieval with self-labelling

    2019

  29. [37]

    Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. 2022. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773

    2022 arXiv

  30. [38]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, and 1 others. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521

    2023 arXiv

  31. [39]

    Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022. Multi-modal answer validation for knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2712--2721

    2022

  32. [40]

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 3081--3089

    2022

  33. [41]

    Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21--29

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.