Pith. sign in

REVIEW 3 major objections 5 minor 49 references

NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight pipeline of retrieved facts and LLM-written explanations lifts 240M-parameter VLMs to the level of 1–4B generative models on commonsense VQA.

desk verdict A useful engineering recipe for small VLMs with solid in-domain gains, but the headline parity claim overreaches on non-comparable baselines and the conclusion's numbers don't match its own tables. read the letter →

arxiv 2508.19724 v2 pith:QOD7QMQS submitted 2025-08-27 cs.CL cs.AI

Aritra Dutta , Swapnanil Mukherjee , Deepanway Ghosal , Somak Aditya This is my paper
classification cs.CLcs.AI
keywords commonsenseVQAsmallvision-languagemodelsretrieval-augmentedreasoningLLM-generatedexplanationsdenseretrievalnoise-robustlosshallucinationreductionparameter-efficientVLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the reason small vision-language models struggle at commonsense VQA is not their size but their lack of the commonsense knowledge that lies outside the image. It proposes NLKI, a modular pipeline that retrieves natural-language facts with a fine-tuned ColBERTv2, asks Llama-3.1-8B to condense those facts plus dense and region captions and detected objects into a short explanation, and feeds that explanation to sub-240M reader models (ViLT, VisualBERT, FLAVA). Across CRIC, AOKVQA, and e-SNLI-VE, the paper reports accuracy gains of several points and claims the small models then match or exceed 1–4B generative models like Qwen2-VL-2B and SmolVLM-2.5B, with additional gains from noise-robust losses such as symmetric and generalized cross-entropy. If true, this matters because it would make parameter-efficient commonsense reasoning practical in the 250M-parameter regime.

What carries the argument

The carrier of the argument is the 'Type-5' explanation: a 15–20 word sentence produced by Llama-3.1-8B from dense captions, region captions, detected objects, the question, and the single retrieved fact, then prepended to the question before the small VLM reads the image. Around it sits a modular pipeline—fine-tuned ColBERTv2 (a late-interaction neural retriever) over the OMCS commonsense corpus, and noise-robust losses (symmetric and generalized cross-entropy) that keep the reader from overfitting to mislabeled examples.

What would settle it

Fine-tune Qwen2-VL-2B and SmolVLM-2.5B on the same AOKVQA multiple-choice task with the same Type-5 explanation prepended, and score them exactly like the small readers; if their accuracy then meets or exceeds the small models, the parity claim fails. Also check whether the conclusion's 24→38 and 46→54 numbers are reproducible from the code release, since the tables list 33.45 and 47.85 as the best values.

Watch

Extended reading notes

Core claim

The paper's central claim is that NLKI closes most of the gap between 250M-parameter and 1–4B-parameter vision-language models on commonsense VQA. It argues that the bottleneck for small models is not parameter count but missing knowledge, and that a pipeline of dense retrieval plus LLM-generated, visually grounded explanations delivers that knowledge in a form the small reader can use. The authors report that on AOKVQA this raises ViLT from 24% to 38% and FLAVA from 46% to 54%, allowing them to match or exceed Qwen2-VL-2B and SmolVLM-2.5B, and that adding symmetric or generalized cross-entropy losses protects those gains from the 10–25% label noise the benchmarks contain. The paper's own ta

Load-bearing premise

The comparison of small fine-tuned models with larger zero-shot generative models is treated as an apples-to-apples measure of what each architecture can do; if the generative models were fine-tuned and scored with the same protocol, the reported 'match or exceed' could disappear.

Editorial extensions

If this is right

  • A 250M-parameter VLM with NLKI can reach AOKVQA accuracy that matches or exceeds zero-shot 1–4B generative models, at a fraction of the FLOPs, making commonsense VQA practical on modest hardware.
  • Knowledge quality is the lever: Type-5 explanations cut hallucination from 51% to 18.5%, and the highest-scoring explanations produce the largest accuracy gains.
  • More retrieved facts are worse than one: accuracy drops sharply as the number of concatenated facts increases, so knowledge integration should be sparse and targeted.
  • Noise-robust losses should be chosen per dataset: SCE helps most on heavily noisy CRIC and AOKVQA, CE+GCE on moderate noise, and plain CE remains best on clean e-SNLI-VE.
  • Retrieval quality can be improved by fine-tuning ColBERTv2 on the target commonsense VQA queries; fine-tuned ColBERTv2 beats FAISS, SBERT, and Stella on recall and relevance metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the same recipe should transfer to any small discriminative VLM: the paper's modular design means a better retriever or a cheaper explainer can be swapped in without retraining the reader, so replacing Llama-3.1-8B with a sub-1B explainer is a natural test of the accuracy/hallucination trade-off.
  • The paper's finding that one retrieved fact outperforms five suggests a token-budget view of knowledge augmentation: for 250M readers, coherence and relevance may matter more than coverage, and retrieval could be optimized for a single high-utility fact rather than top-k similarity.
  • The quoted AOKVQA jumps (24→38 and 46→54 in the conclusion) are larger than the best numbers shown in the tables (33.45 and 47.85); a reader should check which evaluation setup produces the headline numbers before treating them as the robust estimate.
  • The cleanest extension is to give the 1–4B generative models the same fine-tuning and explanation-augmented input; until that is done, 'match or exceed' should be understood as matching zero-shot generative evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes NLKI, a modular retrieval-augmented pipeline for small vision-language models (ViLT, VisualBERT, FLAVA, all under 240M parameters). The pipeline retrieves natural-language commonsense facts from OMCS using a fine-tuned ColBERTv2, prompts Llama-3.1-8B to generate Type-5 explanations conditioned on dense captions, region captions, detected objects, the question, and retrieved facts, and concatenates explanation+question to the VLM input. Experiments on CRIC, AOKVQA, and e-SNLI-VE compare baselines, retrieved-fact augmentation, LLM explanations, and noise-robust losses (SCE, GCE). The paper also benchmarks four generative VLMs under 4B parameters on AOKVQA and claims that NLKI-equipped sVLMs match or exceed these larger models. The main empirical claim—that the full NLKI pipeline plus noise-robust losses improves fine-tuned sVLMs over their no-knowledge baselines—is supported by Table 2, with gains of roughly +2 to +15 points depending on dataset and architecture.

Significance. If the internal gains are reproducible, this is a useful systems contribution: it shows that careful prompt construction, dense retrieval, and noise-robust losses can substantially improve small VLMs on commonsense VQA, with a detailed latency breakdown and public code/checkpoints. The component-level ablations (retriever comparison, explanation types, loss functions) are valuable for practitioners. However, the headline parity claim against 1–4B generative VLMs is not currently established, because the comparison in Section 7 uses zero-shot generative models scored by EM/BERTScore while the sVLMs are fine-tuned and scored as 4-way MCQ classifiers. The conclusion also contains numerical inconsistencies with the reported tables. The paper is best viewed as a strong empirical study of knowledge integration for small VLMs, pending a cleaner controlled comparison with generative baselines.

major comments (3)
  1. [Section 7, Table 4 vs. Table 2] The claim that NLKI-equipped sVLMs 'match or exceed' 1–4B generative VLMs rests on a non-commensurable comparison. Section 7 and Table 4 evaluate Qwen2-VL, SmolVLM, MiniCPM, and Phi3-Vision zero-shot, with Appendix E explicitly stating 'no external knowledge was provided during this benchmarking,' and score them by exact match and BERTScore/cosine threshold. The sVLM results in Table 2 are fine-tuned on training splits and evaluated as multiple-choice accuracy. Fine-tuning status, decoding protocol, and the metric differ simultaneously. A controlled comparison—e.g., fine-tuning the generative baselines on the same splits and evaluating them with the same accuracy protocol, or evaluating the sVLMs generatively with BERTScore—is required before the parity claim can be taken as established.
  2. [Section 10 vs. Tables 2/12] The conclusion states that NLKI lifts ViLT from 24 to 38 and FLAVA from 46 to 54 on AOKVQA, and that it 'outmatches 1–4B parameter generative baselines.' Neither set of numbers appears in Table 2 or Table 12, which report ViLT 24.01→33.45 and FLAVA 33.07→47.85 under SCE/CE. Moreover, Table 4 shows MiniCPM at 58.58 and Phi3-Vision at 53.24, both above the best NLKI FLAVA value of 47.85, so the 'outmatching' statement is not supported by the reported data. Please reconcile the text with the tables and qualify the claim to name the specific generative baselines that are actually exceeded.
  3. [Section 6.2, Table 1 and downstream Table 2] The choice of Type-5 as the strongest explanation variant is based on Table 1, which compares generated explanations to ground-truth explanations using BLEU/ROUGE/cosine on CRIC (and AOKVQA in Appendix H). The table does not state the split used for this comparison. If the explanation-quality scores were computed on the same test split that is later used for the downstream accuracy results in Table 2, then selecting Type-5 on the basis of those scores constitutes test-set selection and the downstream gains are optimistic. The split must be stated explicitly; if a validation split was used, that should be documented.
minor comments (5)
  1. [Abstract and Section 6.4] The abstract says the framework lifts end-to-end answer accuracy 'by up to 7%,' but Section 6.4 reports '+13% for FLAVA on AOKVQA' and Table 2 computes a 14.8-point gain for FLAVA (33.07→47.85). Please ensure the stated magnitude is consistent with the tables.
  2. [Table 2 caption] Table 2 does not specify which split is used for each dataset (train/val/test). Table 3 explicitly says CRIC test, and Table 4 says AOKVQA val, but Table 2 should state the split for all three datasets to allow replication.
  3. [Table 10] Several entries in Table 10 look like column misalignments, e.g., VisualBERT '2 Facts + Question' shows CRIC 20.25 while adjacent entries are near 73; FLAVA '2 Facts + Question' shows 31.51 in CRIC. Please check the alignment of the table and report correct values.
  4. [Section 10 noise audit statement] The conclusion says the noise audits reveal '17–23% faulty labels in CRIC and AOKVQA,' but Section 6.4 and Appendix G report 180/1000 (18%) label-noise cases in CRIC and about 90/1000 (9%) in AOKVQA. Please reconcile the reported percentages.
  5. [Section 2 / Section H] The introduction refers to 'Llama-3-8B' while the experimental sections use 'Llama-3.1-8B'; please use a single consistent model name throughout.

Circularity Check

1 steps flagged · score 4.0 of 10

Hallucination-reduction claim is partly definitional; core accuracy results are empirical, but supporting claims carry selection and metric-comparability risks.

  1. self definitional [Section 8 and Appendix H ('What We Define as Hallucinations and Their Constituents')]
    "we used the term 'Hallucination' repeatedly to indicate any content in an LLM-generated explanation that is not supported by and in some cases directly contradicts the information in the question, the image, or any retrieved knowledge provided as context. ... our gold-label-free Type-5 explanation prompt, which feeds dense and region captions plus retrieved facts, cuts that rate to 18.5% and grounds over 80% of explanations in the scene."

    Hallucination is defined as content not supported by the provided context (question, image, retrieved knowledge). Type-5 is the variant that enriches that context with dense captions, region captions, object lists, and retrieved facts (Section 4.2). Under this definition, adding more 'supporting context' to the prompt mechanically reduces the set of statements counted as unsupported. The 51%-to-18.5% hallucination drop is therefore in part guaranteed by the definition, not an independent measurement of factual accuracy. The downstream VQA accuracy gains, however, are separate empirical measurements and do not reduce to this definition.

full rationale

The paper's main accuracy claims (Table 2: ViLT 24.01→33.45, VisualBERT 23.6→40.12, FLAVA 33.07→47.85 under Type-5+SCE on AOKVQA; CRIC/e-SNLI-VE gains) are empirical fine-tuning results with baselines and a KAT comparison; there is no equation-level derivation that collapses into a fitted parameter or a self-citation. No load-bearing self-citations or imported uniqueness theorems are present. The one clear by-construction element is the hallucination-reduction claim, which uses a definition of hallucination as 'not supported by ... provided context' and then credits Type-5 with adding more provided context; this is a supporting contribution, not the core accuracy result. Two non-circular but important validity concerns are flagged: (1) Table 1 does not state its split, so the selection of Type-5 as best explanation prompt may have used test-split explanation similarity before reporting the same split's accuracy, creating a selection-on-test risk; (2) Table 4's generative benchmark uses a zero-shot BERTScore threshold (0.71) chosen by manual analysis, while sVLMs are fine-tuned MCQ classifiers, so the 'match or exceed' parity claim compares non-commeasurable protocols. Additionally, the Conclusion's specific numbers (ViLT 24→38, FLAVA 46→54) do not match Table 2's reported values (24.01→33.45, 33.07→47.85 under SCE), an internal inconsistency that weakens the summary but is not circularity. These factors keep the score moderate rather than zero.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper is empirical; there are no mathematical axioms. The load-bearing assumptions are about evaluation validity (explanation-quality proxies, noise audits) and about the fairness of the cross-model comparison.

free parameters (6)
  • SCE hyperparameters (alpha, beta, gamma) = alpha=0.1, beta=1.0, gamma=4
    Taken from Wang et al. (2019), but these constants determine the loss used for the reported noise-robust gains.
  • GCE hyperparameters (q, lambda) = q=0.7; lambda=0.4 (0.9 for e-SNLI-VE)
    q from Zhang and Sabuncu (2018), lambda chosen per dataset, affecting the reported robustness gains.
  • Number of retrieved facts k = 1
    Selected as optimal after ablations (Table 10); the central pipeline uses one fact.
  • Truncation length for explanation plus question = 100 tokens
    Chosen to keep inputs under the reader model's context; affects what the VLM sees.
  • Cosine threshold for generative evaluation = 0.71
    Set by manual analysis of outputs (Appendix E); directly affects the ACC score for generative VLMs in Table 4.
  • Type-5 prompt variant = DC + RC + O + Q + RF
    Selected as best by explanation-quality metrics in Table 1; the split used for selection is not specified.
assumptions (4)
  • domain assumption Prepending an LLM-generated explanation to the question is an effective way to deliver commonsense knowledge to sVLMs.
    This is the core mechanism of NLKI, introduced in Section 4.2 and implemented in Appendix B; no alternative integration is compared in depth.
  • domain assumption BLEU, ROUGE, and cosine similarity against ground-truth explanations are valid proxies for explanation usefulness.
    Used to select the prompt type in Section 6.2 and Table 1 and to benchmark retrievers in Table 9; the paper shows n-gram overlap does not always track downstream accuracy (Table 13), so this proxy is partially contradicted.
  • domain assumption Manual noise audit of 1000 samples per dataset is representative of the dataset's label noise.
    The claimed 10-25% label noise and the choice of loss function rest on manual counts (Section 6.4, Appendix G) from a small sample.
  • domain assumption Fine-tuning the retriever on the training split does not leak into the test split through the LLM explanation step.
    The retriever is fine-tuned on 160K CRIC training samples and used to retrieve facts for test-set explanations; the assumption that no test labels were used in retriever training is standard but unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks." pith.science (2026). https://pith.science/paper/QOD7QMQS

@misc{pith2026250819724,
  author       = {Pith},
  title        = {Pith review of: NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOD7QMQS}},
  note         = {Machine review of arXiv:2508.19724}
}
read the original abstract

Commonsense visual-question answering often hinges on knowledge that is missing from the image or the question. Small vision-language models (sVLMs) such as ViLT, VisualBERT and FLAVA therefore lag behind their larger generative counterparts. To study the effect of careful commonsense knowledge integration on sVLMs, we present an end-to-end framework (NLKI) that (i) retrieves natural language facts, (ii) prompts an LLM to craft natural language explanations, and (iii) feeds both signals to sVLMs respectively across two commonsense VQA datasets (CRIC, AOKVQA) and a visual-entailment dataset (e-SNLI-VE). Facts retrieved using a fine-tuned ColBERTv2 and an object information-enriched prompt yield explanations that largely cut down hallucinations, while lifting the end-to-end answer accuracy by up to 7% (across 3 datasets), making FLAVA and other models in NLKI match or exceed medium-sized VLMs such as Qwen-2 VL-2B and SmolVLM-2.5B. As these benchmarks contain 10-25% label noise, additional finetuning using noise-robust losses (such as symmetric cross entropy and generalised cross entropy) adds another 2.5% in CRIC, and 5.5% in AOKVQA. Our findings expose when LLM-based commonsense knowledge beats retrieval from commonsense knowledge bases, how noise-aware training stabilises small models in the context of external knowledge augmentation, and why parameter-efficient commonsense reasoning is now within reach for 250M models.

Figures

Figures reproduced from arXiv: 2508.19724 by the authors.

Figure 1
Figure 1. Comparison of accuracy and model size for various small Vision-Language Models (sVLMs) AOKVQA (Val). EKI: Type-5 Explanation Knowl￾edge Integrated models, as defined in 2, while RKI: Retrieved Knowledge Integrated models. The colour gradient represents execution time, with red indicating longer durations and blue indicating shorter ones. external knowledge sources (Wang et al., 2015; Aditya et al., 2018a, 2019). Fac… view at source ↗
Figure 2
Figure 2. Facts retrieved by pre-trained ColBERTv2 vs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples illustrating that pre-trained COLBERTv2 performs better than FAISS vector search in retrieving [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Noise Distribution of CRIC. We checked 1000 [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Noise Distribution of AOKVQA. We checked [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Examples illustrating the noisy samples from the datasets. where Noise Category is the type of noise we encountered during manual analysis, Explanation Generated From Noisy Sample is the LLaMA-3.1-8B explanation generated as a result of the noise for each sample, which…
Figure 7
Figure 7. Figure 7: Examples from the CRIC dataset showing the difference between the Ground Truth, Type-0, Type-1 and [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: More examples of Type-0, Type-1, and Type-5 generated explanations. in an LLM-generated explanation that is not sup￾ported by and in some cases directly contradicts the information in the question, the image, or any retrieved knowledge provided as context. In other [P…
Figure 9
Figure 9. Figure 9: Examples showing difference between the LLAMA-3.1-8B generated [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: We report some instances where the LLM generates inaccurate [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219

    arXiv 2024

  4. [4]

    Somak Aditya, Yezhou Yang, and Chitta Baral. 2018 a . https://arxiv.org/abs/1803.08896 Explicit reasoning over end-to-end neural architectures for visual question answering . Preprint, arXiv:1803.08896

    work page Pith review arXiv 2018

  5. [5]

    Somak Aditya, Yezhou Yang, and Chitta Baral. 2019. Integrating knowledge and reasoning in image understanding. In 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, pages 6252--6259. International Joint Conferences on Artificial Intelligence

    work page 2019

  6. [6]

    Somak Aditya, Yezhou Yang, Chitta Baral, and Yiannis Aloimonos. 2018 b . Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. In Uncertainty in artificial intelligence

    work page 2018

  7. [7]

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818--1826

    work page 2024

  8. [8]

    Zero-shot Visual Question Answering using Knowledge Graph

    Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, and Huajun Chen. 2021. https://arxiv.org/abs/2107.05348 Zero-shot visual question answering using knowledge graph . Preprint, arXiv:2107.05348

    work page Pith review arXiv 2021

Show all 49 references
  1. [9]

    Pan, Ningyu Zhang, and Huajun Chen

    Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, Jiaqi Li, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, and Huajun Chen. 2024. Knowledge graphs meet multi-modal learning: A comprehensive survey. CoRR, abs/2402.05391

    2024 arXiv

  2. [10]

    Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20

    2021

  3. [11]

    Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. 2023. Reltr: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2023

  4. [12]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library . Preprint, arXiv:2401.08281

    2024 arXiv

  5. [13]

    Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2023. https://doi.org/10.1109/TPAMI.2022.3210780 Cric: A vqa dataset for compositional reasoning on vision and commonsense . IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5561--5578

    2023

  6. [14]

    Deepanway Ghosal, Navonil Majumder, Roy Lee, Rada Mihalcea, and Soujanya Poria. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.809 Language guided visual question answering: Elevate your multimodal language model using knowledge-enriched prompts . In Findings of the Ass...

    2023 doi

  7. [15]

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander G Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. Kat: A knowledge augmented transformer for vision-and-language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis...

    2022

  8. [16]

    Catherine Havasi, Robert Speer, Kenneth Arnold, Henry Lieberman, Jason Alonso, and Jesse Moeller. 2010. Open mind common sense: crowd-sourcing for common sense. In Proceedings of the 2nd AAAI Conference on Collaboratively-Built Knowledge Sources and Artificial Intelligence, AA...

    2010

  9. [17]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024 arXiv

  10. [18]

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2023. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF conference on ...

    2023

  11. [19]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. https://arxiv.org/abs/2004.04906 Dense passage retrieval for open-domain question answering . Preprint, arXiv:2004.04906

    2020 arXiv

  12. [20]

    Omar Khattab and Matei Zaharia. 2020. https://arxiv.org/abs/2004.12832 Colbert: Efficient and effective passage search via contextualized late interaction over bert . Preprint, arXiv:2004.12832

    2020 arXiv

  13. [21]

    Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583--5594. PMLR

    2021

  14. [22]

    Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. https://arxiv.org/abs/2202.01110 A survey on retrieval-augmented text generation . Preprint, arXiv:2202.01110

    2022 arXiv

  15. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . Preprint, arXiv:2301.12597

    2023 arXiv

  16. [24]

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. https://arxiv.org/abs/1908.03557 Visualbert: A simple and performant baseline for vision and language . Preprint, arXiv:1908.03557

    2019 arXiv

  17. [25]

    Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. ECCV

    2018

  18. [26]

    Weizhe Lin and Bill Byrne. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.772 Retrieval augmented visual question answering with outside knowledge . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11238--11254, Abu Dhabi, Uni...

    2022 doi

  19. [27]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  20. [28]

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. https://arxiv.org/abs/2210.07316 Mteb: Massive text embedding benchmark . Preprint, arXiv:2210.07316

    2023 arXiv

  21. [29]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  22. [30]

    Jiahua Rao, Zifei Shan, Longpo Liu, Yao Zhou, and Yuedong Yang. 2023. Retrieval-based knowledge augmented vision language pre-training. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5399--5409

    2023

  23. [31]

    Varun Nagaraj Rao, Siddharth Choudhary, Aditya Deshpande, Ravi Kumar Satzoda, and Srikar Appalaraju. 2024. Raven: Multitask retrieval augmented vision-language learning. arXiv preprint arXiv:2406.19150

    2024 arXiv

  24. [32]

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. https://arxiv.org/abs/1506.02640 You only look once: Unified, real-time object detection . Preprint, arXiv:1506.02640

    2016 arXiv

  25. [33]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084

    2019 arXiv

  26. [34]

    Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. 2024. End-to-end training of multi-document reader and retriever for open-domain question answering. In Proceedings of the 35th International Conference on Neural Information Processing Systems...

    2024

  27. [35]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. https://arxiv.org/abs/2206.01718 A-okvqa: A benchmark for visual question answering using world knowledge . Preprint, arXiv:2206.01718

    2022 arXiv

  28. [36]

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. https://arxiv.org/abs/2112.04482 Flava: A foundational language and vision alignment model . Preprint, arXiv:2112.04482

    2022 arXiv

  29. [37]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. https://arxiv.org/abs/2409.12191 Qwen2-v...

    2024 arXiv

  30. [38]

    Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2015. Explicit knowledge-based reasoning for visual question answering. arXiv preprint arXiv:1511.02570

    2015 arXiv

  31. [39]

    Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2017. https://arxiv.org/abs/1606.05433 Fvqa: Fact-based visual question answering . Preprint, arXiv:1606.05433

    2017 arXiv

  32. [40]

    Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2023. https://openreview.net/forum?id=8IN-qLkl215 Visually-augmented language modeling . In The Eleventh International Conference on Learning Representations

    2023

  33. [41]

    Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. 2019. https://arxiv.org/abs/1908.06112 Symmetric cross entropy for robust learning with noisy labels . CoRR, abs/1908.06112

    2019 arXiv

  34. [42]

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706

    2019 arXiv

  35. [43]

    Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, and Anima Anandkumar. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.793 Re- V ...

    2023 doi

  36. [44]

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. 2023. Retrieval-augmented multimodal language modeling. In International Conference on Machine Learning, pages 39755--39769. PMLR

    2023

  37. [45]

    Shuquan Ye, Yujia Xie, Dongdong Chen, Yichong Xu, Lu Yuan, Chenguang Zhu, and Jing Liao. 2023. Improving commonsense in vision-language models via knowledge graph riddles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2634--2645

    2023

  38. [46]

    Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang, and Meng Jiang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.294 Retrieval augmentation for commonsense reasoning: A unified approach . In Proceedings of the 2022 Conference on Empirical M...

    2022 doi

  39. [47]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  40. [48]

    Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31

    2018

  41. [49]

    Wenfeng Zheng, Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. 2021. https://doi.org/10.1016/j.patcog.2021.108153 Knowledge base graph embedding module design for visual question answering model . Pattern Recognition, 120:108153

    2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.