Pith. sign in

REVIEW 6 cited by

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.08614 v2 pith:WMD5MUUE submitted 2021-12-16 cs.CL

Liangke Gui , Borui Wang , Qiuyuan Huang , Alex Hauptmann , Yonatan Bisk , Jianfeng Gao This is my paper
classification cs.CL
keywords knowledgeexplicitmodelreasoningansweraugmentedimplicitmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a different question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6 points absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AR-RAG: Autoregressive Retrieval Augmentation for Image Generation

    cs.CV 2025-06 conditional novelty 7.0 of 10

    Autoregressive patch-level retrieval augmentation improves text-to-image generation on GenEval, DPG-Bench, and Midjourney-30K, with a training-free decoding variant and a fine-tuned variant.

  2. SoftReason: A Fully Differentiable Neuro-Soft-Symbolic Deductive Reasoning Architecture over High-Dimensional Perceptual Data

    cs.AI 2026-07 conditional novelty 6.0 of 10

    SoftReason learns a differentiable soft deductive closure operator over perceptual facts and reports 94.3% Hit@1 on KVQA entity linking.

  3. Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Controllable retrieval-difficulty curriculum plus reward-propagation sampling lets RL close the pretrain-to-KB-VQA gap and beat prior SOTA on two hard encyclopedic VQA benchmarks.

  4. KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    KG-ViP fuses scene graphs and commonsense graphs via a query-based retrieval-and-fusion pipeline to improve multi-modal LLM performance on visual question answering.

  5. Outside Knowledge Conversational Video (OKCV) Dataset -- Dialoguing over Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    OKCV is a new human-annotated video dialogue dataset where answering questions requires both visual grounding in the video and external knowledge.

  6. Augmented Vision-Language Models: A Systematic Review

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.

Pith tools