Interpretable Visual Question Answering Referring to Outside Knowledge

He Zhu; Miki Haseyama; Ren Togo; Takahiro Ogawa

RECORD

Interpretable Visual Question Answering Referring to Outside Knowledge

He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama

Computer Vision and Pattern Recognition · Submitted 8 Mar 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2303.04388 (version 1)

Pith IDIQXFLIDB, Pith's permanent record of this paper and its review

CitationHe Zhu, Miki Haseyama, Ren Togo, Takahiro Ogawa. "Interpretable Visual Question Answering Referring to Outside Knowledge." arXiv:2303.04388 (2023). https://pith.science/paper/IQXFLIDB

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,881 input tokens.

What the abstract itself states, as probabilities: claims first 0.16 · states limitation 0.09 · human participants 0.06 · controlled comparison 0.54 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.93 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations.	contribution 0.99	0.90	0.29	asserted 0.98
s1	Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image.	gap 0.83	0.06	0.03	asserted 1.00
s2	Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily.	gap 0.56	0.05	0.07	asserted 0.59
s3	The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model.	method 0.92	0.85	0.07	asserted 1.00
s4	The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results.	contribution 1.00	0.82	0.16	asserted 0.99
s5	Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.	result 1.00	0.96	0.75	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model's decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites

Uses the method or tool of 3 papers

We used five different models to generate multiple captions containing various information, including GIT-large [ref] fine-tuned on COCO, GIT-large [ref] fine-tuned on TextCaps, BLIP [ref] , CoCa [ref] and BLIP-2 [ref] .

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models · Experimental Settings · 1.00
In the outside knowledge retrieval, we used the same query and passage encoder as Dense Passage Retrieval [ref] and the Faiss [ref] to realize the retrieval process.

Dense Passage Retrieval for Open-Domain Question Answering · Experimental Settings · 1.00
All the scores of the language metrics were computed by the publicly available code provided by Chen et al. [ref] .

Microsoft COCO Captions: Data Collection and Evaluation Server · Experimental Settings · 1.00

Uses data or values from 1 paper

We finally chose a recently proposed subset from Wikidata [ref] following KAT [ref] as our outside knowledge set $𝒦$ in this paper.

KAT: A Knowledge Augmented Transformer for Vision-and-Language · Introduction of Image Caption and Outside Knowledge · 0.85

Cites as background 2 papers

Recently, pre-trained large-scale models such as the Bidirectional Encoder Representations from Transformers (BERT) [ref] , GPT-2 [ref] , and the Contrastive Language-Image Pre-training (CLIP) [ref] have been developed rapidly, which makes it possible to generate natural language more correctly.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Introduction · 0.92
Visual Question Answering (VQA) [ref] has been widely used in various scenarios such as defect detection [ref] and medical diagnosis [ref] .

Medical Visual Question Answering: A Survey · Introduction · 1.00

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.