From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

Zihao Zhu

RECORD

From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

Zihao Zhu

Computer Vision and Pattern Recognition · Submitted 25 Jun 2022

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,822 input tokens.

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.09 · human participants 0.04 · controlled comparison 0.31 · randomized assignment 0.01 · states quantitative result 0.06 · compares with prior results 0.85 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge.	background 0.98	0.06	0.05	asserted 1.00
s1	Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model.	background 0.90	0.06	0.04	asserted 0.97
s2	It is effortless for human but challenging for machines.	background 0.86	0.06	0.02	asserted 1.00
s3	In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues.	contribution 0.96	0.90	0.12	asserted 0.99
s4	Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities.	method 0.99	0.85	0.03	asserted 0.51
s5	Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs.	method 0.98	0.83	0.06	asserted 0.95
s6	Compared to existing modular networks, we extend visual reasoning from one graph to more graphs.	contribution 0.98	0.78	0.06	asserted 1.00
s7	We can explicitly trace the reasoning process according to module weights and graph attentions.	method 0.58	0.79	0.07	asserted 0.70
s8	Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.	result 1.00	0.96	0.82	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

1 arXiv paper cite this in 1 sentence

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 5 sentences without a confident label

Uses the method or tool of 1 paper

We select 10 types of relations to reduce the computation cost following [ref] .

CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense · Graph Representations · 0.93

Uses data or values from 1 paper

We evaluated our model on the CRIC [ref] , a large VQA dataset contains 1,303,271 deeper questions than other common datasets, which are randomly split into train (60%), validation (20%) and test (20%).

CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense · Experimental Settings · 0.97

Compares with 1 paper

(3) RVC [ref] builds upon neural module networks with supervision of expert layout.

CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense · Experimental Results · 0.91

Cites as background 1 paper

Most of existing VQA models [ref] that exhibit reasoning capabilities can be divided into three categories.

Meta Module Network for Compositional Visual Reasoning · Introduction · 0.93

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.