A dual-stream network combining CNN and vision transformer features, followed by a rule extractor, achieves state-of-the-art accuracy on multiple Raven's Progressive Matrices benchmarks, including large gains on out-of-distribution PGM subsets.
SAViR-T: Spatially Attentive Visual Reasoning with Transformers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We present a novel computational model, "SAViR-T", for the family of visual reasoning problems embodied in the Raven's Progressive Matrices (RPM). Our model considers explicit spatial semantics of visual elements within each image in the puzzle, encoded as spatio-visual tokens, and learns the intra-image as well as the inter-image token dependencies, highly relevant for the visual reasoning task. Token-wise relationship, modeled through a transformer-based SAViR-T architecture, extract group (row or column) driven representations by leveraging the group-rule coherence and use this as the inductive bias to extract the underlying rule representations in the top two row (or column) per token in the RPM. We use this relation representations to locate the correct choice image that completes the last row or column for the RPM. Extensive experiments across both synthetic RPM benchmarks, including RAVEN, I-RAVEN, RAVEN-FAIR, and PGM, and the natural image-based "V-PROM" demonstrate that SAViR-T sets a new state-of-the-art for visual reasoning, exceeding prior models' performance by a considerable margin.
fields
cs.CV 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Learning Visual Abstract Reasoning through Dual-Stream Networks
A dual-stream network combining CNN and vision transformer features, followed by a rule extractor, achieves state-of-the-art accuracy on multiple Raven's Progressive Matrices benchmarks, including large gains on out-of-distribution PGM subsets.