RECORD
Indoor scene recognition from images under visual corruptions
Computer Vision and Pattern Recognition · Submitted 23 Aug 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The classification of indoor scenes is a critical component in various applications, such as intelligent robotics for assistive living. While deep learning has significantly advanced this field, models often suffer from reduced performance due to image corruption. This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion, integrating caption-based semantic features with visual data to enhance both accuracy and robustness against corruption. We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN). Our study shows that this fusion markedly improves model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset. Moreover, while standalone visual models displayed high accuracy on uncorrupted images, their performance deteriorated significantly with increased corruption severity. Conversely, the multimodal models demonstrated improved accuracy in clean conditions and substantial robustness to a range of image corruptions. These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The classification of indoor scenes is a critical component in various applications, such as intelligent robotics for assistive living. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | While deep learning has significantly advanced this field, models often suffer from reduced performance due to image corruption. | background 0.52 | 0.08 | 0.02 | asserted 0.96 |
| s2 | This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion, integrating caption-based semantic features with visual data to enhance both accuracy and robustness against corruption. | contribution 0.95 | 0.75 | 0.20 | asserted 0.99 |
| s3 | We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN). | method 0.97 | 0.79 | 0.04 | asserted 0.96 |
| s4 | Our study shows that this fusion markedly improves model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset. | result 1.00 | 0.97 | 0.48 | asserted 1.00 |
| s5 | Moreover, while standalone visual models displayed high accuracy on uncorrupted images, their performance deteriorated significantly with increased corruption severity. | result 0.99 | 0.83 | 0.07 | asserted 1.00 |
| s6 | Conversely, the multimodal models demonstrated improved accuracy in clean conditions and substantial robustness to a range of image corruptions. | result 1.00 | 0.92 | 0.28 | asserted 1.00 |
| s7 | These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems. | interpretation 1.00 | 0.67 | 0.77 | hedged possibility 0.88 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The classification of indoor scenes is a critical component in various applications, such as intelligent robotics for assistive living. While deep learning has significantly advanced this field, models often suffer from reduced performance due to image corruption. This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion, integrating caption-based semantic features with visual data to enhance both accuracy and robustness against corruption. We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN). Our study shows that this fusion markedly improves model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset. Moreover, while standalone visual models displayed high accuracy on uncorrupted images, their performance deteriorated significantly with increased corruption severity. Conversely, the multimodal models demonstrated improved accuracy in clean conditions and substantial robustness to a range of image corruptions. These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 4 sentences without a confident label
Uses the method or tool of 7 papers
-
We employ the well-known Mean Average Precision (mAP) metric, jointly with Mean Corruption Error (mCE) and Mean Relative Corruption Error (mRCE) metrics, as introduced by [ref] , which are metrics for evaluating robustness against corruptions.
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations · Places148-corrupted: An indoor scenes dataset · 0.95
-
First, we apply ExpansionNet-v2 [ref] , an image captioning model, to generate raw captions for each image in our dataset.
Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning · Extraction of high-level image descriptors · 1.00
-
For this task, we propose adapting the Graph Isomorphism Network (GIN) [ref] , which we chose due to its simple architecture, which could lead to reasonable inference rates in low-energy, low-consumption devices.
How Powerful are Graph Neural Networks? · Extraction of high-level image descriptors · 0.90
Cites as background 1 paper
-
Indoor scene recognition has been shaped by various proposals that fall under two main categories: image-based classification and multimodal techniques that incorporate additional data types, such as depth maps or 3D reconstructions [ref] .
Centroid Based Concept Learning for RGB-D Indoor Scene Classification · Related Works · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.