RECORD
SeMLaPS: Real-time Semantic Mapping with Latent Prior Networks and Quasi-Planar Segmentation
Computer Vision and Pattern Recognition · Submitted 28 Jun 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our system achieves state-of-the-art semantic mapping quality within 2D-3D networks-based systems and matches the performance of 3D convolutional networks on three real indoor datasets, while working in real-time.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The availability of real-time semantics greatly improves the core geometric functionality of SLAM systems, enabling numerous robotic and AR/VR applications. We present a new methodology for real-time semantic mapping from RGB-D sequences that combines a 2D neural network and a 3D network based on a SLAM system with 3D occupancy mapping. When segmenting a new frame we perform latent feature re-projection from previous frames based on differentiable rendering. Fusing re-projected feature maps from previous frames with current-frame features greatly improves image segmentation quality, compared to a baseline that processes images independently. For 3D map processing, we propose a novel geometric quasi-planar over-segmentation method that groups 3D map elements likely to belong to the same semantic classes, relying on surface normals. We also describe a novel neural network design for lightweight semantic map post-processing. Our system achieves state-of-the-art semantic mapping quality within 2D-3D networks-based systems and matches the performance of 3D convolutional networks on three real indoor datasets, while working in real-time. Moreover, it shows better cross-sensor generalization abilities compared to 3D CNNs, enabling training and inference with different depth sensors. Code and data will be released on project page: http://jingwenwang95.github.io/SeMLaPS
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The availability of real-time semantics greatly improves the core geometric functionality of SLAM systems, enabling numerous robotic and AR/VR applications. | background 1.00 | 0.07 | 0.05 | asserted 1.00 |
| s1 | We present a new methodology for real-time semantic mapping from RGB-D sequences that combines a 2D neural network and a 3D network based on a SLAM system with 3D occupancy mapping. | contribution 0.84 | 0.91 | 0.06 | asserted 0.99 |
| s2 | When segmenting a new frame we perform latent feature re-projection from previous frames based on differentiable rendering. | method 0.99 | 0.93 | 0.03 | asserted 0.95 |
| s3 | Fusing re-projected feature maps from previous frames with current-frame features greatly improves image segmentation quality, compared to a baseline that processes images independently. | result 0.99 | 0.94 | 0.10 | asserted 1.00 |
| s4 | For 3D map processing, we propose a novel geometric quasi-planar over-segmentation method that groups 3D map elements likely to belong to the same semantic classes, relying on surface normals. | method 0.63 | 0.94 | 0.04 | asserted 0.93 |
| s5 | We also describe a novel neural network design for lightweight semantic map post-processing. | contribution 0.79 | 0.92 | 0.03 | asserted 0.93 |
| s6 | Our system achieves state-of-the-art semantic mapping quality within 2D-3D networks-based systems and matches the performance of 3D convolutional networks on three real indoor datasets, while working in real-time. | result 1.00 | 0.96 | 0.68 | asserted 1.00 |
| s7 | Moreover, it shows better cross-sensor generalization abilities compared to 3D CNNs, enabling training and inference with different depth sensors. | result 0.98 | 0.92 | 0.10 | asserted 1.00 |
| s8 | Code and data will be released on project page: http://jingwenwang95.github.io/SeMLaPS | future work 0.89 | 0.32 | 0.02 | asserted 0.84 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The availability of real-time semantics greatly improves the core geometric functionality of SLAM systems, enabling numerous robotic and AR/VR applications. We present a new methodology for real-time semantic mapping from RGB-D sequences that combines a 2D neural network and a 3D network based on a SLAM system with 3D occupancy mapping. When segmenting a new frame we perform latent feature re-projection from previous frames based on differentiable rendering. Fusing re-projected feature maps from previous frames with current-frame features greatly improves image segmentation quality, compared to a baseline that processes images independently. For 3D map processing, we propose a novel geometric quasi-planar over-segmentation method that groups 3D map elements likely to belong to the same semantic classes, relying on surface normals. We also describe a novel neural network design for lightweight semantic map post-processing. Our system achieves state-of-the-art semantic mapping quality within 2D-3D networks-based systems and matches the performance of 3D convolutional networks on three real indoor datasets, while working in real-time. Moreover, it shows better cross-sensor generalization abilities compared to 3D CNNs, enabling training and inference with different depth sensors. Code and data will be released on project page: http://jingwenwang95.github.io/SeMLaPS
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites
Uses the method or tool of 3 papers
-
We adopt SSMA [ref] , using separate encoders for RGB and depth inputs, but replace the ResNet-50 encoder with a lightweight MobileNetV3 [ref] , keeping computational complexity in mind, see Fig. [ref].
Rethinking Atrous Convolution for Semantic Image Segmentation · Latent Prior Network · 0.95
-
To get smooth gradient propagation, we employ Pytorch3D for feature re-projection, see Sec. 3.3 in [ref] .
Accelerating 3D Deep Learning with PyTorch3D · Latent Prior Network · 1.00
-
We train LPN on 1201 training sequences of the ScanNet v2 dataset with a step of 20 between adjacent frames, using Adam [ref] for 20 epochs with an initial learning rate of $1e-4$ and one-cycle learning rate scheduler; We use $N=3$ and perform random scale, crop, flip, Gaussian blur and random view-order permutation for data augmentation.
Adam: A Method for Stochastic Optimization · System Implementation Details · 1.00
Compares with 1 paper
-
For 2D semantic segmentation evaluation we use AdapNet++ [ref] , FuseNet [ref] , SSMA [ref] and PS-res-excite [ref] , and our closest baseline MVCNet [ref] based on our LPN backbone, in single-view inference (MVCNet-S) and in X-view inference (MVCNet-MX) which we re-implemented.
Robust Double-Encoder Network for RGB-D Panoptic Segmentation · Experiments · 0.94
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.