DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation

Bowen Yin; Li Liu; Ming-Ming Cheng; Qibin Hou; Xuying Zhang; Zhongyu Li

DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.09668 v2 pith:DU3Q5YQD submitted 2023-09-18 cs.CV

DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation

Bowen Yin , Xuying Zhang , Zhongyu Li , Li Liu , Ming-Ming Cheng , Qibin Hou This is my paper

classification cs.CV

keywords rgb-ddformersegmentationpretrainedsemantictasksbackbonedatasets

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

Share X LinkedIn Reddit HN

read the original abstract

We present DFormer, a novel RGB-D pretraining framework to learn transferable representations for RGB-D segmentation tasks. DFormer has two new key innovations: 1) Unlike previous works that encode RGB-D information with RGB pretrained backbone, we pretrain the backbone using image-depth pairs from ImageNet-1K, and hence the DFormer is endowed with the capacity to encode RGB-D representations; 2) DFormer comprises a sequence of RGB-D blocks, which are tailored for encoding both RGB and depth information through a novel building block design. DFormer avoids the mismatched encoding of the 3D geometry relationships in depth maps by RGB pretrained backbones, which widely lies in existing methods but has not been resolved. We finetune the pretrained DFormer on two popular RGB-D tasks, i.e., RGB-D semantic segmentation and RGB-D salient object detection, with a lightweight decoder head. Experimental results show that our DFormer achieves new state-of-the-art performance on these two tasks with less than half of the computational cost of the current best methods on two RGB-D semantic segmentation datasets and five RGB-D salient object detection datasets. Our code is available at: https://github.com/VCIP-RGBD/DFormer.

discussion (0)

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation
cs.CV 2026-04 unverdicted novelty 6.0

CrossWeaver introduces MIB and SAF modules to enable flexible, reliability-aware cross-modal interaction and fusion, achieving SOTA multimodal semantic segmentation with minimal parameters and generalization to unseen...
Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropout
cs.CV 2026-07 conditional novelty 5.0

Condition Dropout freezes the original RGB-D encoder, trains a copied encoder with random modality dropout, and cuts the mIoU drop under missing inputs from about -25 to about -10 on NYUv2 and SUN RGB-D.
Visual Place Recognition in Forests with Depth-Aware Distillation
cs.CV 2026-06 unverdicted novelty 5.0

Depth-aware distillation improves DINOv2 visual place recognition in forests, yielding gains over appearance-only methods on the WildCross benchmark.
SegSTRONG-C: Segmenting Surgical Tools Robustly On Non-adversarial Generated Corruptions -- An EndoVis'24 Challenge
cs.CV 2024-07 accept novelty 5.0

SegSTRONG-C provides a new benchmark where top models reach 0.9394 DSC and 0.9301 NSD on corrupted surgical tool segmentation tests, showing conventional techniques help but calling for more innovative robustness methods.
Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data
cs.CV 2026-02 reject novelty 4.0

Adding monocular depth to EfficientViT-SAM improves point-prompted segmentation at 3 and 5 clicks after fine-tuning on 11.2k images, but universal gains and data-efficiency are not established.