REVIEW 5 cited by
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels. Recent works mainly utilize complete occupancy labels in 3D voxel space for supervision. However, the expensive annotation process and sometimes ambiguous labels have severely constrained the usability and scalability of 3D occupancy models. To address this, we present RenderOcc, a novel paradigm for training 3D occupancy models only using 2D labels. Specifically, we extract a NeRF-style 3D volume representation from multi-view images, and employ volume rendering techniques to establish 2D renderings, thus enabling direct 3D supervision from 2D semantics and depth labels. Additionally, we introduce an Auxiliary Ray method to tackle the issue of sparse viewpoints in autonomous driving scenarios, which leverages sequential frames to construct comprehensive 2D rendering for each object. To our best knowledge, RenderOcc is the first attempt to train multi-view 3D occupancy models only using 2D labels, reducing the dependence on costly 3D occupancy annotations. Extensive experiments demonstrate that RenderOcc achieves comparable performance to models fully supervised with 3D labels, underscoring the significance of this approach in real-world applications.
Forward citations
Cited by 5 Pith papers
-
VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models
VISA improves closed-set 3D occupancy mIoU on nuScenes by using VLM instance audits as reliability-weighted semantic supervisors during training of existing world models.
-
SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction
SDGOCC improves multimodal 3D occupancy prediction by using LiDAR depth and semantic masks to guide camera-to-BEV transformation, achieving state-of-the-art mIoU on Occ3D-nuScenes.
-
VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction
A training-only Gaussian splatting loss, which renders predicted 3D semantics and motion into 2D camera views, improves semantic occupancy and scene flow prediction across several camera-based models.
-
SliceSemOcc: Vertical Slice Based Multimodal 3D Semantic Occupancy Representation
SliceSemOcc improves 3D semantic occupancy prediction by slicing voxel features into global and local height bands and applying per-height channel attention, yielding modest mIoU gains on nuScenes benchmarks.
-
Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving
Diffusion-based generative models, using discrete categorical diffusion conditioned on BEV features, improve 3D occupancy prediction and downstream planning for autonomous driving.
Discussion (0). Sign in to comment.