Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving

Pengfei Li; Wei Sun; Xiang Li; Yan Wang; Yilun Chen; Yupeng Zheng

REVIEW 4 major objections 6 minor 1 cited by

Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PreWorld shows that a 3D occupancy world model for autonomous driving can be pre-trained on abundant 2D labels, then fine-tuned with 3D labels, and still beat prior state-of-the-art on both 3D occupancy prediction and 4D forecasting.

desk verdict A credible semi-supervised 3D occupancy pipeline with believable pretraining gains, but the 4D forecasting/world-model claims outrun the evidence—the Copy&Paste baseline tells a sobering story. read the letter →

arxiv 2502.07309 v1 pith:5FLN3BDF submitted 2025-02-11 cs.CV

Xiang Li , Pengfei Li , Yupeng Zheng , Wei Sun , Yan Wang , Yilun Chen This is my paper · ORCID

classification cs.CV

keywords 3Doccupancyprediction4Dforecastingworldmodelsemi-supervisedlearningautonomousdrivingvolumerendering2Dlabelsupervision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a 3D occupancy world model for autonomous driving can be trained in two stages—first self-supervised on cheap 2D labels via volume rendering, then fine-tuned on 3D occupancy labels—and that this combination is enough to beat prior state-of-the-art on 3D occupancy prediction and set new state-of-the-art on 4D occupancy forecasting. The motivation is that 3D occupancy labels for outdoor scenes are expensive, while 2D labels (RGB, depth, semantics) are abundant. The two-stage scheme would matter because it offers a path to scale world models for driving without the usual 3D annotation burden, and it also provides a stronger self-supervised baseline for forecasting. The paper reports mIoU 34.69 on Occ3D-nuScenes, ahead of OccFlowNet's 33.86, and an average 4D forecasting mIoU of 9.55 when 2D pre-training is used.

What carries the argument

The load-bearing component is the temporal 2D rendering supervision of future volume features. An attribute projection head maps the occupancy network's volume features into density, semantic, and RGB fields; these fields are volume-rendered into 2D predictions along camera rays, and the loss in Eq. 8 matches the rendered RGB, depth, and semantics of both current and future frames against 2D labels. This lets a simple state-conditioned forecasting module (two MLPs that optionally consume ego-state tokens) be trained to predict future volume features while avoiding the information loss of tokenizing and decoding discrete occupancy. In the fine-tuning stage the projection head is dropped and an occupancy head with focal, lovasz-softmax, and scene-class affinity losses is supervised by 3D occupancy labels.

What would settle it

Remove the future-frame terms of Eq. 8 and retrain; if the 4D forecasting mIoU/IoU on the validation set is unchanged, then the temporal rendering supervision is not teaching dynamics. Alternatively, evaluate on segments with strong ego rotation or acceleration and compare forecasts with a copy-current-occupancy baseline; if PreWorld's advantage disappears on such segments, the forecasting capability is an artifact of the rendering loss rather than learned dynamics.

Watch

Extended reading notes

Core claim

The central claim is that a 3D occupancy world model for autonomous driving can be built as a single end-to-end network that predicts future volume features directly from multi-view images, avoiding the frozen-occupancy plus discrete-token forecasting pipeline of prior world models. The evidence is a two-stage training scheme: pre-training supervises current and future volume features by rendering them into 2D RGB, depth, and semantic images via volume rendering (temporal 2D rendering loss), and fine-tuning supervises an occupancy head with 3D occupancy labels. With this scheme, PreWorld reports a 34.69 mIoU on 3D occupancy prediction on Occ3D-nuScenes, ahead of the 33.86 reported for OccFlowNet, and the strongest 4D occupancy forecasting among the compared methods, with further gains from the 2D pre-training. A self-supervised variant that only uses 2D labels and a density threshold for occupancy also reports markedly better 4D forecasting than the previous self-supervised baseline.

Load-bearing premise

The future-frame 2D rendering loss genuinely pushes the forecasting module to predict scene dynamics rather than allowing it to satisfy the RGB, depth, and semantic losses by outputting a static or average version of the current scene.

Editorial extensions

If this is right

If the claim is correct, large-scale 2D data can pre-train driving world models, and relatively small sets of 3D occupancy labels are enough to reach competitive or state-of-the-art accuracy, directly reducing annotation cost.
The two-MLP state-conditioned forecasting module, trained end-to-end, indicates that the discrete tokenization and autoregressive generation used by earlier world models is not a necessary step for strong 4D occupancy forecasting.
The self-supervised variant PreWorld-S shows that 2D labels alone can support both 4D occupancy forecasting and motion planning, nearly doubling forecast IoU over the OccWorld self-supervised baseline.
Jointly training forecasting and motion planning (with ego-trajectory supervision) improves both outcomes, so a single PreWorld-style model can serve perception, prediction, and planning in one unified network.
Under the RayIoU metric the gains from 2D pre-training persist, which suggests the improvement is real geometry-aware prediction rather than an artifact of predicting thicker surfaces for static classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive ablation the paper does not run is to remove only the future-frame terms of Eq. 8 while keeping current-frame 2D supervision; if 4D forecasting performance is unchanged, the temporal component of the rendering loss is not the source of the forecasting gain.
The same two-stage recipe could plausibly transfer to other sensor modalities or to driving datasets with different weather, since 2D labels are comparatively cheap; testing PreWorld-S on a non-nuScenes dataset would clarify how much of the gain depends on the specific benchmark.
The relatively small average gain over the Copy&Paste baseline (0.46 mIoU) and the below-baseline 3s mIoU in Table 6 suggests long-horizon dynamics are still weak; a forecasting module that explicitly models ego motion or learning scene flow could raise long-horizon accuracy.
The large effect of ego-state input on planning but smaller effect on forecasting raises a question the paper leaves open: whether the planner truly anticipates obstacles or partly relies on trajectory shortcuts, an issue the paper itself flags as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi [Name],

Short version: PreWorld is a credible engineering contribution for semi-supervised 3D occupancy prediction, but the 'world model' and 4D forecasting claims are ahead of the evidence. If you read it for the occupancy results, you'll find a decent paper; if you read it for the forecasting, you'll be disappointed by the gap between the abstract and Table 6.

The genuinely new thing is the two-stage pipeline: pretrain an image-to-volume network with 2D rendering losses (RGB, depth, semantic) on current and future frames, then fine-tune with 3D occupancy labels, with the forecasting module optimized end-to-end in both stages. That's different from OccWorld/OccLLaMA, which rely on frozen occupancy models and discrete tokens. The 3D occupancy numbers are internally consistent: the model reports 34.69 mIoU vs 33.86 for OccFlowNet, the RayIoU analysis is a thoughtful bonus, and the pretraining gain shows up across data scales and metrics. The appendix's extended scaling study is a genuinely honest response to an initially confounded ablation; the authors admit they didn't give the smaller datasets enough epochs, then fix it properly. That's good behavior.

The soft spots are real. The 4D forecasting results are the paper's main footprint, but the Copy&Paste baseline in Table 6 tells a sobering story. The plain model is actually worse than Copy&Paste at the 3s horizon (5.89 vs 6.23 mIoU), and the full model's gain over Copy&Paste is less than one mIoU at 3s. Because the future-frame rendering loss can be satisfied by a static scene plus ego-motion alignment, the model could be learning precisely that, not object dynamics. The authors don't discuss this. They also don't provide error bars, and the differences they rely on are often sub-mIoU. The planning results with ego-state are explicitly attributed to a shortcut (they cite Zhai et al. and Li et al.), so those numbers shouldn't be read as a planning contribution. The 'semi-supervised' label is also a bit generous, since the 2D labels come from the same LiDAR/3D annotation suite, though the pretraining does use a different supervision modality and it does help.

Bottom line: this is a paper for the 3D occupancy and semi-supervised perception crowd, not for anyone building an actual prediction system on the strength of its forecasting results. It deserves a serious referee (and already got one at ICLR), but the forecasting claims should be scaled back or supported by a baseline that separates ego-motion from object dynamics. I'd send it to review and ask for that analysis. For my own work, I'd cite the pretraining result, not the forecasting SOTA.

Best,

Referee Report

4 major / 6 minor

Summary. PreWorld is a vision-centric 3D occupancy prediction and forecasting framework trained in two stages. In a self-supervised pre-training stage, an occupancy network and a state-conditioned forecasting module are optimized with 2D rendering supervision (RGB, depth, semantics) on current and future frames via volume rendering (Eq. 8). In a fully-supervised fine-tuning stage, the model is fine-tuned with 3D occupancy labels through an occupancy head. Experiments on Occ3D-nuScenes report an mIoU of 34.69 for 3D occupancy prediction (above OccFlowNet's 33.86), average future mIoU of 9.55 for 4D forecasting (above OccLLaMA's 8.66), and competitive motion planning, with additional ablations on pre-training attributes, data scale, and model components.

Significance. The two-stage paradigm is a plausible direction for reducing 3D annotation burden, and applying temporal rendering supervision to future frames is a novel and clearly stated idea. The paper includes extensive ablations, a candid discussion of training-time limitations in Appendix A.2, and a second evaluation metric (RayIoU) that reinforces the 3D occupancy results. The simple two-MLP forecasting module is a useful contrast to token-based world models, and the attempt to isolate the benefit of pre-training at different data scales is commendable. However, the central 4D 'world model' claim rests on small improvements over a static Copy&Paste baseline and on a loss that may not force genuine dynamics; the scalability and planning claims are also weakened by epoch-budget mismatches and the ego-trajectory shortcut, respectively. If the forecasting module is not learning real object motion, the headline SOTA claims are not established.

major comments (4)

[§3.3, Eq. (8); Table 6] The load-bearing assumption that the temporal 2D rendering loss in Eq. (8) forces F to learn scene dynamics is not supported. Because rays from future frames are transformed into the current frame using ego poses, a model that predicts a static occupancy volume (or one that only warps by ego-motion) can satisfy much of the RGB/depth/semantic supervision without modeling object motion. Table 6 is consistent with this: the plain PreWorld model (row 2) improves over Copy&Paste by only 0.46 mIoU on average and is worse at the 3s horizon (5.89 vs 6.23); even the full model gains less than 1 mIoU at 3s. To substantiate the 'world model' and SOTA forecasting claims, the paper needs a direct test of learned dynamics (e.g., per-class moving-object mIoU, flow-based evaluation, or a comparison to an ego-motion-only warping baseline), or the claims should be restricted to static extrapolation with learned corrections.
[§4.3 Table 5; Appendix A.2 Table 10] The scalability ablation in Table 5 is confounded by unequal training budgets: the 150-scene and 450-scene models were trained for 24 and 18 epochs, respectively, while the 700-scene model was trained for 12 epochs, as the authors themselves note in A.2. The extended experiments in Table 10 show that at convergence the advantage of pre-training on 150 scenes shrinks from the headline 6.36 mIoU to roughly 1.3 mIoU, and that most of the benefit is faster convergence rather than improved asymptotic performance. The main-text sentence 'pre-training significantly boosts the mIoU from 18.66 to 25.02' is therefore misleading without the epoch context. Please present the best-epoch or converged numbers in the main text and reframe the contribution as 'faster convergence plus a smaller asymptotic gain,' or update the main text to cite Table 10.
[§4.1, §2.3] The claim that the method reduces the need for 3D annotation is not fully tested, because the 2D depth and semantic labels used in pre-training are derived from the same LiDAR/3D annotations that produce the 3D occupancy labels in Occ3D-nuScenes. The two supervision sources are therefore not independent, and the paper does not demonstrate that the 2D labels are obtainable without corresponding 3D annotation effort. The authors should clarify the provenance of the 2D labels and, ideally, include an experiment with genuinely independent 2D labels (e.g., off-the-shelf depth/semantic estimators) to support the semi-supervised claim.
[§4.2 Table 3] The motion planning comparison is difficult to interpret because the strong results depend on ego-state conditioning that the paper itself describes as a 'shortcut' (citing Zhai et al. 2023 and Li et al. 2024). Without ego-state, PreWorld's average collision rate (1.14) is markedly worse than OccLLaMA's (0.70); with ego-state it improves to 0.53, which the paper attributes to the shortcut. Because Table 7 shows that trajectory supervision alone yields L2 0.45 and that 3D occupancy labels add little, the planning experiments do not establish that the occupancy world model improves planning. Please ablate the ego-trajectory shortcut (e.g., corrupting or removing ego-state at inference) or explicitly scope the planning claim to the state-conditioned setting and note that the benefit may come from the trajectory shortcut rather than world-model forecasting.

minor comments (6)

[Abstract and §2.3] Calling the pre-training stage 'self-supervised' when it uses 2D labels (depth, semantics) is inconsistent with the usual meaning of self-supervision; suggest '2D-supervised' or clarify the relationship to self-supervised methods.
[Eq. (8)] The ground-truth quantities d2D_i, s2D_i, c2D_i are used before being defined; define them explicitly at first use.
[Table 6 caption] The word 'copys' should be 'copies'.
[Table 2 caption] The word 'reprersents' should be 'represents'.
[§3.2] The dimensions and exact architecture of the two MLPs and how ego-state tokens are constructed are not specified; please provide details for reproducibility.
[Conclusion] The statement 'demonstrate the robustness of PreWorld' is stronger than the evidence warrants, since no robustness tests (e.g., sensor noise, weather) are reported; consider rewording to 'effectiveness.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PreWorld's claims are benchmarked against held-out Occ3D-nuScenes ground truth with no in-loop fitting of the target.

full rationale

PreWorld's central derivation is a two-stage training recipe: pretraining with 2D rendering losses (Eq. 8) followed by fine-tuning with 3D occupancy losses (Eq. 9). The paper's performance claims are measured on the held-out Occ3D-nuScenes validation set, so the reported mIoU/IoU/RayIoU values are not constructed by the training objective; they are external evaluations. The forecasting module F is a two-MLP state-conditioned module trained end-to-end, and the Copy&Paste baseline in Table 6 is an explicit control, showing the model is not merely copying current occupancy. Although the concern that the future-frame rendering loss in Eq. 8 may be satisfied by a static volume is a legitimate scientific limitation for the 4D world-model claim, it is a correctness/validity risk, not a circular-reasoning defect: nothing in the derivation defines the predicted future occupancy in terms of the ground-truth future occupancy, and no fitted parameter is renamed as a prediction. Self-citations (e.g., to MonoOcc and OccLLaMA) are present but are not used to justify load-bearing premises; the method adopts external baselines and architectures (RenderOcc, BEVStereo, FB-OCC) for implementation details rather than to force the claimed result. The use of 2D labels that may be derived from the same underlying LiDAR annotations as the 3D labels weakens the label-independence aspect of 'semi-supervised' but is a data-dependency observation, not a circular derivation. Overall, no equation or training step reduces to the target metric by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard differentiable volume rendering, an inherited backbone (BEVStereo + FB-OCC), hand-set loss weights, and a threshold for self-supervised inference. No new physical entities are introduced. The data assumption that 2D labels are cheaper than 3D labels is not costed.

free parameters (3)

Loss weights (lambda_dep, lambda_sem, lambda_RGB, lambda_f, lambda_l, lambda_sem_scal, lambda_geo_scal) = 1.0 each
Section 4.1 sets every loss weight to 1.0 without sensitivity analysis; the balance between rendering, occupancy, and affinity losses could affect the reported results.
Density threshold tau = not reported
Appendix A.3, Eq. 10 uses an unspecified threshold to convert density field to occupancy for self-supervised inference; the threshold is chosen by hand and not ablated.
Training schedule (epochs) = 6/12 for 3D, 8/18 for 4D
The number of pretraining and fine-tuning epochs is chosen ad hoc and, per Appendix A.2, varies across data-scale ablations, confounding some comparisons.

assumptions (5)

standard math Volume rendering equations (Eq. 6-7) correctly model ray-marching through the predicted density field.
Taken from NeRF/RenderOcc; no derivation or validation in this paper, but it is a standard differentiable renderer used in prior work.
domain assumption BEVStereo and FB-OCC provide a strong occupancy network and head, and their pretrained weights (if any) are not disclosed.
The entire pipeline inherits the behavior of these external architectures (Section 4.1); the paper does not ablate the choice of backbone.
domain assumption 2D depth and semantic labels available for the pretraining stage are accurate and dense enough to supervise 3D geometry and semantics.
The pretraining loss L2D assumes ground-truth 2D labels (d2D, s2D, c2D) correspond to the scene; errors in these labels directly corrupt the volume rendering signal (Eq. 8).
domain assumption The 2D labels used in pretraining are substantially cheaper than 3D occupancy labels.
Motivates the semi-supervised setup; no cost model or annotation pipeline is provided.
domain assumption The forecasting module, two MLPs with optional ego-state tokens, can model future volume features without suffering from accumulated error.
The simple module is claimed to be sufficient (Section 3.2); Table 6 shows it actually underperforms Copy&Paste at 3s, suggesting possible accumulation of error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving." pith.science (2026). https://pith.science/paper/5FLN3BDF

@misc{pith2026250207309,
  author       = {Pith},
  title        = {Pith review of: Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5FLN3BDF}},
  note         = {Machine review of arXiv:2502.07309}
}

read the original abstract

Understanding world dynamics is crucial for planning in autonomous driving. Recent methods attempt to achieve this by learning a 3D occupancy world model that forecasts future surrounding scenes based on current observation. However, 3D occupancy labels are still required to produce promising results. Considering the high annotation cost for 3D outdoor scenes, we propose a semi-supervised vision-centric 3D occupancy world model, PreWorld, to leverage the potential of 2D labels through a novel two-stage training paradigm: the self-supervised pre-training stage and the fully-supervised fine-tuning stage. Specifically, during the pre-training stage, we utilize an attribute projection head to generate different attribute fields of a scene (e.g., RGB, density, semantic), thus enabling temporal supervision from 2D labels via volume rendering techniques. Furthermore, we introduce a simple yet effective state-conditioned forecasting module to recursively forecast future occupancy and ego trajectory in a direct manner. Extensive experiments on the nuScenes dataset validate the effectiveness and scalability of our method, and demonstrate that PreWorld achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks.

Figures

Figures reproduced from arXiv: 2502.07309 by the authors.

**Figure 1.** (a) Self-Supervised 3D Occupancy Model can be trained using solely 2D labels as supervision. However, it lacks the capability to forecast future occupancy. In contrast, (b) FullySupervised 3D Occupancy World Model can forecast future occupancy, but it relies on 3D occupancy labels for meaningful results due to its indirect architecture, which employs a frozen 3D occupancy model. To tackle these challenges, our (c)… view at source ↗

**Figure 2.** The architecture of our proposed PreWorld. Firstly, volume features are extracted from multi-view images with an occupancy network. Subsequently, a state-conditioned forecasting module is employed to recursively forecast future volume features using historical features. In the self-supervised pre-training stage, volume features are projected into various attribute fields and supervised by 2D labels through volume re… view at source ↗

**Figure 3.** The proposed state-conditioned forecasting module is simply composed of two MLPs. Ego states can be optionally integrated into the network, as denoted by the dashed arrows. Without loss of generality, our forecasting module is simply composed of two MLPs. We demonstrate that even without intricate design, this simple architecture can still achieve comparable and even superior results to state-ofthe-art methods. … view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Qualitative results of 3D occupancy prediction on the Occ3D-nuScenes validation set. The holistic structure and fine-grained details of the scene are highlighted by orange boxes and red boxes respectively. Compared with existing fully-supervised methods and self-superv…

**Figure 5.** Figure 5: More qualitative results of 3D occupancy prediction on the Occ3D-nuScenes validation set. The holistic structure and fine-grained details of the scene are highlighted by orange boxes and red boxes respectively. barrier cons. veh bus trailer motorcycle traffic cone oth…

**Figure 6.** Figure 6: More qualitative results of 3D occupancy prediction on the Occ3D-nuScenes validation set. The shaded area represents occluded regions where the voxels are not included in the evaluation. In contrast to RenderOcc, our PreWorld makes more cautious predictions for occlud…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

3D and 4D World Modeling: A Survey
cs.CV 2025-09 conditional novelty 5.0 of 10

A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow

Simon Boeder, Fabian Gigengack, and Benjamin Risse. Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow. arXiv preprint arXiv:2402.12792,

arXiv
[7]

Hdmapnet: An online hd map construction and evaluation framework

Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA) , pp. 4628–4634. IEEE, 2022a. Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. V oxformer: Sparse voxel transfo...

arXiv 2022
[8]

Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation

Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934,

arXiv 2012
[10]

Renderocc: Vision-centric 3d occupancy prediction with 2d ren- dering supervision

Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Hongwei Xie, Bing Wang, Li Liu, and Shanghang Zhang. Renderocc: Vision-centric 3d occupancy prediction with 2d ren- dering supervision. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12404–12411. IEEE,

work page 2024
[11]

Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 194–210. Springer,

work page 2020
[12]

Scene as occupancy

12 Published as a conference paper at ICLR 2025 Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8406–8415,

work page 2025
[13]

Occllama: An occupancy-language-action generative world model for autonomous driving

Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy-language-action generative world model for autonomous driving. arXiv preprint arXiv:2409.03272,

arXiv
[14]

Drinet++: Efficient voxel- as-point point cloud segmentation

Maosheng Ye, Rui Wan, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Drinet++: Efficient voxel- as-point point cloud segmentation. arXiv preprint arXiv:2111.08318,

arXiv

Show all 24 references

[15]

Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes

Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430,

arXiv
[16]

Occnerf: Self-supervised multi-camera occupancy prediction with neural radiance fields

Chubin Zhang, Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen Lu. Occnerf: Self-supervised multi-camera occupancy prediction with neural radiance fields. arXiv preprint arXiv:2312.09243, 2023a. Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui ...

arXiv
[17]

Monoocc: Digging into monocular semantic occupancy prediction

Yupeng Zheng, Xiang Li, Pengfei Li, Yuhang Zheng, Bu Jin, Chengliang Zhong, Xiaoxiao Long, Hao Zhao, and Qichao Zhang. Monoocc: Digging into monocular semantic occupancy prediction. arXiv preprint arXiv:2403.08766,

arXiv
[18]

13 Published as a conference paper at ICLR 2025 A M ORE EVALUATIONS A.1 3D O CCUPANCY PREDICTION WITH RAYIOU To address the inconsistent depth penalty issue within the mIoU metric, SparseOcc (Liu et al.,

2025
[19]

introduces a novel metric, RayIoU, designed to enhance the evaluation of 3D occupancy model performance. In order to demonstrate the robustness of our approach as a 3D occupancy model across various metrics, we opt to evaluate PreWorld on the 3D occupancy prediction task using...

2023
[20]

by 2.6 RayIoU. Comparing to purely 3D occupancy super- vision, the proposed self-supervised pre-training stage provides a significant boost in RayIoU from 36.4 to 38.7, which reaffirms the effectiveness of our two-stage training paradigm for PreWorld. Altogether, Table 1 and 8...

2024
[21]

While OccFlowNet outperforms SparseOcc in the mIoU metric with 33.86 over 30.90, its performance notably lags behind SparseOcc in terms of RayIoU

have de- creased. While OccFlowNet outperforms SparseOcc in the mIoU metric with 33.86 over 30.90, its performance notably lags behind SparseOcc in terms of RayIoU. These results indicate that the performance of our PreWorld is not inferior in some categories; rather, our mode...

2025
[22]

15 Published as a conference paper at ICLR 2025 Table 11: Detailed 3D occupancy prediction performance of different data scale utilized in pre- training and fine-tuning stage. Fine-tuning Pre-training mIoU Overall Large Static Small 150 Scenes × 30.26 35.82 24.08 700 Scenes 31...

2025
[23]

The red boxes highlight fine-grained details of the 3D occupancy predictions and the ground truth, while the orange boxes mark holistic structure of an area within the scene

and self-supervised method RenderOcc (Pan et al., 2024), further substantiating the robustness of our PreWorld model and the effectiveness of our novel two-stage training paradigm. The red boxes highlight fine-grained details of the 3D occupancy predictions and the ground trut...

2024
[24]

However, while this approach may lead to higher mIoU scores, its predictions for occluded regions are chaotic, indicating a lack of true understanding of the scene structure

tends to predict thicker surfaces for large static categories. However, while this approach may lead to higher mIoU scores, its predictions for occluded regions are chaotic, indicating a lack of true understanding of the scene structure. On the contrary, our PreWorld makes mor...

2025
[2018]

Gaia-1: A generative world model for autonomous driving

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023a. Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, ...

arXiv
[2020]

Fully sparse 3d panoptic occupancy prediction

Haisong Liu, Haiguang Wang, Yang Chen, Zetong Yang, Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. arXiv preprint arXiv:2312.17118,

arXiv
[2021]

Magicdrive: Street view generation with diverse 3d geometry control

Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601,

arXiv
[2022]

Planning-oriented autonomous driving

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17853–17862, 2023b. Junjie...

arXiv
[2023]

World models

David Ha and J¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122,

arXiv
[2024]

Dif- ferentiable raycasting for self-supervised occupancy forecasting

11 Published as a conference paper at ICLR 2025 Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. Dif- ferentiable raycasting for self-supervised occupancy forecasting. In European Conference on Computer Vision, pp. 353–369. Springer,

2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow

Simon Boeder, Fabian Gigengack, and Benjamin Risse. Occflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow. arXiv preprint arXiv:2402.12792,

arXiv

[2] [7]

Hdmapnet: An online hd map construction and evaluation framework

Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA) , pp. 4628–4634. IEEE, 2022a. Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. V oxformer: Sparse voxel transfo...

arXiv 2022

[3] [8]

Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation

Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934,

arXiv 2012

[4] [10]

Renderocc: Vision-centric 3d occupancy prediction with 2d ren- dering supervision

Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Hongwei Xie, Bing Wang, Li Liu, and Shanghang Zhang. Renderocc: Vision-centric 3d occupancy prediction with 2d ren- dering supervision. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12404–12411. IEEE,

work page 2024

[5] [11]

Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 194–210. Springer,

work page 2020

[6] [12]

Scene as occupancy

12 Published as a conference paper at ICLR 2025 Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8406–8415,

work page 2025

[7] [13]

Occllama: An occupancy-language-action generative world model for autonomous driving

Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy-language-action generative world model for autonomous driving. arXiv preprint arXiv:2409.03272,

arXiv

[8] [14]

Drinet++: Efficient voxel- as-point point cloud segmentation

Maosheng Ye, Rui Wan, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. Drinet++: Efficient voxel- as-point point cloud segmentation. arXiv preprint arXiv:2111.08318,

arXiv