Pith. sign in

REVIEW 3 major objections 7 minor 144 references

The paper argues that self-driving perception foundation models should be organized around four core capabilities—generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding—and that no current system i

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 20:48 UTC pith:5OLQUSI2

load-bearing objection A useful capability-organized survey of foundation models for AD perception whose central 'novel taxonomy' claim is contradicted by its own Table 1 — worth a referee after cleanup, but not the field-defining framing it bills itself as. the 3 major comments →

arxiv 2509.08302 v1 pith:5OLQUSI2 submitted 2025-09-10 cs.RO cs.CV

Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities

Rajendramayavan Sathyam , Yueqi Li This is my paper
classification cs.RO cs.CV
keywords autonomous driving perceptionfoundation modelscapability taxonomygeneralized knowledgespatial understandingmulti-sensor robustnesstemporal understandingself-supervised learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This survey argues that foundation models for self-driving perception should be organized around four capabilities—generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding—rather than around tasks or methods. For each capability it reviews the main technical routes: distillation and pseudo-labeling for generalized knowledge; volumetric models, neural rendering, and masked autoencoders for spatial understanding; cross-modal contrastive learning, knowledge distillation, multi-view consistency, and diffusion for multi-sensor robustness; and 4D prediction, diffusion world models, and temporal contrastive learning for temporal understanding. It claims that no existing system seamlessly integrates all four capabilities into a real-time framework. If the taxonomy is right, it gives developers a capability-driven checklist for building and evaluating perception models and names the integration of these capabilities as the central open problem.

Core claim

The paper's central claim is that a capability-based taxonomy—generalized knowledge, spatial understanding, multi-sensor robustness, temporal understanding—captures what a perception foundation model must master to handle dynamic, long-tail driving, and that this taxonomy should steer research better than task- or method-based surveys do. To support this, the survey classifies recent methods into capability-oriented clusters: VFM/VLM/LLM adaptation mechanisms for generalized knowledge; volumetric models, NeRF/3D Gaussian Splatting rendering, and 3D masked autoencoders for spatial understanding; cross-modality contrastive learning, distillation, multi-view consistency, multi-modal masked auto

What carries the argument

The load-bearing object is the four-capability taxonomy itself, plus the coverage matrix (Table 10) that assigns each surveyed method an X for each capability it provides. The taxonomy does the work of partitioning the literature: each of the four sections maps one capability to a family of techniques, such as knowledge distillation and pseudo-labeling for generalized knowledge; NeRF/3D Gaussian Splatting and masked autoencoders for spatial understanding; contrastive learning, distillation, multi-view consistency, and diffusion for multi-sensor robustness; and 4D forecasting and temporal contrastive learning for temporal understanding. The matrix then serves as the survey's evidence that cap

Load-bearing premise

The taxonomy assumes the four capabilities—generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding—are a complete and cleanly separable decomposition of autonomous-driving perception, so every method can be placed in distinct boxes.

What would settle it

A concrete falsifier would be a deployed or benchmarked perception system that demonstrably operates in real time while satisfying all four capabilities on a held-out distribution-shift suite; observing such a system would falsify the paper's claim that no existing system integrates all four. Alternatively, a controlled study showing that improving temporal understanding necessarily changes multi-sensor robustness would falsify the taxonomy's separability assumption.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Researchers can use the four capabilities as a checklist when designing a perception foundation model, choosing whichever pillar is weakest for their deployment target.
  • The integration claim implies that hybrid systems—foundation models handling high-level reasoning at a reduced rate plus a conventional real-time pipeline—are an interim solution, not the end state.
  • Benchmarks should be built to isolate and stress each capability—for example, corruption benchmarks for multi-sensor robustness and accident-focused scenarios for generalized knowledge—rather than reporting only average-case mAP or IoU.
  • If no current system combines all four capabilities, progress depends on closing integration gaps: latency, calibration, synchronization, and the representation mismatch between dense spatial outputs and object-centric planning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The taxonomy's exhaustive-and-separable assumption is the part most worth testing: temporal understanding and multi-sensor robustness both lean on spatial alignment, so a capability overlap, rather than four independent axes, may be the real structure.
  • A quantitative analogue of Table 10—scoring degree of capability instead of binary X marks—would turn the survey's qualitative claim into a testable rubric and could reveal whether the four categories are actually independent.
  • The survey's benchmark table suggests a natural extension: build a benchmark suite that perturbs one capability axis at a time—sensor dropout, occlusion, unseen categories, temporal discontinuity—so capability-level comparisons can be made fairly across models.
  • The integration bottleneck also implies an opportunity: a model that does combine all four, even at modest performance, would be more informative for the field than another state-of-the-art result on a single task.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This manuscript surveys foundation models for autonomous-driving perception and organizes the literature around a proposed taxonomy of four core capabilities: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding. Each capability is motivated, illustrated with representative methods (e.g., VFM/VLM/LLM distillation, occupancy networks, neural rendering, cross-modal contrastive learning, temporal contrastive learning), and accompanied by challenges. The paper further claims that this capability-based taxonomy is novel and that no existing system integrates all four capabilities into a real-time framework. The final sections discuss benchmark limitations, real-time latency, data bias, regulatory issues, and hallucination risks.

Significance. If the taxonomy is accepted, the survey provides a useful organizational frame for a rapidly growing literature and a practical checklist for model development. Its strengths are the breadth of recent work covered, the structured tables (Tables 3, 6, 7, 9, 10), and the attention to deployment issues such as latency, benchmarks, and hallucination. No new quantitative results or code are presented, so the appropriate standard of assessment is internal consistency and accurate characterization of prior work. Measured against that standard, the paper needs revision: the central novelty claim is internally contradicted by Table 1, and the binary capability assignments in Table 10 are not fully aligned with the text.

major comments (3)
  1. [Section I and Table 1] The abstract and Section I claim a 'novel taxonomy' and state that existing surveys 'frequently overlook' multi-sensor robustness and spatial awareness. However, Table 1 lists survey [6] as covering all four capabilities, including exactly those two. This is a direct internal contradiction: either [6] already organizes around these capabilities, which undermines the novelty claim, or Table 1 mischaracterizes [6], which undermines the reliability of the comparison matrix. The paper provides no section-level comparison or criterion for what counts as 'coverage.' Please revise the claim or provide a precise distinction between method-based and capability-based coverage and show that [6] fails the latter.
  2. [Section VII.A and Table 10] The claim that no existing system integrates all four capabilities into a real-time framework rests on Table 10, but the row assignments are not reliable. For example, the text describes SEAL as combining generalized knowledge with multi-sensor robustness and temporal understanding, yet the Table 10 row 'SLidR [43] / SEAL [44]' appears to mark only two capabilities. The text and table must be aligned, and the coding rules for assigning X marks should be stated explicitly.
  3. [Section I and Table 10] The four capabilities are treated as cleanly separable binary attributes, but they are interdependent: cross-modal fusion requires spatial alignment, temporal modeling often depends on multi-view geometric consistency, and generalized knowledge is used to resolve ambiguities in 3D reconstruction. The paper does not define what 'covers a capability' means or how overlap is adjudicated. Since the 'no seamless integration' conclusion is the survey's central claim, I ask for a short scope definition for each capability and a reproducible coding rubric for Table 10.
minor comments (7)
  1. [Abstract vs. Section VI] The abstract uses 'temporal reasoning' while Section VI and the conclusion use 'temporal understanding.' Please use one term consistently.
  2. [Figure 4 caption] Typo: 'supervise a encoder' should be 'supervise an encoder.'
  3. [Section IV.A and V.E] Typos: 'V olumetric' should be 'Volumetric' in Section IV.A; 'defusion' should be 'diffusion' in Section V.E.
  4. [Section II.E] Capitalization: 'Therefore, They become foundational components' should be 'therefore, they become...'
  5. [Section II.B] The distillation loss equations are set with inline expressions for the softened probabilities, which is hard to read. Please display these as numbered equations.
  6. [Table 10] Rows that group multiple methods under one mark, e.g., 'SLidR [43] / SEAL [44]', obscure differences in capability coverage. Use separate rows or per-method marks.
  7. [Section VII] The survey does not state its own limitations, including the literature selection protocol and the coverage cutoff date. Adding a short limitations paragraph would improve reproducibility.

Circularity Check

0 steps flagged

No circularity: the survey's taxonomy is an interpretive frame; there are no fitted inputs, derived predictions, or load-bearing self-citations.

full rationale

This paper is a survey whose contribution is an organizing taxonomy (generalized knowledge, spatial understanding, multi-sensor robustness, temporal understanding), not a derivation of empirical predictions from first principles. There are no fitted parameters, no equations used to derive results, no predictions from fitted inputs, and no load-bearing self-citations: the authors do not cite their own prior work, and all cited systems are external published methods described by the survey. The four capabilities are introduced by definition in Section I and then used to structure the review; Table 10's X-marks are the authors' qualitative literature judgments, not quantities derived from data. Consequently, there is no chain by which an output reduces to an input by construction. The skeptical concern that Table 1 lists reference [6] as covering all four capabilities, conflicting with the Introduction's claim that prior surveys overlook multi-sensor robustness and spatial awareness, is an internal-consistency/novelty-support issue, not a circularity: resolving it requires checking [6]'s actual scope, not exhibiting an equation-level or citation-level loop. Likewise, the subjective coverage judgments in Table 10 are potential evidence-quality concerns but do not constitute circular reasoning. Thus no circularity is found.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The survey introduces no fitted numbers. Its conceptual content rests on accurate summarization of external work, the validity of the chosen capability decomposition, and the standard assumption that foundation-model pretraining transfers downstream. The four-capability taxonomy itself is an invented organizational entity with no independent falsifiable handle.

axioms (3)
  • domain assumption The cited primary papers are accurately summarized, and each is a legitimate instance of the assigned capability.
    The entire survey stands on grouping external works into four categories; no re-implementation or benchmark comparison is provided.
  • ad hoc to paper The four capabilities are jointly exhaustive and largely orthogonal for AD perception.
    This is the paper's central framing, introduced in Section I and operationalized in Table 10; no evidence establishes exhaustiveness or orthogonality.
  • domain assumption Foundation-model pretraining on diverse data transfers to AD perception.
    Sections II and III assume this premise; it is standard in the literature but not verified by the survey.
invented entities (1)
  • Four-capability taxonomy no independent evidence
    purpose: Provides the survey's organizing structure: generalized knowledge, spatial understanding, multi-sensor robustness, temporal understanding.
    A conceptual framework asserted by the authors; no independent benchmark or analysis demonstrates that these four axes are the correct or complete decomposition of AD perception.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities." pith.science (2026). https://pith.science/paper/5OLQUSI2

@misc{pith2026250908302,
  author       = {Pith},
  title        = {Pith review of: Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OLQUSI2}},
  note         = {Machine review of arXiv:2509.08302}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Foundation models are revolutionizing autonomous driving perception, transitioning the field from narrow, task-specific deep learning models to versatile, general-purpose architectures trained on vast, diverse datasets. This survey examines how these models address critical challenges in autonomous perception, including limitations in generalization, scalability, and robustness to distributional shifts. The survey introduces a novel taxonomy structured around four essential capabilities for robust performance in dynamic driving environments: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal reasoning. For each capability, the survey elucidates its significance and comprehensively reviews cutting-edge approaches. Diverging from traditional method-centric surveys, our unique framework prioritizes conceptual design principles, providing a capability-driven guide for model development and clearer insights into foundational aspects. We conclude by discussing key challenges, particularly those associated with the integration of these capabilities into real-time, scalable systems, and broader deployment challenges related to computational demands and ensuring model reliability against issues like hallucinations and out-of-distribution failures. The survey also outlines crucial future research directions to enable the safe and effective deployment of foundation models in autonomous driving systems.

Figures

Figures reproduced from arXiv: 2509.08302 by Rajendramayavan Sathyam, Yueqi Li.

Figure 1
Figure 1. Figure 1: FIGURE 1 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: FIGURE 7 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: FIGURE 8 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: FIGURE 10 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: FIGURE 12 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: FIGURE 13 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: FIGURE 14 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

144 extracted references · 47 canonical work pages · 4 internal anchors

  1. [1]

    3d object detection for autonomous driving: A survey,

    R. Qian, X. Lai, and X. Li, “3d object detection for autonomous driving: A survey,”Pattern Recognition, vol. 130, p. 108796, 2022

    2022

  2. [2]

    Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and out- looks,

    K. Muhammad, T. Hussain, H. Ullah, J. Del Ser, M. Rezaei, N. Kumar, M. Hijji, P. Bellavista, and V . H. C. de Albuquerque, “Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and out- looks,”IEEE Transactions on Intelligent Transporta- tion Systems, vol. 23, no. 12, pp. 22 694–22 715, 2022

    2022

  3. [3]

    A review of deep learning-based visual multi-object tracking algorithms for autonomous driving,

    S. Guo, S. Wang, Z. Yang, L. Wang, H. Zhang, P. Guo, Y . Gao, and J. Guo, “A review of deep learning-based visual multi-object tracking algorithms for autonomous driving,”Applied Sciences, vol. 12, no. 21, p. 10741, 2022

    2022

  4. [4]

    Towards long-tailed 3d detection,

    N. Peri, A. Dave, D. Ramanan, and S. Kong, “Towards long-tailed 3d detection,” inConference on Robot Learning. PMLR, 2023, pp. 1904–1915

    2023

  5. [5]

    On the opportuni- ties and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskillet al., “On the opportuni- ties and risks of foundation models,”arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Forging vision foundation models for autonomous driving: Challenges, methodologies, and opportunities,

    X. Yan, H. Zhang, Y . Cai, J. Guo, W. Qiu, B. Gao, K. Zhou, Y . Zhao, H. Jin, J. Gaoet al., “Forging vision foundation models for autonomous driving: Challenges, methodologies, and opportunities,”arXiv preprint arXiv:2401.08045, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Applications of large scale foundation models for autonomous driving,

    Y . Huang, Y . Chen, and Z. Li, “Applications of large scale foundation models for autonomous driving,” arXiv preprint arXiv:2311.12144, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    A survey for foundation models in au- tonomous driving,

    H. Gao, Z. Wang, Y . Li, K. Long, M. Yang, and Y . Shen, “A survey for foundation models in au- tonomous driving,”arXiv preprint arXiv:2402.01105, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Prospective role of foundation models in advancing autonomous vehicles,

    J. Wu, B. Gao, J. Gao, J. Yu, H. Chu, Q. Yu, X. Gong, Y . Chang, H. E. Tseng, H. Chenet al., “Prospective role of foundation models in advancing autonomous vehicles,”Research, vol. 7, p. 0399, 2024

    2024

  10. [10]

    Llm4drive: A survey of large language models for autonomous driving,

    Z. Yang, X. Jia, H. Li, and J. Yan, “Llm4drive: A survey of large language models for autonomous driving,”arXiv preprint arXiv:2311.01043, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Vision language models in autonomous driving: A survey and outlook,

    X. Zhou, M. Liu, E. Yurtsever, B. L. Zagar, W. Zim- mer, H. Cao, and A. C. Knoll, “Vision language models in autonomous driving: A survey and outlook,” IEEE Transactions on Intelligent Vehicles, 2024

    2024

  12. [12]

    A simple framework for contrastive learning of vi- sual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of vi- sual representations,” inInternational conference on machine learning. PmLR, 2020, pp. 1597–1607. 26 VOLUME 00, 2024

    2020

  13. [13]

    Momentum contrast for unsupervised visual repre- sentation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual repre- sentation learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2020, pp. 9729–9738

    2020

  14. [14]

    Improved baselines with momentum contrastive learning,

    X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,”arXiv preprint arXiv:2003.04297, 2020

    Pith/arXiv arXiv 2003

  15. [15]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009

    2022

  16. [16]

    An image is worth 16x16 words: Transformers for image recogni- tion at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recogni- tion at scale,”arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  17. [17]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  18. [18]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Com- puter Vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  19. [19]

    Self- training with noisy student improves imagenet classi- fication,

    Q. Xie, M.-T. Luong, E. Hovy, and Q. V . Le, “Self- training with noisy student improves imagenet classi- fication,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 687–10 698

    2020

  20. [20]

    Bootstrap your own latent-a new approach to self- supervised learning,

    J.-B. Grill, F. Strub, F. Altch ´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azaret al., “Bootstrap your own latent-a new approach to self- supervised learning,”Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020

    2020

  21. [21]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660

    2021

  22. [22]

    Structure-from- motion revisited,

    J. L. Schonberger and J.-M. Frahm, “Structure-from- motion revisited,” inProceedings of the IEEE con- ference on computer vision and pattern recognition, 2016, pp. 4104–4113

    2016

  23. [23]

    Multi-view stereo: A tutorial,

    Y . Furukawa, C. Hern´andezet al., “Multi-view stereo: A tutorial,”Foundations and trends® in Computer Graphics and Vision, vol. 9, no. 1-2, pp. 1–148, 2015

    2015

  24. [24]

    Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,

    C. Cadena, L. Carlone, H. Carrillo, Y . Latif, D. Scara- muzza, J. Neira, I. Reid, and J. J. Leonard, “Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,”IEEE Transactions on robotics, vol. 32, no. 6, pp. 1309– 1332, 2016

    2016

  25. [25]

    3d-r2n2: A unified approach for single and multi- view 3d object reconstruction,

    C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese, “3d-r2n2: A unified approach for single and multi- view 3d object reconstruction,” inComputer vision– ECCV 2016: 14th European conference, amsterdam, the netherlands, October 11-14, 2016, proceedings, part VIII 14. Springer, 2016, pp. 628–644

    2016

  26. [26]

    Predicting depth, surface normals and semantic labels with a common multi- scale convolutional architecture,

    D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi- scale convolutional architecture,” inProceedings of the IEEE international conference on computer vision, 2015, pp. 2650–2658

    2015

  27. [27]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Bar- ron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99– 106, 2021

    2021

  28. [28]

    3d gaussian splatting for real-time radiance field rendering

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Dret- takis, “3d gaussian splatting for real-time radiance field rendering.”ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023

    2023

  29. [29]

    Yolov3: An incremen- tal improvement,

    J. Redmon and A. Farhadi, “Yolov3: An incremen- tal improvement,”arXiv preprint arXiv:1804.02767, 2018

    Pith/arXiv arXiv 2018

  30. [30]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  31. [31]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  33. [33]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Y . Loet al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  34. [34]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  36. [36]

    Grounding dino: Marrying dino with grounded pre-training for VOLUME 00, 2024 27 Authoret al.: Preparation of Papers for IEEE OPEN JOURNALS open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Suet al., “Grounding dino: Marrying dino with grounded pre-training for VOLUME 00, 2024 27 Authoret al.: Preparation of Papers for IEEE OPEN JOURNALS open-set object detection,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 38–55

    2024

  37. [37]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020

    2020

  38. [38]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  39. [39]

    Video diffusion models,

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” Advances in Neural Information Processing Systems, vol. 35, pp. 8633–8646, 2022

    2022

  40. [40]

    3d shape generation and completion through point-voxel diffusion,

    L. Zhou, Y . Du, and J. Wu, “3d shape generation and completion through point-voxel diffusion,” inPro- ceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 5826–5835

    2021

  41. [41]

    Diffusion-based signed distance fields for 3d shape generation,

    J. Shim, C. Kang, and K. Joo, “Diffusion-based signed distance fields for 3d shape generation,” inProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 20 887–20 897

    2023

  42. [42]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askellet al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  43. [43]

    Image-to-lidar self-supervised distilla- tion for autonomous driving data,

    C. Sautier, G. Puy, S. Gidaris, A. Boulch, A. Bursuc, and R. Marlet, “Image-to-lidar self-supervised distilla- tion for autonomous driving data,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9891–9901

    2022

  44. [44]

    Segment any point cloud sequences by distilling vision foundation models,

    Y . Liu, L. Kong, J. Cen, R. Chen, W. Zhang, L. Pan, K. Chen, and Z. Liu, “Segment any point cloud sequences by distilling vision foundation models,” Advances in Neural Information Processing Systems, vol. 36, pp. 37 193–37 229, 2023

    2023

  45. [45]

    Better call sal: Towards learning to segment anything in lidar,

    A. O ˇsep, T. Meinhardt, F. Ferroni, N. Peri, D. Ra- manan, and L. Leal-Taix ´e, “Better call sal: Towards learning to segment anything in lidar,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 71–90

    2024

  46. [46]

    Sam4udass: When sam meets unsupervised domain adaptive semantic segmentation in intelligent ve- hicles,

    W. Yan, Y . Qian, H. Zhuang, C. Wang, and M. Yang, “Sam4udass: When sam meets unsupervised domain adaptive semantic segmentation in intelligent ve- hicles,”IEEE Transactions on Intelligent Vehicles, vol. 9, no. 2, pp. 3396–3408, 2024

    2024

  47. [47]

    Occnerf: Self-supervised multi- camera occupancy prediction with neural radiance fields,

    C. Zhang, J. Yan, Y . Wei, J. Li, L. Liu, Y . Tang, Y . Duan, and J. Lu, “Occnerf: Self-supervised multi- camera occupancy prediction with neural radiance fields,”CoRR, 2023

    2023

  48. [48]

    Open 3D World in Autonomous Driving

    X. Cheng and L. Li, “Open 3d world in autonomous driving,”arXiv preprint arXiv:2408.10880, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    Ovo: Open-vocabulary occupancy,

    Z. Tan, Z. Dong, C. Zhang, W. Zhang, H. Ji, and H. Li, “Ovo: Open-vocabulary occupancy,”arXiv preprint arXiv:2305.16133, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Clip2scene: Towards label-efficient 3d scene understanding by clip,

    R. Chen, Y . Liu, L. Kong, X. Zhu, Y . Ma, Y . Li, Y . Hou, Y . Qiao, and W. Wang, “Clip2scene: Towards label-efficient 3d scene understanding by clip,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7020–7030

    2023

  51. [51]

    Vlm2scene: Self- supervised image-text-lidar learning with foundation models for autonomous driving scene understanding,

    G. Liao, J. Li, and X. Ye, “Vlm2scene: Self- supervised image-text-lidar learning with foundation models for autonomous driving scene understanding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 4, 2024, pp. 3351–3359

    2024

  52. [52]

    Unsupervised 3d perception with 2d vision-language distillation for autonomous driv- ing,

    M. Najibi, J. Ji, Y . Zhou, C. R. Qi, X. Yan, S. Ettinger, and D. Anguelov, “Unsupervised 3d perception with 2d vision-language distillation for autonomous driv- ing,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8602– 8612

    2023

  53. [53]

    Opensight: A simple open-vocabulary framework for lidar-based object detection,

    H. Zhang, J. Xu, T. Tang, H. Sun, X. Yu, Z. Huang, and K. Yu, “Opensight: A simple open-vocabulary framework for lidar-based object detection,” inEu- ropean Conference on Computer Vision. Springer, 2024, pp. 1–19

    2024

  54. [54]

    SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model

    D. Zhang, D. Liang, H. Yang, Z. Zou, X. Ye, Z. Liu, and X. Bai, “Sam3d: Zero-shot 3d object detec- tion via segment anything model,”arXiv preprint arXiv:2306.02245, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  55. [55]

    Gpt- driver: Learning to drive with gpt,

    J. Mao, Y . Qian, J. Ye, H. Zhao, and Y . Wang, “Gpt- driver: Learning to drive with gpt,”arXiv preprint arXiv:2310.01415, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning,

    S. Wang, Z. Yu, X. Jiang, S. Lan, M. Shi, N. Chang, J. Kautz, Y . Li, and J. M. Alvarez, “Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning,”arXiv preprint arXiv:2405.01533, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,”IEEE Robotics and Automation Letters, 2024

    2024

  58. [58]

    Dol- phins: Multimodal language model for driving,

    Y . Ma, Y . Cao, J. Sun, M. Pavone, and C. Xiao, “Dol- phins: Multimodal language model for driving,” in European Conference on Computer Vision. Springer, 2024, pp. 403–420

    2024

  59. [59]

    Emma: End-to- end multimodal model for autonomous driving,

    J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp, Y . Zhou, J. Guo, D. Anguelov, and M. Tan, “Emma: End-to- end multimodal model for autonomous driving,”arXiv preprint arXiv:2410.23262, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Lidar-llm: Exploring the potential of large language models for 3d lidar understanding,

    S. Yang, J. Liu, R. Zhang, M. Pan, Z. Guo, X. Li, Z. Chen, P. Gao, H. Li, Y . Guoet al., “Lidar-llm: Exploring the potential of large language models for 3d lidar understanding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 9, 2025, pp. 9247–9255

    2025

  61. [61]

    A survey on hallucination in large language mod- els: Principles, taxonomy, challenges, and open ques- tions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qinet al., 28 VOLUME 00, 2024 “A survey on hallucination in large language mod- els: Principles, taxonomy, challenges, and open ques- tions,”ACM Transactions on Information Systems, vol. 43, no. 2, pp. 1–55, 2025

    2024

  62. [62]

    Retrieval-augmented genera- tion for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschelet al., “Retrieval-augmented genera- tion for knowledge-intensive nlp tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  63. [63]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection,

    A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learning to retrieve, generate, and critique through self-reflection,” 2024

    2024

  64. [64]

    Driving with llms: Fusing object-level vector modal- ity for explainable autonomous driving,

    L. Chen, O. Sinavski, J. H ¨unermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, “Driving with llms: Fusing object-level vector modal- ity for explainable autonomous driving,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14 093–14 100

    2024

  65. [65]

    A survey on occupancy perception for autonomous driv- ing: The information fusion perspective,

    H. Xu, J. Chen, S. Meng, Y . Wang, and L.-P. Chau, “A survey on occupancy perception for autonomous driv- ing: The information fusion perspective,”Information Fusion, vol. 114, p. 102671, 2025

    2025

  66. [66]

    Neural vol- umetric world models for autonomous driving,

    Z. Huang, J. Zhang, and E. Ohn-Bar, “Neural vol- umetric world models for autonomous driving,” in European Conference on Computer Vision. Springer, 2024, pp. 195–213

    2024

  67. [67]

    Tri-perspective view for vision-based 3d semantic oc- cupancy prediction,

    Y . Huang, W. Zheng, Y . Zhang, J. Zhou, and J. Lu, “Tri-perspective view for vision-based 3d semantic oc- cupancy prediction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2023, pp. 9223–9232

    2023

  68. [68]

    V oxformer: Sparse voxel transformer for camera-based 3d se- mantic scene completion,

    Y . Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, S. Fidler, C. Feng, and A. Anandkumar, “V oxformer: Sparse voxel transformer for camera-based 3d se- mantic scene completion,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 9087–9098

    2023

  69. [69]

    Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,

    Y . Zhang, Z. Zhu, and D. Du, “Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction,” inProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 2023, pp. 9433–9443

    2023

  70. [70]

    Fully sparse 3d occupancy prediction,

    H. Liu, Y . Chen, H. Wang, Z. Yang, T. Li, J. Zeng, L. Chen, H. Li, and L. Wang, “Fully sparse 3d occupancy prediction,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 54–71

    2024

  71. [71]

    Hybridocc: Nerf enhanced transformer-based multi- camera 3d occupancy prediction,

    X. Zhao, B. Chen, M. Sun, D. Yang, Y . Wang, X. Zhang, M. Li, D. Kou, X. Wei, and L. Zhang, “Hybridocc: Nerf enhanced transformer-based multi- camera 3d occupancy prediction,”IEEE Robotics and Automation Letters, 2024

    2024

  72. [72]

    Renderocc: Vision- centric 3d occupancy prediction with 2d rendering supervision,

    M. Pan, J. Liu, R. Zhang, P. Huang, X. Li, H. Xie, B. Wang, L. Liu, and S. Zhang, “Renderocc: Vision- centric 3d occupancy prediction with 2d rendering supervision,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 12 404–12 411

    2024

  73. [73]

    S-nerf++: Autonomous driving simu- lation via neural reconstruction and generation,

    Y . Chen, J. Zhang, Z. Xie, W. Li, F. Zhang, J. Lu, and L. Zhang, “S-nerf++: Autonomous driving simu- lation via neural reconstruction and generation,”IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 2025

    2025

  74. [74]

    Selfocc: Self-supervised vision-based 3d occupancy prediction,

    Y . Huang, W. Zheng, B. Zhang, J. Zhou, and J. Lu, “Selfocc: Self-supervised vision-based 3d occupancy prediction,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2024, pp. 19 946–19 956

    2024

  75. [75]

    Renderworld: World model with self-supervised 3d label,

    Z. Yan, W. Dong, Y . Shao, Y . Lu, L. Haiyang, J. Liu, H. Wang, Z. Wang, Y . Wang, F. Remondinoet al., “Renderworld: World model with self-supervised 3d label,”arXiv preprint arXiv:2409.11356, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    Gaussian- flowocc: Sparse and weakly supervised occupancy es- timation using gaussian splatting and temporal flow,

    S. Boeder, F. Gigengack, and B. Risse, “Gaussian- flowocc: Sparse and weakly supervised occupancy es- timation using gaussian splatting and temporal flow,” arXiv preprint arXiv:2502.17288, 2025

    Pith/arXiv arXiv 2025

  77. [77]

    Street gaussians: Modeling dynamic urban scenes with gaussian splat- ting,

    Y . Yan, H. Lin, C. Zhou, W. Wang, H. Sun, K. Zhan, X. Lang, X. Zhou, and S. Peng, “Street gaussians: Modeling dynamic urban scenes with gaussian splat- ting,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 156–173

    2024

  78. [78]

    Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction,

    Y . Huang, W. Zheng, Y . Zhang, J. Zhou, and J. Lu, “Gaussianformer: Scene as gaussians for vision-based 3d semantic occupancy prediction,” inEuropean Con- ference on Computer Vision. Springer, 2024, pp. 376– 393

    2024

  79. [79]

    Surroundocc: Multi-camera 3d occupancy pre- diction for autonomous driving,

    Y . Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu, “Surroundocc: Multi-camera 3d occupancy pre- diction for autonomous driving,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 21 729–21 740

    2023

  80. [80]

    Uno: Unsupervised occupancy fields for per- ception and forecasting,

    B. Agro, Q. Sykora, S. Casas, T. Gilles, and R. Ur- tasun, “Uno: Unsupervised occupancy fields for per- ception and forecasting,” inCVPR, 2024

    2024

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.