Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AutoOcc auto-labels 3D occupancy with no human annotation

desk verdict New pipeline, real efficiency gains, but the headline comparison depends on undocumented baseline replications. read the letter →

arxiv 2502.04981 v3 pith:JXLVQBOH submitted 2025-02-07 cs.CV

Xiaoyu Zhou , Jingqi Wang , Yongtao Wang , Yufei Wei , Nan Dong , Ming-Hsuan Yang This is my paper · ORCID
classification cs.CV
keywords semanticoccupancyannotationGaussiansplattingvision-languagemodelsopen-endedsemanticsautonomousdriving3Dreconstructionzero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes AutoOcc, a fully automatic pipeline for generating 3D semantic occupancy labels from camera images, with optional LiDAR support, without any human annotation. The central claim is that this vision-centric pipeline outperforms existing automated occupancy annotation methods on the Occ3D-nuScenes benchmark while also supporting open-ended semantic categories, meaning objects not present in any predefined class list. If correct, AutoOcc would let autonomous driving and robotics systems produce their own dense 3D labels on demand, covering rare or novel object types that current close-set pipelines would fold into an 'other' class. The paper further reports robust performance in static and dynamic scenes, as well as cross-dataset zero-shot generalization to SemanticKITTI.

What carries the argument

The central object is VL-GS (Vision-Language Guided Gaussian Splatting), a set of semantic-aware scalable Gaussians that act as intermediate geometric descriptors of the scene. Each Gaussian carries semantic attributes and a scaling factor, and a semantic rendering equation using alpha-blending yields the 2D semantic map used in optimization. A cumulative Gaussian-to-voxel splatting step aggregates the Gaussians onto a voxel grid, weighting each voxel's label by occupied depth, opacity, and semantic probability. Dynamic objects are handled by a self-estimated flow module that computes Chamfer-distance correspondences between temporally paired Gaussians sharing the same semantics, and an optional LiDAR geometry loss aligns Gaussian centers with multi-frame LiDAR anchor points.

What would settle it

Run AutoOcc on scenes where a visible object is deliberately excluded from the VLM's attention, for instance by prompting the VLM to ignore a specific class, and check whether the resulting occupancy misses that object. Alternatively, introduce a known scale error into the UniDepth estimates and measure how directly it propagates to the occupancy geometry; if the Gaussian optimization corrects for it, the reconstruction is doing real 3D reasoning, and if not, the 2D priors are the true bottleneck.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that vision-language model (VLM) attention maps can serve as a complete semantic prior for automated 3D occupancy annotation when fused through differentiable Gaussian splatting. AutoOcc prompts a VLM to list all objects in a scene, converts the resulting attention maps into prompts for SAM segmentation and UniDepth depth estimation, and uses those 2D priors to build semantic-aware scalable Gaussians (VL-GS) as an intermediate 3D representation. A cumulative Gaussian-to-voxel splatting algorithm then converts the optimized Gaussians into labeled occupancy voxels. The paper reports that this pipeline achieves higher IoU and mIoU than existing automated pipelines, including LiDAR-based ones, while requiring no manual labels and no post-processing.

Load-bearing premise

The whole pipeline stands on the assumption that the 2D semantic attention maps, SAM masks, and UniDepth depth estimates are accurate and complete enough to drive the 3D Gaussian optimization, with no independent 3D verification beyond the optimization itself.

Editorial extensions

If this is right

  • Automated 3D occupancy label generation becomes possible without human annotation, removing a major bottleneck in building training datasets for occupancy perception.
  • Open-ended semantics means the pipeline can label objects outside any predefined category set, which matters for safety-critical rare objects in real-world driving.
  • Vision-centric annotation can match or exceed LiDAR-based pipelines, reducing reliance on expensive sensor setups for label production.
  • Dynamic objects are annotated with spatiotemporal consistency, so moving vehicles and pedestrians do not create ghost occupancies.
  • The pipeline uses fewer parameters and less memory than voxel- or mesh-based alternatives, making it practical on a single moderately sized GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If VLM attention quality continues to improve, AutoOcc's label accuracy should improve without any change to the 3D reconstruction pipeline, since the VLM acts as the sole semantic oracle.
  • The same 2D-to-3D distillation strategy could be extended to video-language models, which might yield even better temporal consistency for dynamic scenes.
  • Because the occupancy output is produced by reconstruction, it may serve as a pseudo-label engine for training downstream occupancy predictors, potentially enabling self-improvement cycles.
  • The method's dependence on 2D priors suggests a testable threshold: annotation quality will scale with the recall of the VLM's object list, so cluttered or rare-object scenes will stress-test the open-ended claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AutoOcc, a fully automatic pipeline for open-ended semantic 3D occupancy annotation. The method uses VLM-generated semantic attention maps as prompts for SAM and UniDepth, introduces semantic-aware scalable Gaussians with a self-estimated flow module for dynamic objects, and produces voxel occupancy via cumulative Gaussian-to-voxel splatting. The authors claim that AutoOcc outperforms existing automated occupancy annotation methods without human labels, supports open-ended categories, and handles static and dynamic scenes. Experiments are reported on Occ3D-nuScenes (Table 2) and zero-shot SemanticKITTI (Table 3), with ablations and an efficiency comparison. The central comparative result depends on a VLM-based replication of SurroundOcc and OpenOcc that is not described in sufficient detail.

Significance. The idea of using language-model attention to guide differentiable reconstruction is timely and, if the claims hold, would be a useful contribution to scalable occupancy annotation. The method explicitly targets open-ended categories, which addresses a real limitation of fixed-vocabulary auto-labelers. Table 4 suggests significant efficiency gains (5.0 GB memory and 0.3 M primitives for AutoOcc versus 32-73 GB and 0.8-3.0 M for the other label-free methods), and Table 3 indicates plausible cross-dataset transfer. These strengths are undercut by the unverifiable baseline replication and the mismatch between the open-ended claim and the fixed-class evaluation. With the replication protocol and open-ended evaluation supplied, the contribution would be solid.

major comments (4)
  1. [§4.2, Table 2] The comparative claim in the abstract is supported by numbers for SurroundOcc* and OpenOcc* that are described only as 'replicated by replacing the manually annotated results with the semantic point clouds projected from VLMs.' The manuscript does not state which VLM was used, which prompt or attention extraction procedure produced the semantic point clouds, how multi-view labels were fused, what thresholds were applied, or whether the original methods' post-processing steps (SurroundOcc's mesh construction and nearest-neighbor densification; OpenOcc's AAP densification) were retained. Because the headline claim is 'outperforms existing automated occupancy annotation methods without human labels', an unfavorable or inconsistent replication protocol would make the reported mIoU gap (20.92 vs 18.59 and 17.76) uninformative. Please provide a complete replication protocol and, ideally, release the code so the comparison can be checked.
  2. [§4.3, Table 3] The 'open-ended' capability is evaluated only against a fixed set of SemanticKITTI classes, and the manuscript never specifies how the dynamically generated open-ended vocabulary from the VLM is matched to those benchmark classes. This tests cross-dataset generalization to known classes, not open-ended annotation of objects outside any predefined category. Moreover, §4.1 states that 'other' and 'other flat' are excluded from evaluation, which removes exactly the classes that would capture unknown semantics. Please add an explicit open-ended evaluation protocol, e.g., measuring detection/annotation accuracy for categories absent from both nuScenes and SemanticKITTI, or analyzing sensitivity to vocabulary size.
  3. [§3.2, Eq. (6)] The self-estimated flow module relies on a dynamic threshold ρ that is never specified, and no ablation or sensitivity analysis is reported for it. As written, the right-hand side of Eq. (6) can be negative when the mean displacement exceeds ρ, so it does not by itself define an indicator function; the subsequent grouping of dynamic Gaussians is therefore underspecified. Since dynamic-object handling is one of the paper's stated contributions, please define the thresholding procedure, report how ρ was chosen, and provide an ablation over its value.
  4. [§3.2, Eqs. (3)-(4), (8)] The scalable Gaussian and the cumulative splatting are not described precisely enough to reproduce. Eq. (3) introduces η, κ, and Σ without a clear definition of the ray geometry, and the expression mixing η^{-1} with covariance entries is dimensionally opaque. Eq. (8) similarly combines an occupied depth d_i, a Gaussian value G(x_i), opacity, and semantic softmax without specifying the voxel coordinate frame or how multi-Gaussian votes are normalized. Please rewrite these definitions with explicit notation and, if possible, include pseudo-code for the splatting step.
minor comments (5)
  1. [§3.1, Eq. (1)] The attention aggregation formula is unclear; please specify what H′ is, why the sum runs over l = 0..L, and how the per-token attention tensor is reduced to a 2D map. As written, it is difficult to connect Eq. (1) to the rasterized attention maps M.
  2. [§4.1] The statement that evaluation excludes 'other' and 'other flat' follows prior work, but for an open-ended method it would be informative to also report mIoU with these classes included.
  3. [Table 4 caption] The 'Number' column is not defined in the caption; specify whether it is the number of Gaussian primitives, points, or voxels, and add the unit (e.g., M).
  4. [References] Reference [29] appears twice in the citation '[29, 29]' in Section 2, and the reference list contains both the arXiv and ECCV versions of Grounding DINO as [30] and [31]; please deduplicate these entries.
  5. [§3.2, after Eq. (6)] The text refers to 'the centroid position at the i-th frame is denoted by o_i', but Eq. (6) uses p_t and p_{t+Δt}; align the notation to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are evaluated against external human-annotated benchmarks and do not reduce to the method's own inputs.

full rationale

The paper's central claim—that AutoOcc outperforms prior automated occupancy annotation pipelines without human labels—is supported by comparison against external human-annotated benchmarks (Occ3D-nuScenes for semantic occupancy and SemanticKITTI for cross-dataset zero-shot transfer). The output semantic occupancy is produced by optimizing semantic-aware Gaussians against 2D VLM attention maps, SAM masks, UniDepth estimates, optional LiDAR anchors, and rendering losses; it is not defined in terms of, nor fitted to, the benchmark occupancy labels used for evaluation. Equations (1), (3), (5), (7), and (8) are computational components (attention aggregation, ellipsoid depth, Chamfer flow, LiDAR geometry loss, and voxel splatting) with no term that reduces to the reported IoU/mIoU. The use of the authors' prior attention-map work [29] and DrivingGaussian [67] is component-level self-citation, not a load-bearing uniqueness theorem, and these components do not define the benchmark metric. The replication of SurroundOcc* and OpenOcc* via VLM-projected semantic points is described only briefly, which is a reproducibility or fairness concern, but it is not a circular reduction because those baseline numbers are not produced by AutoOcc's own equations. No circular step is therefore exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 3 invented entities

The central claim rests on the reliability of 2D priors from VLM, SAM, and UniDepth, plus the convergence of per-scene Gaussian optimization. These are domain assumptions, not proven in the paper. The only explicit hand-set parameters that are load-bearing are the dynamic threshold rho and the semantic scaling factor. The three proposed algorithmic components are new, but their evidence is internal to the paper. This is a moderate assumption burden for an empirical systems paper.

free parameters (2)
  • Dynamic threshold rho = not reported
    Defined in Eq. (6), rho determines whether a Gaussian is classified as dynamic. It is a hand-set threshold without a stated value or fitting procedure. It directly affects the dynamic object handling, which is a key claim.
  • Semantic scaling factor = not reported
    The semantic-aware scalable Gaussian adaptively adjusts a scaling factor based on gradient magnitude and occupied range (Section 3.2), but no explicit formula or numerical value is given. The scaling behavior is central to the multi-granularity representation.
assumptions (4)
  • domain assumption VLM attention maps reliably enumerate and localize all relevant scene objects when prompted with a generic open-ended instruction.
    Section 3.1 builds the whole pipeline on these attention maps. If the VLM misses objects or produces unfaithful attention, the occupancy annotation will be incomplete or wrong. No quantitative verification of attention-map quality is provided.
  • domain assumption SAM instance masks, guided by the attention maps, accurately delineate object boundaries.
    SAM segmentation outputs are used as pseudo instance masks in Section 3.1. Segmentation errors directly propagate to Gaussian semantics.
  • domain assumption UniDepth provides metric depth with correct scale for all relevant scene regions.
    UniDepth is used to produce pseudo 3D points and initialize Gaussians in Section 3.1. If depth has scale biases, the reconstructed 3D geometry will be misaligned with the true scene.
  • domain assumption Gaussian splatting optimization converges to an accurate 3D reconstruction from multi-view 2D images without any 3D supervision.
    The entire reconstruction and semantic assignment in Section 3.2 relies on 2D rendering losses. Convergence to a correct 3D structure is assumed even though per-scene optimization can be trapped in local optima, especially for textureless or dynamic regions.
invented entities (3)
  • Semantic-aware scalable Gaussian
    purpose: Intermediate 3D representation with per-Gaussian semantic labels and scale factors that adapt to object size, used to reconstruct the scene.
    The representation is evaluated only through the paper's own benchmark comparisons. No independent source confirms that it behaves as intended.
  • Self-estimated flow module
    purpose: Identifies dynamic objects and assigns flow vectors to Gaussians so that moving objects can be reconstructed over time.
    The module is demonstrated to improve results in the paper's ablations (Table 5), but there is no external evidence that the flow estimates correspond to true scene motion.
  • Cumulative Gaussian-to-voxel splatting
    purpose: Converts the optimized Gaussian representation into a volumetric semantic occupancy grid.
    This is a new algorithm introduced by the paper; its validity rests on the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting." pith.science (2026). https://pith.science/paper/JXLVQBOH

@misc{pith2026250204981,
  author       = {Pith},
  title        = {Pith review of: AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splatting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXLVQBOH}},
  note         = {Machine review of arXiv:2502.04981}
}
read the original abstract

Obtaining high-quality 3D semantic occupancy from raw sensor data remains an essential yet challenging task, often requiring extensive manual labeling. In this work, we propose AutoOcc, a vision-centric automated pipeline for open-ended semantic occupancy annotation that integrates differentiable Gaussian splatting guided by vision-language models. We formulate the open-ended semantic 3D occupancy reconstruction task to automatically generate scene occupancy by combining attention maps from vision-language models and foundation vision models. We devise semantic-aware Gaussians as intermediate geometric descriptors and propose a cumulative Gaussian-to-voxel splatting algorithm that enables effective and efficient occupancy annotation. Our framework outperforms existing automated occupancy annotation methods without human labels. AutoOcc also enables open-ended semantic occupancy auto-labeling, achieving robust performance in both static and dynamically complex scenarios.

Figures

Figures reproduced from arXiv: 2502.04981 by the authors.

Figure 1
Figure 1. AutoOcc is a fully automatic, vision-centric pipeline for open-ended semantic 3D occupancy annotation. Our method achieves more efficient and effective semantic occupancy auto-labeling by integrating vision-language guidance with differentiable reconstruction. AutoOcc supports open-ended semantic annotation and effectively handles dynamic objects, without relying on any human annotations. Abstract Obtaining high-qua… view at source ↗
Figure 2
Figure 2. Overall pipeline of our method. AutoOcc is a vision-centric automated pipeline for semantic occupancy annotation. Our method starts with multi-view image inputs (optionally with LiDAR), extracts semantic attention maps from VLMs, and refines a dynamic semantic query list. We then propose Vision-Language Guided Gaussian Splatting (VL-GS), incorporating semantic-aware scalable Gaussians and self-estimated flow for dyn… view at source ↗
Figure 3
Figure 3. Vision-Language Guided Gaussian Splatting (VL￾GS) efficiently reconstructs semantic instances using a scalable strategy guided by semantic attention maps from VLMs. Addi￾tionally, VL-GS models dynamic objects through dynamic Gaus￾sians driven by self-estimated flow. propose designing a semantic-aware scalable Gaussian that adaptively scales and reconstructs different semantic ob￾jects. Unlike dense voxels or point c… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative results of semantic occupancy annotation on Occ3D-nuScenes [46]. Our method enables high-quality annotation of semantic 3D occupancy, capturing fine-grained geometry, structurally challenging regions, and dynamic objects across complex scenes. demonstrates …
Figure 6
Figure 6. Figure 6: Semantic occupancy of dynamics. AutoOcc accurately annotates the semantic occupancy of dynamic objects, maintains spatiotemporal consistency, and infers occluded parts [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of our method with human an [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splatting

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A camera-only Gaussian-surfel pipeline reconstructs full Waymo scenes, converts them to binary occupancy labels, and trains CVT-Occ to generalize on Occ3D-Waymo and Occ3D-nuScenes at a level close to or above LiDAR-la...

  2. From 2D to 3D Cognition: A Brief Survey of General World Models

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A survey proposing a two-pillar, three-capability framework that organizes recent AI world models by their transition from 2D visual prediction to 3D cognition.

Reference graph

Works this paper leans on

67 extracted references · 41 canonical work pages · cited by 2 Pith papers

  1. [1]

    Quantifying atten- tion flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying atten- tion flow in transformers. arXiv preprint arXiv:2005.00928,

    arXiv 2005

  2. [2]

    Se- mantickitti: A dataset for semantic scene understanding of lidar sequences

    Jens Behley, Martin Garbade, Andres Milioto, Jan Quen- zel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Se- mantickitti: A dataset for semantic scene understanding of lidar sequences. In ICCV, pages 9297–9307, 2019. 7

    work page 2019

  3. [3]

    Lan- gocc: Self-supervised open vocabulary occupancy estima- tion via volume rendering.arXiv preprint arXiv:2407.17310,

    Simon Boeder, Fabian Gigengack, and Benjamin Risse. Lan- gocc: Self-supervised open vocabulary occupancy estima- tion via volume rendering.arXiv preprint arXiv:2407.17310,

    arXiv

  4. [4]

    Oc- cflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow

    Simon Boeder, Fabian Gigengack, and Benjamin Risse. Oc- cflownet: Towards self-supervised occupancy estimation via differentiable rendering and occupancy flow. arXiv preprint arXiv:2402.12792, 2024. 2, 3

    arXiv 2024

  5. [5]

    Monoscene: Monoc- ular 3d semantic scene completion

    Anh-Quan Cao and Raoul De Charette. Monoscene: Monoc- ular 3d semantic scene completion. In CVPR, pages 3991– 4001, 2022. 3

    work page 2022

  6. [6]

    Pointbev: A sparse approach for bev predictions

    Loick Chambon, Eloi Zablocki, Mickaël Chen, Florent Bar- toccioni, Patrick Pérez, and Matthieu Cord. Pointbev: A sparse approach for bev predictions. InCVPR, pages 15195– 15204, 2024. 3

    work page 2024

  7. [7]

    Seal: Self- supervised embodied active learning using exploration and 3d consistency

    Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhutdinov. Seal: Self- supervised embodied active learning using exploration and 3d consistency. NIPS, 34:13086–13098, 2021. 1

    work page 2021

  8. [8]

    V oxelnext: Fully sparse voxelnet for 3d object detection and tracking

    Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. V oxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR, pages 21674–21683, 2023. 3

    work page 2023

Show all 67 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR,

  2. [10]

    Yolo-world: Real-time open- vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open- vocabulary object detection. In CVPR, 2024. 3

    2024

  3. [11]

    A point set generation network for 3d object reconstruction from a single image

    Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017. 5

    2017

  4. [12]

    Dy- namic 3d gaussian fields for urban areas

    Tobias Fischer, Jonas Kulhanek, Samuel Rota Bulo, Lorenzo Porzi, Marc Pollefeys, and Peter Kontschieder. Dy- namic 3d gaussian fields for urban areas. arXiv preprint arXiv:2406.03175, 2024. 3

    2024 arXiv

  5. [13]

    Gaussianocc: Fully self-supervised and effi- cient 3d occupancy estimation with gaussian splatting.arXiv e-prints, 2024

    Wanshui Gan, Fang Liu, Hongbin Xu, Ningkai Mo, and Naoto Yokoya. Gaussianocc: Fully self-supervised and effi- cient 3d occupancy estimation with gaussian splatting.arXiv e-prints, 2024. 2, 3, 6, 7, 8

    2024

  6. [14]

    A comprehensive framework for 3d occupancy estimation in autonomous driving

    Wanshui Gan, Ningkai Mo, Hongbin Xu, and Naoto Yokoya. A comprehensive framework for 3d occupancy estimation in autonomous driving. IEEE Transactions on Intelligent Vehi- cles, 2024. 2, 3

    2024

  7. [15]

    Simple-bev: What really matters for multi-sensor bev perception? In ICRA, pages 2759–2765

    Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. Simple-bev: What really matters for multi-sensor bev perception? In ICRA, pages 2759–2765. IEEE, 2023. 3

    2023

  8. [16]

    Tri-perspective view for vision-based 3d se- mantic occupancy prediction

    Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d se- mantic occupancy prediction. In CVPR, pages 9223–9232,

  9. [17]

    Selfocc: Self-supervised vision-based 3d oc- cupancy prediction

    Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. Selfocc: Self-supervised vision-based 3d oc- cupancy prediction. In CVPR, pages 19946–19956, 2024. 2, 3

    2024

  10. [18]

    Segment anything in high qual- ity

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity. NIPS, 36, 2024. 4

    2024

  11. [19]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  12. [20]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 2, 4

    2023

  13. [21]

    F-vlm: Open-vocabulary object detec- tion upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detec- tion upon frozen vision and language models. arXiv preprint arXiv:2209.15639, 2022. 3

    2022 arXiv

  14. [22]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, 2022. 3

    2022

  15. [23]

    Unifying voxel-based representation with transformer for 3d object detection

    Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. NIPS, 35:18442–18455,

  16. [24]

    V oxel field fusion for 3d object detection

    Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. V oxel field fusion for 3d object detection. In CVPR, pages 1120–1129, 2022. 3

    2022

  17. [25]

    Fast-bev: A fast and strong bird’s-eye view perception baseline

    Yangguang Li, Bin Huang, Zeren Chen, Yufeng Cui, Feng Liang, Mingzhu Shen, Fenggang Liu, Enze Xie, Lu Sheng, Wanli Ouyang, et al. Fast-bev: A fast and strong bird’s-eye view perception baseline. TPAMI, 2024. 3

    2024

  18. [26]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chong- hao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, pages 1–

  19. [27]

    Fb-occ: 3d occupancy prediction based on forward-backward view transformation

    Zhiqi Li, Zhiding Yu, David Austin, Mingsheng Fang, Shiyi Lan, Jan Kautz, and Jose M Alvarez. Fb-occ: 3d occupancy prediction based on forward-backward view transformation. arXiv preprint arXiv:2307.01492, 2023. 3

    2023 arXiv

  20. [28]

    Generative region-language pretraining for open-ended object detection

    Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In CVPR, 2024. 3

    2024

  21. [29]

    Training-free open-ended object detection and segmentation via attention as prompts

    Zhiwei Lin, Yongtao Wang, and Zhi Tang. Training-free open-ended object detection and segmentation via attention as prompts. arXiv preprint arXiv:2410.05963, 2024. 3, 4

    2024 arXiv

  22. [30]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 4

    2023 arXiv

  23. [31]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024. 3

    2024

  24. [32]

    Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data

    Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boular- ias, and Kostas Bekris. Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data. In Conference on Robot Learning, pages 1610–1620. PMLR, 2023. 2, 6

    2023

  25. [33]

    Bev- guided multi-modality fusion for driving perception

    Yunze Man, Liang-Yan Gui, and Yu-Xiong Wang. Bev- guided multi-modality fusion for driving perception. In CVPR, pages 21960–21969, 2023. 3

    2023

  26. [34]

    Uniocc: Unifying vision-centric 3d oc- cupancy prediction with geometric and semantic rendering

    Mingjie Pan, Li Liu, Jiaming Liu, Peixiang Huang, Lon- glong Wang, Shanghang Zhang, Shaoqing Xu, Zhiyi Lai, and Kuiyuan Yang. Uniocc: Unifying vision-centric 3d oc- cupancy prediction with geometric and semantic rendering. arXiv preprint arXiv:2306.09117, 2023. 3

    2023 arXiv

  27. [35]

    Renderocc: Vision-centric 3d occupancy predic- tion with 2d rendering supervision

    Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Hongwei Xie, Bing Wang, Li Liu, and Shanghang Zhang. Renderocc: Vision-centric 3d occupancy predic- tion with 2d rendering supervision. In ICRA, pages 12404– 12411. IEEE, 2024. 3

    2024

  28. [36]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR,

  29. [37]

    Langsplat: 3d language gaussian splatting

    Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In CVPR, pages 20051–20060, 2024. 3

    2024

  30. [38]

    Occupancy anticipation for efficient exploration and navigation

    Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient exploration and navigation. In ECCV, pages 400–418. Springer, 2020. 1

    2020

  31. [39]

    An exploration of embodied visual exploration

    Santhosh K Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. An exploration of embodied visual exploration. IJCV, 129(5):1616–1649, 2021. 1

    2021

  32. [40]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 3

    2024 arXiv

  33. [41]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  34. [42]

    Toward open set recogni- tion

    Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recogni- tion. TPAMI, pages 1757–1772, 2012. 3

    2012

  35. [43]

    Occupancy as set of points

    Yiang Shi, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Xinggang Wang. Occupancy as set of points. arXiv preprint arXiv:2407.04049, 2024. 3

    2024 arXiv

  36. [44]

    Ovo: Open-vocabulary occupancy

    Zhiyu Tan, Zichao Dong, Cheng Zhang, Weikun Zhang, Hang Ji, and Hao Li. Ovo: Open-vocabulary occupancy. arXiv preprint arXiv:2305.16133, 2023. 7, 8

    2023 arXiv

  37. [45]

    Driv- ingforward: Feed-forward 3d gaussian splatting for driv- ing scene reconstruction from flexible surround-view input

    Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. Driv- ingforward: Feed-forward 3d gaussian splatting for driv- ing scene reconstruction from flexible surround-view input. arXiv preprint arXiv:2409.12753, 2024. 3

    2024 arXiv

  38. [46]

    Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving

    Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. NIPS, 36, 2024. 2, 6, 7

    2024

  39. [47]

    Scene as occupancy

    Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In ICCV, pages 8406–8415, 2023. 1, 2

    2023

  40. [48]

    Learning-based 3d occupancy prediction for autonomous navigation in occluded environments

    Lizi Wang, Hongkai Ye, Qianhao Wang, Yuman Gao, Chao Xu, and Fei Gao. Learning-based 3d occupancy prediction for autonomous navigation in occluded environments. In IROS, pages 4509–4516. IEEE, 2021. 1

    2021

  41. [49]

    Openoccupancy: A large scale benchmark for sur- rounding semantic occupancy perception

    Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for sur- rounding semantic occupancy perception. In ICCV, pages 17850–17859, 2023. 2, 6, 8

    2023

  42. [50]

    Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation

    Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaox- iang Zhang. Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation. In CVPR, pages 17158–17168, 2024. 1

    2024

  43. [51]

    Surroundocc: Multi-camera 3d occu- pancy prediction for autonomous driving

    Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundocc: Multi-camera 3d occu- pancy prediction for autonomous driving. In ICCV, pages 21729–21740, 2023. 2, 6, 7, 8

    2023

  44. [52]

    Towards open vocabulary learning: A survey

    Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey. TPAMI, 2024. 3

    2024

  45. [53]

    Vlm: Task-agnostic video- language model pre-training for video understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video- language model pre-training for video understanding. arXiv preprint arXiv:2105.09996, 2021. 3

    2021 arXiv

  46. [54]

    Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation

    Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707, 2023. 6

    2023 arXiv

  47. [55]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 4

    2024

  48. [56]

    Flashocc: Fast and memory-efficient occupancy prediction via channel-to-height plugin

    Zichen Yu, Changyong Shu, Jiajun Deng, Kangjie Lu, Zong- dai Liu, Jiangyong Yu, Dawei Yang, Hui Li, and Yan Chen. Flashocc: Fast and memory-efficient occupancy prediction via channel-to-height plugin. arXiv preprint arXiv:2311.12058, 2023. 3

    2023 arXiv

  49. [57]

    Mip-splatting: Alias-free 3d gaussian splat- ting

    Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splat- ting. In CVPR, pages 19447–19456, 2024. 3

    2024

  50. [58]

    Faster segment anything: Towards lightweight sam for mo- bile applications

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications. arXiv preprint arXiv:2306.14289 , 2023. 4

    2023 arXiv

  51. [59]

    Occnerf: Self- supervised multi-camera occupancy prediction with neural radiance fields

    Chubin Zhang, Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen Lu. Occnerf: Self- supervised multi-camera occupancy prediction with neural radiance fields. arXiv e-prints, pages arXiv–2312, 2023. 2, 3, 6

    2023

  52. [60]

    Sam3d: Zero-shot 3d object detection via segment anything model

    Dingyuan Zhang, Dingkang Liang, Hongcheng Yang, Zhikang Zou, Xiaoqing Ye, Zhe Liu, and Xiang Bai. Sam3d: Zero-shot 3d object detection via segment anything model. arXiv preprint arXiv:2306.02245, 2023. 2

    2023 arXiv

  53. [61]

    A simple framework for open-vocabulary segmentation and detection

    Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In ICCV, pages 1020–1031, 2023. 2

    2023

  54. [62]

    Occformer: Dual-path transformer for vision-based 3d semantic occu- pancy prediction

    Yunpeng Zhang, Zheng Zhu, and Dalong Du. Occformer: Dual-path transformer for vision-based 3d semantic occu- pancy prediction. In ICCV, pages 9433–9443, 2023. 2, 3

    2023

  55. [63]

    Lif-seg: Lidar and camera image fusion for 3d lidar semantic segmentation

    Lin Zhao, Hui Zhou, Xinge Zhu, Xiao Song, Hongsheng Li, and Wenbing Tao. Lif-seg: Lidar and camera image fusion for 3d lidar semantic segmentation. IEEE Transactions on Multimedia, 26:1158–1168, 2023. 5

    2023

  56. [64]

    Fusion of 3d lidar and camera data for object detection in autonomous vehicle applications

    Xiangmo Zhao, Pengpeng Sun, Zhigang Xu, Haigen Min, and Hongkai Yu. Fusion of 3d lidar and camera data for object detection in autonomous vehicle applications. IEEE Sensors Journal, 20(9):4901–4913, 2020. 5

    2020

  57. [65]

    Fast segment any- thing

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing. arXiv preprint arXiv:2306.12156, 2023. 4

    2023 arXiv

  58. [66]

    Veon: V ocabulary- enhanced occupancy prediction

    Jilai Zheng, Pin Tang, Zhongdao Wang, Guoqing Wang, Xi- angxuan Ren, Bailan Feng, and Chao Ma. Veon: V ocabulary- enhanced occupancy prediction. In ECCV, pages 92–108. Springer, 2025. 6, 7

    2025

  59. [67]

    Drivinggaussian: Composite gaussian splatting for surrounding dynamic au- tonomous driving scenes

    Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic au- tonomous driving scenes. In CVPR, pages 21634–21643,

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.