Pith. sign in

REVIEW 5 major objections 8 minor 64 references

SAM4D: Segment Anything in Camera and LiDAR Streams

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAM4D claims the first promptable model that segments objects jointly in camera and LiDAR streams, so a point or box in either sensor yields masks in both, propagated over time.

desk verdict SAM4D is a credible engineering contribution with a genuinely new task formulation, but its central evaluation is weakened by training and testing on the same VFM-generated pseudo-labels, with no human-verified 3D ground truth. read the letter →

arxiv 2506.21547 v1 pith:AAPGKNC2 submitted 2025-06-26 cs.CV cs.RO

Jianyun Xu , Song Wang , Ziqian Ni , Chunyong Hu , Sheng Yang , Jianke Zhu , Qiang Li This is my paper
classification cs.CVcs.RO
keywords promptablesegmentationmulti-modalfusioncamera-LiDARtemporalconsistencyautonomousdrivingpseudo-labeldataengineWaymo-4DSegmemoryattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that promptable segmentation—the click-or-box-to-mask interaction made familiar in image models—can be extended to operate simultaneously on camera and LiDAR streams in one model. SAM4D accepts a point, box, or mask prompt from either sensor and returns a temporally consistent object masklet in both modalities, which defines the new Promptable Multi-modal Segmentation (PMS) task. The practical motivation is annotation efficiency for autonomous driving: one prompt in a camera frame could label the same object in LiDAR and in all subsequent frames, turning manual multi-modal labeling into a much cheaper interactive process. To avoid a human-annotation bottleneck, the paper builds an automated data engine that turns vision-foundation-model video masklets into camera-LiDAR aligned pseudo-labels, producing the Waymo-4DSeg dataset with over 300k cross-modal masklets. Experiments on Waymo-4DSeg and an unseen nuScenes set are used to argue that cross-modal prompting, temporal propagation, and zero-shot generalization all work.

What carries the argument

The two load-bearing components are UMPE and MCMA, plus the data engine that supplies training signal. UMPE gives every image patch and every LiDAR voxel a position code in a shared 3D space: image pixels are unprojected into pseudo-point clouds using estimated depth and the camera-to-LiDAR extrinsic transform, then passed through an MLP, while LiDAR voxels pass their coordinates through the same MLP, so cross-modal attention can compare positionally aligned tokens. MCMA runs self-attention within each modality, cross-attention between modalities, and temporal attention against a memory bank whose stored positions are transformed by ego-motion before retrieval, allowing long-horizon feature reuse under large vehicle motion. The mask decoder consumes sparse or mask prompts from either modality and outputs both 2D and 3D masks, and training simulates interactive prompting with corrective clicks. The data engine closes the loop by generating Waymo-4DSeg pseudo-labels: Grounding-DINO and SAM2 seed video masklets, 4D reconstruction and ray casting map pixels to voxels, and DBSCAN clustering plus overlap merging fuses and denoises the camera-LiDAR masklets.

What would settle it

Re-annotate a random sample of Waymo-4DSeg objects with human instance masks on both camera frames and LiDAR scans, then recompute SAM4D's mIoU against those human masks rather than the engine's pseudo-labels; if the numbers drop substantially or the ranking against the single-modality baselines changes, the pseudo-label ground truth is carrying the reported result.

Watch

Extended reading notes

Core claim

SAM4D's central claim is that image and LiDAR segmentation need not be separate tasks: with a shared 3D positional encoding, prompts can cross modalities, and with an ego-motion-compensated memory bank, objects can stay consistent over long sequences. The architecture pairs Unified Multi-modal Positional Encoding (UMPE), which lifts image features into a shared 3D space via estimated depth and camera-to-LiDAR transforms while encoding LiDAR voxels with the same MLP-based positional code, with Motion-aware Cross-modal Memory Attention (MCMA), which transforms past-frame features and object pointers into the current coordinate frame before cross-modal and temporal attention. Trained on pseudo-labels from the automated data engine, the model reports that an image prompt yields usable LiDAR masks and a LiDAR prompt yields usable image masks, that first-frame prompts propagate into stream-level masklets, and that zero-shot transfer to nuScenes is strong and improves with fine-tuning. The data engine itself is part of the contribution: it combines VFM-driven video masklets, 4D voxel reconstruction with ray casting, and DBSCAN-based fusion to generate camera-LiDAR aligned pseudo-labels, with an average cross-modal IoU of 0.56 used as a label-quality score.

Load-bearing premise

The load-bearing premise is that the pseudo-labels produced by the automated data engine are accurate enough to serve as both training targets and evaluation ground truth; the paper reports an average cross-modal IoU of 0.56 and filters evaluation to objects with score above 0.5, volume above 50, and away from the ground.

Editorial extensions

If this is right

  • Annotators could label both modalities from a single interaction: a point or box in one sensor returns masks in the other, reducing per-object labeling cost.
  • A first-frame prompt propagates into a temporally consistent masklet over the whole sequence, which is the behavior needed for semi-automatic video-and-LiDAR annotation.
  • The automated data engine can generate large volumes of camera-LiDAR aligned pseudo-labels orders of magnitude faster than human annotation, making scale-up of 4D training data feasible.
  • Ego-motion-compensated memory attention reduces object mismatches over long sequences, making the model more reliable for tracking and for labeling dynamic scenes.
  • Zero-shot transfer to nuScenes suggests the learned cross-modal alignment is not overfit to Waymo, and fine-tuning gives a further gain on new sensor configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its evaluation numbers are upper bounds on objects the data engine already scores well; a human-verified subset or a test on low-score objects would show whether SAM4D's cross-modal prompting also helps where pseudo-labels are noisier.
  • Because the same vision foundation models generate the training pseudo-labels, SAM4D is likely to inherit their failure modes, for example object misassociation between similar nearby instances; a testable check is to prompt SAM4D on cases where the engine's masklets are known to confuse two objects.
  • The shared 3D positional encoding is in principle sensor-agnostic, so the same architecture could accept other depth-bearing sensors or depth-from-mono inputs, suggesting a path toward multi-camera and multi-LiDAR setups that the paper lists only as future work.
  • A natural next step the paper only gestures at is natural-language prompting: text embeddings placed in the same unified 3D code could turn SAM4D into an open-vocabulary 4D segmenter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces SAM4D, described as the first promptable segmentation model that operates jointly on camera and LiDAR streams, unifying multi-modal and temporal segmentation in a single framework. The model builds on SAM2, adding a LiDAR encoder, a Unified Multi-modal Positional Encoding (UMPE) that lifts image features into a shared 3D space, and a Motion-aware Cross-modal Memory Attention (MCMA) module that uses ego-motion compensation for temporal alignment. To support training, the authors construct Waymo-4DSeg, a large pseudo-labeled dataset generated by an automated data engine that combines GroundingDINO/SAM2-derived video masklets, 4D voxel reconstruction, ray casting, and cross-modal fusion. Experiments on Waymo-4DSeg and a nuScenes generalization study report strong cross-modal segmentation performance under various prompting settings.

Significance. If the reported results hold up, the paper would make a meaningful contribution by defining a new task (Promptable Multi-modal Segmentation), offering a concrete architecture that addresses cross-modal prompting and temporal consistency, and providing a large-scale pseudo-labeled dataset for autonomous driving research. The paper is also transparent in some respects: it reports the pseudo-label cross-modal IoU of 0.56, describes the evaluation filters in Sec. C.1, and acknowledges limitations of the pseudo-label strategy in Sec. D.1. These strengths are undermined, however, by the self-referential evaluation protocol: the model is trained and tested against pseudo-labels produced by the same vision-foundation-model pipeline that initializes its image encoder, and no human-verified ground-truth check is provided. The lack of comparisons to strong existing baselines such as SAL or PointSAM and the absence of error bars further reduce the confidence in the quantitative claims.

major comments (5)
  1. [Sec. 5.1 and Sec. C.1] The central evaluation is circular. The data engine in Sec. 5.1 uses GroundingDINO and SAM2 to generate video masklets, then projects and fuses them into LiDAR voxel masklets; the reported quality score is the cross-modal IoU between these two sets of engine-generated masks, not an external ground-truth measurement (mean 0.56, 10th percentile 0.24 per Sec. B.1). SAM4D is trained and evaluated against these same pseudo-labels, and its image encoder is initialized from SA-V, i.e., the SAM2 model family that produced the labels. Sec. C.1 further filters the evaluation to objects with score > 0.5 and volume > 50 and excludes ground regions, removing exactly the cases where the engine is least reliable. Consequently, the numbers in Tables 2 and 4 measure how well SAM4D reproduces the data engine's outputs, not true segmentation accuracy. A human-verified evaluation subset, or an existing human-annotated dataset, is needed to support the claim of powerful cross-modal segmentation.
  2. [Sec. 6.2 and Related Work (Sec. 2)] The paper claims to be the first unified promptable camera-LiDAR segmentation model, and cites SAL [31] and PointSAM [64] as related promptable 3D segmentation approaches. However, no experiments compare against SAL, PointSAM, or other VFM-based 2D-to-3D segmentation methods. The only baseline, SAM2+Project (Table 4), is a simple projection of SAM2 video masks onto per-frame point clouds. Without strong baselines, the claimed advantage over existing modality-specific methods is not established. Please add quantitative comparisons to SAL, PointSAM, and at least one additional projection-based or fusion-based baseline under the same evaluation protocol.
  3. [Sec. 6.3, Table 6] The ego-motion ablation does not convincingly support the claim that MCMA's ego-motion compensation is a major contribution. Table 6 shows that removing ego-motion changes image mIoU from 69.8 to 69.7 and J&F from 80.1 to 80.3 (slightly worse with ego-motion), while LiDAR mIoU changes from 55.7 to 52.2 and NMP from 582 to 746. The text states NMP decreases from 746 to 592, but Table 6 reports 582. More importantly, the near-zero image-domain differences and the absence of error bars or multiple seeds make it impossible to judge whether the LiDAR improvements are statistically significant. Report variance across runs, or temper the claim that ego-motion 'significantly' improves temporal consistency.
  4. [Sec. 6.2, Table 3] The nuScenes generalization results are weak and are overinterpreted. Zero-shot LiDAR mIoU is 25.9, which is only 18.9 points below the fine-tuned value of 44.8, and in absolute terms 25.9 mIoU is a low number for a claimed 'strong cross-modal generalization'. The evaluation relies on nuInsSeg, which provides 2D instance masks for foreground objects; it is not described how the LiDAR ground truth is obtained for nuScenes. Clarify the LiDAR label derivation and discuss these numbers honestly, including the large modality gap between image (58.4 mIoU) and LiDAR (25.9 mIoU) in the zero-shot setting.
  5. [Sec. 6.1 and all result tables] No error bars, confidence intervals, or multiple-seed averages are reported anywhere. The evaluation uses 48 randomly selected clips (Sec. 6.1), but the reader cannot assess the variability of the mIoU, J&F, and NMP metrics. This is especially problematic for ablations like Table 6, where image mIoU differs by only 0.1. Report means and standard deviations over at least three random seeds, or at minimum over several clip subsamples, for all main tables.
minor comments (8)
  1. [Sec. 6.3, Table 6] The sentence reading 'NMP decreases from 746 to 592' contradicts the table entry of 582; the table shows 582, so the text should be corrected to 582.
  2. [Sec. 1] There is a typo 'promotable' in the final paragraph of the Introduction: the word should be 'promptable' to match the rest of the paper.
  3. [Sec. 4.3, Eq. (2)] The intrinsic matrix K is described as a 4x4 matrix, but conventional camera intrinsics are 3x3; if a homogeneous 4x4 form is intended, please define its structure explicitly to avoid ambiguity.
  4. [Sec. 5.1] The main text reports only the average cross-modal IoU of 0.56; including the 10th percentile of 0.24 (given in Sec. B.1) would give readers a more complete picture of label quality and should be mentioned here as well.
  5. [Sec. C.1] The phrase 'temporarily exclude instances near the ground' is vague: it is unclear whether the final model is also evaluated without ground-region instances and whether this restriction remains in any deployed setting.
  6. [Sec. 6.1 and Table A1] The main text says SAM4D is trained for 36 epochs, while Table A1 reports 'steps ~44k'; clarify the correspondence between epochs and steps.
  7. [Sec. A.2] In the sentence 'we sample 8-frame sequences', the LiDAR voxel size is given as '0.15' without units; it should read '0.15 m'.
  8. [Sec. 4.3] The typo 'incluing' should be 'including' in the sentence 'For sparse prompts incluing points or bounding boxes'.

Circularity Check

2 steps flagged · score 6.0 of 10

Waymo-4DSeg main evaluation is self-referential: training targets and test 'ground truth' are both generated by the same VFM data engine, and the test-time quality filter is the engine's self-consistency score.

  1. fitted input called prediction [Sec. 5.1 (automated data engine); Sec. C.1 (training and validation data)]
    "During SAM4D training, targets with a volume greater than 10 and a score greater than 0.3 are used. For testing, to further ensure the reliability of the ground truth, the volume filtering threshold is increased to 50, and the score threshold is raised to 0.5. Furthermore, there is currently significant ambiguity in the pseudo-labels for ground regions. To ensure better convergence of the LiDAR branch, we temporarily exclude instances near the ground during both training and evaluation."

    The Waymo-4DSeg 'ground truth' used for all main mIoU results is the output of the paper's own data engine: Grounding-DINO and SAM/SAM2 produce image masklets, ray casting and 4D reconstruction transfer them to LiDAR voxels, and DBSCAN filters noise. SAM4D is trained with these same pseudo-labels as supervision, then evaluated against a stricter filtered slice of the same label source (volume>50, score>0.5, ground regions removed). The reported image and LiDAR mIoU therefore measure how well SAM4D reproduces the data engine's labels on the subset where the engine is most self-consistent, not agreement with independent 3D ground truth.

  2. self definitional [Sec. B.1 (Supplementary), cross-modal IoU quality score; Sec. C.1 uses score as ground-truth filter]
    "Assuming that a masklet is visible for image i, we calculated the IoU between the voxels mapped by masklet in image i and the visible part of the unified voxel masklet. The average IoU across all images represents one masklet's overall score. The mean score of the masklets in our dataset is 0.56, with a 10th percentile of 0.24."

    The 'unified voxel masklet' is not an independent 3D annotation: it is the merged product of the same per-image video masklets after ray casting and DBSCAN fusion. A masklet's quality score is therefore the agreement of each image-derived mask with the engine's own fused version of those masks, i.e., a self-consistency statistic rather than validation against human or externally verified labels. Section C.1 then uses exactly this score as the criterion for 'reliable ground truth' at evaluation (score>0.5), so the benchmark's quality filter is defined circularly with respect to the label-generation pipeline. This strengthens the first step: the reported Waymo-4DSeg numbers are measured on the labels the engine considers most self-consistent, not on independently verified objects.

full rationale

The model architecture itself — UMPE, MCMA, and the promptable multi-modal segmentation formulation — is an independent engineering contribution, and there is no imported uniqueness theorem or load-bearing self-citation chain. The circularity is concentrated in the Waymo-4DSeg evaluation. The data engine that generates the training pseudo-labels also generates the test 'ground truth,' and the score threshold used to select reliable test objects is the data engine's own cross-modal self-consistency score (mean 0.56, 10th percentile 0.24). Thus the central experimental demonstration of cross-modal segmentation ability is partially a fit to the same VFM-driven label source from which the model's image encoder is also initialized. The nuScenes experiment provides some external grounding, but its zero-shot LiDAR mIoU is only 25.9 and it is not the basis for the paper's main claims. Because the central Waymo evaluation reduces to reproducing the data engine's labels on a filtered, self-consistent subset, a partial circularity score of 6 is appropriate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the accuracy of the automated pseudo-labeling pipeline and the unstated depth source for image lifting. These are domain assumptions that should be validated against human-annotated data.

free parameters (3)
  • Pseudo-label quality thresholds (train/test) = train: volume > 10, score > 0.3; test: volume > 50, score > 0.5
    Hand-chosen filters that exclude noisy pseudo-labels from training and evaluation; directly affect reported mIoU.
  • Training modality presence probabilities = 0.5 both, 0.25 camera-only, 0.25 LiDAR-only
    Sampling probabilities for targets during training to mimic interaction logic; affects cross-modal learning.
  • Memory bank sizes N and M = Not specified in paper
    Number of unprompted (N) and prompted (M) frames retained in memory; key temporal hyperparameters not reported.
assumptions (3)
  • domain assumption Pseudo-labels from the automated data engine are a sufficiently accurate proxy for human ground truth.
    The dataset is generated by VFM-driven masklets and 4D reconstruction without large-scale human verification; average cross-modal IoU is only 0.56. This assumption underlies both training and evaluation in Sec. 6.
  • domain assumption Image depth estimates D(u,v) used to lift image features into 3D are accurate.
    Eq. (2) in Sec. 4.3 assumes known per-pixel depth for the UMPE; the paper does not specify how D is obtained (e.g., LiDAR projection or learned depth), so the alignment quality depends on this unstated input.
  • domain assumption Ego-motion transformations Tt<-t' from vehicle odometry are exact.
    Eq. (8) relies on SE(3) transforms for memory alignment; odometry noise would corrupt temporal feature retrieval.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAM4D: Segment Anything in Camera and LiDAR Streams." pith.science (2026). https://pith.science/paper/AAPGKNC2

@misc{pith2026250621547,
  author       = {Pith},
  title        = {Pith review of: SAM4D: Segment Anything in Camera and LiDAR Streams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AAPGKNC2}},
  note         = {Machine review of arXiv:2506.21547}
}
read the original abstract

We present SAM4D, a multi-modal and temporal foundation model designed for promptable segmentation across camera and LiDAR streams. Unified Multi-modal Positional Encoding (UMPE) is introduced to align camera and LiDAR features in a shared 3D space, enabling seamless cross-modal prompting and interaction. Additionally, we propose Motion-aware Cross-modal Memory Attention (MCMA), which leverages ego-motion compensation to enhance temporal consistency and long-horizon feature retrieval, ensuring robust segmentation across dynamically changing autonomous driving scenes. To avoid annotation bottlenecks, we develop a multi-modal automated data engine that synergizes VFM-driven video masklets, spatiotemporal 4D reconstruction, and cross-modal masklet fusion. This framework generates camera-LiDAR aligned pseudo-labels at a speed orders of magnitude faster than human annotation while preserving VFM-derived semantic fidelity in point cloud representations. We conduct extensive experiments on the constructed Waymo-4DSeg, which demonstrate the powerful cross-modal segmentation ability and great potential in data annotation of proposed SAM4D.

Figures

Figures reproduced from arXiv: 2506.21547 by the authors.

Figure 1
Figure 1. We aim to build a foundation model for 4D segmentation by introducing three interconnected components: (a) a promptable [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Segment Anything Model in 4D (SAM4D) workflow. The image and LiDAR encoders generate modality￾specific embeddings, which are aligned through the proposed Unified Multi-modal Positional Encoding. The Motion-aware Cross￾modal Memory Attention then processes multi-modal and temporal features, incorporating ego-motion for improved feature interaction. Finally, the updated image and LiDAR features are que… view at source ↗
Figure 3
Figure 3. Illustration of the proposed Unified Multi-modal Posi￾tional Encoding. modal Positional Encoding (UMPE) scheme. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of the proposed Motion-aware Cross￾modal Memory Attention. is applied: PLiDAR sin(x, y, z) = SinPE3D(x, y, z) (4) which encodes the spatial structure of the point cloud. To ensure consistency with image features lifted into 3D, we utilize the same MLP-base…
Figure 5
Figure 5. Figure 5: Overview of our data engine, which is composed of three steps to construct high-quality pseudo labels. cross-attention to update current features with aligned tem￾poral information: F final img = CrossAttn(F ′′ img,(Mt←t ′ img , O t ′ img)), F final LiDAR = CrossAttn(F…
Figure 6
Figure 6. Figure 6: Performance comparison with different promptable frames in interactive offline and online evaluation settings. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 50 canonical work pages

  1. [31]

    Better call sal: Towards learning to segment anything in lidar

    Aljo ˇsa O ˇsep, Tim Meinhardt, Francesco Ferroni, Neehar Peri, Deva Ramanan, and Laura Leal-Taix ´e. Better call sal: Towards learning to segment anything in lidar. In European Conference on Computer Vision, pages 71–90, 2024. 2

    work page 2024

  2. [64]

    Point-SAM: Promptable 3d segmentation model for point clouds

    Yuchen Zhou, Jiayuan Gu, Tung Yen Chiang, Fanbo Xiang, and Hao Su. Point-SAM: Promptable 3d segmentation model for point clouds. In International Conference on Learning Representations, 2025. 2 11 SAM4D: Segment Anything in Camera and LiDAR Streams Supplementary Material In this document, we further provide the following ma- terials to support the finding...

    work page 2025

  3. [1]

    Lan- gocc: Self-supervised open vocabulary occupancy estima- tion via volume rendering.arXiv preprint arXiv:2407.17310,

    Simon Boeder, Fabian Gigengack, and Benjamin Risse. Lan- gocc: Self-supervised open vocabulary occupancy estima- tion via volume rendering.arXiv preprint arXiv:2407.17310,

    arXiv

  4. [2]

    Window Attention is Bugged: How not to Interpolate Position Embeddings

    Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and Christoph Feichtenhofer. Window attention is bugged: how not to interpolate position embeddings. arXiv preprint arXiv:2311.05613, 2023. 4

    work page Pith review arXiv 2023

  5. [3]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11621–11631, 2020. 8, 16

    work page 2020

  6. [4]

    Mopa: Multi-modal prior aided do- main adaptation for 3d semantic segmentation

    Haozhi Cao, Yuecong Xu, Jianfei Yang, Pengyu Yin, Sheng- hai Yuan, and Lihua Xie. Mopa: Multi-modal prior aided do- main adaptation for 3d semantic segmentation. In IEEE In- ternational Conference on Robotics and Automation , pages 9463–9470, 2024. 2

    work page 2024

  7. [5]

    Rsprompter: Learning to prompt for remote sensing instance segmenta- tion based on visual foundation model

    Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmenta- tion based on visual foundation model. IEEE Transactions on Geoscience and Remote Sensing, 62:1–17, 2024. 1, 2

    work page 2024

  8. [6]

    Clip2scene: Towards label-efficient 3d scene under- standing by clip

    Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene under- standing by clip. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 7020–7030, 2023. 3

    work page 2023

Show all 64 references
  1. [7]

    Sam2-adapter: Evaluating & adapting seg- ment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more

    Tianrun Chen, Ankang Lu, Lanyun Zhu, Chaotao Ding, Chu- nan Yu, Deyi Ji, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam2-adapter: Evaluating & adapting seg- ment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more. arXiv preprint arXi...

    2024 arXiv

  2. [8]

    Futr3d: A unified sensor fusion framework for 3d detection

    Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 172–181, 2023. 2

    2023

  3. [9]

    Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion

    Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 5559–5568, 2021. 7

    2021

  4. [10]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019. 4, 7, 12

    2019

  5. [11]

    Benchmarking robustness of 3d object detection to common corruptions

    Yinpeng Dong, Caixin Kang, Jinlai Zhang, Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, and Jun Zhu. Benchmarking robustness of 3d object detection to common corruptions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1022–1032, 2023. 1

    2023

  6. [12]

    Interactive4d: Interactive 4d lidar segmentation

    Ilya Fradlin, Idil Esen Zulfikar, Kadir Yilmaz, Theodora Kontogianni, and Bastian Leibe. Interactive4d: Interactive 4d lidar segmentation. arXiv preprint arXiv:2410.08206 ,

    arXiv

  7. [13]

    Scale dispar- ity of instances in interactive point cloud segmentation

    Chenrui Han, Xuan Yu, Yuxuan Xie, Yili Liu, Sitong Mao, Shunbo Zhou, Rong Xiong, and Yue Wang. Scale dispar- ity of instances in interactive point cloud segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2660–2667, 2024. 2

    2024

  8. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770–778, 2016. 12

    2016

  9. [15]

    Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels

    Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engel- mann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. In European Confer- ence on Computer Vision, pages 278–295, 2024. 2

    2024

  10. [16]

    Segment anything in high qual- ity

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity. Advances in Neural Information Processing Systems, 36: 29914–29934, 2023. 2 9

    2023

  11. [17]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 1, 2, 3, 5, 6

    2023

  12. [18]

    Mseg3d: Multi-modal 3d semantic segmentation for autonomous driv- ing

    Jiale Li, Hang Dai, Hao Han, and Yong Ding. Mseg3d: Multi-modal 3d semantic segmentation for autonomous driv- ing. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 21694–21704, 2023. 2

    2023

  13. [19]

    Pmafusion: Projection-based multi-modal alignment for 3d semantic oc- cupancy prediction

    Shiyao Li, Wenming Yang, and Qingmin Liao. Pmafusion: Projection-based multi-modal alignment for 3d semantic oc- cupancy prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3627–3634, 2024. 2

    2024

  14. [20]

    Lwsis: Lidar-guided weakly super- vised instance segmentation for autonomous driving

    Xiang Li, Junbo Yin, Botian Shi, Yikang Li, Ruigang Yang, and Jianbing Shen. Lwsis: Lidar-guided weakly super- vised instance segmentation for autonomous driving. In AAAI Conference on Artificial Intelligence , pages 1433– 1441, 2023. 8, 16

    2023

  15. [21]

    Unifying voxel-based representation with transformer for 3d object detection

    Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. Advances in Neural In- formation Processing Systems, 35:18442–18455, 2022. 2

    2022

  16. [22]

    Bevfusion: A simple and robust lidar-camera fusion framework

    Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Sys- tems, 35:10421–10434, 2022. 2

    2022

  17. [23]

    Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding

    Guibiao Liao, Jiankun Li, and Xiaoqing Ye. Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding. In AAAI Conference on Artificial Intelligence , pages 3351– 3359, 2024. 3

    2024

  18. [24]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Eu- ropean Conference on Computer Vision, pages 38–55, 2024. 6, 13

    2024

  19. [25]

    Segment any point cloud sequences by distilling vision foundation models

    Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wen- wei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. Advances in Neural Information Processing Sys- tems, 36:37193–37229, 2023. 2

    2023

  20. [26]

    Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation

    Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation. In IEEE International Conference on Robotics and Automation, pages 2774–2781, 2023. 1, 2

    2023

  21. [27]

    See more and know more: Zero- shot point cloud segmentation via multi-modal visual data

    Yuhang Lu, Qi Jiang, Runnan Chen, Yuenan Hou, Xinge Zhu, and Yuexin Ma. See more and know more: Zero- shot point cloud segmentation via multi-modal visual data. In IEEE/CVF International Conference on Computer Vision, pages 21674–21684, 2023. 3

    2023

  22. [28]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 1, 2

    2024

  23. [29]

    Segment anything model for medical image analysis: an experimental study

    Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 89:102918, 2023. 1, 2

    2023

  24. [30]

    Robust 3d semantic segmentation based on multi-phase multi-modal fusion for intelligent vehicles

    Peizhou Ni, Xu Li, Wang Xu, Dong Kong, Yue Hu, and Kun Wei. Robust 3d semantic segmentation based on multi-phase multi-modal fusion for intelligent vehicles. IEEE Transac- tions on Intelligent Vehicles, 9(1):1602–1614, 2023. 2

    2023

  25. [32]

    Co-occ: Coupling explicit feature fusion with volume rendering regularization for multi-modal 3d semantic occupancy prediction

    Jingyi Pan, Zipeng Wang, and Lin Wang. Co-occ: Coupling explicit feature fusion with volume rendering regularization for multi-modal 3d semantic occupancy prediction. IEEE Robotics and Automation Letters, 2024. 2

    2024

  26. [33]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 815–824, 2023. 3

    2023

  27. [34]

    Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d

    Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision, pages 194–210, 2020. 4

    2020

  28. [35]

    The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017

    Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Ar- bel´aez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation.arXiv preprint arXiv:1704.00675, 2017. 7

    2017 arXiv

  29. [36]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  30. [37]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 1, 2, 3, 4, 5, 6, 7, 12

    2024 arXiv

  31. [38]

    Grounding dino 1.5: Advance the” edge” of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wen- long Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the” edge” of open-set object detection. arXiv preprint arXiv:2405.10300, 2024. 6

    2024 arXiv

  32. [39]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  33. [40]

    Hi- era: A hierarchical vision transformer without the bells-and- whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hi- era: A hierarchical vision transformer without the bells-and- whistles. In International Conference on Machine Learn...

    2023

  34. [41]

    Mm-tta: multi-modal test-time adaptation for 3d 10 semantic segmentation

    Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk- Jin Yoon. Mm-tta: multi-modal test-time adaptation for 3d 10 semantic segmentation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 16928–16937,

  35. [42]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InIEEE/CVF Conference on Computer Vision and Pattern Rec...

    2020

  36. [43]

    Ovo: Open-vocabulary occupancy

    Zhiyu Tan, Zichao Dong, Cheng Zhang, Weikun Zhang, Hang Ji, and Hao Li. Ovo: Open-vocabulary occupancy. arXiv preprint arXiv:2305.16133, 2023. 3

    2023 arXiv

  37. [44]

    TorchSparse: Efficient Point Cloud Inference Engine

    Haotian Tang, Zhijian Liu, Xiuyu Li, Yujun Lin, and Song Han. TorchSparse: Efficient Point Cloud Inference Engine. In Conference on Machine Learning and Systems , 2022. 4, 7, 12

    2022

  38. [45]

    TorchSparse++: Efficient Point Cloud Engine

    Haotian Tang, Shang Yang, Zhijian Liu, Ke Hong, Zhong- ming Yu, Xiuyu Li, Guohao Dai, Yu Wang, and Song Han. TorchSparse++: Efficient Point Cloud Engine. In Computer Vision and Pattern Recognition Workshops, 2023. 4, 12

    2023

  39. [46]

    Can sam segment any- thing? when sam meets camouflaged object detection

    Lv Tang, Haoke Xiao, and Bo Li. Can sam segment any- thing? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709, 2023. 1, 2

    2023 arXiv

  40. [47]

    Vdbfusion: Flexible and efficient tsdf integration of range sensor data

    Ignacio Vizzo, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. Vdbfusion: Flexible and efficient tsdf integration of range sensor data. Sensors, 22(3):1296, 2022. 13

    2022

  41. [48]

    Pop-3d: Open-vocabulary 3d occupancy prediction from im- ages

    Antonin V obecky, Oriane Sim ´eoni, David Hurych, Spyri- don Gidaris, Andrei Bursuc, Patrick P ´erez, and Josef Sivic. Pop-3d: Open-vocabulary 3d occupancy prediction from im- ages. Advances in Neural Information Processing Systems , 36, 2024. 3

    2024

  42. [49]

    Occgen: Gener- ative multi-modal 3d occupancy prediction for autonomous driving

    Guoqing Wang, Zhongdao Wang, Pin Tang, Jilai Zheng, Xi- angxuan Ren, Bailan Feng, and Chao Ma. Occgen: Gener- ative multi-modal 3d occupancy prediction for autonomous driving. In European Conference on Computer Vision, pages 95–112, 2024. 2

    2024

  43. [50]

    Meta- rangeseg: Lidar sequence semantic segmentation using mul- tiple feature aggregation

    Song Wang, Jianke Zhu, and Ruixiang Zhang. Meta- rangeseg: Lidar sequence semantic segmentation using mul- tiple feature aggregation. IEEE Robotics and Automation Letters, 7(4):9739–9746, 2022. 2

    2022

  44. [51]

    Lidar2map: In defense of lidar-based semantic map construction using online camera distillation

    Song Wang, Wentong Li, Wenyu Liu, Xiaolu Liu, and Jianke Zhu. Lidar2map: In defense of lidar-based semantic map construction using online camera distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5186–5195, 2023. 2

    2023

  45. [52]

    Robobev: Towards robust bird’s eye view perception under corruptions

    Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robobev: Towards robust bird’s eye view perception under corruptions. arXiv preprint arXiv:2304.06719, 2023. 1

    2023 arXiv

  46. [53]

    Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection

    Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection. In IEEE/CVF International Conference on Computer Visio...

    2023

  47. [54]

    Efficientsam: Leveraged masked image pretraining for efficient segment anything

    Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xi- ang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In IEEE/CVF Conference on Computer Vision and Patt...

    2024

  48. [55]

    Cross modal trans- former: Towards fast and robust 3d object detection

    Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal trans- former: Towards fast and robust 3d object detection. In IEEE/CVF International Conference on Computer Vision , pages 18268–18278, 2023. 1, 2

    2023

  49. [56]

    Sam3d: Segment anything in 3d scenes.arXiv preprint arXiv:2306.03908, 2023

    Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes.arXiv preprint arXiv:2306.03908, 2023. 2

    2023 arXiv

  50. [57]

    Clip2: Contrastive language- image-point pretraining from real-world point cloud data

    Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language- image-point pretraining from real-world point cloud data. In IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  51. [58]

    Faster segment anything: Towards lightweight sam for mo- bile applications

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mo- bile applications. arXiv preprint arXiv:2306.14289 , 2023. 2

    2023 arXiv

  52. [59]

    Sparselif: High-performance sparse lidar- camera fusion for 3d object detection

    Hongcheng Zhang, Liu Liang, Pengxin Zeng, Xiao Song, and Zhe Wang. Sparselif: High-performance sparse lidar- camera fusion for 3d object detection. In European Confer- ence on Computer Vision, pages 109–128, 2024. 1, 2

    2024

  53. [60]

    Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip

    Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In IEEE/CVF International Conference on Com- puter Vision, pages 2048–2059, 2023. 3

    2023

  54. [61]

    Fusionocc: Multi-modal fusion for 3d occupancy prediction

    Shuo Zhang, Yupeng Zhai, Jilin Mei, and Yu Hu. Fusionocc: Multi-modal fusion for 3d occupancy prediction. In ACM International Conference on Multimedia , pages 787–796,

  55. [62]

    Fast segment any- thing

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing. arXiv preprint arXiv:2306.12156, 2023. 2

    2023 arXiv

  56. [63]

    Veon: V ocabulary- enhanced occupancy prediction

    Jilai Zheng, Pin Tang, Zhongdao Wang, Guoqing Wang, Xi- angxuan Ren, Bailan Feng, and Chao Ma. Veon: V ocabulary- enhanced occupancy prediction. In European Conference on Computer Vision, pages 92–108, 2024. 3

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.