{"id":"b0b9b8ca-c311-4f45-b0ff-5628995fab01","arxiv_id":"2506.21547","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SAM4D is a promptable model that segments and tracks objects across camera and LiDAR streams with cross-modal prompts, trained on pseudo-labels generated by an automated data engine.","lead":"SAM4D extends the Segment Anything approach to simultaneous camera and LiDAR segmentation, letting a user prompt in either sensor and get temporally consistent masks in both. If the approach holds up, it could speed up multi-sensor annotation for autonomous driving by a large margin.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central evaluation is confounded: SAM4D is trained and scored against pseudo-labels from the same VFM-based pipeline (cross-modal IoU 0.56), and Sec. C.1 filters evaluation to score>0.5, volume>50, excluding ground regions; no human-verified 3D ground truth checks this.","rationale":"The strongest claim is architectural and task-level novelty, but its empirical support is what makes it credible. The reader's weakest assumption pinpoints exactly the same risk: the pseudo-labels serve as both training targets and evaluation ground truth, with filters that remove low-score, small, and ground-region objects. I see no internal contradiction in the model design or the UMPE/MCMA equations; the architecture is a plausible extension of SAM2. The concern is evidential rather than formal. The paper does provide independent signals: zero-shot evaluation on nuScenes and a large LiDAR gap over SAM2+Project, which is real credit where due. However, nuScenes LiDAR evaluation is restricted to foreground instance masks, and zero-shot LiDAR mIoU of 25.9 is modest. The reported mean cross-modal IoU of 0.56 (10th percentile 0.24) is low for a training and evaluation target, and the evaluation filters in Sec. C.1 mean that Tables 1, 2, and 4 measure segmentation of easy, self-consistent masklets. If a user prompts a hard object—small, ground-adjacent, or occluded—there is currently no evidence. The concrete test of human-verified evaluation on an unfiltered subset would settle whether the concern lands. If it passes, conditional acceptance is appropriate; if it fails, the central empirical claim should be downgraded. Since the reader already made the verdict CONDITIONAL based on this same weakness, the verdict should remain UNCHANGED.","tokens_in":19033,"tokens_out":3652,"duration_ms":41665,"concrete_test":"Take 10 Waymo-4DSeg validation clips and have two annotators independently produce dense 3D instance masks, including ground-adjacent and grass/curb classes, with no score or volume filtering. Recompute Table 2 and Table 4 on this human-verified subset using the same prompt protocol, and also recompute on the unfiltered pseudo-labels. If SAM4D's LiDAR mIoU advantage over SAM2+Project shrinks substantially (e.g., below +10 points) or its image gains vanish, the filtered pseudo-label evaluation is the source of the headline result; if the gap persists on human labels, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim—that SAM4D is the first promptable multi-modal segmentation model and demonstrates powerful cross-modal segmentation on Waymo-4DSeg—rests on numbers computed against its own pseudo-labels. Sec. 5.1 describes a data engine in which Grounding-DINO and SAM produce image masklets, ray casting and 4D reconstruction transfer them to LiDAR voxels, and DBSCAN filters noise; masklet quality is then measured by cross-modal IoU, yielding a mean of 0.56 with a 10th percentile of 0.24. That score is not an external ground-truth check; it measures consistency between the engine's video masklets and the engine's fused voxel masklets. Sec. C.1 then removes exactly the hard cases during evaluation: only objects with volume >50 and score >0.5 are kept, and ground regions are excluded, while training used volume >10 and score >0.3. Reported gains in Table 4 (SAM4D vs SAM2+Project: +1.6 image mIoU, +23.7 LiDAR mIoU) are therefore measured on the subset of objects where the engine's labels are most self-consistent. The image branch is initialized from SAM2 trained on SA-V, the same model family that produced the labels, so high image IoU is partly expected; LiDAR labels are derived from those image masks rather than independent 3D annotation. The external nuScenes evaluation is partial: nuInsSeg provides foreground instance masks, and zero-shot LiDAR mIoU is only 25.9 (Table 3). Without a human-annotated holdout, the central evidence cannot distinguish genuine multi-modal segmentation from learning the pseudo-label distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAM4D, described as the first promptable segmentation model that operates jointly on camera and LiDAR streams, unifying multi-modal and temporal segmentation in a single framework. The model builds on SAM2, adding a LiDAR encoder, a Unified Multi-modal Positional Encoding (UMPE) that lifts image features into a shared 3D space, and a Motion-aware Cross-modal Memory Attention (MCMA) module that uses ego-motion compensation for temporal alignment. To support training, the authors construct Waymo-4DSeg, a large pseudo-labeled dataset generated by an automated data engine that combines GroundingDINO/SAM2-derived video masklets, 4D voxel reconstruction, ray casting, and cross-modal fusion. Experiments on Waymo-4DSeg and a nuScenes generalization study report strong cross-modal segmentation performance under various prompting settings.","tokens_in":19500,"tokens_out":5164,"duration_ms":51968,"significance":"If the reported results hold up, the paper would make a meaningful contribution by defining a new task (Promptable Multi-modal Segmentation), offering a concrete architecture that addresses cross-modal prompting and temporal consistency, and providing a large-scale pseudo-labeled dataset for autonomous driving research. The paper is also transparent in some respects: it reports the pseudo-label cross-modal IoU of 0.56, describes the evaluation filters in Sec. C.1, and acknowledges limitations of the pseudo-label strategy in Sec. D.1. These strengths are undermined, however, by the self-referential evaluation protocol: the model is trained and tested against pseudo-labels produced by the same vision-foundation-model pipeline that initializes its image encoder, and no human-verified ground-truth check is provided. The lack of comparisons to strong existing baselines such as SAL or PointSAM and the absence of error bars further reduce the confidence in the quantitative claims.","major_comments":[{"comment":"The central evaluation is circular. The data engine in Sec. 5.1 uses GroundingDINO and SAM2 to generate video masklets, then projects and fuses them into LiDAR voxel masklets; the reported quality score is the cross-modal IoU between these two sets of engine-generated masks, not an external ground-truth measurement (mean 0.56, 10th percentile 0.24 per Sec. B.1). SAM4D is trained and evaluated against these same pseudo-labels, and its image encoder is initialized from SA-V, i.e., the SAM2 model family that produced the labels. Sec. C.1 further filters the evaluation to objects with score > 0.5 and volume > 50 and excludes ground regions, removing exactly the cases where the engine is least reliable. Consequently, the numbers in Tables 2 and 4 measure how well SAM4D reproduces the data engine's outputs, not true segmentation accuracy. A human-verified evaluation subset, or an existing human-annotated dataset, is needed to support the claim of powerful cross-modal segmentation.","section":"Sec. 5.1 and Sec. C.1"},{"comment":"The paper claims to be the first unified promptable camera-LiDAR segmentation model, and cites SAL [31] and PointSAM [64] as related promptable 3D segmentation approaches. However, no experiments compare against SAL, PointSAM, or other VFM-based 2D-to-3D segmentation methods. The only baseline, SAM2+Project (Table 4), is a simple projection of SAM2 video masks onto per-frame point clouds. Without strong baselines, the claimed advantage over existing modality-specific methods is not established. Please add quantitative comparisons to SAL, PointSAM, and at least one additional projection-based or fusion-based baseline under the same evaluation protocol.","section":"Sec. 6.2 and Related Work (Sec. 2)"},{"comment":"The ego-motion ablation does not convincingly support the claim that MCMA's ego-motion compensation is a major contribution. Table 6 shows that removing ego-motion changes image mIoU from 69.8 to 69.7 and J&F from 80.1 to 80.3 (slightly worse with ego-motion), while LiDAR mIoU changes from 55.7 to 52.2 and NMP from 582 to 746. The text states NMP decreases from 746 to 592, but Table 6 reports 582. More importantly, the near-zero image-domain differences and the absence of error bars or multiple seeds make it impossible to judge whether the LiDAR improvements are statistically significant. Report variance across runs, or temper the claim that ego-motion 'significantly' improves temporal consistency.","section":"Sec. 6.3, Table 6"},{"comment":"The nuScenes generalization results are weak and are overinterpreted. Zero-shot LiDAR mIoU is 25.9, which is only 18.9 points below the fine-tuned value of 44.8, and in absolute terms 25.9 mIoU is a low number for a claimed 'strong cross-modal generalization'. The evaluation relies on nuInsSeg, which provides 2D instance masks for foreground objects; it is not described how the LiDAR ground truth is obtained for nuScenes. Clarify the LiDAR label derivation and discuss these numbers honestly, including the large modality gap between image (58.4 mIoU) and LiDAR (25.9 mIoU) in the zero-shot setting.","section":"Sec. 6.2, Table 3"},{"comment":"No error bars, confidence intervals, or multiple-seed averages are reported anywhere. The evaluation uses 48 randomly selected clips (Sec. 6.1), but the reader cannot assess the variability of the mIoU, J&F, and NMP metrics. This is especially problematic for ablations like Table 6, where image mIoU differs by only 0.1. Report means and standard deviations over at least three random seeds, or at minimum over several clip subsamples, for all main tables.","section":"Sec. 6.1 and all result tables"}],"minor_comments":[{"comment":"The sentence reading 'NMP decreases from 746 to 592' contradicts the table entry of 582; the table shows 582, so the text should be corrected to 582.","section":"Sec. 6.3, Table 6"},{"comment":"There is a typo 'promotable' in the final paragraph of the Introduction: the word should be 'promptable' to match the rest of the paper.","section":"Sec. 1"},{"comment":"The intrinsic matrix K is described as a 4x4 matrix, but conventional camera intrinsics are 3x3; if a homogeneous 4x4 form is intended, please define its structure explicitly to avoid ambiguity.","section":"Sec. 4.3, Eq. (2)"},{"comment":"The main text reports only the average cross-modal IoU of 0.56; including the 10th percentile of 0.24 (given in Sec. B.1) would give readers a more complete picture of label quality and should be mentioned here as well.","section":"Sec. 5.1"},{"comment":"The phrase 'temporarily exclude instances near the ground' is vague: it is unclear whether the final model is also evaluated without ground-region instances and whether this restriction remains in any deployed setting.","section":"Sec. C.1"},{"comment":"The main text says SAM4D is trained for 36 epochs, while Table A1 reports 'steps ~44k'; clarify the correspondence between epochs and steps.","section":"Sec. 6.1 and Table A1"},{"comment":"In the sentence 'we sample 8-frame sequences', the LiDAR voxel size is given as '0.15' without units; it should read '0.15 m'.","section":"Sec. A.2"},{"comment":"The typo 'incluing' should be 'including' in the sentence 'For sparse prompts incluing points or bounding boxes'.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the evaluation circularity: the model is trained and scored against pseudo-labels from the same VFM pipeline, with filtering that removes hard cases. This is not a fatal flaw in the architecture or data-engine contributions, but it is load-bearing for the paper's central claim. If the authors cannot provide a human-verified evaluation subset or otherwise break the circularity, I would not be able to support acceptance. The lack of comparison to SAL/PointSAM and the absence of error bars further weaken the empirical support. On a separate note, the paper does not state whether code or the Waymo-4DSeg annotations will be released; given the dataset is a major claimed contribution, the authors should clarify release plans."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. The task formulation—Promptable Multi-modal Segmentation, where a prompt in either camera or LiDAR produces masks in both—is new and useful for autonomous-driving annotation. The model itself is a sensible extension of SAM2: UMPE is a reasonable way to share 3D positional encoding between image patches and LiDAR voxels, and MCMA adds ego-motion compensation to memory attention. The data engine is also a serious piece of engineering: it produces a large pseudo-labeled dataset (300k cross-modal masklets) and the authors are transparent about the pipeline. The nuScenes zero-shot result, while modest on LiDAR, does suggest some generalization beyond Waymo.\n\nThe soft spots are real and center on evaluation. The model is trained and scored against pseudo-labels produced by the same VFM family (SAM2 + GroundingDINO) that powers the data engine. The engine's own cross-modal IoU averages only 0.56, and the evaluation filters out low-score objects (score > 0.5, volume > 50) and excludes ground regions, while training uses looser thresholds. So the reported mIoU on Waymo-4DSeg measures how well the model reproduces the engine's labels on the easiest subset, not true segmentation quality. There is no human-verified 3D ground truth anywhere in the loop. The nuScenes external check is partial: LiDAR mIoU at zero shot is 25.9, which is not strong. Also, the main baseline (SAM2+Project) is weak; there is no comparison with SAL or PointSAM, the existing promptable LiDAR methods. The ego-motion ablation is marginal for image mIoU, though it does cut LiDAR NMP meaningfully. Minor point: the table and text disagree on the NMP value (582 vs 592).\n\nTo be fair, the authors acknowledge in the limitations section that pseudo-label ambiguities persist and that human-annotated subsets are future work. That honesty counts, but it does not fix the current evidence gap. The core idea is plausible, and the engineering is solid, but the claim of \"powerful cross-modal segmentation\" needs a clean evaluation on a human-annotated holdout or an external dataset with 3D labels.\n\nWho should read this: anyone working on promptable segmentation for driving or on using VFMs for auto-labeling. It would be a good reading-group paper because it raises exactly the right methodological questions about self-scoring with pseudo-labels. I would not cite it yet for its numbers, but I would cite it for the task formulation and dataset potential once the evaluation is cleaned up. A serious referee should engage with it, but with the expectation that the evaluation needs rework before acceptance.","headline":"SAM4D is a credible engineering contribution with a genuinely new task formulation, but its central evaluation is weakened by training and testing on the same VFM-generated pseudo-labels, with no human-verified 3D ground truth.","tokens_in":19959,"tokens_out":1349,"would_cite":true,"duration_ms":16563,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAM4D claims the first promptable model that segments objects jointly in camera and LiDAR streams, so a point or box in either sensor yields masks in both, propagated over time.","keywords":["promptable segmentation","multi-modal fusion","camera-LiDAR","temporal consistency","autonomous driving","pseudo-label data engine","Waymo-4DSeg","memory attention"],"falsifier":"Re-annotate a random sample of Waymo-4DSeg objects with human instance masks on both camera frames and LiDAR scans, then recompute SAM4D's mIoU against those human masks rather than the engine's pseudo-labels; if the numbers drop substantially or the ranking against the single-modality baselines changes, the pseudo-label ground truth is carrying the reported result.","tokens_in":18804,"feed_emoji":"🚗","tokens_out":11687,"duration_ms":107920,"temperature":0.7,"pith_summary":"The paper sets out to show that promptable segmentation—the click-or-box-to-mask interaction made familiar in image models—can be extended to operate simultaneously on camera and LiDAR streams in one model. SAM4D accepts a point, box, or mask prompt from either sensor and returns a temporally consistent object masklet in both modalities, which defines the new Promptable Multi-modal Segmentation (PMS) task. The practical motivation is annotation efficiency for autonomous driving: one prompt in a camera frame could label the same object in LiDAR and in all subsequent frames, turning manual multi-modal labeling into a much cheaper interactive process. To avoid a human-annotation bottleneck, the paper builds an automated data engine that turns vision-foundation-model video masklets into camera-LiDAR aligned pseudo-labels, producing the Waymo-4DSeg dataset with over 300k cross-modal masklets. Experiments on Waymo-4DSeg and an unseen nuScenes set are used to argue that cross-modal prompting, temporal propagation, and zero-shot generalization all work.","feed_headline":"Prompt either sensor, get masks in both camera and LiDAR","feed_subtitle":"SAM4D turns one prompt in either sensor into temporally consistent 2D and 3D masks across a whole drive.","key_machinery":"The two load-bearing components are UMPE and MCMA, plus the data engine that supplies training signal. UMPE gives every image patch and every LiDAR voxel a position code in a shared 3D space: image pixels are unprojected into pseudo-point clouds using estimated depth and the camera-to-LiDAR extrinsic transform, then passed through an MLP, while LiDAR voxels pass their coordinates through the same MLP, so cross-modal attention can compare positionally aligned tokens. MCMA runs self-attention within each modality, cross-attention between modalities, and temporal attention against a memory bank whose stored positions are transformed by ego-motion before retrieval, allowing long-horizon feature reuse under large vehicle motion. The mask decoder consumes sparse or mask prompts from either modality and outputs both 2D and 3D masks, and training simulates interactive prompting with corrective clicks. The data engine closes the loop by generating Waymo-4DSeg pseudo-labels: Grounding-DINO and SAM2 seed video masklets, 4D reconstruction and ray casting map pixels to voxels, and DBSCAN clustering plus overlap merging fuses and denoises the camera-LiDAR masklets.","core_discovery":"SAM4D's central claim is that image and LiDAR segmentation need not be separate tasks: with a shared 3D positional encoding, prompts can cross modalities, and with an ego-motion-compensated memory bank, objects can stay consistent over long sequences. The architecture pairs Unified Multi-modal Positional Encoding (UMPE), which lifts image features into a shared 3D space via estimated depth and camera-to-LiDAR transforms while encoding LiDAR voxels with the same MLP-based positional code, with Motion-aware Cross-modal Memory Attention (MCMA), which transforms past-frame features and object pointers into the current coordinate frame before cross-modal and temporal attention. Trained on pseudo-labels from the automated data engine, the model reports that an image prompt yields usable LiDAR masks and a LiDAR prompt yields usable image masks, that first-frame prompts propagate into stream-level masklets, and that zero-shot transfer to nuScenes is strong and improves with fine-tuning. The data engine itself is part of the contribution: it combines VFM-driven video masklets, 4D voxel reconstruction with ray casting, and DBSCAN-based fusion to generate camera-LiDAR aligned pseudo-labels, with an average cross-modal IoU of 0.56 used as a label-quality score.","pith_inferences":["The paper leaves implicit that its evaluation numbers are upper bounds on objects the data engine already scores well; a human-verified subset or a test on low-score objects would show whether SAM4D's cross-modal prompting also helps where pseudo-labels are noisier.","Because the same vision foundation models generate the training pseudo-labels, SAM4D is likely to inherit their failure modes, for example object misassociation between similar nearby instances; a testable check is to prompt SAM4D on cases where the engine's masklets are known to confuse two objects.","The shared 3D positional encoding is in principle sensor-agnostic, so the same architecture could accept other depth-bearing sensors or depth-from-mono inputs, suggesting a path toward multi-camera and multi-LiDAR setups that the paper lists only as future work.","A natural next step the paper only gestures at is natural-language prompting: text embeddings placed in the same unified 3D code could turn SAM4D into an open-vocabulary 4D segmenter."],"forward_implications":["Annotators could label both modalities from a single interaction: a point or box in one sensor returns masks in the other, reducing per-object labeling cost.","A first-frame prompt propagates into a temporally consistent masklet over the whole sequence, which is the behavior needed for semi-automatic video-and-LiDAR annotation.","The automated data engine can generate large volumes of camera-LiDAR aligned pseudo-labels orders of magnitude faster than human annotation, making scale-up of 4D training data feasible.","Ego-motion-compensated memory attention reduces object mismatches over long sequences, making the model more reliable for tracking and for labeling dynamic scenes.","Zero-shot transfer to nuScenes suggests the learned cross-modal alignment is not overfit to Waymo, and fine-tuning gives a further gain on new sensor configurations."],"supporting_citations":[{"why":"SAM2 is the video-segmentation architecture and training recipe that SAM4D extends to multi-modal streams, and its mask-propagation is reused in the data engine.","marker":"[37]"},{"why":"SAM defines the promptable point/box/mask segmentation task and decoder design that SAM4D generalizes to LiDAR.","marker":"[17]"},{"why":"Waymo Open Dataset supplies the raw camera and LiDAR sequences plus pre-annotated 3D boxes that Waymo-4DSeg is built on.","marker":"[42]"},{"why":"Lift-Splat-Shoot's depth-based unprojection is the basis for UMPE lifting image pixels into 3D.","marker":"[34]"},{"why":"MinkUNet is the sparse LiDAR encoder that turns point clouds into voxel tokens in SAM4D.","marker":"[10]"},{"why":"Grounding-DINO provides open-vocabulary object detections that seed the camera masklets in the data engine.","marker":"[24]"},{"why":"VDBFusion is used for the 4D voxel reconstruction that links image pixels to LiDAR points in the data engine.","marker":"[47]"},{"why":"nuScenes (with nuInsSeg) is the unseen dataset used for zero-shot and fine-tuning generalization tests.","marker":"[3]"}],"fun_headline_variants":["One prompt, both sensors, 2D and 3D masks","Prompt either sensor, get masks in both streams","Cross-modal prompting: one click, masks in both","One prompt, consistent masks across time and sensors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-labels produced by the automated data engine are accurate enough to serve as both training targets and evaluation ground truth; the paper reports an average cross-modal IoU of 0.56 and filters evaluation to objects with score above 0.5, volume above 50, and away from the ground.","fun_headline_variants_meta":{"raw":{"variants":["One prompt, both sensors, 2D and 3D masks","Prompt either sensor, get masks in both streams","Cross-modal prompting: one click, masks in both","One prompt, consistent masks across time and sensors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3577,"prompt_tokens":983,"completion_tokens":2594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2529}},"tokens_in":599,"tokens_out":2594,"duration_ms":23935,"temperature":1.0,"reasoning_tokens":2529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:22:47.532676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of Waymo-4DSeg objects with human instance masks on both camera frames and LiDAR scans, then recompute SAM4D's mIoU against those human masks rather than the engine's pseudo-labels; if the numbers drop substantially or the ranking against the single-modality baselines changes, the pseudo-label ground truth is carrying the reported result.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM defines the promptable point/box/mask segmentation task and decoder design that SAM4D generalizes to LiDAR."},{"cited_title":"Scalability in perception for autonomous driving: Waymo open dataset","cited_arxiv_id":null,"evidence_quote":"Waymo Open Dataset supplies the raw camera and LiDAR sequences plus pre-annotated 3D boxes that Waymo-4DSeg is built on."},{"cited_title":"Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d","cited_arxiv_id":null,"evidence_quote":"Lift-Splat-Shoot's depth-based unprojection is the basis for UMPE lifting image pixels into 3D."},{"cited_title":"4d spatio-temporal convnets: Minkowski convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"MinkUNet is the sparse LiDAR encoder that turns point clouds into voxel tokens in SAM4D."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","cited_arxiv_id":null,"evidence_quote":"Grounding-DINO provides open-vocabulary object detections that seed the camera masklets in the data engine."},{"cited_title":"Vdbfusion: Flexible and efficient tsdf integration of range sensor data","cited_arxiv_id":null,"evidence_quote":"VDBFusion is used for the 4D voxel reconstruction that links image pixels to LiDAR points in the data engine."},{"cited_title":"nuscenes: A multi- modal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"nuScenes (with nuInsSeg) is the unseen dataset used for zero-shot and fine-tuning generalization tests."}],"review_version":1}