Pith. sign in

REVIEW 4 major objections 6 minor 25 references

LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing depth from LiDAR or stereo into Mask2Former with image-conditioned weighting improves PQ by up to five points and matches a video-supervised baseline in VPQ without video training.

desk verdict Modest depth-fusion results and a fixable but real overclaim: the video-free VPS story is contradicted by the paper's own fine-tuning on Cityscapes-VPS. read the letter →

arxiv 2412.20881 v1 pith:J7BT775U submitted 2024-12-30 cs.CV

Fardin Ayar , Ehsan Javanmardi , Manabu Tsukada , Mahdi Javanmardi , Mohammad Rahmati This is my paper · ORCID
classification cs.CV
keywords panopticsegmentationLiDAR-camerafusionvideodynamicfeatureweightinglocation-awarequeriestime-awareMask2Formerautonomousvehicles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that adding depth information, whether from stereo or LiDAR, to a camera-only panoptic segmentation network improves both image and video panoptic segmentation, and that the video version can rival video-supervised models without any video training. The authors build on Mask2Former and fuse depth features through a learned, image-conditioned weighting of depth features, which they report raises panoptic quality from 57.18 to 62.12 on Cityscapes. For video, they add two small query modifications—location-aware queries that predict each segment's position and time-aware queries that reuse previous-frame queries at inference—and report VPQ of 57.24, essentially matching the video-supervised Video K-Net's 57.08. The practical significance is that autonomous-vehicle perception could get better segmentation from cheap depth sensors and skip expensive video annotations.

What carries the argument

The image-side mechanism is the fusion function $\phi(F_I, F_D) = F_I + \sigma(\mathrm{conv}_{1\times1}(F_I)) \cdot \gamma F_D$, where $\sigma$ is a sigmoid gate computed from image features, so the model decides per spatial location how much depth to trust. The video-side mechanisms are location-aware queries (an MLP trained with L1 loss to predict each thing segment's bounding-box center, making query matching position-sensitive) and time-aware queries (feeding only non-empty queries from the previous frame as the initial decoder queries for the current frame, applied only at evaluation). Segment association across frames is done by Hungarian matching of output queries, inherited from the video-free baseline.

What would settle it

Run the same Mask2Former training protocol on a dataset with real projected LiDAR and panoptic labels, such as nuScenes or SemanticKITTI, and compare PQ and VPQ against the paper's simulated-LiDAR results; the claim is falsified if the real-LiDAR gain over the camera-only baseline drops below the reported margin or the VPQ advantage disappears.

Watch

Extended reading notes

Core claim

The central claim is that depth is a cheap, effective auxiliary modality for panoptic segmentation, and that the depth signal can be injected at the feature level without redesigning the segmentation head. Concretely, the paper reports that replacing simple feature summation with a learned gate, formed by a 1x1 convolution over image features that scales depth features before adding them, raises panoptic quality on Cityscapes from 57.18 to 62.12 when the depth branch is pretrained. For video, the same model with two query-level changes—location-aware queries that predict segment positions, and time-aware queries that carry non-empty queries from the previous frame into the current decoder at inference—reaches 57.24 VPQ on Cityscapes-VPS, effectively matching the video-supervised Video K-Net at 57.08. The authors interpret this as evidence that video-supervised training is not strictly necessary for video panoptic segmentation in autonomous driving settings.

Load-bearing premise

The whole LiDAR-camera fusion claim rides on treating simulated LiDAR, created by angle-based downsampling of stereo depth, as an accurate stand-in for real LiDAR's noise, sparsity, and ray-drop; if that simulation does not transfer, the reported PQ and VPQ gains may not appear with a real sensor.

Editorial extensions

If this is right

  • Over the batch-size-6 camera-only baseline, stereo depth adds about 2.1 PQ points and simulated LiDAR depth with a pretrained depth backbone adds about 4.9 PQ points (57.18 to 62.12).
  • In video panoptic segmentation, depth fusion alone improves VPQ from 51.71 to 54.36, location-aware queries raise it to 55.41, and time-aware queries push it to 57.24, essentially tying the video-supervised Video K-Net's 57.08.
  • The gains are concentrated in thing classes such as vehicles and people, whose boundaries are geometry-defined, while stuff classes improve less.
  • A camera-plus-depth perception stack can approach video-supervised performance while skipping video annotation and video-specific losses.
  • The remaining gap to video-supervised methods is specifically in tracking thing segments (39.86 versus 45.0 VPQ for things), not in stuff segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the LiDAR experiments are simulations from stereo depth, the natural next test is whether the gains survive with real LiDAR noise and occlusions, where the fusion gate may need to be more conservative.
  • The fusion formula is architecture-agnostic, so the same dynamic weighting could be transplanted into other transformer-based or convolutional segmentation and detection heads with minimal change.
  • Since time-aware queries are used only at inference, the proposed video model adds no training-time cost, so the recipe is directly usable when video labels are scarce.
  • The stronger improvement on things than stuff suggests depth contributes mostly geometric boundary cues; one could probe this by ablating on datasets with many thin or partially occluded objects, where such cues should matter most.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a LiDAR-camera fusion module for Mask2Former-based panoptic segmentation, using two separate ResNet50 backbones and a dynamic feature-weighting function, plus location-aware queries (LAQ) and time-aware queries (TAQ) for video panoptic segmentation. It claims these modifications improve image panoptic quality by up to 5 points and achieve a VPQ of 57.24 on Cityscapes-VPS without training on video data. Experiments compare the proposed model against a Mask2Former baseline and the video-supervised Video K-Net on Cityscapes and Cityscapes-VPS.

Significance. If the central claim held, the contribution would be valuable for autonomous driving, since video panoptic segmentation without video supervision would remove the need for expensive video annotations. The paper honestly reports that VPQ_things still lags behind Video K-Net (39.86 vs 45.0), and the architecture is simple and easy to reproduce. However, the claim is currently undermined by the experimental protocol: all VPS results in Table 2 include fine-tuning on the Cityscapes-VPS video dataset, and all LiDAR results use simulated rather than real LiDAR. These issues must be resolved before the significance of the contribution can be assessed.

major comments (4)
  1. [§4.5, Table 2, Abstract, §3.1.1] The paper's central claim that the proposed model performs video panoptic segmentation 'without video training' is not supported by the experimental protocol. Section 4.5 states that 'all models in this section are first trained on the Cityscapes dataset and then fine-tuned on the Cityscapes-vps for an additional 5,000 iterations'; Cityscapes-VPS is a video dataset, so the VPQ values in Table 2 are obtained after training on the evaluation benchmark's video frames. The manuscript never reports results for models trained only on Cityscapes without this fine-tuning, so the observed VPQ gains from LAQ/TAQ could be confounded with the additional video training. The claims in the abstract, Section 3.1.1, and the conclusion must be revised or supported by experiments without VPS fine-tuning.
  2. [Table 1] The claimed 'up to 5 points' improvement in image panoptic quality is based on an unfair baseline comparison. The first row of Table 1 reports the original Mask2Former with batch size 16 (PQ 61.10), while all subsequent rows use batch size 6, giving the base model PQ 57.18; the batch-size reduction alone accounts for a 3.92 PQ drop. The LiDAR fusion gain (62.12 vs 57.18) should be compared against a batch-size-matched baseline, or the paper must explicitly justify why the batch-6 baseline is the appropriate reference.
  3. [§4.1, §4.4] All LiDAR results in the paper are obtained from simulated LiDAR, not real sensor data. Section 4.1 explains that LiDAR is mimicked by taking stereo disparity, applying angle-based downsampling to resemble a Velodyne-64, and randomly dropping 30% of rays; Section 4.4 then presents these as 'LiDAR' results. The conclusion that LiDAR-camera fusion improves panoptic segmentation may not transfer to real LiDAR sensors with different noise, sparsity, and systematic errors. The paper should either evaluate on a real LiDAR dataset (e.g., nuScenes or SemanticKITTI) or explicitly restrict the claims to simulated depth and LiDAR.
  4. [§3.3.2, Table 2] The time-aware query (TAQ) component is applied only during evaluation, as stated in Section 3.3.2 ('will only be applied during the evaluation phase while network training remains unchanged'). This means the 'video-free' model nevertheless uses temporal information from the previous frame at inference. The paper should clarify the distinction between 'no video training' and 'no video data at inference,' and should discuss the potential for error accumulation from reusing non-empty queries at test time.
minor comments (6)
  1. [§4.5] Section 4.5 refers to 'Table ??'; the table number must be filled in.
  2. [§1] In the Introduction, 'segmenting (e.g. sky, road, etc.)' should read 'segments' or 'stuff classes'.
  3. [§2.3, References] Reference [9] is cited as 'Gang et al.' but the first author is Keke Geng; the citation should be corrected.
  4. [§4.3] The description of pretraining the depth network is unclear: it says a depth image classification network was trained from scratch using estimated depth [1] from ImageNet, but reference [1] is a metric-depth estimator, not an ImageNet classification pretraining procedure; please specify the architecture and training data.
  5. [§1, §4.4] The footnote in Section 1 states that 'there is no difference in our proposed model in terms of what the 3D modality is,' yet Section 4.4 treats stereo and LiDAR separately with different preprocessing; these statements should be reconciled.
  6. [Figure 3] The qualitative discussion mentions 'the second image' for the red-boxed mistakes, but the red boxes appear in the third image of the figure; verify the wording.

Circularity Check

1 steps flagged · score 6.0 of 10

The VPS contribution's 'without video training' claim is evaluated after fine-tuning on the video benchmark; image-PS fusion results are independent and not circular.

  1. fitted input called prediction [§3.1.1 (Extension to Video Panoptic Segmentation) vs. §4.5 and Table 2]
    ""the network also does not require video datasets for training" (§3.1.1); "Following the common practice of related works [17], all models in this section are first trained on the Cityscapes dataset and then fine-tuned on the Cityscapes-vps for an additional 5,000 iterations" (§4.5)."

    The paper's central VPS claim is that location/time-aware queries give strong VPQ "without being trained on video data". The Table 2 VPQ numbers, including 57.24 for the proposed model, are produced by models that were fine-tuned on Cityscapes-vps, a video dataset, directly before evaluation. The reported VPQ is therefore a video-fine-tuned result, not a video-free prediction. The claim "does not require video datasets for training" is contradicted by the protocol, and the observed gains from the query modifications are confounded with additional video-domain fine-tuning; no video-free ablation is reported.

full rationale

The image panoptic segmentation contribution is self-contained: Equation (3-2) defines a learnable dynamic weighting (sigma(conv1x1(F_I)) * gamma F_D) trained on Cityscapes in the usual supervised sense, and the 5-point PQ gain is an empirical result, not an identity or a renamed fitted parameter. The LAQ and TAQ modules are also architectural proposals with supervised losses, not derivations from the quantities they predict. There are no load-bearing self-citations: the cited MinVIS and Video K-Net baselines are external and are used as prior work, not to forbid alternatives. The sole circularity is the video panoptic 'without video training' claim: §3.1.1 asserts no video datasets are needed, but §4.5 fine-tunes every model in that section on Cityscapes-vps before evaluating VPQ, so the headline VPS result reduces to a video-supervised fine-tuning setup. This makes the VPS portion partially circular by construction; the image-side fusion results remain independent evidence. The paper also has a separate validity concern (simulated LiDAR), but that is not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The empirical claims depend on assumptions about data simulation (simulated LiDAR representing real LiDAR), about what counts as video training, and about the sufficiency of depth-only geometric input. The method adds trained parameters (gating vector gamma, LAQ MLP) and a hand-set loss weight. No new physical entities are introduced.

free parameters (4)
  • L1 loss weight for location-aware queries = 5
    Set to 5 in Section 4.2 with no sensitivity analysis; the location loss only applies to thing segments.
  • LiDAR ray-drop downsampling factor = 0.3
    Section 4.1 randomly downsamples the simulated LiDAR by 0.3 to mimic ray-drop; the value is chosen by hand and not validated against real LiDAR.
  • Learned gating vector gamma = learned during training (dimension C_D)
    Equation (3-2) uses a per-channel vector gamma to weight depth features after a 1x1-conv gate; it is a learned parameter, so the fusion strength is fitted to the training data.
  • Angle-based downsampling pattern for Velodyne-64 mimicry = not fully specified
    Section 4.1 mentions angle-based downsampling to mimic a Velodyne-64 LiDAR but does not provide the exact angular pattern, leaving a free choice that affects results.
assumptions (5)
  • domain assumption Simulated LiDAR generated by downsampling estimated stereo disparity is representative of real LiDAR sensor data for fusion evaluation.
    Section 4.1 creates 'simulated LiDAR' from a learning-based stereo matcher [22] with angle-based downsampling to mimic Velodyne-64 and random 0.3 ray-drop. All LiDAR results in Tables 1 and 2 depend on this premise.
  • domain assumption Fine-tuning on the Cityscapes-VPS video dataset for 5,000 iterations does not count as video training.
    Section 4.5 fine-tunes all models on Cityscapes-vps, yet the title and abstract claim 'without video training.' The claim only holds if this fine-tuning is excluded from 'video training.'
  • domain assumption Depth alone is sufficient as the extra geometric input within image boundaries.
    Section 3.2(c) states that given depth z and pixel coordinates (u,v), the other 3D coordinates are determined, so no other LiDAR channels (e.g., intensity) are needed. This justifies the single-channel depth input.
  • domain assumption Bipartite matching of transformer queries across consecutive frames is sufficient for object association without video loss.
    Section 3.1.1 adopts the MinVIS [11] approach as the base for video extension; the reliability of this matching is assumed, and the LAQ and TAQ modifications build on it.
  • domain assumption ZoeDepth-estimated depth on ImageNet is a valid pretraining source for the depth backbone.
    Section 4.3 trains the depth ResNet50 from scratch on depth maps estimated by ZoeDepth [1] from ImageNet; it is assumed this pretraining transfers to Cityscapes LiDAR-style depth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training." pith.science (2026). https://pith.science/paper/J7BT775U

@misc{pith2026241220881,
  author       = {Pith},
  title        = {Pith review of: LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7BT775U}},
  note         = {Machine review of arXiv:2412.20881}
}
read the original abstract

Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied for cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS).This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.

Figures

Figures reproduced from arXiv: 2412.20881 by the authors.

Figure 1
Figure 1. (a) Example image from the cityscapes dataset. (b) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of our proposed method, which is based on Mask2Former [ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Panoptic segmentation output for a video sequence. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 17 canonical work pages

  1. [1]

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller

  2. [2]

    Luca Caltagirone, Mauro Bellone, Lennart Svensson, and Mattias Wahde. 2019. LIDAR–camera fusion for road detection using fully convolutional neural net- works. Robotics and Autonomous Systems 111 (2019), 125–131

    work page 2019

  3. [3]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision . Springer, 213–229

    2020

  4. [4]

    Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. 2020. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 12475–12485

    work page 2020

  5. [5]

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1290–1299

    2022

  6. [6]

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classifica- tion is not all you need for semantic segmentation.Advances in neural information processing systems 34 (2021), 17864–17875

    2021

  7. [7]

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus En- zweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3213–3223

    2016

  8. [8]

    Söhnke Benedikt Fischedick, Daniel Seichter, Robin Schmidt, Leonard Rabes, and Horst-Michael Gross. 2023. Efficient multi-task scene analysis with rgb-d transformers. In 2023 International Joint Conference on Neural Networks (IJCNN) . IEEE, 1–10

    work page 2023

Show all 25 references
  1. [9]

    Keke Geng, Ge Dong, Guodong Yin, and Jingyu Hu. 2020. Deep dual-modal traffic objects instance segmentation method using camera and lidar data for autonomous driving. Remote Sensing 12, 20 (2020), 3274

    2020

  2. [10]

    Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik. 2014. Learn- ing rich features from RGB-D images for object detection and segmentation. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13...

    2014

  3. [11]

    De-An Huang, Zhiding Yu, and Anima Anandkumar. 2022. Minvis: A minimal video instance segmentation framework without video-based training. Advances in Neural Information Processing Systems 35 (2022), 31265–31277

    2022

  4. [12]

    Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. 2020. Video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9859–9868

    2020

  5. [13]

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dol- lár. 2019. Panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9404–9413

    2019

  6. [14]

    Jason Ku, Ali Harakeh, and Steven L Waslander. 2018. In defense of classical image processing: Fast depth completion on the cpu. In 2018 15th Conference on Computer and Robot Vision (CRV) . IEEE, 16–22

    2018

  7. [15]

    Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. 2023. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2023

  8. [16]

    Junlong Li, Bingyao Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. 2023. Tcovis: Temporally consistent online video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 1097–1107

    2023

  9. [17]

    Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. 2022. Video k-net: A simple, strong, and unified baseline for video segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18847–18857

    2022

  10. [18]

    Matteo Sodano, Federico Magistri, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. 2023. Robust double-encoder network for rgb-d panoptic segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 4953–4959

    2023

  11. [19]

    Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen

  12. [20]

    Zhangjing Wang, Yu Wu, and Qingqing Niu. 2019. Multi-sensor fusion in auto- mated driving: A survey. Ieee Access 8 (2019), 2847–2868

    2019

  13. [21]

    Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. 2022. In defense of online models for video instance segmentation. InEuropean Conference on Computer Vision. Springer, 588–605

    2022

  14. [22]

    Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. 2023. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)

    2023

  15. [23]

    Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. 2023. CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. IEEE Transactions on intelligent transportation systems (2023)

    2023

  16. [2021]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Max-deeplab: End-to-end panoptic segmentation with mask transform- ers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5463–5474

  17. [2023]

    Zoedepth: Zero-shot transfer by combining relative and metric depth.arXiv preprint arXiv:2302.12288 (2023)

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.