pith. sign in

arxiv: 2404.05307 · v2 · pith:RA3NWJFNnew · submitted 2024-04-08 · 💻 cs.CV · cs.RO

4D Radar Semantic Segmentation of People in Field Conditions Using Temporal Multi-View Networks

Mikael Skog , Oleksandr Kotlyar , Vladim\'ir Kubelka , Martin Magnusson This is my paper

Pith reviewed 2026-05-24 02:07 UTC · model grok-4.3

classification 💻 cs.CV cs.RO
keywords 4D radarsemantic segmentationpeople detectionConvLSTMmulti-view projectionlow-visibility conditionsrobot perceptiontemporal networks
0
0 comments X

The pith

Temporal multi-view networks turn 4D radar projections into person segmentation that works in dust and fog.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that 4D radar can support reliable people detection for robots when cameras and lidars cannot operate. It introduces TMVA4D networks that take multiple 2D projections of range, azimuth, elevation and Doppler data and feed them to CNN and ConvLSTM encoders. The resulting models separate person points from background across real industrial sites, reaching 75.9 percent Dice and 61.2 percent IoU even under low visibility. This approach keeps the Doppler velocity cue and the temporal dimension without needing full 4D volumetric processing.

Core claim

CNN and ConvLSTM encoders applied to elevation, azimuth, range and Doppler 2D projections of 4D radar point clouds produce semantic segmentation masks that distinguish people from background with Dice 75.9 percent and IoU 61.2 percent across multiple operational field sites.

What carries the argument

TMVA4D, a family of CNN-plus-ConvLSTM architectures that process a set of 2D projections of the four-dimensional radar cube to perform per-point semantic segmentation.

If this is right

  • Robots can maintain people detection in dust, fog and smoke where vision and lidar fail.
  • The same projection-plus-temporal-encoding approach can be retrained for other object classes in radar data.
  • Per-point Doppler velocity is retained as an explicit input channel alongside spatial projections.
  • Public release of data and code will allow direct replication and extension on new radar hardware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be combined with existing lidar or camera pipelines for sensor fusion without requiring full 4D convolution.
  • ConvLSTM layers may support frame-to-frame tracking of moving people in addition to static segmentation.
  • Performance may degrade when people are stationary and lack distinct Doppler signatures, suggesting a need for explicit velocity-augmented loss terms.

Load-bearing premise

The chosen 2D projections keep enough information for the networks to separate person points from background without critical loss of the original 4D structure.

What would settle it

A new test set collected at an unseen industrial site that yields Dice scores below 60 percent for the person class would falsify the claim of promising performance under field conditions.

Figures

Figures reproduced from arXiv: 2404.05307 by Martin Magnusson, Mikael Skog, Oleksandr Kotlyar, Vladim\'ir Kubelka.

Figure 1
Figure 1. Figure 1: Predicted mask in the camera (elevation-azimuth) view, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Point cloud projected to the EA view with corresponding [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the proposed TMVA4D architecture. The ar [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison between ground truth and TMVA4D predictions [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

Reliable people detection is crucial for the safe autonomy of mobile robots and heavy vehicles, both on roads and in industrial settings like mining and construction. However, common sensors like cameras or lidars are prone to failure in adverse conditions such as dust, fog, or smoke, which limits their use in real-world robotic systems. Radar, on the other hand, delivers robust measurements in a wide range of environmental conditions. In particular, modern high-resolution 4D imaging radars provide 4D point clouds across range, azimuth, and elevation, as well as per-point Doppler velocity data, well suited for robot perception. We propose TMVA4D, a family of artificial neural network architectures based on CNN and ConvLSTM encoders that leverage the 4D radar modality for semantic segmentation. The architectures are trained to distinguish between background and person classes using a series of 2D projections of the 4D radar data, encompassing elevation, azimuth, range, and Doppler velocity dimensions. Evaluated across several operational sites, our models achieve promising performance (Dice 75.9%, IoU 61.2% for class person) even in low-visibility conditions. The data and code will be made publicly available upon publication.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces TMVA4D, a family of CNN- and ConvLSTM-based architectures for semantic segmentation of 4D radar point clouds into person and background classes. The networks operate on multiple 2D projections of the 4D data (range-azimuth, range-elevation, Doppler, etc.) and are evaluated on field data from several operational sites, reporting Dice 75.9% and IoU 61.2% for the person class even under low-visibility conditions. The authors state that data and code will be released publicly.

Significance. If the reported performance is reproducible and generalizes beyond the evaluated sites, the work would provide a concrete demonstration that 4D radar can support reliable person detection in conditions where cameras and lidars degrade. The explicit plan to release data and code is a positive contribution to reproducibility in radar perception research.

major comments (2)
  1. [Abstract / §4] Abstract and §4 (Experiments): the central performance numbers (Dice 75.9%, IoU 61.2%) are presented without any description of training protocol, choice of baselines, cross-validation procedure, number of independent runs, error bars, or data exclusion criteria. These omissions make it impossible to assess whether the quoted figures support the claim of “promising performance.”
  2. [§3] §3 (Method): the paper states that 2D projections are used but does not quantify information loss relative to the native 4D representation (e.g., via an ablation that compares projected vs. volumetric or point-cloud inputs). This directly affects the weakest assumption identified in the review—that the chosen projections retain sufficient discriminative power.
minor comments (2)
  1. [Abstract] The abstract claims evaluation “across several operational sites” but does not specify how many sites, their diversity, or whether any site was held out for testing.
  2. [§3] Notation for the four projection planes (elevation, azimuth, range, Doppler) should be defined once in §3 and used consistently in figures and equations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below, indicating where revisions will be made to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract / §4] Abstract and §4 (Experiments): the central performance numbers (Dice 75.9%, IoU 61.2%) are presented without any description of training protocol, choice of baselines, cross-validation procedure, number of independent runs, error bars, or data exclusion criteria. These omissions make it impossible to assess whether the quoted figures support the claim of “promising performance.”

    Authors: We agree that the experimental details are insufficient for full reproducibility assessment. In the revised manuscript we will expand §4 with a complete description of the training protocol (including optimizer, learning rate schedule, loss function, and hardware), the baselines evaluated, the cross-validation procedure, the number of independent runs, standard error bars on all metrics, and explicit data exclusion criteria. These additions will directly support evaluation of the reported Dice and IoU figures. revision: yes

  2. Referee: [§3] §3 (Method): the paper states that 2D projections are used but does not quantify information loss relative to the native 4D representation (e.g., via an ablation that compares projected vs. volumetric or point-cloud inputs). This directly affects the weakest assumption identified in the review—that the chosen projections retain sufficient discriminative power.

    Authors: We acknowledge that an explicit quantification of information loss would strengthen the justification for the multi-view projection approach. Our design choice is motivated by computational tractability and the established effectiveness of 2D radar projections in the literature; direct 4D volumetric processing would incur prohibitive memory and compute costs for the target robotic platforms. In the revision we will add a dedicated paragraph in §3 discussing this rationale, citing supporting evidence from prior radar perception work, and noting that the public release of data and code will enable future comparisons. A full ablation against native 4D inputs is not feasible within the current experimental scope but will be flagged as future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical performance claim only

full rationale

The paper reports an empirical result from training CNN/ConvLSTM networks on 2D projections of 4D radar point clouds and measuring segmentation metrics (Dice/IoU) on held-out field data. No derivation, equation, or uniqueness theorem is invoked; the central claim is a measured performance number on external test sites, not a quantity forced by fitting or self-citation. The architecture description is a standard encoder design choice with no self-referential reduction. This matches the default expectation for an applied ML paper whose output is an experimental benchmark rather than a closed-form derivation.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

The performance claim rests on empirical training of deep networks whose weights are fitted to the collected radar datasets; no additional invented entities or non-standard axioms are introduced beyond standard deep-learning assumptions.

free parameters (1)
  • network weights and hyperparameters
    CNN and ConvLSTM parameters are fitted during training on the radar projection data; exact count and selection procedure not stated in abstract.

pith-pipeline@v0.9.0 · 5761 in / 1105 out tokens · 28979 ms · 2026-05-24T02:07:55.936350+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

  1. [1]

    Boreas: A multi-season au- tonomous driving dataset

    Keenan Burnett, David J Yoon, Yuchen Wu, Andrew Z Li, Haowei Zhang, Shichen Lu, Jingxing Qian, Wei-Kang Tseng, Andrew Lambert, Keith YK Leung, Angela P Schoel- lig, and Timothy D Barfoot. “Boreas: A multi-season au- tonomous driving dataset”. In: The International Journal of Robotics Research 42.1-2 (2023), pp. 33–42. DOI: 10.1177/ 02783649231160195. epri...

    work page 2023

  2. [2]

    ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

    Anjun Chen, Xiangyu Wang, Kun Shi, Shaohao Zhu, Bin Fang, Yingfeng Chen, Jiming Chen, Yuchi Huo, and Qi Ye. “ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions”. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023. DOI: 10.1109/icra48891.2023.10161428

    work page doi:10.1109/icra48891.2023.10161428 2023

  3. [3]

    DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”. In: IEEE Trans- actions on Pattern Analysis and Machine Intelligence 40.4 (2018), pp. 834–848. DOI: 10.1109/TPAMI.2017.2699184

    work page doi:10.1109/tpami.2017.2699184 2018

  4. [4]

    Merits and Limitations of Automotive Radar for Land Vehicle Positioning in Challenging Environ- ments

    Emma Dawson, Eslam Mounier, Mohamed Elhabiby, and Aboelmagd Noureldin. “Merits and Limitations of Automotive Radar for Land Vehicle Positioning in Challenging Environ- ments”. In: IEEE Sensors Journal 23.21 (2023), pp. 26691– 26700. DOI: 10.1109/JSEN.2023.3318069

    work page doi:10.1109/jsen.2023.3318069 2023

  5. [5]

    RAMP-CNN: A Novel Neural Network for Enhanced Auto- motive Radar Object Recognition

    Xiangyu Gao, Guanbin Xing, Sumit Roy, and Hui Liu. “RAMP-CNN: A Novel Neural Network for Enhanced Auto- motive Radar Object Recognition”. In: IEEE Sensors Journal 21.4 (2021), pp. 5119–5132. DOI: 10 . 1109 / JSEN . 2020 . 3036047

    work page 2021

  6. [6]

    Safety Perfor- mance: Benchmarking Progress of ICMM Company Members In 2022

    International Council on Mining and Metals. Safety Perfor- mance: Benchmarking Progress of ICMM Company Members In 2022 . International Council on Mining and Metals, 2023

    work page 2022

  7. [7]

    RSS-Net: Weakly-Supervised Multi-Class Seman- tic Segmentation with FMCW Radar

    Prannay Kaul, Daniele de Martini, Matthew Gadd, and Paul Newman. “RSS-Net: Weakly-Supervised Multi-Class Seman- tic Segmentation with FMCW Radar”. In: 2020 IEEE In- telligent V ehicles Symposium (IV) . 2020, pp. 431–436. DOI: 10.1109/IV47402.2020.9304674

    work page doi:10.1109/iv47402.2020.9304674 2020

  8. [8]

    Empirical Anal- ysis of Autonomous Vehicle’s LiDAR Detection Performance Degradation for Actual Road Driving in Rain and Fog

    Jiyoon Kim, Bum-jin Park, and Jisoo Kim. “Empirical Anal- ysis of Autonomous Vehicle’s LiDAR Detection Performance Degradation for Actual Road Driving in Rain and Fog”. In: Sensors 23.6 (2023). DOI: 10.3390/s23062972

    work page doi:10.3390/s23062972 2023

  9. [9]

    Adam: A Method for Stochastic Optimization

    Diederik Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: International Conference on Learning Representations (2014)

    work page 2014

  10. [10]

    PointPillars: Fast Encoders for Object Detection From Point Clouds

    Alex H. Lang, Sourabh V ora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. “PointPillars: Fast Encoders for Object Detection From Point Clouds”. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, pp. 12689–12697. DOI: 10.1109/CVPR.2019. 01298

    work page doi:10.1109/cvpr.2019 2019

  11. [11]

    V-Net: Fully Convolutional Neural Networks for V olumetric Medical Image Segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. “V-Net: Fully Convolutional Neural Networks for V olumetric Medical Image Segmentation”. In: 2016 F ourth International Conference on 3D Vision (3DV) . 2016, pp. 565–571. DOI: 10.1109/3DV .2016.79

    work page doi:10.1109/3dv 2016

  12. [12]

    Plenoc- trees for real-time rendering of neural radiance fields,

    Arthur Ouaknine, Alasdair Newson, Patrick P ´erez, Florence Tupin, and Julien Rebut. “Multi-View Radar Semantic Seg- mentation”. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . 2021, pp. 15651–15660. DOI: 10. 1109/ICCV48922.2021.01538

    work page arXiv 2021

  13. [13]

    CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations

    Arthur Ouaknine, Alasdair Newson, Julien Rebut, Florence Tupin, and Patrick P ´erez. “CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations”. In: 2020 25th International Conference on Pattern Recogni- tion (ICPR). 2020, pp. 5068–5075. DOI: 10.1109/ICPR48806. 2021.9413181

    work page doi:10.1109/icpr48806 2020

  14. [14]

    K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions

    Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wi- jaya. “K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions”. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. V ol. 35. Curran Associates, Inc., 2022, pp. 3819–3829

    work page 2022

  15. [15]

    VINSEval: Evaluation Framework for Unified Testing of Consistency and Robustness of Visual-Inertial Navigation System Algorithms,

    Marcel Sheeny, Emanuele De Pellegrin, Saptarshi Mukher- jee, Alireza Ahrabian, Sen Wang, and Andrew Wallace. “RADIATE: A Radar Dataset for Automotive Perception in Bad Weather”. In: 2021 IEEE International Conference on Robotics and Automation (ICRA) . 2021, pp. 1–7. DOI: 10 . 1109/ICRA48506.2021.9562089

    work page arXiv 2021

  16. [16]

    Going deeper with convolutions

    Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. “Going deeper with convolutions”. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2015, pp. 1–9. DOI: 10 . 1109/CVPR.2015.7298594

    work page arXiv 2015

  17. [17]

    RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera- Radar Fused Object 3D Localization

    Yizhou Wang, Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu. “RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera- Radar Fused Object 3D Localization”. In: IEEE Journal of Selected Topics in Signal Processing 15.4 (2021), pp. 954–

    work page 2021

  18. [18]

    DOI: 10.1109/JSTSP.2021.3058895

    work page doi:10.1109/jstsp.2021.3058895 2021

  19. [19]

    In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

    Yizhou Wang, Gaoang Wang, Hung-Min Hsu, Hui Liu, and Jenq-Neng Hwang. “Rethinking of Radar’s Role: A Camera- Radar Dataset and Systematic Annotator via Coordinate Alignment”. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . 2021, pp. 2809–2818. DOI: 10.1109/CVPRW53098.2021.00316

    work page doi:10.1109/cvprw53098.2021.00316 2021

  20. [20]

    Relatively lazy: Indoor-outdoor navigation using vision and GNSS,

    Ao Zhang, Farzan Erlik Nowruzi, and Robert Laganiere. “RADDet: Range-Azimuth-Doppler based Radar Object De- tection for Dynamic Road Users”. In: 2021 18th Conference on Robots and Vision (CRV) . 2021, pp. 95–102. DOI: 10.1109/ CRV52889.2021.00021

    work page arXiv 2021

  21. [21]

    TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving

    Lianqing Zheng, Zhixiong Ma, Xichan Zhu, Bin Tan, Sen Li, Kai Long, Weiqi Sun, Sihan Chen, Lu Zhang, Mengyue Wan, Libo Huang, and Jie Bai. “TJ4DRadSet: A 4D Radar Dataset for Autonomous Driving”. In: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) . 2022, pp. 493–498. DOI: 10.1109/ITSC55140.2022.9922539

    work page doi:10.1109/itsc55140.2022.9922539 2022