REVIEW 5 major objections 5 minor 1 cited by
RadarRGBD A Multi-Sensor Fusion Dataset for Perception with RGB-D and mmWave Radar
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper introduces RadarRGBD, a dataset coupling RGB-D imagery with raw, high-angular-resolution millimeter-wave radar data across indoor and outdoor low-light scenes, and shows a fine-tuned depth estimator can fill Kinect depth gaps.
desk verdict A genuinely useful dataset niche, but the paper currently hinges on unverified calibration and synchronization claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dataset itself: a stationary trolley rig with a Kinect V2 and a cascaded TI AWR2243 radar, aligned through stored timestamps and through radar-to-infrared-camera extrinsics estimated by least-squares optimization over multiple retro-reflector views. The raw 128x86x64 radar cube is what makes the fusion claim operational, since it lets researchers work with range, angle, and Doppler information rather than only CFAR-extracted point clouds. For the depth-completion half, the carrying mechanism is a fine-tuned DepthAnythingV2 model whose loss stacks SiLog, smooth L1, and gradient terms, with pseudo-relative depth maps supplying scale supervision in pixels where Kinect has no valid measurement.
What would settle it
Place retro-reflectors at known positions in a scene, record both sensors, and project the radar points into the RGB-D images: if the mean reprojection error exceeds a few pixels or the timestamp offset is larger than one radar frame period (0.2 s), the fused data cannot serve as reliable ground truth. For the depth-completion claim, scan the same scenes with a second depth sensor such as lidar or multi-view stereo and compare the filled hole regions against it; if the filled depth is not closer to that reference than the original Kinect holes, the method's central effect is not confirmed.
Extended reading notes
Core claim
The central claim is that high-resolution millimeter-wave radar data — an AWR2243 cascade with 12 transmit and 16 receive antennas forming 86 effective virtual channels with about 1° azimuth resolution — can be synchronized with RGB-D frames in a single dataset that supplies both processed point clouds and the raw ADC cube. The dataset covers over 2,700 frames across indoor and outdoor scenes including low-light conditions, and includes calibration parameters and timestamps for each sensor. The paper also claims that fine-tuning DepthAnythingV2 with a combination of SiLog loss, smooth L1 loss on valid depth points, and pseudo-relative depth scale supervision effectively fills missing regions in Kinect depth maps, reporting depth-completion RMSE down to 0.122 and MAE down to 0.083 on the three tested model sizes.
Load-bearing premise
The load-bearing premise is that the stationary collection rig and the stored timestamps really keep the 5 Hz radar and 30 Hz Kinect aligned well enough for fusion, and that the radar-to-camera alignment is accurate; the paper does not report a quantitative check of either.
Editorial extensions
If this is right
- If the dataset is sound, radar-vision fusion research can use raw ADC frames, not just point clouds, enabling signal-level methods such as custom range-Doppler features or learned radar representations.
- The roughly 1° azimuth-resolution radar paired with RGB-D can serve as a benchmark for depth completion, 3D object detection, and scene understanding in indoor and outdoor low-light conditions.
- The depth-completion recipe of absolute depth at valid pixels plus pseudo-relative scale supervision could be applied to other time-of-flight or RGB-D sensors that suffer from hole artifacts.
- The released calibration files and timestamps allow cross-modal projection between radar points and RGB-D images without requiring researchers to re-calibrate the rig.
Reading between the lines
- The paper does not report a synchronization-error analysis; a natural extension is to quantify the temporal offset between the 5 Hz radar and 30 Hz Kinect using a moving target, which would tell users how much temporal misalignment to expect in fusion.
- Because both raw ADC and processed point clouds are provided, the dataset could support a controlled study of how much information CFAR-based point-cloud extraction discards, a comparison the paper mentions but does not itself run.
- If the depth-completion results generalize, pseudo-relative scale supervision could be combined with radar points as additional absolute anchors to further improve depth maps in regions where both Kinect and radar see the scene.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RadarRGBD, a multi-sensor fusion dataset combining Kinect V2 RGB-D data with millimeter-wave radar data (point clouds and raw ADC matrices) from a TI AWR2243 12Tx-16Rx cascade radar, over approximately 2,700 frames spanning indoor, outdoor, and low-light scenes. The authors position this as filling a gap in existing datasets, which typically lack either visual depth data, raw radar data, or high angular-resolution radar. As a secondary contribution, the paper proposes a depth-completion method that fine-tunes DepthAnythingV2 with absolute depth supervision from the Kinect plus pseudo-relative depth labels generated by the pretrained model, and reports quantitative results in Table III. The dataset is announced to be publicly released at a GitHub URL, but no data or code is currently accessible. The central dataset claim depends on spatial and temporal synchronization between the 5 Hz radar and the 30 Hz Kinect, and on the radar-to-camera extrinsic calibration, none of which are quantitatively validated in the manuscript.
Significance. If the dataset is delivered as described, it would be a useful resource: the combination of raw ADC radar data, radar point clouds, and RGB-D imagery across indoor and outdoor settings is not covered by existing public datasets such as RADIal, RaDICaL, or Coloradar, and the 86-virtual-channel, 1.05° azimuth-resolution radar is a meaningful step up from many low-resolution alternatives. The paper also provides a clear comparison table identifying this gap. The depth-completion method is a modest incremental contribution and its evaluation is currently not convincing, but the dataset itself is the primary value. Because the dataset is not yet available and the calibration and synchronization are asserted rather than demonstrated, the significance cannot presently be verified; these are the load-bearing points that need to be strengthened.
major comments (5)
- [Section III-C, Eq. (7)] The radar-to-infrared-camera extrinsic calibration is stated as a least-squares optimization in Eq. (7), but no quantitative validation is reported. There is no reprojection error, no hold-out validation, no number of calibration frames, and no final residual. Given the 1.05° azimuth resolution and the 512x424 depth image, an unquantified extrinsic error of even a few degrees would displace projected radar points by many pixels at typical operating distances. Please report quantitative calibration accuracy, for example the mean and maximum reprojection error on held-out retro-reflector frames, and state how many frames were used.
- [Section II and Section III-A] The claim of temporal synchronization is not supported. The statement that the system 'remained stationary, ensuring temporal and spatial synchronization' (Section II) does not establish synchronization; a stationary scene does not remove clock offset or drift between a 5 Hz radar and a 30 Hz camera. The timestamps.txt file is mentioned in Section III-E, but no trigger mechanism, timestamp-offset measurement, or synchronization error bound is described. In outdoor scenes with moving objects, 100-200 ms misalignment would visibly break fusion. Please document the synchronization procedure quantitatively, or explicitly state the assumed maximum timestamp misalignment and its effect on fusion.
- [Section IV, Eqs. (8)-(10) and Table III] The depth-completion evaluation is partly self-referential and does not substantiate the claim that missing regions are 'effectively filled.' Pseudo-relative depth labels are generated by a pretrained DepthAnythingV2 model and then used as supervision for the same missing regions (Eqs. (8)-(10)), with no independent ground truth in those holes. Table III reports RMSE, MAE, iRMSE, and iMAE, but the manuscript does not state whether these metrics are computed only on valid Kinect depth pixels or also on filled regions. Please clarify the evaluation protocol and add an experiment with independent ground truth (for example, masking out valid depth regions and comparing the completion against the held-out true depth) to validate the hole-filling claim.
- [Section IV, Eqs. (8)-(9)] Equations (8) and (9) are garbled and appear inconsistent with the accompanying text. For example, Eq. (8) mixes SiLog loss notation with scaling and translation factors without a clear definition of how they interact, and Eq. (9) introduces symbols (rho, E_x, E_y, R) without precise definitions. The duplicated paragraph at the beginning of Section IV further hampers reproducibility. Please rewrite these equations using standard notation and define every symbol.
- [Abstract and Section III-E] The core dataset contribution cannot be independently assessed because no data are currently available. The paper states the dataset and documentation 'will be publicly available' at a GitHub URL, but no download link or review copy is provided. For a dataset paper, access to the data is essential for evaluating the central claim. Please provide an anonymous review copy or a preprint-release link, and ensure the repository contains the calibration files and timestamps described in the paper.
minor comments (5)
- [Section III-E] The text says 'We compared RadarRGB with some of the currently most commonly used multi-sensor datasets'; this should say 'RadarRGBD'.
- [Section IV] References are mislabeled: in the second paragraph, '[1]' is used for Transformer, DINOv2, and DPT, which are actually references [20] and other works, not the Mapillary Vistas reference. Please correct the citation numbering.
- [Table III] The table lacks a baseline row for DepthAnythingV2 without the proposed fine-tuning or with only absolute-depth supervision; without this baseline, the reported gains from the pseudo-relative depth loss cannot be attributed to the proposed modification.
- [Section III-A] The phrase 'the optical image resolution is 1920x1080 pixels' is ambiguous; the Kinect V2 RGB image is 1920x1080, but the paper later uses the term 'infrared image' for the 512x424 depth/IR sensor. Please clarify which sensor produces which resolution.
- [Section V] The sentence 'All input images are cropped and resized to a consistent size of 518x518' would benefit from specifying how training and test splits are defined, since the dataset contains 2,700 frames and the quantitative results presumably use a subset.
Circularity Check
The dataset itself is constructed independently, but the depth-completion claim is partially circular: pseudo-relative depth labels from the pretrained DepthAnythingV2 model are scaled and used as the only supervision inside Kinect holes for a model initialized from that same pretrained model.
-
fitted input called prediction
[Section IV, Eq. (8)-(10) (depth-completion fine-tuning)]
"To ensure more reasonable depth values are regressed in the missing regions, we use pseudo-relative depth labels generated by the inference of an open-source relative depth estimation model. For the valid depth points, we compute the scaling factor between the model's predictions and the pseudo-relative depth labels. Then, for the remaining points, the labels are scaled and the L1 loss function is applied between the scaled labels and the predicted depth."
The pseudo-relative depth labels come from the pretrained DepthAnythingV2 model, and the fine-tuned depth-completion model is initialized from that same pre-trained model (Section IV). A scale/shift is fit on valid Kinect pixels where absolute depth exists, then the pretrained model's pseudo-labels are scaled and used as the supervision targets for the missing (hole) regions. Since Eq. (10) trains the output to match these scaled pseudo-labels with no independent ground truth in the holes, the reported 'filling' of missing regions is largely a measure of how well the model reproduces its own pretrained prior, anchored only globally by the Kinect-valid pixels.
full rationale
The central dataset contribution is not circular: camera intrinsics and RGB-IR extrinsics use independent Zhang calibration; radar extrinsics are solved from corner-reflector correspondences via least squares (Eq. 7); radar point clouds are produced by standard FFT/CFAR/DOA processing. No author self-citations are load-bearing; all cited datasets and methods are external. The weak temporal-synchronization justification ('the system remained stationary, ensuring temporal and spatial synchronization') and the absence of reported calibration residuals/reprojection errors are correctness and validation risks, but they are not circular reductions. The only circular element is the depth hole-filling step: pseudo-relative labels generated by the pretrained DepthAnythingV2 model supervise a model initialized from the same pretrained model, with no independent ground truth in the Kinect holes. Because this affects only the secondary depth-completion claim and the dataset claim remains externally checkable, the overall circularity score is a moderate 4.
Assumptions & free parameters
free parameters (2)
- Depth-completion loss weights =
unspecified (defaults implied in Eq. 10)
- High-loss region masking ratio =
10%
assumptions (5)
- standard math Zhang's camera calibration and standard FFT, CFAR, and DOA radar processing are valid for this platform.
- domain assumption Kinect V2 depth values at valid pixels are accurate enough to serve as ground truth for depth supervision.
- ad hoc to paper Pseudo-relative depth maps from pretrained DepthAnythingV2 provide meaningful supervision in Kinect hole regions.
- domain assumption The stationary collection setup and timestamps.txt provide sufficient temporal synchronization between radar and camera.
- domain assumption The radar-to-infrared-camera extrinsic parameters estimated by least squares in Eq. (7) are accurate enough for fusion.
Cite this review
Pith. "Pith review of RadarRGBD A Multi-Sensor Fusion Dataset for Perception with RGB-D and mmWave Radar." pith.science (2026). https://pith.science/paper/KWQSKTNR
@misc{pith2026250515860,
author = {Pith},
title = {Pith review of: RadarRGBD A Multi-Sensor Fusion Dataset for Perception with RGB-D and mmWave Radar},
year = {2026},
howpublished = {\url{https://pith.science/paper/KWQSKTNR}},
note = {Machine review of arXiv:2505.15860}
}
read the original abstract
Multi-sensor fusion has significant potential in perception tasks for both indoor and outdoor environments. Especially under challenging conditions such as adverse weather and low-light environments, the combined use of millimeter-wave radar and RGB-D sensors has shown distinct advantages. However, existing multi-sensor datasets in the fields of autonomous driving and robotics often lack high-quality millimeter-wave radar data. To address this gap, we present a new multi-sensor dataset:RadarRGBD. This dataset includes RGB-D data, millimeter-wave radar point clouds, and raw radar matrices, covering various indoor and outdoor scenes, as well as low-light environments. Compared to existing datasets, RadarRGBD employs higher-resolution millimeter-wave radar and provides raw data, offering a new research foundation for the fusion of millimeter-wave radar and visual sensors. Furthermore, to tackle the noise and gaps in depth maps captured by Kinect V2 due to occlusions and mismatches, we fine-tune an open-source relative depth estimation framework, incorporating the absolute depth information from the dataset for depth supervision. We also introduce pseudo-relative depth scale information to further optimize the global depth scale estimation. Experimental results demonstrate that the proposed method effectively fills in missing regions in sensor data. Our dataset and related documentation will be publicly available at: https://github.com/song4399/RadarRGBD.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
RadarSFD: Single-Frame Diffusion with Pretrained Priors for Radar Point Clouds
A latent diffusion model initialized with monocular depth priors reconstructs dense LiDAR-like point clouds from single radar frames, achieving state-of-the-art results on RadarHD.
Reference graph
Works this paper leans on
-
[1]
Constructed a multi-sensor fusion dataset that includes RGB-D and high- quality millimeter -wave radar data, containing both point clouds and raw radar matrices
-
[2]
Proposed and validated a depth completion method based on relative depth estimation and absolute scale supervision
-
[3]
Developed a multi-scenario depth estimation benchmark platform with high practical value. II. R ELATED WORK A. Multi-Sensor Datasets in Autonomous Driving Multi-sensor fusion technology has been widely applied in fields such as autonomous driving, robot navigation, and indoor positioning, playing a crucial role in a wide range of perception tasks. Existin...
work page 1920
-
[4]
The mapillary vistas dataset for semantic understanding of street scenes,
G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder, “The mapillary vistas dataset for semantic understanding of street scenes,” in Proc. IEEE Int. Conf. Comput. Vision, 2017, pp. 4990–4999
work page 2017
-
[5]
The cityscapes dataset for semantic urban scene understanding,
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2016, pp. 3213–3223
work page 2016
-
[6]
Bdd100k: A diverse driving dataset for heterogeneous multitask learning,
F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” in Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognit., 2020, pp. 2636–2645
work page 2020
-
[7]
Are we ready for autonomous driving? the kitti vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2012, pp. 3354–3361
work page 2012
-
[8]
Multispectral pedestrian detection: Benchmark dataset and baseline,
S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon, “Multispectral pedestrian detection: Benchmark dataset and baseline,” in Proc. IEEE Conf. Comput. Vision Pattern Recognit., 2015, pp. 1037–1045
work page 2015
Show all 23 references
-
[9]
Arkitscenes: A diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data,
G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, et al., “Arkitscenes: A diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data,” arXiv preprint arXiv:2111.08897, 2021
2021 arXiv
-
[10]
Automotive radar dataset for deep learning based 3D object detection,
M. Meyer and G. Kuschk, “Automotive radar dataset for deep learning based 3D object detection,” in Proc. 16th Eur. Radar Conf. (EuRAD), 2019, pp. 129–132
2019
-
[11]
NuScenes: A multimodal dataset for autonomous driving,
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “NuScenes: A multimodal dataset for autonomous driving,” in Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognit., 2020, pp. 11621–11631
2020
-
[12]
Pixset: An opportunity for 3d computer vision to go beyond point clouds with a full-waveform lidar dataset,
J.-L. Déziel, P. Merriaux, F. Tremblay, D. Lessard, D. Plourde, J. Stanguennec, P. Goulet, and P. Olivier, “Pixset: An opportunity for 3d computer vision to go beyond point clouds with a full-waveform lidar dataset,” in Proc. 2021 IEEE Int. Intell. Transp. Syst. Conf. (ITSC), ...
2021
-
[13]
RadarScenes: A real-world radar point cloud data set for automotive applications,
O. Schumann, M. Hahn, N. Scheiner, F. Weishaupt, J. F. Tilly, J. Dickmann, and C. Wöhler, “RadarScenes: A real-world radar point cloud data set for automotive applications,” in Proc. 2021 IEEE 24th Int. Conf. Inf. Fusion (FUSION), 2021, pp. 1–8
2021
-
[14]
TJ4DRadSet: A 4D radar dataset for autonomous driving,
L. Zheng, Z. Ma, X. Zhu, B. Tan, S. Li, K. Long, W. Sun, S. Chen, L. Zhang, M. Wan, et al., “TJ4DRadSet: A 4D radar dataset for autonomous driving,” in Proc. 2022 IEEE 25th Int. Conf. Intell. Transp. Syst. (ITSC), 2022, pp. 493–498
2022
-
[15]
NTU4DRadLM: 4D radar-centric multi-modal dataset for localization and mapping,
J. Zhang, H. Zhuge, Y. Liu, G. Peng, Z. Wu, H. Zhang, Q. Lyu, H. Li, C. Zhao, D. Kircali, et al., “NTU4DRadLM: 4D radar-centric multi-modal dataset for localization and mapping,” in Proc. 2023 IEEE 26th Int. Conf. Intell. Transp. Syst. (ITSC), 2023, pp. 4291–4296
2023
-
[16]
Raw high-definition radar for multi-task learning,
J. Rebut, A. Ouaknine, W. Malik, and P. Pérez, “Raw high-definition radar for multi-task learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 17021–17030
2022
-
[17]
Dual radar: A multi-modal dataset with dual 4D radar for autonomous driving,
X. Zhang, L. Wang, J. Chen, C. Fang, L. Yang, Z. Song, G. Yang, Y. Wang, X. Zhang, J. Li, et al., “Dual radar: A multi-modal dataset with dual 4D radar for autonomous driving,” arXiv preprint arXiv:2310.07602, 2023
2023 arXiv
-
[18]
Coloradar: The direct 3D millimeter wave radar dataset,
A. Kramer, K. Harlow, C. Williams, and C. Heckman, “Coloradar: The direct 3D millimeter wave radar dataset,” Int. J. Robot. Res., vol. 41, no. 4, pp. 351–360, 2022, SAGE Publications Sage UK: London, England
2022
-
[19]
DREAM-PCD: Deep reconstruction and enhancement of mmwave radar pointcloud,
R. Geng, Y. Li, D. Zhang, J. Wu, Y. Gao, Y. Hu, and Y. Chen, “DREAM-PCD: Deep reconstruction and enhancement of mmwave radar pointcloud,” IEEE Trans. Image Process., 2024, IEEE
2024
-
[20]
Milipoint: A point cloud dataset for mmwave radar,
H. Cui, S. Zhong, J. Wu, Z. Shen, N. Dahnoun, and Y. Zhao, “Milipoint: A point cloud dataset for mmwave radar,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 62713–62726, 2023
2023
-
[21]
Radical: A synchronized FMCW radar, depth, IMU and RGB camera data dataset with low-level FMCW radar signals,
T.-Y. Lim, S. A. Markowitz, and M. N. Do, “Radical: A synchronized FMCW radar, depth, IMU and RGB camera data dataset with low-level FMCW radar signals,” IEEE J. Sel. Topics Signal Process., vol. 15, no. 4, pp. 941–953, 2021, IEEE
2021
-
[22]
A flexible new technique for camera calibration,
Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330–1334, 2002, IEEE
2002
-
[23]
Depth Anything V2,
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth Anything V2,” Adv. Neural Inf. Process. Syst., vol. 37, pp. 21875–21911, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.