REVIEW 4 major objections 6 minor 31 references
Lidar Variability: A Novel Dataset and Comparative Study of Solid-State and Spinning Lidars
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that a low-cost dome-shaped solid-state lidar, the Livox Mid-360, consistently yields the most accurate and stable lidar odometry and point-cloud registration across SLAM and ICP methods, and it introduces a dataset that…
desk verdict A genuinely new multi-lidar dataset configuration, but the central claim about Mid-360's consistent superiority needs stronger statistical, synchronization, and uncertainty evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the new dataset itself: simultaneous, time-synchronized (PTP) point clouds from three lidar types mounted rigidly on a mobile quadruped robot, with motion-capture ground truth indoors and GNSS-RTK ground truth outdoors. What carries the argument is the controlled single-platform configuration, which removes platform and synchronization differences and lets APE differences be attributed to the sensors and the registration methods. On top of that, the standardized evaluation protocol—same SLAM pipelines, same ICP variants, fixed range constraints per environment, and APE as the common metric—is what makes the Mid-360's consistency a measurable, comparative result rather than a single anecdote.
What would settle it
Run the same SLAM and ICP pipelines on the released sequences with an independent, higher-accuracy ground-truth reference (for example, a survey-grade total station or recalibrated motion capture) and recompute the APE rankings: if the Livox Mid-360 is no longer consistently the lowest-error sensor across most methods, or if the ordering shifts when time synchronization offsets are deliberately perturbed, then the paper's central claim would be overturned.
Extended reading notes
Core claim
The paper's central claim is that the Livox Mid-360, a dome-shaped solid-state lidar with a 360° horizontal field of view and roughly 25° vertical coverage, consistently produced the lowest mean Absolute Pose Error (APE) and the most stable results across both SLAM algorithms (FAST-LIO2, FASTER-LIO, S-FAST-LIO, GLIM, FAST-LIO-SAM) and ICP variants (point-to-point KISS-ICP, point-to-plane Open3D-GICP, hybrid GenZ-ICP) in indoor and outdoor tests. In indoor offices, the Mid-360 beat the Ouster OS0-128 and the Livox Avia on most algorithms; outdoors, it remained the most consistent, whereas the Avia, which has the longest range (450 m), won only under FASTER-LIO, and the Ouster, which has the highest resolution, never dominated. The authors attribute the Mid-360's advantage to its dome-shaped design, which provides a wide vertical field of view and dense scan coverage of the immediate surroundings, and they emphasize that this finding holds in IMU-free odometry, isolating the lidar's geometric contribution.
Load-bearing premise
The comparison assumes that the motion-capture and GNSS-RTK reference trajectories, plus the PTP time synchronization and extrinsic calibration, are accurate enough that the reported APE differences reflect lidar characteristics rather than time offsets or calibration errors, and that two office sequences and one outdoor road are representative enough to support general conclusions.
Editorial extensions
If this is right
- A low-cost dome-shaped solid-state lidar such as the Livox Mid-360 can serve as a primary odometry sensor in GNSS-denied indoor environments, matching or outperforming spinning sensors that cost several times more.
- In IMU-free odometry, scan geometry—vertical field of view and coverage of nearby surfaces—can matter more for registration accuracy than maximum range or nominal resolution.
- Narrow-FoV solid-state lidars like the Avia retain value for long-range outdoor SLAM, but their poor local registration means a multi-lidar system pairing them with a dome sensor would exploit both strengths.
- The dataset offers a common platform, ground truth, and evaluation protocol that future lidar odometry and registration work can benchmark against, closing the gap left by datasets that exclude dome-shaped sensors.
Reading between the lines
- If the consistency result generalizes, integrators of indoor service robots could replace high-end spinning lidars with dome-shaped solid-state units, cutting sensor cost sharply without sacrificing odometry accuracy.
- Because the environment set is small (two offices, one road), a natural next experiment is to add corridors, vegetation-heavy areas, and weather variation to test whether the Mid-360's geometric advantage persists across scene types.
- A direct corollary of the wide-vertical-FoV explanation is that tilting the spinning Ouster to increase vertical coverage should shrink its APE gap to the Mid-360, which can be checked on the released sequences.
- The benchmark strips out IMU assistance to isolate sensor geometry; fusing the same data with IMUs in the SLAM pipelines would reveal how much of the dome lidar's advantage survives tight sensor coupling.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a new multi-lidar dataset collected on a Unitree B1 platform carrying a Livox Avia, a Livox Mid-360, and an Ouster OS0-128, with MoCap ground truth indoors and GNSS-RTK ground truth outdoors. It benchmarks five lidar-inertial SLAM algorithms and three ICP variants on two indoor sequences and one outdoor road sequence, reporting mean and standard deviation of Absolute Pose Error (APE). The central scientific claim is that the Livox Mid-360, a low-cost dome-shaped solid-state lidar, consistently achieves the highest accuracy and stability across both SLAM and ICP methods, and that the dataset is the first to integrate dome-shaped, solid-state, and spinning lidars on a single ground platform.
Significance. The dataset concept addresses a real gap: no existing public dataset combines a dome-shaped Livox Mid-360 with a limited-FOV solid-state lidar and a high-end spinning lidar on the same platform, and the comparison of ICP variants in an IMU-free setting is potentially useful for practitioners. If the dataset is released and the reference trajectories and time synchronization are validated, the work could provide a solid reference for heterogeneous lidar benchmarking and for selecting low-cost sensors for odometry. The paper also gives credit to related work (TIERS, GEODE, CTE-MLO) and is careful to distinguish its contribution from prior datasets. However, the headline comparative claim currently rests on very small APE differences computed from single runs, with no repeated trials and no uncertainty analysis for the ground-truth references, so the significance is conditional on additional validation.
major comments (4)
- [Section III.C and III.F, Table III] The reference trajectories are load-bearing for the central comparison, but their uncertainty is not quantified. Section III.C states that MoCap and Xsens GNSS-RTK provide ground truth, and Section III.F states that APE is computed against these references, yet no accuracy numbers, no per-sensor time-synchronization validation, and no trajectory-alignment protocol are given. This matters because the claimed indoor margins are extremely small: in Table III, FAST_LIO2 on IndoorOffice1 gives Mid-360 0.0451 ± 0.0150 m versus Ouster 0.0446 ± 0.0298 m, a difference of 0.5 mm, and several other entries differ by less than 1 cm. A constant synchronization offset of a few milliseconds at walking speed shifts the reference by centimeters, which is the same magnitude as the reported differences. The paper should provide a quantitative accuracy assessment of both reference systems, an explicit time-sync validation per sensor (the heterogeneous timestamps described in Section IV.A make this non-trivial), and a statistical comparison over repeated runs.
- [Section IV.A and Tables III–IV] The empirical basis for the 'consistently' claim is too thin. Only two indoor sequences and one outdoor sequence were collected, with no repeated trials; the reported standard deviation is computed over poses along a single run, not over independent runs. Consequently, the claim that Mid-360 consistently outperforms the other sensors is not statistically supported. The authors should add multiple trials per condition (or at least per trajectory) and report distributions across runs, or perform significance tests such as paired comparisons across trajectories. Without this, the rankings in Tables III and IV cannot be distinguished from run-to-run variability, especially where differences are sub-centimeter.
- [Section III.D and IV.A, Table II] The SLAM benchmark conflates lidar geometry with IMU quality. Section III.D benchmarks FAST-LIO2, FASTER-LIO, S-FAST-LIO, GLIM, and FAST-LIO-SAM using each sensor's built-in IMU, and Section IV.A states that the Livox sensors publish IMU data at 200 Hz while the Ouster publishes at 100 Hz, with different IMU models listed in Table II (IAM-20680HT, BMI088, ICM40609). Because these are lidar-inertial odometry systems, differences in APE can be caused by IMU quality or rate rather than by lidar scanning pattern. The abstract highlights performance 'particularly without an IMU in odometry,' but that condition is only realized in the ICP evaluation, not in the SLAM benchmark. The authors should either run IMU-free variants of the SLAM pipelines or clearly separate the lidar-geometry effect from the IMU effect in the analysis.
- [Section IV.A (dataset availability)] The dataset is a primary contribution, but it is not yet available: Section IV.A says it 'will appear online soon.' Since the paper's novelty and reproducibility depend on the raw bags, extrinsic calibration, PTP configuration, and exact preprocessing parameters, the comparative results cannot currently be verified. The authors should provide a public release or, at minimum, a detailed data card with download links, sensor calibration files, and the exact tuning parameters used for each SLAM and ICP method, before the benchmark claims can be fully assessed.
minor comments (6)
- [Throughout] There are several typos and formatting issues: 'Universtiy' in the author affiliation, 'limitted FoV' in the conclusion, 'ourdoor' in the conclusion, 'UA V' in the abstract and Figure 1, and the reference to 'Section VI presents the experimental results' while the actual sections are IV and V. These should be corrected.
- [Table I] The row for 'Ouster (2020)' lists 'OSDome' but does not clearly indicate whether the dome-shaped column is checked; also, the note about the CTE-MLO dataset should be integrated into the table caption or main text for readability.
- [Section III.F] The APE definition is incomplete: the paper should specify whether the error is computed over full SE(3) poses or only positions, and state the trajectory alignment method (e.g., Umeyama alignment with or without scale) and the timestamp interpolation used when comparing to the ground truth.
- [Section IV.A] For the Livox Avia point cloud, the listed fields do not include a per-point timestamp, unlike the Mid-360 and Ouster entries; this asymmetry is relevant to the synchronization discussion and should be documented explicitly.
- [Section III.E and Table IV] The ICP hyperparameters, such as the 20 cm indoor and 60 m outdoor correspondence ranges, are presented as fixed choices without a sensitivity analysis. Since ICP results are known to depend on these thresholds and the three lidars have very different point densities and FoVs, a supplementary experiment varying the thresholds would strengthen the comparison.
- [Figure 5] The grouped boxplots lack a clear description of what quantity is plotted (per-pose APE across all times? per-script errors?) and how many samples each box contains; the caption should state this and define the whiskers and outliers.
Circularity Check
No significant circularity: the benchmark conclusions are measured against external MoCap and GNSS-RTK references, not derived from the paper's own fitted parameters.
full rationale
This paper is an empirical dataset-and-benchmark study rather than a derivation. The central claims about Livox Mid-360 accuracy are supported by Absolute Pose Error (APE) measurements computed with the EVO package against ground truth from a motion capture system (indoor) and an Xsens GNSS-RTK unit (outdoor). These references are external to the lidar trajectories being evaluated, so the ranking is not forced by construction. The paper also evaluates public SLAM and ICP pipelines with stated, fixed preprocessing constraints; no parameter is fitted to a subset of data and then relabeled as a prediction. The self-citations (TIERS dataset in [17] and related lidar-image works) appear in the related-work and motivation sections and are not the evidence for the headline performance comparison. Concerns about unquantified reference uncertainty or time-synchronization artifacts are validity threats for the empirical conclusions, but they do not constitute circular reasoning in the sense of a claim reducing to its own inputs. Consequently, no specific circular step can be quoted, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Indoor ICP correspondence distance =
20 cm
- Outdoor ICP range constraint =
60 m
- Indoor SLAM max range =
60 m
assumptions (4)
- domain assumption Ground truth from MoCap (indoor) and GNSS-RTK (outdoor) is accurate enough to compute meaningful APE
- domain assumption PTP timestamp synchronization and extrinsic calibration among sensors are correct
- domain assumption The environment and trajectories are representative of indoor and outdoor lidar operation
- domain assumption The chosen ICP implementations faithfully represent point-to-point, point-to-plane, and hybrid categories
Cite this review
Pith. "Pith review of Lidar Variability: A Novel Dataset and Comparative Study of Solid-State and Spinning Lidars." pith.science (2026). https://pith.science/paper/V45VB4U5
@misc{pith2026250704321,
author = {Pith},
title = {Pith review of: Lidar Variability: A Novel Dataset and Comparative Study of Solid-State and Spinning Lidars},
year = {2026},
howpublished = {\url{https://pith.science/paper/V45VB4U5}},
note = {Machine review of arXiv:2507.04321}
}
read the original abstract
Lidar technology has been widely employed across various applications, such as robot localization in GNSS-denied environments and 3D reconstruction. Recent advancements have introduced different lidar types, including cost-effective solid-state lidars such as the Livox Avia and Mid-360. The Mid-360, with its dome-like design, is increasingly used in portable mapping and unmanned aerial vehicle (UAV) applications due to its low cost, compact size, and reliable performance. However, the lack of datasets that include dome-shaped lidars, such as the Mid-360, alongside other solid-state and spinning lidars significantly hinders the comparative evaluation of novel approaches across platforms. Additionally, performance differences between low-cost solid-state and high-end spinning lidars (e.g., Ouster OS series) remain insufficiently examined, particularly without an Inertial Measurement Unit (IMU) in odometry. To address this gap, we introduce a novel dataset comprising data from multiple lidar types, including the low-cost Livox Avia and the dome-shaped Mid-360, as well as high-end spinning lidars such as the Ouster series. Notably, to the best of our knowledge, no existing dataset comprehensively includes dome-shaped lidars such as Mid-360 alongside both other solid-state and spinning lidars. In addition to the dataset, we provide a benchmark evaluation of state-of-the-art SLAM algorithms applied to this diverse sensor data. Furthermore, we present a quantitative analysis of point cloud registration techniques, specifically point-to-point, point-to-plane, and hybrid methods, using indoor and outdoor data collected from the included lidar systems. The outcomes of this study establish a foundational reference for future research in SLAM and 3D reconstruction across heterogeneous lidar platforms.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Event-based sensor fusion and application on odometry: A survey
Jiaqiang Zhang, Xianjia Yu, Ha Sier, Haizhou Zhang, and Tomi West- erlund. Event-based sensor fusion and application on odometry: A survey. In 2025 IEEE 6th International Conference on Image Processing, Applications and Systems (IPAS) , pages 1–6. IEEE, 2025. (a) Avia (450 m range) (b) Mid-360 (100 m range) (c) Ouster OS0-128 (150 m range) Fig. 6: Outdoor...
work page 2025
-
[3]
Deep learning for lidar- only and lidar-fusion 3d perception: A survey
Danni Wu, Zichen Liang, and Guang Chen. Deep learning for lidar- only and lidar-fusion 3d perception: A survey. Intelligence & Robotics , 2(2):105–129, 2022
work page 2022
-
[4]
Point-lio: Robust high-bandwidth light detection and ranging inertial odometry
Dongjiao He, Wei Xu, Nan Chen, Fanze Kong, Chongjian Yuan, and Fu Zhang. Point-lio: Robust high-bandwidth light detection and ranging inertial odometry. Advanced Intelligent Systems , 5(7):2200459, 2023
work page 2023
-
[5]
Jiarong Lin and Fu Zhang. Loam livox: A fast, robust, high-precision lidar odometry and mapping package for lidars of small fov. In 2020 IEEE international conference on robotics and automation (ICRA) , pages 3126–3131. IEEE, 2020
work page 2020
-
[6]
Stream-based ground segmentation for real- time lidar point cloud processing on fpga, 2024
Xiao Zhang, Zhanhong Huang, Garcia Gonzalez Antony, Witek Jachim- czyk, and Xinming Huang. Stream-based ground segmentation for real- time lidar point cloud processing on fpga, 2024
work page 2024
-
[7]
Analysis of Deep Learning-Based Colorization and Super-Resolution Techniques for Lidar Imagery
Sier Ha, Honghao Du, Xianjia Yu, Jian Song, and Tomi Westerlund. Enhancing the reliability of lidar point cloud sampling: A colorization and super-resolution approach based on lidar-generated images. arXiv preprint arXiv:2409.11532, 2024
work page Pith review arXiv 2024
-
[8]
Haizhou Zhang, Xianjia Yu, Sier Ha, and Tomi Westerlund. Lidar- generated images derived keypoints assisted point cloud registration scheme in odometry estimation. Remote Sensing, 15(20):5074, 2023
work page 2023
-
[9]
Uav tracking with lidar as a camera sensor in gnss-denied environments
Ha Sier, Xianjia Yu, Iacopo Catalano, Jorge Pena Queralta, Zhuo Zou, and Tomi Westerlund. Uav tracking with lidar as a camera sensor in gnss-denied environments. In 2023 International Conference on Localization and GNSS (ICL-GNSS) , pages 1–7. IEEE, 2023
work page 2023
Show all 31 references
-
[10]
General-purpose deep learning detection and segmentation models for images from a lidar-based camera sensor
Xianjia Yu, Sahar Salimpour, Jorge Pena Queralta, and Tomi Westerlund. General-purpose deep learning detection and segmentation models for images from a lidar-based camera sensor. Sensors, 23(6):2936, 2023
2023
-
[11]
Solid-state-lidar- inertial-visual odometry and mapping via quadratic motion model and reflectivity information
Tao Yin, Jingzheng Yao, Yan Lu, and Chunrui Na. Solid-state-lidar- inertial-visual odometry and mapping via quadratic motion model and reflectivity information. Electronics, 12(17), 2023
2023
-
[12]
The affordable diy mandeye lidar system for surveying caves, and how to convert 3d clouds into traditional cave ground plans and extended profiles
Loris Redovnikovi ´c, Antun Jakopec, Janusz B˛ edkowski, and Jurica Jageti´c. The affordable diy mandeye lidar system for surveying caves, and how to convert 3d clouds into traditional cave ground plans and extended profiles. International Journal of Speleology , 53(3):7, 2025
2025
-
[13]
Vision meets robotics: The kitti dataset
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR) , 2013
2013
-
[14]
1 Year, 1000km: The Oxford RobotCar Dataset
Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman. 1 Year, 1000km: The Oxford RobotCar Dataset. The International Journal of Robotics Research (IJRR) , 36(1):3–15, 2017
2017
-
[15]
The apolloscape open dataset for autonomous driving and its application
Peng Wang, Xinyu Huang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE transactions on pattern analysis and machine intelligence, 2019
2019
-
[16]
Ouster dataset, 2020
Ouster_copyright. Ouster dataset, 2020
2020
-
[17]
Multi-modal lidar dataset for benchmarking general-purpose localization and mapping algorithms
Qingqing Li, Xianjia Yu, Jorge Peña Queralta, and Tomi Westerlund. Multi-modal lidar dataset for benchmarking general-purpose localization and mapping algorithms. arXiv preprint arXiv:2203.03454 , 2022
2022 arXiv
-
[18]
Livox dataset, 2020
Livox_copyright. Livox dataset, 2020
2020
-
[19]
Heterogeneous lidar dataset for benchmarking robust localization in diverse degenerate scenarios, 2024
Zhiqiang Chen, Yuhua Qi, Dapeng Feng, Xuebin Zhuang, Hongbo Chen, Xiangcheng Hu, Jin Wu, Kelin Peng, and Peng Lu. Heterogeneous lidar dataset for benchmarking robust localization in diverse degenerate scenarios, 2024
2024
-
[20]
Cte-mlo: Continuous-time and efficient multi-lidar odometry with localizability- aware point cloud sampling, 2025
Hongming Shen, Zhenyu Wu, Yulin Hui, Wei Wang, Qiyang Lyu, Tianchen Deng, Yeqing Zhu, Bailing Tian, and Danwei Wang. Cte-mlo: Continuous-time and efficient multi-lidar odometry with localizability- aware point cloud sampling, 2025
2025
-
[21]
Besl and Neil D
P.J. Besl and Neil D. McKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 14(2):239–256, 1992
1992
-
[22]
Chen and G
Y . Chen and G. Medioni. Object modeling by registration of multiple range images. In Proceedings. 1991 IEEE International Conference on Robotics and Automation , pages 2724–2729 vol.3, 1991
1991
-
[23]
Iterative point matching for registration of free-form curves
E Recherche, Et Automatique, Sophia Antipolis, and Zhengyou Zhang. Iterative point matching for registration of free-form curves. Int. J. Comput. Vision, 13, 07 1992
1992
-
[24]
Fast-lio2: Fast direct lidar-inertial odometry, 2021
Wei Xu, Yixi Cai, Dongjiao He, Jiarong Lin, and Fu Zhang. Fast-lio2: Fast direct lidar-inertial odometry, 2021
2021
-
[25]
Faster-lio: Lightweight tightly coupled lidar-inertial odometry using parallel sparse incremental voxels
Chunge Bai, Tao Xiao, Yajie Chen, Haoqian Wang, Fang Zhang, and Xiang Gao. Faster-lio: Lightweight tightly coupled lidar-inertial odometry using parallel sparse incremental voxels. IEEE Robotics and Automation Letters, 7(2):4861–4868, 2022
2022
-
[26]
A simplified implementation of fast_lio, 2023
Zlwang7. A simplified implementation of fast_lio, 2023
2023
-
[27]
Glim: 3d range-inertial localization and mapping with gpu-accelerated scan matching factors
Kenji Koide, Masashi Yokozuka, Shuji Oishi, and Atsuhiko Banno. Glim: 3d range-inertial localization and mapping with gpu-accelerated scan matching factors. Robotics and Autonomous Systems , 179:104750, 2024
2024
-
[28]
A slam implementation combining fast-lio2 with pose graph optimization and loop closing based on lio-sam paper, 2023
Zlwang7. A slam implementation combining fast-lio2 with pose graph optimization and loop closing based on lio-sam paper, 2023
2023
-
[29]
Kiss-icp: In defense of point-to-point icp – simple, accurate, and robust registration if done the right way.IEEE Robotics and Automation Letters , 8(2):1029–1036, February 2023
Ignacio Vizzo, Tiziano Guadagnino, Benedikt Mersch, Louis Wiesmann, Jens Behley, and Cyrill Stachniss. Kiss-icp: In defense of point-to-point icp – simple, accurate, and robust registration if done the right way.IEEE Robotics and Automation Letters , 8(2):1029–1036, February 2023
2023
-
[30]
Open3D: A modern library for 3D data processing
Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018
2018 arXiv
-
[31]
Open3d slam: Point cloud based mapping and localization for education
Edo Jelavic, Julian Nubert, and Marco Hutter. Open3d slam: Point cloud based mapping and localization for education. In Robotic Perception and Mapping: Emerging Techniques, ICRA 2022 Workshop , page 24. ETH Zurich, Robotic Systems Lab, 2022
2022
-
[32]
Genz-icp: Generalizable and degeneracy-robust lidar odometry using an adaptive weighting.IEEE Robotics and Automation Letters , 10(1):152–159, January 2025
Daehan Lee, Hyungtae Lim, and Soohee Han. Genz-icp: Generalizable and degeneracy-robust lidar odometry using an adaptive weighting.IEEE Robotics and Automation Letters , 10(1):152–159, January 2025
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.