Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Performance Assessment of Lidar Odometry Frameworks: A Case Study at the Australian Botanic Garden Mount Annan

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A looped drive through the Australian Botanic Garden Mount Annan shows COIN-LIO keeping the horizontal trajectory nearly drift-free while LIO-SAM drifts by metres in the final quarter of the path.

desk verdict Useful first COIN-LIO vs LIO-SAM comparison in a vegetation-rich garden, but the drift and segment-error numbers contradict each other, so the quantitative claims need a rework before citation. read the letter →

arxiv 2411.16931 v2 pith:DLRWD4OM submitted 2024-11-25 cs.RO

Mohamed Mourad Ouazghire , Julie Stephany Berrio , Mao Shan , Stewart Worrall This is my paper · ORCID
classification cs.RO
keywords LiDAR-inertialodometrySLAMevaluationnaturalenvironmentdatasetCOIN-LIOLIO-SAMloopclosuretrajectorydriftyawerror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish how two state-of-the-art LiDAR-inertial odometry systems, COIN-LIO and LIO-SAM, hold up in a natural, unstructured environment rather than the urban or indoor settings where such systems are usually tested. Using a 140-second, roughly 890-metre looped drive through the Australian Botanic Garden Mount Annan, recorded with a 128-beam LiDAR plus GPS and IMU, the authors compare trajectory estimates against GNSS ground truth in the horizontal and vertical planes. They find that both systems track elevation well, but COIN-LIO keeps the horizontal path close to ground truth while LIO-SAM accumulates noticeable x-y drift after about 700 metres and larger yaw errors over longer distances. A sympathetic reader would care because it supplies one of the few head-to-head evaluations in dense vegetation and points to loop-closure robustness as the deciding factor for long natural-environment runs.

What carries the argument

The comparison is carried by a looped natural-environment dataset and two error metrics. The dataset, a 140-second, 9.7 GB recording on a roughly 890-metre path with open grass, paved paths, dense vegetation, and a moving car, is the test object that supplies the loop closures and geometrically challenging stretches. The metrics are the Absolute Trajectory Error, which aligns the whole estimated path to ground truth and averages positional deviation, and the Relative Error, which aligns short sub-trajectory pairs and measures local translation and yaw drift over increasing distances. On top of those, the frameworks themselves are the machinery: COIN-LIO augments LiDAR intensity into image patches and fuses photometric error with point-to-plane registration in an iterated extended Kalman filter, while LIO-SAM couples LiDAR and IMU through a factor graph with scan-to-map matching and loop-closure detection. The paper uses these tools to separate vertical performance, where the IMU dominates, from horizontal performance, where loop-closure quality and scan registration decide the outcome.

What would settle it

Survey several fixed points along the vegetated final quarter with a total station or an RTK base station, then compare both estimated trajectories to those surveyed checkpoints. If LIO-SAM's x-y drift tracks the GNSS solution's own multipath error rather than the surveyed path, the paper's central comparison collapses; if it tracks away from the surveyed path, the conclusion stands.

Watch

Extended reading notes

Core claim

On its own terms, the paper reports that COIN-LIO outperforms LIO-SAM in maintaining trajectory fidelity in the horizontal plane. In plots, COIN-LIO's estimated loop stays near the ground truth, with small deviations in densely vegetated stretches that are corrected at loop closure, while LIO-SAM aligns well early on and then diverges clearly in x and y during the final quarter of the loop. Position-drift curves put COIN-LIO within about plus or minus one metre in x and y and 0.8 metres in z, versus LIO-SAM exceeding four metres of drift in x and y by the end while remaining within about 1.5 metres in z. Relative translation and yaw errors grow with distance for both frameworks, but COIN-LIO's means are lower (about 3.5 percent translation error and 0.5 degrees yaw versus 4.5 percent and 0.7 degrees for LIO-SAM), with the differences statistically significant at the distances tested. The authors attribute the gap to COIN-LIO's continuous-time, intensity-augmented registration and effective loop-closure correction, and suggest LIO-SAM's factor-graph scan-to-map matching is less robust to feature-sparse or repetitive vegetation at this scale.

Load-bearing premise

The whole comparison rests on assuming the GNSS receiver used as ground truth stays centimetre-accurate even under the garden's dense tree cover; if the reference drifts there, the reported horizontal advantage could be an artifact of which system the reference agrees with.

Editorial extensions

If this is right

  • For long, looped routes in vegetation-dense parks or forests, COIN-LIO-style intensity-augmented registration is the better default if heading and horizontal position matter.
  • LIO-SAM can still be used in such environments, but its loop-closure parameters and detection range would need retuning before trusting the final segment of a long run.
  • Vertical accuracy is the less discriminating axis: both frameworks stayed within roughly one to one-and-a-half metres in z, so elevation-only tasks would not reveal the difference.
  • Computational cost does not decide the choice here, since the two frameworks ran in nearly identical time on the same hardware.
  • The new dataset gives the community a compact, repeatable 890-metre benchmark for natural-environment LiDAR-inertial odometry, even though the data are not yet public.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test would be to re-run LIO-SAM with a larger loop-closure search radius and finer voxel filtering; if its final-quarter drift shrinks, the reported gap is partly a configuration effect rather than an architectural limit.
  • Because the GNSS ground truth was collected under tree cover without independent validation, the absolute drift numbers should be treated as comparative rather than metrological until surveyed checkpoints confirm the reference trajectory.
  • The intensity-image mechanism that helps COIN-LIO in this garden may transfer to other feature-poor or repetitively textured scenes such as orchards, tunnels, or snow-covered fields, since the same photometric cues do not depend on geometric structure.
  • One could quantify how much of the horizontal advantage comes from loop closure versus the continuous-time front end by disabling loop closure in COIN-LIO and measuring the drift curve again.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a comparative evaluation of two LiDAR-inertial odometry frameworks, COIN-LIO and LIO-SAM, on a newly recorded dataset from the Australian Botanic Garden Mount Annan. The dataset includes 128-beam LiDAR, IMU, and GNSS ground truth over an approximately 890 m looped trajectory with open, paved, and densely vegetated sections. The authors analyze horizontal and vertical trajectory estimates, position drift, relative translation and yaw errors at different traveled distances, and report a paired t-test as evidence that COIN-LIO outperforms LIO-SAM. The central claim is that COIN-LIO maintains superior accuracy in the horizontal plane and over longer trajectories, while both frameworks perform comparably in the vertical plane.

Significance. If the findings are valid, the paper would provide useful comparative evidence on two state-of-the-art LIO systems in an unstructured natural environment, a setting underrepresented in existing benchmarks. The dataset itself (vehicle-mounted, 128-beam LiDAR, garden environment) complements public datasets such as WildPlaces and MulRan. The trajectory visualizations and qualitative observations about LIO-SAM's late-trajectory divergence are informative. However, the quantitative basis for the central claim is undermined by an internal inconsistency between the drift plots and the reported box plots, and the statistical analysis is reported without sufficient detail. The dataset is proprietary and only available on request, which limits reproducibility. The paper's value is as a case study rather than a general benchmark, and it needs substantial revision before its claims can be accepted.

major comments (4)
  1. [§5.3, §5.4, §5.5] The drift plots and the box plots are numerically irreconcilable. Section 5.3 states that COIN-LIO's x/y drift remained within ±1 m and z drift within ±0.8 m (Fig. 9), and that LIO-SAM's x/y drift exceeded -4 m near the end (Fig. 10). Yet Sections 5.4 and 5.5 report for both frameworks translation errors around 5 m at 95.22 m and increasing to 12–15 m at 476.13 m (Figs. 11b and 12a). If COIN-LIO's global position error is at most about 1.3 m in any axis, a relative translation error of 15 m over a 476 m segment cannot be reconciled with standard definitions of ATE or RPE unless the two sets of plots measure different quantities, use different trajectory alignment procedures, or use different distance conventions. The manuscript does not define which quantity each figure reports. This inconsistency must be resolved before the central claim that COIN-LIO demonstrates superior horizontal accuracy can be considered supported.
  2. [§5.6] The statistical test reporting is incomplete. The paired t-test is summarized only by mean values (3.5% vs. 4.5% relative translation error; 0.5° vs. 0.7° yaw error) and p < 0.05 thresholds. No information is given about the sample size (presumably the three repeated runs), the pairing structure, the distribution of differences, effect sizes, or whether multiple comparisons across distances were accounted for. The claim that COIN-LIO 'consistently outperformed' LIO-SAM at all distances is not accompanied by distance-wise statistics. These details are necessary to evaluate whether the observed differences are statistically meaningful.
  3. [§3.1] The ground-truth accuracy claim is not validated under canopy. Section 3.1 says the NovAtel PwrPak7D-E1 GNSS receiver ensures 'centimeter-level accuracy,' but the data were collected in a garden with densely vegetated sections where multipath and occlusion can degrade GNSS accuracy to well below that level. If the ground-truth trajectory itself contains horizontal errors, the relative ranking of the frameworks could be an artifact. The authors should provide quantitative validation of the ground truth in the vegetated segments—for example, by checking residuals at the known loop-closing point or comparing with an independent estimator—and should temper the accuracy claim accordingly.
  4. [§3.1, §6.4] The configuration of both frameworks is described only qualitatively. The COIN-LIO column-shift calibration and the LIO-SAM loop closure, voxel grid, and scan matching parameters are said to be 'adjusted' or 'set to suitable values,' but no concrete parameter values or sensitivity analysis are provided. Because both algorithms are sensitive to such settings, the absence of this information makes it unclear whether the observed performance difference is intrinsic to the methods or an artifact of the chosen configuration. Please include the exact parameter values and, if possible, an ablation or sensitivity study.
minor comments (5)
  1. [Figure 8 captions and §5.2] Figure 8 subfigures are captioned as 'x-y Plane' and the parenthetical says 'widened vertically for more zoom,' but the plots and text refer to the x-z plane for vertical trajectory analysis. The captions should be corrected to 'x-z Plane.'
  2. [§2.2 and §4.4] The RPG Trajectory Evaluation Toolbox is cited as [8] in §2.2, but §4.4 refers to 'RPG-Trajectory [11]' where reference 11 is the KITTI benchmark paper. The citation numbering is inconsistent and should be corrected throughout.
  3. [§2.3] SegMatch is cited as [9], but reference 9 is the EVO package; the citation for SegMatch should be replaced with the appropriate place-recognition reference.
  4. [§2.2] The sentence beginning 'The RPG Trajectory Evaluation Toolbox... offers a comprehensive framework for trajectory analysis.' is immediately followed by 'Provide detailed metrics...' with an incorrect verb form; change to 'It provides detailed metrics...'.
  5. [Throughout] There are typographical artifacts such as 'F AST-LIO2' and 'T rajectory' with irregular spacing; these should be cleaned up in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the study is an external benchmark of two third-party LiDAR-inertial odometry frameworks against GNSS ground truth, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper does not derive any predicted quantity from its own model. It executes two published frameworks, COIN-LIO and LIO-SAM, on a newly collected dataset and compares the resulting trajectories against NovAtel GNSS ground truth using standard ATE/RPE-style metrics from the RPG trajectory evaluation toolbox and EVO. The central claim that COIN-LIO outperforms LIO-SAM is a measurement, not a derivation. There is no fitted parameter that is later called a prediction, no self-defined quantity that is then 'confirmed', and no uniqueness theorem or prior author work invoked to force the conclusion. The COIN-LIO and LIO-SAM frameworks are cited from external authors (Pfreundschuh et al. and Shan et al.), and the present authors have no overlap with those works. The three repeated runs and paired t-tests are statistical comparisons of observed errors, not circular constructions. Concerns about GNSS accuracy under vegetation, the internal consistency of drift plots versus translation error box plots, and the absence of public dataset access are correctness and evidence-quality issues, not circularity. The paper's derivation chain is self-contained as an experimental evaluation, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the external GNSS ground truth, the fairness of framework parameter tuning, and the appropriateness of the statistical and alignment procedures. These are reasonable starting assumptions, but the paper does not provide enough evidence to verify them. No new physical entities or conserved quantities are introduced.

free parameters (3)
  • COIN-LIO image projection column shift = unspecified
    COIN-LIO requires a calibration step to correct image projection for the Ouster sensor; the paper does not report the calibrated shift value.
  • LIO-SAM loop closure detection parameters = unspecified
    The paper states parameters were adjusted for the larger scale trajectory ('The framework's loop closure detection was enabled, with parameters adjusted to accommodate the larger scale of the trajectory') but does not give values.
  • LIO-SAM filtering and scan matching parameters = unspecified
    The paper says voxel grid and scan matching parameters were set to 'suitable values for outdoor environments' without reporting them.
assumptions (4)
  • domain assumption GNSS ground truth is accurate to centimeter level even under dense vegetation
    Section 3.1 states the NovAtel receiver ensures centimeter-level accuracy, but no validation under canopy is provided. This is load-bearing because the comparison depends on the ground truth being reliable in the horizontal plane.
  • domain assumption Similarity transformation alignment for ATE does not mask systematic drift differences
    Section 4.3 describes aligning trajectories to ground truth; this is standard, but the choice of alignment can affect ATE values and the perceived drift.
  • domain assumption The paired t-test assumptions are satisfied
    Section 5.6 reports p<0.05 but does not describe the sample construction or verify normality and independence. The box plots show overlapping distributions, so the significance claim needs more support.
  • domain assumption The parameter settings for both frameworks are comparably fair
    The authors tuned LIO-SAM for the larger scale and calibrated COIN-LIO's projection, but the paper does not establish that the tuning effort is balanced. Uneven tuning could bias the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Assessment of Lidar Odometry Frameworks: A Case Study at the Australian Botanic Garden Mount Annan." pith.science (2026). https://pith.science/paper/DLRWD4OM

@misc{pith2026241116931,
  author       = {Pith},
  title        = {Pith review of: Performance Assessment of Lidar Odometry Frameworks: A Case Study at the Australian Botanic Garden Mount Annan},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLRWD4OM}},
  note         = {Machine review of arXiv:2411.16931}
}
read the original abstract

Autonomous vehicles are being tested in diverse environments worldwide. However, a notable gap exists in evaluating datasets representing natural, unstructured environments such as forests or gardens. To address this, we present a study on localisation at the Australian Botanic Garden Mount Annan. This area encompasses open grassy areas, paved pathways, and densely vegetated sections with trees and other objects. The dataset was recorded using a 128-beam LiDAR sensor and GPS and IMU readings to track the ego-vehicle. This paper evaluates the performance of two state-of-the-art LiDARinertial odometry frameworks, COIN-LIO and LIO-SAM, on this dataset. We analyse trajectory estimates in both horizontal and vertical dimensions and assess relative translation and yaw errors over varying distances. Our findings reveal that while both frameworks perform adequately in the vertical plane, COINLIO demonstrates superior accuracy in the horizontal plane, particularly over extended trajectories. In contrast, LIO-SAM shows increased drift and yaw errors over longer distances.

Figures

Figures reproduced from arXiv: 2411.16931 by the authors.

Figure 1
Figure 1. Australian Botanic Garden Mount Annan 3D [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Body frame’s image projection visualized [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 2
Figure 2. Comparison table of Lidar Inertial Odometry Methods [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Australian Botanic Garden’s Trajectory Loop [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Stereo camera shot’s point cloud equivalent [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Trajectory of COIN-LIO in the x-y Plane For LIO-SAM, the initial portion of the trajectory ( [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Trajectory of LIO-SAM in the x-y Plane [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 10
Figure 10. Figure 10: Position Drift/Distance of LIO-SAM 5.4 Translation and Yaw Error Analysis for LIO-SAM Translation and Yaw Errors Across Different Distances [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 8
Figure 8. Figure 8: Comparison of Trajectories in the x-y Plane [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Position Drift/Distance of COIN-LIO In contrast, LIO-SAM exhibited an increasing drift in the x and y axes after 700 meters. The drift exceeded -4 meters in both axes at the end of the trajectory. The z-axis drift remained within ±1.5 meters, indicating that vertical e…
Figure 11
Figure 11. Figure 11: Translation and Yaw Errors for LIO-SAM Overall Conclusion: Together, these subfigures demonstrate how the performance of LIO-SAM degrades over longer trajectories. Although the relative transla￾tion error stabilizes as a percentage of distance, the ab￾solute translati…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [9]

    Grupp, EVO: Python package for the evaluation of odometry and SLAM, 2017

    M. Grupp, EVO: Python package for the evaluation of odometry and SLAM, 2017. https://github.com/MichaelGrupp/evo

    work page 2017

  2. [1]

    Coin-lio: Complemen- tary intensity-augmented lidar inertial odometry

    P. Pfreundschuh, H. Oleynikova, C. Cadena, R. Siegwart and O. Andersson, “Coin-lio: Complemen- tary intensity-augmented lidar inertial odometry”, in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 1730–1737, IEEE, 2024

    work page 2024

  3. [2]

    Lio-sam: Tightly-coupled lidar in- ertial odometry via smoothing and mapping,

    T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and R. Daniela, “Lio-sam: Tightly-coupled lidar in- ertial odometry via smoothing and mapping,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5135–5142, IEEE, 2020

    work page 2020

  4. [3]

    ”Wild-places: A large- scale dataset for lidar place recognition in un- structured natural environments.” 2023 IEEE in- ternational conference on robotics and automation (ICRA)

    Knights, Joshua, et al. ”Wild-places: A large- scale dataset for lidar place recognition in un- structured natural environments.” 2023 IEEE in- ternational conference on robotics and automation (ICRA). IEEE, 2023

    work page 2023

  5. [4]

    G. Kim, Y. S. Park, Y. Cho, J. Jeong and A. Kim, ”MulRan: Multimodal Range Dataset for Urban Place Recognition,” 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), Paris, France, 2020, pp. 6246-6253, doi: 10.1109/ICRA40945.2020.9197298

    arXiv 2020

  6. [5]

    Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain,

    T. Shan and B. Englot, “Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain,” in 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 4758–4765, 2018

    work page 2018

  7. [6]

    Robust place recognition using an imaging lidar,

    T. Shan, B. Englot, F. Duarte, C. Ratti, and D. Rus, “Robust place recognition using an imaging lidar,” 2021

    work page 2021

  8. [7]

    Fast- lio2: Fast direct lidar-inertial odometry,

    W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “Fast- lio2: Fast direct lidar-inertial odometry,” 2021

    work page 2021

Show all 13 references
  1. [8]

    A tutorial on quan- titative trajectory evaluation for visual(-inertial) odometry,

    Z. Zhang and D. Scaramuzza, “A tutorial on quan- titative trajectory evaluation for visual(-inertial) odometry,” in IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS), 2018

    2018

  2. [10]

    A benchmark for the evaluation of rgb-d slam systems,

    J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Sys- tems, pp. 573–580, IEEE, 2012

    2012

  3. [11]

    Are we ready for autonomous driving? the kitti vision bench- mark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision bench- mark suite,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012

    2012

  4. [12]

    A method for registra- tion of 3-d shapes,

    P. J. Besl and N. D. McKay, “A method for registra- tion of 3-d shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239–256, 1992

    1992

  5. [13]

    The normal distributions transform: A new approach to laser scan matching,

    P. Biber and W. Straßer, “The normal distributions transform: A new approach to laser scan matching,” in Proceedings 2003 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS 2003), vol. 3, pp. 2743–2748, IEEE, 2003

    2003

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.