REVIEW 3 major objections 6 minor 35 references
Multi-LVI-SAM: A Robust LiDAR-Visual-Inertial Odometry for Multiple Fisheye Cameras
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Multi-LVI-SAM claims that projecting all fisheye views onto one normalized sphere, with a geometric compensation for the offset between each camera and the sphere center, yields more accurate and robust LiDAR-visual-inertial odometry than p
desk verdict Solid engineering, shaky theory: the extrinsic compensation is a far-field approximation, not the rigorous deduction claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Panoramic visual feature model: a sphere centered at S; each camera's pixel u_c maps to sphere point u_s = λ(R_i u_c + t_i), so all cameras share one reference frame and one set of feature constraints. Extrinsic compensation: for each frame, plane normals n_i = (S_i C_i) × (S_i u_i); the two planes intersect along direction m = n_1 × n_2; the sine-law depth correction λ_PP' = λ_P' sin α / sin(α+γ) moves the triangulated point along m. Together these let a multi-fisheye rig behave as one panoramic sensor inside the factor-graph optimization that fuses IMU pre-integration, visual residuals, LiDAR scan matching, and loop closure.
What would settle it
Mount two fisheye cameras with known offsets from a sphere center, place a structured-light or LiDAR-measured target at several distances (e.g., 0.3, 1, 3, 10 m), and compare compensated triangulation depths against the ground-truth depths. If the per-point error increases as the ratio of camera-to-sphere offset to target depth increases, the far-field approximation in Eq. 6 is falsified.
Extended reading notes
Core claim
Multi-LVI-SAM's central claim is that multi-camera visual information should be lifted onto a common normalized sphere, the panoramic model, and then treated as one feature set in the same tightly coupled factor-graph pipeline used by single-camera LVIO, rather than maintaining per-camera pose estimates. The paper shows that naive common-sphere triangulation is geometrically wrong when camera centers are offset from the sphere center: the true point P is not the plane-intersection point P' obtained from sphere-centered planes. Its extrinsic compensation computes the plane-intersection direction m from the two planes spanned by camera center, sphere center, and feature, then applies a depth c
Load-bearing premise
The load-bearing assumption is that the sine-law depth correction (Eq. 6) is geometrically exact; the paper omits the derivation, and the formula holds exactly only when the feature point is far from the camera relative to the camera-to-sphere offset. Close-range features could retain triangulation bias that the reported RMSE gains partly hide.
Editorial extensions
If this is right
- Multi-camera LVIO can be built by changing only the visual front end: one sphere, one feature map, one residual type, independent of how many cameras are mounted.
- Wide-FoV perception is affordable in real time: four cameras cost about 2.15 times the runtime of one camera, and the system keeps estimating pose when a single camera points at a textureless wall or is occluded.
- Where LiDAR degenerates, visual triangulation carries the depth signal, and the compensation step is what makes that triangulation trustworthy; removing it raises RMSE on every reported sequence, most sharply on Stairs.
- The framework inherits LiDAR-based metric depth and loop-closure verification, so scale remains observable even when individual visual or LiDAR constraints are weak.
Reading between the lines
- Beyond the paper: the same panoramic projection and compensation should transfer to any multi-camera rig, including mixed-FoV or stereo systems, and to visual-inertial odometry without LiDAR; a direct test would disable LiDAR depth association and see whether multi-view triangulation alone holds scale.
- The sine-law correction is a far-field approximation; at target depths comparable to the camera-to-sphere offset, residual bias should appear. A range-resolved test with near and far targets would show where the formula's benefit stops.
- The paper does not exploit overlapping fields of view; using the overlap as an additional constraint could refine camera-to-sphere extrinsics and feature depth in the regions where two cameras see the same point.
- The paper notes its point-to-line LiDAR matching is weaker under fast motion than point-to-plane matching; pairing the panoramic visual front end with a plane-based LiDAR backend is a natural combination not tested here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Multi-LVI-SAM, a tightly coupled LiDAR-visual-inertial odometry system with multiple fisheye cameras. Feature observations from all cameras are projected onto a spherical panoramic model to form a single representation, and an 'extrinsic compensation' step is introduced to correct triangulation bias caused by the offset between individual camera centers and the panoramic sphere center. The system builds on LVI-SAM/VINS-Mono/LIO-SAM and is evaluated on Newer College, M2DGR, and Hilti'2022 datasets against FAST-LIO2, LVI-SAM, FAST-LIVO2, and R2Live. Ablations compare single-camera, LIO-only, and with/without compensation configurations, and report runtime overhead.
Significance. The panoramic multi-camera model is a practically useful contribution: it avoids per-camera redundancy, widens the effective field of view, and the ablation indicates improved robustness in cases where individual cameras fail (e.g., M2DGR walk-01, Hilti'2022 sequences). The public-dataset evaluation and comparisons with strong open-source baselines are appropriate. However, the paper's most distinctive algorithmic component—the extrinsic compensation formula—is not rigorously derived as claimed and relies on an unstated far-field approximation. Because the 40.6% Stairs RMSE reduction is used as the main quantitative evidence for this component, the current version does not fully support the paper's central claim. With a corrected derivation and a depth-aware validation, the contribution would be significant.
major comments (3)
- [Section III-B.2, Eqs. (5)-(6)] Equation (6) is presented as a rigorous geometric deduction, but the stated derivation is incomplete and the formula is only an approximation. In triangle S1–P–P', side S1P' lies along S1u1 and side C1P lies along C1u1. The angle at S1 between S1P' and S1P is not equal to α (the angle between S1u1 and C1u1) unless S1P is parallel to C1P, i.e., in the far-field limit |C1P| >> |S1C1|. The law-of-sines step leading to λ_PP' = λ_P' sinα / sin(α+γ) therefore substitutes an approximate angle. The paper neither states the far-field assumption nor provides exact expressions. Since Section IV-C attributes a 40.6% RMSE improvement on the Stairs sequence to this compensation, the central quantitative claim is not yet supported. Please supply a complete derivation (or the exact correction obtained by intersecting the ray C1 + d·c with the line P' + λ·m), state the approximation explicitly, and quant
- [Section III-B, Eqs. (2) and (8)] The panoramic visual feature model is described only by the coordinate transformation in Eq. (2) and the plane normals in Eq. (8). The paper does not specify how observations enter the optimizer: is the visual residual defined on the unit sphere (e.g., tangent-plane reprojection), and how are the per-camera extrinsics R_i, t_i incorporated into the factor graph? Without these definitions, the claimed unification of multi-camera constraints and the effect of the compensation are not reproducible. Please state the exact residual, the optimization variables, and the relevant Jacobian structure, even if the system follows VINS-Mono.
- [Section IV-C, Table IV] The ablation evidence is partly confounded by the approximation issue. In the Stairs row, the compensation reduces RMSE from 0.759511 to 0.451100, but this sequence is exactly one where nearby features (stairs, walls) make the far-field assumption questionable. The paper should report, for this sequence, the distribution of feature depths entering triangulation, or provide an experiment in which features are stratified by depth, to confirm that the correction is not an accidental improvement. In addition, clarify how the 'w/o compensation' configuration is obtained while keeping all other system components identical.
minor comments (6)
- [Table II] In the Math-Hard row, 'Ours (w/ loop)' is reported as 0.88219; from Table IV the correct value appears to be 0.088219. This typo affects a central comparison table and should be fixed.
- [Table III] In the room-02 row, the FAST-LIO2 entry reads '0.314317l' with a stray 'l'. Please correct.
- [Section IV] The Hilti'2022 dataset appears only in the ablation study (Section IV-C) but is not introduced in the experimental setup at the beginning of Section IV. Include its sensor configuration and state why it is not included in the main comparison against external baselines.
- [Eq. (2)] Clarify whether u_c is a pixel coordinate after undistortion, a normalized bearing vector, or a point on the fisheye image plane. The equation omits camera intrinsics and distortion models, which are needed to reproduce the transformation.
- [Figures 3 and 4] The geometric quantities S1, C1, u1, α, and γ are hard to identify. Enlarge the figures and label these symbols explicitly, including the direction of the extrinsic offset.
- [Table IV] The header mixes 'camera0/camera1/camera2/camera3' with 'camera-left/camera-right/camera-midleft/camera-midright'. Make the mapping explicit for readability.
Circularity Check
No circularity: the extrinsic compensation is a geometric correction built from calibrated extrinsics, and all accuracy claims are evaluated on external public benchmarks.
full rationale
The paper's central derivation chain is self-contained and externally anchored. The panoramic visual feature model (Eq. 2) is a direct coordinate transformation of each fisheye camera onto a sphere using calibrated extrinsics; the extrinsic compensation (Eqs. 3-7) is constructed from camera-to-sphere offsets and observed bearing vectors, not from the pose errors it is meant to reduce. The ablation in Table IV simply turns the compensation on/off, and the reported 40.6% Stairs RMSE reduction is an empirical comparison on a public benchmark, not a fitted parameter relabeled as a prediction. Comparisons against FAST-LIO2, LVI-SAM, FAST-LIVO2, and R2Live use public datasets with external ground truth, so the claimed improvements are falsifiable outside the paper's own parameters. The only author-overlapping citation (Ref. [16]) appears in related work with the non-load-bearing remark that it 'remains prone to motion drift,' so it does not carry the derivation. The one legitimate concern is rigor, not circularity: Eq. (6) is announced as a 'rigorous geometric deduction' without derivation, and the stated formula appears to correspond to a far-field limit of the exact line-intersection solution; this is a soundness/approximation risk for close-range features, not a reduction of the result to its own inputs. No step defines a quantity in terms of the very quantity it is supposed to predict, and no fitted value is renamed as a prediction.
Assumptions & free parameters
free parameters (1)
- Depth blurring threshold =
not reported
assumptions (3)
- domain assumption Plane coplanarity: for each frame, the camera center, the panoramic sphere center, the spherical projection, the true point P, and the erroneous point P' all lie in one plane.
- ad hoc to paper The correction formula Eq. (6) assumes the angle at the sphere center between rays to P' and P equals the camera-ray angle alpha (far-field approximation).
- domain assumption LiDAR depth association via a 2D K-D tree on the sphere, with occlusion filtering by maximum depth spread, is reliable.
Cite this review
Pith. "Pith review of Multi-LVI-SAM: A Robust LiDAR-Visual-Inertial Odometry for Multiple Fisheye Cameras." pith.science (2026). https://pith.science/paper/CUIFJGIH
@misc{pith2026250905740,
author = {Pith},
title = {Pith review of: Multi-LVI-SAM: A Robust LiDAR-Visual-Inertial Odometry for Multiple Fisheye Cameras},
year = {2026},
howpublished = {\url{https://pith.science/paper/CUIFJGIH}},
note = {Machine review of arXiv:2509.05740}
}
read the original abstract
We propose a multi-camera LiDAR-visual-inertial odometry framework, Multi-LVI-SAM, which fuses data from multiple fisheye cameras, LiDAR and inertial sensors for highly accurate and robust state estimation. To enable efficient and consistent integration of visual information from multiple fisheye cameras, we introduce a panoramic visual feature model that unifies multi-camera observations into a single representation. The panoramic model serves as a global geometric optimization framework that consolidates multi-view constraints, enabling seamless loop closure and global pose optimization, while simplifying system design by avoiding redundant handling of individual cameras. To address the triangulation inconsistency caused by the misalignment between each camera's frame and the panoramic model's frame, we propose an extrinsic compensation method. This method improves feature consistency across views and significantly reduces triangulation and optimization errors, leading to more accurate pose estimation. We integrate the panoramic visual feature model into a tightly coupled LiDAR-visual-inertial system based on a factor graph. Extensive experiments on public datasets demonstrate that the panoramic visual feature model enhances the quality and consistency of multi-camera constraints, resulting in higher accuracy and robustness than existing multi-camera LiDAR-visual-inertial systems.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Vins-mono: A robust and versatile monoc- ular visual-inertial state estimator,
T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monoc- ular visual-inertial state estimator,”IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004–1020, 2018
2018
-
[2]
Openvins: A research platform for visual-inertial estimation,
P. Geneva, K. Eckenhoff, W. Lee, Y . Yang, and G. Huang, “Openvins: A research platform for visual-inertial estimation,” in2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 4666–4672
work page 2020
-
[3]
Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,
T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus, “Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,” in2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 5135–5142
work page 2020
-
[4]
Fast-lio2: Fast direct lidar- inertial odometry,
W. Xu, Y . Cai, D. He, J. Lin, and F. Zhang, “Fast-lio2: Fast direct lidar- inertial odometry,”IEEE Transactions on Robotics, vol. 38, no. 4, pp. 2053–2073, 2022
-
[5]
Lvi-sam: Tightly-coupled lidar-visual-inertial odometry via smoothing and mapping,
T. Shan, B. Englot, C. Ratti, and D. Rus, “Lvi-sam: Tightly-coupled lidar-visual-inertial odometry via smoothing and mapping,” in2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 5692–5698
work page 2021
-
[6]
Fast-livo2: Fast, direct lidarâ ˘A¸ Sinertialâ˘A¸ Svisual odometry,
C. Zheng, W. Xu, Z. Zou, T. Hua, C. Yuan, D. He, B. Zhou, Z. Liu, J. Lin, F. Zhu, Y . Ren, R. Wang, F. Meng, and F. Zhang, “Fast-livo2: Fast, direct lidarâ ˘A¸ Sinertialâ˘A¸ Svisual odometry,”IEEE Transactions on Robotics, vol. 41, pp. 326–346, 2025
work page 2025
-
[7]
Panoramic slam from a multiple fisheye camera rig,
S. Ji, Z. Qin, J. Shan, and M. Lu, “Panoramic slam from a multiple fisheye camera rig,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 159, 11 2019
work page 2019
-
[8]
Mcov-slam: A multicamera omnidirectional visual slam system,
Y . Yang, M. Pan, D. Tang, T. Wang, Y . Yue, T. Liu, and M. Fu, “Mcov-slam: A multicamera omnidirectional visual slam system,” IEEE/ASME Transactions on Mechatronics, vol. 29, no. 5, pp. 3556– 3567, 2024
work page 2024
Show all 35 references
-
[9]
Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,
R. Mur-Artal and J. D. Tardos, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,”IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255–1262, Oct. 2017. [Online]. Available: http://dx.doi.org/10.1109/TRO.2017.2705103
2017
-
[10]
Mcvo: A generic visual odometry for arbitrarily arranged multi-cameras,
H. Yu, J. Wang, Y . He, W. Yang, and G.-S. Xia, “Mcvo: A generic visual odometry for arbitrarily arranged multi-cameras,”
-
[11]
Panoramic visual-inertial slam tightly coupled with a wheel encoder,
F. Jiang, J. Chen, and S. Ji, “Panoramic visual-inertial slam tightly coupled with a wheel encoder,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 182, pp. 96–111, 12 2021
2021
-
[12]
Towards robust visual-inertial odometry with multiple non-overlapping monocular cameras,
Y . He, H. Yu, W. Yang, and S. Scherer, “Towards robust visual-inertial odometry with multiple non-overlapping monocular cameras,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 9452–9458
2022
-
[13]
Bamf-slam: Bundle adjusted multi-fisheye visual-inertial slam using recurrent field transforms,
W. Zhang, S. Wang, X. Dong, R. Guo, and N. Haala, “Bamf-slam: Bundle adjusted multi-fisheye visual-inertial slam using recurrent field transforms,” in2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 6232–6238
2023
-
[14]
Mavis: Multi-camera augmented visual-inertial slam using se2(3) based exact imu pre-integration,
Y . Wang, Y . Ng, I. Sa, Ã. Parra, C. Rodriguez-Opazo, T. Lin, and H. Li, “Mavis: Multi-camera augmented visual-inertial slam using se2(3) based exact imu pre-integration,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 1694– 1700
2024
-
[15]
Rmsc-vio: Robust multi-stereoscopic visual-inertial odometry for local visually challeng- ing scenarios,
T. Zhang, J. Xu, H. Shen, R. Yang, and T. Yang, “Rmsc-vio: Robust multi-stereoscopic visual-inertial odometry for local visually challeng- ing scenarios,”IEEE Robotics and Automation Letters, vol. 9, no. 5, pp. 4130–4137, 2024
2024
-
[16]
Panoramic direct lidar-assisted visual odometry,
Z. Yuan, T. Xu, X. Wang, J. Geng, and X. Yang, “Panoramic direct lidar-assisted visual odometry,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09287
2024
-
[17]
Efficient and accurate tightly-coupled visual-lidar slam,
C.-C. Chou and C.-F. Chou, “Efficient and accurate tightly-coupled visual-lidar slam,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 14 509–14 523, 2022
2022
-
[18]
A high-precision lidar-inertial odometry via invariant extended kalman filtering and efficient surfel mapping,
H. Zhang, R. Xiao, J. Li, C. Yan, and H. Tang, “A high-precision lidar-inertial odometry via invariant extended kalman filtering and efficient surfel mapping,”IEEE Transactions on Instrumentation and Measurement, vol. 73, pp. 1–11, 2024
2024
-
[19]
Visual-lidar odometry and mapping: low-drift, robust, and fast,
J. Zhang and S. Singh, “Visual-lidar odometry and mapping: low-drift, robust, and fast,” in2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 2174–2181
2015
-
[20]
Lvio-fusion:tightly- coupled lidar-visual-inertial odometry and mapping in degenerate environments,
H. Zhang, L. Du, S. Bao, J. Yuan, and S. Ma, “Lvio-fusion:tightly- coupled lidar-visual-inertial odometry and mapping in degenerate environments,”IEEE Robotics and Automation Letters, vol. 9, no. 4, pp. 3783–3790, 2024
2024
-
[21]
Camvox: A low-cost and accurate lidar-assisted visual slam system,
Y . Zhu, C. Zheng, C. Yuan, X. Huang, and X. Hong, “Camvox: A low-cost and accurate lidar-assisted visual slam system,” in2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 5049–5055
2021
-
[22]
R 2 live: A robust, real- time, lidar-inertial-visual tightly-coupled state estimator and mapping,
J. Lin, C. Zheng, W. Xu, and F. Zhang, “R 2 live: A robust, real- time, lidar-inertial-visual tightly-coupled state estimator and mapping,” IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 7469–7476, 2021
2021
-
[23]
R3live: A robust, real-time, rgb-colored, lidar- inertial-visual tightly-coupled state estimation and mapping package,
J. Lin and F. Zhang, “R3live: A robust, real-time, rgb-colored, lidar- inertial-visual tightly-coupled state estimation and mapping package,” in2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 10 672–10 678
2022
-
[24]
Fast-livo: Fast and tightly-coupled sparse-direct lidar-inertial-visual odometry,
C. Zheng, Q. Zhu, W. Xu, X. Liu, Q. Guo, and F. Zhang, “Fast-livo: Fast and tightly-coupled sparse-direct lidar-inertial-visual odometry,” in2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 4003–4009
2022
-
[25]
Livissfm: Accurate and robust structure-from-motion with lidar and visual cues,
H. Jiang, L. Zhou, Z. Zhang, Y . Yu, and G. Zhang, “Livissfm: Accurate and robust structure-from-motion with lidar and visual cues,” 2024. [Online]. Available: https://arxiv.org/abs/2410.22213
2024 arXiv
-
[26]
Liver: A tightly coupled lidar-inertial-visual state estimator with high robustness for underground environments,
T. Wen, Y . Fang, B. Lu, X. Zhang, and C. Tang, “Liver: A tightly coupled lidar-inertial-visual state estimator with high robustness for underground environments,”IEEE Robotics and Automation Letters, vol. 9, no. 3, pp. 2399–2406, 2024
2024
-
[27]
mvlins: A multilevel visual-lidar-inertial navigation system with completely decoupled odometry and adaptive environmental mapping,
B. Zhang, W. Yao, Y . Wang, P. Li, X. Shao, and G. Sun, “mvlins: A multilevel visual-lidar-inertial navigation system with completely decoupled odometry and adaptive environmental mapping,”IEEE Transactions on Intelligent Vehicles, pp. 1–13, 2024
2024
-
[28]
isam2: Incremental smoothing and mapping with fluid relinearization and incremental variable reordering,
M. Kaess, H. Johannsson, R. Roberts, V . Ila, J. Leonard, and F. Dellaert, “isam2: Incremental smoothing and mapping with fluid relinearization and incremental variable reordering,” in2011 IEEE International Conference on Robotics and Automation, 2011, pp. 3281–3288
2011
-
[29]
Good features to track,
J. Shi and Tomasi, “Good features to track,” in1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1994, pp. 593–600
1994
-
[30]
An Iterative Image Registration Technique with an Application to Stereo Vision,
B. D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” inIJCAI’81: 7th international joint conference on Artificial intelligence, vol. 2, Vancouver, Canada, Aug. 1981, pp. 674–679. [Online]. Available: https://hal.science/h...
1981
-
[31]
Bags of binary words for fast place recognition in image sequences,
D. Galvez-Lø spez and J. D. Tardos, “Bags of binary words for fast place recognition in image sequences,”IEEE Transactions on Robotics, vol. 28, no. 5, pp. 1188–1197, 2012
2012
-
[33]
M2dgr: A multi-sensor and multi-scenario slam dataset for ground robots,
J. Yin, A. Li, T. Li, W. Yu, and D. Zou, “M2dgr: A multi-sensor and multi-scenario slam dataset for ground robots,”IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2266–2273, 2021
2021
-
[34]
Hilti-oxford dataset: A millimeter-accurate benchmark for simultaneous localization and mapping,
L. Zhang, M. Helmberger, L. F. T. Fu, D. Wisth, M. Camurri, D. Scaramuzza, and M. Fallon, “Hilti-oxford dataset: A millimeter-accurate benchmark for simultaneous localization and mapping,”IEEE Robotics and Automation Letters, vol. 8, no. 1, p. 408â ˘A¸ S415, Jan. 2023. [Online...
2023
-
[2021]
Available: https://arxiv.org/abs/2112.08854
[Online]. Available: https://arxiv.org/abs/2112.08854
-
[2024]
Available: https://arxiv.org/abs/2412.03146
[Online]. Available: https://arxiv.org/abs/2412.03146
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.