Pith. sign in

REVIEW 4 major objections 7 minor 30 references

Multi-Spectral Visual Odometry without Explicit Stereo Matching

T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A visible–thermal camera pair can do metric visual odometry with no stereo matching.

desk verdict A credible direct VO extension to RGB+LWIR with a genuinely new fixed-baseline trick, but the metric-scale claim is under-supported by unspecified trajectory alignment and an under-specified scale-recovery step. read the letter →

arxiv 1908.08814 v1 pith:B5E5ISEL submitted 2019-08-23 cs.CV

Weichen Dai , Yu Zhang , Donglei Sun , Naira Hovakimyan , Ping Li This is my paper · ORCID
classification cs.CV
keywords multi-spectralvisualodometrylong-waveinfrareddirectimagealignmentmetricscalerecoverybundleadjustmentsemi-densereconstructionthermalimagingstereomatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a visible-light camera and a long-wave infrared camera, rigidly fixed with a known baseline, can serve as a metric visual odometry system without ever finding stereo correspondences between the two spectra. Because the two image types share almost no texture, explicit matching is unreliable; the paper instead runs direct image alignment separately on each camera's temporal stream and couples the streams through the fixed baseline geometry. If correct, this allows visible and thermal images to be fused for localization in low-illumination or smoky conditions, and yields a semi-dense 3D reconstruction with both spectral measurements per point. The paper supports the claim with seven indoor and outdoor sequences, comparing against established monocular and stereo baselines.

What carries the argument

The load-bearing object is the static-stereo direct image alignment term $E^{i,b}_{l,k}$, which reprojects a map point from the visible camera into the LWIR camera through the fixed extrinsic transformation $T_{b,a}$ and compares photoconsistency between two LWIR frames. This term supplies the metric-scale information that temporal monocular alignment alone cannot provide. The paper also derives two critical-motion conditions, Equations (9) and (10), under which the scale becomes unobservable, and argues that the sliding-window optimization generally avoids them.

What would settle it

Record a sequence with motion-capture ground truth, then rerun the pipeline with the extrinsic baseline rotated by a known small angle (say 1 degree) and translated by a few millimeters while keeping all other settings unchanged. If the trajectory error and the reconstructed map scale do not degrade as the perturbation grows, the static-stereo term is not actually carrying the metric-scale recovery described in the paper.

Watch

Extended reading notes

Core claim

The central discovery is that photometric bundle adjustment over multi-view stereo can replace explicit stereo matching in a multi-spectral rig. Each 3D point is initialized with random depth from one camera and projected into subsequent frames of the same camera (temporal multi-view stereo) and, through the fixed extrinsic transform $T_{b,a}$, into the other camera's frames (static stereo). The photometric residual is evaluated only between images from the same camera, so no cross-spectral similarity is ever required. The fixed-baseline constraint makes the otherwise unobservable metric scale observable, provided the motion avoids two critical conditions; over a sliding window with many points and keyframes those conditions generically fail, and the scale converges. The method also discards LWIR frames corrupted by non-uniformity correction and keeps tracking with the visible camera alone, giving odometry continuity during thermal-camera maintenance.

Load-bearing premise

The fixed baseline transform $T_{b,a}$ between the visible and LWIR cameras must be accurately calibrated and the two cameras must be hardware synchronized; if that transform is wrong, the recovered metric scale is biased even though the photometric optimization may still look consistent.

Editorial extensions

If this is right

  • A multi-spectral rig can be treated as scale-aware without solving cross-modal correspondence, so operation in darkness, smoke, or glare no longer requires thermal-to-visible feature matching.
  • Thermal non-uniformity correction outages stop being fatal: corrupted thermal frames are dropped and the visible stream alone carries tracking, with scale held by the fixed baseline.
  • Because scale converges only during sliding-window optimization, the first few frames of a run are not metrically reliable; the scale becomes trustworthy after a motion that excites the baseline direction.
  • The semi-dense output includes a metric 3D map with per-point visible and thermal intensities, which the paper notes is not produced by earlier multi-spectral methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate testable extension is to compute, from Equation (10), the minimal two-frame maneuver that guarantees scale observability, and use it as a startup script for practical deployments.
  • The method's scale accuracy is only as good as the unquantified extrinsic calibration and synchronization between the two cameras; an ablation that perturbs $T_{b,a}$ would quantify how much of the reported accuracy rests on that assumption.
  • The same architecture should transfer to other sensor pairs with low texture correlation and a known rigid transform, such as visible plus near-infrared or visible plus event camera, where the static-stereo term would play the same scale-pinning role.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes MSVO, a visual odometry method that uses a rigidly mounted visible-light camera and a long-wave infrared camera without explicit cross-spectral stereo matching. A visible-camera map point is projected into both LWIR frames through the fixed extrinsic T_{b,a}, so that photometric residuals are computed between temporal views of each camera separately. Sliding-window bundle adjustment optimizes keyframe poses, affine brightness parameters, and inverse depths, and the paper argues that the fixed baseline makes the metric scale observable. Experiments on seven self-recorded indoor/outdoor sequences report ATE against motion-capture ground truth, with qualitative demonstrations of metric-scale recovery, fused multi-spectral point clouds, and handling of uncooled-LWIR NUC corruption. The paper claims accurate odometry and semi-dense metric 3D reconstruction without stereo matching.

Significance. If the claims hold, the paper makes a useful contribution to multi-spectral visual odometry: it avoids brittle cross-spectral feature matching, exploits fixed-baseline geometry through direct alignment, provides metric-scale recovery, and addresses the practical NUC-corruption problem of uncooled LWIR cameras. The cost functions and optimization structure are clearly specified, and the reported ATE numbers indicate that the system functions on the recorded sequences. The observability discussion is an attempt to characterize critical motions, which is valuable for a scale-recovering method. However, the evidence for the central metric-scale claim is currently incomplete: the role of the scale-carrying residual in the optimization is described inconsistently, the calibration and synchronization accuracy of the fixed baseline is not quantified, and the ATE evaluation does not isolate scale error. These issues are fixable and do not invalidate the overall idea, but they need to be addressed before the claims can be accepted.

major comments (4)
  1. [Section IV-D and Eq. (13)] The statement near the end of Section IV-D that "Equation (6) is not used until the metric scale is recovered" contradicts the definition of the full cost in Eq. (13), which sums E_{l,k} over active keyframes, with E_{l,k} defined in Eq. (12) as the sum of E^{i,a}_{l,k} and E^{i,b}_{l,k}. Since Eq. (6) is the only scale-carrying residual introduced in the paper, the manuscript never explains how the metric scale is initially recovered if Eq. (6) is excluded from the optimization. Please clarify the optimization schedule: specify which cost function is used before scale convergence, which cost is used in the convergence test, and exactly when Eq. (6) enters the optimization.
  2. [Section IV-B3, Eqs. (9)-(10)] The observability derivation is too terse and contains notation errors that make it impossible to verify. After defining T_b = [R^T, -R^T t; 0^T, 1], the text states t_b = -R t, but the translation vector of that matrix is -R^T t; the definitions of t'_b, t'^*_b, R', s, and t are not stated precisely. The sentence just before Eq. (10), "When t_b is on the surface formed by t_b, t'_b, and p_w," is self-referential and presumably should refer to a different point. As written, the critical-condition equations (9) and (10) cannot be checked, yet they are the only formal support for the claim that the metric scale is observable in the sliding window. Please rewrite the derivation with consistent notation and a complete argument.
  3. [Section V-A and Eqs. (7)-(8)] The metric-scale claim rests on the fixed extrinsic T_{b,a} between the visible and LWIR cameras. A bias in the magnitude of the baseline directly scales the recovered world, and any inter-camera timing offset acts as a motion-dependent baseline error. The paper states that the cameras are hardware synchronized but reports no calibration reprojection error, no extrinsic uncertainty, and no synchronization jitter, despite the LWIR camera being rolling-shutter and subject to geometric distortion and motion blur. Please report quantitative calibration and synchronization accuracy, and validate the recovered scale against an independent distance measurement (e.g., known object sizes or an external reference) rather than relying on the fixed baseline alone.
  4. [Table II and Section V-B] The ATE evaluation is described as aligning the estimation results with the groundtruth trajectories, but the alignment model is not specified. Since monocular ORB-SLAM2 and DSO cannot recover metric scale, a Sim(3) alignment is likely used; if so, the ATE values are invariant to a global scale error and do not evaluate the paper's central metric-scale claim. Please state explicitly whether the alignment is SE(3) or Sim(3), and report per-sequence scale error (e.g., the estimated scale factor after SE(3) alignment) for MSVO. Without this, the quantitative evidence for recovered metric scale is incomplete.
minor comments (7)
  1. [Section III] "Li Algebra" should be "Lie algebra" in the sentence introducing se(3).
  2. [Section IV-D] The variable used for the state in Eq. (14) is written as f in se(3)^n x R^m, but the update rule in Eq. (15) acts on SE(3) poses; the domain should be SE(3)^n x R^m.
  3. [Section IV-E] "Duing NUC" should be "During NUC."
  4. [Table I] The first column header says "Method" but the entries are sequence names; the header should be "Sequence."
  5. [Section V-B] The sentence describing ATE computation is ambiguous: "the Euclidean distance is computed between the estimation results and the groundtruth" should specify that the RMSE is over trajectory positions after alignment, not a single Euclidean distance.
  6. [Section V-B] No error bars or repeated trials are reported for the ATE values in Table II; given the short sequences and handheld device, reporting run-to-run variability would strengthen the comparison.
  7. [General] The manuscript does not mention whether the dataset or code will be released, which limits reproducibility given that the evaluation is on a self-built dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: metric scale is anchored by a calibrated fixed baseline and evaluated against external ground truth.

full rationale

The derivation chain is self-contained. The metric-scale argument rests on the static-stereo photometric residual (Eq. 6) built from the fixed extrinsic T_{b,a} (Eqs. 7-8); the baseline is an input calibration, not a quantity fitted to the evaluation data. Map points are initialized at random depth (Sec. IV-A), so no output quantity is presupposed by the formulation. The observability test (Eqs. 9-10) is an independent geometric condition and does not incorporate the ATE results. Baselines are evaluated against motion-capture/GPS ground truth (Tables I-II, Figs. 8, 15) with no parameter fitted to those trajectories. No load-bearing self-citations occur; the cited prior work (DSO, ORB-SLAM2, non-overlapping-FOV methods) is used for standard components rather than to justify the novelty. Two concerns noted in review are correctness issues, not circularity: (i) the note in Sec. IV-D that Eq. (6) is not used until metric scale is recovered is ambiguous because Eq. (6) is the scale-carrying residual, and (ii) Table II reports ATE after alignment without stating whether alignment is SE(3) or Sim(3), so it may not separately validate metric scale. Neither concern involves fitting an input and then calling it a prediction, nor importing a self-citation as an external fact; hence the score remains 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the fixed-baseline assumption, photometric consistency, and accuracy of camera models, all standard in stereo and direct VO. No fitted constants are introduced to produce the claimed results; the method estimates scale from the calibrated baseline. No new physical entities are postulated.

assumptions (4)
  • domain assumption The extrinsic calibration T_{b,a} between the visible and LWIR cameras is accurate and the cameras are hardware synchronized.
    Entered via Equations (7) and (8); scale recovery depends on this fixed baseline.
  • domain assumption Photometric consistency holds across temporal frames of each camera, with brightness modeled by a per-frame affine transfer.
    The direct alignment residuals in Equations (3) and (6) rely on this assumption, inherited from DSO.
  • domain assumption The camera motion during initialization avoids the critical motions characterized by Equations (9) and (10).
    The paper notes the device was translated and rotated simultaneously to avoid critical motion and facilitate scale recovery.
  • domain assumption Intrinsic camera models and distortion for both cameras are calibrated accurately.
    Projection equations (1) require accurate intrinsics; the paper calibrates using a special chessboard but reports no calibration accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Spectral Visual Odometry without Explicit Stereo Matching." pith.science (2026). https://pith.science/paper/B5E5ISEL

@misc{pith2026190808814,
  author       = {Pith},
  title        = {Pith review of: Multi-Spectral Visual Odometry without Explicit Stereo Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5E5ISEL}},
  note         = {Machine review of arXiv:1908.08814}
}
read the original abstract

Multi-spectral sensors consisting of a standard (visible-light) camera and a long-wave infrared camera can simultaneously provide both visible and thermal images. Since thermal images are independent from environmental illumination, they can help to overcome certain limitations of standard cameras under complicated illumination conditions. However, due to the difference in the information source of the two types of cameras, their images usually share very low texture similarity. Hence, traditional texture-based feature matching methods cannot be directly applied to obtain stereo correspondences. To tackle this problem, a multi-spectral visual odometry method without explicit stereo matching is proposed in this paper. Bundle adjustment of multi-view stereo is performed on the visible and the thermal images using direct image alignment. Scale drift can be avoided by additional temporal observations of map points with the fixed-baseline stereo. Experimental results indicate that the proposed method can provide accurate visual odometry results with recovered metric scale. Moreover, the proposed method can also provide a metric 3D reconstruction in semi-dense density with multi-spectral information, which is not available from existing multi-spectral methods.

Figures

Figures reproduced from arXiv: 1908.08814 by the authors.

Figure 1
Figure 1. Potential (green) and unfeasible (red) correspondences. For [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The IR images of a human head in different wavelengths [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. MSVO system overview. The red and blue color denote two types of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: One point with two temporal matching results. The red and the blue [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: The observability of metric scale. The red and the blue color denote [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Factor graph for the full cost function. The red and the blue colored [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Multi-spectral device: The sensors from left to right are RGB camera, [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Depth map of multi-spectral images on Sq7. The right image shows [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Example of a fused point cloud. The left top picture is the RGB image [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Normal and corrupted thermal images. If the LWIR camera enters [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 14
Figure 14. Figure 14: The point cloud and stereo association results. The top left image [PITH_FULL_IMAGE:figures/full_fig_p008_14.png]
Figure 15
Figure 15. Figure 15: Qualitative result. The estimated trajectory (green) is aligned to GPS [PITH_FULL_IMAGE:figures/full_fig_p008_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    F. Arnell. Vision-based pedestrian detection system for use in smart cars. Master’s thesis, Royal Institute of Technology, Dept. of Numerical Analysis and Computer Science, SE-100 , 44, 2005. 2

    work page 2005

  2. [2]

    Beauvisage, N

    A. Beauvisage, N. Aouf, and H. Courtois. Multi-spectral visual odometry for unmanned air vehicles. In Systems, Man, and Cybernetics (SMC), 2016 IEEE International Conference on , pages 001994–001999. IEEE, 2016. 2

    work page 2016

  3. [3]

    P. V . K. Borges and S. Vidas. Practical infrared visual odometry. IEEE Transactions on Intelligent Transportation Systems , 17(8):2205–2213,

    work page

  4. [4]

    L. Chen, L. Sun, T. Yang, L. Fan, K. Huang, and Z. Xuanyuan. Rgb-t slam: A flexible slam framework by combining appearance and thermal information. In Robotics and Automation (ICRA), 2017 IEEE International Conference on , pages 5682–5687. IEEE, 2017. 2

    work page 2017

  5. [5]

    Chiuso, P

    A. Chiuso, P. Favaro, H. Jin, and S. Soatto. Structure from motion causally integrated over time. IEEE transactions on pattern analysis and machine intelligence , 24(4):523–535, 2002. 2

    work page 2002

  6. [6]

    Clipp, J.-H

    B. Clipp, J.-H. Kim, J.-M. Frahm, M. Pollefeys, and R. Hartley. Robust 6dof motion estimation for non-overlapping, multi-camera systems. In Applications of Computer Vision, 2008. WACV 2008. IEEE Workshop on, pages 1–8. IEEE, 2008. 2

    work page 2008

  7. [7]

    A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse. Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelligence , 29(6):1052–1067, 2007. 2

    work page 2007

  8. [8]

    Engel, V

    J. Engel, V . Koltun, and D. Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017. 2, 3, 5, 6

    work page 2017

Show all 30 references
  1. [9]

    Engel, T

    J. Engel, T. Sch ¨ops, and D. Cremers. Lsd-slam: Large-scale direct monocular slam. In European Conference on Computer Vision , pages 834–849. Springer, 2014. 2, 3

    2014

  2. [10]

    Engel, J

    J. Engel, J. Sturm, and D. Cremers. Semi-dense visual odometry for a monocular camera. In Computer Vision (ICCV), 2013 IEEE International Conference on , pages 1449–1456. IEEE, 2013. 3

    2013

  3. [11]

    Forster, Z

    C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and D. Scaramuzza. Svo: Semidirect visual odometry for monocular and multicamera systems. IEEE Transactions on Robotics , 33(2):249–265, 2017. 2

    2017

  4. [12]

    Hajebi and J

    K. Hajebi and J. S. Zelek. Structure from infrared stereo images. In Computer and Robot Vision, 2008. CRV’08. Canadian Conference on , pages 105–112. IEEE, 2008. 2

    2008

  5. [13]

    S.-H. Jung, J. Eledath, S. Johansson, and V . Mathevon. Egomotion estimation in monocular infra-red image sequence for night vision applications. In Applications of Computer Vision, 2007. WACV’07. IEEE Workshop on, pages 8–8. IEEE, 2007. 2

    2007

  6. [14]

    Kazik, L

    T. Kazik, L. Kneip, J. Nikolic, M. Pollefeys, and R. Siegwart. Real- time 6d stereo visual odometry with non-overlapping fields of view. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 1529–1536. IEEE, 2012. 2

    2012

  7. [15]

    J.-H. Kim, H. Li, and R. Hartley. Motion estimation for nonoverlapping multicamera rigs: Linear algebraic and l ∞ geometric solutions. IEEE Transactions on Pattern Analysis and Machine Intelligence , 32(6):1044– 1059, 2010. 2

    2010

  8. [16]

    Klein and D

    G. Klein and D. Murray. Parallel tracking and mapping for small ar workspaces. In Mixed and Augmented Reality, 2007. ISMAR 2007. 6th IEEE and ACM International Symposium on , pages 225–234. IEEE,

    2007

  9. [17]

    Leutenegger, S

    S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, and P. Furgale. Keyframe-based visual–inertial odometry using nonlinear optimization. The International Journal of Robotics Research , 34(3):314–334, 2015. 5

    2015

  10. [18]

    S.-S. Lin. Review: Extending visible band computer vision techniques to infrared band images. Rapport Technique MS-CIS-01-04, Computer and Information Science Department , 2001. 2

    2001

  11. [19]

    Magnabosco and T

    M. Magnabosco and T. P. Breckon. Cross-spectral visual simultaneous localization and mapping (slam) with sensor handover. Robotics and Autonomous Systems , 61(2):195–208, 2013. 2

    2013

  12. [20]

    Mouats, N

    T. Mouats, N. Aouf, L. Chermak, and M. A. Richardson. Thermal stereo odometry for uavs. IEEE Sensors Journal , 15(11):6335–6347, 2015. 2

    2015

  13. [21]

    Mouats, N

    T. Mouats, N. Aouf, A. D. Sappa, C. Aguilera, and R. Toledo. Multispectral stereo odometry. IEEE Transactions on Intelligent Transportation Systems, 16(3):1210–1224, 2015. 1, 2

    2015

  14. [22]

    Mouragnon, M

    E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd. Real time localization and 3d reconstruction. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on , volume 1, pages 363–370. IEEE, 2006. 2

    2006

  15. [23]

    Mur-Artal and J

    R. Mur-Artal and J. D. Tard ´os. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics , 2017. 2, 6

    2017

  16. [24]

    R. A. Newcombe, S. J. Lovegrove, and A. J. Davison. Dtam: Dense tracking and mapping in real-time. In Computer Vision (ICCV), 2011 IEEE International Conference on , pages 2320–2327. IEEE, 2011. 2

    2011

  17. [25]

    Nist ´er, O

    D. Nist ´er, O. Naroditsky, and J. Bergen. Visual odometry. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on , volume 1, pages I–I. Ieee,

    2004

  18. [26]

    Owens and L

    K. Owens and L. Matthies. Passive night vision sensor comparison for unmanned ground vehicle stereo vision navigation. In Computer Vision Beyond the Visible Spectrum: Methods and Applications, 1999.(CVBVS’99) Proceedings. IEEE Workshop on, pages 59–68. IEEE,

    1999

  19. [27]

    Pinggera12, T

    P. Pinggera12, T. Breckon, and H. Bischof. On cross-spectral stereo matching using dense gradient features. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , 2012. 2

    2012

  20. [28]

    Poujol, C

    J. Poujol, C. A. Aguilera, E. Danos, B. X. Vintimilla, R. Toledo, and A. D. Sappa. A visible-thermal fusion based monocular visual odometry. In Robot 2015: Second Iberian Robotics Conference , pages 517–528. Springer, 2016. 2

    2015

  21. [29]

    Rankin, A

    A. Rankin, A. Huertas, L. Matthies, M. Bajracharya, C. Assad, S. Brennan, P. Bellutta, and G. W. Sherwin. Unmanned ground vehicle perception using thermal infrared cameras. In Unmanned Systems Technology XIII, volume 8045, page 804503. International Society for Optics and Phot...

    2011

  22. [30]

    R. Wang, M. Schworer, and D. Cremers. Stereo dso: Large-scale direct sparse visual odometry with stereo cameras. In Proceedings of the IEEE International Conference on Computer Vision , pages 3903–3911, 2017. 3

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.