Cross-Spectral Stereo Inertial Odometry

Ayoung Kim; Hyeonjae Gil; Hyunho Song; Hyunsoo Jang; Seungsang Yun; Tai Hyoung Rhee

Cross-Spectral Stereo Inertial Odometry

T0 review · 2 major / 0 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A cross-spectral visual-thermal-inertial odometry system uses asynchronous deep matching and dynamic weighting to avoid simultaneous failures from spectral redundancy.

desk verdict The paper assembles a practical asynchronous cross-spectral VTI system with entropy weighting and NUC handling, but the abstract supplies no numbers to check whether the weighting actually delivers the claimed robustness. read the letter →

arxiv 2606.29757 v1 pith:S7PFKY5L submitted 2026-06-29 cs.RO

Seungsang Yun , Hyunsoo Jang , Tai Hyoung Rhee , Hyunho Song , Hyeonjae Gil , Ayoung Kim This is my paper

classification cs.RO

keywords cross-spectralodometryvisual-thermalfusionstereoinertialspectral-awareweightingasynchronousstateestimationthermalnon-uniformitycorrectionreal-timeVIO

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a real-time cross-spectral VTI odometry architecture that temporally separates high-latency deep feature matching from high-rate state estimation. Standard single-spectrum stereo VIO fails when both cameras encounter the same environmental degradation because their data are redundant. The system counters this by applying a spectral-aware weighting that shifts reliance between visual and thermal streams according to photometric entropy and thermal noise, while also providing continuous handling of thermal non-uniformity correction. Experiments across varied conditions show the approach delivers higher accuracy under normal daylight and sustained operation when lighting or thermal conditions degrade.

What carries the argument

Spectral-aware weighting scheme that balances visual and thermal modalities using photometric entropy and thermal noise, paired with asynchronous decoupling of deep matching from state estimation.

What would settle it

A test environment where photometric entropy or thermal noise no longer tracks actual sensor quality, or where decoupling-induced timing offsets produce measurable drift in the state estimate, would show the weighting and decoupling fail to deliver claimed robustness.

Watch

Extended reading notes

Core claim

The architecture incorporates a spectral-aware weighting scheme that dynamically balances modality reliance based on photometric entropy and thermal noise, ensuring robustness against both abrupt lighting changes and thermal artifacts. Asynchronous decoupling of deep matching from state estimation preserves real-time performance, and seamless NUC handling maintains tracking continuity, allowing the system to overcome spectral redundancy.

Load-bearing premise

Photometric entropy and thermal noise metrics remain reliable indicators of each modality's quality and asynchronous decoupling does not introduce unmodeled timing errors that degrade the fused estimate.

Editorial extensions

If this is right

The system achieves superior accuracy in nominal daylight compared with single-spectrum baselines.
Robustness is maintained in visually degraded environments where both modalities would otherwise fail together.
Seamless NUC handling prevents loss of tracking continuity during thermal camera recalibration.
Spectral redundancy is overcome by complementary use of visual and thermal data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The weighting logic could be adapted to additional modalities such as event cameras or radar without redesigning the core fusion pipeline.
The asynchronous structure suggests potential for deployment on embedded hardware where deep matching would otherwise violate timing budgets.
Extension to longer-duration missions might require explicit modeling of how NUC events accumulate bias in the inertial integration.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main point is a real-time visual-thermal-inertial odometry pipeline that runs deep matching asynchronously so it does not slow the high-rate VIO loop, then fuses the two modalities with a weighting scheme driven by photometric entropy and thermal noise, plus a fix for NUC events that keeps tracking alive. The authors correctly identify that single-spectrum stereo VIO can fail in correlated ways when lighting or thermal conditions hit both cameras at once, and they try to solve it with an engineering stack rather than a new theory.

The integration looks sensible on paper: decoupling the slow matcher buys real-time performance, the weighting tries to drop the bad modality when conditions change, and NUC handling is a concrete detail that matters for thermal cameras. Releasing code and data is the right move for this kind of applied work.

The abstract states superiority in daylight and robustness in degraded scenes but shows none of the supporting numbers, baselines, ablations, or timing checks. Without those, it is impossible to tell whether the entropy and noise metrics actually track modality quality or whether the asynchronous split creates unmodeled latency that hurts the fused estimate. The stress-test concern about those proxies therefore stands until the full experiments are examined.

This is the sort of paper that matters to people who need to run odometry on robots that move between bright and dark or hot and cold environments. A practitioner who wants a working starting point once the repo is live could get something out of it. The thinking is coherent and the problem is real, so the paper deserves a serious referee even if the final verdict depends on the quantitative results that are missing from the abstract.

Referee Report

2 major / 0 minor

Summary. The paper introduces an asynchronous cross-spectral visual-thermal-inertial (VTI) odometry system that temporally decouples high-latency deep matching from high-rate state estimation. It incorporates a spectral-aware weighting scheme based on photometric entropy and thermal noise for dynamic modality balancing, a mechanism for handling thermal Non-uniformity Correction (NUC), and claims through experiments to overcome spectral redundancy with superior accuracy in daylight and robustness in degraded environments. The work plans to release code and data.

Significance. If the weighting scheme and asynchronous design are shown to be effective with supporting quantitative evidence, the approach could meaningfully advance robust real-time odometry by addressing correlated failures in single-spectrum systems. The open-sourcing commitment strengthens potential impact for reproducibility in the robotics community.

major comments (2)

[Abstract] Abstract: The central claim that the photometric entropy and thermal noise metrics enable reliable dynamic weighting and robustness against spectral redundancy lacks any referenced validation (e.g., correlation with ground-truth modality quality, ablation on metric failure cases, or cross-validation results). Without these, the attribution of superior accuracy and robustness to the architecture cannot be assessed.
[Abstract] Abstract: The asynchronous decoupling of deep matching from high-rate VIO is asserted to avoid real-time violations and unmodeled timing errors, yet no timing histograms, latency distributions, or synchronization error analysis are described to confirm that the fused state estimate remains unaffected.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments on the abstract below, proposing targeted revisions to improve clarity and evidence presentation while preserving the manuscript's core contributions.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim that the photometric entropy and thermal noise metrics enable reliable dynamic weighting and robustness against spectral redundancy lacks any referenced validation (e.g., correlation with ground-truth modality quality, ablation on metric failure cases, or cross-validation results). Without these, the attribution of superior accuracy and robustness to the architecture cannot be assessed.

Authors: We agree the abstract would benefit from explicit linkage to validation. The manuscript's Section V reports extensive experiments demonstrating superior accuracy and robustness across scenarios, including comparisons that implicitly support the weighting scheme's role. We will revise the abstract to reference these experiments (e.g., 'as shown via ablation studies in Section V') and expand the experiments section with a dedicated ablation on the entropy/noise metrics versus ground-truth modality quality where space allows. This strengthens attribution without altering the reported results. revision: yes
Referee: [Abstract] Abstract: The asynchronous decoupling of deep matching from high-rate VIO is asserted to avoid real-time violations and unmodeled timing errors, yet no timing histograms, latency distributions, or synchronization error analysis are described to confirm that the fused state estimate remains unaffected.

Authors: The asynchronous design is detailed in Section III, with real-time operation asserted based on the system architecture and experimental runs. We acknowledge the abstract lacks quantitative timing evidence. We will revise the manuscript to include timing histograms, latency distributions, and synchronization analysis (either in the main text or supplementary material) to explicitly confirm the fused estimate is unaffected by the decoupling. This directly addresses the concern while aligning with the existing real-time claims. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: system description with independent empirical validation

full rationale

The paper presents an engineering architecture (asynchronous decoupling of deep matching from VIO, spectral-aware weighting via photometric entropy/thermal noise, NUC handling) whose central claims rest on experimental results across scenarios rather than any closed mathematical derivation. No equations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text; the weighting scheme is a design choice justified by robustness goals, not reduced to its own inputs by construction. The derivation chain is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The abstract provides no explicit free parameters, axioms, or invented entities; the weighting scheme and NUC handling are presented as engineering choices whose internal details are not visible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cross-Spectral Stereo Inertial Odometry." pith.science (2026). https://pith.science/paper/S7PFKY5L

@misc{pith2026260629757,
  author       = {Pith},
  title        = {Pith review of: Cross-Spectral Stereo Inertial Odometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S7PFKY5L}},
  note         = {Machine review of arXiv:2606.29757}
}

read the original abstract

Standard stereo VIO focuses exclusively on the benefit of metric scale via single-spectrum baselines, often overlooking the risks of spectral redundancy. This structural limitation leads to correlated failures, where both sensors simultaneously fail in degraded environments that affect their shared spectrum. Leveraging a cross-spectral system presents a complementary solution to this issue, yet the significant appearance gap between modalities renders standard matching ineffective. Existing deep learning-based matchers, while effective, introduce inference latencies that violate real-time constraints. To bridge this gap, we present an asynchronous real-time cross-spectral visual-thermal-inertial (VTI) system that temporally decouples high-latency deep matching from high-rate state estimation. Our architecture incorporates a spectral-aware weighting scheme that dynamically balances modality reliance based on photometric entropy and thermal noise, ensuring robustness against both abrupt lighting changes and thermal artifacts. Furthermore, we introduce a seamless handling mechanism for thermal Non-uniformity Correction (NUC) to maintain tracking continuity. Extensive experiments across diverse scenarios confirm that our system overcomes spectral redundancy, yielding superior accuracy in nominal daylight while ensuring robustness in visually degraded environments. We will open source our code and data: https://github.com/seungsang07/cross-spectral-stereo-inertial-odometry

Figures

Figures reproduced from arXiv: 2606.29757 by the authors.

**Figure 1.** Comparison of RGB and TIR inputs under challenging illumination. TIR preserves structure under low light and glare, enabling robust cross-spectral VTI in both degraded and nominal conditions. both views capture identical spectral information, they generate highly correlated photometric gradients. Consequently, instead of sharpening the optimization basin, the second view merely duplicates the existing constraints, o… view at source ↗

**Figure 3.** SRW map of RGB-TIR measurements. Higher values denote greater photometric reliability. The proposed SRW combines point-wise EWG with frame-level HNR to suppress noise-dominated TIR responses while preserving reliable structures. For comparison, the Difference of Gaussian (DoG) response is also visualized; it captures structural edges but also responds to highfrequency thermal noise and FPN-like artifacts. {𝑐 (1) 𝐻 … view at source ↗

**Figure 5.** Cross-spectral stereo calibration. Epipolar lines induced by TIR points accurately correspond to their RGB counterparts, validating the estimated crossspectral stereo geometry. DAY-OUT NIGHT-OUT GLARE-OUT TIR-NOISE-IN INOUT-TRANS [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

Figures from the paper (4 more)

**Figure 6.** Figure 6: Sequences from our Cross-Spectral Dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Qualitative trajectory comparison on DAY-OUT. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Qualitative trajectory on INOUT-TRANS. This sequence includes an indoor-to-outdoor transition with abrupt illumination changes between a dark indoor environment and a brighter outdoor scene. Trajectories are visualized after aligning the initial pose and starting point…

**Figure 10.** Figure 10: Temperature-aware 3D reconstruction. Outdoor RGB and thermal maps, where the thermal cloud (Low to High) reveals details invisible in RGB. TABLE VI: Runtime statistics on DAY-OUT. Latency values are reported in milliseconds. The camera input runs at 25 Hz(40 ms). Modu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages

[1]

Robust multispectral visual- inertial navigation with visual odometry failure recovery,

A. Beauvisage, K. Ahiska, and N. Aouf, “Robust multispectral visual- inertial navigation with visual odometry failure recovery,”IEEE Trans. Intell. Transport. Sys., vol. 23, no. 7, pp. 9089–9101, 2022

2022
[2]

ROVTIO: Robust visual thermal inertial odometry,

H. D. Flemmen, “ROVTIO: Robust visual thermal inertial odometry,” Master’s thesis, NTNU, 2021

2021
[3]

Online temporal calibration for monocular visual- inertial systems,

T. Qin and S. Shen, “Online temporal calibration for monocular visual- inertial systems,” inProc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys., 2018, pp. 3662–3669

2018
[4]

ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,

C. Campos, R. Elvira, J. J. G. Rodr ´ıguez, J. M. Montiel, and J. D. Tard´os, “ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,”IEEE Trans. Robot., vol. 37, no. 6, pp. 1874–1890, 2021

2021
[5]

arXiv preprint arXiv:2202.09199 (2022)

S. Leutenegger, “OKVIS2: Realtime scalable visual-inertial SLAM with loop closure,”arXiv preprint arXiv:2202.09199, 2022

work page arXiv 2022
[6]

DM-VIO: Delayed marginalization visual-inertial odometry,

L. Von Stumberg and D. Cremers, “DM-VIO: Delayed marginalization visual-inertial odometry,”IEEE Robot. and Automat. Lett., vol. 7, no. 2, pp. 1408–1415, 2022

2022
[7]

Direct sparse odometry,

J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,”IEEE Trans. Pattern Analysis and Machine Intell., vol. 40, no. 3, pp. 611–625, 2018

2018
[8]

DeepTIO: A deep thermal-inertial odometry with visual hallucination,

M. R. U. Saputra, P. P. De Gusmao, C. X. Lu, Y. Almalioglu, S. Rosa, C. Chen, J. Wahlstr¨om, W. Wang, A. Markham, and N. Trigoni, “DeepTIO: A deep thermal-inertial odometry with visual hallucination,”IEEE Robot. and Automat. Lett., vol. 5, no. 2, pp. 1672–1679, 2020

2020

Show all 25 references

[9]

TP-TIO: A robust thermal-inertial odometry with deep ThermalPoint,

S. Zhao, P. Wang, H. Zhang, Z. Fang, and S. Scherer, “TP-TIO: A robust thermal-inertial odometry with deep ThermalPoint,” inProc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys., 2020, pp. 4505–4512

2020
[10]

Registering multimodal imagery with occluding objects using mutual information: application to stereo tracking of humans,

S. Krotosky and M. Trivedi, “Registering multimodal imagery with occluding objects using mutual information: application to stereo tracking of humans,” inAugmented Vision Perception in Infrared: Algorithms and Applied Systems, 2009, pp. 321–347

2009
[11]

MultiPoint: Cross-spectral registration of thermal and optical aerial imagery,

F. Achermann, A. Kolobov, D. Dey, T. Hinzmann, J. J. Chung, R. Siegwart, and N. Lawrance, “MultiPoint: Cross-spectral registration of thermal and optical aerial imagery,” inConf. on Robot Learning, 2021, pp. 1746–1760

2021
[12]

XoFTR: Cross-modal feature matching transformer,

¨O. Tuzcuo˘glu, A. K¨oksal, B. Sofu, S. Kalkan, and A. A. Alatan, “XoFTR: Cross-modal feature matching transformer,” inProc. IEEE Conf. on Comput. Vision and Pattern Recog., 2024, pp. 4275–4286

2024
[13]

Minima: Modality invariant image matching,

J. Ren, X. Jiang, Z. Li, D. Liang, X. Zhou, and X. Bai, “Minima: Modality invariant image matching,” inProc. IEEE Conf. on Comput. Vision and Pattern Recog., 2025, pp. 23 059–23 068

2025
[14]

Multispec- tral stereo odometry,

T. Mouats, N. Aouf, A. D. Sappa, C. Aguilera, and R. Toledo, “Multispec- tral stereo odometry,”IEEE Trans. Intell. Transport. Sys., vol. 16, no. 3, pp. 1210–1224, 2014

2014
[15]

Multi-spectral visual odometry without explicit stereo matching,

W. Dai, Y. Zhang, D. Sun, N. Hovakimyan, and P. Li, “Multi-spectral visual odometry without explicit stereo matching,” inProc. Intl. Conf. on 3D Vis., 2019, pp. 443–452

2019
[16]

Multimodal tracking framework for visual odometry in challenging illumination conditions,

A. Beauvisage, K. Ahiska, and N. Aouf, “Multimodal tracking framework for visual odometry in challenging illumination conditions,” inProc. IEEE Intl. Conf. on Robot. and Automat., 2020, pp. 11 133–11 139

2020
[17]

Fieldscale: Locality-aware field-based adaptive rescaling for thermal infrared image,

H. Gil, M.-H. Jeon, and A. Kim, “Fieldscale: Locality-aware field-based adaptive rescaling for thermal infrared image,”IEEE Robot. and Automat. Lett., vol. 9, no. 7, pp. 6424–6431, 2024

2024
[18]

Proactive camera attribute control using bayesian optimization for illumination-resilient visual navigation,

J. Kim, Y. Cho, and A. Kim, “Proactive camera attribute control using bayesian optimization for illumination-resilient visual navigation,”IEEE Trans. Robot., vol. 36, no. 4, pp. 1256–1271, 2020

2020
[19]

Unbiased estimator for distorted conics in camera calibration,

C. Song, J. Shin, M.-H. Jeon, J. Lim, and A. Kim, “Unbiased estimator for distorted conics in camera calibration,” inProc. IEEE Conf. on Comput. Vision and Pattern Recog., 2024, pp. 373–381

2024
[20]

Roma: Robust dense feature matching,

J. Edstedt, Q. Sun, G. B ¨okman, M. Wadenb¨ack, and M. Felsberg, “Roma: Robust dense feature matching,” inProc. IEEE Conf. on Comput. Vision and Pattern Recog., 2024, pp. 19 790–19 800

2024
[21]

IMU prein- tegration on manifold for efficient visual-inertial maximum-a-posteriori estimation,

C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “IMU prein- tegration on manifold for efficient visual-inertial maximum-a-posteriori estimation,” inProc. Robot.: Science & Sys. Conf., 2015

2015
[22]

Cart: Caltech aerial rgb-thermal dataset in the wild,

C. Lee, M. Anderson, N. Raganathan, X. Zuo, K. Do, G. Gkioxari, and S.-J. Chung, “Cart: Caltech aerial rgb-thermal dataset in the wild,” 2024

2024
[23]

M2p2: A multi-modal passive perception dataset for off-road mobility in extreme low-light conditions,

A. Datar, A. Pokhrel, M. Nazeri, M. B. Rao, H. Rangwala, C. Pan, Y. Zhang, A. Harrison, M. Wigness, P. R. Osteen, J. Ye, and X. Xiao, “M2p2: A multi-modal passive perception dataset for off-road mobility in extreme low-light conditions,” in2025 IEEE/RSJ International Confer- e...

2025
[24]

Deep depth estimation from thermal image,

U. Shin, J. Park, and I. S. Kweon, “Deep depth estimation from thermal image,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 1043–1053

2023
[25]

evo: Python package for the evaluation of odometry and slam

M. Grupp, “evo: Python package for the evaluation of odometry and slam.” https://github.com/MichaelGrupp/evo, 2017

2017

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.

[1] [1]

Robust multispectral visual- inertial navigation with visual odometry failure recovery,

A. Beauvisage, K. Ahiska, and N. Aouf, “Robust multispectral visual- inertial navigation with visual odometry failure recovery,”IEEE Trans. Intell. Transport. Sys., vol. 23, no. 7, pp. 9089–9101, 2022

2022

[2] [2]

ROVTIO: Robust visual thermal inertial odometry,

H. D. Flemmen, “ROVTIO: Robust visual thermal inertial odometry,” Master’s thesis, NTNU, 2021

2021

[3] [3]

Online temporal calibration for monocular visual- inertial systems,

T. Qin and S. Shen, “Online temporal calibration for monocular visual- inertial systems,” inProc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys., 2018, pp. 3662–3669

2018

[4] [4]

ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,

C. Campos, R. Elvira, J. J. G. Rodr ´ıguez, J. M. Montiel, and J. D. Tard´os, “ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,”IEEE Trans. Robot., vol. 37, no. 6, pp. 1874–1890, 2021

2021

[5] [5]

arXiv preprint arXiv:2202.09199 (2022)

S. Leutenegger, “OKVIS2: Realtime scalable visual-inertial SLAM with loop closure,”arXiv preprint arXiv:2202.09199, 2022

work page arXiv 2022

[6] [6]

DM-VIO: Delayed marginalization visual-inertial odometry,

L. Von Stumberg and D. Cremers, “DM-VIO: Delayed marginalization visual-inertial odometry,”IEEE Robot. and Automat. Lett., vol. 7, no. 2, pp. 1408–1415, 2022

2022

[7] [7]

Direct sparse odometry,

J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,”IEEE Trans. Pattern Analysis and Machine Intell., vol. 40, no. 3, pp. 611–625, 2018

2018

[8] [8]

DeepTIO: A deep thermal-inertial odometry with visual hallucination,

M. R. U. Saputra, P. P. De Gusmao, C. X. Lu, Y. Almalioglu, S. Rosa, C. Chen, J. Wahlstr¨om, W. Wang, A. Markham, and N. Trigoni, “DeepTIO: A deep thermal-inertial odometry with visual hallucination,”IEEE Robot. and Automat. Lett., vol. 5, no. 2, pp. 1672–1679, 2020

2020