pith. sign in

arxiv: 2606.21115 · v1 · pith:LQBHUDAKnew · submitted 2026-06-19 · 💻 cs.CV · cs.AI· eess.IV

MS-rPPG: Multi-spectral State Space Model for Remote Photoplethysmography in Driver Monitoring Systems

Jiho Choi , Sang Jun Lee This is my paper

Pith reviewed 2026-06-26 14:44 UTC · model grok-4.3

classification 💻 cs.CV cs.AIeess.IV
keywords remote photoplethysmographymulti-spectral imagingstate space modelheart rate estimationdriver monitoringnear-infrared videoMamba architecture
0
0 comments X

The pith

Multi-spectral RGB-NIR fusion with state space modeling yields more accurate remote heart rate estimates during driving.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents MS-rPPG, a framework that processes paired RGB and near-infrared face videos to estimate heart rate from drivers despite changing cabin light and head motion. It introduces cross-spectral linear modulation to blend frequency-domain information across spectra and MS-Mamba, a state-space architecture that jointly tracks long temporal sequences and cross-channel feature interactions. Experiments on the public MR-NIRP Car dataset and a newly collected MS-Drive set of 50 real driving sessions show higher accuracy and robustness than earlier single-spectrum or convolutional approaches. The work targets continuous, camera-only physiological monitoring inside vehicles where contact sensors are impractical.

Core claim

MS-rPPG combines RGB and NIR face videos using a cross-spectral linear modulation strategy derived from frequency-domain analysis together with the MS-Mamba state-space model to produce heart-rate estimates that remain accurate under the illumination shifts and head movements typical of real driving.

What carries the argument

MS-Mamba, a multi-spectral extension of the state-space model that processes long-range temporal dependencies while exchanging information across RGB and NIR channels, paired with cross-spectral linear modulation that aligns features in the frequency domain.

If this is right

  • Heart-rate signals extracted from vehicle cameras become reliable enough to support real-time driver health alerts without added hardware.
  • Combining visible and near-infrared channels supplies complementary pulsatile information that single-spectrum methods miss under motion or varying light.
  • State-space models can replace recurrent or convolutional backbones for long video sequences in physiological signal recovery.
  • The same multi-spectral pipeline can be retrained on other vehicle-camera datasets to monitor additional vital signs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the frequency-domain modulation generalizes, the same alignment step could be applied to other multi-camera physiological setups such as RGB plus thermal.
  • Embedding MS-Mamba into existing vehicle perception stacks would allow heart-rate monitoring to run on the same compute budget already allocated to driver attention systems.
  • The MS-Drive collection protocol could be replicated by other groups to create standardized benchmarks for in-car rPPG under naturalistic motion.

Load-bearing premise

The performance gains from cross-spectral linear modulation and MS-Mamba will persist when illumination patterns, head-motion statistics, or participant demographics differ markedly from those recorded in the MR-NIRP Car and MS-Drive collections.

What would settle it

Collecting a new driving dataset under substantially different cabin lighting or with a different age or skin-tone distribution and finding that MS-rPPG accuracy falls below the best prior single-spectrum baseline.

Figures

Figures reproduced from arXiv: 2606.21115 by Jiho Choi, Sang Jun Lee.

Figure 1
Figure 1. Figure 1: Comparison with recent Mamba-based rPPG frameworks [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of proposed framework. Our method integrates RGB and NIR face videos for robust remote physiological measurement under real-world [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of the Mamba [13] and the proposed CSLM. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example images from the MS-Drive dataset, which contains simultaneously captured RGB, NIR, and reliable ECG sensor data. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Number of subjects per Fitzpatrick skin type in the MS-Drive [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of test results on the MR-NIRP dataset, showing the ground-truth and predicted PPG signals in red and blue, respectively. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cross-dataset evaluation results of MS-rPPG on the MS-Drive [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
read the original abstract

Remote photoplethysmography (rPPG) is a camera-based technique for measuring physiological signals, particularly cardiac activity. From the remotely measured signals, heart rate can be estimated, which is crucial for health monitoring. In this study, we investigate a driver health monitoring system based on remote heart rate estimation. However, driving environments represent uncontrolled settings where videos are subject to varying illumination conditions and frequent head movements. We introduce MS-rPPG, a multi-spectral framework that combines RGB with near-infrared (NIR) face video to alleviate rPPG estimation under challenging driving conditions. To combine the complementary features from two spectral videos, we propose a cross-spectral linear modulation (CSLM) strategy based on frequency-domain analysis. Moreover, we introduce MS-Mamba, a novel state space model designed to effectively model long-range temporal dependencies while jointly capturing cross-channel interactions between multi-spectral features. We collected a real-world dataset called MS-Drive, which was recorded from 50 participants while driving the vehicle. The proposed method was evaluated on the MR-NIRP Car dataset and MS-Drive datasets. The experimental results indicate that MS-rPPG shows better robustness and heart rate estimation accuracy than previous methods, highlighting its promise for driver health monitoring. The codes are available at github.com/ziiho08/MS-rPPG.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper proposes MS-rPPG, a multi-spectral framework for remote photoplethysmography (rPPG) in driver monitoring systems. It integrates RGB and near-infrared (NIR) video using a cross-spectral linear modulation (CSLM) strategy based on frequency-domain analysis and introduces the MS-Mamba state space model to model long-range temporal dependencies and cross-channel interactions. A new dataset MS-Drive with 50 participants is collected, and the method is evaluated on MR-NIRP Car and MS-Drive datasets, claiming improved robustness and heart rate estimation accuracy over previous methods. Code is made available.

Significance. If the reported performance improvements are validated, this work could advance the field of non-contact physiological signal measurement in uncontrolled environments such as driving. The combination of multi-spectral data and state space models addresses key challenges like varying illumination and head movements. The introduction of a new real-world dataset and open-sourcing of code are notable strengths that could facilitate further research.

major comments (1)
  1. [Experimental results] Experimental results section: The central claim that MS-rPPG shows better robustness and heart rate estimation accuracy than previous methods is unsupported by any reported quantitative metrics (MAE, RMSE, etc.), baseline implementations, ablation tables, error bars, or statistical tests. Without these, it is impossible to determine whether gains arise from CSLM + MS-Mamba, the NIR channel, or evaluation choices.
minor comments (2)
  1. The abstract states evaluation on MR-NIRP Car and MS-Drive but provides no details on participant demographics, illumination conditions, or head motion ranges in the test sets.
  2. Clarify how the frequency-domain analysis in CSLM is implemented and whether it introduces any tunable parameters.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for stronger quantitative evidence in the experimental results. We address this point directly below.

read point-by-point responses

  1. Referee: [Experimental results] Experimental results section: The central claim that MS-rPPG shows better robustness and heart rate estimation accuracy than previous methods is unsupported by any reported quantitative metrics (MAE, RMSE, etc.), baseline implementations, ablation tables, error bars, or statistical tests. Without these, it is impossible to determine whether gains arise from CSLM + MS-Mamba, the NIR channel, or evaluation choices.

    Authors: We agree that the current manuscript does not provide the requested quantitative details to fully substantiate the claims. In the revised version we will add comprehensive tables reporting MAE, RMSE, and correlation metrics on both MR-NIRP Car and MS-Drive, direct comparisons against re-implemented baselines, ablation studies isolating CSLM and MS-Mamba, error bars across multiple runs or subjects, and statistical tests (e.g., paired t-tests) to demonstrate significance. These additions will allow readers to assess the source of the reported improvements. revision: yes

Circularity Check

0 steps flagged

No circularity: new architecture and dataset yield independent empirical claims

full rationale

The provided abstract and context describe a new multi-spectral framework (CSLM + MS-Mamba), a newly collected MS-Drive dataset from 50 participants, and evaluation on MR-NIRP Car plus MS-Drive. No equations, fitted parameters, self-citations, or uniqueness theorems are quoted that would reduce any performance number to a definition or prior fit by construction. The reported robustness and accuracy gains are presented as experimental outcomes on external benchmarks, with code released for reproducibility. This satisfies the default expectation of a self-contained empirical contribution without load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no explicit free parameters, axioms, or invented entities beyond the high-level claim that the new components improve performance; the ledger is therefore empty pending full text.

pith-pipeline@v0.9.1-grok · 5778 in / 1035 out tokens · 16095 ms · 2026-06-26T14:44:58.473587+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Contactless health monitoring: An overview of video-based techniques utilising machine/deep learning,

    A. Hajr, B. Tarvirdizadeh, K. Alipour, and M. Ghamari, “Contactless health monitoring: An overview of video-based techniques utilising machine/deep learning,”IET Wireless Sensor Systems, vol. 15, no. 1, p. e70009, 2025

    2025

  2. [2]

    A comparative study of principled rppg-based pulse rate tracking algorithms for fitness activities,

    Q. Zhu, C.-W. Wong, Z. M. Lazri, M. Chen, C.-H. Fu, and M. Wu, “A comparative study of principled rppg-based pulse rate tracking algorithms for fitness activities,”IEEE Transactions on Biomedical Engineering, 2024

    2024

  3. [3]

    End-to-end multimodal emotion recognition based on facial expressions and remote photoplethysmography signals,

    J. Li and J. Peng, “End-to-end multimodal emotion recognition based on facial expressions and remote photoplethysmography signals,” IEEE Journal of Biomedical and Health Informatics, 2024

    2024

  4. [4]

    Facial-video-based physiological signal measurement: Recent advances and affective applications,

    Z. Yu, X. Li, and G. Zhao, “Facial-video-based physiological signal measurement: Recent advances and affective applications,”IEEE Sig- nal Processing Magazine, vol. 38, no. 6, pp. 50–58, 2021

    2021

  5. [5]

    Sparseppg: Towards driver monitoring using camera-based vital signs estimation in near-infrared,

    E. Magdalena Nowara, T. K. Marks, H. Mansour, and A. Veer- araghavan, “Sparseppg: Towards driver monitoring using camera-based vital signs estimation in near-infrared,” inProceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 1272–1281

    2018

  6. [6]

    Ai innovations in rppg systems for driver monitoring: Comprehensive systematic review and future prospects,

    S. G. Ahmed, K. Verbert, N. Siedahmed, A. Khalil, H. Aljassmi, and F. Alnajjar, “Ai innovations in rppg systems for driver monitoring: Comprehensive systematic review and future prospects,”Ieee Access, 2025

    2025

  7. [7]

    Robust pulse rate from chrominance- based rppg,

    G. De Haan and V . Jeanne, “Robust pulse rate from chrominance- based rppg,”IEEE transactions on biomedical engineering, vol. 60, no. 10, pp. 2878–2886, 2013

    2013

  8. [8]

    Detecting pulse from head motions in video,

    G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 3430–3437

    2013

  9. [9]

    Deepphys: Video-based physiological measurement using convolutional attention networks,

    W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” inProceedings of the european conference on computer vision (ECCV), 2018, pp. 349–365

    2018

  10. [10]

    Efficientphys: En- abling simple, fast and accurate camera-based cardiac measurement,

    X. Liu, B. Hill, Z. Jiang, S. Patel, and D. McDuff, “Efficientphys: En- abling simple, fast and accurate camera-based cardiac measurement,” inProceedings of the IEEE/CVF winter conference on applications of computer vision, 2023, pp. 5008–5017

    2023

  11. [11]

    Phys- former: Facial video-based physiological measurement with temporal difference transformer,

    Z. Yu, Y . Shen, J. Shi, H. Zhao, P. H. Torr, and G. Zhao, “Phys- former: Facial video-based physiological measurement with temporal difference transformer,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 4186–4196

    2022

  12. [12]

    Rhythmformer: Extracting rppg signals based on hierarchical temporal periodic transformer,

    B. Zou, Z. Guo, J. Chen, and H. Ma, “Rhythmformer: Extracting rppg signals based on hierarchical temporal periodic transformer,”CoRR, 2024

    2024

  13. [13]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    Physmamba: Efficient remote physio- logical measurement with slowfast temporal difference mamba,

    C. Luo, Y . Xie, and Z. Yu, “Physmamba: Efficient remote physio- logical measurement with slowfast temporal difference mamba,” in Chinese Conference on Biometric Recognition. Springer, 2024, pp. 248–259

    2024

  15. [15]

    Rhythmmamba: Fast, lightweight, and accurate remote physiological measurement,

    B. Zou, Z. Guo, X. Hu, and H. Ma, “Rhythmmamba: Fast, lightweight, and accurate remote physiological measurement,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 10, 2025, pp. 11 077–11 085

    2025

  16. [16]

    Blending camera and 77 ghz radar sensing for equitable, robust plethysmography

    A. Vilesov, P. Chari, A. Armouti, A. B. Harish, K. Kulkarni, A. De- oghare, L. Jalilian, and A. Kadambi, “Blending camera and 77 ghz radar sensing for equitable, robust plethysmography.”ACM Trans. Graph., vol. 41, no. 4, pp. 36–1, 2022

    2022

  17. [17]

    Non-contact, automated cardiac pulse measurements using video imaging and blind source separation

    M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.”Optics express, vol. 18, no. 10, pp. 10 762–10 774, 2010

    2010

  18. [18]

    Algo- rithmic principles of remote ppg,

    W. Wang, A. C. Den Brinker, S. Stuijk, and G. De Haan, “Algo- rithmic principles of remote ppg,”IEEE Transactions on Biomedical Engineering, vol. 64, no. 7, pp. 1479–1491, 2016

    2016

  19. [19]

    Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,

    Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” arXiv preprint arXiv:1905.02419, 2019

    work page arXiv 1905

  20. [20]

    Self-supervised rgb-nir fusion video vision transformer framework for rppg estimation,

    S. Park, B.-K. Kim, and S.-Y . Dong, “Self-supervised rgb-nir fusion video vision transformer framework for rppg estimation,”IEEE Trans- actions on Instrumentation and Measurement, vol. 71, pp. 1–10, 2022

    2022

  21. [21]

    Memory- efficient low-latency remote photoplethysmography through temporal- spatial state space duality,

    K. Wang, J. Tang, Y . Fan, J. Ji, Y . Shi, and Y . Wang, “Memory- efficient low-latency remote photoplethysmography through temporal- spatial state space duality,”arXiv preprint arXiv:2504.01774, 2025

    work page arXiv 2025

  22. [22]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” inProceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018

    2018

  23. [23]

    The validity and practicality of sun-reactive skin types i through vi,

    T. B. Fitzpatrick, “The validity and practicality of sun-reactive skin types i through vi,”Archives of dermatology, vol. 124, no. 6, pp. 869– 871, 1988

    1988

  24. [24]

    rppg-toolbox: Deep remote ppg toolbox,

    X. Liu, G. Narayanswamy, A. Paruchuri, X. Zhang, J. Tang, Y . Zhang, R. Sengupta, S. Patel, Y . Wang, and D. McDuff, “rppg-toolbox: Deep remote ppg toolbox,”Advances in Neural Information Processing Systems, vol. 36, pp. 68 485–68 510, 2023

    2023