pith. sign in

arxiv: 2606.31736 · v1 · pith:SNOZ5SJSnew · submitted 2026-06-30 · 💻 cs.CV

Rhythm-Structured Predictive Learning for Remote Photoplethysmography

Ba-Thinh Nguyen , Huu-Dung Nguyen , Thi-Duyen Ngo , Thanh-Ha Le This is my paper

Pith reviewed 2026-07-01 05:25 UTC · model grok-4.3

classification 💻 cs.CV
keywords remote photoplethysmographyself-supervised learningjoint-embedding predictive architecturecyclic rhythm-state plannerdual order mamba encodermasked video predictionphysiological signal estimationpulse rhythm modeling
0
0 comments X

The pith

RhythmJEPA predicts latent teacher representations from masked facial videos using cyclic state planning and dual-order scanning to learn physiological dynamics for remote photoplethysmography.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that self-supervised rPPG models learn better physiological signals when they predict high-level latent representations from masked videos instead of reconstructing pixels or low-level visuals. Existing approaches risk the model latching onto face appearance rather than the subtle pulse-driven color changes, and most Mamba methods scan only in time order without using the repeating structure of heartbeats. RhythmJEPA adds a Cyclic Rhythm-State Planner that estimates states per frame and finds the best cyclic path with dynamic programming under a transition grammar, plus a Dual Order Mamba Encoder that scans both chronologically and by state order. A Spatial Pulse Mixer then pulls out compact pulse-sensitive tokens. If this holds, rPPG systems could extract heart rate and related signals more reliably from ordinary videos without contact sensors.

Core claim

RhythmJEPA performs joint-embedding predictive learning by training a student network to predict the latent representations produced by a teacher network on masked facial video inputs, rather than reconstructing the RGB frames. The Cyclic Rhythm-State Planner estimates frame-wise latent physiological states and recovers the most plausible cyclic path through dynamic programming constrained by a transition grammar. The Dual Order Mamba Encoder processes tokens in both chronological order and state-ordered sequence to capture local continuity and long-range rhythm-consistent dependencies. The Spatial Pulse Mixer extracts compact pulse-sensitive facial tokens. This combination yields competitiv

What carries the argument

The joint-embedding predictive loss that targets latent teacher representations, combined with the Cyclic Rhythm-State Planner that decodes cyclic state paths via dynamic programming and the Dual Order Mamba Encoder that adds state-ordered scanning to chronological scanning.

If this is right

  • rPPG training can avoid direct pixel reconstruction and its associated bias toward visual appearance.
  • Explicit cyclic state modeling through dynamic programming improves capture of periodic pulse structure beyond simple chronological scanning.
  • Dual-order processing in the encoder combines local temporal continuity with long-range rhythm dependencies.
  • The Spatial Pulse Mixer provides a compact way to isolate pulse-relevant tokens without heavy computation.
  • The overall framework delivers competitive results across multiple public rPPG benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same latent-prediction plus cyclic-planning pattern could be tested on respiration or other periodic signals extractable from video.
  • Relaxing the strict cyclic grammar might allow the planner to handle irregular rhythms seen in clinical recordings.
  • The method's reliance on a pre-trained teacher suggests experiments swapping the teacher for different self-supervised backbones to measure sensitivity.
  • Deployment on mobile devices would require checking whether the dual-order Mamba still runs efficiently at video frame rates.

Load-bearing premise

Predicting latent representations from masked videos will steer the network toward physiological dynamics instead of facial appearance, and dynamic programming on the constrained transition grammar will reliably recover the true cyclic state sequence.

What would settle it

Running the same architecture and training budget with a standard masked pixel reconstruction loss instead of the latent prediction loss and finding equal or higher accuracy on the three evaluation datasets.

Figures

Figures reproduced from arXiv: 2606.31736 by Ba-Thinh Nguyen, Huu-Dung Nguyen, Thanh-Ha Le, Thi-Duyen Ngo.

Figure 1
Figure 1. Figure 1: Stage I: Given a masked facial video, the student en [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stage II: RhythmJEPA fine-tuning. 3. Methodology 3.1. Problem Formulation We represent a facial video clip as X = {Xt} T −1 t=0 , Xt ∈ R H×W×3 . (1) Its synchronized rPPG waveform is represented as y = [y0, . . . , yT −1] ∈ R T . (2) We learn a video-to-waveform mapping: fθ : R T ×H×W×3 → R T , yˆ = fθ(X), (3) where yˆ = [ˆy0, . . . , yˆT −1] denotes the predicted rPPG waveform. As shown in [PITH_FULL_IMA… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative analysis of SPM on PURE and UBFC-rPPG. We jointly visualize the recovered rPPG waveform, its power spectrum, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity analysis of the JEPA mask ratio on PURE [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
read the original abstract

Remote photoplethysmography (rPPG) estimates physiological signals from facial videos by analyzing subtle pulse induced skin color variations. Despite recent progress, existing self-supervised rPPG methods mainly reconstruct masked pixels or low-level visual representations, which can bias the model toward facial appearance rather than latent physiological dy namics. Moreover, most recent Mamba-based approaches scan facial video tokens only in chronological order, limiting their ability to exploit the cyclic structure of pulse signals. To ad dress these limitations, we propose RhythmJEPA, a rhythm structured joint-embedding predictive learning framework for rPPG. Instead of reconstructing RGB frames, RhythmJEPA predicts latent teacher representations from masked facial videos, thereby encouraging physiology-aware representation learning in the embedding space. To explicitly model pulse-related tem poral structure, we introduce a Cyclic Rhythm-State Plan ner (CRSP), which estimates frame-wise latent physiological states and decodes the most plausible cyclic state path via dynamic programming with a constrained transition grammar. Guided by the decoded states, we further design a Dual Order Mamba Encoder (DOM), which combines conventional chronological scanning with state-ordered scanning to capture both local temporal continuity and long-range rhythm-consistent dependencies. Finally, a lightweight Spatial Pulse Mixer (SPM) extracts compact pulse-sensitive facial tokens with a favorable balance between complexity and performance. Experiments on PURE, UBFC-rPPG, and MMPD show competitive performance over representative rPPG methods. The codes are available at https://github.com/deconasser/RhythmJEPA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes RhythmJEPA, a rhythm-structured joint-embedding predictive learning framework for remote photoplethysmography (rPPG). Instead of pixel reconstruction, it predicts latent teacher representations from masked facial videos to encourage physiology-aware embeddings. It introduces a Cyclic Rhythm-State Planner (CRSP) that estimates frame-wise states and decodes the most plausible cyclic path via dynamic programming under a constrained transition grammar; a Dual Order Mamba Encoder (DOM) that performs both chronological and state-ordered scanning; and a Spatial Pulse Mixer (SPM) for compact pulse-sensitive tokens. Experiments on PURE, UBFC-rPPG, and MMPD are reported to show competitive performance versus representative rPPG methods, with code released.

Significance. If the quantitative results and ablations hold, the shift from low-level reconstruction to latent physiological prediction combined with explicit cyclic modeling via CRSP and dual-order Mamba scanning could address key biases in current self-supervised rPPG. Code availability supports reproducibility. However, the absence of any metrics, error bars, dataset statistics, or ablation tables in the visible text prevents assessment of whether these contributions deliver measurable gains.

major comments (2)
  1. Abstract: the central claim that RhythmJEPA 'achieves competitive performance' on PURE, UBFC-rPPG, and MMPD is unsupported by any numerical results, tables, error bars, or baseline comparisons. Without these data the claim cannot be verified and the soundness of the contribution remains unevaluable.
  2. Method description (CRSP and DOM sections): the assumption that predicting latent teacher representations will bias the model toward physiological dynamics rather than appearance, and that the constrained transition grammar plus dynamic programming will reliably recover the most plausible cyclic state path, is stated but not accompanied by any supporting derivation, proof of correctness, or ablation that isolates this effect from facial appearance cues.
minor comments (2)
  1. Abstract: typographical spacing errors ('dy namics', 'tem poral', 'ad dress') should be corrected for readability.
  2. Abstract: dataset statistics (number of subjects, video lengths, frame rates) and exact evaluation metrics (MAE, RMSE, Pearson correlation, etc.) are omitted, hindering direct comparison with prior work.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments. We provide point-by-point responses to the major comments below.

read point-by-point responses

  1. Referee: Abstract: the central claim that RhythmJEPA 'achieves competitive performance' on PURE, UBFC-rPPG, and MMPD is unsupported by any numerical results, tables, error bars, or baseline comparisons. Without these data the claim cannot be verified and the soundness of the contribution remains unevaluable.

    Authors: We agree that the abstract claim would be stronger with explicit numerical support. The full manuscript reports detailed results, tables with MAE/RMSE/Pearson metrics, error bars, and baseline comparisons in the Experiments section. To address the concern directly, we will revise the abstract to include key quantitative highlights (specific performance values and brief baseline notes) so the claim is verifiable from the abstract. revision: yes

  2. Referee: Method description (CRSP and DOM sections): the assumption that predicting latent teacher representations will bias the model toward physiological dynamics rather than appearance, and that the constrained transition grammar plus dynamic programming will reliably recover the most plausible cyclic state path, is stated but not accompanied by any supporting derivation, proof of correctness, or ablation that isolates this effect from facial appearance cues.

    Authors: The components are motivated by the cyclic physiology of pulse signals and the goal of shifting from appearance to latent dynamics. We acknowledge that the current text would benefit from stronger justification. We will revise the Method section to expand the rationale for the latent-prediction bias and the transition grammar, and add ablation studies that isolate CRSP/DOM effects (with and without the rhythm components) from appearance cues. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The manuscript abstract and context introduce RhythmJEPA as a joint-embedding predictive framework that predicts latent teacher representations rather than pixels, augmented by CRSP for cyclic state planning via dynamic programming and DOM for dual-order scanning. No equations, parameter-fitting steps, or self-citation chains are present in the supplied text that would reduce any claimed prediction or uniqueness result to a tautology or fitted input. The performance claims are benchmarked against external datasets (PURE, UBFC-rPPG, MMPD) and are therefore falsifiable outside the model's own definitions, satisfying the criteria for a self-contained derivation with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The method implicitly assumes that pulse signals possess recoverable cyclic state structure and that masked-video prediction in embedding space isolates physiology from appearance.

pith-pipeline@v0.9.1-grok · 5812 in / 1257 out tokens · 24135 ms · 2026-07-01T05:25:53.617596+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bo- janowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15619–15629, 2023. 2

    2023

  2. [2]

    Revisiting Feature Prediction for Learning Visual Representations from Video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learn- ing visual representations from video.arXiv preprint arXiv:2404.08471, 2024. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Unsupervised skin tissue seg- mentation for remote photoplethysmography.Pattern recog- nition letters, 124:82–90, 2019

    Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsupervised skin tissue seg- mentation for remote photoplethysmography.Pattern recog- nition letters, 124:82–90, 2019. 2, 5, 6

    2019

  4. [4]

    Deepphys: Video- based physiological measurement using convolutional atten- tion networks

    Weixuan Chen and Daniel McDuff. Deepphys: Video- based physiological measurement using convolutional atten- tion networks. InProceedings of the european conference on computer vision (ECCV), pages 349–365, 2018. 1, 2, 6

    2018

  5. [5]

    Robust pulse rate from chrominance-based rppg.IEEE transactions on biomedical engineering, 60(10):2878–2886, 2013

    Gerard De Haan and Vincent Jeanne. Robust pulse rate from chrominance-based rppg.IEEE transactions on biomedical engineering, 60(10):2878–2886, 2013. 1, 2, 6

    2013

  6. [6]

    Retinaface: Single-shot multi- level face localisation in the wild

    Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kot- sia, and Stefanos Zafeiriou. Retinaface: Single-shot multi- level face localisation in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203–5212, 2020. 5

    2020

  7. [7]

    Fan: Fourier analysis networks.Advances in Neural Information Processing Systems, 38:66267–66296,

    Yihong Dong, Ge Li, Yongding Tao, Xue Jiang, Kechi Zhang, Jia Li, Jinliang Deng, Jing Su, Jun Zhang, and Jingjing Xu. Fan: Fourier analysis networks.Advances in Neural Information Processing Systems, 38:66267–66296,

  8. [8]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Challenges and prospects of visual contactless physiologi- cal monitoring in clinical study.NPJ Digital Medicine, 6(1): 231, 2023

    Bin Huang, Shen Hu, Zimeng Liu, Chun-Liang Lin, Jun- feng Su, Changchen Zhao, Li Wang, and Wenjin Wang. Challenges and prospects of visual contactless physiologi- cal monitoring in clinical study.NPJ Digital Medicine, 6(1): 231, 2023. 1

    2023

  10. [10]

    Titong Jiang, Yuan Ma, Jiaqi Li, Qing Dong, Xuewu Ji, and Yahui Liu. Lsts: Periodicity learning via long short-term temporal shift for remote physiological measurement.IEEE Transactions on Circuits and Systems for Video Technology, 35(7):6452–6465, 2025. 2, 6, 7, 8

    2025

  11. [11]

    Measuring pulse rate with a webcam.Journal of Medical Imaging and Health Informatics, 2(1):87–92, 2012

    Magdalena Lewandowska and J˛ edrzej Nowak. Measuring pulse rate with a webcam.Journal of Medical Imaging and Health Informatics, 2(1):87–92, 2012. 2

    2012

  12. [12]

    Multi-task temporal shift attention networks for on-device contactless vitals measurement.Advances in Neural Infor- mation Processing Systems, 33:19400–19411, 2020

    Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention networks for on-device contactless vitals measurement.Advances in Neural Infor- mation Processing Systems, 33:19400–19411, 2020. 1, 2, 6, 8

    2020

  13. [13]

    Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement

    Xin Liu, Brian Hill, Ziheng Jiang, Shwetak Patel, and Daniel McDuff. Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 5008–5017, 2023. 1, 2, 5, 6, 7, 8

    2023

  14. [14]

    rppg-mae: Self-supervised pretraining with masked autoencoders for remote physiological mea- surements.IEEE Transactions on Multimedia, 26:7278– 7293, 2024

    Xin Liu, Yuting Zhang, Zitong Yu, Hao Lu, Huanjing Yue, and Jingyu Yang. rppg-mae: Self-supervised pretraining with masked autoencoders for remote physiological mea- surements.IEEE Transactions on Multimedia, 26:7278– 7293, 2024. 1, 2

    2024

  15. [15]

    Vmamba: Visual state space model.Advances in neural information processing systems, 37:103031–103063, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model.Advances in neural information processing systems, 37:103031–103063, 2024. 2

    2024

  16. [16]

    Physmamba: Effi- cient remote physiological measurement with slowfast tem- poral difference mamba

    Chaoqi Luo, Yiping Xie, and Zitong Yu. Physmamba: Effi- cient remote physiological measurement with slowfast tem- poral difference mamba. InChinese Conference on Biomet- ric Recognition, pages 248–259. Springer, 2024. 1, 2

    2024

  17. [17]

    Ba-Thinh Nguyen, Thach-Ha Ngoc Pham, Hoang-Long Duc Nguyen, Thi-Duyen Ngo, and Thanh-Ha Le. Reperio-rppg: Relational temporal graph neural networks for periodicity learning in remote physiological measurement.Journal of Visual Communication and Image Representation, page 104823, 2026. 1, 2, 5, 7, 8 PREPRINT

    2026

  18. [18]

    Local group invariance for heart rate esti- mation from face videos in the wild

    Christian S Pilz, Sebastian Zaunseder, Jarek Krajewski, and Vladimir Blazek. Local group invariance for heart rate esti- mation from face videos in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1254–1262, 2018. 2, 6

    2018

  19. [19]

    Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.Optics express, 18(10):10762–10774, 2010

    Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.Optics express, 18(10):10762–10774, 2010. 1, 2

    2010

  20. [20]

    Rethinking rppg su- pervision: Distribution and temporal relation alignment for domain generalization

    Wei Qian, Zhipeng Li, Yuchen Ding, Xiangyuan Jia, Gaoji Su, Fei Wang, Kun Li, and Meng Wang. Rethinking rppg su- pervision: Distribution and temporal relation alignment for domain generalization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1338–1348, 2026. 4

    2026

  21. [21]

    Rs+ rppg: Robust strongly self-supervised learning for rppg.IEEE Transactions on Cir- cuits and Systems for Video Technology, 2025

    Marko Savic and Guoying Zhao. Rs+ rppg: Robust strongly self-supervised learning for rppg.IEEE Transactions on Cir- cuits and Systems for Video Technology, 2025. 1, 2

    2025

  22. [22]

    Super-convergence: Very fast training of neural networks using large learn- ing rates

    Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learn- ing rates. InArtificial intelligence and machine learning for multi-domain operations applications, pages 369–386. SPIE, 2019. 5

    2019

  23. [23]

    Non-contact video-based pulse rate measurement on a mo- bile service robot

    Ronny Stricker, Steffen Müller, and Horst-Michael Gross. Non-contact video-based pulse rate measurement on a mo- bile service robot. InThe 23rd IEEE international sympo- sium on robot and human interactive communication, pages 1056–1062. IEEE, 2014. 2, 5, 6

    2014

  24. [24]

    Zhaodong Sun and Xiaobai Li. Contrast-phys+: Unsuper- vised and weakly-supervised video-based remote physiolog- ical measurement via spatiotemporal contrast.IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 46(8): 5835–5851, 2024. 1, 6

    2024

  25. [25]

    Mmpd: Multi- domain mobile video physiology dataset

    Jiankai Tang, Kequan Chen, Yuntao Wang, Yuanchun Shi, Shwetak Patel, Daniel McDuff, and Xin Liu. Mmpd: Multi- domain mobile video physiology dataset. In2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 1–5. IEEE,

  26. [26]

    Re- mote plethysmographic imaging using ambient light.Optics express, 16(26):21434–21445, 2008

    Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. Re- mote plethysmographic imaging using ambient light.Optics express, 16(26):21434–21445, 2008. 1

    2008

  27. [27]

    Algorithmic principles of remote ppg

    Wenjin Wang, Albertus C Den Brinker, Sander Stuijk, and Gerard De Haan. Algorithmic principles of remote ppg. IEEE Transactions on Biomedical Engineering, 64(7):1479– 1491, 2016. 1, 2, 6

    2016

  28. [28]

    Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,

    Zitong Yu, Xiaobai Li, and Guoying Zhao. Re- mote photoplethysmograph signal measurement from fa- cial videos using spatio-temporal networks.arXiv preprint arXiv:1905.02419, 2019. 1, 2, 6

    work page arXiv 1905

  29. [29]

    Physformer: Facial video-based physiological measurement with temporal dif- ference transformer

    Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip HS Torr, and Guoying Zhao. Physformer: Facial video-based physiological measurement with temporal dif- ference transformer. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 4186–4196, 2022. 1, 2, 6, 7, 8

    2022

  30. [30]

    Fvbpsr- mamba: Facial video-based physiological signal recovery using mamba for remote heart rate measurement.Biomed- ical Signal Processing and Control, 123:110523, 2026

    Baihua Zhang, Yan Gao, and Yudong Yao. Fvbpsr- mamba: Facial video-based physiological signal recovery using mamba for remote heart rate measurement.Biomed- ical Signal Processing and Control, 123:110523, 2026. 2

    2026

  31. [31]

    Yizhu Zhang, Jingang Shi, Jiayin Wang, Yuan Zong, Wen- ming Zheng, and Guoying Zhao. Maskfusionnet: A dual- stream fusion model with masked pre-training mechanism for rppg measurement.IEEE Transactions on Circuits and Systems for Video Technology, 34(11):11521–11534, 2024. 1, 2

    2024

  32. [32]

    Rhythmformer: Extract- ing patterned rppg signals based on periodic sparse attention

    Bochao Zou, Zizheng Guo, Jiansheng Chen, Junbao Zhuo, Weiran Huang, and Huimin Ma. Rhythmformer: Extract- ing patterned rppg signals based on periodic sparse attention. Pattern Recognition, 164:111511, 2025. 1, 2, 5, 6

    2025

  33. [33]

    Rhythmmamba: Fast, lightweight, and accurate remote physiological measurement

    Bochao Zou, Zizheng Guo, Xiaocheng Hu, and Huimin Ma. Rhythmmamba: Fast, lightweight, and accurate remote physiological measurement. InProceedings of the AAAI Conference on Artificial Intelligence, pages 11077–11085,