Pith. sign in

REVIEW 4 major objections 7 minor 39 references

Heart rate and respiratory rate prediction from noisy real-world smartphone based on Deep Learning methods

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read On noisy, real-world smartphone fingertip videos, a purpose-built 3D convolutional network estimates heart rate and respiratory rate more accurately than a standard signal-processing baseline, cutting RMS error by 68% and 75%.

desk verdict A genuinely new real-world fingertip video dataset and a plausible deep learning win over EEMD-PCA, but the headline 68%/75% error reductions rest on a best-of-four-fold point estimate over 15 test videos with no confidence intervals. read the letter →

arxiv 2506.22460 v1 pith:ICQCSMNK submitted 2025-06-17 eess.SP cs.AI

classification eess.SPcs.AI
keywords vitalsignsdeeplearningregressionmobilephonesmHealthphotoplethysmography3Dconvolutionalneuralnetworkfingertipvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mobile fingertip videos are a cheap source of vital signs, but past accuracy claims were mostly measured in controlled lab conditions. This paper tests that assumption with smartphone videos collected in everyday life, finds that a standard signal-processing method performs much worse there, and shows that a purpose-built 3D convolutional network (DVR 3) cuts heart-rate RMS error from 9.02 to 2.92 beats/min and respiratory-rate RMS error from 19.62 to 4.80 breaths/min—a 68% and 75% reduction. The result suggests regressor-based deep learning, rather than hand-crafted spectral analysis, should be the basis for real-world smartphone vital-sign estimation. The paper also introduces the quality-filtered dataset of 71 videos from 46 subjects used for training and testing.

What carries the argument

The central object is the Deep Video Regression 3 (DVR 3) network, a C3D-inspired 3D convolutional neural network built for temporal counting. It takes 360 frames (24 seconds at 30 fps) of $32\times32\times1$ video—either the red channel or a grayscale composite—and stacks 3D convolutions and max pools whose temporal-depth kernels shrink from 90 to 10, giving the network a long temporal context in which to count color oscillations. Splitting each convolution into two layers with $1\times1$ filters cuts trainable parameters from 470.6M in DVR 2 to 157.5M in DVR 3 while keeping the same error. The work it does is replacing hand-crafted spectral decomposition (EEMD-PCA) with a learned mapping from raw spatiotemporal color variation to beats and breaths per minute.

What would settle it

Collect a new uncontrolled smartphone-fingertip dataset where heart rate is recorded by ECG and respiratory rate by capnography or respiratory inductance plethysmography, then run DVR 3 against EEMD-PCA under the paper's training protocol. If the respiratory-rate RMS advantage over EEMD-PCA disappears once labels are reference-grade, or if HR RMS rises above the ~4 beats/min label-noise floor, the central claim would be refuted.

Watch

Extended reading notes

Core claim

On the quality-filtered subset of a new real-world dataset—71 fingertip videos from 46 participants, recorded without experimenter supervision before and after exercise—the Deep Video Regression 3 (DVR 3) network predicts heart rate with a root-mean-square error of 2.92 beats/min and respiratory rate with an RMS error of 4.80 breaths/min. Against the EEMD-PCA signal-processing baseline on the same test videos, those are reductions of 68% and 75% respectively (from 9.02 beats/min and 19.62 breaths/min). The paper's central claim is that this gap reflects the failure of laboratory-tuned spectral methods to generalize to noisy daily-life video, and that a 3D CNN that learns to count color-variation frequency from raw frames is the more robust approach. The HR error approaches the roughly 4 beats/min error of the manual pulse-count labels themselves, while RR remains the harder target.

Load-bearing premise

The load-bearing premise is that the manual labels are accurate enough to train and evaluate against: heart rate was counted as radial/carotid pulse over 30 seconds (a method with about 4 beats/min error), and respiratory rate was self-counted by participants with no independent validation.

Editorial extensions

If this is right

  • A regressor-based deep network should replace spectral decomposition as the default estimator for HR and RR from fingertip phone video in uncontrolled settings.
  • Heart-rate predictions at 2.92 BPM RMS are close to the ~4 BPM error of the 30-second palpation labels, so further HR improvements will likely require better ground truth, not just better networks.
  • Respiratory rate remains the limiting vital sign; the paper's analysis points to longer windows (30 seconds) and explicitly modeling respiratory sinus arrhythmia and baseline/amplitude modulation as next steps.
  • Any practical deployment needs an automated quality gate, since 56% of collected videos lacked a detectable pulse and were discarded.
  • Predicting HR and RR jointly reduces the RR error bias but costs HR accuracy (RMS rises from 2.92 to 6.36 BPM), so the choice between single-task and multi-task models depends on the use case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the reported numbers come from the best of four validation folds evaluated on 15 held-out videos, a single fixed model would likely show a smaller gain on a new population; a pre-registered single-model evaluation would give the deployable expectation.
  • Beyond the paper: the 44% retention rate means the reported accuracies apply only to videos with a clear pulsatile signal; end-to-end performance including rejected videos is likely lower and should be measured.
  • Beyond the paper: a natural testable extension is the same architecture on 30-second windows, which the paper itself suggests; if temporal context is the limiting factor for RR, longer input should raise RR accuracy.
  • Beyond the paper: the learned color-variation-frequency representation might transfer to other PPG-like inputs such as webcam facial video or wearable optical sensors, since the network does not assume a particular sensor geometry.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The manuscript proposes a deep 3D convolutional architecture (DVR 1/2/3) for estimating heart rate (HR) and respiratory rate (RR) from 30-second smartphone fingertip videos recorded in uncontrolled daily-life settings. A dataset of 161 videos was collected from participants who self-counted RR while an expert measured HR by pulse palpation; after automated quality filtering, 71 videos from 46 subjects remained. The authors hold out 15 videos by subject, train with four-fold validation on the remaining data, and select the validation-best fold as the final model. They report that the proposed method reduces HR RMS error from 9.02 to 2.92 beats/min (68%) and RR RMS error from 19.62 to 4.80 breaths/min (75%) relative to an EEMD-PCA baseline, and they additionally report single-task and simultaneous HR/RR prediction results with Bland-Altman and correlation plots.

Significance. If the point estimates are reliable, this is a useful real-world stress test of camera-based vital-sign estimation: it shows that a classical EEMD-PCA method degrades sharply on uncontrolled smartphone video and that a purpose-built 3D CNN can recover much of that gap. The subject-disjoint split and the direct comparison against a published signal-processing baseline are strengths, and the paper is explicit about label-adjustment procedures and augmentation choices. However, the central quantitative claim rests on 15 test videos with best-of-four-fold model selection and no uncertainty quantification, and the RR ground truth is self-counted with no validation. The result is therefore promising but not yet statistically secured; the contribution is closer to a benchmark report than a definitive demonstration of the stated generalization to daily life.

major comments (4)
  1. [Section 4.2, Tables 3 and 5] The headline 68% and 75% error reductions are point estimates from the validation-best of four cross-validation folds evaluated on 15 held-out test videos. With n = 15, the RMS is highly sensitive to individual clips, and selecting the fold with the best validation loss adds optimistic selection bias. No per-fold test errors, confidence intervals, bootstrap estimates, or significance tests are reported, so the central claim is not statistically quantified. Please report per-fold test results, confidence intervals for the test RMS differences, and ideally repeated subject-disjoint splits or leave-one-subject-out validation.
  2. [Section 3.1] The RR ground truth is self-counted by participants and doubled, with no validation of this procedure, while the HR label uncertainty is at least quantified by the cited MAE of slightly over 4 BPM. Because these labels serve as both training targets and the evaluation reference, label noise or bias in RR directly contaminates the reported 75% RR reduction. A label-quality assessment (e.g., expert re-counting on a subset, duplicate measurements, or a sensitivity analysis) is needed before the RR comparison can be considered reliable. In addition, the participant counts are inconsistent: the abstract states N=111, Section 3.1 states N=105, and Section 4.2 uses 52 subjects while Section 3.3 reports 46 unique subjects.
  3. [Sections 3.2 and 5 (Discussion)] The automated quality filter discarded 90 of 161 videos (56%) for lacking a detectable pulsatile pattern, and the evaluation is performed only on the remaining 71 high-quality clips. The conclusion that the method works on noisy real-world daily-life video is therefore limited to the 44% of recordings that pass quality filtering. The manuscript should either report performance on the unfiltered set or explicitly characterize the deployment-time filtering step, its accuracy, and the failure rate; otherwise the abstract's generalization claim overstates what is demonstrated.
  4. [Section 4.3, Tables 5 and 7] The text states that the simultaneous predictor improves RR performance slightly (RMS of 5.20 vs. 4.80 breaths/min) compared with predicting RR alone, but 5.20 is worse than 4.80, and Table 5 reports the single-task red-channel model at 4.80. This contradicts the reported direction of the comparison and should be corrected, along with a clearer explanation of why simultaneous training is beneficial when it appears to hurt HR performance.
minor comments (7)
  1. [Abstract and Section 3.1] The number of participants differs between the abstract (N=111) and the data collection section (N=105); this should be reconciled.
  2. [Discussion] The Discussion says the RR RMS of 5 breaths/min is shown in Table 3, but the RR single-task results appear in Table 5; the cross-reference is incorrect.
  3. [Table 1] Table 1 repeats 'DVR 1' three times in the header; the three columns should be labeled DVR 1, DVR 2, and DVR 3 so that the parameter and training-time comparisons are interpretable.
  4. [Section 4.2] The statement that K=4 was chosen to provide approximately a 55-25-20 train-validation-test split is arithmetically unclear; a four-fold split on the 80% training pool gives roughly 60-20-20 splits, not 55-25-20, and the sampling procedure from HR×RR tiers should be described more precisely.
  5. [Throughout] The phrase 'fingerprint videos' appears in the Introduction and should be 'fingertip videos' for consistency with the rest of the paper.
  6. [Tables 3, 5, and 7] The table captions say 'DVR 2 and 3' but the architecture described in the text is DVR 3; no separate results for DVR 2 versus DVR 3 are presented, so the captions should state which architecture produced the numbers.
  7. [Section 4.2] The training section says both 'Lookahead' and 'DP-SGD' were used as the optimizer; unless these are composed in a specific way, the description is contradictory and should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline HR/RR improvements are an external benchmark comparison on subject-disjoint held-out test video, not a restatement of training inputs.

full rationale

The central claim is an empirical benchmark comparison, not a derivation. The DVR models are trained with a mean-squared-error loss on labeled videos, and the reported HR RMS of 2.92 and RR RMS of 4.80 are evaluated on 15 held-out test videos from 9 subjects who were excluded from training. No equation in the paper defines the predicted HR or RR in terms of the ground-truth labels or the test statistics; the models take video frames as input and output a single real number. The EEMD-PCA comparator is an external algorithm taken from Motin et al. and is applied to the same test videos, so its errors are not fitted or constructed to make the deep network look better. The tunable choices identified in the paper—the four-fold validation-based model selection, augmentation parameters, and the 0.75/0.25 simultaneous-loss weights—are set without access to the held-out test labels and do not encode the 68% and 75% reductions by construction. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. Concerns about the small test set, best-of-fold selection bias, and the absence of confidence intervals are statistical reliability issues, not circularity under the stated definitions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper contributes an empirical architecture and a private dataset. Its central claim rests on hand-chosen network hyperparameters, augmentation and loss-weight parameters, and unvalidated ground-truth labels. No new physical entities are introduced. The number of free parameters is large relative to the 71-video dataset, so the 68% and 75% reductions should be treated as evidence for a method, not a settled result.

free parameters (4)
  • Network architecture hyperparameters = Hand-chosen depths 90/60/30/15/10, channels 64/128/256, pooling strides
    The DVR 2/3 depth schedule and kernel sizes are motivated heuristically rather than derived; the central result depends on this specific configuration.
  • Simultaneous loss weights = 0.75 * HRLoss + 0.25 * RRLoss
    The paper states 'by empirical analysis, the loss function is determined as given below'; these weights are fit to training and validation data and affect simultaneous HR/RR results.
  • Augmentation parameters = z = v = h = 50%, brightness [0.1, 1.0]
    Zoom, shift, and brightness ranges are chosen parameters that affect regularization and final model performance.
  • Optimizer and training hyperparameters = Learning rate 1e-5, Lookahead step size 0.5, dropout 0.15, batch of 5 720-frame sequences, 40 epochs
    He initialization, Lookahead rate, dropout, batch size, and early stopping patience are selected choices that influence the trained weights and thus the reported errors.
assumptions (4)
  • domain assumption Fingertip video color changes encode pulse and respiratory information.
    The method assumes a PPG-like signal is recoverable from mobile phone camera video; this is taken from the cited prior literature and not validated within this paper.
  • domain assumption Manual 30-second pulse palpation and self-counted breaths provide accurate ground truth.
    Data Collection section uses doubled 30-second counts as labels; only the HR side is supported by a cited MAE just over 4 BPM, while RR self-counting is unvalidated.
  • domain assumption The automated quality filter correctly identifies usable pulse videos.
    Section 3.2 relies on McGinnis et al. to keep 71 of 161 clips; the test set is drawn only from filtered videos, so results may not extend to rejected videos.
  • standard math Standard deep learning training behaves as expected on this small dataset.
    The network relies on backpropagation, batch normalization, ReLU, and dropout; no proof is offered that these generalize from 71 videos.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Heart rate and respiratory rate prediction from noisy real-world smartphone based on Deep Learning methods." pith.science (2026). https://pith.science/paper/ICQCSMNK

@misc{pith2026250622460,
  author       = {Pith},
  title        = {Pith review of: Heart rate and respiratory rate prediction from noisy real-world smartphone based on Deep Learning methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ICQCSMNK}},
  note         = {Machine review of arXiv:2506.22460}
}
read the original abstract

Using mobile phone video of the fingertip as a data source for estimating vital signs such as heart rate (HR) and respiratory rate (RR) during daily life has long been suggested. While existing literature indicates that these estimates are accurate to within several beats or breaths per minute, the data used to draw these conclusions are typically collected in laboratory environments under careful experimental control, and yet the results are assumed to generalize to daily life. In an effort to test it, a team of researchers collected a large dataset of mobile phone video recordings made during daily life and annotated with ground truth HR and RR labels from N=111 participants. They found that traditional algorithm performance on the fingerprint videos is worse than previously reported (7 times and 13 times worse for RR and HR, respectively). Fortunately, recent advancements in deep learning, especially in convolutional neural networks (CNNs), offer a promising solution to improve this performance. This study proposes a new method for estimating HR and RR using a novel 3D deep CNN, demonstrating a reduced error in estimated HR by 68% and RR by 75%. These promising results suggest that regressor-based deep learning approaches should be used in estimating HR and RR.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 34 canonical work pages

  1. [1]

    Pelegris K

    P. Pelegris K. Banitsas, T. Orbach, and K. Marias. A novel method to detect heart beat rate using a mobile phone. In Conf. Proc. IEEE Eng. Med. Biol. Soc., 2010

  2. [2]

    Belagiannis, C

    V . Belagiannis, C. Rupprecht, G. Carneiro, and N. Navab. Robust optimization for deep regression. In ICCV, 2015

  3. [3]

    J. B. Bolkhovsky, C. G. Scully, and K. H. Chon. Statistical analysis of heart rate and heart rate variability monitoring through the use of smart phone cameras. In Conf. Proc. IEEE Eng. Med. Biol. Soc., 2012

  4. [4]

    M. Chan, D. Esteve, J. Y . Fourniols, C. Escriba, and E. Campo. Smart wearable systems: Current status and future challenges. Artif. Intell. Med., 56:137–156, 2012

  5. [5]

    ffmpeg tool (version be1d324), 2016

    FFmpeg Developers. ffmpeg tool (version be1d324), 2016. Available: http://www. ffmpeg.org

  6. [6]

    Donahue, L

    J. Donahue, L. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell., 2017

  7. [7]

    J. Du, S. Li, X. Chen, S. Chen, and M. Hong. Dynamic differential-privacy preserving sgd, 2021. arXiv preprint arXiv:2111.00173

  8. [8]

    C. G. Scully et al. Physiological parameter monitoring from optical recordings with a mobile phone. IEEE Trans. Biomed. Eng., 2012

Show all 39 references
  1. [9]

    Grieten et al

    L. Grieten et al. Evaluating smartphone based photoplethysmography as a screening solu- tion for atrial fibrillation: A digital tool to detect afib? J. Am. Coll. Cardiol., 2017

  2. [10]

    Cao et al

    S. Cao et al. Morphology extraction of fetal ecg using temporal cnn-based nonlinear adap- tive noise cancelling. PLOS ONE, 17(12):e0278917, 2022

  3. [11]

    Li et al

    T. Li et al. A pilot study of respiratory rate derived from a wearable biosensor compared with capnography in emergency department patients. Open Access Emerg. Med., 11:103– 108, 2019

  4. [12]

    Girshick, J

    R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014

  5. [13]

    T. Han, X. Xiao, L. Shi, J. Canny, and J. Wang. Balancing accuracy and fun: Designing camera based mobile games for implicit heart rate monitoring. In Proc. ACM CHI’15 Conf. Hum. Factors Comput. Syst., 2015

  6. [14]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. Available: https://arxiv. org/abs/1502.01852

  7. [15]

    A. D. Hollerbach and N. V . Sneed. Accuracy of radial pulse assessment by length of counting interval. Heart and Lung, 19(3):258–264, 1990

  8. [16]

    Ilbeigipour, A

    S. Ilbeigipour, A. Albadvi, and E. Akhondzadeh Noughabi. Real-time heart arrhythmia detection using apache spark structured streaming. J. Healthc. Eng., 2021:6624829, 2021. doi: 10.1155/2021/6624829

  9. [17]

    Jonathan and M

    E. Jonathan and M. Leahy. Investigating a smartphone imaging unit for photoplethysmog- raphy. Physiol. Meas., 2010

  10. [18]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012

  11. [19]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In ICCV, 2011. Farabi 19

  12. [20]

    Lazaro, Y

    J. Lazaro, Y . Nam, E. Gil, P. Laguna, and K. H. Chon. Respiratory rate derived from smartphone-camera-acquired pulse photoplethysmographic signals. Physiol. Meas. , 36 (11):2317, 2015

  13. [21]

    H. Lee, A. Cho, and M. Whang. Fusion method to estimate heart rate from facial videos based on rppg and rbcg. Sensors, 21(20):6764, 2021. doi: 10.3390/s21206764

  14. [22]

    X. Liu, W. Liang, Y . Wang, S. Li, and M. Pei. 3d head pose estimation with convolutional neural network trained on synthetic images. In ICIP, pages 1289–1293, 2016

  15. [23]

    Lázaro, E

    J. Lázaro, E. Gil, R. Bailón, A. Mincholé, and P. Laguna. Deriving respiration from pho- toplethysmographic pulse width. Med. Biol. Eng. Comput., 2013

  16. [24]

    R. S. McGinnis, E. W. McGinnis, C. Petrillo, J. Ferri, J. Scism, and M. Price. Validation of smartphone based heart rate tracking for remote treatment of panic attacks. IEEE J. Biomed. Health Inform., 25(3):656–662, 2020

  17. [25]

    M. A. Motin, C. K. Karmakar, and M. Palaniswami. Ensemble empirical mode decompo- sition with principal component analysis: A novel approach for extracting respiratory rate and heart rate from photoplethysmographic signal. IEEE J. Biomed. Health Inform. , 22 (3):766–774, 2018

  18. [26]

    Y . Nam, Y . Kong, B. Reyes, N. Reljin, and K. H. Chon. Monitoring of heart and breathing rates using dual cameras on a smartphone. PLOS ONE, 11(3):e0151013, 2016. doi: 10. 1371/journal.pone.0151013

  19. [27]

    M.-Z. Poh, D. J. McDuff, and R. W. Picard. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng., 58:7–11, 2011

  20. [28]

    Y . Ren, B. Syrnyk, and N. Avadhanam. Improving video-based heart rate and respiratory rate estimation via pulse-respiration quotient. In Workshop on Healthcare AI and COVID- 19, PMLR, 2022

  21. [29]

    B. S. Schumacher. Monitoring vital signs to identify postoperative complications. Academy of Medical-Surgical Nurses, 4(2):142–145, 1995

  22. [30]

    Sermanet, D

    P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y . Lecun. Overfeat: In- tegrated recognition, localization and detection using convolutional networks. In ICLR, 2014

  23. [31]

    Y . Song, J. Chen, and R. Zhang. Heart rate estimation from incomplete electrocardiogra- phy signals. Sensors, 23(2):597, 2023. doi: 10.3390/s23020597

  24. [32]

    Soomro, A

    K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human action classes from videos in the wild, 2012. arXiv preprint arXiv:1212.0402

  25. [33]

    Y . Sun, X. Wang, and X. Tang. Deep convolutional network cascade for facial point detection. In CVPR, 2013

  26. [34]

    Szegedy, W

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015

  27. [35]

    Toshev and C

    A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, 2014

  28. [36]

    Ullah, J

    A. Ullah, J. Ahmad, K. Muhammad, M. Sajjad, and S. W. Baik. Action recognition in video sequences using deep bi-directional lstm with cnn features. IEEE Access, 6:1155– 1166, 2018

  29. [37]

    Zaman, C

    R. Zaman, C. H. Cho, K. Hartmann-Vaccarezza, T. N. Phan, G. Yoon, and J. W. Chong. Novel fingertip image-based heart rate detection methods for a smartphone. Sensors, 17 (2):358, 2017. Farabi 20

  30. [38]

    Zhang, J

    M. Zhang, J. Lucas, J. Ba, and G. E. Hinton. Lookahead optimizer: k steps forward, 1 step back. In Advances in Neural Information Processing Systems (NeurIPS) , pages 9593–9604, 2019

  31. [39]

    Zheng, K

    K. Zheng, K. Ci, H. Li, L. Shao, G. Sun, J. Liu, and J. Cui. Heart rate prediction from facial video with masks using eye location and corrected by convolutional neural networks. Biomed. Signal Process. Control, 75:103609, 2022. doi: 10.1016/j.bspc.2022.103609

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.