{"id":"07297699-900c-45fe-ab33-f76de1651ea2","arxiv_id":"2607.19999","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A consortium guide that standardizes how to quantify and validate uncertainty for ML models applied to wearable PPG signals, with benchmarks, datasets, and software.","lead":"This paper is a practical guide for adding uncertainty estimates to machine-learning models that read pulse signals (photoplethysmography) from wearables. It maps out model choices, uncertainty methods, validation metrics, benchmark datasets, and open-source code for medical-device developers and clinicians.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Guide's own §9.3 reports poor UQ reliability in the practical calibration-free setting, undercutting the claim that reliable UQ is attainable.","rationale":"The reader's weakest assumption concerned label noise in benchmark datasets and its potential to bias calibration metrics. That is a legitimate concern, but I identified a more direct and load-bearing issue within the manuscript itself: the guide's own empirical summary reports poor UQ reliability for all methods in the calibration-free setting, which is the realistic deployment scenario. This internal evidence does not depend on external validation or speculation about label noise; it is stated explicitly in §9.3. It directly undercuts the reader's strongest claim that reliable, validated UQ is attainable by following the guide's recommendations. I also considered the definite mathematical error in §3.2.1 — a mixture of Gaussians is not generally Gaussian — but that is localized and fixable, whereas the reported failure of the entire recommended UQ pipeline in the relevant setting is more consequential. The proper verdict remains CONDITIONAL, as the reader assigned, because the guide still has value as a framework and a catalog of methods; however, the condition should explicitly include a demonstration of reliable UQ in a calibration-free, external-data setting, or a reframing of the central claim as aspirational. Since the reader's verdict already demands conditions before acceptance, I do not change it.","tokens_in":45955,"tokens_out":4893,"duration_ms":53198,"concrete_test":"Reproduce the QUMPHY benchmark in the calibration-free setting using the software in [115] on an external dataset not used in [3] (e.g., TriggersAF for AF detection or AuroraBP for BP regression). Compute the recommended reliability metrics — ECE/ACE/VCE for classification and ENCE/PICP/CRPS for regression — for each UQ method (MLE, MAP, deep ensembles, Monte Carlo dropout, temperature scaling, isotonic regression, conformal prediction). If no method achieves acceptable reliability (e.g., PICP within 5% of nominal coverage, ENCE ≤ 0.2, ECE ≤ 0.1), then the guide's own §9.3 conclusion is confirmed and the central claim that reliable UQ is attainable remains unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, as framed by the reader, is that combining the described UQ methods, validation metrics, and benchmark resources makes reliable, validated UQ attainable for PPG ML. A condition for this claim is that the guide's own recommended combination demonstrably produces reliable uncertainties, at least in the primary wearable deployment scenario. The guide's own summary in §9.3 undercuts this condition: 'The use of bivariate histograms reported in [3] to qualitatively assess small scale reliability revealed poor reliability for all models and uncertainty quantification techniques, where the worst reliability was exhibited in the “calibration-free” setting.' It also states, 'in the “calibration-free” setting, model-independent calibration methods for UQ did not generally improve uncertainty reliability according to the chosen metrics.' The calibration-free setting — training and testing on disjoint subjects — is the realistic wearable scenario. Additionally, §9.3 concedes that 'there is a lack of robust quantitative metrics to assess individual reliability, and so there is a need for new methods to be developed,' despite §4.1 stating that individual reliability is what matters when a single prediction informs diagnosis. Thus the guide's own evidence indicates that the recommended combination does not yet yield reliable UQ, and the validation framework is explicitly incomplete for the most clinically relevant evaluation. This is an internal inconsistency with the central attainability claim, not merely a lack of external validation. The guide remains useful as a taxonomy and evaluation checklist, but as a specification for producing reliable UQ it is not supported by the reported results.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a project deliverable (Good Practice Guide) from the QUMPHY consortium, aimed at practitioners and clinicians who use machine learning on photoplethysmography (PPG) signals. It reviews three ML modelling approaches (raw-signal deep learning, image-representation deep learning, and feature-based ML), surveys model-dependent and model-independent uncertainty quantification methods, discusses validation metrics for classification and regression, lists six benchmark problems with associated datasets, describes the accompanying open-source software, and includes a chapter on ethics. The guide does not present new benchmark experiments itself; it synthesizes results from companion papers [2] and [3] and from project reports. Its central assertion is that any prediction should be accompanied by a reliable uncertainty statement, and it aims to provide a practical framework for producing and validating such uncertainties.","tokens_in":46226,"tokens_out":4085,"duration_ms":46683,"significance":"If the guidance is followed, this document could serve as a useful specification for developers and regulators in the wearable-PPG domain. The guide is commendably explicit about many limitations, including label noise in benchmark datasets and the absence of robust individual-reliability metrics. It ships with publicly accessible software repositories and provides a structured comparison of UQ methods and metrics. The ethical discussion, while standard, is integrated into the project workflow rather than added as an afterthought. The main scientific value is in the synthesis and the identification of open problems, not in new empirical results.","major_comments":[{"comment":"The text states that a uniformly-weighted mixture of Gaussian distributions is 'also Gaussian' with mean and variance given by the displayed formulas. This is mathematically incorrect: the mixture of Gaussians is a Gaussian mixture, not a Gaussian, unless all component means are equal. The mean and variance formulas are correct for the mixture, but the distribution itself is not Gaussian. This matters because the guide later treats the ensemble predictive distribution as Gaussian for constructing coverage intervals and for evaluating Gaussian NLL. Please correct the statement and indicate how coverage intervals and NLL should be computed for the mixture (or explicitly justify a Gaussian approximation). The phrase 'assuming independence between the distributions from the individual models' is also inaccurate for the marginal mixture distribution and should be removed or qualified.","section":"§3.2.1, Eqs. (5)–(6)"},{"comment":"The guide's own summary reports that, in the calibration-free setting (training and testing on disjoint subjects, which is the realistic wearable scenario), 'model-independent calibration methods for UQ did not generally improve uncertainty reliability according to the chosen metrics,' and that bivariate histograms 'revealed poor reliability for all models and uncertainty quantification techniques, where the worst reliability was exhibited in the “calibration-free” setting.' At the same time, §4.1 argues that individual reliability is what matters when a single prediction informs a clinical decision, and §9.3 concedes that 'there is a lack of robust quantitative metrics to assess individual reliability.' This tension undercuts the framing that reliable, validated UQ is attainable by following the recommended combination. The authors should either temper the central claim or provide a mor","section":"§9.3, §4.1"}],"minor_comments":[{"comment":"The sentence 'assuming independence between the distributions from the individual models' is conceptually misleading: the mixture distribution is a convex combination, not an independent sum. Please rephrase.","section":"§3.2.1"},{"comment":"For the regression version of histogram binning, the interval [b0, bM] for the variance is not defined. State how the lower and upper bounds are chosen in practice.","section":"§3.3.1.1"},{"comment":"The ENCE definition uses an unweighted average over bins. Many published definitions weight by bin frequency. Please clarify the convention used and justify the choice.","section":"§4.3, Eq. (17)"},{"comment":"DeepBeat is described as 'preferred due to its large scale ... despite some label noise,' and later as having AF labels based on patient history rather than active AF within the signal window. Please make the nature of the label noise explicit at the first mention to avoid confusion.","section":"§5.2"},{"comment":"The checkmark notation (e.g., '✓/✓' and '✗/✓') in the R/C and E/A columns is difficult to parse. Add a footnote explaining each pair.","section":"Table 1"},{"comment":"Some references are to arXiv preprints and project deliverables that may not be permanently available. Consider adding DOIs where possible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The guide leans heavily on companion papers [2] and [3] for the benchmark results that underpin its recommendations. The editor may wish to verify that those papers are publicly available and have been peer-reviewed, since the guide's practical value depends on them. The mathematical error in §3.2.1 is the most obvious technical issue and should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a project deliverable from the QUMPHY consortium, written as a good-practice guide rather than a novel research paper. Read it with that frame. What is new: two benchmark datasets (MIMIC-III-Ext-PPG, MIMIC PERform Large) and public software with model training and UQ implementations. Those are reproducible and likely useful to the PPG community. The document also does something rare: it actually reports where UQ methods fail, in its own benchmarking. Section 9.3 is candid that in the calibration-free (disjoint subjects) setting—the realistic wearable scenario—model-independent calibration methods did not generally improve reliability, and small-scale reliability was poor for all methods. That honesty is a strength, not a weakness.\n\nThe main parts—model-dependent UQ (MLE, MAP, ensembles, dropout), model-independent methods (calibration, conformal), and validation metrics (ECE, ACE, UCE, VCE, ENCE, PICP, CRPS)—are standard material, but organized cleanly for a practitioner audience. The guide's own caveats about label noise in DeepBeat and MIMIC are appropriate.\n\nSoft spots. The mathematical error in §3.2.1 is real and should be fixed before this becomes a reference: a uniformly-weighted mixture of Gaussian densities is not itself Gaussian. The mean and variance aggregation formulas are correct—the guide's variance decomposition into epistemic and aleatoric parts is right—but the distributional claim is false and could mislead someone constructing Gaussian intervals from an ensemble.\n\nThe stress-test note about §9.3 claims an internal inconsistency with an 'attainability' claim. I don't think that lands. The guide never demonstrates or promises reliable UQ; its stated position is that robust UQ is needed, and it provides evaluation methods. It then reports that current methods often fail—which is the kind of negative result that should be published. So the central framing is consistent.\n\nThe heavier concern is the degree to which load-bearing recommendations (VCE over UCE, model benchmarking conclusions) come from companion papers [2], [3], [4] that are not included here. That is normal for a project guide, but for someone outside the project it limits independent verification. Fine for a guide; would need the supporting papers for a journal article.\n\nWho is this for? Researchers and device manufacturers working on PPG-based ML who need a practical checklist for choosing and validating UQ. For them it is worth reading and likely worth citing as a reference. It deserves a serious referee—ideally someone who will catch the Gaussian mixture error and prod the authors to clarify what the guide does and does not establish empirically.\n\nRecommendation: accept for peer review with revisions requested. Fix §3.2.1, and where the guide summarizes results from companion papers, flag them more explicitly as external rather than as the guide's own validated results.","headline":"A useful, honest synthesis of UQ practice for PPG-ML; the central claims hold up, but fix the mixture-of-Gaussians paragraph and don't oversell the empirical support.","tokens_in":46762,"tokens_out":3097,"would_cite":false,"duration_ms":35027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This guide establishes that a machine learning prediction from a PPG signal is incomplete without a quantified, validated uncertainty, and gives a practical recipe for producing and checking one.","keywords":["uncertainty quantification","photoplethysmography","machine learning","model calibration","conformal prediction","wearable devices","benchmark datasets","validation metrics"],"falsifier":"Take a benchmark such as AF detection and replace chart-derived or history-based labels with expert PPG/ECG-annotated labels for the same windows; if the guide's recommended calibration metrics report well-calibrated uncertainties on the original labels but systematically worse coverage or reliability on the expert-labelled subset, that would show the framework's label-reliability assumption is violated.","tokens_in":45873,"feed_emoji":"📊","tokens_out":3687,"duration_ms":40799,"temperature":0.7,"pith_summary":"The paper is a metrology-style good practice guide for anyone building or using machine learning models on photoplethysmography (PPG) signals from wearables. It claims that a model output is not a complete measurement result unless it comes with a quantitative statement of its own quality, and that such an uncertainty statement can be made reliable through a combination of uncertainty quantification methods, validation metrics, and benchmark datasets. The guide lays out three modelling routes—deep learning on raw signals, deep learning on image representations, and feature-based machine learning—together with model-dependent and model-independent ways to attach uncertainties to predictions. Its central recommendation is that uncertainty reliability must be validated with local and global calibration metrics, and that this is now practically achievable thanks to software and benchmarks released with the guide.","feed_headline":"Every wearable-pulse prediction needs a validated uncertainty","feed_subtitle":"A practical guide shows how to quantify and verify uncertainty for machine learning on PPG signals.","key_machinery":"The central object is the validated uncertainty statement: a predictive distribution for regression, or a probability mass function for classification, which is checked against observed errors using calibration metrics. Two mechanisms do the heavy lifting: post-hoc calibration methods (binning, isotonic regression, temperature scaling) and conformal prediction, which from a calibration set produces prediction intervals or sets with a user-specified coverage guarantee that is distribution-free. Around these, the guide organises a validation workflow—split data into training, validation, calibration and test sets; choose an uncertainty expression; compute local/global reliability metrics; insp","core_discovery":"The paper's core claim is that a prediction, by itself, is not enough: a PPG-based diagnostic decision is only as good as the reliability of the uncertainty attached to the prediction. The guide argues that uncertainty can be quantified either through model-dependent methods (maximum likelihood and maximum a posteriori estimation, deep ensembles, Monte Carlo dropout) or model-independent methods (calibration, temperature scaling, conformal prediction), and that in both cases the results must be validated using calibration metrics such as ECE, ACE, smECE, UCE, VCE and NLL for classification and ENCE, PICP, CCE, CRPS and MPIW for regression. Its contribution is to assemble this toolkit into a","pith_inferences":["An implicit consequence: because the framework judges uncertainty by calibration against reference labels, its recommendations inherit any label noise; the guide itself flags that some AF labels are based on patient history rather than active AF in the window, and that some rhythm annotations were inferred from chart events rather than expert PPG labelling.","The paper leaves open the development of quantitative individual-reliability metrics; a natural extension is to adapt per-prediction calibration tests to this framework.","The same validation workflow could transfer to other physiological signals or sensor modalities, provided the benchmark labels are built with the same care; a testable extension would be to run the recommended metrics on a modality where ground truth is unambiguous.","The recommendation that modern CNNs on raw time series are the safest default is based on two benchmark tasks; if transferred to the other four benchmark problems, the ranking may shift, especially where interpretable features are clinically validated."],"forward_implications":["If followed, wearable device manufacturers can ship models whose predictions carry coverage intervals with known reliability, not just point estimates.","Clinicians can decide whether a single measurement is good enough to act on, because the stated uncertainty has been checked against observed errors rather than assumed.","The six benchmark problems and accompanying datasets let different models and uncertainty quantification methods be compared on common ground, making published accuracy claims harder to inflate.","The software implementations give practitioners a low-cost way to adopt the recommended methods.","The guide's distinction between global, local and individual reliability shifts evaluation practice toward what matters in real use: how reliable is this particular prediction?"],"fun_headline_variants":["Wearable pulse predictions need validated uncertainty","Guide to quantifying uncertainty in PPG machine learning","PPG diagnostics: reliability hinges on validated uncertainty","Uncertainty quantification for wearable pulse signals"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reference labels used to validate uncertainty are reliable enough to serve as ground truth; the guide itself flags that some DeepBeat AF labels are based on patient history rather than active AF in the window, and that MIMIC rhythm annotations were inferred from chart events rather than expert PPG labelling.","fun_headline_variants_meta":{"raw":{"variants":["Wearable pulse predictions need validated uncertainty","Guide to quantifying uncertainty in PPG machine learning","PPG diagnostics: reliability hinges on validated uncertainty","Uncertainty quantification for wearable pulse signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2313,"prompt_tokens":667,"completion_tokens":1646,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":411,"tokens_out":1646,"duration_ms":12444,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:03:19.808046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a benchmark such as AF detection and replace chart-derived or history-based labels with expert PPG/ECG-annotated labels for the same windows; if the guide's recommended calibration metrics report well-calibrated uncertainties on the original labels but systematically worse coverage or reliability on the expert-labelled subset, that would show the framework's label-reliability assumption is violated.","supporting_citations":[],"review_version":1}