{"id":"0bfc1b55-586f-4a8c-bf9d-9e11da77b416","arxiv_id":"2505.11412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Hyperparameter choice strongly alters the quality and composition of Monte Carlo Dropout and IVON uncertainty estimates for PPG-based AF and blood pressure models, and per-class calibration can differ sharply from global calibration.","lead":"Two uncertainty quantification methods for wearable photoplethysmography are tested on atrial fibrillation classification and blood pressure regression: Monte Carlo Dropout and the newer IVON optimizer. The paper shows that hyperparameters such as dropout rate and Hessian initialization substantially change both predictive performance and the balance between aleatoric and epistemic uncertainty.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated uncertainty disentanglement leaves the paper's composition claim unsupported, per its own correlation analysis and caveats.","rationale":"The reader's weakest assumption was the patient-overlapping VitalDB split for BP regression, which is a real limitation affecting the generalization claims for that task. However, the patient-overlap issue primarily weakens the BP regression results, whereas the central abstract claim about 'composition' of uncertainties is directly undermined by the paper's own admission that its disentanglement method is unvalidated and that the estimated aleatoric and epistemic components are highly correlated. The paper flags this in Secs. 3.4.3, 7.1, and 7.3, so this is a self-acknowledged weak point rather than an external attack. Because the reader's verdict was already CONDITIONAL and partly based on this same limitation, the concern reinforces the conditional verdict without moving it to ACCEPT or REJECT. The novel disentanglement method is a claimed contribution (Sec. 2.3), and if it is invalid, the 'proportion of total uncertainty that is aleatoric' claim is not empirically established. A synthetic validation test would settle whether the composition claim is trustworthy or should be explicitly qualified as decomposition-dependent.","tokens_in":25931,"tokens_out":6470,"duration_ms":68128,"concrete_test":"Validate Algorithm 3 and Eqs. 3-4 on a synthetic benchmark with known aleatoric and epistemic ground truth, e.g. label noise of known input-dependent magnitude plus a covariate-shift test set, following the framework of Mucsányi et al. (2024). For each hyperparameter setting (MCD dropout rates 5/10/40%, IVON h0 0.001/0.01/0.5), compute the recovered aleatoric fraction from the proposed disentanglement and compare it to the true noise-driven fraction in the synthetic data. If the recovered aleatoric fraction does not track the true fraction as sampling stochasticity varies, the composition claim should be removed from the abstract and conclusion, or replaced with a statement that the reported composition is only with respect to a specific, unvalidated decomposition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim includes the 'quality and composition' of predicted uncertainties: 'the stochasticity of the model parameter sampling determines the proportion of the total uncertainty that is aleatoric' (Abstract, also Conclusion). This composition claim rests entirely on the validity of the disentanglement procedure in Algorithm 3 for classification and Eqs. 3-4 for regression. The paper never validates that procedure against ground-truth aleatoric/epistemic components. Instead, it reports high Pearson correlations between the disentangled components (Table 5: 0.72-0.81 for BP regression) and explicitly states in Sec. 7.3: 'without verifying our method for disentangling uncertainties, we cannot be certain that the observed increases in our predicted epistemic uncertainty is truly epistemic uncertainty.' In Sec. 7.1 it concludes 'these results suggest that our own estimates are likely not completely disentangled.' If the decomposition is invalid, then the 'proportion of the total uncertainty that is aleatoric' is an artifact of the chosen formulae, not a property of the models. This directly undermines the composition component of the headline claim, which is featured in the abstract and conclusion. The regression decomposition is the standard Kendall-Gal law-of-total-variance approach, which the paper itself notes is known to be entangled (Sec. 3.4.3, citing ref. 29); the classification version is a new method that is presented as a contribution but never validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates two approximate variational uncertainty quantification techniques, Monte Carlo Dropout (MCD) and Improved Variational Online Newton (IVON), for deep learning models that detect atrial fibrillation from raw PPG time series and predict blood pressure from VitalDB PPG signals. The authors train AF classifiers with several MCD dropout rates and IVON Hessian initializations, and BP regression models with several MCD dropout rates, then assess predictive performance and calibration using ECE, UCE, ENCE, coverage calibration error, and per-class calibration curves. The central claims are that hyperparameter choices substantially affect both predictive performance and the quality and composition of predicted uncertainties, that the degree of sampling stochasticity determines the estimated aleatoric/epistemic split, and that per-class calibration differs meaningfully from population-level calibration. The manuscript also proposes a new entropy-based decomposition of classification uncertainty and cautions that standard disentanglement methods may not separate aleatoric from epistemic uncertainty.","tokens_in":26135,"tokens_out":4912,"duration_ms":52759,"significance":"If the central findings hold, the paper provides a practically useful, large-scale comparison of MCD and IVON on two clinically relevant PPG tasks and makes a strong case that calibration should be evaluated per class and with multiple metrics. The authors are explicit about several limitations, including the known entanglement of the regression disentanglement and the uncertainty about their own classification decomposition, which is a genuine strength. The novelty is incremental rather than foundational, but the empirical observations about hyperparameter sensitivity and adaptive calibration are valuable for practitioners building uncertainty-aware wearable-health models. The main contributions that would survive scrutiny are the evaluation protocol and the specific empirical comparisons; the compositional and generalization claims need additional support before they can be accepted.","major_comments":[{"comment":"The compositional component of the headline claim, that sampling stochasticity determines the proportion of total uncertainty that is aleatoric, rests on the disentanglement procedures in Eqs. (3)-(4) and Algorithm 3, but these procedures are never validated against ground-truth aleatoric and epistemic components. The manuscript itself reports high Pearson correlations between the estimated components (0.72-0.81 in Table 5), states in §7.3 that \"without verifying our method for disentangling uncertainties, we cannot be certain that the observed increases in our predicted epistemic uncertainty is truly epistemic uncertainty,\" and concludes in §7.1 that the estimates are \"likely not completely disentangled.\" Because the classification decomposition in Algorithm 3 is a new construct and its interpretation as pure aleatoric uncertainty is not established, the composition results should be presented as properties of the decomposition formulas rather than as properties of the models unless a synthetic or otherwise ground-truth benchmark is provided.","section":"§3.4.3, Table 5, §7.1, §7.3"},{"comment":"The VitalDB blood pressure evaluation uses the same 1,293 patients in the training, validation, and test splits, as stated in §4.2 and Table 2. The BP regression performance and calibration numbers in Tables 3-4 and Fig. 2 therefore do not measure generalization to previously unseen patients; the text in §4.2 acknowledges this overlap, but Section 1's general framing about \"poor generalisation to previously unseen test data\" and the clinical motivation imply a generalization claim. Because within-patient correlation can inflate apparent performance and calibration, the authors should either add a patient-disjoint evaluation or explicitly reframe the BP results as describing a patient-specific calibration scenario.","section":"§4.2, Table 2"},{"comment":"All reported comparisons are single training/evaluation runs without repeated seeds, confidence intervals, or other variance estimates. Several differences that support the paper's central message are small in magnitude, such as the AF MCD AUC of 0.84 vs. 0.85 and F1 of 0.70 vs. 0.72 in Table 6, and the IVON total ECE of 0.073 vs. 0.074 in Table 9. Without repeated runs or interval estimates, the claim that hyperparameters have a \"considerable effect\" on predictive performance and calibration is not quantitatively supported for these close comparisons. At minimum, the key configurations should be repeated over multiple seeds and reported as means with standard deviations or confidence intervals.","section":"Tables 3-9"}],"minor_comments":[{"comment":"There are numerous typographical errors, including \"sigificant\" in §1.1, \"a a single\" in §1, \"uncertainies\" in §3.2.1, and \"incompatable\" in §6; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The entropy operator H is used in Algorithm 3 and Eqs. (5)-(6) but is not defined until after the algorithm; it would be clearer to define it immediately before the algorithm.","section":"§3.2.2, Algorithm 3"},{"comment":"Several axis labels in Figure 2 are truncated or missing symbols, such as \"DBP: vs. Error\" and \"Predicted \" with no variance symbol shown; these should be corrected for readability.","section":"Figure 2"},{"comment":"The decision not to train IVON on the BP regression task is explained, but the phrase \"given our interest in comparing calibration quality... for similar model architectures\" is a bit confusing because no IVON regression comparison is presented; consider clarifying that the comparison applies only to classification.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and empirically rich, but it overstates the compositional and generalization aspects of its findings. The disentanglement caveats are already in the text, which is good, but the abstract and conclusion still present composition as a main result. The patient-overlapping BP split and the absence of repeated seeds are standard but important methodological weaknesses that the editors may want the authors to address substantively. The paper is not a rejection candidate, because the core empirical observations about hyperparameter sensitivity and per-class calibration are likely reproducible and useful; however, the load-bearing claims need reframing or additional evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a useful and unusually honest empirical study of MCD and IVON on wearable PPG tasks (AF classification, BP regression). The genuinely new content is the first application of IVON with heteroscedastic aleatoric output heads to raw 1D PPG time series, plus the per-class adaptive calibration analysis, which shows that global calibration metrics can hide large per-class miscalibration—in several rows of Tables 7 and 9 the per-class ECE is roughly double the total. That point matters for clinical deployment and is worth taking seriously.\n\nWhat the paper does well: it varies the right hyperparameters (dropout rate, IVON Hessian initialisation), uses a broad set of calibration metrics, and is candid about the limits of its own tools. The authors explicitly say their regression disentanglement is likely not fully disentangled (Sec. 7.1, Table 5 correlations 0.72–0.81) and that without validating their classification disentanglement they cannot be sure the estimated epistemic increases are truly epistemic (Sec. 7.3). That honesty is a real strength.\n\nThe soft spots are real but manageable. The abstract and conclusion claim that hyperparameters determine the composition of predicted uncertainty into aleatoric and epistemic parts. That claim rests on the disentanglement procedure in Algorithm 3 and Eqs. 3–4, which is never validated against known ground truth. Given the paper's own correlations and caveats, I would not treat the composition finding as established. A revision should either validate the decomposition or downgrade the claim to 'changes in two computed summaries' rather than true aleatoric/epistemic separation. Second, every comparison is a single run without repeated seeds or confidence intervals, so the quantitative ordering across hyperparameters is fragile. Third, the VitalDB split is patient-overlapping by design (stated in Sec. 4.2), but the intro and abstract language about 'previously unseen test data' is misleading for the BP regression results. That is a framing fix.\n\nBottom line: this deserves a real peer review, not a desk reject. The per-class calibration message and the PPG-specific UQ comparison are useful to the community. I'd want major revisions on the disentanglement claim, repeated runs, code release, and a clearer statement about the VitalDB split, but the core empirical story is plausible and the authors' willingness to flag their own limitations makes it a good candidate for a serious referee.","headline":"Useful, honest empirical study of MCD/IVON hyperparameters for PPG uncertainty, but the headline 'composition' claim rests on a disentanglement method the authors themselves have not validated.","tokens_in":26748,"tokens_out":3239,"would_cite":true,"duration_ms":31790,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For PPG heart predictions, sampling stochasticity controls which uncertainty you get and how well it is calibrated.","keywords":["uncertainty quantification","Monte Carlo Dropout","Improved Variational Online Newton","photoplethysmography","atrial fibrillation","blood pressure estimation","model calibration","aleatoric and epistemic uncertainty"],"falsifier":"Retrain the blood-pressure regression models on a VitalDB split in which no patient appears in more than one set, and recompute the ENCE and CCE orderings across dropout rates (1%, 5%, 40%). If the ordering of calibration quality with dropout rate changes or the local calibration differences vanish, the paper's regression conclusions describe per-patient adaptation rather than generalisable model behaviour.","tokens_in":25624,"feed_emoji":"🫀","tokens_out":7732,"duration_ms":69175,"temperature":0.7,"pith_summary":"This paper tests two scalable uncertainty-quantification techniques—Monte Carlo Dropout and Improved Variational Online Newton—on deep networks that classify atrial fibrillation and regress blood pressure from raw wearable photoplethysmography (PPG) time series. It finds that the hyperparameter controlling how randomly the model parameters are sampled at evaluation (dropout rate for MCD, Hessian initialisation for IVON) has a large effect on both predictive accuracy and the quality and composition of the predicted uncertainties. In particular, more stochastic sampling raises the fraction of uncertainty attributed to the model (epistemic) rather than the data (aleatoric), and on the whole population of test examples it improves calibration of class probabilities. The paper also shows that per-class calibration can differ sharply from global calibration, so an evaluation protocol that checks local and adaptive calibration is needed before these uncertainties are used in clinical decisions.","feed_headline":"Uncertainty in PPG models turns on one sampling dial","feed_subtitle":"Tuning one hyperparameter changes which uncertainty you get and whether probabilities are calibrated.","key_machinery":"The central mechanism is the stochasticity of model parameter sampling at evaluation time. For Monte Carlo Dropout this is set by the dropout rate $p$; for Improved Variational Online Newton it is set by the Hessian initialisation $h_0$, which controls the width of the approximate Gaussian posterior. This single dial determines how much of the total predictive variance is labelled epistemic (model uncertainty) as opposed to aleatoric (inherent data noise), and it correlates with how well the final class probabilities are calibrated. The supporting machinery is the standard heteroscedastic likelihood modelling from Kendall and Gal (Gaussian negative log-likelihood for regression; Monte-Carlo-averaged softmax for classification), the law-of-total-variance decomposition for regression and the entropy-concavity decomposition $H_{\\text{ale}}\\le H_{\\text{total}}$ for classification, and a multi-metric evaluation protocol (ECE, UCE, ENCE, CCE, bivariate error-vs-uncertainty histograms, per-class curves) that exposes where calibration fails locally.","core_discovery":"The central claim is that the reliability of uncertainty estimates from MCD and IVON on PPG prediction tasks is governed largely by the stochasticity of the parameter sampling, controlled by the dropout rate and by the Hessian initialisation respectively. Larger dropout rates (and smaller $h_0$ for IVON) increase the proportion of total uncertainty labelled epistemic and produce better-calibrated class probabilities as measured by the expected calibration error (ECE) over the whole test population, while the best predictive accuracy occurs at intermediate settings. The same stochasticity also shifts the balance between aleatoric and epistemic components, and the two components, when disentangled with standard formulas, turn out to be highly correlated, so the decomposition must be interpreted cautiously. Because per-class calibration curves diverge from global ones, the paper argues that evaluating uncertainty quality requires both individual and adaptive calibration metrics, not just aggregate scores.","pith_inferences":["If the sampling-stochasticity/ECE relationship holds on other physiological time-series tasks, hyperparameter scans for UQ should report both ECE and an entropy-based calibration metric such as UCE, because the two can rank configurations oppositely (in the AF results, dropout 40% had the best ECE but a worse UCE than dropout 10%).","The high correlation between aleatoric and epistemic estimates hints that a single total-uncertainty score may be a more honest summary than the separated components for clinical use, until disentanglement methods are validated on physiological data.","An external, patient-disjoint evaluation of the BP models would test whether the dropout-rate effects on regression calibration persist under true generalization; this follows naturally from the paper's current split, which shares all 1,293 patients across sets.","A similar tuning of sampling stochasticity could be tried with other approximate Bayesian schemes, such as deep ensembles with different initialisations, to see whether calibration quality likewise improves with wider posterior sampling."],"forward_implications":["Uncertainty estimates from MCD and IVON on PPG tasks are not trustworthy as-is: the dropout rate and $h_0$ must be tuned on the chosen calibration metric, because the optimal setting for accuracy, for ECE, and for UCE are not the same.","The proportion of predicted uncertainty attributed to epistemic versus aleatoric sources is largely an artefact of the sampling-stochasticity hyperparameter, not a stable property of the data or model.","Whole-test-set calibration can hide severe per-class miscalibration; for AF screening, low-uncertainty predictions are predominantly non-AF, so a highly confident decision rule would produce many false negatives.","For BP regression, the two disentangled uncertainty components are highly correlated, so users should not interpret the separated numbers as independent sources of doubt.","Greater stochasticity in model sampling produced better whole-population calibration of class probabilities for both techniques, suggesting a common principle: more posterior exploration helps probability calibration on this task."],"supporting_citations":[{"why":"Supplies the basis for Monte Carlo Dropout as approximate variational inference, motivating its use for epistemic uncertainty.","marker":"[26]"},{"why":"Provides the heteroscedastic aleatoric-uncertainty modelling and the variance/entropy disentanglement formulas used in both tasks.","marker":"[30]"},{"why":"Introduces IVON, the optimiser whose Hessian initialisation is the hyperparameter under study.","marker":"[38]"},{"why":"Benchmarking analysis that the paper cites when cautioning that standard disentanglement may not separate aleatoric and epistemic uncertainty.","marker":"[29]"},{"why":"Defines consistency and adaptivity, shaping the per-class and conditional calibration evaluation protocol.","marker":"[56]"},{"why":"Supplies the dataset splits and evaluation context for both tasks, including the patient-overlapping VitalDB split used for blood pressure.","marker":"[62]"},{"why":"Source of the VitalDB PPG and blood-pressure data used to train the regression models.","marker":"[63]"}],"fun_headline_variants":["One dial defines uncertainty kind in PPG models","PPG uncertainty: sampling stochasticity rules calibration","Single hyperparameter sets PPG uncertainty balance","Dropout and Hessian noise shape PPG uncertainty","Sampling dial splits PPG uncertainty and calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The blood-pressure regression results use the same 1,293 ICU patients in training, validation, and test sets, so the claims about \"previously unseen test data\" assume that this patient-overlapping split behaves like a true generalization test.","fun_headline_variants_meta":{"raw":{"variants":["One dial defines uncertainty kind in PPG models","PPG uncertainty: sampling stochasticity rules calibration","Single hyperparameter sets PPG uncertainty balance","Dropout and Hessian noise shape PPG uncertainty","Sampling dial splits PPG uncertainty and calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2893,"prompt_tokens":977,"completion_tokens":1916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1859}},"tokens_in":593,"tokens_out":1916,"duration_ms":14729,"temperature":1.0,"reasoning_tokens":1859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:30.737474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the blood-pressure regression models on a VitalDB split in which no patient appears in more than one set, and recompute the ENCE and CCE orderings across dropout rates (1%, 5%, 40%). If the ordering of calibration quality with dropout rate changes or the local calibration differences vanish, the paper's regression conclusions describe per-patient adaptation rather than generalisable model behaviour.","supporting_citations":[{"cited_title":"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning","cited_arxiv_id":null,"evidence_quote":"Supplies the basis for Monte Carlo Dropout as approximate variational inference, motivating its use for epistemic uncertainty."},{"cited_title":"What uncertainties do we need in Bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the heteroscedastic aleatoric-uncertainty modelling and the variance/entropy disentanglement formulas used in both tasks."},{"cited_title":"Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts","cited_arxiv_id":null,"evidence_quote":"Defines consistency and adaptivity, shaping the per-class and conditional calibration evaluation protocol."},{"cited_title":"PulseDB: A large, cleaned dataset based on MIMIC-III and VitalDB for benchmarking cuff-less blood pressure estimation methods.Frontiers in Digital Health, 4:1090854, 2023","cited_arxiv_id":null,"evidence_quote":"Source of the VitalDB PPG and blood-pressure data used to train the regression models."}],"review_version":1}