{"id":"f7e3fa31-b0a9-41ce-a452-cd1d610d825f","arxiv_id":"1908.07483","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-step neural network that uses both daily sleep timing and same-day light, skin temperature, and activity data estimates DLMO with lower root-mean-square error than models using one time scale alone.","lead":"This paper combines a week of sleep timing data with one day of wrist sensor readings to estimate when a person's melatonin rises at night. The combined model reduced estimation error to about 1.4 hours on a test group of college students, better than using either data source alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical superiority claim is based on validation-fold tests after model selection, not on the independent test set.","rationale":"The reader's verdict is CONDITIONAL, and I agree the paper should not be rejected outright: the framework is reasonable, the results are plausible, and the authors explicitly list limitations. My concern is narrower but decisive for the headline claim: the 'statistically significantly better' phrasing refers to validation-fold statistics, not the held-out test set, and those statistics were computed after selecting the winning model using the same folds. This makes the p-values optimistic and does not directly support the strongest claim as stated. The proposed test on the 31 test samples would settle whether the improvement generalizes. This does not change the overall CONDITIONAL verdict, but it sharpens the condition: the claim of statistical superiority should be re-evaluated on independent data, not on the folds used for model selection. The reader's rationale flagged the validation-versus-test issue, though the formal weakest_assumption field focused on the sufficiency of the scalar sleep-midpoint summary, so my agreement is partial.","tokens_in":15907,"tokens_out":3737,"duration_ms":39498,"concrete_test":"Run a paired permutation or bootstrap test on the 31 held-out test samples, comparing absolute errors (or RMSE) for RNNEMA versus EMA and RNNEMA versus RNN24-hour. For example, resample test subjects 10,000 times, compute the RMSE difference each time, and report the bootstrap confidence interval and a paired permutation p-value. If the observed test-RMSE advantages (1.38 vs 1.42 and 1.55) do not reach p<0.05 or the CI includes zero, the 'statistically significantly better' claim should be downgraded to 'not supported on the independent test set.' If the test-set comparison is significant in this pre-specified analysis, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2's central comparison is not a test-set comparison. Table 2 reports RMSEtest (1.38 vs 1.42 vs 1.55) and <1h only descriptively; the significance tests cited for 'statistically significantly better model performance' are repeated-measures ANOVA and paired t-tests on the 10 cross-validation RMSEval values. Two problems arise. First, RNNEMA was selected among RNNSMA, RNNEMA, and RNNMA based on the same validation RMSEval and <1h after the three models did not differ significantly on those folds (p=0.66), so the follow-up comparisons on the same folds are conditioned on that selection, and the p-values are not corrected for it. Second, validation-fold RMSE is a model-selection criterion, not an unbiased estimate of test performance. With only 31 test samples, a 0.04-hour RMSE difference is well within sampling noise, and no inferential test is reported on the test set. Thus the headline claim that the two-time-scale model is statistically significantly better than EMA or RNN24-hour is not actually established by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-step framework for estimating dim light melatonin onset (DLMO) from wearable sensor data. The first step summarizes daily sleep timing data from the week before the target day using a moving average (SMA, EMA, or MA), producing a scalar circadian-phase proxy; the second step feeds this summary together with the current day's frequently sampled light exposure, skin temperature, and activity data into a gated recurrent unit (GRU) network to regress DLMO directly. The framework is evaluated on data from 207 undergraduates, with a temporal split (2013–2016 for training, 2017 for testing). The paper reports that the two-step model RNNEMA outperforms an EMA-only model and an RNN model using only 24-hour frequent data on cross-validation RMSE and on the test-set metrics RMSE and percentage of predictions within one hour, and claims that these differences are statistically significant.","tokens_in":16111,"tokens_out":7955,"duration_ms":76036,"significance":"If the claimed improvements are real, the paper offers a practical, low-cost way to estimate circadian phase from wearables without requiring full overnight melatonin profiles, which is an important step toward clinical and self-tracking applications. The two-step framework is a natural generalization of existing single-time-scale approaches and is, to the authors' knowledge, the first to predict DLMO directly from frequently sampled data, avoiding the need to fit entire melatonin curves. The use of a temporally separated holdout set is a methodological strength, and the comparison of three moving-average summarizers plus an error analysis are useful contributions. However, the central statistical claim is not yet supported by the evidence as presented: the significance tests are performed on cross-validation folds rather than on the independent test set, and model selection is not accounted for. The practical significance of the test-set gain (about 0.04 h in RMSE over EMA) also remains unclear without inferential analysis on the 31 test subjects.","major_comments":[{"comment":"The headline claim that the two-time-scale model is \"statistically significantly better\" than the EMA and RNN24-hour baselines is not supported by the reported tests. The repeated-measures ANOVA and paired t-tests cited in Section 5.2 are applied to RMSEval values from the 10 cross-validation folds, not to the 31-sample test set. The test-set metrics in Table 2 (RMSEtest: 1.38 vs. 1.42 vs. 1.55; <1h: 64.5% vs. 54.8% vs. 41.9%) are presented only descriptively, with no significance test or confidence interval. Because RNNEMA was selected after comparing it with RNNSMA and RNNMA on the same validation folds (Section 5.2 reports p=0.66 for that comparison), the subsequent paired tests of the selected model against the baselines on those same folds are conditional on the selection and are not corrected for it. The abstract's statement of statistically significantly lower RMSE is therefore not established.","section":"Section 5.2, Table 2"},{"comment":"With a test set of only 31 samples, the observed RMSEtest difference between RNNEMA and EMA (1.38 vs. 1.42 h, a 0.04 h difference) is well within plausible sampling noise, and the reported <1h difference (64.5% vs. 54.8%) is based on a handful of samples. The manuscript does not report paired significance tests (e.g., Wilcoxon signed-rank test or paired t-test on per-subject absolute errors) or bootstrap confidence intervals for the test-set differences. Without such test-set inference, the conclusion that the two-step model is superior to one-scale models is overstated. The authors should either provide this analysis or temper the claims appropriately.","section":"Section 4.2, Table 2"},{"comment":"The hyperparameters alpha=0.9 (EMA decay) and n=7 (window size) are selected using the same dataset on which the main comparison is made. Alpha is chosen based on 'preliminary experiments (data not shown)', and n=7 is retained even though Section 5.1 reports no significant effect of window size. This additional layer of model selection is not reflected in the significance testing of Section 5.2, further inflating the risk of a spurious finding. The authors should fix these parameters a priori, use nested cross-validation, or at least report the sensitivity of the central comparison to the choice of alpha and n.","section":"Section 3.1, Section 5.1"}],"minor_comments":[{"comment":"The theoretical comparison of SMA and EMA variances assumes i.i.d. Gaussian noise on sleep midpoints, which is a strong simplification given the autocorrelation and missingness of sleep timing data. The statement that EMA 'could be a better method when the noise is small' is framed too strongly; the simulation in Figure 4 is the more relevant evidence.","section":"Section 3.2"},{"comment":"The definition of RMSEval is inconsistent: Section 4.1 uses leave-one-participant-out cross-validation, while Section 4.2 switches to 10-fold cross-validation. Please harmonize the terminology to avoid confusion, especially because the results in Sections 5.1 and 5.2 are based on different validation protocols.","section":"Section 4.1 and Section 4.2"},{"comment":"The y-axis label 'r2' should be rendered as 'R²' for readability.","section":"Figure 4"},{"comment":"The model name is written as 'RNNEMA' in tables and equations but sometimes as 'RNN EMA' in the text (e.g., the first sentence of Section 5.2 and Figure 5). Please use consistent notation.","section":"Section 5.2 and throughout"},{"comment":"The comparison with prior work [4, 35] is described as using 'a subset of the data in this datasets', but the manuscript does not specify whether the same train/test split, preprocessing, and evaluation protocol were used for the cited methods. Please clarify the matching conditions for these comparisons.","section":"Table 4"},{"comment":"The manuscript does not include a code availability statement. Given that the method is empirical and relies on several preprocessing and hyperparameter choices, providing code (or at least a detailed reproducibility checklist) would strengthen the paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical application, and the temporally separated train/test split is a genuine strength. The central statistical claim, however, currently rests on validation-fold comparisons after model selection, which is not an acceptable basis for the abstract's assertion of significance. I would like the authors to add test-set inference (e.g., paired tests or confidence intervals) and to account for model selection, either by nested validation or by framing the results more cautiously. If the test-set analysis does not support the claim, the paper should be revised to reflect a weaker, descriptive conclusion, which would still be of interest if presented honestly. The manuscript is otherwise within the scope of the journal and has the potential to be a useful contribution after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The two-step framework is a genuine contribution, but the headline statistical claim doesn't survive close reading. The abstract and Section 5.2 say the two-time-scale model performs \"statistically significantly\" better than single-scale models. Those p-values come from repeated-measures ANOVA and paired t-tests on the 10 cross-validation RMSE values of the training set, after RNNEMA was chosen from three variants that did not differ significantly on those same folds (p = 0.66). The independent test set (31 samples) shows better numbers for RNNEMA — RMSE 1.38 h vs. 1.42 h and 1.55 h; within 1 h, 64.5% vs. 54.8% and 41.9% — but no significance test is reported on that test set, and with n = 31 a 0.04 h difference is within sampling noise. So the statistical claim is overstated.\n\nWhat is genuinely new: combining a summary of the past week's sleep midpoints (SMA/EMA/MA) with the current day's hourly light exposure, skin temperature, and activity in a GRU to regress DLMO directly. This is a reasonable extension over either daily-sampled sleep timing models or the current-day sensor stream alone, and differs from earlier pipelines that fit a full melatonin curve before deriving DLMO. The training/test split is temporal (2013–2016 vs. 2017), which is a real holdout. The authors also compare against a prior neural network and a limit-cycle oscillator on a subset of the data, and they openly list limitations: single-college student cohort, inconclusive feature combination results. The noise analysis of moving averages is a nice touch.\n\nThe main soft spots are the significance-testing issue, plus a few reproducibility gaps: alpha = 0.9 and n = 7 were selected via \"preliminary experiments (data not shown)\", no code or data are included, and the comparison to prior work on \"a subset of the data\" is not fully described (were those baselines retrained on the same split or quoted from their papers?). None of these are fatal — the empirical pattern is internally consistent — but they mean the paper's central claim needs to be reframed as \"RNNEMA performs better on the test set in this sample\" rather than \"statistically significantly better.\"\n\nThis is a paper for the chronobiology/sleep-technology community, and the ML crowd as a cautionary example of validation-fold p-values being treated as test evidence. It deserves peer review, not desk rejection: a serious referee can ask for a corrected statistical analysis, a confidence interval on the test RMSE, and a fuller description of the baseline comparison. Send it out.","headline":"Real contribution undercut by validation-fold significance testing; test set shows promise but not proof.","tokens_in":16672,"tokens_out":5065,"would_cite":false,"duration_ms":50395,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-step model combining a week of sleep timing with one day of wearable sensor data estimates DLMO within one hour for 64.5% of held-out test cases.","keywords":["circadian rhythm","dim light melatonin onset","DLMO estimation","wearable sensors","sleep timing","recurrent neural network","moving average","two time scales"],"falsifier":"Run a cohort in which the past week's light exposure is varied while sleep midpoints are held fixed, for example bright evening light before a constant sleep schedule. If a model that also receives prior-week light-exposure summaries beats the two-step model on test RMSE, the scalar sleep-timing summary is not sufficient. A simpler check: add prior-day summaries of light, skin temperature, and activity to the first step and see whether test RMSE drops below the reported 1.38 hours.","tokens_in":15680,"feed_emoji":"🌙","tokens_out":11204,"duration_ms":99003,"temperature":0.7,"pith_summary":"Dim light melatonin onset (DLMO) is the gold-standard marker of the body's circadian phase, but measuring it requires hours of timed saliva or blood samples under dim light, followed by laboratory assay. This paper argues that DLMO can be estimated instead from two time scales of wearable data: a week of daily sleep-timing records compressed into one summary value, combined with a full day of hourly light, skin-temperature, and activity readings. On data from 207 undergraduates, the two-step model reached a test root-mean-square error of 1.38 hours and placed 64.5% of predictions within one hour of the true DLMO. The paper reports that the two-scale model was statistically significantly better than models using either time scale alone, which would make circadian phase practical to track for shift work, jet lag, and personalized treatment timing.","feed_headline":"Wearable model estimates circadian phase within one hour 64.5%","feed_subtitle":"DLMO normally needs overnight lab sampling; two-scale wearable models estimate it from daily life.","key_machinery":"The machinery is a two-step feature-combination rule. In the first step, the previous week of daily sleep midpoints $T_i$ is compressed into a single circadian-phase summary $\\psi$ by a moving average; the paper compares simple moving average, exponential moving average with decay rate $\\alpha = 0.9$, and a general weighted moving average, and reports that they perform similarly on validation data, so the exponential version is carried forward. In the second step, a recurrent neural network whose cells are gated recurrent units (a standard type of sequence-learning cell) consumes the 24 hourly values of the current day's light exposure, skin temperature, and physical activity, and at every hourly step receives the scalar $\\psi$ as an extra input, letting the network condition the day's phase estimate on prior sleep timing. Training proceeds in three stages: least squares for the moving-average parameters, RNN training with those parameters frozen, and finally joint fine-tuning of all parameters by back-propagation.","core_discovery":"The central claim is that estimating DLMO directly from a combination of daily and frequently sampled sensor data works better than estimating it from either time scale alone, and that this can be done without ever fitting a full melatonin profile. The paper formalizes the target as $\\phi = f(\\psi, x)$, where $\\psi$ is a summary of data from before the day of interest and $x$ is that day's 24 hours of hourly light exposure, wrist skin temperature, and physical activity. It implements $\\psi$ as a moving average of the previous seven days' sleep midpoints, comparing simple, exponential, and fully weighted moving averages, and feeds $\\psi$ together with $x$ into a recurrent neural network with gated recurrent unit cells. In the evaluated configuration, the model achieved a validation RMSE of 1.21 hours, a test RMSE of 1.38 hours, and 64.5% of test predictions within one hour of measured DLMO, statistically significantly better on validation sets than either the exponential moving average alone or a 24-hour RNN without the sleep-timing summary (repeated-measures ANOVA, $p=0.0022$, with paired comparisons $p=0.048$ and $p=0.013$ after multiple-comparison correction). The authors present this as a generalization of earlier one-time-scale approaches and as the first model to predict DLMO directly from frequently sampled data rather than first estimating melatonin concentrations.","pith_inferences":["The paper's own error analysis shows that the two-step model's errors are strongly correlated with those of the sleep-midpoint summary, so the largest remaining accuracy gain likely lies in enriching the first step (for example, summaries of prior light exposure or activity timing) rather than in the network architecture.","Because the test set is one cohort of undergraduates at one college, the absolute numbers (1.38-hour RMSE, 64.5% within one hour) should be read as an upper bound of expected field performance; shift workers, jet-lagged travelers, or clinical populations may need recalibration.","The choice of the exponential moving average over the fully weighted moving average is pragmatic rather than strongly supported by validation RMSE; a larger test set could plausibly change that ordering, and the paper's noise analysis suggests simple averaging becomes preferable as sleep-midpoint noise grows.","A clean test of the framework's core premise is to add previous-day light-exposure summaries to the first step: if accuracy improves, the scalar sleep-timing summary is losing information; if it does not, the compression is sufficient."],"forward_implications":["A wearable-derived DLMO estimate can replace expensive overnight saliva or blood collections for many uses, since all inputs come from wrist sensors and sleep diaries.","Because the two-scale model outperforms both single-scale baselines, circadian-phase estimators should include both a recent-history summary and same-day frequent sampling.","The model with light exposure and skin temperature alone still reaches 61.3% within one hour on the held-out set, so a two-sensor setup could nearly match the full three-sensor version.","Training a model that predicts DLMO directly avoids the need for full melatonin profiles, lowering the cost of building and updating such models.","The framework is described as general to other regression or classification tasks with sensor data at different sample rates, so the same two-step structure could transfer to other wearable health targets."],"supporting_citations":[{"why":"Establishes the relationship between fixed and free sleep schedules and DLMO; the simple moving average baseline comes from this line of work.","marker":"[5]"},{"why":"Shows that sleep logs with self-selected sleep times predict DLMO, providing another daily-sampled baseline.","marker":"[27]"},{"why":"Provides the limit-cycle-oscillator comparison (43% within one hour) and the DLMO threshold protocol used in this data set.","marker":"[35]"},{"why":"Neural-network baseline using sleep timing, sensor data, and demographics that the paper compares against.","marker":"[4]"},{"why":"Multi-channel ambulatory monitoring study supporting the use of 24-hour frequently sampled sensor data for circadian phase.","marker":"[24]"},{"why":"Autoregressive model estimating human circadian phase from ambulatory signals, representing the frequent-sampling-only approach.","marker":"[17]"},{"why":"Mathematical model in which prior circadian phase enters the prediction, the conceptual precursor of the scalar summary term.","marker":"[21]"},{"why":"Light-based mathematical model of the human circadian pacemaker used to motivate the equation form.","marker":"[26]"}],"fun_headline_variants":["Two-scale sensor model sharpens DLMO estimates","Wearable data plus sleep history estimates DLMO within an hour","Combining time scales cuts DLMO estimation error","Two-scale model: 64.5% of DLMO predictions within the hour"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one scalar summary of the past week's sleep midpoints contains all the circadian-relevant information from before the current day, so anything that happens earlier (light exposure, activity, sleep irregularity) matters only through that average; if prior patterns carry independent information, the claimed improvement may not generalize beyond this student cohort.","fun_headline_variants_meta":{"raw":{"variants":["Two-scale sensor model sharpens DLMO estimates","Wearable data plus sleep history estimates DLMO within an hour","Combining time scales cuts DLMO estimation error","Two-scale model: 64.5% of DLMO predictions within the hour"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2837,"prompt_tokens":1072,"completion_tokens":1765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":1695}},"tokens_in":688,"tokens_out":1765,"duration_ms":12767,"temperature":1.0,"reasoning_tokens":1695,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:17:37.745813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a cohort in which the past week's light exposure is varied while sleep midpoints are held fixed, for example bright evening light before a constant sleep schedule. If a model that also receives prior-week light-exposure summaries beats the two-step model on test RMSE, the scalar sleep-timing summary is not sufficient. A simpler check: add prior-day summaries of light, skin temperature, and activity to the first step and see whether test RMSE drops below the reported 1.38 hours.","supporting_citations":[{"cited_title":"Burgess and Charmane I","cited_arxiv_id":null,"evidence_quote":"Establishes the relationship between fixed and free sleep schedules and DLMO; the simple moving average baseline comes from this line of work."},{"cited_title":"Martin and Charmane I","cited_arxiv_id":null,"evidence_quote":"Shows that sleep logs with self-selected sleep times predict DLMO, providing another daily-sampled baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the limit-cycle-oscillator comparison (43% within one hour) and the DLMO threshold protocol used in this data set."},{"cited_title":"Brown, Melissa A","cited_arxiv_id":null,"evidence_quote":"Neural-network baseline using sleep timing, sensor data, and demographics that the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multi-channel ambulatory monitoring study supporting the use of 24-hour frequently sampled sensor data for circadian phase."},{"cited_title":"Gil, Xavier L","cited_arxiv_id":null,"evidence_quote":"Autoregressive model estimating human circadian phase from ambulatory signals, representing the frequent-sampling-only approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mathematical model in which prior circadian phase enters the prediction, the conceptual precursor of the scalar summary term."},{"cited_title":"Kronauer, Daniel B","cited_arxiv_id":null,"evidence_quote":"Light-based mathematical model of the human circadian pacemaker used to motivate the equation form."}],"review_version":1}