{"id":"add81e7f-d55a-4193-a349-5ee042a303dc","arxiv_id":"2505.03039","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LSTM autoencoder trained on normal-period Fitbit data detected self-reported depression and anxiety worsening episodes with adjusted F1 0.80, with resting heart rate as the top contributing feature.","lead":"This paper tests whether a machine learning model trained only on people's normal wearable data can flag weeks when their depression or anxiety symptoms get meaningfully worse. It reports strong detection accuracy on 2,023 UK adults and says resting heart rate is the most useful signal.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline F1 may not exceed a null detector: the lenient episode-level metric and in-sample threshold choice leave the reported recall uninterpretable without a sham-episode control.","rationale":"The reader's strongest claim is the quantitative performance, so the most load-bearing condition is that the evaluation actually measures signal rather than the leniency of the metric. The reader's weakest-assumption field points to the 35-day window, but the paper's own text and the adjusted F-score definition make the evaluation protocol the more decisive issue: with up to 29 windows per episode and a 5% per-window false-positive rate, the null detection rate can be high, and the threshold percentile was selected after looking at performance on the same episodes. I therefore partially agree with the reader: their rationale already lists threshold selection and missing baselines, but the window definition is secondary to the missing null control. The proposed sham-episode check settles the concern directly: if sham recall is comparable to 0.88, the reported F1 is not attributable to symptom-related changes; if it is low, the model is genuinely sensitive. The reader's CONDITIONAL verdict already requires this kind of re-evaluation, so my read does not move the verdict; it sharpens the condition by adding a specific null experiment. Verdict remains UNCHANGED.","tokens_in":13248,"tokens_out":11592,"duration_ms":126529,"concrete_test":"Run the trained LSTM-AE with the same 95th-percentile threshold on matched sham episodes: randomly sample 393 non-overlapping 35-day windows from the normal periods of the same participants (or shuffle the 393 episode start dates across the timeline), then compute the same adjusted recall, precision, and F1 treating those sham windows as true episodes. Report the distribution over at least 100 sham draws with a confidence interval. If the sham recall approaches the reported 0.88 (e.g., >0.5), the headline recall is not evidence of sensitivity to symptom worsening; if the sham recall is substantially lower (e.g., <0.3), the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—adjusted F1 = 0.80 (precision 0.73, recall 0.88) on 393 episodes—rests on an evaluation that does not yet rule out chance-level detection. An episode counts as detected if at least one of up to 29 overlapping 7-day windows in the 35-day anomalous period crosses the threshold, and the threshold is a percentile (90–100) of reconstruction error on normal validation data; the paper reports the best F1 (95th percentile) without a held-out split for this selection. Because the 95th-percentile threshold flags 5% of normal windows by construction, a trivial detector would, if windows were independent, flag at least one window in about 77% of random 35-day normal periods—close to the reported recall of 0.88. The authors provide no null-model, no baseline, and no confidence interval, and the event-based adjusted metric is lenient by design. The fixed 35-day window in the reader's weakest assumption is a secondary modeling choice; the load-bearing problem is that the headline metric does not demonstrate that detected anomalies are specific to symptom-worsening episodes rather than ordinary variation in normal data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an unsupervised anomaly detection framework for identifying clinically meaningful worsening of depression and anxiety from consumer wearable data (sleep duration, step count, resting heart rate). A LSTM autoencoder is trained exclusively on 'normal' periods (defined by eight consecutive weeks of low PHQ-8/GAD-7 scores and no COVID-19 infection) and detects anomalies when the reconstruction error of a 7-day sliding window exceeds a percentile threshold of the validation error distribution. Episodes are defined as ≥5-point increases in PHQ-8 or GAD-7 relative to the participant's normal baseline, and an episode is considered detected if at least one 7-day window within the 35-day anomalous period is flagged. The authors report an adjusted F1 of 0.80 (precision 0.73, recall 0.88) over 393 episodes, with SHAP-based explanations identifying resting heart rate as the most influential feature in 71.4% of detected anomalies. The paper claims the framework enables personalized, scalable, and proactive mental health monitoring.","tokens_in":13434,"tokens_out":3925,"duration_ms":38969,"significance":"If the reported performance is valid, this work would make a meaningful contribution: it uses a large real-world dataset (2,023 participants), adopts an unsupervised training strategy that avoids noisy labels, and provides interpretability through SHAP, which is valuable for clinical translation. The study also honestly acknowledges several limitations, including the empirical selection of detection thresholds. However, the central performance claim is currently undermined by a load-bearing methodological issue: the detection threshold was selected to maximize the F1 on the very episodes used for evaluation, and no null-model or false-positive control is provided for the lenient event-based metric. These issues must be addressed before the headline numbers can be interpreted. The strengths of the study design and the availability of code on request are noted, but the evaluation methodology needs revision.","major_comments":[{"comment":"The detection threshold is not fixed a priori but is selected as the percentile (90th–100th) of validation reconstruction error that yields the highest adjusted F1 on the 393 labeled anomalous episodes, with the 95th percentile reported as 'highest performance.' This constitutes direct tuning of the decision boundary on the evaluation episodes: the reported F1 of 0.80 is therefore an in-sample, optimistically biased estimate. The authors should either select the threshold on a held-out labeled development set, or justify a fixed threshold (e.g., based on a target false-positive rate on normal validation data) and report performance with that threshold applied to a separate test set. Confidence intervals should also be provided for the reported metrics.","section":"Methods, 'Anomaly Detection Model'; Results, Figure 2b"},{"comment":"The adjusted F-score treats an episode as detected if at least one of the 29 overlapping 7-day windows in the 35-day anomalous period exceeds the threshold. At the 95th-percentile threshold, a trivial detector that flags 5% of normal windows by construction would flag at least one window in about 77% of random 35-day periods (if windows were independent), which is close to the reported recall of 0.88. The paper does not report the false-positive rate on non-anomalous (control) periods, nor does it compare against a null detector or a random baseline. Without such controls, the reported recall and F1 do not demonstrate that the detector is specific to symptom-worsening episodes. The authors should report specificity on matched normal periods, provide a null-model comparison, and consider a more stringent episode-level criterion (e.g., requiring a minimum number of flagged windows or temporal contiguity).","section":"Methods, 'Evaluation Metrics'; Results, Figure 2b and 2c"},{"comment":"The model is trained on normal-period data from all 2,023 participants, including the 341 participants whose anomalous episodes are used for evaluation. Although the unsupervised training does not use episode labels, per-participant z-score normalization and the threshold selection described above are performed on the same participants, which may inflate apparent performance relative to a deployment scenario. A participant-wise or temporal split (e.g., training on a random subset of participants and evaluating on held-out participants) is needed to estimate generalizability to new individuals.","section":"Methods, 'Study Samples and Settings'; Results, 'Performance metrics'"}],"minor_comments":[{"comment":"The abstract states '71.4 percentage,' which should be '71.4%.'","section":"Abstract"},{"comment":"The caption refers to 'Distribution of adjusted F1-scores across all 393 anomalous episodes' and states that 54 episodes had an F1 of 0. This is confusing because F1 is normally an aggregate metric rather than an episode-level value; the authors should clarify how an episode-level F1 is computed or rephrase to describe detection status per episode.","section":"Results, Figure 2c"},{"comment":"The rows '5-9-point increase in PHQ-8' (214) and '≥10-point increase in PHQ-8' (34) sum to 248, which equals the number of episodes with a PHQ-8 increase (PHQ-only 148 + BOTH 100). The authors should state explicitly that these counts refer to episodes with a PHQ-8 increase of the given magnitude, not to distinct episodes, to avoid apparent inconsistency with the total of 393.","section":"Table 1"},{"comment":"The discussion appropriately acknowledges that 'some definitions for anomaly detection (such as the magnitude of change, anomaly duration, and detection thresholds) were determined empirically and require further investigation.' This is an honest statement, but it further underscores that the reported performance should be treated as exploratory until the threshold-selection procedure is corrected.","section":"Discussion, 'Limitations'"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, but the headline F1=0.80 is not interpretable because the threshold was selected on the evaluation episodes and no null-model baseline is provided. These are fixable with a proper held-out evaluation design, so I recommend major revision rather than rejection. The authors should be encouraged to report performance with a pre-specified threshold, include false-positive rates on non-anomalous periods, and provide confidence intervals. If, after re-analysis, the corrected performance is much lower, the manuscript may still be publishable as a methods/feasibility study, but the current claims would need to be moderated accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know the central claim before reading: the adjusted F1 of 0.80 for detecting depression/anxiety worsening is not currently well supported. The evaluation has a load-bearing flaw, and the authors need to redo it before the number means anything.\n\nWhat is genuinely new and good: this is one of the largest consumer-wearable mental health anomaly detection studies I've seen, with 2,023 normal-period participants and 393 annotated episodes. The framework is simple and clearly described: an LSTM autoencoder trained only on normal data, with SHAP for interpretability. The qualitative findings—elevated resting heart rate, reduced step count, shorter sleep around symptom worsening—are consistent with previous literature, and the SHAP analysis adds useful granularity about which features dominate in different episode types. The authors are honest about several limitations, including the empirical choice of threshold and window.\n\nHere is the problem. The detection threshold is not fixed in advance. The paper explores the 90th to 100th percentiles of validation reconstruction error and reports the best F1, which happened to be the 95th percentile. That is tuning the threshold to the labeled episodes without a held-out split, which inflates performance. More importantly, the headline recall may not beat a null detector. A 95th-percentile threshold flags 5% of normal windows by construction. Each anomalous episode contains 29 overlapping 7-day windows in its 35-day period, and an episode counts as detected if any window crosses the threshold. If normal windows were independent, a random detector would flag at least one window in about 77% of random normal periods. The reported recall is 0.88. That gap is not enough to conclude the model is detecting something specific, especially with no baselines and no confidence intervals. The event-based adjusted F-score is lenient by design, and the absence of a sham-episode control makes the recall uninterpretable.\n\nWhat would fix this: lock the threshold on a true held-out validation set or via nested cross-validation; compare against simple baselines (e.g., thresholding raw step count or heart rate, or a window-level null model); run a permutation/sham-episode analysis to estimate the false-positive rate on normal periods; and report confidence intervals. Code and data availability are currently only 'upon request,' which is not sufficient for replication.\n\nI would send this to peer review—the dataset and framing are worth referee time—but the verdict should be 'revise before acceptance,' with the evaluation redone. As it stands, the paper is a promising framework and a useful case study in evaluation pitfalls, not a validated detection result.","headline":"Large cohort and clearly described framework, but the headline F1 of 0.80 is not yet interpretable because the detection threshold is chosen on the validation set and no null/baseline comparison is reported.","tokens_in":14056,"tokens_out":1918,"would_cite":false,"duration_ms":20322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Passive wearable data can flag clinically significant depression and anxiety worsening, with adjusted F1 = 0.80 and resting heart rate dominant in 71.4% of episodes.","keywords":["anomaly detection","wearable devices","depression and anxiety","LSTM autoencoder","SHAP explainability","resting heart rate","digital mental health monitoring","passive sensing"],"falsifier":"Re-run the same model on the same data with the anomalous window shifted or widened, for example to the 14 days before and 21 days after the assessment; if the adjusted recall of 0.88 changes materially or the set of undetected episodes shifts, the detection is tied to the fixed window rather than to the underlying symptom change. A stronger test would use an independent cohort with daily symptom ratings and check that flagged anomalies align with actual symptom-onset dates.","tokens_in":13005,"feed_emoji":"🧠","tokens_out":11914,"duration_ms":109025,"temperature":0.7,"pith_summary":"The paper sets out to show that everyday wearable data can catch the moment when depression or anxiety symptoms begin to worsen, without waiting for a questionnaire or a clinic visit. The authors train an LSTM autoencoder exclusively on each participant's stable, low-symptom periods, then flag any week-long stretch whose reconstructed sleep, step count, or resting heart rate deviates strongly from that normal pattern. Across 393 symptom-worsening episodes in 341 participants, the model detects episodes with 88% recall and 73% precision (adjusted F1 = 0.80), and it performs better when both depression and anxiety worsen together or when symptom increases are more severe. Resting heart rate is the leading explanatory feature in 71.4% of detections, with both unusually high and unusually low values raising the anomaly score. This points to passive wearables serving as a scalable early-warning layer for mental health monitoring.","feed_headline":"Wearable algorithm catches 88% of depression-anxiety flare-ups","feed_subtitle":"Model trained only on healthy baselines flags symptom worsening; resting heart rate leads in 71% of detections.","key_machinery":"The load-bearing object is an LSTM autoencoder: an encoder-decoder network that compresses a 7-day sliding window of three per-participant z-scored daily features (sleep duration, total steps, resting heart rate) into a latent vector and reconstructs it. Because the model is trained only on normal-period data, high reconstruction error marks deviation from the person's own healthy pattern; the detection threshold is the 95th percentile of validation-set reconstruction error, and evaluation uses an event-based adjusted F-score in which an episode counts as detected if at least one flagged 7-day window falls inside it. SHAP values applied to the reconstruction error supply explanations, ranking resting heart rate first in 71.4% of episodes and enabling time-dynamic tracing of individual alarms.","core_discovery":"The central claim is that clinically meaningful worsening of depression and anxiety leaves a recoverable trace in daily behavior and physiology, and that an unsupervised reconstruction model can find that trace. Trained on 2,023 participants' normal periods (at least 8 consecutive weeks with PHQ-8 and GAD-7 scores both below 5, excluding COVID-19 windows), the LSTM autoencoder learns normal daily patterns of sleep duration, step count, and resting heart rate. An anomalous episode is defined as a 5-point or larger increase in PHQ-8 or GAD-7 over the participant's own normal-period average, with a fixed anomalous window of 21 days before to 14 days after the flagged assessment. Using the 95th percentile of validation reconstruction error as the threshold, the model detects 393 episodes with adjusted F1 = 0.80 (precision 0.73, recall 0.88), with higher performance for episodes involving both depression and anxiety (F1 = 0.84) and for 10-point or larger increases (F1 around 0.85). SHAP attribution identifies resting heart rate as the most influential feature overall, with a U-shaped relationship, followed by step count and sleep duration, which show negative associations.","pith_inferences":["Editorial extension: the fixed 35-day anomalous window means the reported 88% recall applies only to changes that fall inside that window; a learnable or individually calibrated window is the natural next test and could change the recall estimate.","Editorial extension: per-participant z-score normalization removes baseline differences, so the framework as presented cannot distinguish a person with low baseline activity from one with high baseline activity; modelling baseline types explicitly could sharpen precision.","Editorial extension: the SHAP rank pattern suggests a testable clinical hypothesis that sleep disturbance is an earlier or more specific marker for anxiety-only episodes, while resting heart rate is a general arousal marker; this could be checked with prospective daily symptom diaries."],"forward_implications":["A monitor running this framework could work between fortnightly self-report assessments, raising an early alarm when a clinically significant worsening may already be underway.","The threshold and window choices are directly reusable, and the reported performance by episode type gives concrete expectations for severe and comorbid cases.","Resting heart rate should be treated as a priority candidate digital biomarker, with both unusually high and unusually low values considered clinically relevant.","Individual-level explanations can reveal whether an alarm was driven by sleep, activity, or heart rate, and whether one signal preceded another.","Because the model trains only on healthy-baseline data, the approach avoids reliance on noisy worsening labels and could transfer to other conditions with definable stable periods."],"supporting_citations":[{"why":"Supplies the large longitudinal mHealth dataset and study protocol from which normal periods and wearable data are drawn.","marker":"[51]"},{"why":"Defines the PHQ-8 depression scale used to label symptom-worsening episodes.","marker":"[41]"},{"why":"Defines the GAD-7 anxiety scale used to label symptom-worsening episodes.","marker":"[55]"},{"why":"Supplies the LSTM autoencoder architecture and training configuration for time-series anomaly detection.","marker":"[61]"},{"why":"Supplies the LSTM encoder-decoder reconstruction approach for multi-sensor anomaly detection.","marker":"[62]"},{"why":"Defines the adjusted F-score evaluation that counts an episode as detected when at least one segment is flagged.","marker":"[64]"},{"why":"Provides the SHAP attribution method used to explain reconstruction error by feature.","marker":"[67]"},{"why":"Sets the daily resting-heart-rate calculation and the COVID-19 exclusion window used in defining normal periods.","marker":"[47]"}],"fun_headline_variants":["Wearable AI flags 88% of depression-anxiety flare-ups","Resting heart rate predicts mental health worsening in wearables","Explainable wearable model catches 88% of mood episodes","Wearable anomaly detection finds 88% of symptom increases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that every symptom-related change in sleep, steps, or resting heart rate lands inside the fixed 35-day window around the flagged questionnaire (21 days before to 14 days after); if behavioral changes happen earlier or later, the model cannot see them, and the reported detection rate depends on this empirically chosen definition.","fun_headline_variants_meta":{"raw":{"variants":["Wearable AI flags 88% of depression-anxiety flare-ups","Resting heart rate predicts mental health worsening in wearables","Explainable wearable model catches 88% of mood episodes","Wearable anomaly detection finds 88% of symptom increases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1670,"prompt_tokens":1037,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":653,"tokens_out":633,"duration_ms":6436,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:00:53.273553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same model on the same data with the anomalous window shifted or widened, for example to the 14 days before and 21 days after the assessment; if the adjusted recall of 0.88 changes materially or the set of undetected episodes shifts, the detection is tied to the fixed window rather than to the underlying symptom change. A stronger test would use an independent cohort with daily symptom ratings and check that flagged anomalies align with actual symptom-onset dates.","supporting_citations":[{"cited_title":"JMIR research protocols, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the large longitudinal mHealth dataset and study protocol from which normal periods and wearable data are drawn."},{"cited_title":"Journal of affective disorders, 2009","cited_arxiv_id":null,"evidence_quote":"Defines the PHQ-8 depression scale used to label symptom-worsening episodes."},{"cited_title":"Archives of internal medicine, 2006","cited_arxiv_id":null,"evidence_quote":"Defines the GAD-7 anxiety scale used to label symptom-worsening episodes."},{"cited_title":"Long short term memory networks for anomaly detection in time series","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM autoencoder architecture and training configuration for time-series anomaly detection."},{"cited_title":"Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications","cited_arxiv_id":null,"evidence_quote":"Defines the adjusted F-score evaluation that counts an episode as detected when at least one segment is flagged."},{"cited_title":"and S.-I","cited_arxiv_id":null,"evidence_quote":"Provides the SHAP attribution method used to explain reconstruction error by feature."},{"cited_title":"Engineering Applications of Artificial Intelligence, 2023","cited_arxiv_id":null,"evidence_quote":"Sets the daily resting-heart-rate calculation and the COVID-19 exclusion window used in defining normal periods."}],"review_version":1}