{"id":"aa513ce5-1821-48a8-a4e8-575c59a50746","arxiv_id":"1908.06013","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Activity and phone-use features can separate psychiatric diagnostic groups and mood states in a single research cohort, with reported accuracies from 67% to 95% and mood-score errors of 1.4 to 3.3 points.","lead":"This thesis tests whether motion data from phones and wrist sensors can detect psychiatric symptoms and mood swings in people with bipolar disorder, borderline personality disorder, and schizophrenia. It reports classification accuracies between 67% and 95% and personalized mood predictions, but the results come from a single unshared dataset with no external validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"State labels are self-report and can be missing as a function of clinical state; classifiers may separate questionnaire response patterns rather than mood episodes.","rationale":"The reader's weakest-assumption analysis identified reliance on self-report labels as the critical vulnerability, and the manuscript itself contains explicit admissions that questionnaire compliance can depend on clinical state and that missing responses were mean-imputed. That makes the ground-truth assumption genuinely load-bearing: the strongest accuracy claims are for differentiation between euthymia, mania, and depression, and those labels are derived entirely from self-report instruments. If missingness is MNAR, the training labels are systematically distorted, and the reported accuracies cannot be read as evidence about clinical episodes. I considered whether cross-validation leakage from pre-CV feature selection or SMOTE is a more serious threat, but the label problem is more fundamental: even a perfectly executed CV cannot repair labels that are biased by state-dependent non-response. The proposed concrete test directly addresses the weakness by comparing against clinician-validated labels, which is the appropriate standard for the clinical claim. Since the current verdict is already CONDITIONAL and this concern reinforces the need for external validation rather than overturning the framework, no verdict adjustment is needed.","tokens_in":51416,"tokens_out":2752,"duration_ms":32910,"concrete_test":"Obtain or reconstruct the AMoSS episode labels and re-run the Chapter 7 state classifiers under two changes: (1) exclude all weeks where the participant did not complete the QIDS/ASRM prompt rather than mean-imputing them as euthymic, and (2) re-label a validation subset (e.g. 20-30 participants) using clinician-administered YMRS/HAMD or SCID interviews conducted within ±3 days of the sensor window. Compare the LOOCV accuracies in Table 7.5 on the re-labelled subset. If accuracy falls substantially or the classifier no longer separates clinician-defined states, the self-report label dependence is confirmed. If the accuracies hold on clinician labels, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the self-report questionnaires used to define clinical states are valid ground truth. Section 3.2.1.2 explicitly states that 'compliance can be a function of the clinical state, i.e. the patient may stop responding during a manic or depressive episode,' and Section 4.3.1 imputes missing questionnaire responses with the unconditional mean. If non-response is state-dependent (MNAR), then the euthymia/mania/depression labels in Chapter 7 are not a random sample of episodes: severe episodes during which patients stop responding are either missing or assigned mean-imputed euthymic scores, while the labels that remain may track ease of self-report rather than clinical state. Because the same missingness mechanism can also affect accelerometer wear/non-wear (Section 4.3.2), the classifier could be separating response/wear behaviour from true mood. The diagnosis-level accuracies (HC vs BD/BPD) rely on clinician diagnoses and are less exposed, but the state-discrimination and personalised mood claims, which are the core of the abstract, inherit the label problem. This is not an internal inconsistency, but it makes the headline accuracies uninterpretable as clinical-state discrimination without an external label source.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, a DPhil thesis deposited on arXiv, proposes a computational framework for automated symptom assessment in mental health from physical activity and phone-use data collected in ambulatory settings. It describes the AMoSS study, data pre-processing and segmentation methods, a symptom-driven feature framework, and three application areas: differentiation between healthy controls, bipolar disorder, and borderline personality disorder; differentiation between euthymic, manic, and depressive states; and personalised prediction of mood questionnaire scores. It also applies the framework to a separate schizophrenia dataset with heart-rate fusion. The headline results reported under leave-one-out cross-validation are 67--90% accuracy for disorder/state discrimination, 95.3% accuracy for schizophrenia versus controls, and mean absolute errors of 1.36--3.32 points for personalised mood regression.","tokens_in":51698,"tokens_out":4770,"duration_ms":52749,"significance":"If the reported results hold, this work would strengthen the evidence that passive sensor data contain clinically usable information about psychiatric state, and the proposed symptom-driven feature framework is a useful organizing principle for mHealth psychiatry. The strengths of the manuscript include a comparatively large longitudinal ambulatory cohort, the use of consumer devices alongside research-grade sensors, a principled mapping from Liddle's symptom dimensions to objective features, and an independent external schizophrenia dataset. However, the headline claims are conditional on two unresolved issues: the validity of self-report questionnaires as ground truth for mood state, and the integrity of the cross-validated performance estimates in small, resampled cohorts. The diagnosis-level comparisons are less exposed to the label-validity problem, but the state-discrimination and personalised mood claims, which are central to the abstract, currently inherit it.","major_comments":[{"comment":"The state-discrimination results in Chapter 7 are not yet interpretable as clinical-state discrimination because the ground-truth labels are derived from self-report questionnaires whose missingness is state-dependent. The manuscript itself states that 'compliance can be a function of the clinical state, i.e. the patient may stop responding during a manic or depressive episode', and §4.3.1 then imputes missing weekly questionnaire scores with the unconditional mean. Under missing-not-at-random nonresponse, this procedure assigns missing episodes an average or euthymic-like score, so the remaining labels may track ease of self-report or device wear rather than true mood. The authors should quantify questionnaire missingness by state, test whether missingness is associated with concurrent self-report scores, and re-run the Chapter 7 and Chapter 8 analyses using only observed labels or an external clinician-rated episode source.","section":"§3.2.1.2, §4.3.1"},{"comment":"The cross-validated accuracy numbers in Tables 6.5 and 7.5 are point estimates from a protocol that does not, as presented, nest under-sampling, SMOTE, and feature selection inside each training fold. Under-sampling the majority class and applying mRMR or LASSO on the full data subset before LOOCV can inflate accuracy because information from the held-out subject leaks into feature selection and resampling. Please clarify the exact order of operations; if these steps are not nested within folds, the analyses should be redone with a nested or fully independent cross-validation pipeline. In addition, report bootstrap confidence intervals or per-subject prediction tables, given the small cohort sizes.","section":"§6.2.2, §6.3.1, §6.4.3, §7.4.3"},{"comment":"The selection of BOCPD with an expected segment length of 60 seconds for human accelerometer data is based on synthetic RR tachograms generated by the McSharry--Clifford model, with the rationale that heart rate correlates with physical activity. This is an unvalidated assumption: the model generates heart-rate dynamics, not accelerometer segment statistics, and the paper does not demonstrate that the segment-duration distributions of the two signals are similar. Since the BOCPD segment features feed into the psychomotor and disorganisation feature sets used in later chapters, the algorithm choice should be validated against human activity data with annotated change points, or at least cross-checked against the Fitbit bed/wake annotations described in §4.5.3.","section":"§4.5.1.5–§4.5.1.6"},{"comment":"The personalised mood regression results inherit the label-validity problem of the daily Mood Zoom scores. The reported mean absolute errors are computed against the same self-report scores that are subject to state-dependent missingness and subjective bias, so the errors are not necessarily errors in an external clinical state. In addition, Table 8.5 does not report how many days per participant had imputed versus observed labels, which matters because unconditional-mean imputation will artificially improve apparent agreement when compliance is low. Please provide observed-only results and a missingness analysis.","section":"§8.4.2, §8.5, Table 8.5"},{"comment":"The schizophrenia classification result of 95.3% and the claimed 10--17% improvement from adding heart-rate features are presented without per-fold breakdowns or confidence intervals. With the small number of participants in the Nuffield and Proteus datasets, a few individuals can drive the difference between feature sets, especially when feature selection is performed on the full cohort. Please report leave-one-out predictions per participant for the activity-only, HR-only, and combined feature sets, and state explicitly whether feature selection and any resampling were performed inside each fold.","section":"§9.4, Table 9.4"}],"minor_comments":[{"comment":"The BOCPD expected-segment-length hyperparameter is denoted λ in the equations and Table 4.2, but the text that selects the value refers to it as τ = 60 seconds; please unify the notation.","section":"§4.5.1.3, §4.5.1.6"},{"comment":"The statistical test is spelled 'Wilcox rank sum test' in several places; the correct name is the Wilcoxon rank sum test.","section":"Tables 3.2, 3.5, 3.7"},{"comment":"There are numerous typographical errors that should be corrected in a copyedit, including 'biploar disorder' in Figure 3.3, 'primarely' in the Glossary, and inconsistent hyphenation and encoding artifacts in 'Na¨ıve'.","section":"Throughout"},{"comment":"The evaluation of change-point algorithms reports TPR and FPR, but the captions do not state the number of generated tachograms, the exact definition of the tolerance interval δ, or how true change points were defined from the synthetic model; please add these details.","section":"§4.5.1.4–§4.5.1.6"},{"comment":"The mobile application section notes that 'no formal comparison were performed between smart phone characteristics with and without AMoSS mobile application'; this usability limitation should be acknowledged in the conclusions as a potential source of battery- or performance-related non-adherence.","section":"§3.1.3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a thesis-length document rather than a conventional journal article, and the central results may overlap with the listed publications in §1.6, particularly Osipov et al. (2015) and Palmius et al. (2014). The editor should ask the authors to clarify the intended publication venue and the novel contribution relative to those papers. I am also concerned that the headline accuracies, while plausible as point estimates, are not yet established because of the label-missingness and cross-validation issues described in the major comments; these are fixable in principle but may require a substantial reanalysis or additional external validation data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a DPhil thesis, not a typical arXiv paper, and it reads like one — comprehensive, careful in places, and honest about its own limitations. The genuinely new asset is the AMoSS dataset: over 100 participants with up to a year of phone and wearable sensor data, plus weekly and daily self-report mood questionnaires. The framework that maps Liddle's symptom dimensions (psychomotor, disorganisation, mood) onto concrete objective features is a real contribution, and the methods are mostly established but sensibly combined. The HMM bed/wake segmentation, validated against Fitbit button presses, is solid work, and the schizophrenia cohort with heart-rate fusion showing a 10% accuracy gain over activity alone is the cleanest result in the thesis.\n\nThe soft spots are where the abstract overreaches. The state-level accuracies (euthymia vs mania 80%, euthymia vs depression 85%, mania vs depression 90%) come from labels that are entirely self-report — QIDS-SR16, ASRM, Mood Zoom. The thesis itself admits (Section 3.2.1.2) that compliance can drop during manic or depressive episodes, and Section 4.3.1 imputes missing questionnaire responses with the unconditional mean. That is a textbook MNAR setup: severe episodes disappear or get pulled toward euthymic means, and the classifier may be separating questionnaire response patterns from true mood. The stress-test note is right. The diagnostic-level results (HC vs BD/BPD) use clinician diagnoses and are less exposed, but the abstract's core claims about state discrimination and personalised mood prediction inherit the label problem. LOOCV point estimates without confidence intervals, plus under-sampling and SMOTE in small cohorts, make the numbers look more precise than they are. No code or data sharing, and no external validation, so these accuracies are in-sample.\n\nThat said, the thesis is not sloppy. It has a thoughtful limitations chapter (Section 10.2) and the analyses are competently executed. The problem is interpretation, not execution. This is a paper for researchers in digital psychiatry and mHealth who want to see what a serious longitudinal effort looks like and where the field's pitfalls are. It deserves a serious referee, but the verdict should be conditional: reframe the claims as exploratory, add confidence intervals, and ideally validate on clinician-rated states or an external cohort before calling anything clinically usable.","headline":"A substantial thesis with a new longitudinal dataset and a sensible symptom-driven framework, but the headline state-discrimination accuracies rest on self-report labels whose missingness is likely state-dependent, so the clinical-readiness claims need to be dialed back.","tokens_in":52166,"tokens_out":2136,"would_cite":true,"duration_ms":25269,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis claims that passive physical-activity and phone-use data can differentiate psychiatric diagnoses and clinical mood states with reported accuracies of 67–95.3%, and can predict personalised mood scores with errors of 1.36–3.32…","keywords":["mental health","actigraphy","physical activity","bipolar disorder","borderline personality disorder","schizophrenia","mood state classification","mHealth monitoring"],"falsifier":"Run the same feature pipeline on a cohort where clinical states are independently adjudicated by structured clinical interview rather than self-report, and check whether the classifier's sensitivity and specificity for clinician-confirmed mania and depression remain at the reported levels when questionnaires are missing, delayed, or contradicted by clinician ratings.","tokens_in":51221,"feed_emoji":"📱","tokens_out":5133,"duration_ms":46750,"temperature":0.7,"pith_summary":"This thesis sets out to show that continuous, passively collected measurements of physical activity, sleep, and phone use carry enough information about psychiatric state to serve as objective symptom markers. Using accelerometer data from wrist-worn devices and smartphones, together with questionnaire-based mood labels, it reports classifiers that separate healthy controls from bipolar disorder (67%), healthy controls from borderline personality disorder (70%), bipolar from borderline personality disorder (80%), and the bipolar mood states euthymia, mania, and depression from each other (80–90%). A separate schizophrenia-versus-controls analysis reports 95.3% accuracy under leave-one-out cross-validation, with heart rate adding substantial predictive power to activity-based features. The broader claim is that disorders and clinical episodes leave measurable behavioural signatures that could enable early detection of deterioration in ambulatory patients.","feed_headline":"Passive sensor data sort psychiatric diagnoses and mood episodes","feed_subtitle":"A thesis reports 67–95% accuracy using wrist and phone activity, plus near-point mood prediction.","key_machinery":"The central object is a feature pipeline built from accelerometer time series. Tri-axial acceleration is converted into the Euclidean Norm Minus One (ENMO) metric for day-time activity, while epoch-based activity counts are used for sleep analysis. Non-stationarity is treated as signal rather than noise: the Bayesian Online Change Point Detection algorithm segments activity into stationary segments, and the durations and transitions of those segments become features. Sleep and wakefulness are identified with an Explicit Duration Hidden semi-Markov Model whose parameters are trained on device-based bed-time annotations. These features are then ranked with minimum Redundancy Maximum Relevance or LASSO, and classified with logistic regression or support vector machines under leave-one-out cross-validation.","core_discovery":"The central claim is that objective features extracted from activity and behaviour time series can differentiate mental health diagnoses and clinical mood states with clinically useful accuracy. The features are organised around three symptom dimensions from a five-cluster symptoms model: psychomotor (activity level and intensity), disorganisation (multiscale entropy, activity persistence, and day-to-day pattern variability), and mood (sleep-wake segmentation, circadian amplitude, and non-parametric rest-activity characteristics). Personalised regression models predict mood scores with a mean absolute error of 1.36 to 3.32 points, which falls within the 4–5 point ranges that psychiatric questionnaires reserve for distinct identifiable mood states. Adding heart-rate features to locomotor features improves schizophrenia classification by almost 10% over activity alone and by almost 17% over heart-rate features alone. The thesis argues that these results support a framework for computational behaviour analysis that could identify clinical deterioration earlier than routine clinic visits.","pith_inferences":["The accuracy figures should be read as separating questionnaire-labelled states, not clinician-validated episodes; if questionnaire compliance is itself state-dependent, the classifiers may partly be detecting response behaviour rather than the underlying episode, a limitation the thesis acknowledges.","A natural next test is an external cohort with clinician-rated episodes and dense sensor data to see whether the 67%–95.3% accuracies survive independent adjudication.","The framework implies that symptom dimensions, rather than diagnostic categories, are the more tractable prediction target; the same features that separate mood states could extend to other conditions with psychomotor or circadian disruption.","A testable extension is applying the pipeline to consumer wristbands with lower sampling rates, to see whether the accuracy gains persist outside research-grade accelerometers."],"forward_implications":["If a patient's sensor stream can flag mood episodes between clinic visits, clinicians could be alerted to deterioration earlier than weekly self-report alone would allow.","The reported accuracy levels suggest sensor-based features could act as a screening layer that prompts targeted clinical interviews, rather than replacing clinician judgement.","Personalised mood models imply that each patient's behavioural baseline can be learned, making deviations from that baseline more informative than population-level thresholds.","The schizophrenia result, where fusing heart rate and activity improved classification, suggests multi-modal sensing may be needed for disorders whose behavioural signature is weaker.","A standardised feature framework tied to symptom dimensions gives future studies a common language for comparing objective mental-health monitoring results."],"supporting_citations":[{"why":"Supplies the five-cluster symptom dimensions (reality distortion, disorganisation, psychomotor, mood, anxiety) that structure the thesis's feature framework.","marker":"Liddle, 2001"},{"why":"Provided the motivating premise that activity and motion analysis has potential as a diagnostic tool for mental disorders.","marker":"Teicher, 1995"},{"why":"Validates the ENMO metric and auto-calibration procedures used to convert raw tri-axial acceleration into day-time activity features.","marker":"van Hees et al., 2013"},{"why":"Defines the Bayesian Online Change Point Detection algorithm used to segment activity into stationary segments for disorganisation features.","marker":"Adams and MacKay, 2007"},{"why":"Provides the realistic heart-rate tachogram generator used to evaluate change-point detection algorithms before applying them to activity data.","marker":"McSharry et al., 2002"},{"why":"QIDS-SR16 self-report questionnaire that serves as ground truth for depression severity in the mood-state analyses.","marker":"Rush et al., 2003"},{"why":"ASRM self-report scale for mania that serves as ground truth for identifying manic episodes.","marker":"Altman et al., 1997"},{"why":"Prior actigraphy study showing structural activity measures associate with disorganisation symptoms, motivating the disorganisation feature cluster.","marker":"Walther et al., 2014"},{"why":"Introduced the activity-derived Vulnerability Index distinguishing bipolar disorder risk and states, supporting the use of activity structure for mood assessment.","marker":"Indic et al., 2011"},{"why":"Earlier smartphone-based monitoring of bipolar disorder that provides the mHealth monitoring baseline the thesis extends.","marker":"Bardram et al., 2012"}],"fun_headline_variants":["Activity patterns separate psychiatric diagnoses and mood states","Heart-rate data improves schizophrenia diagnosis by 10 percent","Wearable sensors predict mood scores in clinical ranges","Objective activity metrics spot mental health changes early"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth clinical states are defined by self-report questionnaire scores (QIDS-SR16, ASRM, and Mood Zoom), and the thesis concedes that questionnaire compliance can itself change with clinical state; if the labels are wrong or missing during episodes, the reported accuracies may reflect questionnaire response patterns rather than true clinical states.","fun_headline_variants_meta":{"raw":{"variants":["Activity patterns separate psychiatric diagnoses and mood states","Heart-rate data improves schizophrenia diagnosis by 10 percent","Wearable sensors predict mood scores in clinical ranges","Objective activity metrics spot mental health changes early"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2726,"prompt_tokens":923,"completion_tokens":1803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1744}},"tokens_in":539,"tokens_out":1803,"duration_ms":14464,"temperature":1.0,"reasoning_tokens":1744,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:16:59.248467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same feature pipeline on a cohort where clinical states are independently adjudicated by structured clinical interview rather than self-report, and check whether the classifier's sensitivity and specificity for clinician-confirmed mania and depression remain at the reported levels when questionnaires are missing, delayed, or contradicted by clinician ratings.","supporting_citations":[],"review_version":1}