{"id":"730001fe-cdf8-4fb8-8103-382e9cc1a326","arxiv_id":"2506.06306","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A LightGBM model using sensor features plus time-of-day and current agitation status predicts next-6-hour agitation with AUC-ROC 0.972 and AUC-PR 0.432 on the TIHM dataset, but random-fold evaluation and label-derived features leave the true predictive value unclear.","lead":"This study benchmarks machine learning models that predict agitation episodes in people with dementia up to six hours ahead using in-home sensor data from the TIHM dataset. The best model reaches high ROC scores when time-of-day and current agitation status are added, but the paper lacks a time-only baseline and may overstate early prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.9720 AUC-ROC headline is unsupported as a generalizable estimate because the feature-augmented results in Figure 3 use only random 5-fold CV without participant or temporal blocking; the paper's own LOPO analysis shows 0.8712 AUC-ROC for the same base LightGBM setting, so the central…","rationale":"The reader's diagnosis is sound: Table 1 provides both 5-fold and LOPO results, which makes the inflation visible. That transparency is a genuine strength; the paper is not hiding the problem, but it presents the random-CV number as the headline. I do not see grounds for rejection: LOPO AUC-ROC values of roughly 0.85-0.89 for several models are still above chance and clinically interesting, and the dataset is valuable. However, the abstract's 0.9720 figure is specifically the random-CV, all-features LightGBM, and no participant-blocked or temporally blocked version of that exact setting is reported. In addition, because Section 3 shows agitation is strongly concentrated in afternoon and evening and Section 5.2 explicitly adds day-quarter as a feature, the paper lacks a context-only baseline that would show the sensor features themselves carry signal beyond time of day. Both issues are fixable with additional analyses and code release, so conditional acceptance is the right outcome: the promised LOPO/temporal-block evaluation and context-only baseline would settle whether the central claim survives. My read does not move the verdict; it sharpens the condition under which the paper should be accepted.","tokens_in":13163,"tokens_out":4379,"duration_ms":39812,"concrete_test":"Re-run the Figure 3 feature-augmented LightGBM comparison under (a) leave-one-participant-out CV and (b) temporal-block CV, e.g., train on the first K days per participant and test on later days. Report AUC-ROC and AUC-PR for both protocols, and additionally include a baseline trained only on day-quarter and current-agitation features with no sensor inputs. If the sensor-plus-context model does not retain a substantial advantage over the context-only baseline under grouped CV, then the headline 0.9720 is not evidence for sensor-based early prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim turns on treating 6-hour samples as exchangeable units. Section 5 (Table 1) describes '5-fold CV' without grouping by participant or by contiguous time blocks, and Figure 3, the source of the 0.9720/0.4320 numbers, reports only 5-fold CV. Consecutive 6-hour windows from the same participant are not independent: they share physiology, activity routines, and can lie inside the same agitation episode. Random splits therefore place near-duplicate samples in both training and test folds, allowing the model to memorize participant- and episode-specific patterns. This is not hypothetical: the LOPO rows of Table 1, which respect participant independence, drop LightGBM W from 0.9099 to 0.8712 AUC-ROC and Gradient Boosting W from 0.8930 to 0.8588. No LOPO or time-blocked result is reported for the feature-augmented setting in Figure 3. The paper is internally honest in reporting both protocols, but the abstract's 'highest AUC-ROC of 0.9720' is specifically the random-CV, all-features LightGBM, so as published the central quantitative claim is optimistic and not established for unseen participants or future time periods.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks machine learning models for early agitation prediction in community-dwelling people with dementia using the TIHM dataset. It introduces contextual activity features, compares binary classification (tabular and sequential) and anomaly detection formulations, and evaluates models with 5-fold and leave-one-participant-out cross-validation. The best reported result is an AUC-ROC of 0.9720 achieved by LightGBM with day-quarter and current agitation status features under 5-fold CV. The authors claim this is the first comprehensive benchmarking of agitation prediction in community-based dementia care.","tokens_in":13453,"tokens_out":4699,"duration_ms":33281,"significance":"If the reported performance is reliable, the paper would provide a useful baseline for a clinically important prediction task using a public dataset. The study's strengths include the use of a real-world multimodal dataset, a wide model comparison, and SHAP-based explainability. However, the central quantitative claim rests on a 5-fold CV protocol that does not respect participant or temporal independence; the paper's own LOPO results are substantially lower. The manuscript is therefore not yet ready in its current form.","major_comments":[{"comment":"The 5-fold CV appears to split the 6-hour samples randomly across the full pooled dataset, without grouping by participant or by contiguous time blocks. Consecutive samples from the same participant are highly autocorrelated, so random splits can place near-duplicates of test samples in the training folds, inflating AUC. The paper's own LOPO rows in Table 1 show LightGBM W dropping from 0.9099 to 0.8712 AUC-ROC, and Gradient Boosting W from 0.8930 to 0.8588, when participant independence is respected. Because Figure 3 reports only 5-fold CV, the headline 0.9720 is not a valid estimate of performance for unseen participants or future time periods. Please report LOPO or temporally blocked CV for the augmented feature settings and adjust the abstract and conclusions accordingly.","section":"Section 5, Table 1 and Figure 3"},{"comment":"The statement that including day-quarter and current agitation status \"does not introduce information leakage\" is correct only in the narrow sense of not using future labels. Current agitation status is the clinician-labeled outcome at time t, which is the same label stream as the target at t+1; if an agitation episode spans multiple 6-hour windows, the model can learn to predict continuation rather than onset. Under the random-split 5-fold CV used in Figure 3, this autoregressive feature can be memorized per participant or per episode. The paper should quantify how much of the improvement from 0.9099 to 0.9720 is due to the current agitation feature under LOPO or episode-aware splitting.","section":"Section 5.2"},{"comment":"Several unweighted models report F1-score 0 and sensitivity 0 (e.g., Gradient Boosting and LightGBM under 5-fold), meaning they predict no positive cases at the default threshold. Reporting accuracy 0.9875 for these models is misleading without noting that they are trivial classifiers; AUC values may be acceptable, but the table should either report threshold-independent metrics only or include a note on default thresholds. This affects comparability across models and protocols.","section":"Table 1"},{"comment":"The SHAP feature importance analysis is performed only under the same 5-fold CV protocol, so the feature rankings may be influenced by the same sample-autocorrelation issue. Recomputing the SHAP analysis under LOPO or a temporally blocked split would clarify whether the identified features (e.g., hallway-count-std) are stable predictors for unseen participants.","section":"Section 5.3"}],"minor_comments":[{"comment":"The phrase \"up to six hours in advance\" is ambiguous; the prediction is for the next 6-hour timestamp, so the actual horizon is 0 to 6 hours after the current window. Please clarify the prediction horizon.","section":"Abstract"},{"comment":"There are typos in the axis labels: \"Numbe\" in Figure 1 and \"agiatation\" (twice) in Figure 3. Please correct these.","section":"Figures 1 and 3"},{"comment":"The sentence reporting the interval between consecutive agitation episodes states \"the mean and standard deviation of the interval ... were 2.43 days and 4.15 days\" but does not specify which value is the mean and which is the standard deviation. Please clarify.","section":"Section 3"},{"comment":"Sleep data are excluded because of high missingness (70.21%), but the decision rule is not stated. Please justify the exclusion threshold and cite the referenced guidance [36,37] more specifically.","section":"Section 4.1"},{"comment":"The literature review is qualitative; a table comparing prior work on agitation prediction/detection (sample size, prediction vs detection, setting, sensor modalities, performance) would help substantiate the \"first comprehensive benchmarking\" claim.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a compact benchmark study with a potentially useful comparison across many models and problem formulations. The main concern is the evaluation protocol for the headline results: Figure 3 uses only random 5-fold CV, while the paper's own LOPO analysis shows a substantial drop in performance. I recommend requesting LOPO or temporal-blocked CV results for the augmented features, and softening the abstract claims if those results are materially lower. The claims of being the 'first comprehensive benchmarking' should also be checked against related work on agitation prediction (e.g., Homdee, HekmatiAthar) and calibrated accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it is the first to turn the public TIHM dataset toward agitation prediction rather than detection, and it benchmarks a wide range of model families (tabular, sequential, anomaly) on 56 participants with 2,803 days of data. The eight contextual activity features are a modest but reasonable addition, and the paper deserves credit for reporting leave-one-participant-out (LOPO) results alongside 5-fold CV. That internal honesty makes the central claim easier to assess.\n\nThe assessment, though, is not kind to the abstract. The headline 0.9720 AUC-ROC comes from random 5-fold CV on 6-hour blocks, which leaks both participant identity and episode continuity into training folds. The paper's own LOPO table shows the same LightGBM setting dropping to 0.8712. And the feature-augmented results in Figure 3 — the ones that reach 0.9720 — are reported only under 5-fold CV, never LOPO. So the abstract's best number is exactly the one most likely to be inflated.\n\nThe bigger issue is the 'current agitation status' input feature. It is the clinician label stream shifted one step back. When agitation episodes span multiple 6-hour intervals, that feature mostly tells the model the episode is already underway — continuation, not onset. The paper itself notes that 86.67% of episodes are not preceded by agitation, so this feature has limited value for early warning. Using it to claim early prediction is misleading.\n\nI also miss a time-of-day-only baseline. The paper adds day-quarter as a feature and shows improvement, but never quantifies how much a simple 'predict afternoon/evening' rule would get. Given the well-documented sundowning pattern in this very dataset, that baseline could eat a large chunk of the apparent gain.\n\nThese are fixable problems, and the paper's LOPO results (AUC-ROC in the 0.87–0.89 range) still suggest real signal in the sensors. But as published, the central quantitative claim is not established for unseen participants or future time periods.\n\nWho is this for? Clinical ML and gerontechnology researchers, and anyone who wants a concrete case study in why random CV fails on longitudinal health data. It deserves a serious referee, but the verdict should be major revision: group CV by participant and time block, report LOPO for the augmented features, add a time-of-day baseline, drop or reframe the current-agitation feature, and give error bars. With those changes, this would be a citable benchmark.","headline":"Useful first benchmark for agitation prediction on TIHM, but the headline 0.9720 AUC-ROC is an artifact of random 5-fold CV and a label-derived feature; the LOPO numbers are the honest ones.","tokens_in":14001,"tokens_out":2596,"would_cite":false,"duration_ms":23716,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agitation in dementia can be forecast six hours ahead using passive in-home sensors, with a LightGBM model reaching AUC-ROC 0.9720 on a public benchmark.","keywords":["agitation prediction","early prediction of agitation","multimodal sensors","people living with dementia","machine learning","deep learning","community-based dementia care","TIHM dataset"],"falsifier":"Recompute the LightGBM result under participant-grouped cross-validation, keeping all samples from one participant in either the training or test set; if the AUC-ROC drops toward the leave-one-participant-out value of 0.8712 or lower, the 0.9720 figure reflects information leakage between consecutive time blocks rather than true predictive skill.","tokens_in":12945,"feed_emoji":"🧠","tokens_out":4600,"duration_ms":35887,"temperature":0.7,"pith_summary":"This paper claims that agitation episodes in people with dementia living at home can be predicted up to six hours in advance by machine learning models trained on passive in-home motion and physiology sensors. Using the TIHM dataset of 2,803 days from 56 community-dwelling participants, it benchmarks multiple models and problem formulations, with the strongest result from binary classification on 6-hour sensor windows: LightGBM reaches AUC-ROC 0.9720 and AUC-PR 0.4320 when day-quarter and current-agitation status are added as inputs. If correct, this would mean caregivers could be alerted before a distressing episode begins, using only privacy-preserving sensors rather than video or wearable devices that require active participation. The authors present this as the first broad benchmark of agitation prediction in community-based dementia care.","feed_headline":"Sensor data predicts dementia agitation six hours ahead","feed_subtitle":"Passive in-home motion and physiology monitoring flags agitation risk up to six hours early with an AUC of 0.9720.","key_machinery":"The central object is a feature pipeline over the TIHM dataset: 32 statistical activity features (sum, maximum, mean, and standard deviation of hourly motion counts across eight household locations), 8 novel contextual activity features (total events, unique locations, location entropy, private-to-public ratio, location dominance ratio, back-and-forth count, active-location ratio, and number of transitions), and 8 physiology features (means of the eight physiological measurements per 6-hour timestamp). The prediction target is the clinician-verified agitation label at timestamp $t+1$, and the task is binary classification on the feature vector at time $t$. The argument is carried by the LightGBM gradient boosting classifier with a weighted loss function, evaluated under 5-fold cross-validation, with SHAP feature attributions identifying hallway movement variability as the top predictor.","core_discovery":"The central claim is that the current 6-hour timestamp of multimodal in-home sensor data is enough to predict whether agitation will occur in the next 6-hour window, and that adding contextual information (time of day and whether agitation is already occurring at the current timestamp) materially improves that prediction. In the binary tabular formulation with 5-fold cross-validation, the light gradient boosting machine achieves AUC-ROC 0.9720 and AUC-PR 0.4320, the highest reported in the paper. The paper also finds that a transformer model using the two most recent timestamps reaches AUC-ROC 0.9531 and AUC-PR 0.2277, and that anomaly-detection formulations are weaker overall, though One-Class SVM yields higher AUC-PR than the classification models. Taken together, the authors argue that agitation prediction, not just detection, is feasible in community settings with non-intrusive, privacy-preserving sensors.","pith_inferences":["Inference: The headline 5-fold CV results likely overstate real-world performance if consecutive 6-hour blocks from the same participant are not independent; re-evaluating with grouped or temporal cross-validation would probably lower the reported AUC-ROC.","Inference: Because time of day (sundowning) is such a strong predictor, a testable extension is to compare this model against a baseline that uses only clock time and current agitation status, with no sensor features, to quantify how much sensor data actually contributes.","Inference: The modest AUC-PR of 0.4320, despite a high AUC-ROC, implies that the positive class is hard to detect with precision; in practice, alert systems may need a high false-alarm tolerance or additional filters.","Inference: Excluding sleep data, which had 70% missingness, removes a modality that is plausibly informative for agitation; better imputation or dedicated sleep collection could improve predictions beyond what this benchmark shows."],"forward_implications":["Caregivers or monitoring systems could receive alerts up to six hours before a likely agitation episode, creating a window for behavioral or environmental intervention.","The approach relies only on passive motion sensors and routine physiological measurements such as blood pressure, heart rate, and weight, avoiding video and audio privacy concerns.","The strong contribution of day-quarter and current agitation status suggests that simple temporal context, rather than richer sensing modalities, drives much of the predictive accuracy.","The drop from 5-fold to leave-one-participant-out CV (LightGBM AUC-ROC from 0.9099 to 0.8712) shows that performance for unseen participants is lower but still substantial, indicating partial generalization across individuals.","The public TIHM benchmark can serve as a common testbed for future agitation-prediction methods, allowing direct comparison of new approaches against the reported numbers."],"supporting_citations":[{"why":"Supplies the TIHM dataset, including the sensor modalities, 6-hour labels, and the agitation episodes used for all experiments.","marker":"[11]"},{"why":"Established the earlier systematic review of sensor-based agitation detection and prediction, defining the gap this paper addresses.","marker":"[23]"},{"why":"Prior deep-learning agitation forecasting on a single participant, used as a motivating comparison for scalable prediction.","marker":"[12]"},{"why":"Earlier small-sample prediction study using environmental sensors, cited to show limited prior work on agitation prediction.","marker":"[34]"},{"why":"The LightGBM algorithm that produces the best reported AUC-ROC and AUC-PR results.","marker":"[44]"},{"why":"Gradient boosting baseline used throughout the benchmarking tables and figures.","marker":"[43]"},{"why":"SMOTE oversampling used to address the highly imbalanced distribution of agitation labels.","marker":"[50]"},{"why":"Class-weighted loss function used to improve minority-class sensitivity in the classification models.","marker":"[49]"},{"why":"SHAP feature attributions used to identify which sensor features drive the agitation predictions.","marker":"[51]"}],"fun_headline_variants":["6-hour sensor window predicts next dementia agitation","In-home sensors forecast agitation 6 hours early in dementia","Multimodal sensors predict dementia agitation with 0.972 AUC","Sensor data flags dementia agitation risk six hours ahead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results assume that 6-hour data segments from the same participant are independent and exchangeable, so randomly splitting them into training and test sets does not leak information about the same person or the same agitation episode into both sides of the experiment.","fun_headline_variants_meta":{"raw":{"variants":["6-hour sensor window predicts next dementia agitation","In-home sensors forecast agitation 6 hours early in dementia","Multimodal sensors predict dementia agitation with 0.972 AUC","Sensor data flags dementia agitation risk six hours ahead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1316,"prompt_tokens":1032,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":648,"tokens_out":284,"duration_ms":2922,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:31:27.932731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the LightGBM result under participant-grouped cross-validation, keeping all samples from one participant in either the training or test set; if the AUC-ROC drops toward the leave-one-participant-out value of 0.8712 or lower, the 0.9720 figure reflects information leakage between consecutive time blocks rather than true predictive skill.","supporting_citations":[{"cited_title":"Tihm: An open dataset for remote healthcare monitoring in dementia,","cited_arxiv_id":null,"evidence_quote":"Supplies the TIHM dataset, including the sensor modalities, 6-hour labels, and the agitation episodes used for all experiments."},{"cited_title":"Applications and outcomes of internet of things for patients with alzheimer’s disease/dementia: A scoping review,","cited_arxiv_id":null,"evidence_quote":"Established the earlier systematic review of sensor-based agitation detection and prediction, defining the gap this paper addresses."},{"cited_title":"Data-driven forecasting of agitation for persons with dementia: A deep learning-based ap- proach,","cited_arxiv_id":null,"evidence_quote":"Prior deep-learning agitation forecasting on a single participant, used as a motivating comparison for scalable prediction."},{"cited_title":"Leveraging Self-Training and Variational Autoencoder for Agitation Detection in People with Dementia Using Wearable Sensors","cited_arxiv_id":"2412.19254","evidence_quote":"Earlier small-sample prediction study using environmental sensors, cited to show limited prior work on agitation prediction."},{"cited_title":"Class-balanced loss based on effective number of samples,","cited_arxiv_id":null,"evidence_quote":"SMOTE oversampling used to address the highly imbalanced distribution of agitation labels."},{"cited_title":"One-class classification with node embedding type fea- tures,","cited_arxiv_id":null,"evidence_quote":"Class-weighted loss function used to improve minority-class sensitivity in the classification models."},{"cited_title":"Smote: syn- thetic minority over-sampling technique,","cited_arxiv_id":null,"evidence_quote":"SHAP feature attributions used to identify which sensor features drive the agitation predictions."}],"review_version":1}