{"id":"e2ffea58-a2da-4071-a8e8-f43ac02ab4e9","arxiv_id":"2412.19254","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A variational autoencoder plus self-training pipeline is reported to detect agitation in dementia patients from wristband sensor data, reaching 90.18% balanced accuracy with XGBoost.","lead":"The authors combine a variational autoencoder with self-training to detect agitation in dementia patients from wristband sensor data, using a mix of labeled and unlabeled recordings. If valid, the approach would ease the labeling bottleneck that limits AI-based agitation monitoring in hospitals.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 90.18% balanced accuracy is not supported because the 70/30 split appears to include the model's own pseudo-labels in the test set, making the evaluation measure self-consistency rather than clinical AA detection.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the test set may contain pseudo-labels and the split is not participant-level. My reading of Section 3.3 and the label-count tables confirms this is not a minor ambiguity but a likely evaluation flaw. Table 5's counts sum to nearly the full dataset, which strongly suggests the test partition included pseudo-labeled unlabeled windows unless the authors silently excluded them. Since the paper provides no code, data, or explicit statement that pseudo-labels were kept out of the test set, the central numerical claim is not verifiable and is likely inflated. This does not impugn the clinical motivation or the general idea of combining VAEs with self-training; it simply means the reported 90.18% balanced accuracy cannot be taken as evidence of real-world AA detection. The reader's REJECT verdict remains appropriate, so I recommend no change.","tokens_in":14435,"tokens_out":2414,"duration_ms":26151,"concrete_test":"Re-run the self-training + VAE pipeline but construct the test set strictly from the 20,279 originally clinician-labeled minutes (18,804 normal + 1,475 AA) from the 5 fully labeled participants, using leave-one-participant-out splitting and excluding every pseudo-labeled sample from the test partition. Report balanced accuracy and AUC ROC on those original labels only. If XGBoost's balanced accuracy is substantially below 90.18% or drops below the supervised baseline, the claimed improvement is an artifact of test-set contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VAE feature extraction plus self-training significantly improves agitation detection, with XGBoost reaching 90.18% balanced accuracy and 99.6% AUC ROC (Section 4.4, Table 5). The load-bearing condition is that the evaluation uses only genuine clinician-provided labels on data not seen during training. The paper does not establish this. Section 3.3 says the dataset is split into 70% training and 30% testing, but does not say that pseudo-labeled samples are excluded from the test partition. The label counts in Tables 3–5 show that after self-training the dataset is augmented to near the full 57,742 minutes (e.g., Table 5: 54,712 + 3,026 = 57,738), so a 30% test slice taken after augmentation necessarily contains many windows whose labels were generated by the self-training model itself, unless an explicit exclusion is made. No such exclusion is described. Because self-training adds pseudo-labels based on model confidence, evaluating on those same pseudo-labels inflates balanced accuracy and AUC: the model is being scored on its own predictions. Additionally, the split is at the one-minute window level rather than the participant level, so even for originally labeled data, windows from the same participant appear in both training and test sets, further weakening generalization claims. Without a test set composed only of original clinical labels from held-out participants, the headline numbers do not support the conclusion that the pipeline detects agitation in new patients.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a semi-supervised pipeline for detecting agitation/aggression (AA) in people with dementia from Empatica E4 physiological data. The dataset includes 57,742 one-minute windows from 14 participants across three studies, of which 18,804 minutes are labeled normal, 1,475 are labeled AA, and 37,463 are unlabeled. The proposed system extracts features with the Flirt toolkit, learns a compact representation with a variational autoencoder (VAE), and then applies self-training with three classifiers (Random Forest, Extra Trees, XGBoost). The authors report that XGBoost with VAE and self-training achieves a balanced accuracy of 90.18% and an AUC ROC of 99.6%, and they conclude that combining VAE with self-training significantly improves AA detection.","tokens_in":14713,"tokens_out":7553,"duration_ms":69942,"significance":"The application is practically relevant, and the multi-site partially labeled dataset is a potentially useful resource. If the reported results were trustworthy, the paper would offer evidence that self-training can mitigate the labeling bottleneck in wearable-based agitation detection. However, the evaluation protocol does not support the central claim: the test set appears to contain pseudo-labeled windows, the split is at the window rather than participant level, and the supervised baseline in Table 3 is internally inconsistent. The quantitative headline is therefore not established, although the underlying idea and dataset may be salvageable with a rigorous re-analysis.","major_comments":[{"comment":"The manuscript does not state that pseudo-labeled samples are excluded from the 30% test set. Section 3.2.3 describes adding pseudo-labeled samples to the training set, and Section 3.3 describes only a random 70/30 split. After self-training, the Normal/AA counts in Table 4 (e.g., 55,214/2,523 for Random Forest) and Table 5 (e.g., 54,712/3,026 for XGBoost) sum to nearly the full 57,742-minute dataset, so a 30% slice of the augmented data necessarily contains many windows whose labels were produced by the classifier itself. Unless the authors document an explicit exclusion of all pseudo-labeled windows and all unlabeled windows from the test partition, balanced accuracy and AUC measure agreement with the model's own pseudo-labels rather than clinical AA detection.","section":"§3.3, §3.2.3, Tables 4–5"},{"comment":"The split is performed at the level of one-minute windows, not participants. With only five fully labeled participants, windows from the same participant are almost certainly present in both training and testing, so the reported metrics do not measure generalization to new patients or sessions. A participant-level split, or at least leave-one-participant-out evaluation, is required before any claim about detecting AA in unseen individuals can be made.","section":"§3.3"},{"comment":"Table 3 is titled 'Supervised Learning and Variational Autoencoder' and Section 4.2 says the model was trained on 18,804 normal and 1,475 AA labeled minutes, but the table's Normal/AA row reports counts (55,214/2,523, 55,648/2,092, 54,768/2,974) identical to the self-training counts in Table 4. Either the row is mislabeled or the 'supervised' experiment already includes pseudo-labeled data; as written, the comparison in Section 4.4 between 'fully supervised learning with VAE' and the proposed system is not interpretable.","section":"§4.2, Table 3"},{"comment":"The claim that self-training and VAE 'significantly improve' performance is not supported by any significance test, confidence interval, or repeated-split analysis; the single 70/30 split with one random seed is insufficient. Additionally, the abstract reports an accuracy of 90.16% while Section 4.4 and Section 5 report 90.18%, and the test-set size for the augmented datasets is never given.","section":"§4.4, Abstract"},{"comment":"The VAE's training procedure does not state whether the VAE was fit before or after the 70/30 split. If the encoder was trained on the entire dataset, test-window features are influenced by test data, which is a further form of leakage. The VAE should be trained on the training partition only, and the manuscript should specify this explicitly.","section":"§3.2.2"}],"minor_comments":[{"comment":"The abstract reports an accuracy of 90.16% while Section 4.4 and Section 5 report 90.18%; these numbers should be reconciled.","section":"Abstract vs. §4.4"},{"comment":"Equations (2) and (3) contain the citation marker '[44]' inside the displayed formulas; the citations should be moved to the surrounding text.","section":"Equations (2) and (3)"},{"comment":"The sentence 'as shown in 4.4' appears to refer to Equation (4), not to a numbered section; the cross-reference should be corrected.","section":"§3.2.2"},{"comment":"The meaning of the 'Normal/AA Labels' row is ambiguous: it should state clearly whether the counts refer to the training set, the full augmented dataset, or the test set.","section":"Tables 2–5"},{"comment":"The text says the authors 'choose 100 features from the 182 original features' but also that the VAE produces a 100-dimensional latent representation; the relationship between the selected features and the latent dimension should be clarified.","section":"§3.2.2"},{"comment":"The comparison in Table 6 is across different datasets, different participants, and different feature sets; the caption should more strongly warn the reader that the AUC values are not directly comparable.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the evaluation protocol: the headline numbers cannot be interpreted unless the authors can show that the test set contains only original clinical labels from held-out participants, with no pseudo-labeled windows and no VAE training on test data. This is fixable by re-running the experiments with a participant-level split and reporting uncertainty, but the current manuscript should not be accepted as is. If the authors cannot provide such a re-analysis, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the headline result, 90.18% balanced accuracy, is not supported by the evaluation as described. The test set appears to include the model's own pseudo-labels. That said, the dataset is genuinely new and clinically motivated, and the pipeline is a reasonable thing to try.\n\nWhat's new: they assembled a 14-participant, multi-hospital Empatica E4 dataset with only 5 participants fully labeled, and applied a standard recipe—VAE feature compression followed by self-training with XGBoost/Random Forest/Extra Trees. As an application paper it's fine; novelty is narrow, but they don't claim a new algorithm.\n\nWhat's good: they acknowledge the label-scarcity problem, use balanced accuracy, and compare supervised, VAE-only, self-training-only, and combined conditions. The writing is mostly clear.\n\nSoft spots, in order of severity:\n\n1. Test contamination. Section 3.3 describes a random 70/30 split but never says pseudo-labeled samples are excluded from the test set. The label counts in Tables 3–5 sum to near the full 57,742 minutes (e.g., Table 5: 54,712 + 3,026 = 57,738), which only makes sense if augmented data is in the test partition. If so, balanced accuracy measures agreement with the model's own pseudo-labels, not clinical truth. The text in 4.2 says the supervised VAE model uses only 18,804/1,475 labels, yet Table 3 lists 55,214/2,523—directly contradictory.\n\n2. Split is at window level, not participant level. Even with genuine labels, one-minute windows from the same participant appear in both train and test. That inflates performance and tells you nothing about generalization to new patients.\n\n3. No error bars, no code/data, and hyperparameters (latent dim 100, threshold 0.7, 100 iterations) are fixed without sensitivity analysis.\n\n4. Minor but telling: reference [32] is 'J. Doe et al.'—looks like a placeholder; and [20] is cited for 'Khan et al.' but is actually Spasojevic et al. Abstract says 90.16%, text says 90.18%.\n\nWho benefits: someone looking for a cautionary example of semi-supervised evaluation pitfalls, or a reader mining this dataset for future work. As evidence that VAE plus self-training works for AA detection, it shouldn't be cited.\n\nMy call: send it to peer review, but expect heavy revision. The dataset is real and the clinical problem matters; the flaws are fixable with a participant-level split and explicit exclusion of pseudo-labels from the test set. If the authors can't redo that analysis, reject.","headline":"The 90.18% balanced accuracy claim is undermined by a test set that appears to include the model's own pseudo-labels, though the new dataset and clinical angle are worth a look.","tokens_in":15293,"tokens_out":3442,"would_cite":false,"duration_ms":30472,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a variational autoencoder for feature compression plus a self-training loop for pseudo-labeling significantly improves agitation and aggression detection in people with dementia from wristband sensor data, with…","keywords":["dementia","agitation detection","wearable sensors","variational autoencoder","self-training","semi-supervised learning","XGBoost","physiological signals"],"falsifier":"Train the identical pipeline but evaluate with leave-one-participant-out on the five fully labeled participants, and audit the 30% test windows to confirm they carry nurse-confirmed clinical labels rather than model-generated pseudo-labels; also reconcile the abstract's 90.16% figure with the 90.18% reported in Table 5. If balanced accuracy drops materially below 90% under these conditions, the central claim would not survive.","tokens_in":14201,"feed_emoji":"🧠","tokens_out":6395,"duration_ms":56975,"temperature":0.7,"pith_summary":"This paper tries to show that a semi-supervised pipeline can detect agitation and aggression episodes in people with dementia from wristband physiology when only a fraction of the data has clinical labels. The proposed system first compresses 182 engineered features into a 100-dimensional representation with a variational autoencoder, then runs self-training to generate pseudo-labels for unlabeled minutes and retrains three classifiers. The authors report that this combination beats fully supervised baselines, with XGBoost reaching a balanced accuracy of 90.18% and an AUC-ROC of 99.6%. If the result holds, continuous monitoring from routinely collected wearable data becomes a practical route to early agitation alerts without requiring exhaustive manual labeling.","feed_headline":"VAE plus self-training lifts agitation detection to 90%","feed_subtitle":"Wristband models exploit unlabeled minutes and beat fully supervised baselines with a VAE plus pseudo-labeling.","key_machinery":"The load-bearing mechanism is a two-stage pipeline. First, a variational autoencoder, a probabilistic encoder-decoder trained to reconstruct input while regularizing a latent distribution, maps 182 hand-crafted features from heart rate, acceleration, electrodermal activity, and temperature into a 100-dimensional latent code per one-minute window. Second, a self-training loop trains a classifier on the labeled windows, predicts pseudo-labels for the 37,463 unlabeled windows, accepts those with predicted probability above a 0.7 threshold, and retrains, up to 100 iterations. The classifiers compared are Random Forest, Extra Trees, and XGBoost, with balanced accuracy as the headline metric because the agitation class is heavily outnumbered.","core_discovery":"The central claim is that combining variational autoencoder representation learning with self-training materially improves agitation and aggression classification compared with supervised learning on the labeled portion alone. On a dataset of 57,742 one-minute windows from 14 participants (5 fully labeled and 9 unlabeled), XGBoost trained on VAE-encoded features with self-training achieved a balanced accuracy of 90.18%, precision, recall, and F1-score of 99.0%, and an AUC-ROC of 99.6%, versus 85.03% balanced accuracy for the best fully supervised baseline. The paper interprets this as evidence that the VAE's compact latent representation, combined with confident pseudo-labels from self-training, lets models exploit unlabeled data rather than being limited by scarce clinical labels.","pith_inferences":["A reader should verify the composition of the 30% test set: Table 3 reports label counts, such as 55,214 normal and 2,523 agitation minutes, that sum to nearly the full 57,742-minute dataset even in the fully supervised condition, which suggests some rows may already include pseudo-labeled or unlabeled windows; if so, the reported balanced accuracy is not a clean measure of clinical generalization","The natural next experiment is leave-one-participant-out evaluation, because the random window split can place windows from the same participants in both training and test, inflating performance relative to unseen patients.","The abstract's 90.16% and Section 4.4's 90.18% differ slightly; reconciling the exact reported number and the model version behind it would be a useful check.","The same VAE-plus-self-training recipe is a plausible template for other wearable monitoring tasks with sparse labels, such as stress or seizure detection, but the transfer depends on the pseudo-label threshold and class-imbalance handling being re-tuned per task."],"forward_implications":["A semi-supervised pipeline can improve agitation detection over supervised baselines when labeled data are scarce.","The VAE dimension reduction from 182 to 100 features does not hurt and appears to help downstream classification.","The self-training label expansion roughly doubles the number of positive agitation windows in the training set before final classification.","XGBoost is the best-performing classifier in this setting, while Extra Trees is the fastest, indicating a practical speed-accuracy trade-off.","The system could support continuous monitoring in long-term care by alerting caregivers to probable agitation episodes."],"supporting_citations":[{"why":"This supplies the variational autoencoder formulation, including the reparameterization trick and KL-divergence objective, used for feature compression.","marker":"[17]"},{"why":"This is the authors' prior work establishing the preprocessing and feature-engineering pipeline on similar Empatica E4 agitation data.","marker":"[41]"},{"why":"This is the authors' prior pilot establishing the classification models and neuropsychiatric symptom detection setting that this paper extends.","marker":"[42]"},{"why":"This provides the Flirt toolkit used for cleaning, resampling, and extracting the raw features from the wristband signals.","marker":"[43]"},{"why":"This is the multimodal sensor pilot for agitation detection used as the main comparison point for AUC-ROC values.","marker":"[20]"},{"why":"This is the prior semi-supervised approach for identifying agitation likelihood that motivates the semi-supervised framing of the study.","marker":"[12]"},{"why":"This survey of semi-supervised learning grounds the self-training and pseudo-labeling procedure used in the proposed system.","marker":"[46]"}],"fun_headline_variants":["VAE + pseudo-labeling: 90% accuracy for dementia agitation","Semi-supervised VAE model detects agitation with 90% accuracy","Self-training and VAE turn unlabeled wristband data into agitation signals","Dementia agitation detection boosted by VAE self-training approach","VAE self-training method reaches 90% on agitation detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes that the 70/30 split of one-minute windows creates independent training and test sets whose test labels are clinically confirmed rather than pseudo-labels, and that windows from the same participant do not appear in both sets.","fun_headline_variants_meta":{"raw":{"variants":["VAE + pseudo-labeling: 90% accuracy for dementia agitation","Semi-supervised VAE model detects agitation with 90% accuracy","Self-training and VAE turn unlabeled wristband data into agitation signals","Dementia agitation detection boosted by VAE self-training approach","VAE self-training method reaches 90% on agitation detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2894,"prompt_tokens":1016,"completion_tokens":1878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1787}},"tokens_in":632,"tokens_out":1878,"duration_ms":14368,"temperature":1.0,"reasoning_tokens":1787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:47:17.889976+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical pipeline but evaluate with leave-one-participant-out on the five fully labeled participants, and audit the 30% test windows to confirm they carry nurse-confirmed clinical labels rather than model-generated pseudo-labels; also reconcile the abstract's 90.16% figure with the 90.18% reported in Table 5. If balanced accuracy drops materially below 90% under these conditions, the central claim would not survive.","supporting_citations":[{"cited_title":"Inves- tigating multimodal sensor features importance to detect agitation in people with dementia,","cited_arxiv_id":null,"evidence_quote":"This is the authors' prior work establishing the preprocessing and feature-engineering pipeline on similar Empatica E4 agitation data."},{"cited_title":"Artificial intelligence and features investigating to detect neuropsychiatric symptoms in patients with dementia: A pilot study,","cited_arxiv_id":null,"evidence_quote":"This is the authors' prior pilot establishing the classification models and neuropsychiatric symptom detection setting that this paper extends."},{"cited_title":"Flirt: A feature generation toolkit for wearable data,","cited_arxiv_id":null,"evidence_quote":"This provides the Flirt toolkit used for cleaning, resampling, and extracting the raw features from the wristband signals."},{"cited_title":"A pilot study to detect agitation in people living with dementia using multi-modal sensors,","cited_arxiv_id":null,"evidence_quote":"This is the multimodal sensor pilot for agitation detection used as the main comparison point for AUC-ROC values."},{"cited_title":"Semi-supervised learning for identifying the likelihood of agitation in people with dementia,","cited_arxiv_id":null,"evidence_quote":"This is the prior semi-supervised approach for identifying agitation likelihood that motivates the semi-supervised framing of the study."},{"cited_title":"A survey on semi-supervised learning,","cited_arxiv_id":null,"evidence_quote":"This survey of semi-supervised learning grounds the self-training and pseudo-labeling procedure used in the proposed system."}],"review_version":1}