{"id":"59b407b2-4082-44ce-b400-6c2e2b61fcff","arxiv_id":"2504.12236","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"First-week passive sensing and self-report data can predict end-of-term low academic performance with accuracy comparable to prior month-long models, but only within the same term; cross-year generalization is limited and fairness varies.","lead":"Using smartphone and Fitbit data from the first week of two Spring terms, the authors trained logistic regression, 1D-CNN, and multi-task CNN models to flag students whose end-of-term GPA fell below 3.2. The models match the accuracy of prior work that needed a month of data, but cross-year generalization is weak for the simpler models and fairness varies across protected groups.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Week-one prediction claim depends on same-term end-of-term labels; Table 5 shows LR and 1D-CNN at chance in the cross-year deployment protocol, so the real-world early-warning claim is unsupported as stated.","rationale":"","tokens_in":38571,"tokens_out":4544,"duration_ms":49821,"concrete_test":"Run the deployment protocol for LR and 1D-CNN: train on 2018 only, freeze the entire pipeline (imputation, SMOTE, correlation threshold r, feature set, architecture, and hyperparameters) using only 2018 training/validation data, then apply to 2019 first-week data and report balanced accuracy with bootstrap confidence intervals. Also rerun the within-term LR LOSO-CV with r selected inside each training fold only, never on the held-out test subject; if balanced accuracy drops materially, Table 2 is inflated by leakage. If cross-year balanced accuracy remains about 0.500 for LR and 0.559 for 1D-CNN, the abstract's week-one claim should be revised to describe retrospective same-term prediction, not deployment-ready early identification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that first-week passive sensing identifies at-risk students, with balanced accuracy 0.81–0.92 (Table 2). Those numbers come from within-term evaluation: LR and 1D-CNN are trained and tested on students from the same spring term (Section 5.1 and Figure 2a). In a real deployment, an early-warning system at week one of a new term cannot use that term's end-of-term grades for training; it must apply a model trained on prior terms. Table 5 tests this exact protocol for LR and 1D-CNN trained on 2018 and tested on 2019: balanced accuracy is 0.500 and 0.559, at or near the 0R majority-class baseline of 0.500. The abstract and Section 1 omit this failure when claiming 'identify at-risk students as early as week one' and 'provide predictions at least three weeks earlier.' The MTL variant reaches 0.712, but it is a different model, its secondary task trains on first-week data from both 2018 and 2019, and its primary task still requires 2018 end-of-term labels; it does not rescue the LR/1D-CNN claim. Section 5.4 itself acknowledges that LR and 1D-CNN do not outperform the baselines on unseen data, and that qualification should govern the abstract's wording. A secondary concern is that Appendix C.2.4 admits using the held-out test subject's data to select the correlation threshold r in each LOSO-CV fold; this could inflate the within-term numbers, so even the retrospective claim needs re-examination.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents three modeling approaches (logistic regression, 1D-CNN, and MTL-1D-CNN) to classify college students as low or high academic performers (GPA below/above 3.2) using passive sensing and self-report data collected within the first week of two Spring terms (2018 and 2019). The authors report high within-year balanced accuracy (0.81-0.92) and evaluate the models on explainability, fairness, and generalizability, concluding that trade-offs exist across these human-centered principles. The central claim is that at-risk students can be identified as early as week one, which is stated in the abstract and Section 1; however, the paper's own cross-year evaluation (Table 5) shows the LR and 1D-CNN models trained on 2018 and tested on 2019 perform at or near the majority-class baseline (balanced accuracy 0.500 and 0.559, respectively).","tokens_in":1525,"tokens_out":1954,"duration_ms":59335,"significance":"The work addresses an important gap in the learning analytics and HCI literature by combining passive behavioral sensing with explicit consideration of explainability, fairness, and generalizability, and by evaluating on real longitudinal data. The within-year results, if interpreted as a retrospective feasibility study, are valuable: they show that first-week behavioral features carry signal about end-of-term GPA. The paper also honestly reports the cross-year failure of the two simpler models and includes a multi-task approach that achieves more encouraging cross-year balanced accuracy (0.712). These strengths are, however, undermined by the mismatch between the abstract's 'week one' claim and the deployment-relevant protocol, as well as by acknowledged test-data leakage in the LR feature-selection loop. The empirical contribution is real, but the claims need re-scoping before the paper can be accepted.","major_comments":[{"comment":"The abstract's claim that 'these models can identify at-risk students as early as week one' is not supported by the deployment-relevant protocol. The balanced accuracies in Table 2 (LR 0.814/0.858, 1D-CNN 0.918/0.866) come from within-year training and testing, meaning the model uses end-of-term labels of the same term it is predicting. In a real-world week-one early-warning deployment, those labels do not exist at prediction time; the appropriate protocol is training on past terms and testing on a new term. Under that protocol (Table 5), LR achieves 0.500 and 1D-CNN achieves 0.559 balanced accuracy, essentially matching the majority-class baseline of 0.500. Section 5.4.1 explicitly concedes that neither approach outperforms the baseline on unseen data. The abstract and Section 1 should therefore re-scope the claim to distinguish retrospective within-year feasibility from cross-year deployment performance, or make the MTL model, which reaches 0.712, the focus of the early-prediction claim with its additional assumptions explicitly stated.","section":"Abstract and Section 1 vs. Section 5.4 and Table 5"},{"comment":"The LR feature-selection step selects the correlation threshold r by maximizing a_test - a_train on the held-out test subject within each LOSO-CV fold. This is a test-data leak in the model-selection loop and can inflate the 2018 within-year LR results reported in Table 2. The authors acknowledge the leakage but dismiss it as having 'no leakage' for the 2019 data. Since the within-year LR numbers are a primary support for the week-one claim, the paper must either re-run the 2018 analysis with r chosen only from training folds, or explicitly report the 2018 LR result as an optimistic upper bound and base the claim on the frozen-pipeline 2019 result. As written, the evidence for the headline claim is overstated.","section":"Appendix C.2.4"},{"comment":"The statement that the robust performance 'across the 2018 and 2019 datasets provides strong evidence that early predictions of student performance ... are possible' conflates pipeline-level generalization with model-level generalization. In Table 2, the models are re-fit in each year using that year's own end-of-term labels, so the across-year consistency does not demonstrate that a model pre-trained on one term can make week-one predictions in a new term. The paper introduces the pipeline-level versus model-level distinction only later in Section 5.4; Section 5.1.1 should either be qualified at the point of the claim or the generalizability section should be moved earlier so readers can correctly interpret the early-prediction results.","section":"Section 5.1.1"}],"minor_comments":[{"comment":"The word 'approachs' appears in the final paragraph of Section 5.3.2 and the word 'reffered' appears in Section 5.4; these typos should be corrected.","section":"Section 5"},{"comment":"The caption repeats '(a) 2018 spring term GPA' twice and is missing a description for one panel; this should be fixed for clarity.","section":"Figure 5 caption"},{"comment":"The 'Earliest predictable time' column lists 'before Spring term' for 1R-SVM, which uses prior-term GPA; since this baseline does not use first-week data, the column header should be clarified (e.g., 'earliest data time') to avoid implying a direct comparison on prediction timing.","section":"Table 2"},{"comment":"The fairness thresholds (difference between -0.1 and 0.1, ratio between 0.8 and 1.2) are taken from demographic-parity conventions and extended to equalized odds and equal opportunity without justification; the authors should add a sentence explaining why these thresholds are appropriate for the other two metrics.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical study but the central claim is not supported by the deployment-relevant experiment. The acknowledge leak in Appendix C.2.4 is also concerning. If the authors re-scope the abstract and conclusions to correctly separate within-year feasibility from cross-year deployment, and re-run the 2018 LR analysis without the leakage, the paper would be much stronger and potentially acceptable. The fit with a CS/HCI venue is reasonable given the focus on human-centered principles, but the novelty relative to prior work (e.g., SmartGPA) should be carefully positioned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the within-term first-week result is plausible, but the paper's headline that these models 'identify at-risk students as early as week one' is only true in a retrospective sense. In the only prospective test the authors run—train on 2018, test on 2019—LR and 1D-CNN are at chance (balanced accuracy 0.500 and 0.559), and the paper only owns this in Section 5.4, not in the abstract.\n\nWhat's genuinely new: using first-week passive sensing for GPA category, and systematically comparing three model families on explainability, fairness, and generalizability in one study. The MTL-1D-CNN design—secondary task predicting prior-term GPA from first-week data of both years to help the primary task—is a reasonable idea, and it does improve cross-year transfer (0.712 balanced accuracy). The paper also deserves credit for re-implementing the LSTM baseline from prior work and for discussing trade-offs honestly. The behavioral feature tables are rich, and the observation that students' routines shift as early as Thursday is a useful detail.\n\nSoft spots, in order. First, the key deployment claim is unsupported as stated. Same-term training means the model sees that term's end-of-term labels; a week-one early warning system in a new term cannot. The within-year CV numbers in Table 2 don't answer the question the abstract implies. Second, Appendix C.2.4 admits that the LR pipeline's LOSO-CV uses the held-out test subject's data to select the correlation threshold r. That is test leakage; it can inflate the within-term LR numbers. The authors call it 'some leakage' and say it wasn't applied to 2019, but the 2018 result needs re-analysis without touching the test fold. Third, no error bars or confidence intervals on Table 2; for a sample of 188/196, this matters.\n\nThese are not fatal to every claim. The within-term retrospective finding may survive a clean re-run, and the paper's own Section 5.4 is more careful than the abstract. But the abstract and contributions need rewriting to separate retrospective accuracy from prospective deployment. The data and the MTL experiment are worth a serious referee's time.\n\nRecommendation: send to peer review. It will need major revision before acceptance, but this is a real empirical study with a real weakness, not a desk reject.","headline":"Promising within-term first-week prediction, but the abstract's week-one early-warning claim only works with same-term labels; the honest cross-year test puts LR and 1D-CNN at chance.","tokens_in":39469,"tokens_out":2438,"would_cite":false,"duration_ms":26665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the first week of passively sensed phone and wearable data plus self-reports, the paper claims end-of-term GPA category can be predicted within the same term with 0.81–0.92 balanced accuracy, though cross-year prediction remains…","keywords":["academic performance prediction","early prediction","passive sensing","human-centered machine learning","explainability","fairness","generalizability","multi-task learning"],"falsifier":"Run a prospective deployment in a new Spring term: freeze the LR and 1D-CNN pipelines on data from previous terms before the term starts, collect first-week sensing, and compare week-one predictions against end-of-term GPA. The paper's Table 5 already supplies the predicted outcome for this test—balanced accuracy 0.500 and 0.559, equal to the majority-class baseline—so a prospective run would settle whether the within-term numbers transfer to real early warning.","tokens_in":38350,"feed_emoji":"🎓","tokens_out":8682,"duration_ms":79607,"temperature":0.7,"pith_summary":"This paper asks whether end-of-term academic risk can be detected from a single week of passively sensed phone and wearable data combined with self-reports, and whether the resulting models are explainable, fair, and generalizable enough for real use. Its central claim is that logistic regression and a one-dimensional convolutional network classify students as low or high performers from first-week data with balanced accuracy around 0.81 to 0.92, matching earlier studies that required four or more weeks of data and therefore shifting early prediction three weeks earlier. The paper reports that this holds when models are trained and tested within the same term; when trained on 2018 and applied to 2019, these two models drop to majority-class baseline level (balanced accuracy 0.500 and 0.559), and only a multi-task variant reaches 0.712. The paper's main finding is a trade-off map: the transparent logistic model is reasonably fair but does not transfer across years, the 1D-CNN is the most accurate within a term but opaque, and the multi-task model transfers best but shows the largest fairness gaps for protected groups. It also flags that about 21% of students participated in both years, a caveat for cross-year comparisons.","feed_headline":"Week-one sensing hits 92% balanced accuracy on term GPA","feed_subtitle":"Models trained on the first week match prior accuracy three weeks sooner, but only within the same term.","key_machinery":"The load-bearing object is the first-week feature set built from continuous phone and Fitbit streams: phone unlock counts, Bluetooth scans, location bouts, sleep, steps, physical activity, and self-reported stress/health scales, organized into five daily epochs (morning, afternoon, evening, night, full day) and augmented with behavioral-change slopes and breakpoints. Three pipelines sit on top: logistic regression with correlation-based feature selection and SMOTE oversampling; a 1D-CNN over daily time series; and a multi-task 1D-CNN whose secondary task predicts prior-term Winter GPA, letting it train on 2018 and 2019 data without using the test year's end-of-term labels. The features supply the week-one signal; the multi-task auxiliary task is what gives the only demonstrated cross-year transfer.","core_discovery":"On the paper's own terms, the discovery is that end-of-term GPA category can be predicted from the first week of a term: in Spring 2018 and Spring 2019, logistic regression and a 1D-CNN achieve balanced accuracy between 0.814 and 0.918 in classifying students as low or high performers, comparable to the prior four-week benchmark and three weeks earlier. The paper argues this demonstrates that at-risk students can be identified at week one. It is equally explicit, in Section 5.4, that the same two approaches do not carry that accuracy across terms: trained on 2018 and tested on 2019, they perform at the level of a majority-class baseline, while the multi-task 1D-CNN, which adds prior-term Winter GPA prediction as a secondary task, reaches 0.712 balanced accuracy. Across explainability, fairness, and generalizability, no single approach dominates: LR is transparent and fair in several comparisons but weak cross-year, 1D-CNN is strongest within a term but a black box, and MTL-1D-CNN generalizes best while showing the largest fairness gaps. The paper also notes the presence of returning students (about 21.3% year-to-year retention) as a reliability caveat.","pith_inferences":["The paper's week-one claim is a same-term result; in a realistic deployment where the model must be trained before the term's grades exist, the Table 5 cross-year numbers (0.500 and 0.559 balanced accuracy for LR and 1D-CNN) are the more relevant estimate.","Prior-term GPA alone (the 1R-SVM baseline) performs at 0.672 balanced accuracy cross-year and perfectly identifies students who stay low performers, so the marginal value of week-one sensing over transcripts deserves a direct held-out comparison.","The Thursday/Friday behavioral breakpoint is testable as an intervention design: randomize students to a Friday-focused support message versus a Monday-focused one and compare end-of-term GPA.","The association between phone service provider and GPA, which the paper reads as a proxy for unmeasured socioeconomic context, could be resolved by collecting direct income or family-background measures and checking whether the association disappears."],"forward_implications":["If the within-term results hold, student support teams could begin outreach by the end of the first week of a term, roughly three weeks earlier than the previous state of the art, widening the window for intervention.","The logistic-regression feature rankings suggest concrete early warning signs—class attendance, timing of phone use, restless sleep, and self-reported stressors—that advisors could act on even without the model.","The Friday breakpoint seen in both years implies that many students' weekend behavior starts on Friday, so end-of-week check-ins may be a better-timed intervention point than midweek messages.","Because LR and 1D-CNN fail on the cross-year test, a deployed early-warning system cannot reuse last year's model; the MTL variant is the paper's only candidate for that role, and it still requires further validation.","Fairness varies by protected group and metric across all approaches, so an institution adopting any of these models should audit it on its own student population before acting on its outputs."],"supporting_citations":[{"why":"Provides the LSTM baseline and the only earlier model-level generalizability result (AUC 0.75 dropping to 0.63 on unseen data); used as a comparison baseline for early prediction.","marker":"[32]"},{"why":"Prior study using four weeks of sensing data that reported about 92% accuracy; the paper claims its own week-one predictions are comparable but three weeks earlier.","marker":"[133]"},{"why":"SmartGPA study supplying high-level behavioral features and evidence that passive sensing plus self-reports can predict GPA; also a comparison point.","marker":"[153]"},{"why":"Defines the feature extraction framework used to compute low-level behavioral features from phone and Fitbit data.","marker":"[49]"},{"why":"Fairness evaluation toolkit used to compute demographic parity, equalized odds, and equal opportunity differences and ratios.","marker":"[18]"},{"why":"Introduces the pipeline-level versus model-level generalizability distinction that structures the evaluation.","marker":"[158]"},{"why":"Longitudinal study that is a source of the behavioral, EMA, and self-report data used for modeling.","marker":"[134]"},{"why":"GLOBEM multi-year dataset publication covering the same longitudinal data collection, used as the data source for both years.","marker":"[160]"}],"fun_headline_variants":["Week-one prediction hits 92% but only for same term","Multi-task CNN generalizes across terms, but fairness gaps","Week-one data alone predicts GPA, but trade-offs exist","Week-one accuracy: high within term, mixed across years","Week-one prediction: same-term win, cross-term drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model trained and tested on students from the same term can stand in for real early-warning use; in deployment, the model would have to be trained before the term's grades exist, and the paper's own cross-year test shows two of its three approaches then perform no better than predicting everyone as a high performer.","fun_headline_variants_meta":{"raw":{"variants":["Week-one prediction hits 92% but only for same term","Multi-task CNN generalizes across terms, but fairness gaps","Week-one data alone predicts GPA, but trade-offs exist","Week-one accuracy: high within term, mixed across years","Week-one prediction: same-term win, cross-term drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001716,"raw_usage":{"total_tokens":6853,"prompt_tokens":1071,"completion_tokens":5782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":5700}},"tokens_in":687,"tokens_out":5782,"duration_ms":46437,"temperature":1.0,"reasoning_tokens":5700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:20.137966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a prospective deployment in a new Spring term: freeze the LR and 1D-CNN pipelines on data from previous terms before the term starts, collect first-week sensing, and compare week-one predictions against end-of-term GPA. The paper's Table 5 already supplies the predicted outcome for this test—balanced accuracy 0.500 and 0.559, equal to the majority-class baseline—so a prospective run would settle whether the within-term numbers transfer to real early warning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior study using four weeks of sensing data that reported about 92% accuracy; the paper claims its own week-one predictions are comparable but three weeks earlier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SmartGPA study supplying high-level behavioral features and evidence that passive sensing plus self-reports can predict GPA; also a comparison point."},{"cited_title":"Extraction of Behavioral Features from Smartphone and Wearable Data","cited_arxiv_id":"1812.10394","evidence_quote":"Defines the feature extraction framework used to compute low-level behavioral features from phone and Fitbit data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Longitudinal study that is a source of the behavioral, EMA, and self-report data used for modeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GLOBEM multi-year dataset publication covering the same longitudinal data collection, used as the data source for both years."}],"review_version":1}