{"id":"2c192cc3-fa88-4b6e-b3f7-cc444b3dbe13","arxiv_id":"2508.12013","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Using survey answers from 388 students, XGBoost predicts ChatGPT assignment use with 80% accuracy, and using ChatGPT for new learning is the top predictor.","lead":"This study used survey answers from 388 university students to train a machine learning model that predicts whether they use ChatGPT for assignments, reaching about 80% accuracy. It finds that the strongest signal is already using ChatGPT when learning new topics, which could help universities rethink how they test and assign work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The top predictor may be a near-duplicate of the target, making the headline behavioral finding largely tautological; the 80.1% accuracy could reflect self-report consistency rather than distinct behavioral predictors.","rationale":"I read the paper as a descriptive predictive modeling study with modest, internally plausible performance metrics: train/test agreement is close, and a logistic regression baseline nearly matches XGBoost, which argues against gross overfitting. The weakest point is not the modeling mechanics but the interpretation. The authors' own variable descriptions show that the top predictor and the target are both self-reported ChatGPT usage frequencies measured on the same scale; the SHAP finding may therefore reduce to 'students who say they use ChatGPT also say they use ChatGPT.' This is a construct-validity problem rather than a statistical artifact, and it is more load-bearing than the secondary issue of IV-based feature selection outside cross-validation, because even a perfectly executed pipeline would produce the same misleading headline if the predictor is a near-duplicate of the label. The reader's weakest_assumption identifies exactly this concern. I would keep the conditional recommendation: the paper's central claim can be accepted only if the authors release the data and demonstrate that the result survives removal of tautological predictors. The limitation statements in the paper are honest about self-report and cross-sectional design, but they do not acknowledge this specific threat.","tokens_in":8157,"tokens_out":4207,"duration_ms":47316,"concrete_test":"Obtain the dataset and code from the authors (or run an independent replication using the linked survey; the manuscript states data/code are available on request). First compute Cramér's V between ChatGPT_Used_in_New_Learning and binarized ChatGPT_Assignment. Then retrain the binary XGBoost pipeline exactly as described, but with an ablated feature set that removes ChatGPT_Used_in_New_Learning and the other ChatGPT-frequency/attitude items with nearly identical wording (ChatGPT_Helps_New_Learning, ChatGPT_Explicit_learning, ChatGPT_4_Non_Performer, ChatGPT_OR_SE), leaving demographics, subject preferences, involvement, and social-media variables. If test accuracy falls to the ~53.6% majority-class baseline and the SHAP ranking is reshuffled, the reported 80.1% is driven by the tautological predictor; if accuracy remains above ~75%, the behavioral claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central behavioral claim is that ChatGPT_Used_in_New_Learning (mean |SHAP| = 0.500, gain 20.30) is a meaningful predictor of ChatGPT_Assignment. But in Section 3 the two variables are defined almost identically: ChatGPT_Assignment is 'Frequency of ChatGPT use for assignments' and ChatGPT_Used_in_New_Learning is 'ChatGPT usage frequency for new topics'. Both are self-reported frequency items measured on the same five-point scale in the same survey. A student who reports using ChatGPT 'Frequently' for learning new topics is overwhelmingly likely to report using it 'Frequently' for assignments; this is response consistency, not a discovered behavioral mechanism. If this is tautological, then the headline 80.1% accuracy is real but uninformative, and the abstract's inference about 'overreliance' has no independent support. The concern is not that the model is overfit (train/test scores are close), but that the model's best feature carries the label's own variance. The policy conclusions in Section 5—differentiating responsible use from misuse—therefore rest on a construct-validity assumption the paper never tests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes survey responses from 388 university students (predominantly in Russia) to predict self-reported ChatGPT use in academic assignments. Using XGBoost with SHAP interpretation, the authors report a binary classifier with 80.1% test accuracy (80.2% sensitivity, 79.9% specificity) and a multiclass classifier with 64.5% test accuracy, and identify ChatGPT_Used_in_New_Learning as the dominant predictor (mean |SHAP| = 0.500, gain 20.30). The paper frames these results as evidence of a behavioral continuum of ChatGPT adoption and draws implications for AI-aware assessment design, including a distinction between responsible use and overreliance.","tokens_in":8393,"tokens_out":3610,"duration_ms":40070,"significance":"If the reported predictive performance and feature interpretations were valid, the study would provide a useful quantitative complement to qualitative work on student AI use, and the dual-task (binary plus multiclass) formulation is a reasonable design choice for exploratory analysis. The paper also demonstrates appropriate use of interpretability tools (SHAP, gain-based importance) and clearly reports class distributions. However, the central finding is currently undermined by three load-bearing problems: the top predictor is nearly the same self-reported construct as the target; feature selection by information value appears to be performed before the train-test split, which can inflate the reported accuracy; and the test set contains only about 97 samples, yet no confidence intervals are given. In addition, the language repeatedly overstates the evidence as 'causal' when the data are cross-sectional self-reports. These issues must be resolved before the behavioral and policy conclusions can be accepted.","major_comments":[{"comment":"The stress-test concern about construct overlap is confirmed by the manuscript's own variable descriptions. ChatGPT_Assignment is defined as 'Frequency of ChatGPT use for assignments' and ChatGPT_Used_in_New_Learning as 'ChatGPT usage frequency for new topics'; both are five-point self-reported frequency items from the same survey. A student who reports frequent use for learning new topics will almost inevitably report frequent use for assignments, so the top-ranked predictor (mean |SHAP| = 0.500) largely carries the target's own variance. The abstract's statement that 'frequent use of ChatGPT for learning new concepts correlates with potential overreliance' is therefore not an independent behavioral discovery: no measure of overreliance (e.g., reduced independent effort or learning outcomes) is included, and the correlation is between two items of the same self-report scale. To support the paper's central claim, the authors must either exclude this near-duplicate predictor and rerun the analysis, or demonstrate discriminant validity (e.g., show that the two items are empirically distinct beyond shared method variance). As written, the headline accuracy and the policy implications in Section 5 rest on an untested construct-validity assumption.","section":"Section 3, variable definitions; Section 5 SHAP analysis"},{"comment":"Feature selection using Information Value (IV) appears to be applied to the full dataset before the train-test split: the EDA subsection states that variables with IV < 0.1 were excluded from modeling, and the model training subsection then describes partitioning the data into 75/25. This is a leakage pathway because the excluded variables are chosen using information from both training and test partitions, which can inflate test accuracy and distort feature importance. The authors should either move the IV-based selection inside the cross-validation loop (or use nested CV), or report the performance with all features retained. This issue is load-bearing because the reported 80.1% accuracy and the SHAP ranking may both be affected.","section":"Section 4, EDA and Model Training"},{"comment":"The binary test set contains approximately 97 samples (25% of 388), so the reported test accuracy of 80.1% has a standard error of roughly 4 percentage points and a wide binomial confidence interval (roughly 71% to 88%). The paper reports no confidence intervals or significance tests for any metric. The claim of 'strong predictive performance' should be qualified with uncertainty estimates, and the comparison with logistic regression in Table 2 should also include such intervals. Without this, the reader cannot assess whether XGBoost's small accuracy advantage (80.1% vs. 79.0%) is meaningful.","section":"Section 5, Table 1(a)"},{"comment":"The manuscript repeatedly uses causal language: Section 4 calls the model a 'causal model prototype,' and Section 6 says future work should 'uncover causal and contextual factors.' The data are cross-sectional self-reports, so even a well-specified predictive model cannot support causal claims about ChatGPT use causing overreliance or eroding critical thinking. The phrase 'potential overreliance' in the abstract is also not operationalized or measured anywhere in the study. The authors should either reframe all conclusions as descriptive correlations or add a clearly identified causal analysis with appropriate identification assumptions.","section":"Section 4 and Section 6"},{"comment":"There is an internal inconsistency in the reported multiclass results: the text says the multiclass model reached 'an overall accuracy of approximately 68%, with a weighted F1-score of 0.68,' but Table 1(b) reports test accuracy 0.6450 and weighted F1 0.6450. The authors should clarify whether the 68% figure refers to training, validation, or a different hyperparameter setting, and reconcile the discrepancy. The claim about 'similar training scores, indicating potential data scarcity' is also not substantiated; if training and test scores are close, that suggests underfitting or leakage, not data scarcity, so the interpretation needs revision.","section":"Section 5, Model Performance (multiclass)"}],"minor_comments":[{"comment":"The footnote promises that 'the full dataset, code and the complete set of survey questions' may be provided upon reasonable request, but there is no repository or access mechanism; the Google Drive link in Section 3 is not a stable DOI and may not be available after review. The authors should provide an anonymized repository or supplementary material for reproducibility.","section":"Footnote 1 and Section 3"},{"comment":"Reference [5] contains 'doi: 10.1109/TE.2024.XXXXXXX', which is a placeholder and not a valid DOI. Also, the manuscript does not include a data availability statement beyond the footnote.","section":"References"},{"comment":"The text says values below 2 are labeled as 'rear or non-users'; 'rear' is a typo for 'rare.' Also, 'inconstant responses' should be 'inconsistent responses.'","section":"Section 4, Data Preprocessing"},{"comment":"Figures 4, 5, 6, and 7 are referenced in the text but not all are discussed in sufficient detail; for example, Figure 5's right panel is not explicitly described in the multiclass paragraphs. The captions could more clearly state what each subplot shows (e.g., class labels).","section":"Section 5, Figures"},{"comment":"The text reports hyperparameters for the binary model (learning rate 0.153, max depth 2, 10 estimators, subsample 0.8) and later mentions different settings for the multiclass model (reg_lambda = 3). Clarify which hyperparameter set applies to each model and whether the reported CV performance used the same final parameters.","section":"Section 4, Hyperparameters"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely topic, but the current version has load-bearing validity issues. The construct-overlap concern raised by the stress-test note is real and is visible in the manuscript's own variable definitions. The pre-split IV feature selection and the absence of confidence intervals compound it. These issues are fixable in principle by reanalysis (excluding the near-duplicate predictor, moving selection inside CV, and reporting intervals), but the revision needs to be substantive rather than cosmetic. If the authors can show that the predictive performance persists without ChatGPT_Used_in_New_Learning, or thoroughly rework the conclusions to avoid overclaims, the paper could become publishable. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the dataset and the dual binary/multiclass formulation, not the method. XGBoost plus SHAP on survey data is standard, but a 388-student Russian-university sample with a five-level target split two ways is at least a fresh empirical datapoint. The paper also does a few things well: it compares against logistic regression, reports cross-validated scores, and includes a reasonable EDA. The writing is clear and the limitations section is honest.\n\nThe soft spots, in rough order of importance. First, the top predictor, ChatGPT_Used_in_New_Learning, is nearly the same construct as the target ChatGPT_Assignment. Both are self-reported frequency-of-use items on the same scale in the same survey. The stress-test note is right: the headline finding is more response consistency than behavioral discovery. That doesn't sink the paper, but it does shrink the interpretive payoff. Second, feature selection via information value appears to be done before the train–test split, which risks optimistic generalization. That is fixable but should have been handled inside the cross-validation loop. Third, the abstract's 'overreliance' claim is not measured; the survey does not capture academic independence. Fourth, there is a numeric inconsistency in the multiclass results: the text says ~68% accuracy, while Table 1b reports 64.5% test accuracy. Minor but sloppy. Finally, data and code are only 'available on request'—for a paper trying to establish a predictive benchmark, that is a real limitation.\n\nLogistic regression gets 79% versus XGBoost's 80%, so the nonlinear model adds little; the authors acknowledge this, which is fair. Demographics show near-zero SHAP values, which is a decent null result within the sample. But the sample is narrow and self-selected, so generalizability is thin.\n\nWho gets value? Researchers working on AI-in-education surveys will find a clean template and a cautionary tale about predictor–target overlap. It is not a breakthrough, but it is not empty either. I'd send it to a serious referee, but conditional on the authors releasing data and code, moving feature selection into CV, and softening the causal and 'overreliance' language. If those revisions happen, the paper could be a useful descriptive study.","headline":"A modest, readable survey-modeling study whose headline predictor is too close to the target to support the 'overreliance' conclusion; still merits a serious referee if the authors fix the leakage and framing.","tokens_in":8884,"tokens_out":2523,"would_cite":false,"duration_ms":26030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Students who use ChatGPT to learn new topics are substantially more likely to use it for assignments, and this behavior can be predicted at 80% accuracy.","keywords":["Generative AI","Higher education","Learning Analytics","Machine learning","Educational technology","ChatGPT","XGBoost","SHAP"],"falsifier":"Re-run the analysis with ChatGPT_Used_in_New_Learning removed, or with a latent-variable control for general ChatGPT reliance, and check whether the remaining predictors, especially ChatGPT_4_Non_Performer and ChatGPT_Important_4_Good_Student, still show substantial SHAP values and whether test accuracy drops sharply; if the model holds, the behavioral 'continuum' interpretation is supported.","tokens_in":7958,"feed_emoji":"🎓","tokens_out":5269,"duration_ms":51413,"temperature":0.7,"pith_summary":"This paper builds a machine-learning model on self-reported survey responses from 388 university students and claims that ChatGPT use in academic assignments can be predicted at 80.1% test accuracy from learning habits, subject attitudes, and AI attitudes. Its central behavioral finding is that the strongest predictor by far is whether a student already uses ChatGPT to learn new topics, with a mean SHAP value of 0.500. If true, this matters for assessment design because it reframes assignment-time AI use as an extension of everyday AI-aided learning rather than an isolated cheating behavior, suggesting that policies should target the whole learning habit, not just the assignment moment. The authors use this to propose discipline-specific guidelines and reimagined assessment strategies.","feed_headline":"80% accuracy: ChatGPT learning use predicts assignment use","feed_subtitle":"Survey of 388 students shows the strongest predictor is how often ChatGPT is used to learn new topics.","key_machinery":"The load-bearing object is an XGBoost gradient-boosted decision tree classifier trained on a stratified 75/25 split of the survey data, paired with SHAP values from TreeExplainer for global and class-specific interpretation. The SHAP attribution is what turns the model's prediction into the paper's behavioral conclusion: it ranks ChatGPT_Used_in_New_Learning as the dominant predictor (mean |SHAP| = 0.500) and separates non-users from habitual users along that variable, which is the basis for the proposed assessment implications.","core_discovery":"On its own terms, the paper establishes that a binary XGBoost classifier can separate students who report using ChatGPT for assignments at least 'sometimes' from those who do not, achieving about 80% accuracy, sensitivity, and specificity. The interpretable feature-attribution analysis shows that the single strongest driver of the classification is ChatGPT_Used_in_New_Learning, a survey item measuring how often ChatGPT is used to learn new topics, followed by use for uninteresting or underperforming subjects and endorsement of ChatGPT for competent students. The authors read this as evidence that assignment-level use is part of a behavioral continuum: students who habitually turn to ChatGPT as a learning tool carry that behavior into assignment completion, with compensatory and productivity motives layered on top. Demographic variables such as sex, institution, and country contribute almost nothing, arguing that the behavior is driven by learning strategies and attitudes rather than identity.","pith_inferences":["The top predictor's dominance may be partly a measurement artifact: if ChatGPT_Used_in_New_Learning and ChatGPT_Assignment tap the same self-reported propensity, the headline finding would overstate how informative the other behavioral predictors are; a test that removes the top predictor would reveal whether compensatory and productivity signals carry independent weight.","Because the survey is cross-sectional and self-reported, the model predicts association, not causation; a panel study with actual usage logs could determine whether new-learning use precedes and drives assignment use or simply accompanies it.","The reported 80% accuracy could be inflated by shared survey context and method variance; comparing self-report predictions against logged ChatGPT usage in an institution-approved setting would give a cleaner estimate of true predictive power."],"forward_implications":["If the central claim holds, AI-aware assessment policies should address the learning-habit continuum rather than single-assignment cheating, for example by integrating AI literacy and explicit usage expectations into courses.","The strong predictive role of attitudinal items such as support for AI integration and perceived helpfulness suggests that shaping student attitudes toward generative AI may directly shift assignment-level usage.","Near-zero importance of demographics implies that universal policies are likely to be as effective as policies targeted at specific student subgroups.","The multiclass model's lower accuracy (about 65% versus 80%) indicates that fine-grained frequency prediction is harder, so practical policy distinctions should be coarse, such as users versus non-users, until data quality improves.","The authors' stated limitations caution that the model may not transfer to other populations or to longitudinal settings, so any policy use should be accompanied by local validation."],"supporting_citations":[{"why":"Supplies the 100-million-users-in-two-months adoption statistic that frames ChatGPT as an unprecedented force in education.","marker":"[10]"},{"why":"Provides the perceived-usefulness construct the paper uses to interpret why attitudinal items like ChatGPT_Helps_New_Learning predict usage.","marker":"[2]"},{"why":"Extends the technology-acceptance account, cited when the paper aligns its SHAP findings with perceived usefulness and ease of integration.","marker":"[3]"},{"why":"Rogers's diffusion theory is invoked to explain non-users' low experiential engagement as unfamiliarity impeding adoption.","marker":"[1]"},{"why":"Provides the educator-perspective interviews that motivate the concern about productivity and critical thinking that the study quantifies.","marker":"[15]"}],"fun_headline_variants":["ChatGPT learning use predicts assignment use (80% acc)","AI study: How students learn predicts ChatGPT on homework","Using ChatGPT to learn? It signals assignment use too","Key predictor of ChatGPT assignment use: learning habits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if the survey item about using ChatGPT for learning new topics and the item about using it for assignments are really measuring the same underlying self-reported habit, because then the strongest predictor simply restates the thing being predicted.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT learning use predicts assignment use (80% acc)","AI study: How students learn predicts ChatGPT on homework","Using ChatGPT to learn? It signals assignment use too","Key predictor of ChatGPT assignment use: learning habits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1473,"prompt_tokens":978,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":594,"tokens_out":495,"duration_ms":6065,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:25:50.675252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the analysis with ChatGPT_Used_in_New_Learning removed, or with a latent-variable control for general ChatGPT reliance, and check whether the remaining predictors, especially ChatGPT_4_Non_Performer and ChatGPT_Important_4_Good_Student, still show substantial SHAP values and whether test accuracy drops sharply; if the model holds, the behavioral 'continuum' interpretation is supported.","supporting_citations":[{"cited_title":"Particle Physics in High School Part II: Nuclear Physics","cited_arxiv_id":"2303.10296","evidence_quote":"Supplies the 100-million-users-in-two-months adoption statistic that frames ChatGPT as an unprecedented force in education."},{"cited_title":"Perceived usefulness, perceived ease of use, and user accep- tance of information technology,","cited_arxiv_id":null,"evidence_quote":"Provides the perceived-usefulness construct the paper uses to interpret why attitudinal items like ChatGPT_Helps_New_Learning predict usage."},{"cited_title":"Consumer acceptance and use of information technology,","cited_arxiv_id":null,"evidence_quote":"Extends the technology-acceptance account, cited when the paper aligns its SHAP findings with perceived usefulness and ease of integration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rogers's diffusion theory is invoked to explain non-users' low experiential engagement as unfamiliarity impeding adoption."}],"review_version":2}