REVIEW 5 major objections 5 minor 15 references
Predicting ChatGPT Use in Assignments: Implications for AI-Aware Assessment Design
T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Students who use ChatGPT to learn new topics are substantially more likely to use it for assignments, and this behavior can be predicted at 80% accuracy.
desk verdict A modest, readable survey-modeling study whose headline predictor is too close to the target to support the 'overreliance' conclusion; still merits a serious referee if the authors fix the leakage and framing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is an XGBoost gradient-boosted decision tree classifier trained on a stratified 75/25 split of the survey data, paired with SHAP values from TreeExplainer for global and class-specific interpretation. The SHAP attribution is what turns the model's prediction into the paper's behavioral conclusion: it ranks ChatGPT_Used_in_New_Learning as the dominant predictor (mean |SHAP| = 0.500) and separates non-users from habitual users along that variable, which is the basis for the proposed assessment implications.
What would settle it
Re-run the analysis with ChatGPT_Used_in_New_Learning removed, or with a latent-variable control for general ChatGPT reliance, and check whether the remaining predictors, especially ChatGPT_4_Non_Performer and ChatGPT_Important_4_Good_Student, still show substantial SHAP values and whether test accuracy drops sharply; if the model holds, the behavioral 'continuum' interpretation is supported.
Extended reading notes
Core claim
On its own terms, the paper establishes that a binary XGBoost classifier can separate students who report using ChatGPT for assignments at least 'sometimes' from those who do not, achieving about 80% accuracy, sensitivity, and specificity. The interpretable feature-attribution analysis shows that the single strongest driver of the classification is ChatGPT_Used_in_New_Learning, a survey item measuring how often ChatGPT is used to learn new topics, followed by use for uninteresting or underperforming subjects and endorsement of ChatGPT for competent students. The authors read this as evidence that assignment-level use is part of a behavioral continuum: students who habitually turn to ChatGPT as a learning tool carry that behavior into assignment completion, with compensatory and productivity motives layered on top. Demographic variables such as sex, institution, and country contribute almost nothing, arguing that the behavior is driven by learning strategies and attitudes rather than identity.
Load-bearing premise
The claim collapses if the survey item about using ChatGPT for learning new topics and the item about using it for assignments are really measuring the same underlying self-reported habit, because then the strongest predictor simply restates the thing being predicted.
Editorial extensions
If this is right
- If the central claim holds, AI-aware assessment policies should address the learning-habit continuum rather than single-assignment cheating, for example by integrating AI literacy and explicit usage expectations into courses.
- The strong predictive role of attitudinal items such as support for AI integration and perceived helpfulness suggests that shaping student attitudes toward generative AI may directly shift assignment-level usage.
- Near-zero importance of demographics implies that universal policies are likely to be as effective as policies targeted at specific student subgroups.
- The multiclass model's lower accuracy (about 65% versus 80%) indicates that fine-grained frequency prediction is harder, so practical policy distinctions should be coarse, such as users versus non-users, until data quality improves.
- The authors' stated limitations caution that the model may not transfer to other populations or to longitudinal settings, so any policy use should be accompanied by local validation.
Reading between the lines
- The top predictor's dominance may be partly a measurement artifact: if ChatGPT_Used_in_New_Learning and ChatGPT_Assignment tap the same self-reported propensity, the headline finding would overstate how informative the other behavioral predictors are; a test that removes the top predictor would reveal whether compensatory and productivity signals carry independent weight.
- Because the survey is cross-sectional and self-reported, the model predicts association, not causation; a panel study with actual usage logs could determine whether new-learning use precedes and drives assignment use or simply accompanies it.
- The reported 80% accuracy could be inflated by shared survey context and method variance; comparing self-report predictions against logged ChatGPT usage in an institution-approved setting would give a cleaner estimate of true predictive power.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes survey responses from 388 university students (predominantly in Russia) to predict self-reported ChatGPT use in academic assignments. Using XGBoost with SHAP interpretation, the authors report a binary classifier with 80.1% test accuracy (80.2% sensitivity, 79.9% specificity) and a multiclass classifier with 64.5% test accuracy, and identify ChatGPT_Used_in_New_Learning as the dominant predictor (mean |SHAP| = 0.500, gain 20.30). The paper frames these results as evidence of a behavioral continuum of ChatGPT adoption and draws implications for AI-aware assessment design, including a distinction between responsible use and overreliance.
Significance. If the reported predictive performance and feature interpretations were valid, the study would provide a useful quantitative complement to qualitative work on student AI use, and the dual-task (binary plus multiclass) formulation is a reasonable design choice for exploratory analysis. The paper also demonstrates appropriate use of interpretability tools (SHAP, gain-based importance) and clearly reports class distributions. However, the central finding is currently undermined by three load-bearing problems: the top predictor is nearly the same self-reported construct as the target; feature selection by information value appears to be performed before the train-test split, which can inflate the reported accuracy; and the test set contains only about 97 samples, yet no confidence intervals are given. In addition, the language repeatedly overstates the evidence as 'causal' when the data are cross-sectional self-reports. These issues must be resolved before the behavioral and policy conclusions can be accepted.
major comments (5)
- [Section 3, variable definitions; Section 5 SHAP analysis] The stress-test concern about construct overlap is confirmed by the manuscript's own variable descriptions. ChatGPT_Assignment is defined as 'Frequency of ChatGPT use for assignments' and ChatGPT_Used_in_New_Learning as 'ChatGPT usage frequency for new topics'; both are five-point self-reported frequency items from the same survey. A student who reports frequent use for learning new topics will almost inevitably report frequent use for assignments, so the top-ranked predictor (mean |SHAP| = 0.500) largely carries the target's own variance. The abstract's statement that 'frequent use of ChatGPT for learning new concepts correlates with potential overreliance' is therefore not an independent behavioral discovery: no measure of overreliance (e.g., reduced independent effort or learning outcomes) is included, and the correlation is between two items of the same self-report scale. To support the paper's central claim, the authors must either exclude this near-duplicate predictor and rerun the analysis, or demonstrate discriminant validity (e.g., show that the two items are empirically distinct beyond shared method variance). As written, the headline accuracy and the policy implications in Section 5 rest on an untested construct-validity assumption.
- [Section 4, EDA and Model Training] Feature selection using Information Value (IV) appears to be applied to the full dataset before the train-test split: the EDA subsection states that variables with IV < 0.1 were excluded from modeling, and the model training subsection then describes partitioning the data into 75/25. This is a leakage pathway because the excluded variables are chosen using information from both training and test partitions, which can inflate test accuracy and distort feature importance. The authors should either move the IV-based selection inside the cross-validation loop (or use nested CV), or report the performance with all features retained. This issue is load-bearing because the reported 80.1% accuracy and the SHAP ranking may both be affected.
- [Section 5, Table 1(a)] The binary test set contains approximately 97 samples (25% of 388), so the reported test accuracy of 80.1% has a standard error of roughly 4 percentage points and a wide binomial confidence interval (roughly 71% to 88%). The paper reports no confidence intervals or significance tests for any metric. The claim of 'strong predictive performance' should be qualified with uncertainty estimates, and the comparison with logistic regression in Table 2 should also include such intervals. Without this, the reader cannot assess whether XGBoost's small accuracy advantage (80.1% vs. 79.0%) is meaningful.
- [Section 4 and Section 6] The manuscript repeatedly uses causal language: Section 4 calls the model a 'causal model prototype,' and Section 6 says future work should 'uncover causal and contextual factors.' The data are cross-sectional self-reports, so even a well-specified predictive model cannot support causal claims about ChatGPT use causing overreliance or eroding critical thinking. The phrase 'potential overreliance' in the abstract is also not operationalized or measured anywhere in the study. The authors should either reframe all conclusions as descriptive correlations or add a clearly identified causal analysis with appropriate identification assumptions.
- [Section 5, Model Performance (multiclass)] There is an internal inconsistency in the reported multiclass results: the text says the multiclass model reached 'an overall accuracy of approximately 68%, with a weighted F1-score of 0.68,' but Table 1(b) reports test accuracy 0.6450 and weighted F1 0.6450. The authors should clarify whether the 68% figure refers to training, validation, or a different hyperparameter setting, and reconcile the discrepancy. The claim about 'similar training scores, indicating potential data scarcity' is also not substantiated; if training and test scores are close, that suggests underfitting or leakage, not data scarcity, so the interpretation needs revision.
minor comments (5)
- [Footnote 1 and Section 3] The footnote promises that 'the full dataset, code and the complete set of survey questions' may be provided upon reasonable request, but there is no repository or access mechanism; the Google Drive link in Section 3 is not a stable DOI and may not be available after review. The authors should provide an anonymized repository or supplementary material for reproducibility.
- [References] Reference [5] contains 'doi: 10.1109/TE.2024.XXXXXXX', which is a placeholder and not a valid DOI. Also, the manuscript does not include a data availability statement beyond the footnote.
- [Section 4, Data Preprocessing] The text says values below 2 are labeled as 'rear or non-users'; 'rear' is a typo for 'rare.' Also, 'inconstant responses' should be 'inconsistent responses.'
- [Section 5, Figures] Figures 4, 5, 6, and 7 are referenced in the text but not all are discussed in sufficient detail; for example, Figure 5's right panel is not explicitly described in the multiclass paragraphs. The captions could more clearly state what each subplot shows (e.g., class labels).
- [Section 4, Hyperparameters] The text reports hyperparameters for the binary model (learning rate 0.153, max depth 2, 10 estimators, subsample 0.8) and later mentions different settings for the multiclass model (reg_lambda = 3). Clarify which hyperparameter set applies to each model and whether the reported CV performance used the same final parameters.
Circularity Check
The top predictor ChatGPT_Used_in_New_Learning is a near-duplicate of the target ChatGPT_Assignment; the headline 'overreliance' finding restates the survey's own response consistency rather than an independent behavioral mechanism.
-
self definitional
[Section 3 (variable definitions) and Section 5, 'SHAP Analysis & Gain for Binary Class Classifications']
"– ChatGPT_Assignment: Frequency of ChatGPT use for assignments. – ChatGPT_Used_in_New_Learning: ChatGPT usage frequency for new topics. ... The most impactful variable was ChatGPT_Used_in_New_Learning (mean| SHAP value| = 0.500), indicating that students who frequently use generative AI tools for learning new topics are substantially more likely to use it for assignment preparation as well."
The target and the top predictor are the same self-reported construct—ChatGPT usage frequency—differing only in the activity context named by the two survey items. Both are frequency ratings collected from the same respondents on the same questionnaire. The paper's central behavioral conclusion, that frequent ChatGPT use for learning new topics predicts assignment use and signals 'potential overreliance', therefore reduces by construction to the internal consistency of respondents' frequency ratings: ChatGPT users report using ChatGPT. The strong SHAP value and 80.1% accuracy reflect this construct/response overlap, not an independent, externally validated behavioral mechanism.
full rationale
The paper's technical pipeline (XGBoost, train/test split, SHAP) is not itself circular: the model is fit and evaluated on held-out data, and no parameter is fitted to the test set. However, the load-bearing substantive claim—that 'experiential familiarity' with ChatGPT, measured by ChatGPT_Used_in_New_Learning, predicts assignment use and indicates overreliance—is undermined because that predictor is defined as another ChatGPT-usage-frequency item on the same survey. The model's top feature carries the target's own variance, making the headline finding substantially tautological. Self-citation is present (reference [6] includes author S. Das) but is only related-work context and is not load-bearing. The central issue is construct-level circularity, not data leakage or overfitting, so the score reflects partial circularity of the interpretive result.
Assumptions & free parameters
free parameters (7)
- learning_rate =
0.153
- max_depth =
2
- n_estimators =
10
- subsample =
0.8
- reg_lambda =
3
- IV_threshold =
0.1
- binary_threshold =
>=2 on Likert scale
assumptions (4)
- domain assumption Self-reported survey answers reflect actual ChatGPT usage behavior
- domain assumption Information value based exclusion of features does not leak target information into model training
- domain assumption XGBoost feature importance and SHAP values reveal causal behavioral drivers
- domain assumption The sample of 388 students, predominantly Russian, is sufficient to make general claims about ChatGPT use in assignments
Cite this review
Pith. "Pith review of Predicting ChatGPT Use in Assignments: Implications for AI-Aware Assessment Design." pith.science (2026). https://pith.science/paper/NVZ6J6LY
@misc{pith2026250812013,
author = {Pith},
title = {Pith review of: Predicting ChatGPT Use in Assignments: Implications for AI-Aware Assessment Design},
year = {2026},
howpublished = {\url{https://pith.science/paper/NVZ6J6LY}},
note = {Machine review of arXiv:2508.12013}
}
read the original abstract
The rise of generative AI tools like ChatGPT has significantly reshaped education, sparking debates about their impact on learning outcomes and academic integrity. While prior research highlights opportunities and risks, there remains a lack of quantitative analysis of student behavior when completing assignments. Understanding how these tools influence real-world academic practices, particularly assignment preparation, is a pressing and timely research priority. This study addresses this gap by analyzing survey responses from 388 university students, primarily from Russia, including a subset of international participants. Using the XGBoost algorithm, we modeled predictors of ChatGPT usage in academic assignments. Key predictive factors included learning habits, subject preferences, and student attitudes toward AI. Our binary classifier demonstrated strong predictive performance, achieving 80.1\% test accuracy, with 80.2\% sensitivity and 79.9\% specificity. The multiclass classifier achieved 64.5\% test accuracy, 64.6\% weighted precision, and 64.5\% recall, with similar training scores, indicating potential data scarcity challenges. The study reveals that frequent use of ChatGPT for learning new concepts correlates with potential overreliance, raising concerns about long-term academic independence. These findings suggest that while generative AI can enhance access to knowledge, unchecked reliance may erode critical thinking and originality. We propose discipline-specific guidelines and reimagined assessment strategies to balance innovation with academic rigor. These insights can guide educators and policymakers in ethically and effectively integrating AI into education.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
E. M. Rogers,Diffusion of Innovations, 5th ed. Free Press, 2003
work page 2003
-
[2]
Perceived usefulness, perceived ease of use, and user accep- tance of information technology,
F. D. Davis, “Perceived usefulness, perceived ease of use, and user accep- tance of information technology,”MIS Quarterly, vol. 13, no. 3, pp. 319–340, 1989
work page 1989
-
[3]
Consumer acceptance and use of information technology,
V. Venkatesh, J. Y. L. Thong, and X. Xu, “Consumer acceptance and use of information technology,”MIS Quarterly, vol. 36, no. 1, pp. 157–178, 2012
work page 2012
-
[4]
Guest editorial: Education in the world of ChatGPT and generative AI,
M. Tan, S. Tan, and C. Krishnan, “Guest editorial: Education in the world of ChatGPT and generative AI,”IEEE Transactions on Learning Technolo- gies, vol. 17, no. 1, pp. 1–4, 2024, doi: 10.1109/TLT.2024.3384765
arXiv 2024
-
[5]
Framework for adop- tion of generative artificial intelligence (GenAI) in education,
S. Shailendra, R. Kadel, and A. Sharma, “Framework for adop- tion of generative artificial intelligence (GenAI) in education,” IEEE Transactions on Education , vol. 67, no. 5, pp. 777–785, 2024, doi: 10.1109/TE.2024.XXXXXXX
-
[6]
Orchestratingactivelearning in hybrid classroom: A case study and recommendations for instructors,
U.Khwaja,S.Sadhukhan,S.Das,andS.Iyer,“Orchestratingactivelearning in hybrid classroom: A case study and recommendations for instructors,” in Proceedings of the International Conference on Technology for Education (T4E), 2023, IEEE Xplore
work page 2023
-
[7]
The impact and application of gen- erative artificial intelligence within education,
S. Gutiérrez and R. Strachan, “The impact and application of gen- erative artificial intelligence within education,” 2024. [Online]. Avail- able: https://teaching.ieee.org/the-impact-and-application-of- generative-artificial-intelligence-within-education/ (accessed Jul. 1, 2025)
work page 2024
-
[8]
How 3 educators are using generative AI,
C. Fleury et al., “How 3 educators are using generative AI,” 2024, IEEE Transmitter. [Online]. Available:https://transmitter.ieee.org/how-3- educators-are-using-generative-ai/ (accessed Jul. 1, 2025)
work page 2024
Show all 15 references
-
[9]
Education 5.0: Transforming engineering education in the age of generative AI,
Anonymous, “Education 5.0: Transforming engineering education in the age of generative AI,” inProc. of the IEEE International Conference on Tech- nology for Education (T4E), 2024, IEEE Xplore, Document ID: 10814745
2024
-
[10]
The rise of ChatGPT: Exploring its impact on education,
S. Hu, S. Liu, and B. Xu, “The rise of ChatGPT: Exploring its impact on education,” arXiv preprint arXiv:2303.10296, 2023. [Online]. Available: https://arxiv.org/abs/2303.10296 (accessed Jul. 1, 2025)
2023 arXiv
-
[11]
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT,
D. R. E. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of ChatGPT,”Innovations in Educa- tion and Teaching International,2023,doi:10.1080/14703297.2023.2190148
2023
-
[12]
TheimpactofChatGPTonstudents’learningstrate- gies and academic writing,
A.M.Bettayeb et al.,“TheimpactofChatGPTonstudents’learningstrate- gies and academic writing,” inProceedings of the 2024 International Con- ference on Education and AI, 2024, doi: 10.1145/3610401.3611234
2024
-
[13]
Web mining generative AI policies in universities: A global analysis,
A. Rejeb et al., “Web mining generative AI policies in universities: A global analysis,” Education and Information Technologies, 2024, doi: 10.1007/s10639-024-12317-9. Title Suppressed Due to Excessive Length 15
2024 doi
-
[14]
Effect of AI-assisted tools on learning performance: A meta-analysis,
T. Wang, Y. Zhang, and L. Liu, “Effect of AI-assisted tools on learning performance: A meta-analysis,”Computers and Education: Artificial Intel- ligence, 2025, doi: 10.1016/j.caeai.2025.100157
2025
-
[2023]
Available: https://arxiv.org/abs/2308.04309, accessed Jul
[Online]. Available: https://arxiv.org/abs/2308.04309, accessed Jul. 1, 2025
2025 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.