REVIEW 4 major objections 4 minor 51 references
LLM-Assisted Cheating Detection in Korean Language via Keystrokes
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Keystroke timing can distinguish Korean essays written from scratch, paraphrased from ChatGPT, or transcribed from ChatGPT, with accuracy up to about 93 percent.
desk verdict Useful new Korean keystroke dataset for LLM-cheating detection, but the fixed scenario order threatens the main comparison and the human-baseline 'significance' isn't tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pairing of two interpretable keystroke feature sets with a cognitive-load-aware evaluation design. Temporal features—Key Hold Time (KHT) and Key Interval Time (KIT) summarized by quartiles, mean, and standard deviation—capture motor fluency and physical execution. Rhythmic features capture cognitively driven planning and revision: fifteen types including nine binned pause durations, P-bursts (text production segments ended by a pause of at least two seconds), R-bursts (writing sequences ending in deletion), delete bursts, inter-word and inter-sentence pauses, and pause-before-delete, summarized by seven statistics plus entropy, yielding 107 features. The evaluation splits prompts into low-load (Bloom's remember, understand, apply) and high-load (analyze, evaluate, create) groups, and compares Cognition-Unaware, Cognition-Aware (H→H, L→L), and cross-cognition (H→L, L→H) settings using user-independent splits of the 69 participants and MLP, SVM, and XGBoost classifiers with genetic-algorithm hyperparameter tuning.
What would settle it
Run the identical three-condition protocol with participants randomly assigned to different scenario orders and compare per-class accuracy; if accuracy drops or the temporal-feature advantage disappears when order is controlled, the central claim is an artifact of session ordering. A simpler check is to add elapsed time since session start as a feature and see whether it alone predicts the labels.
Extended reading notes
Core claim
The paper's central discovery is that the way a person types carries a reliable behavioral signature of how the text was produced, even when the text itself reads fluently. Using Key Hold Times and Key Interval Times as temporal features, and binned pauses, bursts, and deletion patterns as rhythmic features, the authors report that classifiers distinguish bona fide Korean writing from paraphrased and transcribed ChatGPT responses under both Cognition-Unaware and Cognition-Aware splits. Temporal features perform best when training and test tasks share the same cognitive load, while rhythmic features generalize better when cognitive load changes across train and test. Detecting transcribed responses was consistently easiest, with recall near or above 94 percent; paraphrased responses were the most confusable, especially with bona fide writing. A human baseline rated by 18 fluent Korean speakers was far weaker, correctly identifying only 69.44 percent of bona fide, 41.67 percent of paraphrased, and 49.07 percent of transcribed responses, which the paper takes as evidence that keystroke-based behavioral detection adds information beyond what text inspection provides.
Load-bearing premise
The three writing conditions were performed in a fixed order without reported counterbalancing, so condition is entangled with session time, fatigue, and practice; if that is true, the classifiers may be detecting time-on-task rather than the writing strategy.
Editorial extensions
If this is right
- In online Korean assessments, keystroke logging could flag suspected ChatGPT paraphrasing for human review, catching a form of cheating that text-matching tools miss.
- Transcribed ChatGPT text is the easiest to detect, so proctoring systems can prioritize high-confidence transcription flags with low false-positive risk.
- Because temporal features excel when cognitive load is known and rhythmic features excel when it varies, a fusion of both feature sets would likely outperform either alone.
- User-independent detection at 85–93 percent accuracy suggests the approach does not require enrolling a user's typing profile in advance.
- The public dataset gives other researchers a Korean-language benchmark for keystroke-based authorship and academic-integrity studies.
Reading between the lines
- If the three writing scenarios were always administered in the same order, part of the reported accuracy may reflect elapsed session time, fatigue, or practice rather than the cognitive writing strategy; a randomized-order replication would show how much of the signal is truly behavioral.
- The rhythmic features' robustness across cognitive-load transfer hints that pause and burst statistics approximate stable motor-cognitive traits, so similar detectors might transfer to other languages or to typing on different keyboards—an extension the paper does not test.
- The finding that human raters struggle most with paraphrase suggests that LLM-detection systems relying only on text content will keep failing; combining keystroke behavior with text-level markers is a concrete next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a Korean keystroke dataset from 69 participants who completed writing tasks under three conditions: bona fide writing, paraphrasing ChatGPT responses, and transcribing ChatGPT responses, with six prompts per condition aligned to Bloom's Taxonomy. It extracts interpretable temporal and rhythmic keystroke features and evaluates MLP, SVM, and XGBoost classifiers under user-independent splits in Cognition-Unaware and Cognition-Aware settings. The authors report roughly 85–93% classification accuracy, with transcribed responses easiest to detect, paraphrased responses hardest, temporal features stronger under Cognition-Aware settings, rhythmic features more robust under cross-cognition transfer, and models outperforming human raters. The dataset and code are publicly available.
Significance. If the results hold, this is a useful contribution: it provides the first Korean-language keystroke dataset for LLM-assisted cheating detection, uses a finer-grained three-class labeling of cheating behavior, and evaluates generalization to unseen users. The internal numbers are broadly consistent with the main claim, and the evaluation is not circular: accuracies are computed on held-out users, and features are not fitted to the labels. The public dataset and code are a concrete reproducibility asset. The main risks are experimental-design confounds and missing statistical support for headline comparisons, rather than methodological circularity.
major comments (4)
- [Section 3.1 and 3.2] The paper never states that the order of the three writing scenarios was randomized or counterbalanced. Section 3.2 reports that only Phase 1 data are used and Section 3.1 describes a five-minute break between scenarios, but if the scenario order was fixed, then the scenario label is confounded with session position, fatigue, and practice with the data-collection portal. Every accuracy reported in Figures 2–6 could then partly reflect a classifier reading the session clock or interface familiarity rather than the cognitive writing strategy. Because the dataset and code are public, please either state the randomization/counterbalancing protocol or run an order-effect check (e.g., compare feature distributions or classifier performance by session position within each scenario) and report the result.
- [Section 4.4 and Abstract] The abstract and Section 4.4 claim that the models 'significantly outperform' human evaluators, but no significance test is reported. The human baseline uses only 18 raters and 18 responses, provides no chance-level benchmark or confidence intervals, and compares text-only human judgments with keystroke-based model predictions, which are different information modalities. Please add a proper statistical comparison (e.g., bootstrap or a mixed-effects model with rater and item random effects), report inter-rater agreement, and qualify the claim if the comparison is not statistically supported.
- [Section 3.2] The central claim depends on the scenario labels being correct, but the paper does not quantify adherence. Section 3.2 lists three validation steps, including verifying word counts and checking paraphrase quality, yet it reports no statistics from these checks (e.g., how many ChatGPT responses had to be re-prompted, how many participant responses were rejected, or how much paraphrases differed textually from their ChatGPT source). Please report adherence statistics or a text-similarity analysis between paraphrased responses and their source; otherwise label noise is unmeasured and the reported accuracies may be optimistic.
- [Section 3.5] The Cognition-Aware analysis defines high and low cognitive load by question index (questions 1–3 vs 4–6) without any independent validation of the load manipulation. The Section 4.2 and 4.3 claims about high-load discriminability and cross-cognition transfer rest entirely on this assignment. Please justify the mapping with pilot ratings, response-time data, or a manipulation check, or relabel the axis as 'question set' rather than cognitive load.
minor comments (4)
- [Section 2] There is a typographical issue in 'difficult to fake ,' with a stray space before the comma; please proofread the manuscript.
- [Section 4.4] The phrase 'minimal contextual guidance and explicit definitions of each label were provided' is internally contradictory; please clarify exactly what guidance and definitions the human raters received.
- [Section 3.6 and Figures 2, 4, 6] The text refers to confidence bands and varying training sizes, but the details of how the confidence bands were computed are not specified; please state the estimator and the number of repetitions used.
- [Section 3.4.2] The sentence listing the seven statistics reads as if 'first and third quartiles' is a single item; please rephrase as 'first quartile, third quartile' to avoid confusion.
Circularity Check
No significant circularity: the central claim is supported by held-out user evaluation, and no fitted parameter is reused as a prediction.
full rationale
I walked the claimed derivation chain: scenario labels are set by the experimental protocol; keystroke features are extracted from raw event timestamps; classifiers are evaluated user-independently on non-overlapping train/test user splits; feature selection and genetic-algorithm hyperparameter tuning are confined to training splits. The reported accuracies are therefore not constructed from the labels or from test-set information. The self-citations present in the paper (e.g., Kundu et al. [2], Kumar et al. [24,25,28], Khurana et al. [44]) are used for background, feature conventions, or planned future methodology, and none carries the load of the central transfer claim. The possible session-order confound identified during review is a validity or experimental-design concern, not a circular reduction: it does not make the derivation equivalent to its inputs by definition. No equation, fitted parameter, or renamed known result is reused as the predicted outcome, so no circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- Pause bin width =
300 ms
- P-burst pause threshold =
2 s
- Feature selection ratio =
50%
- Classifier hyperparameters =
Not reported
assumptions (4)
- domain assumption Keystroke behavioral features are valid proxies for cognitive processes such as planning, revision, and evaluation.
- domain assumption Bloom's Taxonomy questions classified as low (1-3) and high (4-6) induce distinct cognitive load.
- domain assumption Participants complied with the assigned writing condition and manual validation captured noncompliance.
- standard math Multiple responses from the same participant are treated as independent samples after user-level splitting.
Cite this review
Pith. "Pith review of LLM-Assisted Cheating Detection in Korean Language via Keystrokes." pith.science (2026). https://pith.science/paper/QHLRXYFK
@misc{pith2026250722956,
author = {Pith},
title = {Pith review of: LLM-Assisted Cheating Detection in Korean Language via Keystrokes},
year = {2026},
howpublished = {\url{https://pith.science/paper/QHLRXYFK}},
note = {Machine review of arXiv:2507.22956}
}
read the original abstract
This paper presents a keystroke-based framework for detecting LLM-assisted cheating in Korean, addressing key gaps in prior research regarding language coverage, cognitive context, and the granularity of LLM involvement. Our proposed dataset includes 69 participants who completed writing tasks under three conditions: Bona fide writing, paraphrasing ChatGPT responses, and transcribing ChatGPT responses. Each task spans six cognitive processes defined in Bloom's Taxonomy (remember, understand, apply, analyze, evaluate, and create). We extract interpretable temporal and rhythmic features and evaluate multiple classifiers under both Cognition-Aware and Cognition-Unaware settings. Temporal features perform well under Cognition-Aware evaluation scenarios, while rhythmic features generalize better under cross-cognition scenarios. Moreover, detecting bona fide and transcribed responses was easier than paraphrased ones for both the proposed models and human evaluators, with the models significantly outperforming the humans. Our findings affirm that keystroke dynamics facilitate reliable detection of LLM-assisted writing across varying cognitive demands and writing strategies, including paraphrasing and transcribing LLM-generated responses.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
S. Crossley, Y . Tian, J. Choi, L. Holmes, and W. Morris. Plagiarism detection using keystroke logs. In IEEE ICEDM,
- [2]
-
[3]
M. Hoq, Y . Shi, J. Leinonen, Damilola B., Collin L., T. Price, and B. Akram. Detecting chatgpt-generated code submissions in a cs1 course using machine learning models. SIGCSE, 2024. 1
work page 2024
-
[4]
Using keystroke dynamics to detect nonoriginal text
Paul Deane, Mo Zhang, Jiangang Hao, and Chen Li. Using keystroke dynamics to detect nonoriginal text. Journal of Educational Measurement, 2025. 1, 5, 6
work page 2025
-
[5]
Keystroke dynamics as signal for shallow syntactic parsing
Barbara Plank. Keystroke dynamics as signal for shallow syntactic parsing. In COLING, Osaka, Japan, 2016. The COLING 2016 Organizing Committee. 1, 2
work page 2016
-
[6]
N. Agarwal, N. Danielsen, P. Gravdal, and P. Bours. Contract cheat detection using biometric keystroke dynamics. In ICETA, 2022. 1
work page 2022
-
[7]
Elliott Casal and Matt Kessler
J. Elliott Casal and Matt Kessler. Can linguists distinguish between chatgpt/ai and human writing?: A study of research ethics and academic publishing. Research Methods in Applied Linguistics, 2023. 1, 2
work page 2023
-
[8]
A large-scale comparison of human-written versus chatgpt-generated essays
Steffen Herbold, Annette Hautli-Janisz, Ute Heuer, Zlata Kikteva, and Alexander Trautsch. A large-scale comparison of human-written versus chatgpt-generated essays. Scientific Reports, 2023. 1
work page 2023
Show all 51 references
-
[9]
E-cheating prevention measures: detection of cheating at online examinations using deep learning approach–a case study
Leslie Ching Ow Tiong and HeeJeong Jasmine Lee. E-cheating prevention measures: detection of cheating at online examinations using deep learning approach–a case study. https://arxiv.org/abs/2101.09841, 2021. 1
2021 arXiv
-
[10]
Automated cheating detection based on video surveillance in the examination classes
M Roa’a, Ibtisam A Aljazaery, and AHM Alaidi. Automated cheating detection based on video surveillance in the examination classes. iJIM, 2022. 1
2022
-
[11]
Student cheating detection in higher education by implementing machine learning and lstm techniques
Waleed Alsabhan. Student cheating detection in higher education by implementing machine learning and lstm techniques. Sensors, 2023. 1
2023
-
[12]
Machine learning-based approach to exam cheating detection
Firuz Kamalov, Hana Sulieman, and David Santandreu Calonge. Machine learning-based approach to exam cheating detection. Plos one, 2021. 1
2021
-
[13]
Keystroke analysis: Reflections on procedures and measures
Veerle M Baaijen, David Galbraith, and Kees De Glopper. Keystroke analysis: Reflections on procedures and measures. Written Communication, 2012. 1
2012
-
[14]
Using keystroke analytics to understand cognitive processes during writing
Mo Zhang, Hongwen Guo, and Xiang Liu. Using keystroke analytics to understand cognitive processes during writing. IEDMS, 2021. 1, 2
2021
-
[15]
A cognitive process theory of writing
Linda Flower and John R Hayes. A cognitive process theory of writing. College Composition and Communication, 1981. 1, 2
1981
-
[16]
Modeling and remodeling writing
John R Hayes. Modeling and remodeling writing. Written communication, 2012. 1, 2, 5
2012
-
[17]
From keystrokes to cognitive processes: Analyzing morphological knowledge using keystroke logging
Andreas Sellstone. From keystrokes to cognitive processes: Analyzing morphological knowledge using keystroke logging. Master’s thesis, Ume ˚a University, 2023. 1, 2, 5
2023
-
[18]
Investigating cognitive rhythms as a new modality for continuous authentication
Kiran S Balagani. Investigating cognitive rhythms as a new modality for continuous authentication. Technical report, NYIT, 2013. 1, 2, 4, 5
2013
-
[19]
Detecting deception by manipulating cognitive load
Aldert Vrij, Ronald Fisher, Samantha Mann, and Sharon Leal. Detecting deception by manipulating cognitive load. Trends in cognitive sciences, 2006. 1
2006
-
[20]
Anderson and David R
Lorin W. Anderson and David R. Krathwohl, editors. A Taxonomy for Learning, Teaching, and Assessing. A Revision of Bloom’s Taxonomy of Educational Objectives . Allyn & Bacon, 2001. 1, 2, 3
2001
-
[21]
Identity authentication based on keystroke latencies
Rick Joyce and Gopal Gupta. Identity authentication based on keystroke latencies. Commun. ACM, 1990. 2
1990
-
[22]
Verifying identity via keystroke characterstics
John Leggett and Glen Williams. Verifying identity via keystroke characterstics. International Journal of Man-Machine Studies, 1988. 2
1988
-
[23]
Fabian Monrose and Aviel D. Rubin. Keystroke dynamics as a biometric for authentication. FGCS, 2000. 2
2000
-
[24]
Phoha, and Abdul Serwadda
Rajesh Kumar, Vir V . Phoha, and Abdul Serwadda. Continuous authentication of smartphone users by fusing typing, swiping, and phone movement patterns. In IEEE BTAS, 2016. 2
2016
-
[25]
Rajesh Kumar, Partha Pratim Kundu, and Vir V . Phoha. Continuous authentication using one-class classifiers and their fusion. In IEEE ISBA, 2018. 2
2018
-
[26]
On continuous user authentication via typing behavior
Joseph Roth, Xiaoming Liu, and Dimitris Metaxas. On continuous user authentication via typing behavior. IEEE Transactions on Image Processing, 2014. 2
2014
-
[27]
Investigating the discriminative power of keystroke sound
Joseph Roth, Xiaoming Liu, Arun Ross, and Dimitris Metaxas. Investigating the discriminative power of keystroke sound. IEEE TIFS, 2015. 2
2015
-
[28]
Spotting fake profiles in social networks via keystroke dynamics
Alvin Kuruvilla, Rojanaye Daley, and Rajesh Kumar. Spotting fake profiles in social networks via keystroke dynamics. In IEEE-CCNC, 2024. 2, 5
2024
-
[29]
Ieee-bigmm
Vishaal Udandarao, Mohit Agrawal, Rajesh Kumar, and Rajiv Ratn Shah. Ieee-bigmm. In IEEE-BigMM, 2020. 2
2020
-
[30]
Monaco, Ruben Vera-Rodriguez, and Julian Fierrez
Alejandro Acien, Aythami Morales, John V . Monaco, Ruben Vera-Rodriguez, and Julian Fierrez. Typenet: Deep learning keystroke biometrics. IEEE TBIOM, 2022. 2, 4
2022
-
[31]
Plagiarism deterrence in cs1 through keystroke data
Kaden Hart, Chad Mano, and John Edwards. Plagiarism deterrence in cs1 through keystroke data. In SIG-CSE, 2023. 2
2023
-
[32]
Shared multi-keyboard and bilingual datasets to support keystroke dynamics research
Ahmed Anu Wahab, Daqing Hou, Mahesh Banavar, Stephanie Schuckers, Kenneth Eaton, Jacob Baldwin, and Robert Wright. Shared multi-keyboard and bilingual datasets to support keystroke dynamics research. InACM CODASPY,
-
[33]
Alsuhibany, Muna Almushyti, Noorah Alghasham, and Fatimah Alkhudier
Suliman A. Alsuhibany, Muna Almushyti, Noorah Alghasham, and Fatimah Alkhudier. Analysis of free-text keystroke dynamics for arabic language using euclidean distance. In 12th International Conference on Innovations in Information Technology (IIT), 2016. 2
2016
-
[34]
Keystroke timing analysis for individual identification in japanese free text typing
Toshiharu Samura and Haruhiko Nishimura. Keystroke timing analysis for individual identification in japanese free text typing. In ICCAS-SICE, 2009. 2
2009
-
[35]
Language dependent challenge-based keystroke dynamics
Patrick Bours and Sukanya Brahmanpally. Language dependent challenge-based keystroke dynamics. In IEEE ICCST, 2017. 2
2017
-
[36]
User-representative feature selection for keystroke dynamics
Eesa Al Solami, Colin Boyd, Andrew Clark, and Irfan Ahmed. User-representative feature selection for keystroke dynamics. In 5th International Conference on Network and System Security, 2011. 2
2011
-
[37]
Shared data set for free-text keystroke dynamics authentication algorithms
Catalin Iapa and Vladimir Cretu. Shared data set for free-text keystroke dynamics authentication algorithms. In Preprint,
-
[38]
On continuous user authentication via hidden free-text based monitoring
Elena Kochegurova, Elena Luneva, and Ekaterina Gorokhova. On continuous user authentication via hidden free-text based monitoring. In IITI. Springer International Publishing, 2019. 2
2019
-
[39]
Freely typed keystroke dynamics-based user authentication for mobile devices based on heterogeneous features
Junhong Kim and Pilsung Kang. Freely typed keystroke dynamics-based user authentication for mobile devices based on heterogeneous features. Pattern Recognition, 2020. 2
2020
-
[40]
Stragapede
Giuseppe et al. Stragapede. Ieee bigdata 2023 keystroke verification challenge (kvc). In 2023 IEEE International Conference on Big Data (BigData), 2023. 4
2023
-
[41]
Typeformer: Transformers for mobile keystroke biometrics
Giuseppe Stragapede, Paula Delgado-Santos, Ruben Tolosana, Ruben Vera-Rodriguez, Richard Guest, and Aythami Morales. Typeformer: Transformers for mobile keystroke biometrics. Neural Computing and Applications ,
-
[42]
Wahab and D
A. Wahab and D. Hou. When simple statistical algorithms outperform deep learning: A case of keystroke dynamics. In ICPRAM, 2023. 4
2023
-
[43]
Survey of review spam detection using machine learning techniques
Michael Crawford, Taghi M Khoshgoftaar, Joseph D Prusa, Aaron N Richter, and Hamzah Al Najada. Survey of review spam detection using machine learning techniques. Journal of Big Data, 2015. 4
2015
-
[44]
Synthesizing human gaze feedback for improved NLP performance
Varun Khurana, Yaman Kumar, Nora Hollenstein, Rajesh Kumar, and Balaji Krishnamurthy. Synthesizing human gaze feedback for improved NLP performance. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2023. 5
2023
-
[45]
Alejandro et al. Acien. Detection of mental fatigue in the general population: Feasibility study of keystroke dynamics as a real-world biomarker. JMIR Biomedical Engineering ,
-
[46]
Keystroke dynamics: Concepts, techniques, and applications
Rashik Shadman, Ahmed Anu Wahab, Michael Manno, Matthew Lukaszewski, Daqing Hou, and Faraz Hussain. Keystroke dynamics: Concepts, techniques, and applications. ACM Computing Survey, 2025. 5
2025
-
[47]
Examining Pauses in Writing: Theory, Methods and Empirical Data
Wengelin, ˚Asa. Examining Pauses in Writing: Theory, Methods and Empirical Data. Computer key-stroke logging and writing: methods and applications (Studies in Writing) ,
-
[48]
Keystroke patterns as prosody in digital writings: A case study with deceptive reviews and essays
Ritwik Banerjee, Song Feng, Jun Seok Kang, and Yejin Choi. Keystroke patterns as prosody in digital writings: A case study with deceptive reviews and essays. In EMNLP, 2014. 5
2014
-
[49]
Detecting contract cheating using learning analytics
Kelly Trezise, Tracii Ryan, Paula de Barba, and Gregor Kennedy. Detecting contract cheating using learning analytics. Journal of Learning Analytics, 2019. 5
2019
-
[50]
Using keystroke behavior patterns to detect nonauthentic texts in writing assessments: Evaluating the fairness of predictive models
Yang Jiang, Mo Zhang, Jiangang Hao, Paul Deane, and Chen Li. Using keystroke behavior patterns to detect nonauthentic texts in writing assessments: Evaluating the fairness of predictive models. Journal of Educational Measurement ,
-
[51]
Young and Derek C
Steven R. Young and Derek C. et al. Rose. Optimizing deep learning hyperparameters through an evolutionary algorithm. In MLHPC. ACM, 2015. 6
2015
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.