Pith. sign in

REVIEW 3 major objections 4 minor 35 references

How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MOOC essays grew longer and less varied after ChatGPT

desk verdict A solid descriptive study of MOOC essays around ChatGPT's release; the causal reading is not secured, but the paper is worth engaging seriously. read the letter →

arxiv 2504.13038 v1 pith:YIO5DMFQ submitted 2025-04-17 cs.CY cs.CL

Leo Leppänen , Lili Aunimo , Arto Hellas , Jukka K. Nurminen , Linda Mannila This is my paper · ORCID
classification cs.CYcs.CL
keywords largelanguagemodelsMOOCstudentessaysessaylengthtype-tokenratioFleschReadingEaseAI-generatedtextlongitudinalanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the arrival of ChatGPT changed how students write in a free online course on AI ethics. Comparing 56,878 essay submissions from November 2020 to October 2024, it finds that essays submitted after ChatGPT's release are on average longer, written in shorter and simpler sentences, and draw on a narrower vocabulary. The share of tell-tale LLM words such as 'delve' and 'foster' rose roughly tenfold. The authors argue these shifts strongly suggest that a meaningful proportion of MOOC participants now rely on large language models to produce their essay answers, even though individual essays cannot be identified as AI-generated.

What carries the argument

The argument rests on a longitudinal comparison of essay statistics before and after ChatGPT's release, with the dataset split into pre-November 2022, the first year after, and at least one year after. The carrying instruments are token and sentence counts, Flesch Reading Ease, type-token ratio, and relative prevalence of LLM-associated words, each tested for significance with the Mann-Whitney U test. The time-series plots with monthly bins show the shift arriving around March 2023 rather than gradually, which is what gives the paper its temporal claim.

What would settle it

If a similar MOOC with identical prompts but no exposure to ChatGPT showed the same essay-length increase during 2020-2024, or if essay lengths were already trending upward before November 2022, the central claim would be false. More directly, if forensic AI-detection on individual essays found that the post-2023 length increase is concentrated in essays written by students who demonstrably did not use LLMs, the inference from style to LLM use would be undermined.

Watch

Extended reading notes

Core claim

The central discovery is a measured before-and-after shift in student writing that lines up with ChatGPT's release. Mean answer length rose from 150.5 to 230.1 tokens, mean sentence count from 6.85 to 9.76, and mean Flesch Reading Ease from 12.70 to 15.31, while type-token ratio fell from 0.617 to 0.577. Relative prevalence of 'delve' increased 10.45-fold and 'foster' 10.84-fold. Essay topics, measured by topic modeling, stayed broadly stable. The paper interprets these statistics as signs that many students are submitting at least partially LLM-generated answers, while carefully noting that cohort changes and shifting public discourse are alternative explanations it cannot fully rule out.

Load-bearing premise

The pre/post comparison assumes that the student population, course materials, and peer-review behavior stayed essentially the same across the four years, so that the measured changes come from ChatGPT's arrival rather than from a shifting cohort; the authors note that demographics are unavailable.

Editorial extensions

If this is right

  • If the shift is genuine, MOOC providers can no longer assume peer-reviewed essays reflect a student's own writing, which pressures them to adopt proctored exams or other verification for certificate value.
  • Essay length and vocabulary statistics can serve as cheap, aggregate signals for monitoring LLM adoption in a course over time.
  • The narrowing vocabulary and rising readability imply that writing produced with LLM assistance is, on average, more homogeneous and easier to read, which may change what instructors can infer from style.
  • Topic stability despite term shifts suggests that students now discuss the same AI-ethics themes but with a different technical vocabulary, so syllabus content may need fewer updates than stylistic expectations do.
  • The near-synonym standardization (e.g., 'recommendation system' replacing 'recommender system') hints that LLM use may be pushing terminology toward one canonical form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: apply an AI-detector calibrated on known human and LLM essays to this corpus and check whether the post-2023 length increase concentrates in essays flagged as machine-generated; the paper's argument predicts it should.
  • The same method could be applied to other written assignments, such as discussion forum posts or short-answer exams, to see whether the style shift is specific to high-stakes essays or generalizes to low-stakes writing.
  • If the vocabulary shift is driven by non-native speakers using LLMs as translators, as the authors speculate, then the length increase should also appear in courses taught in other languages; comparing multilingual versions of the same MOOC would separate translation use from wholesale generation.
  • The data imply that 'LLM-indicator' words like 'delve' may soon stop being reliable signals, because students who learn from LLM output will adopt them into genuinely human writing, so future detection work will need continuously refreshed baselines.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper presents a longitudinal observational study of 56,878 English-language essay submissions from 3,582 participants in a free University of Helsinki MOOC on AI ethics, spanning November 2020 to October 2024. The authors compare essays submitted before ChatGPT's release with essays submitted at least one year after, and report statistically significant increases in mean token count (150.5 to 230.1), sentence count (6.85 to 9.76), Flesch Reading Ease, and in the relative prevalence of words previously identified as LLM-associated (e.g., 'delve' and 'foster' increased roughly tenfold), alongside a decrease in type-token ratio (0.617 to 0.577). They also report changes in AI-related terminology but no meaningful changes in topic-model-derived topics. The paper interprets these shifts as evidence that a meaningful proportion of MOOC participants rely on LLMs, while acknowledging that the data cannot rule out coincidence or identify specific AI-generated essays.

Significance. The study's descriptive contribution is valuable: to my knowledge it is one of the first longitudinal analyses of MOOC essay text across the ChatGPT launch, using a large naturalistic corpus and externally motivated indicator words. The main findings are large, internally consistent, and in the direction predicted by prior work on LLM writing style. However, the inferential leap from aggregate before/after changes to 'students rely on LLMs' is not secured by the design, and the statistical analysis overstates precision by ignoring clustering. If the authors reframe the contribution as descriptive evidence and weaken the causal conclusion, the paper would be a solid empirical contribution.

major comments (3)
  1. [Section 5.2 and Section 6] The central claim that 'a meaningful proportion of MOOC participants rely on LLMs such as ChatGPT to produce essay answers' is not identified by the pooled before/after comparison in a single course. Section 5.3 states that demographic information is unavailable, and Section 5.1 concedes that the terminology changes could derive from 'shifting focuses of interest and public discourse applied to even a static essay prompt.' Without a comparison group, an interrupted time-series design, or within-participant panel analysis, the observed length and vocabulary shifts could equally reflect changes in who enrolls, in course materials, or in peer-review norms. The March 2023 break, rather than November 2022, further complicates attribution to ChatGPT's release. Please either add a control or placebo analysis (e.g., essays from a course on an unrelated topic over the same period) or explicitly downgrade the conclusion to a descriptive claim.
  2. [Section 4.1 and Section 4.2] All Mann-Whitney U tests treat the 56,878 essays as independent observations even though they come from only 3,582 participants, with multiple essays per participant. This violates the independence assumption and makes the reported p-values (e.g., U=165056438.0, p<0.0001 for token count) artificially small. Please aggregate at the participant level or use a mixed-effects model with participant as a random effect, and report effect sizes or standardized differences in addition to p-values. This is load-bearing for the RQ1/RQ2 significance claims, though the large raw differences may survive the correction.
  3. [Section 3.3 and Section 4.3] The topic-modeling analysis is described as showing no meaningful changes, but the manuscript does not report the Gensim hyperparameters (number of topics, passes, chunksize) or any measure of topic-model stability or validation. Please report these details and, if possible, provide a quantitative comparison (e.g., topic coherence or divergence) rather than relying on visual inspection, so that the negative result for RQ3 is interpretable. This comment does not affect the main descriptive findings but matters for the completeness of the RQ3 analysis.
minor comments (4)
  1. [Section 2.1] There is a duplicated word in the sentence 'have mainly mainly concerned experts'; please correct it.
  2. [Tables 1 and 3 and Section 4.3] The typo 'prevalance' appears in the tables and the typo 'occurences' appears in Section 4.3; the column label 'pM W U' should be formatted consistently.
  3. [Reproducibility] The manuscript does not state whether the data or analysis code will be made available; given the novelty of the dataset, an availability statement would strengthen reproducibility.
  4. [Figure 1] The figure captions state that the shaded area indicates the first and third quartiles, but the text does not define the exact period windows used for the pre/post tests; please clarify how the 'at least one year post' window maps onto the monthly bins.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the before/after essay comparisons are descriptive and do not reduce to their own inputs or to self-citations.

full rationale

The paper is an observational study comparing MOOC essay statistics before and after ChatGPT's release. It does not fit parameters to a target quantity and then re-predict that quantity; the reported differences in token counts, sentence counts, type-token ratio, and word prevalences are computed directly from the corpus using standard text-processing tools. The indicator words and stylometric expectations are borrowed from independent external studies (Kobak et al. and Andre et al.), and the paper reports outcomes that could have gone against those expectations, such as the non-significant change for "important." The few self-citations in the reference list (e.g., Denny et al. and Kauttonen et al.) appear only as background context and are not load-bearing for the central inference. The conclusion that a meaningful proportion of MOOC participants rely on LLMs is an interpretive claim about an observed correlation, not a quantity defined by the analysis itself, and the paper explicitly acknowledges the absence of demographic data and the possibility of alternative explanations such as shifting public discourse. No step in the paper's reasoning equates an input with an output by construction, and no load-bearing premise is justified solely by a self-citation chain. Therefore, no significant circularity is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No new entities are postulated. The study is a purely observational text analysis with no fitted constants; the main analytic choices are the borrowed stylistic markers and unreported topic model settings.

free parameters (1)
  • Gensim topic model hyperparameters (number of topics, passes, chunksize) = Not reported
    The null result for RQ3 depends on the configuration of the topic models, but the chosen settings are not given in the paper.
assumptions (5)
  • domain assumption The essay prompts and course structure remained essentially constant across 2020-2024.
    The authors state in Section 5.1 that essay prompts have stayed constant; if prompts or course materials changed, length and vocabulary shifts could reflect the course rather than LLM influence.
  • domain assumption The student cohort composition did not change systematically over time.
    Demographic data are unavailable (Section 5.3), so pre/post differences could partly reflect shifts in who takes the course.
  • standard math Statistical tests treat essays as independent observations.
    Mann-Whitney U tests in Section 4 assume independent samples, but 56,878 essays come from 3,582 participants and include repeated submissions, making the independence assumption questionable.
  • domain assumption The flagged-spam filtering produces a clean corpus without biasing pre/post comparisons.
    Data preprocessing in Section 3.2 removes essays flagged by peers or staff; if flagging behavior changed over time, the comparison could be biased.
  • domain assumption Words identified in prior academic-writing studies are valid LLM markers in this learner population.
    The paper applies 'delve', 'foster', and similar words from Kobak et al. to MOOC essays written largely by non-native English speakers; their diagnostic value in this population is assumed, not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses." pith.science (2026). https://pith.science/paper/YIO5DMFQ

@misc{pith2026250413038,
  author       = {Pith},
  title        = {Pith review of: How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YIO5DMFQ}},
  note         = {Machine review of arXiv:2504.13038}
}
read the original abstract

The release of ChatGPT in late 2022 caused a flurry of activity and concern in the academic and educational communities. Some see the tool's ability to generate human-like text that passes at least cursory inspections for factual accuracy ``often enough'' a golden age of information retrieval and computer-assisted learning. Some, on the other hand, worry the tool may lead to unprecedented levels of academic dishonesty and cheating. In this work, we quantify some of the effects of the emergence of Large Language Models (LLMs) on online education by analyzing a multi-year dataset of student essay responses from a free university-level MOOC on AI ethics. Our dataset includes essays submitted both before and after ChatGPT's release. We find that the launch of ChatGPT coincided with significant changes in both the length and style of student essays, mirroring observations in other contexts such as academic publishing. We also observe -- as expected based on related public discourse -- changes in prevalence of key content words related to AI and LLMs, but not necessarily the general themes or topics discussed in the student essays as identified through (dynamic) topic modeling.

Figures

Figures reproduced from arXiv: 2504.13038 by the authors.

Figure 1
Figure 1. Key text statistics. Line indicates mean, and the shaded area indicates the first and third quartiles of the data. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 26 canonical work pages

  1. [1]

    C. M. André, H. F. Eriksen, E. J. Jakobsen, L. C. Mingolla, and N. B. Thomsen. Detecting AI authorship: Analyzing descriptive features for AI detection. In Proceedings of the Seventh Workshop on Natural Language for Artificial Intelligence (NL4AI 2023) co-located with 22th International Conference of the Italian Association for Artificial Intelligence (AI...

    work page 2023

  2. [2]

    S. Bird. NLTK: the Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72, 2006

    work page 2006

  3. [3]

    D. Bo, A. A. Ma’rof, and Z. Zaremohzzabieh. The influence of negative stereotypes in science fiction and fantasy on public perceptions of artificial intelligence: A systematic review. Studies in Media and Communication, 13(1): 180 – 190, 2025. doi:10.11114/smc.v13i1.7212

    work page doi:10.11114/smc.v13i1.7212 2025

  4. [4]

    Bochkarev, V

    V . Bochkarev, V . Solovyev, and S. Wichmann. Universals versus historical contingencies in lexical evolution. Journal of The Royal Society Interface, 11(101):20140841, 2014

    work page 2014

  5. [5]

    Campbell

    L. Campbell. Historical linguistics. Edinburgh University Press, 2021

    work page 2021

  6. [6]

    Denny, H

    P. Denny, H. Khosravi, A. Hellas, J. Leinonen, and S. Sarsa. Can we trust AI-generated educational content? Comparative analysis of human and AI-generated learning resources. arXiv preprint arXiv:2306.10509, 2023

    arXiv 2023

  7. [7]

    genlangs

    J. Diamond. “genlangs” and Zipf’s Law: Do languages generated by ChatGPT statistically look human? arXiv preprint arXiv:2304.12191, 2023. doi:https://doi.org/10.48550/arXiv.2304.12191

    arXiv doi:10.48550/arxiv.2304.12191 2023

  8. [8]

    F. Duarte. Number of ChatGPT users. https://explodingtopics.com/blog/chatgpt-users. Accessed: 2025-02-11

    work page 2025

Show all 35 references
  1. [9]

    Y . Fan, L. Tang, H. Le, K. Shen, S. Tan, Y . Zhao, Y . Shen, X. Li, and D. Gaševi´c. Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 2024

    2024

  2. [10]

    A. V . Ferreiro.The influence of social media in language change: Changes in vocabulary. PhD thesis, Universidade de Santiago de Compostela, 2019

    2019

  3. [11]

    F. J. García-Peñalvo and A. Vázquez-Ingelm. What do we mean by GenAI? a systematic mapping of the evolution, trends, and techniques involved in Generative AI. International Journal of Interactive Multimedia and Artificial Intelligence, 8(4):7–16, 2023

    2023

  4. [12]

    Geng and R

    M. Geng and R. Trotta. Is ChatGPT transforming academics’ writing style?, 2024. URL https://arxiv.org/ abs/2404.08627

    2024 arXiv

  5. [13]

    M. Gerlich. Ai tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15 (1):6, 2025

    2025

  6. [14]

    Goulart, M

    L. Goulart, M. L. Matte, A. Mendoza, L. Alvarado, and I. Veloso. AI or student writing? analyzing the situational and linguistic characteristics of undergraduate student writing and AI-generated assignments. Journal of Second Language Writing, 66:101160, 2024

    2024

  7. [15]

    Haenlein and A

    M. Haenlein and A. Kaplan. A brief history of artificial intelligence: On the past, present, and future of artificial intelligence. California Management Review, 61:000812561986492, 07 2019. doi:10.1177/0008125619864925

    2019 doi

  8. [16]

    Kauttonen, U

    J. Kauttonen, U. A. Khan, L. Aunimo, A. Nyqvist, and A. Klemetti. Topic mining for theses and job ads in ICT sector: can higher education institutes respond to job market demands? In Frontiers in Education, volume 9, page 1322774. Frontiers Media SA, 2024

    2024

  9. [17]

    Kobak, R

    D. Kobak, R. G. Márquez, E.-Á. Horvát, and J. Lause. Delving into ChatGPT usage in academic writing through excess vocabulary. arXiv preprint arXiv:2406.07016, 2024

    2024 arXiv

  10. [18]

    H.-P. H. Lee, A. Sarkar, L. Tankelevitch, I. Drosos, S. Rintel, R. Banks, and N. Wilson. The impact of generative ai on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge 9 workers. In CHI Conference on Human Facto...

    2025

  11. [19]

    Liang, Y

    W. Liang, Y . Zhang, Z. Wu, H. Lepp, W. Ji, X. Zhao, H. Cao, S. Liu, S. He, Z. Huang, D. Yang, C. Potts, C. D. Manning, and J. Y . Zou. Mapping the increasing use of LLMs in scientific papers, 2024. URL https: //arxiv.org/abs/2404.01268

    2024 arXiv

  12. [20]

    E. A. M., F. L. D., G. E., and F. Martínez Plumed. Glossary of human-centric artificial intelligence. Technical Report JRC129614, 2022

    2022

  13. [21]

    Martínez, L

    G. Martínez, L. Watson, P. Reviriego, J. A. Hernández, M. Juarez, and R. Sarkar. Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet. In F. Cuzzolin and M. Sultana, editors, Epistemic Uncertainty in Artificial Intelligence, pages 59–73, C...

    2024

  14. [22]

    Mindner, T

    L. Mindner, T. Schlippe, and K. Schaaff. Classification of human-and ai-generated texts: Investigating features for chatgpt. In International Conference on Artificial Intelligence in Education Technology, pages 152–170. Springer, 2023

    2023

  15. [23]

    Mizumoto, S

    A. Mizumoto, S. Yasuda, and Y . Tamura. Identifying ChatGPT-generated texts in EFL students’ writing: Through comparative analysis of linguistic fingerprints. Applied Corpus Linguistics, 4(3):100106, 2024

    2024

  16. [24]

    ’Selfie’ named Word of the Year 2013

    Oxford University Press. ’Selfie’ named Word of the Year 2013. URL https://global.oup.com/ news-items/11001744/word-of-the-year-selfie-2013

    2013

  17. [25]

    J. Reed. How social media is changing language, June 2014. URL https://blog.oup.com/2014/06/ social-media-changing-language/

    2014

  18. [26]

    R. Rehak. The language labyrinth: Constructive critique on the terminology used in the ai discourse. In P. Verdegem, editor,AI for Everyone? Critical Perspectives , pages 87–102. University of Westminster Press, London, 2021. doi:10.16997/book55.f

    2021 doi

  19. [27]

    Reviriego, J

    P. Reviriego, J. Conde, E. Merino-Gómez, G. Martínez, and J. A. Hernández. Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans. Machine Learning with Applications, 18:100602, 2024. ISSN 2666-8270. doi:https://doi.org/10.1016/j.mlwa.2024.100602

    2024

  20. [28]

    Safi and A

    R. Safi and A. J. Naini. The work of students and ChatGPT compared: Using machine learning to detect and characterize AI-generated text. In AMCIS, 2023

    2023

  21. [29]

    T. B. Sardinha. AI-generated vs human-authored texts: A multidimensional comparison. Applied Corpus Linguistics, 4(1):100083, 2024

    2024

  22. [30]

    Sentance and J

    S. Sentance and J. Waite. Teachers’ perspectives on talk in the programming classroom: Language as a mediator. In Proceedings of the 17th ACM Conference on International Computing Education Research , ICER 2021, page 266–280, New York, NY , USA, 2021. Association for Computing...

    2021

  23. [31]

    S. A. Tagliamonte. So sick or so cool? the language of youth on the internet. Language in Society, 45(1):1–32,

  24. [32]

    C. C. Tossell, N. L. Tenhundfeld, A. Momen, K. Cooley, and E. J. de Visser. Student perceptions of ChatGPT use in a college essay assignment: Implications for learning, grading, and trust in artificial intelligence. IEEE Transactions on Learning Technologies, 2024

    2024

  25. [33]

    Uchida, Y

    S. Uchida, Y . Arase, and T. Kajiwara. Profiling English sentences based on CEFR levels. ITL-International Journal of Applied Linguistics (Belgium), pages 103–126, Mar. 2024. doi:10.1075/itl.22018.uch

    2024 doi

  26. [34]

    Yildiz Durak, F

    H. Yildiz Durak, F. E˘gin, and A. Onan. A comparison of human-written versus AI-generated text in discussions at educational settings: Investigating features for ChatGPT, Gemini and BingAI. European Journal of Education, 60 (1):e70014, 2025. 10

    2025

  27. [2016]

    doi:10.1017/S0047404515000780

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.