"I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing

Arfa Naeem; Samra Zafar; Shaheer Minhas; Syed Ali Hassan Zaidi; Zahra Ali

REVIEW 4 major objections 5 minor 6 references

"I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that giving LLMs the keystroke logs and timed snapshots of an essay as it evolves yields feedback that learners find more accurate, personal, and aligned with their own revision thinking than feedback on the final draft…

desk verdict A usable process-aware feedback demo with a headline claim the design can't support; the abstract promises an analysis the body never delivers. read the letter →

arxiv 2506.08221 v1 pith:AM3HTVJN submitted 2025-06-09 cs.CL

Samra Zafar , Shaheer Minhas , Syed Ali Hassan Zaidi , Arfa Naeem , Zahra Ali This is my paper · ORCID

classification cs.CL

keywords LLMswritingprocesskeystrokeloggingAIfeedbackessayrevisioneducationaltechnologyprocess-awarestudent

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that essay feedback from large language models is better when the model sees the writing process, not just the final text. The authors built a browser-based writing tool that logs keystrokes and captures snapshots of the essay every three minutes, and fed those traces along with the final essay to an LLM to generate two-part feedback: rubric-based scores for thesis, language, relevance, and organization, plus a commentary on the writer's pauses and revisions tied to specific timestamps. Twenty undergraduates wrote timed essays with the tool and then rated the feedback; eighteen answered the follow-up survey. Results are presented as an early, mostly positive signal: most students found the process-aware feedback accurate, said it understood why they revised certain parts, and the majority preferred it for being 'in tune with their own thinking.' The intended contribution is a proof-of-concept that process traces make AI writing feedback feel more personal and pedagogically grounded.

What carries the argument

The process trace: a time-stamped record of every keystroke event (each backspace press archives the content typed so far; a new log starts on backspace release after a 3-second pause) plus a snapshot of the full textbox every 3 minutes (6 snapshots in a 20-minute session). The LLM is invoked on each snapshot or log event to stay context-aware in real time, and at submission the backend passes the final essay plus the complete trace to the model. The feedback has two parts: rubric-based evaluation (Thesis and Arguments, Language Use, Prompt Relevance, Organization/Structure) and a revision-behavior commentary that explicitly cites timestamps and time-tagged snapshots. The trace is the load-bearing object: without it the feedback has no access to pauses, deletions, or reorganizations.

What would settle it

Conduct an ablation study with the same writing tool: one arm receives feedback from the final essay alone, the other from the essay plus the keystroke and snapshot traces, and both arms rate how well the feedback reflected their own thinking; if the final-essay-only arm rates the feedback as equally accurate and personal, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that writing-process data—keystroke logging plus periodic snapshots—enables an LLM to read between the lines of a student essay, and that learners experience the resulting feedback as more meaningful than feedback derived from a final draft alone. Specifically, the authors argue that temporal and revision features (when a student paused, what they deleted, how a paragraph was reorganized, which word choice was negotiated) give the model evidence about the writer's cognitive state, so the feedback can name the struggle rather than just the error. The survey results are cited in support: 15 of 18 respondents (83%) said the tool captured their core writing issues, 13 (72%) felt it understood their revision journey, and Likert ratings for grammar relevance, helpfulness, and thesis accuracy averaged between 4.1 and 4.7 on a 5-point scale. The paper concludes that keylogging and snapshot analysis improve the scoring and qualitative feedback of LLM-based essay evaluation.

Load-bearing premise

The study's central comparative claim rests on the untested assumption that the 20 participants' positive ratings of the process-aware feedback were caused by the writing-process data itself, rather than by the general quality, detail, or length of the feedback, because every participant saw only the process-aware condition and there was no final-essay-only control arm.

Editorial extensions

If this is right

Writing tools that already run in a browser can collect the needed process data without new infrastructure, making process-aware feedback a practical upgrade rather than a research prototype.
Feedback that cites timestamps can point a student to the exact moment of decision, turning generic advice into a replay of their own writing moves.
Rubric feedback and revision commentary can be generated in a single pass from the same trace, so process awareness does not require a separate scoring pipeline.
Students' appreciation of the revision summary suggests that process-aware feedback may support metacognitive growth—seeing one's own pause-and-revise pattern—not just error correction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The design cannot prove that the process data caused the favorable ratings: with every participant receiving the same process-aware feedback and no final-essay-only control arm, an equally plausible explanation is that longer or more detailed feedback felt more helpful; a randomized A/B comparison is the natural test.
The abstract's claim that edit types such as adding content or reorganizing paragraphs align with higher coherence and elaboration scores is not shown in the body; verifying it would require linking logged edit events to rubric scores at the essay or passage level.
Process traces could be repurposed for just-in-time intervention: detecting a prolonged pause before a major edit could trigger a prompt at the moment of difficulty rather than after submission.
The 56% who felt earlier fixed mistakes were still over-flagged suggest a design refinement: separate 'resolved' issues from 'live' issues so the feedback reads as a growth narrative rather than a list of past faults.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the system is real and the qualitative findings are interesting, but the paper's central comparative claim—that learners preferred process-aware feedback over final-essay-only feedback—can't be tested from the design. There is no control condition. The abstract also says edit types correlated with higher scores, but no such analysis appears in the paper.

What's genuinely useful here: the interface captures keystrokes and snapshots, and the Gemini-generated feedback examples show real temporal awareness ('the 15:56–16:00 timestamp... a moment of decision-making'). The thematic analysis, even though it's LLM-assisted, surfaces honest negatives: 56% felt the system dwelled on already-fixed errors, 44% felt it missed rhetorical nuance. That's a credible exploratory picture of how students react to process-aware feedback.

The soft spots are load-bearing. All 20 participants got the same condition. High Likert means (e.g., 4.7 for grammar feedback) are absolute ratings, not preference judgments. Without a final-essay-only arm, you can't attribute the ratings to the process data rather than to the feedback being longer, more specific, or just fresh. The edit-type/score correlation claim is simply absent—no rubric scores by edit category are reported anywhere in the paper. The future work section calls for comparing LLM vs human feedback but never mentions the missing final-essay-only control, which is the comparison the headline demands.

The paper is honest about being early, and the qualitative themes (fairness, nuance) are worth reading if you work on writing tutors. But as submitted, it overstates what the study found. I'd treat it as an exploratory system paper, not as evidence for the 'preferred' claim. A serious referee could help the authors reframe it, but the current version's main claim is untestable, so I wouldn't send it to a full review. I'd desk-reject with an invitation to resubmit as a workshop paper with a control condition.

Referee Report

4 major / 5 minor

Summary. The paper presents a custom web-based writing interface that logs keystrokes and periodic textbox snapshots, and it feeds this process trace along with the final essay to Gemini for feedback generation. Twenty undergraduates wrote timed essays, received the process-aware feedback, and then completed a Likert and open-ended survey. The authors report generally positive Likert means and a thematic analysis of open-ended responses, and they claim in the abstract that learners preferred the process-aware feedback and that certain edit types correlate with rubric scores. The body, however, contains only descriptive survey statistics and qualitative themes; there is no comparison condition, no rubric-score analysis, and no inferential testing.

Significance. If the central claims were valid, the work would be a useful step toward process-aware automated writing feedback, with implications for educational NLP and human-AI interaction. The interface and logging design are sensible, and the authors are transparent about using machine-assisted thematic analysis with human oversight, which is a positive credibility feature. However, the headline contribution—that process-aware feedback is preferred or more aligned with learners' thinking—is not empirically established because the design is single-arm. The additional claim about edit types and scores is absent from the body. The paper's current evidentiary basis is too weak to support its advertised conclusions, though the system prototype might be a foundation for a better-controlled future study.

major comments (4)

[Abstract; §3.2–3.3] The abstract states that "learners preferred the process-aware LLM feedback, finding it more in tune with their own thinking." This comparative preference claim is unsupported: all 20 participants were given only the process-aware feedback condition, with no final-essay-only control arm, no random assignment, and no crossover. The Likert ratings in §4.1 are absolute satisfaction scores (e.g., M = 4.7 for grammar feedback), not preference judgments relative to a baseline. The positive ratings could plausibly reflect the general quality, length, or tone of the LLM feedback rather than the presence of process data. Because the central advertised contribution depends on this comparison, this is a load-bearing design flaw.
[Abstract; §4] The abstract also asserts that "certain types of edits, like adding new content or reorganizing paragraphs, aligned closely with higher scores in areas like coherence and elaboration." The body reports no rubric-score analysis by edit category; §4 contains only survey responses and thematic codes. This claim is not substantiated anywhere in the manuscript and should be either removed or supported with the relevant quantitative analysis.
[§4.1; §4.2] The quantitative results are purely descriptive means of six Likert items, with no standard deviations, no confidence intervals, no statistical tests, and no effect sizes. Even as descriptive evidence, the means cluster around 4–5 and could reflect ceiling effects or a general evaluative positivity bias. The paper does not demonstrate that the observed ratings are attributable to the writing-process trace, which is the core variable of interest. A more rigorous analysis would require at least a within-subjects comparison or an appropriate statistical model.
[§5] The Future Work section proposes "a structured comparison between LLM-generated and human-generated feedback" but never mentions the obvious missing control condition: feedback derived from the final essay alone without process data. This omission reinforces that the experiment was not designed to answer the paper's headline comparison, and it suggests the authors may not have recognized the control-condition requirement for their stated claim.

minor comments (5)

[§3.3] The text says 20 participants were recruited, but the thematic analysis is reported for 18 respondents; the discrepancy is never explained. Please clarify how many participants completed the survey and whether two were excluded.
[§3.3] The description of the ChatGPT-4o-assisted thematic analysis is vague about the division of labor. It says humans prepared exemplars and the LLM discovered categories, and that human researchers updated codes, but it is unclear whether the human researchers independently coded all responses or only reviewed the LLM's output. More detail on inter-rater reliability or validation steps would improve confidence in the qualitative findings.
[Throughout] The manuscript contains numerous typographical and formatting errors, including words run together (e.g., "forthiswork," "keyloggeddata") and incomplete figure captions. The template placeholders ("Conference acronym 'XX," "Received 20 February 2007") should be removed before submission.
[References] The references list two very similar entries by Meyer/Jansen and colleagues with nearly identical titles (Meyer et al. 2025 and Jansen et al. 2025). Please verify that these are distinct works and cite each appropriately; the in-text citations in §1 and §2 also conflate them.
[§4.1] The Likert items are described but the exact wording is not provided in a table or appendix, making it difficult to interpret the means. Consider including the full survey instrument.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central claims are empirically evaluated, not reduced to their inputs; the absence of a control arm is a validity threat, not circularity.

full rationale

The paper's derivation chain is descriptive and empirical rather than formal: it collects keystroke/snapshot traces, feeds them to an LLM, and surveys participants. There is no fitted parameter whose value is then reused as a prediction, no equation in which an output equals an input by construction, and no load-bearing self-citation (the cited references are external studies, and the closest internal 'citation' is a dangling template marker [1] that is not used as evidence). The abstract's comparative wording, 'learners preferred the process-aware LLM feedback,' is not supported by a final-essay-only control condition in Methods 3.2–3.3, and the edit-type/score alignment claim in the abstract has no corresponding analysis in the body; however, unsupported or overreached inferences are correctness/validity problems, not circularity. Using ChatGPT-4o to thematically code open-ended responses about LLM feedback is a methodological choice with external precedent (Dai et al. 2023; Qiao et al. 2025), and it does not make the survey responses an input to a derivation that is then claimed as output. Therefore no circular step meeting the quoting-and-reduction threshold can be exhibited, and the honest non-finding is a score of 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on proxy assumptions about process data and self-report. No free parameters are fit to data, and no new physical or conceptual entities are introduced. The main burden is the untested assumption that process data causes the observed positive ratings.

assumptions (3)

domain assumption Keystroke logs and three-minute snapshots are a valid representation of the student's cognitive writing process.
The entire intervention assumes that pausing, backspacing, and periodic text states reflect mental effort and revision intent; this is asserted in Sections 1 and 3.1 without validation against other measures of cognition.
domain assumption Self-reported survey ratings from 18 respondents are sufficient evidence for feedback quality.
Section 3.3 and Section 4 use Likert and open-ended responses as the outcome measure; no external measure of feedback quality, writing improvement, or learning gain is used.
domain assumption LLM-assisted thematic analysis with human review preserves the reliability of qualitative coding.
Section 3.4 uses ChatGPT-4o to generate themes from open-ended responses; the paper cites supporting literature but provides no inter-rater reliability statistics or independent human coding comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of "I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing." pith.science (2026). https://pith.science/paper/AM3HTVJN

@misc{pith2026250608221,
  author       = {Pith},
  title        = {Pith review of: "I Wrote, I Paused, I Rewrote" Teaching LLMs to Read Between the Lines of Student Writing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AM3HTVJN}},
  note         = {Machine review of arXiv:2506.08221}
}

read the original abstract

Large language models(LLMs) like Gemini are becoming common tools for supporting student writing. But most of their feedback is based only on the final essay missing important context about how that text was written. In this paper, we explore whether using writing process data, collected through keystroke logging and periodic snapshots, can help LLMs give feedback that better reflects how learners think and revise while writing. We built a digital writing tool that captures both what students type and how their essays evolve over time. Twenty students used this tool to write timed essays, which were then evaluated in two ways: (i) LLM generated feedback using both the final essay and the full writing trace, and (ii) After the task, students completed surveys about how useful and relatable they found the feedback. Early results show that learners preferred the process-aware LLM feedback, finding it more in tune with their own thinking. We also found that certain types of edits, like adding new content or reorganizing paragraphs, aligned closely with higher scores in areas like coherence and elaboration. Our findings suggest that making LLMs more aware of the writing process can lead to feedback that feels more meaningful, personal, and supportive.

Figures

Figures reproduced from arXiv: 2506.08221 by the authors.

**Figure 3.** Perceived Fairness and Insightfulness of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Students’ View on the System’s Ability to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Overall Helpfulness of Feedback in Improving [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 4 canonical work pages

[3]

Computers & Education 205 (2025), 105434

Understanding Individual Differences in Students’ Responses to Technology-Based Feedback on a Writing Task: The Role of Achievement Motives and Initial Task Performance. Computers & Education 205 (2025), 105434. doi:10.1016/j.compedu.2024.105434 MinLee,L.T.Chau,andD.Kang.2024. Process-AwareWritingAssistance:Modeling the Student Revision Journey with LLMs....

work page arXiv 2025
[4]

Understandingindividualdifferencesinstudents’responsestotechnology- basedfeedbackonawritingtask:theroleofachievementmotivesandinitialtask performance.JournalofResearchonTechnologyinEducation 0,0(2025),1–31. doi:10. 1080/15391523.2025.2471765 arXiv:https://doi.org/10.1080/15391523.2025.2471765 Shan Qiao, Xingyu Fang, Junbo Wang, Ran Zhang, Xiaoming Li, and...

work page arXiv 2025
[5]

medRxiv (2025)

Generative AI for Thematic Analysis in a Mater- nal Health Study: Coding Semi-structured Interviews using Large Lan- guage Models (LLMs). medRxiv (2025). doi:10.1101/2024.09.16.24313707 arXiv:https://www.medrxiv.org/content/early/2025/04/23/2024.09.16.243137 07.full.pdf RonjaSchiller,JohannaFleckenstein,UteMertens,AndreaHorbach,andJennifer Meyer

work page doi:10.1101/2024.09.16.24313707 2025
[2006]

IWrote,IPaused,IRewrote

Usingthematic analysis in psychology. Qual- itative Research in Psychology 3 (01 2006), 77–101. doi:10.1191/1478088706qp063oa A.Chan,M.Lee,andJ.Kim.2024. EnhancingStudentEngagementwithLLM-Generated Feedback:PromiseandLimitations.InProceedingsoftheAnnualMeetingofthe As- sociation for Computational Linguistics. Association for Computational Linguistics. Shi...

doi:10.1191/1478088706qp063oa 2024
[2024]

223,C(Dec

Understanding the effectiveness of automated feedback: Using process data to uncover the role of behavioral engagement.Comput.Educ. 223,C(Dec. 2024),16pages. doi:10.1016/j.compedu.2024.105163 RobertTai,LillianBentley,XinXia,JasonSitt,SarahFankhauser,AnaChicas-Mosier, andBarnasMonteith.2024. AnExaminationoftheUseofLargeLanguageModels toAidAnalysisofTextual...

work page arXiv 2024
[2025]

doi:10.1016/j.caeai.2024.100353 VirginiaBraun and Victoria Clarke

Assessing how accurately large language models encode and apply the common European framework of reference for languages.Computers and Education: Artificial Intelli- gence 8 (Jun 2025), 100353. doi:10.1016/j.caeai.2024.100353 VirginiaBraun and Victoria Clarke

arXiv 2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Computers & Education 205 (2025), 105434

Understanding Individual Differences in Students’ Responses to Technology-Based Feedback on a Writing Task: The Role of Achievement Motives and Initial Task Performance. Computers & Education 205 (2025), 105434. doi:10.1016/j.compedu.2024.105434 MinLee,L.T.Chau,andD.Kang.2024. Process-AwareWritingAssistance:Modeling the Student Revision Journey with LLMs....

work page arXiv 2025

[2] [4]

Understandingindividualdifferencesinstudents’responsestotechnology- basedfeedbackonawritingtask:theroleofachievementmotivesandinitialtask performance.JournalofResearchonTechnologyinEducation 0,0(2025),1–31. doi:10. 1080/15391523.2025.2471765 arXiv:https://doi.org/10.1080/15391523.2025.2471765 Shan Qiao, Xingyu Fang, Junbo Wang, Ran Zhang, Xiaoming Li, and...

work page arXiv 2025

[3] [5]

medRxiv (2025)

Generative AI for Thematic Analysis in a Mater- nal Health Study: Coding Semi-structured Interviews using Large Lan- guage Models (LLMs). medRxiv (2025). doi:10.1101/2024.09.16.24313707 arXiv:https://www.medrxiv.org/content/early/2025/04/23/2024.09.16.243137 07.full.pdf RonjaSchiller,JohannaFleckenstein,UteMertens,AndreaHorbach,andJennifer Meyer

work page doi:10.1101/2024.09.16.24313707 2025

[4] [2006]

IWrote,IPaused,IRewrote

Usingthematic analysis in psychology. Qual- itative Research in Psychology 3 (01 2006), 77–101. doi:10.1191/1478088706qp063oa A.Chan,M.Lee,andJ.Kim.2024. EnhancingStudentEngagementwithLLM-Generated Feedback:PromiseandLimitations.InProceedingsoftheAnnualMeetingofthe As- sociation for Computational Linguistics. Association for Computational Linguistics. Shi...

doi:10.1191/1478088706qp063oa 2024

[5] [2024]

223,C(Dec

Understanding the effectiveness of automated feedback: Using process data to uncover the role of behavioral engagement.Comput.Educ. 223,C(Dec. 2024),16pages. doi:10.1016/j.compedu.2024.105163 RobertTai,LillianBentley,XinXia,JasonSitt,SarahFankhauser,AnaChicas-Mosier, andBarnasMonteith.2024. AnExaminationoftheUseofLargeLanguageModels toAidAnalysisofTextual...

work page arXiv 2024

[6] [2025]

doi:10.1016/j.caeai.2024.100353 VirginiaBraun and Victoria Clarke

Assessing how accurately large language models encode and apply the common European framework of reference for languages.Computers and Education: Artificial Intelli- gence 8 (Jun 2025), 100353. doi:10.1016/j.caeai.2024.100353 VirginiaBraun and Victoria Clarke

arXiv 2025