Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In a Python tutoring environment, students without prior programming experience spent nearly twice as much of their visual attention on feedback generated by a large language model as experienced students did, yet they understood it less…

desk verdict A small but honest eye-tracking study of GenAI vs. compiler feedback in a custom tutor; the descriptive findings are plausible and new, but the unvalidated manual gaze mapping for 3 of 11 participants is the soft spot to press on in revision. read the letter →

arxiv 2506.20433 v2 pith:OAVVHYU4 submitted 2025-06-25 cs.CY

Sven Jacobs , Maurice Kempf , Natalie Kiesler This is my paper
classification cs.CY
keywords ProgrammingeducationFeedbackLargeLanguageModelsGenerativeAIGenEye-trackingThink-aloudLearnerexperience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show how students actually engage with feedback generated by a large language model while solving Python tasks, rather than how good the feedback looks to experts. Using eye-tracking, think-aloud verbalization, and follow-up interviews with 11 undergraduates in the custom Tutor Kai environment, the authors measure attention to the GenAI feedback pane versus the compiler output pane. The central result is that GenAI feedback draws far more visual attention than compiler feedback, and the pattern splits by prior experience: students without prior programming experience fixated on the AI feedback nearly twice as long, requested it more often, understood it less, and benefited less from it, while often skipping the compiler output entirely. A sympathetic reader would care because this suggests that the value of GenAI feedback depends on the learner's background, and that naive reliance on it may widen, rather than close, the gap between prepared and unprepared students.

What carries the argument

The apparatus that carries the argument is a purpose-built web application, the Tutor Kai, whose screen is divided into four Areas of Interest: task description, code editor, GenAI feedback, and compiler feedback. Gaze recordings from eye-tracking glasses are mapped onto these areas to yield fixation times, and the synchronized gaze video is triangulated with think-aloud transcripts to classify each feedback instance as 'has helped,' 'has not helped,' or 'has not been read.' This pairing of where students look with what they say and do is what lets the authors distinguish attention from comprehension and from actual problem-solving benefit.

What would settle it

Re-analyze or replicate the eye-tracking data with a single automated gaze-mapping setup and a larger sample. If, excluding the three manually mapped participants, the gap between inexperienced (30.71%) and experienced (15.49%) fixation time on GenAI feedback shrinks to statistical noise, or if a larger study finds no comprehension difference (22 of 107 versus 1 of 64), then the experience-split claim would be an artifact of the mapping procedure or the small sample.

Watch

Extended reading notes

Core claim

The study's central claim is that in the Tutor Kai, engagement with GenAI feedback is high overall but unequal across experience levels. Specifically, the GenAI feedback captured 23.79% of total fixation time compared to 7.00% for compiler feedback, and inexperienced students spent 30.71% of their fixation time on the GenAI feedback versus 15.49% for experienced students. Inexperienced students also requested the AI feedback more often (107 versus 64 requests), were helped by it less often (43.0% versus 60.9% of the time), and failed to comprehend 22 of 107 messages compared to 1 of 64 for experienced students. The paper argues these numbers show that GenAI feedback can support problem-solving, but only when learners have enough foundational knowledge to interpret it, and that feedback systems need to adapt to learners' prior knowledge.

Load-bearing premise

The whole attention comparison rests on assuming that where students fix their gaze, measured by eye-tracking and mapped onto screen regions, accurately reflects what they are actually reading and processing; for three of the eleven participants that gaze mapping was done manually frame by frame after a tracker change, so small mapping errors could shift the reported fixation percentages and the 'read' versus 'not read' classifications.

Editorial extensions

If this is right

  • If the central claim is right, a student's prior programming experience is a decisive factor in whether GenAI feedback helps or confuses, so feedback tools must adapt explanations to the learner's level.
  • Inexperienced students' habit of requesting GenAI feedback before reading compiler output means that adding LLM-based explanations of compiler errors alone will not help the students who skip the compiler pane entirely.
  • The finding that 22 of 107 AI messages were incomprehensible to novices, often because they referenced concepts like loops without defining them, implies that feedback prompts should be constrained to avoid unexplained terminology and to include concrete code examples.
  • A feedback system that withholds or delays AI feedback until the student has engaged with compiler output could push novices toward the interpretation skills they currently bypass.
  • Because students rated the AI feedback highly (average 8.59/10) even when it did not objectively help, perceived helpfulness cannot be used as a proxy for actual learning support; tools should be evaluated with behavioral measures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to run the same Tutor Kai study with adaptive feedback that expands unfamiliar terms and provides syntax examples, predicting that the comprehension gap between experience groups narrows and the 'has helped' rate for novices rises.
  • The eye-tracking result suggests that in naturalistic settings, novices may be spending their most productive problem-solving time reading fluent but under-specific AI explanations instead of iterating with the compiler, which could slow the development of debugging skills over a full course.
  • One implicit consequence is that LLM-generated feedback should perhaps be gated by a capability check, e.g., pop-up definitions when the feedback mentions concepts outside the student's demonstrated vocabulary, since the paper's category analysis shows 'incomprehensible' and 'missing syntax example' failures are both addressable at generation time.
  • The authors' claim that the sample is representative rests on a course survey in which about half of 97 respondents reported no prior knowledge; a larger replication with more than 11 participants would be needed to see whether the 30.71% versus 15.49% fixation split generalizes beyond this single course.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a mixed-methods lab study with 11 undergraduate students using the authors' custom web application, Tutor Kai, which presents Python programming tasks, a code editor, GenAI feedback from GPT-4 Turbo, and compiler feedback. Eye-tracking, think-aloud protocols, and semi-structured interviews are used to investigate (RQ1) how much attention learners pay to GenAI feedback relative to compiler feedback and (RQ2) to what extent the GenAI feedback is helpful, with comparisons between inexperienced (n=6) and experienced (n=5) students. The headline descriptive findings are that GenAI feedback received 23.79% of overall fixation time, that inexperienced students fixated on it nearly twice as long as experienced students (30.71% vs. 15.49%), that inexperienced students requested GenAI feedback 107 times vs. 64 times for experienced students, and that GenAI feedback was coded as helpful in 43.0% of cases for inexperienced students vs. 60.9% for experienced students. The authors also report that inexperienced students failed to comprehend 22 of 107 GenAI feedback messages, versus 1 of 64 for experienced students, and that compiler feedback was frequently not read by inexperienced students. The paper concludes with concerns about over-reliance on GenAI feedback and suggestions for adaptive feedback design.

Significance. If the descriptive findings hold, the paper makes a useful empirical contribution to computing education research by directly measuring visual attention to GenAI feedback in a realistic programming-task environment and by triangulating gaze with think-aloud and code-change evidence. Strengths include a transparent coding scheme that links gaze, verbalization, and code improvements; the availability of an open repository with the prompt, tasks, and interview guide; and an explicit threats-to-validity section that acknowledges think-aloud disruption, eye-tracker distraction, and LLM nondeterminism. The main limitation is that the gaze-to-AoI mapping, which underpins both research questions, is insecure for three of eleven participants and is not validated; this tempers the strength of the quantitative comparisons. The paper is best read as an exploratory descriptive study rather than as providing robust statistical evidence of group differences.

major comments (4)
  1. [Section 4.2.1; Section 4.2.2; Section 5.2; Section 5.3] The central quantitative claims depend on the gaze-to-AoI mapping, but for participants S01, S02, and S10 this mapping was done manually, frame by frame, after a hardware change to a different Tobii tracker, and no validation, inter-coder agreement, drift correction, or sensitivity analysis is reported. Because the Section 4.2.2 coding scheme defines 'Has helped,' 'Has not helped,' and 'Has not been read' using the precondition that the feedback 'was read (gaze visualization),' any mapping error can reclassify an instance across the helpfulness categories. The three manually mapped participants are distributed across both comparison groups (S01 in the inexperienced group; S02 and S10 in the experienced group), so systematic error would not cancel. Please report a robustness check (e.g., recomputing the headline fixation percentages and helpfulness rates with S01/S02/S10 excluded) or provide evidence of mapping reliability.
  2. [Section 5.3, paragraph beginning 'Our analysis of the 77 instances'] The count of 'has not helped' instances is internally inconsistent. The text states that the analysis covers 77 such instances, but the four listed categories sum to 78 (25 + 23 + 17 + 13). In addition, the relationship between this count, the stated 49.7% helpfulness rate for 171 GenAI feedback outputs, and any 'has not been read' classifications is not fully specified. Please reconcile these numbers and clarify the denominator for each category.
  3. [Section 4.2.2] The helpfulness coding is load-bearing for RQ2, but no inter-rater reliability or second-coder check is reported. The categories require judgment about whether a code change 'directly related' to provided information and whether a verbalization indicates help, so coder subjectivity is nontrivial. Please report agreement on at least a subset of the 171 GenAI feedback instances and the 287 compiler feedback instances, or justify why a single-coder scheme is sufficient in this context.
  4. [Section 5.2; Section 5.3] The paper makes comparative claims such as 'nearly twice as much fixation time,' 'almost four times more helpful,' and 'substantial differences between the two groups' without any confidence intervals or inferential statistics. With n=6 versus n=5 and highly variable per-student behavior, these percentage differences may be fragile. At minimum, the claims should be reframed as exploratory descriptive observations, or accompanied by effect sizes and uncertainty estimates.
minor comments (6)
  1. [Table 2] The text in Section 5.3 reports an average feedback helpfulness rating of 8.59, but the eleven values in Table 2 sum to 95, giving a mean of 8.64; please correct the stated average or the table.
  2. [Table 1 caption] The caption contains a typo: 'Fixiated' should be 'Fixated'.
  3. [Section 4.1.2] The sentence 'They took about 40 minutes to complete all the tasks' has an unclear antecedent; specify that the two student tutors took about 40 minutes.
  4. [Section 7] The statement 'With the sample size of 11 students and 171 GenAI feedbacks, we have reached a large enough sample [4]' is overgeneralized. The cited source supports adequacy for qualitative analysis, not for the quantitative comparative claims; please temper the wording.
  5. [Section 3.2] Temperature=0 reduces but does not eliminate output variability; Section 7's statement that LLM output generation 'remains intransparent' is fine, but the earlier 'maximize output consistency' wording could be softened.
  6. [Section 5.2] In the last sentence of Section 5.2, 'experience students' should be 'experienced students'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the engagement results are direct eye-tracking, log, and think-aloud measurements; the manual gaze mapping for three participants is a validity threat, not a circular derivation.

full rationale

This paper reports an eye-tracking, think-aloud, and interview study of how 11 students engage with GenAI feedback in a custom tutor. The central numerical claims (23.79% of fixation time on GenAI feedback; 30.71% vs 15.49% for inexperienced vs experienced students; 22/107 vs 1/64 incomprehension cases) are direct measurements obtained from gaze-to-AoI mapping, system request logs, and coded think-aloud protocols, not quantities derived from a fitted model or from the cited prior work. The self-citations to Jacobs and Jaschke [11-13] describe the earlier development and evaluation of the Tutor Kai and the iterative prompt design; they provide context and tool provenance, but they do not constitute the evidence for the present engagement findings. The only substantive weakness is the manual frame-by-frame gaze mapping for participants S01, S02, and S10 reported in Section 4.2.1, which is a measurement-validity and reliability threat that could shift fixation percentages and therefore affect RQ1 and the gaze-dependent half of the RQ2 coding. That is an evidentiary limitation rather than circularity: the coding categories 'Has helped', 'Has not helped', and 'Has not been read' are operationalized from observed gaze and behavior rather than being assumed as inputs, and the helpfulness rates are not redefinitions of the eye-tracking data. No equation is defined in terms of a target result, no fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation forces the conclusions. Aside from minor, non-load-bearing self-citation for the tool and prompt lineage, the derivation chain is self-contained, so the circularity burden is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions about measurement validity: eye-tracking, think-aloud, the causal coding of helpfulness, and self-reported experience. There are no free parameters or fitted constants, and no new theoretical entities are introduced. Tutor Kai is an author-built experimental instrument rather than a postulated entity.

assumptions (4)
  • domain assumption Fixation time on an Area of Interest is a valid measure of learner attention to that interface element.
    Used in Section 4.2.1 to answer RQ1; if fixation does not correspond to cognitive attention or reading, the central attention comparisons collapse.
  • domain assumption Think-aloud verbalizations are a reliable window into problem-solving during feedback processing.
    Used in Section 4.2.2 and Section 4.1; the helpfulness coding relies on verbalized thoughts combined with gaze data.
  • domain assumption The helpfulness coding rule that 'read feedback plus subsequent related code improvement' implies the feedback caused the improvement.
    Section 4.2.2 defines 'Has helped'; this assumes a causal interpretation of the gaze-to-code sequence without a control condition.
  • domain assumption Self-reported prior programming experience is an accurate grouping variable.
    Section 5.1 uses self-reports to split 11 students into 6 inexperienced and 5 experienced; the authors call task-performance data confirmation, but the grouping itself is self-reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai." pith.science (2026). https://pith.science/paper/OAVVHYU4

@misc{pith2026250620433,
  author       = {Pith},
  title        = {Pith review of: That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAVVHYU4}},
  note         = {Machine review of arXiv:2506.20433}
}
read the original abstract

The potential of Generative AI (GenAI) for generating feedback in computing education has been the subject of numerous studies. However, there is still limited research on how computing students engage with this feedback and to what extent it supports their problem-solving. For this reason, we built a custom web application providing students with Python programming tasks, a code editor, GenAI feedback, and compiler feedback. Via a think-aloud protocol including eye-tracking and a post-interview with 11 undergraduate students, we investigate (1) how much attention the generated feedback received from learners and (2) to what extent the generated feedback is helpful (or not). In addition, students' attention to GenAI feedback is compared with that towards the compiler feedback. We further investigate differences between students with and without prior programming experience. The findings indicate that GenAI feedback generally receives a lot of visual attention, with inexperienced students spending twice as much fixation time. More experienced students requested GenAI less frequently, and could utilize it better to solve the given problem. It was more challenging for inexperienced students to do so, as they could not always comprehend the GenAI feedback. They often relied solely on the GenAI feedback, while compiler feedback was not read. Understanding students' attention and perception toward GenAI feedback is crucial for developing educational tools that support student learning.

Figures

Figures reproduced from arXiv: 2506.20433 by the authors.

Figure 1
Figure 1. Web application with highlighted AoIs In the center of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Fixation Times on Areas of Interest Comparative analysis reveals that experienced students spent 15.49% of the time on the GenAI feedback, which is approximately half as much as inexperienced students spent on the GenAI feedback (30.71%). The inverse pattern emerged for compiler feedback, where experienced students exhibited nearly double the fixation time (9.35%) compared to inexperienced students (5.05%). Regardin… view at source ↗
Figure 3
Figure 3. Resulting Helpfulness of GenAI and Compiler Feedback for Experienced and Inexperienced Students (RQ2) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?

    cs.AI 2025-07 conditional novelty 6.0 of 10

    In a 136-student trial, novice programmers' feedback requests to a purpose-built ChatGPT tutor followed a consistent sequence, and the tutor's responses aligned with requested feedback types in 75% of exchanges.

Reference graph

Works this paper leans on

44 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [2]

    Brett A Becker, Michelle Craig, Paul Denny, Hieke Keuning, Natalie Kiesler, Juho Leinonen, Andrew Luxton-Reilly, James Prather, and Keith Quille. 2024. Generative AI in Introductory Programming. InComputer Science Curricula 2023. ACM, New York, USA, 438–439

    work page 2024

  2. [4]

    Clive Roland Boddy. 2016. Sample size for qualitative research.Qualitative market research: An international journal19, 4 (2016), 426–432

    work page 2016

  3. [5]

    Teresa Busjahn, Carsten Schulte, and Andreas Busjahn. 2011. Analysis of Code Reading to Gain More Insight in Program Comprehension. InProceedings of the 11th Koli Calling International Conference on Computing Education Research. ACM, Koli Finland, 1–9. https://doi.org/10.1145/2094131.2094133

    work page arXiv 2011

  4. [6]

    Teresa Busjahn, Carsten Schulte, Bonita Sharif, Simon, Andrew Begel, Michael Hansen, Roman Bednarik, Paul Orlov, Petri Ihantola, Galina Shchekotova, and Maria Antropova. 2014. Eye Tracking in Computing Education. InProceedings of the Tenth Annual Conference on International Computing Education Research. ACM, 3–10. https://doi.org/10.1145/2632320.2632344

    work page arXiv 2014

  5. [8]

    Paul Denny, Stephen MacNeil, Jaromir Savelka, Leo Porter, and Andrew Luxton- Reilly. 2024. Desirable Characteristics for AI Teaching Assistants in Programming Student Engagement with GenAI Feedback UKICER ’25, September 04–05, 2025, Edinburgh, UK Education. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. ACM, 408...

    work page arXiv 2024

  6. [9]

    Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. InProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1. ACM, New York, USA, 93–105. https://doi.org/10.1145/3568813.3600139

    arXiv 2023

  7. [10]

    Sven Jacobs. 2025. Data: That’s Not the Feedback I Need! - Student Engagement with GenAI Feedback in the Tutor Kai. https://doi.org/10.17605/OSF.IO/US9HJ

    work page doi:10.17605/osf.io/us9hj 2025

  8. [11]

    Sven Jacobs and Steffen Jaschke. 2024. Evaluating the Application of Large Language Models to Generate Feedback in Programming Education. In2024 IEEE Global Engineering Education Conference (EDUCON). IEEE. https://doi.org/10. 1109/EDUCON60312.2024.10578838

    arXiv 2024

Show all 44 references
  1. [12]

    Sven Jacobs and Steffen Jaschke. 2024. Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation. In2024 36th International Conference on Software Engineering Education and Training. IEEE. https://doi.org/10.1109/CSEET62301.20...

    2024

  2. [13]

    Sven Jacobs, Henning Peters, Steffen Jaschke, and Natalie Kiesler. 2025. Unlimited Practice Opportunities: Automated Generation of Comprehensive, Personalized Programming Tasks. InProceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education...

    2025

  3. [14]

    Johan Jeuring, Hieke Keuning, Samiha Marwan, Dennis Bouvier, Cruz Izu, Natalie Kiesler, Teemu Lehtinen, Dominic Lohr, Andrew Peterson, and Sami Sarsa. 2022. Towards Giving Timely Formative Feedback and Hints to Novice Programmers. InProceedings of the 2022 Working Group Report...

    2022

  4. [15]

    Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs. InProc. of the CHI Conference ...

    2024

  5. [16]

    Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2019. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises.ACM ToCE19 (2019), 1–43. https://doi.org/10.1145/3231711

    2019 doi

  6. [18]

    Natalie Kiesler. 2023. Investigating the Use and Effects of Feedback in Coding- Bat Exercises: An Exploratory Thinking Aloud Study.2023 Future of Educa- tional Innovation-Workshop Series Data in Action: Digital Ecosystem and Emerging Tools for Education(2023), 1–12. https://do...

    2023

  7. [19]

    Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023. Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. In 2023 IEEE Frontiers in Education Conference (FIE). IEEE, College Station, USA, 1–5. https://doi.org/10.1109/FIE58773.2023.10343457

    2023

  8. [20]

    Natalie Kiesler and Daniel Schiffner. 2023. Large Language Models in Intro- ductory Programming Education: ChatGPT’s Performance and Implications for Assessments. https://doi.org/10.48550/arXiv.2308.08572 arXiv:2308.08572 [cs.SE]

    2023 doi arXiv

  9. [21]

    Natalie Kiesler, Ingo Scholz, Jens Albrecht, Friedhelm Stappert, and Uwe Wienkop

  10. [22]

    Charles Koutcheme, Nicola Dainese, Arto Hellas, Sami Sarsa, Juho Leinonen, Syed Ashraf, and Paul Denny. 2024. Evaluating Language Models for Generating and Judging Programming Feedback. https://doi.org/10.48550/arXiv.2407.04873

    2024 doi arXiv

  11. [23]

    2019.Analyzing qualitative data with MAXQDA

    Udo Kuckartz and Stefan Rädiker. 2019.Analyzing qualitative data with MAXQDA. Springer

    2019

  12. [24]

    Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Program- ming Error Messages. InProc. of the 54th ACM Technical Symposium on Computer Science Education V. 1. ACM, 563–569. https:...

    2023

  13. [25]

    Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. 2024. Code- Help: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. InProceedings of the 23rd Koli Calling International Conference on Computing Education Research. ACM, New Y...

    2024

  14. [26]

    Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024. Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. InProceedings of the 55th ACM Technical Symposium on Computer Science Educatio...

    2024

  15. [27]

    Rongxin Liu, Julianna Zhao, Benjamin Xu, Christopher Perez, Yuliia Zhukovets, and David J. Malan. 2025. Improving AI in CS50: Leveraging Human Feedback for Better Learning. InProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1(Pittsburgh, PA, USA...

    2025

  16. [28]

    Dominic Lohr, Hieke Keuning, and Natalie Kiesler. 2025. You’re (Not) My Type- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?Journal of Computer Assisted Learning41, 1 (2025), e13107. https://doi. org/10.1111/jcal.13107 e13107 JCAL-24-434.R1

    2025 doi

  17. [29]

    Wenhan Lyu, Yimeng Wang, Tingting Rachel Chung, Yifan Sun, and Yixuan Zhang. 2024. Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study.arXiv:2404.13414(2024)

    2024 arXiv

  18. [30]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help With Code Understanding. InProc. of the IEEE/ACM 46th ICSE. ACM. https://doi.org/10.1145/3597503.3639187

    2024

  19. [31]

    Susanne Narciss. 2008. Feedback strategies for interactive learning tasks.Hand- book of research on educational communications and technology3 (2008), 125–144

    2008

  20. [32]

    Unaizah Obaidellah, Mohammed Al Haek, and Peter C.-H. Cheng. 2019. A Survey on the Usage of Eye-Tracking in Computer Programming.Comput. Surveys51, 1 (Jan. 2019), 1–58. https://doi.org/10.1145/3145904

    2019 doi

  21. [33]

    Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. 2024. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hi...

    2024

  22. [34]

    Becker, Bailey Kimmel, Jared Wright, and Ben Briggs

    James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs

  23. [35]

    Kumar, Bonnie MacKellar, Renée McCauley, Syed Waqar Nabi, and Michael Oudshoorn

    Rajendra Raj, Mihaela Sabin, John Impagliazzo, David Bowers, Mats Daniels, Felienne Hermans, Natalie Kiesler, Amruth N. Kumar, Bonnie MacKellar, Renée McCauley, Syed Waqar Nabi, and Michael Oudshoorn. 2021. Professional Com- petencies in Computing Education: Pedagogies and Ass...

    2021

  24. [36]

    The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. InProc. of ICER 2024, Vol. 1. ACM, New York, 469–486. https://doi.org/10.1145/3632620.3671116

    2024

  25. [37]

    Lianne Roest, Hieke Keuning, and Johan Jeuring. 2023. Next-Step Hint Generation for Introductory Programming Using Large Language Models. InProceedings of the 26th Australasian Computing Education Conference. ACM, Sydney, Australia, 144–153. https://doi.org/10.1145/3636243.3636259

    2023

  26. [38]

    Reeves, James Prather, Paul Denny, Juho Leinonen, Stephen MacNeil, Brett A

    Brent N. Reeves, James Prather, Paul Denny, Juho Leinonen, Stephen MacNeil, Brett A. Becker, and Andrew Luxton-Reilly. 2024. Prompts First, Finally. https: //doi.org/10.48550/arXiv.2407.09231

    2024 doi arXiv

  27. [39]

    Andreas Scholl and Natalie Kiesler. 2024. How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course. In2024 IEEE Frontiers in Education Conference (FIE). 1–9. https://doi.org/ 10.1109/FIE61694.2024.10893442

    2024

  28. [41]

    Andreas Scholl, Daniel Schiffner, and Natalie Kiesler. 2024. Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT. InProceedings of DELFI 2024, Sandra Schulz and Natalie Kiesler (Eds.). 63–79. https://doi.org/10.18420/delfi2024_05

    2024 doi

  29. [42]

    Andreas Scholl and Natalie Kiesler. 2025. SCRIPT - Supportive Chatbot for Resolving Introductory Programming Tasks. InProceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 2. ACM, New York, 759. https://doi.org/10.1145/3724389.3730786

    2025

  30. [43]

    Valerie J. Shute. 2008. Focus on formative feedback.Review of Educational Research78, 1 (2008)

    2008

  31. [44]

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, and Chenglei et al. Si. 2024. The Prompt Report: A Systematic Survey of Prompting Techniques. https://doi.org/10.48550/arXiv.2406.06608

    2024 doi arXiv

  32. [45]

    Irene Stone. 2024. Exploring Human-Centered Approaches in Generative AI and Introductory Programming Research: A Scoping Review. InProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research. ACM, Manchester United Kingdom, 1–7. https://doi.org/...

    2024

  33. [46]

    1994.The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes

    Paul Solomon. 1994.The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. Academic Press, London

    1994

  34. [47]

    Ruiwei Xiao, Xinying Hou, and John Stamper. 2024. Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices. In Extended Abstracts of the 2024 CHI Conference. ACM, New York. https://doi.org/ 10.1145/3613905.3650937

    2024

  35. [48]

    Jacqueline Whalley, Amber Settle, and Andrew Luxton-Reilly. 2023. A Think- Aloud Study of Novice Debugging.ACM Trans. Comput. Educ.23, 2 (June 2023), 28:1–28:38. https://doi.org/10.1145/3589004

    2023 doi

  36. [2024]

    InProceedings of the 24th Koli Calling International Conference on Com- puting Education Research

    Novice Learners of Programming and Generative AI - Prior Knowledge Matters. InProceedings of the 24th Koli Calling International Conference on Com- puting Education Research. ACM, Koli Finland, 1–2. https://doi.org/10.1145/ 3699538.3699580

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.