Pith. sign in

REVIEW 4 major objections 4 minor 37 references

A controlled 24-student study finds ChatGPT can sharply raise coding-task performance without producing corresponding knowledge gains, especially for beginners.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:49 UTC pith:UJQQ4GBU

load-bearing objection ChatGPT boosts task completion but not measured explanation-quality; the knowledge-gain conclusion is undercut by a single post-task measure. the 4 major comments →

arxiv 2511.13271 v2 pith:UJQQ4GBU submitted 2025-11-17 cs.SE cs.AIcs.IR

Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming

Rufeng Chen , Shuaishuai Jiang , Jiyun Shen , AJung Moon , Lili Wei This is my paper
classification cs.SE cs.AIcs.IR
keywords generative AI in educationcomputing educationChatGPTknowledge gainstask performanceprogramming learnershuman-AI interactioncontrolled user study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that using ChatGPT to solve a programming task can be good for completing the task and bad for learning from it. In a controlled experiment, 24 undergraduates watched a short video on the dynamic sliding window technique, then solved a programming problem either with ChatGPT or with conventional online resources. Beginner programmers with ChatGPT passed far more automated tests (about 411 of 500 vs 81 without), yet most were rated weak when asked to explain their solution and its connection to the concept. Intermediate programmers performed the same with or without ChatGPT, but those without it showed stronger understanding. The authors conclude that task performance and knowledge gains can decouple, and that students need guidance to use GenAI as a learning tool, not just an answer generator.

Core claim

The paper's central claim is that using ChatGPT to generate complete solutions sharply improves task performance, especially for beginners, without producing comparable knowledge gains. Beginner ChatGPT users averaged 410.7 passed tests versus 80.7 for beginners without it (p = 0.015); complete-solution users averaged 495, and five of six passed all 500 tests—yet only one of the six showed strong understanding when asked to explain their approach and its link to the concept. Intermediate participants scored about 291 tests with or without ChatGPT, but those without ChatGPT were more often rated strong on solution and concept understanding. The authors read this as a decoupling of task comple

What carries the argument

The argument is carried by a 2x2 between-subjects experiment (beginner/intermediate proficiency x with/without ChatGPT) built around a single programming task—applying the dynamic sliding window technique—with three measurement instruments: a 500-case automated test suite for task performance; two open-ended post-task questions (explain your approach; relate your solution to the concept) coded by multiple raters into strong/partial/weak understanding; and qualitative coding of chat logs into interaction patterns such as complete solution generation, stepwise code generation, code explanation, API usage, and concept explanation. The load-bearing comparison is between students who generated co

Load-bearing premise

The 'no knowledge gains' conclusion rests on coding two open-ended survey answers given minutes after the task into strong/partial/weak, with no pre-test and no delayed retention measure; if explanation quality is confounded with verbal skill, time pressure, or motivation, the learning deficit could be overstated, and the single-task design limits how far the results generalize.

What would settle it

Give complete-solution ChatGPT users and non-ChatGPT users a pre-test before the learning session and a delayed transfer test (a new problem requiring the same dynamic sliding window concept) days later; if ChatGPT users score at least as well as non-ChatGPT users on the delayed test, the claim that AI assistance does not produce knowledge gains is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Assessments that score only final code will overstate learning for AI-assisted beginners; adding explanation or transfer tasks changes the picture.
  • Beginners who use ChatGPT for complete solutions may need interventions that force engagement with generated code, such as requiring an explanation before submission.
  • Intermediate students may learn more from conventional resources than from ChatGPT as currently used, suggesting proficiency-dependent guidance.
  • Interaction patterns (concept explanation requests, stepwise code generation) are observable early and could be used to detect and redirect over-reliance.
  • Instructor guidance should aim for a middle path: neither banning GenAI nor allowing unguided use, because both extremes in the data produced weak knowledge gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling generalizes, standardized coding assessments administered with AI access will increasingly measure prompt-and-accept skill rather than programming skill, and credential value may shift accordingly.
  • A natural extension is a delayed retention or transfer test; the paper's own logic predicts complete-solution users would show little or no advantage, which would strengthen the central claim and could be tested with minimal added cost.
  • The cohort-timing observation suggests a generational effect: students first exposed to GenAI before building programming foundations may develop different help-seeking habits, so the beginner/intermediate difference may widen over time unless curricula adapt.
  • The data hint that requesting concept explanations correlates with understanding (only one intermediate did), so a teachable intervention—prompting learners to ask 'why' before 'how'—is a concrete, testable design the paper leaves implicit.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a controlled 2×2 experiment with 24 undergraduates (beginner/intermediate × with/without ChatGPT) solving a dynamic sliding-window programming task. It measures task performance via 500 Randoop-generated tests, 'knowledge gains' via post-task open-ended explanations coded as strong/partial/weak, and interaction patterns via chat-history coding. The main claims are that ChatGPT access substantially raises beginners' task performance (410.7 vs. 80.7 passed tests, p=0.015) but not their conceptual understanding; that intermediates perform similarly with or without ChatGPT but show stronger understanding without it; and that both over-reliance and under-use of GenAI are associated with weaker learning outcomes. The paper concludes with practical recommendations for students and educators.

Significance. If the results hold, the paper makes a useful contribution to the computing-education debate: it provides a controlled demonstration that task completion and conceptual learning can decouple when students use generative AI, and it contrasts two proficiency levels within one design. Strengths include the randomized between-subjects design, the use of an automated 500-test oracle, inter-rater reliability for the qualitative coding (Cohen's κ=0.81 and 0.84), and the public release of materials, data, and scripts. However, the statistical evidence is thin (six participants per cell with no variance/effect-size reporting), and the 'knowledge gains' measure is a single post-task self-explanation with no pre-test, no delayed test, and no transfer task. The central learning-related claims are therefore more tentative than the text suggests.

major comments (4)
  1. [IV-C, Table I] The headline performance result (410.7 vs. 80.7, p=0.015) is reported as group means only, with no standard deviations, confidence intervals, or effect sizes. With n=6 per cell, a single outlier can dominate the mean, and the p-value alone does not indicate the magnitude or robustness of the difference. Please report full distributions, CIs, and effect sizes (e.g., Hedges' g), and state which test was used and whether any multiple-comparison correction was applied for the four-group comparison. The same issue applies to the Complete vs. Stepwise comparison in Section V-A (p=0.032, n=12).
  2. [III-B, RQ3] The label 'knowledge gains' is not supported by the measurement. Participant understanding is assessed only after the task, via two open-ended survey questions; there is no pre-session baseline of conceptual knowledge, no delayed retention measure, and no transfer task. Consequently, the data can show post-task explanation quality, but not a change in knowledge. Findings 1 and 2 ('does not lead to statistically significant improvements in knowledge gains'; 'it did not improve their understanding') overstate what the evidence supports. Please either add a pre-test and delayed measure, or reframe the claims as 'post-task explanation quality' and temper the causal language.
  3. [III-B, RQ3 coding] The two authors who coded the open-ended responses were not blind to condition, and the rubric itself treats 'Let ChatGPT write the solution' as Weak Understanding. This makes the outcome measure partially dependent on the exact usage strategy being studied, which risks a tautological association. Cohen's kappa establishes inter-rater reliability, not construct validity. Please add blind coding (e.g., a coder unaware of condition) or, failing that, explicitly analyze and report how the rubric's dependence on the interaction strategy affects the knowledge-gain conclusions.
  4. [V-B, Finding 3] Finding 3 claims that differences in usage intent are influenced by the timing of GenAI exposure relative to programming experience. This narrative is inferred from ChatGPT's release date (late 2022) and cohort demographics; no participant-level data on first exposure or prior resource use are reported. This is a plausible hypothesis but it is not measured or tested. Please present it as a hypothesis for future work, or add the missing per-participant measures.
minor comments (4)
  1. [IV-C] Figure 3 is difficult to read in the provided version; the five score-range bins are not clearly separated by group. Consider a table or a clearer stacked visualization.
  2. [IV-A] The demographics paragraph says most participants reported 1–2 years of coding experience (14/24), but the beginner/intermediate classification criteria allow '3+ years' for intermediate. Please clarify whether self-reported 'years of coding' is consistent with the course-based classification, since the two criteria could in principle conflict.
  3. [V-A] The p=0.032 for Complete vs. Stepwise Code Generation is reported without naming the statistical test or providing any measure of effect size. Please add the test name and a confidence interval or effect size.
  4. [V-B] The sentence 'half of Intermediate w/o ChatGPT got only partial understanding, and the other half were categorized as weak' describes the Intermediate w/ ChatGPT group according to Figure 4. Please correct the group label to avoid confusion.

Circularity Check

0 steps flagged

No significant circularity: empirical study with separately measured treatment, performance, and understanding outcomes.

full rationale

This paper is a controlled user experiment, not a derivation or modeling exercise, so the circularity patterns do not apply. The central comparisons—ChatGPT access versus conventional resources (RQ2), knowledge-gain ratings from post-task open-ended explanations (RQ3), and interaction patterns from chat histories (RQ4)—use outcome measures that are collected independently of the treatment assignment. Task performance is measured by Randoop-generated test cases, not by any quantity fitted to the ChatGPT condition. Knowledge-gain coding is based on participants' written explanations and code, with inter-rater reliability reported (Cohen's kappa 0.81 and 0.84); although the measure captures post-task explanation quality rather than a pre-post gain, that is a construct-validity limitation, not a circular reduction. The paper does not fit parameters and then call them predictions, nor does it invoke a self-citation chain as the load-bearing justification for its findings. References to the authors' own materials are limited to the GitHub repository of study artifacts and do not support any theoretical premise. The interpretation that ChatGPT improves task completion more than conceptual understanding is an empirical claim, and its evidentiary weakness (e.g., no delayed retention test) belongs to correctness or validity risk, not circularity. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No fitted constants, invented entities, or mathematically constructed quantities appear. The paper is an empirical user study whose conclusions rest on measurement assumptions (proficiency grouping, explanation-based knowledge coding, self-selected strategies) rather than on derivation or free parameters.

axioms (5)
  • domain assumption Intermediate/expert classification by course enrollment or 3+ years of coding experience is a valid proxy for programming proficiency.
    Section III: participants are split into beginner/intermediate based on year in program or years coding; all proficiency comparisons depend on this grouping being meaningful and roughly balanced.
  • domain assumption Post-task open-ended explanations measure knowledge gain without a pre-test or delayed retention check.
    Section III-B RQ3: knowledge gains are coded as strong/partial/weak from two survey answers; the central 'no knowledge gain' conclusion rests on this proxy being valid.
  • domain assumption Participants had no prior knowledge of the Dynamic Sliding Window concept.
    Section III-A: the authors state they verified none of the participants had encountered the concept, so post-task understanding is attributed to the experimental session.
  • domain assumption Randoop-generated test outcomes from a correct reference solution are a valid measure of task performance.
    Section III-B RQ2: 500 randomly generated tests are used to score solutions; the assumption is that these tests capture meaningful correctness rather than coincidental edge cases.
  • ad hoc to paper The timing-of-exposure narrative (Finding 3) is inferred from ChatGPT's release date and cohort demographics, not measured per participant.
    Section V-B: the explanation that beginners encountered GenAI before building programming foundations is inferred from ChatGPT availability and course level; no individual-level data on when each student first used GenAI or started programming is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 23617 in / 9861 out tokens · 92554 ms · 2026-08-03T21:49:24.317747+00:00 · methodology

0 comments
read the original abstract

The rise of Generative AI (GenAI) tools like ChatGPT has created new opportunities and challenges for computing education. Existing research has primarily focused on GenAI's ability to complete educational tasks and its impact on student performance, often overlooking its effects on knowledge gains. In this study, we investigate how GenAI assistance compares to conventional online resources in supporting knowledge gains across different proficiency levels. We conducted a controlled user experiment with 24 undergraduate students of two different levels of programming experience (beginner, intermediate) to examine how students interact with ChatGPT while solving programming tasks. We analyzed task performance, conceptual understanding, and interaction behaviors. Our findings reveal that generating complete solutions with GenAI significantly improves task performance, especially for beginners, but does not consistently result in knowledge gains. Importantly, usage strategies differ by experience: beginners tend to rely heavily on GenAI toward task completion often without knowledge gain in the process, while intermediates adopt more selective approaches. We find that both over-reliance and minimal use result in weaker knowledge gains overall. Based on our results, we call on students and educators to adopt GenAI as a learning rather than a problem solving tool. Our study highlights the urgent need for guidance when integrating GenAI into programming education to foster deeper understanding.

Figures

Figures reproduced from arXiv: 2511.13271 by AJung Moon, Jiyun Shen, Lili Wei, Rufeng Chen, Shuaishuai Jiang.

Figure 1
Figure 1. Figure 1: Participant Demographics engineering. 20 participants (83.3%) indicated that they rarely or never used LeetCode. In contrast, ChatGPT was widely used, with 17 participants (70.8%) coding with it at least weekly. B. User Perceptions of ChatGPT (RQ1) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: User Perceptions of ChatGPT TABLE I: Mean Number of Passed Tests per Group Group Mean Passed Tests Beginner w/ ChatGPT 410.7 Beginner w/o ChatGPT 80.7 Intermediate w/ ChatGPT 291.0 Intermediate w/o ChatGPT 290.8 All Participants 268.3 0 1 100 100 400 400 499 500 Number of Passed Tests 0 2 4 Number of Participants Distribution of Passed Tests by Group Group Beginner w/ ChatGPT Beginner w/o ChatGPT Intermedi… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of Passed Tests by Group [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Participants’ understanding levels on solution and concept. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

  1. [1]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, and W. Zaremba, “Evaluating large language models trained on code,” 07 2021

    2021

  2. [2]

    Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” inExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’22. New York, NY , USA: Association for Computing Machinery,

    2022

  3. [3]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, A. Babiker, N. Sch ¨arli, A. Chowdhery, P. Mansfield, D. Demner-Fushman, B. Ag ¨uera y Arcas, D. Webster, G. S. Corrado, Y . Matias, K. Chou, J. Gottweis, N. Tomasev, Y . Liu, A. Rajkomar, J. Barra...

    2023

  4. [4]

    Chatgpt for good? on opportunities and challenges of large language models for education,

    E. Kasneci, K. Sessler, S. K ¨uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G ¨unnemann, E. H ¨ullermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, “Chatgpt for good? on opportunities and challenges of large language m...

    2023

  5. [5]

    Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals,

    P. Mirowski, K. W. Mathewson, J. Pittman, and R. Evans, “Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals,” inProceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23. New York, NY , USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi-org.pr...

    arXiv 2023

  6. [6]

    A recipe for arbitrary text style transfer with large language models,

    E. Reif, D. Ippolito, A. Yuan, A. Coenen, C. Callison-Burch, and J. Wei, “A recipe for arbitrary text style transfer with large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2109.03910

    Pith/arXiv arXiv 2022

  7. [7]

    Generative disco: Text-to- video generation for music visualization,

    V . Liu, T. Long, N. Raw, and L. Chilton, “Generative disco: Text-to- video generation for music visualization,” 2023. [Online]. Available: https://arxiv.org/abs/2304.08551

    Pith/arXiv arXiv 2023

  8. [8]

    Experiences with remote examination formats in light of gpt-4,

    F. Dobslaw and P. Bergh, “Experiences with remote examination formats in light of gpt-4,” inProceedings of the 5th European Conference on Software Engineering Education, ser. ECSEE ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 220–225. [Online]. Available: https://doi.org/10.1145/3593663.3593695

    arXiv 2023

  9. [9]

    Evaluating chatgpt: Generative ai in ux design and web development pedagogy,

    E. York, “Evaluating chatgpt: Generative ai in ux design and web development pedagogy,” inProceedings of the 41st ACM International Conference on Design of Communication, ser. SIGDOC ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 197–201. [Online]. Available: https://doi.org/10.1145/3615335.3623035

    arXiv 2023

  10. [11]

    The robots are coming: Exploring the implications of openai codex on introductory programming,

    J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather, “The robots are coming: Exploring the implications of openai codex on introductory programming,” inProceedings of the 24th Australasian Computing Education Conference, ser. ACE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 10–19. [Online]. Available: https...

    arXiv 2022

  11. [12]

    My ai wants to know if this will be on the exam: Testing openai’s codex on cs2 programming exercises,

    J. Finnie-Ansley, P. Denny, A. Luxton-Reilly, E. A. Santos, J. Prather, and B. A. Becker, “My ai wants to know if this will be on the exam: Testing openai’s codex on cs2 programming exercises,” inProceedings of the 25th Australasian Computing Education Conference, ser. ACE ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 97–104. [Onl...

    arXiv 2023

  12. [13]

    Teaching cs50 with ai: Leveraging generative artificial intelligence in computer science education,

    R. Liu, C. Zenke, C. Liu, A. Holmes, P. Thornton, and D. J. Malan, “Teaching cs50 with ai: Leveraging generative artificial intelligence in computer science education,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 750–756. [Online]...

    arXiv 2024

  13. [14]

    Evaluating llm-generated worked examples in an introductory programming course,

    B. Jury, A. Lorusso, J. Leinonen, P. Denny, and A. Luxton- Reilly, “Evaluating llm-generated worked examples in an introductory programming course,” inProceedings of the 26th Australasian Computing Education Conference, ser. ACE ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 77–86. [Online]. Available: https://doi.org/10.1145/36362...

    arXiv 2024

  14. [15]

    Cheating in the age of generative ai: A high school survey study of cheating behaviors before and after the release of chatgpt,

    V . R. Lee, D. Pope, S. Miles, and R. C. Z ´arate, “Cheating in the age of generative ai: A high school survey study of cheating behaviors before and after the release of chatgpt,”Computers and Education: Artificial Intelligence, vol. 7, p. 100253, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2666920X24000560

    2024

  15. [16]

    My learnings from allowing large language models in introductory computer science classes,

    R. Bhalerao, “My learnings from allowing large language models in introductory computer science classes,” 03 2024, pp. 1574–1575

    2024

  16. [17]

    May we consult chatgpt in our human-computer interaction written exam? an experience report after a professor answered yes,

    A. P. Freire, P. C. F. Cardoso, and A. d. L. Salgado, “May we consult chatgpt in our human-computer interaction written exam? an experience report after a professor answered yes,” inProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, ser. IHC ’23. New York, NY , USA: Association for Computing Machinery,

  17. [18]

    Kogi: A seamless integration of chatgpt into jupyter environments for programming education,

    K. Kuramitsu, Y . Obara, M. Sato, and M. Obara, “Kogi: A seamless integration of chatgpt into jupyter environments for programming education,” inProceedings of the 2023 ACM SIGPLAN International Symposium on SPLASH-E, ser. SPLASH-E 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 50–59. [Online]. Available: https://doi.org/10.1145/3...

    arXiv 2023

  18. [19]

    How beginning programmers and code llms (mis)read each other,

    S. Nguyen, H. M. Babe, Y . Zi, A. Guha, C. J. Anderson, and M. Q. Feldman, “How beginning programmers and code llms (mis)read each other,” inProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, ser. CHI ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi-org.proxy3.library.mcgill.ca...

    arXiv 2024

  19. [20]

    An empirical study on usage and perceptions of llms in a software engineering project,

    S. Rasnayaka, G. Wang, R. Shariffdeen, and G. N. Iyer, “An empirical study on usage and perceptions of llms in a software engineering project,” inProceedings of the 1st International Workshop on Large Language Models for Code, ser. LLM4Code ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 111–118. [Online]. Available: https://doi.org...

    arXiv 2024

  20. [21]

    Analysis of student-llm interaction in a software engineering project,

    A. Naman, R. Shariffdeen, G. Wang, S. Rasnayaka, and G. N. Iyer, “Analysis of student-llm interaction in a software engineering project,” in2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), 2025, pp. 112–119

    2025

  21. [22]

    Examining-the-usage-of-generative-ai-models-in-student-learning,

    “Examining-the-usage-of-generative-ai-models-in-student-learning,”

  22. [24]

    Using large language models to enhance programming error messages,

    J. Leinonen, A. Hellas, S. Sarsa, B. Reeves, P. Denny, J. Prather, and B. A. Becker, “Using large language models to enhance programming error messages,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 563–569. [Online]. Available: ht...

    arXiv 2023

  23. [25]

    Evaluating the performance of code generation models for solving parsons problems with small prompt variations,

    B. Reeves, S. Sarsa, J. Prather, P. Denny, B. A. Becker, A. Hellas, B. Kimmel, G. Powell, and J. Leinonen, “Evaluating the performance of code generation models for solving parsons problems with small prompt variations,” inProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V . 1, ser. ITiCSE 2023. New York, NY , ...

    arXiv 2023

  24. [26]

    Github copilot ai pair programmer: Asset or liability?

    A. M. Dakhel, V . Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, Z. Ming, and Jiang, “Github copilot ai pair programmer: Asset or liability?” 2023. [Online]. Available: https://arxiv.org/abs/2206.15331

    Pith/arXiv arXiv 2023

  25. [27]

    Promptly: Using prompt problems to teach learners how to effectively utilize ai code generators,

    P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves, “Promptly: Using prompt problems to teach learners how to effectively utilize ai code generators,” 2023. [Online]. Available: https://arxiv.org/abs/2307.16364

    Pith/arXiv arXiv 2023

  26. [28]

    How do programming students use generative ai?

    C. Rahe and W. Maalej, “How do programming students use generative ai?”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715762

    doi:10.1145/3715762 2025

  27. [29]

    “it’s weird that it knows what i want

    J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ““it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers,”ACM Trans. Comput.- Hum. Interact., vol. 31, no. 1, Nov. 2023. [Online]. Available: https://doi-org.proxy3.library.mcgill.ca...

    doi:10.1145/3617367 2023

  28. [30]

    Studying the effect of ai code generators on supporting novice learners in introductory programming,

    M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman, “Studying the effect of ai code generators on supporting novice learners in introductory programming,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23. New York, NY , USA: Association for Computing Machinery, 2023. [Online]. Av...

    arXiv 2023

  29. [31]

    Generative ai in programming education: Evaluating chatgpt’s effect on computational thinking,

    M. Sivasakthi and A. Meenakshi, “Generative ai in programming education: Evaluating chatgpt’s effect on computational thinking,”SN Computer Science, vol. 6, no. 5, p. 541, 2025. [Online]. Available: https://doi.org/10.1007/s42979-025-04051-9

    work page doi:10.1007/s42979-025-04051-9 2025

  30. [32]

    Outcomes, perceptions, and interaction strategies of novice programmers studying with chatgpt,

    J. Penney, P. Acharya, P. Hilbert, P. Parekh, A. Sarma, I. Steinmacher, and M. A. Gerosa, “Outcomes, perceptions, and interaction strategies of novice programmers studying with chatgpt,” inProceedings of the 7th ACM Conference on Conversational User Interfaces, ser. CUI ’25. New York, NY , USA: Association for Computing Machinery, 2025. [Online]. Availabl...

    arXiv 2025

  31. [33]

    Chatgpt vs. stack overflow: An exploratory comparison of programming assistance tools,

    J. Liu, X. Tang, L. Li, P. Chen, and Y . Liu, “Chatgpt vs. stack overflow: An exploratory comparison of programming assistance tools,” in2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), 2023, pp. 364–373

    2023

  32. [34]

    Randoop - automatic unit test generation for java,

    “Randoop - automatic unit test generation for java,” 2025. [Online]. Available:{https://randoop.github.io/randoop/}

    2025

  33. [35]

    Openai - introducing chatgpt,

    “Openai - introducing chatgpt,” 2022. [Online]. Available:{https: //openai.com/index/chatgpt/}

    2022

  34. [36]

    Combiner l’analyse statique et la g ´en´eration de tests pour une assurance qualit´e am ´elior´ee : Une exploration des probl `emes de compatibilit ´e android,

    “Combiner l’analyse statique et la g ´en´eration de tests pour une assurance qualit´e am ´elior´ee : Une exploration des probl `emes de compatibilit ´e android,” 2025

    2025

  35. [2022]

    Available: https://doi.org/10.1145/3491101.3519665

    [Online]. Available: https://doi.org/10.1145/3491101.3519665

    arXiv

  36. [2024]

    Available: https://doi.org/10.1145/3638067.3638100

    [Online]. Available: https://doi.org/10.1145/3638067.3638100

    arXiv

  37. [2025]

    Available:{https://github.com/RafferyChen/ Examining-the-Usage-of-Generative-AI-Models-in-Student-Learning

    [Online]. Available:{https://github.com/RafferyChen/ Examining-the-Usage-of-Generative-AI-Models-in-Student-Learning. git}