Pith. sign in

REVIEW 3 major objections 4 minor 64 references

Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pair programming with an AI collaborator outperformed solo AI programming in a semester-long classroom study.

desk verdict Worth a careful read, but the headline PAI-over-SAI claim rests on an independence-violating test and an overreaching abstract. read the letter →

arxiv 2505.08119 v1 pith:2D7UJ77D submitted 2025-05-12 cs.HC

Wenhan Lyu , Yimeng Wang , Yifan Sun , Yixuan Zhang This is my paper · ORCID
classification cs.HC
keywords generativeAIlargelanguagemodelspairprogrammingcomputerscienceeducationhuman-AIcollaborationclassroomstudylearningoutcomesassignmentperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a generative-AI tool can act as a pair-programming partner rather than a mere coding aid, and whether that changes what students gain from the activity. In a semester-long undergraduate web-development course, 39 students completed six in-class assignments under three randomly ordered conditions: traditional two-human pair programming, pair programming with a generative-AI partner alongside the human pair, and solo programming with generative AI. The authors report that assignment scores were highest when students worked with both a human teammate and an AI tool, and lowest when they worked alone with the AI, with the best-versus-worst contrast statistically significant. They also find that students' attitudes toward LLMs' programming ability improved significantly after early collaboration, while students divided the labor: AI for syntax, concepts, and debugging; humans for idea exchange. The paper positions this as one of the first classroom evaluations of generative AI as an active collaborator in pair programming.

What carries the argument

The machinery that carries the argument is a within-subjects classroom protocol: each of 39 students completed six autograded assignments, one under each of the three conditions in randomized order within each stage, so every student serves as their own control. Performance differences are detected with nonparametric statistics—Kruskal-Wallis followed by Dunn's test—because assignment scores are not normally distributed; the qualitative arm uses a general inductive approach to code weekly reports and open-ended survey responses, explaining the quantitative ordering through students' own accounts. The LLM tools involved an inline-completion tool and a conversational tool, with students encouraged to use a provided web platform for conversational access.

What would settle it

Look at the released per-assignment scores and fit a model that respects repeated measures, such as a Friedman test on students' condition means or a mixed-effects model with student and assignment as random effects. If the PAI-versus-SAI contrast loses $p<0.05$, the headline ordering remains a descriptive average but the paper's claim of a significant benefit does not.

Watch

Extended reading notes

Core claim

The paper's central claim is that, across six autograded in-class assignments, students in the pair-programming-with-GenAI condition achieved the highest scores, students in traditional pair programming scored in the middle, and students programming solo with GenAI scored lowest. The paper reports this ordering as statistically meaningful: a Kruskal-Wallis test over all score observations gave $\chi^2 = 6.69$ ($p < 0.05$), and Dunn's post-hoc test found a significant difference between the best and worst conditions ($p < 0.05$). Median scores were 60 for human-plus-AI, 47.14 for traditional pairs, and 33.33 for solo-with-AI. The same data show that students' agreement that LLMs are good at programming and give high-quality answers rose significantly from the initial to the midterm survey. In their own words, students used LLM partners for syntax, concepts, and debugging, while using human partners for idea exchange and discussion, and treated AI as a useful but limited collaborator with context, hallucination, and outdated-knowledge problems.

Load-bearing premise

The finding assumes that treating each student's six assignment scores as separate independent data points is valid; if the scores from the same student are correlated, the reported statistical difference between human-plus-AI and solo-AI may not survive.

Editorial extensions

If this is right

  • If the ordering holds, classroom pair-programming designs should treat generative AI as a complement to a human partner, not as a substitute for the partner.
  • The lowest scores in solo-with-AI mean that giving students an LLM without a human collaborator is not automatically helpful, so curricula should pair AI access with structured support or collaboration.
  • Significant attitude gains after early collaboration suggest that structured early exposure can shape students' views of LLM capabilities, while the later rise in perceived need for guidance implies scaffolding matters.
  • Preference modeling shows that students who value pair programming's skill benefits prefer human-plus-AI setups, so pedagogical framing of pair programming's value may drive adoption more than raw tool performance.
  • Students' workflow friction with separate windows and their multi-screen adaptations point toward integrating AI tools into shared, multi-device coding environments rather than standalone chatbots.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the human partner in the PAI condition may function as a filter who catches and corrects AI-generated code; this can be tested by comparing pairs instructed to verify AI output with pairs that accept it without review.
  • Editorial inference: because scores come from autograders measuring output correctness, the PAI advantage may reflect task completion rather than durable learning; a delayed no-AI transfer assessment would distinguish acquisition from reliance.
  • Editorial inference: students' division of labor—AI for syntax and humans for ideas—suggests that future tools should preserve an idea-exchange space for human partners rather than positioning the AI as the sole teammate.
  • Editorial inference: the classroom environment may also shape the result, since students knew a human teammate was present and accountable; a replication with anonymous or remote human partners could test whether social accountability drives the PAI benefit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a semester-long mixed-methods study of 39 undergraduates in an advanced web-development course, comparing three conditions: traditional pair programming (PP), pair programming with generative AI tools (PAI), and solo programming with generative AI tools (SAI). Students completed six autograded assignments under these conditions in randomized order, with each student experiencing each condition once in Stage 2 and once in Stage 3. The quantitative analyses include Friedman/Wilcoxon tests for attitude change, a Kruskal-Wallis/Dunn analysis of assignment scores, and cumulative link mixed models for condition preference; qualitative analysis of weekly reports and open-ended responses examines collaboration dynamics. The central performance claim is that PAI students achieved the highest and SAI the lowest assignment scores, with a statistically significant PAI-versus-SAI difference, and the qualitative findings identify complementary roles of human and AI partners.

Significance. If the performance finding were robustly established, the study would be a valuable real-classroom contribution to the emerging literature on GenAI in CS education, and the qualitative material on role differentiation, multi-screen adaptation, and tool modality preferences offers concrete design guidance. The study's strengths include the semester-long authentic setting, randomized order of conditions within two stages, use of autograded assignments, and OSF materials with data and code. However, the headline performance claim rests on a statistical analysis that treats repeated, clustered observations as independent, so the core empirical result is not currently supported at the standard claimed confidence; the qualitative and attitudinal findings are less affected by this problem.

major comments (3)
  1. [Section 4.2.1] The comparative score analysis violates the independence assumption of the Kruskal-Wallis and Dunn tests. The 'full dataset' pools 234 observations (39 students x 6 assignments), but each student contributes two scores per condition and, in the two pair conditions, the two partners submit the same solution and receive the same autograder score; the effective number of independent units is closer to 39 students (or fewer, at pair level) than 234. The reported p<0.05 for the PAI-versus-SAI contrast is therefore not trustworthy as presented, and no effect sizes or confidence intervals are given. Because the abstract's 'highest in PAI / lowest in SAI' claim and the 'consistently outperformed' statement in Section 4.2.1 rest on this contrast, the analysis must be redone with a repeated-measures or mixed-effects model that includes student and pair random effects and assignment/period fixed effects.
  2. [Section 4.2.1 and Abstract] The full ordering PAI > PP > SAI is not supported by the reported pairwise results: only the PAI-versus-SAI Dunn contrast reached significance, and no significant differences are reported for PAI versus PP or PP versus SAI. The abstract's 'highest' and 'lowest' wording and the Section 4.2.2 takeaway that PAI 'outperformed both' PP and SAI overstate the evidence; the claim should be limited to the significant contrast unless the reanalysis supports the additional comparisons.
  3. [Section 4.2.1] The design also confounds condition with assignment and time. Although the order of conditions is randomized, each condition appears on different assignments for different students, and the six assignments are not exchangeable: topic difficulty and learning trajectory vary across the semester. The statement that analyzing the full dataset 'minimize[s] the risk of short-term fluctuations caused by topic difficulty' does not control for systematic assignment effects; a mixed-effects model with assignment fixed effects is needed to separate condition effects from assignment difficulty.
minor comments (4)
  1. [Figure 3] The figure caption identifies this as a Dunn's test result, but the figure appears to show only score distributions; please include the pairwise p-values, effect sizes, and the number of observations in the figure or its caption.
  2. [Section 3.2.2] The procedure states that students were 'randomly paired with one or two teammates' and that Stage 3 used 'the same human teammate(s) throughout'; please clarify whether each student had one fixed partner for all six assignments and how groups of three were handled in the pair conditions.
  3. [Section 4.2.1] The reported score values (e.g., means from 41.33 to 60 and medians of 60, 47.14, 33.33) are presented without stating the score scale; please state whether these are raw points, percentages, or normalized scores.
  4. [Section 3.2.2] Students were 'strongly encouraged' to use the research team's web-based tool exclusively for conversational LLM interactions; please clarify whether compliance was verified or logged, because variation in tools used could affect the comparison of conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical score comparison, and the paper's self-citations are contextual, not load-bearing.

full rationale

The paper's central claim (PAI highest, SAI lowest assignment scores) is a direct empirical comparison of measured autograder scores across three conditions, not a derived quantity or a prediction from fitted parameters. Section 4.2.1 reports a Kruskal-Wallis test and Dunn's post-hoc test on observed assignment scores; the tests are standard inference procedures and do not assume the conclusion they establish. No equation in the paper defines the conditions in terms of the outcome, and no fitted parameter is later renamed as a prediction. The only self-citations appear in the Related Work section (e.g., Lyu et al. [31] describing a prior LLM teaching-assistant study and Yue et al. [62] on MathVC); these are contextual descriptions and do not supply the performance comparison or any uniqueness/validity argument. The limitations section acknowledges small scale and generalizability concerns but does not reveal a circular step. The non-independence of repeated observations per student in the Kruskal-Wallis/Dunn analysis is a statistical validity concern, but it is not a form of circular reasoning under the specified rubric because the scoring data are observed rather than constructed from the hypothesis.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This empirical study does not introduce theoretical constants. Its central claims rest on standard assumptions about measurement validity, self-report accuracy, and the adequacy of the within-subject design to control confounds.

assumptions (4)
  • domain assumption Autograder scores are a valid measure of programming performance.
    The paper uses predefined autograders (Section 3.2.2) to grade all assignments; if the autograders do not equivalently capture quality across conditions, the score comparisons are compromised.
  • domain assumption Self-reported survey attitudes are accurate reflections of students' perceptions.
    RQ1 and preference analyses rely on Likert-scale self-reports (Sections 3.2, 3.3); social desirability or misunderstanding could bias results.
  • domain assumption The within-subject randomized ordering of conditions isolates the effect of condition from assignment difficulty and learning over time.
    The design randomizes order (Section 3.2.2) but assignment topics and difficulty are not counterbalanced or measured, so this assumption is strong.
  • domain assumption Student groups and pairings do not systematically bias condition assignments.
    Students were paired within self-formed final project groups (Section 3.2.2), so randomization was constrained and group composition may correlate with ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting." pith.science (2026). https://pith.science/paper/2D7UJ77D

@misc{pith2026250508119,
  author       = {Pith},
  title        = {Pith review of: Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2D7UJ77D}},
  note         = {Machine review of arXiv:2505.08119}
}
read the original abstract

Generative AI (GenAI), especially Large Language Models (LLMs), is rapidly reshaping both programming workflows and computer science education. Many programmers now incorporate GenAI tools into their workflows, including for collaborative coding tasks such as pair programming. While prior research has demonstrated the benefits of traditional pair programming and begun to explore GenAI-assisted coding, the role of LLM-based tools as collaborators in pair programming remains underexamined. In this work, we conducted a mixed-methods study with 39 undergraduate students to examine how GenAI influences collaboration, learning, and performance in pair programming. Specifically, students completed six in-class assignments under three conditions: Traditional Pair Programming (PP), Pair Programming with GenAI (PAI), and Solo Programming with GenAI (SAI). They used both LLM-based inline completion tools (e.g., GitHub Copilot) and LLM-based conversational tools (e.g., ChatGPT). Our results show that students in PAI achieved the highest assignment scores, whereas those in SAI attained the lowest. Additionally, students' attitudes toward LLMs' programming capabilities improved significantly after collaborating with LLM-based tools, and preferences were largely shaped by the perceived usefulness for completing assignments and learning programming skills, as well as the quality of collaboration. Our qualitative findings further reveal that while students appreciated LLM-based tools as valuable pair programming partners, they also identified limitations and had different expectations compared to human teammates. Our study provides one of the first empirical evaluations of GenAI as a pair programming collaborator through a comparison of three conditions (PP, PAI, and SAI). We also discuss the design implications and pedagogical considerations for future GenAI-assisted pair programming approaches.

Figures

Figures reproduced from arXiv: 2505.08119 by the authors.

Figure 1
Figure 1. Overall structure of our study. Students completed three assignments in both Stage 2 and Stage 3, each under a different [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (A) An overview of students’ average attitudes across five different opinion statements about programming with [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Non-parametric pairwise comparison test (Dunn’s [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 22 canonical work pages

  1. [1]

    Ibrahim Adeshola and Adeola Praise Adepoju. 2024. The opportunities and challenges of ChatGPT in education. Interactive Learning Environments 32, 10 (2024), 6159–6172. doi:10.1080/10494820.2023.2253858

    arXiv 2024

  2. [2]

    Gökçe Akçayır and Murat Akçayır. 2018. The flipped classroom: A review of its advantages and challenges. Computers & Education 126 (2018), 334–345. doi:10.1016/j.compedu.2018.07.021

    work page doi:10.1016/j.compedu.2018.07.021 2018

  3. [3]

    Prashant Baheti, Edward Gehringer, and David Stotts. 2002. Exploring the Efficacy of Distributed Pair Programming. In Extreme Programming and Agile Methods — XP/Agile Universe 2002, Don Wells and Laurie Williams (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 208–220. doi:10.1007/3-540-45672-4_20

    work page doi:10.1007/3-540-45672-4_20 2002

  4. [4]

    James, and Nadia Polikarpova

    Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang. 7, OOPSLA1, Article 78 (April 2023), 27 pages. doi:10.1145/3586030

    doi:10.1145/3586030 2023

  5. [5]

    Kent Beck. 2000. Extreme programming explained: embrace change . Addison- Wesley. doi:doi/10.5555/318762

    work page doi:10.5555/318762 2000

  6. [6]

    Christopher Bull and Ahmed Kharrufa. 2024. Generative Artificial Intelligence Assistants in Software Development Education: A Vision for Integrating Genera- tive Artificial Intelligence Into Educational Practice, Not Instinctively Defending Against It. IEEE Software 41, 2 (2024), 52–59. doi:10.1109/MS.2023.3300574

    arXiv 2024

  7. [7]

    Edgar Acosta Chaparro, Aybala Yuksel, Pablo Romero, and Sallyann Bryant. 2005. Factors Affecting the Perceived Effectiveness of Pair Programming in Higher Education.. In PPIG. 2

    work page 2005

  8. [8]

    Rune H. B. Christensen. 2023. ordinal—Regression Models for Ordinal Data . https: //CRAN.R-project.org/package=ordinal R package version 2023.12-4.1

    work page 2023

Show all 64 references
  1. [9]

    Lawrence Chung, Brian A Nixon, Eric Yu, and John Mylopoulos. 2012. Non- functional requirements in software engineering . Vol. 5. Springer Science & Busi- ness Media

    2012

  2. [10]

    Cursor. [n. d.]. https://www.cursor.com/ Accessed: 2025-02-19

    2025

  3. [11]

    Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. 2024. A complete survey on llm-based ai chatbots. arXiv preprint arXiv:2406.16937 (2024). doi:10.48550/arXiv.2406.16937

    2024 doi arXiv

  4. [12]

    Becker, and Brent N

    Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt Problems: A New Programming Exercise for the Generative AI Era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Educati...

    2024

  5. [13]

    Paul Denny, Stephen MacNeil, Jaromir Savelka, Leo Porter, and Andrew Luxton- Reilly. 2024. Desirable Characteristics for AI Teaching Assistants in Programming Education. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (Milan, Italy) (...

    2024

  6. [14]

    Christof Ebert and Panos Louridas. 2023. Generative AI for Software Practitioners. IEEE Software 40, 4 (2023), 30–38. doi:10.1109/MS.2023.3265877

    2023

  7. [15]

    Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Comput- ers, Software, and Applications Conference (COMPSAC) . 87...

    2023

  8. [16]

    Pauline Gibbons. 2002. Scaffolding language, scaffolding learning . Heinemann Portsmouth, NH

    2002

  9. [17]

    GitHub. [n. d.]. https://github.com/features/copilot Accessed: 2025-02-19

    2025

  10. [18]

    Brian Hanks, Sue Fitzgerald, Renée McCauley, Laurie Murphy, and Carol Zander

  11. [19]

    Jo E Hannay, Tore Dybå, Erik Arisholm, and Dag IK Sjøberg. 2009. The effective- ness of pair programming: A meta-analysis. Information and software technology 51, 7 (2009), 1110–1122. doi:10.1016/j.infsof.2009.02.001

    2009 doi

  12. [20]

    Hanna Hulkko and Pekka Abrahamsson. 2005. A multiple case study on the impact of pair programming on product quality. In Proceedings of the 27th International Conference on Software Engineering (St. Louis, MO, USA) (ICSE ’05). Association for Computing Machinery, New York, NY,...

    2005

  13. [21]

    Rasha Ahmad Husein, Hala Aburajouh, and Cagatay Catal. 2024. Large language models for code completion: A systematic literature review. Computer Standards & Interfaces (2024), 103917. doi:10.1016/j.csi.2024.103917

    2024

  14. [22]

    Saki Imai. 2022. Is GitHub copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machiner...

    2022

  15. [23]

    A Jo. 2023. The promise and peril of generative AI. Nature 614, 1 (2023), 214–216. doi:10.1038/d41586-023-00340-6

    2023 doi

  16. [24]

    Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2025. Explainable automated debugging via large language model-driven scientific debugging. Empirical Software Engineering 30, 2 (2025), 1–28. doi:10.1007/s10664-024-10594-x

    2025 doi

  17. [25]

    Ericson, David Weintrop, and Tovi Grossman

    Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Comp...

    2023

  18. [26]

    Ban It Till We Understand It

    Sam Lau and Philip Guo. 2023. From "Ban It Till We Understand It" to "Re- sistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as Chat- GPT and GitHub Copilot. In Proceedings of the 2023 AC...

    2023

  19. [28]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing System...

    2023

  20. [29]

    Chung Kwan Lo. 2023. What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences 13, 4 (2023), 410. doi:10.3390/ educsci13040410

    2023

  21. [30]

    Kim Man Lui and Keith CC Chan. 2006. Pair programming productivity: Novice– novice vs. expert–expert. International Journal of Human-computer studies 64, 9 (2006), 915–925. doi:10.1016/j.ijhcs.2006.04.010

    2006 doi

  22. [31]

    Wenhan Lyu, Yimeng Wang, Tingting Chung, Yifan Sun, and Yixuan Zhang. 2024. Evaluating the effectiveness of llms in introductory computer science education: A semester-long field study. In Proceedings of the Eleventh ACM Conference on Learning@ Scale. 63–74. doi:10.1145/365760...

    2024

  23. [32]

    Wenhan Lyu, Shuang Zhang, Tingting Chung, Yifan Sun, and Yixuan Zhang. 2025. Understanding the practices, perceptions, and (dis)trust of generative AI among instructors: A mixed-methods study in the U.S. higher education. Computers and Education: Artificial Intelligence 8 (202...

    2025

  24. [33]

    Qianou Ma, Tongshuang Wu, and Kenneth Koedinger. 2023. Is ai the better programming partner? human-human pair programming vs. human-ai pair pro- gramming. arXiv preprint arXiv:2306.05153 (2023). doi:10.48550/arXiv.2306.05153

    2023 doi arXiv

  25. [34]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Sy...

    2023

  26. [35]

    Atharva Mehta, Nipun Gupta, Aarav Balachandran, Dhruv Kumar, Pankaj Jalote, et al. 2023. Can chatgpt play the role of a teaching assistant in an introductory programming course? arXiv preprint arXiv:2312.07343 (2023). doi:10.48550/arXiv. 2312.07343

    2023 doi arXiv

  27. [36]

    Microsoft. [n. d.]. https://marketplace.visualstudio.com/items?itemName=MS- vsliveshare.vsliveshare Accessed: 2025-02-19

    2025

  28. [37]

    Matija Mikac, Miroslav Horvatić, Robert Logožar, and Emil Dumić. 2024. Chat- GPT in Education-Use Cases in an Introductory Web Programming Course. In INTED2024 Proceedings. IATED, 3173–3182. doi:10.21125/inted.2024.0853

    2024

  29. [38]

    Ogle, Jason C

    Derek H. Ogle, Jason C. Doll, A. Powell Wheeler, and Alexis Dinno. 2025. FSA: Simple Fisheries Stock Assessment Methods. https://CRAN.R-project.org/package= FSA R package version 0.9.6

    2025

  30. [39]

    OpenAI. [n. d.]. https://chatgpt.com/ Accessed: 2025-02-19

    2025

  31. [40]

    Zhang, Mark Harman, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. 34, 2, Article 42 (Jan. 2025), 28 pages. doi:10.1145/3697010

    2025 doi

  32. [41]

    Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, To- bias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. In Proceedings of the 2023 ACM Conference on In...

    2023

  33. [42]

    Becker, Bailey Kimmel, Jared Wright, and Ben Briggs

    James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randri- anasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. In Proceedings of the 2024 ACM Confere...

    2024

  34. [43]

    Nermin Punar Özçelik and Gonca Yangın Ekşi. 2024. Cultivating writing skills: the role of ChatGPT as a learning assistant—a case study. Smart Learning Envi- ronments 11, 1 (2024), 10. doi:10.1186/s40561-024-00296-8

    2024 doi

  35. [44]

    Fardin Ahsan Sakib, Saadat Hasan Khan, and A. H. M. Rezaul Karim. 2024. Ex- tending the Frontier of ChatGPT: Code Generation and Debugging. In 2024 International Conference on Electrical, Computer and Energy Technologies (ICECET . L@S ’25, July 21–23, 2025, Palermo, Italy Wenh...

    2024

  36. [45]

    S Saroja and S Haseena. 2023. Functional and Non-Functional Requirements in Agile Software Development. Agile Software Development: Trends, Challenges and Applications (2023), 71–86

    2023

  37. [46]

    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 27–43. doi:10.1145/3...

    2022

  38. [47]

    Andreas Scholl and Natalie Kiesler. 2024. How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course. arXiv preprint arXiv:2407.20792 (2024). doi:10.48550/arXiv.2407.20792

    2024 doi arXiv

  39. [48]

    Andreas Scholl, Daniel Schiffner, and Natalie Kiesler. 2024. Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT. arXiv preprint arXiv:2405.19132 (2024). doi:10.48550/arXiv.2405.19132

    2024 doi arXiv

  40. [49]

    Carlos Alexandre Gouvea da Silva, Felipe Negrelle Ramos, Rafael Veiga de Moraes, and Edson Leonardo dos Santos. 2024. ChatGPT: Challenges and benefits in software programming for higher education. Sustainability 16, 3 (2024), 1245. doi:10.3390/su16031245

    2024 doi

  41. [50]

    Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D

    Jiao Sun, Q. Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D. Weisz. 2022. Investigating Explainability of Generative AI for Code through Scenario-based Design. In Proceedings of the 27th International Conference on Intelligent Use...

    2022

  42. [51]

    Jinbo Tan, Lei Wu, and Shanshan Ma. 2024. Collaborative dialogue patterns of pair programming and their impact on programming self-efficacy and coding performance. British Journal of Educational Technology 55, 3 (2024), 1060–1081. doi:10.1111/bjet.13412

    2024 doi

  43. [52]

    R Core Team. 2024. R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project. org/

    2024

  44. [53]

    The Svelte team. [n. d.]. Svelte 5 is alive. https://svelte.dev/blog/svelte-5-is-alive Accessed: 2025-02-19

    2025

  45. [54]

    ChatGPT is the companion, not enemies

    Mark Feng Teng. 2024. “ChatGPT is the companion, not enemies”: EFL learners’ perceptions and experiences in using ChatGPT for feedback in writing.Computers and Education: Artificial Intelligence 7 (2024), 100270. doi:10.1016/j.caeai.2024. 100270

    2024 doi

  46. [55]

    Emily Theophilou, Cansu Koyutürk, Mona Yavari, Sathya Bursic, Gregor Don- abauer, Alessia Telari, Alessia Testa, Raffaele Boiano, Davinia Hernandez-Leo, Martin Ruskov, et al. 2023. Learning to prompt in the classroom to understand AI limits: a pilot study. In International Con...

    2023 doi

  47. [56]

    David R Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation 27, 2 (2006), 237–246. doi:10.1177/ 1098214005283748

    2006

  48. [57]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, ...

    2022

  49. [58]

    Tianjia Wang, Daniel Vargas Díaz, Chris Brown, and Yan Chen. 2023. Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 92–102. do...

    2023

  50. [59]

    Upchurch

    Laurie Williams and Richard L. Upchurch. 2001. In support of student pair- programming. In Proceedings of the Thirty-Second SIGCSE Technical Symposium on Computer Science Education (Charlotte, North Carolina, USA) (SIGCSE ’01). Association for Computing Machinery, New York, NY...

    2001

  51. [60]

    Bai, Robert Tairas, and Yu Huang

    Yuankai Xue, Hanlin Chen, Gina R. Bai, Robert Tairas, and Yu Huang. 2024. Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education...

    2024

  52. [61]

    Yi-Miao Yan, Chuang-Qi Chen, Yang-Bang Hu, and Xin-Dong Ye. 2025. LLM- based collaborative programming: impact on students’ computational thinking and self-efficacy. Humanities and Social Sciences Communications 12, 1 (2025), 1–12. doi:10.1057/s41599-025-04471-1

    2025 doi

  53. [62]

    Murong Yue, Wenhan Lyu, Wijdane Mifdal, Jennifer Suh, Yixuan Zhang, and Ziyu Yao. 2024. Mathvc: An llm-simulated multi-character virtual classroom for mathematics education. arXiv preprint arXiv:2404.06711 (2024)

    2024

  54. [63]

    Cynthia Zastudil, Magdalena Rogalska, Christine Kapp, Jennifer Vaughn, and Stephen MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. In 2023 IEEE Frontiers in Education Conference (FIE) . 1–9. doi:10.1109/FIE58773.2023.10343467

    2023

  55. [64]

    Chunpeng Zhai, Santoso Wibowo, and Lily D Li. 2024. The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: a systematic review.Smart Learning Environments 11, 1 (2024), 28. doi:10.1186/s40561-024-00316-7

    2024 doi

  56. [2011]

    Computer Science Education 21, 2 (2011), 135–173

    Pair programming in education: A literature review. Computer Science Education 21, 2 (2011), 135–173. doi:10.1080/08993408.2011.579808

    2011

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.