Pith. sign in

REVIEW 4 major objections 5 minor 50 references

The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GitHub Copilot makes students faster on legacy-code tasks, at the cost of understanding.

desk verdict A genuinely useful brownfield Copilot study with a transparent but load-bearing order confound; the process analysis is citable, the performance claims are not causal. read the letter →

arxiv 2506.10051 v1 pith:5EPBGEET submitted 2025-06-11 cs.SE

Md Istiak Hossain Shihab , Christopher Hundhausen , Ahsun Tariq , Summit Haque , Yunhan Qiao , Brian Mulanda This is my paper
classification cs.SE
keywords GitHubCopilotbrownfieldprogramminglegacycodeAI-assistedcomputingeducationstudentprogrammersempiricalsoftwareengineeringprocessanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether GitHub Copilot helps undergraduate computing students do brownfield programming, meaning adding new features to an unfamiliar legacy code base written by someone else, and what changes in how they work. In a within-subjects experiment, ten students each implemented two nearly identical features in a legacy web application, once with Copilot and once without. The authors report that with Copilot, students completed the common first task about 35% faster, passed more automated tests (6.30 vs. 4.20 on average), spent less time typing code and searching the web, and shifted from a read-understand-implement workflow to a prompt-view-response-implement cycle. They also report that students who accepted Copilot output in small, selective pieces did better than those who pasted whole chunks, and that students worried they did not understand why Copilot's code worked. The point of the paper is that these productivity gains in brownfield settings are real and large, but they create a new set of educational demands around evaluating and integrating generated code.

What carries the argument

The load-bearing apparatus is a matched-pair brownfield task design. Two feature additions to the same 3,818-line legacy front-end application, the Add Distance feature and the Add Picture feature, were engineered to be highly similar in cognitive demand, implementation size, and structure, each decomposing into three parallel tasks (UI construction, interactive behavior, and persistence) and scored by a shared-style automated test suite. Around that task pair, the authors built a second-by-second video coding scheme distinguishing eleven primary activity categories (write code, view code, view web, view response, and so on), verified with inter-rater reliability, plus a secondary scheme for how code was written (manual entry, accepting a suggestion, pasting chat output, or modifying a suggestion). The comparisons are carried by non-parametric statistics with a false-discovery-rate correction, permutation-based chi-square tests, and a Markov transition analysis of activity sequences, so the central evidence is the contrast between two well-matched tasks in the same code base, measured at the level of single-second behavior.

What would settle it

Run the same two feature tasks with treatment order counterbalanced across ten or more students: assign half the participants to implement the first feature with Copilot and the second without, and the other half in the reverse order. If the 35% time advantage and the test-score gap shrink or reverse once task order is controlled, the reported Copilot effect is largely an artifact of familiarity with the code base and task format.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that GitHub Copilot materially improves undergraduate performance on brownfield programming tasks: a 34.9% reduction in completion time on the first task (W = 7.0, p = 0.04, r = 0.66) and a rise in average tests passed from 4.20 to 6.30 (W = 1.5, p = 0.01, r = 0.80), which the authors interpret as significantly more progress toward complete feature implementations. The behavioral analysis shows that time spent writing code fell from about 30% to about 20% of session time, that viewing web content fell from about 13.5% to about 1.9%, and that prompting and reading Copilot responses together took about 27% of session time. A Markov transition analysis found that the dominant workflow became prompt, view response, implement, replacing a read, understand, implement sequence, and the difference between the two workflow patterns was statistically significant. The authors further claim that selective, suggestion-level adoption of Copilot output, rather than wholesale pasting, was associated with higher performance, and that participants' own reports reveal a tension between efficiency and understanding: students completed more but often did not know why the generated code worked or where it belonged.

Load-bearing premise

The results assume that the learning that happened during the no-Copilot condition was too small to matter, because every participant tried Copilot only after already working through the code base once without it.

Editorial extensions

If this is right

  • If the 35% time saving holds beyond this sample, brownfield feature work by students becomes an activity where the bottleneck shifts from writing code to deciding which generated code to accept and how to integrate it.
  • Automated test suites can absorb this workflow: students ran tests more often with Copilot, so assessment based on functional correctness remains feasible even when a large fraction of the code is machine-generated.
  • The observed prompt-to-view-response-to-implement cycle implies that assignments that never ask students to read and understand the existing code will let them skip the comprehension step entirely, so task design must force that step if understanding is a goal.
  • Students with lower performance spent more time pasting whole Copilot chat responses; if that association is causal, teaching students to ask for smaller suggestions and edit them could be a direct performance intervention.
  • Because students themselves report not understanding why suggestions work, courses that adopt Copilot should pair its use with exercises requiring explanation or modification of generated code rather than treating acceptance as completion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A counterbalanced replication, with half the students doing the Copilot feature first, would separate the tool's effect from the familiarity students gained in the no-Copilot condition; the paper did not counterbalance treatment order, so part of the advantage could in principle be practice on the code base.
  • An untested extension is whether the selective-adoption advantage is causal: an experiment that randomly assigns students to whole-paste versus suggestion-level interaction modes could determine whether granular use improves outcomes or merely correlates with better students.
  • The same measurement apparatus could be applied to other generative AI assistants or other programming languages to ask whether the workflow shift is specific to Copilot's autocomplete-plus-chat interface or a general property of AI-assisted programming.
  • If Copilot-style inline edit features become standard, the distinction between pasting chat output and accepting suggestions may blur; the one participant with access to the newer insertion feature used it heavily, so future versions of this study should standardize the tool version.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a within-subjects controlled experiment with 10 undergraduate students comparing brownfield programming task performance with and without GitHub Copilot. Participants completed two highly similar feature-implementation tasks on an unfamiliar legacy JavaScript/HTML/CSS web application: one without Copilot and one with Copilot, in that fixed order. The authors report statistically significant reductions in Task 1 completion time (34.9%, W=7.0, p=0.04, r=0.66) and increases in total tests passed (6.30 vs 4.20, W=1.5, p=0.01, r=0.80), along with behavioral coding showing shifts in time allocation, a prompt→view response→implement workflow, and differences in Copilot usage between higher and lower performers. Exit interviews revealed concerns about superficial understanding. The paper claims to provide the first empirical account of GenAI assistance in brownfield programming tasks.

Significance. If the causal interpretation were warranted, this would be a useful contribution to computing education research: it extends prior greenfield studies to brownfield settings, uses objective automated tests rather than self-report, applies FDR correction, reports effect sizes, and provides inter-rater reliability for the behavioral coding and public replication materials. However, the headline RQ1 claims are conditional because the unmitigated learning/order confound prevents causal attribution, and the RQ3 grouping is post-hoc. The contribution is therefore primarily descriptive and exploratory in its current form.

major comments (4)
  1. [Section 3.5 and 6.1] The fixed treatment order (No Copilot first for all 10 participants) confounds the treatment contrast with practice and codebase-familiarity effects. The 34.9% time reduction and 2.1-test improvement in Section 4.1 may therefore reflect learning rather than Copilot. Counterbalancing the two feature implementations does not remove this because the No Copilot task always came first; the authors acknowledge the unmitigated learning effect in Section 6.1, but the abstract and Section 5.1 nonetheless state causal conclusions ('Copilot enabled,' 'led to'). Please either add an empirical check of the learning-effect magnitude (e.g., compare task completion times across the three subtasks within each condition, or run a between-subjects control), or substantially weaken the causal phrasing throughout and frame the study as an exploratory comparison. This is the load-bearing issue for RQ1.
  2. [Section 4.1.1 and 4.1.2] The efficiency analysis is based only on Task 1 (the only task all participants completed), while the correctness analysis uses total tests passed across all three tasks. This asymmetry should be explained: for participants who did not finish Tasks 2 or 3, how is incomplete work scored, and does the time comparison omit censored data? If the main time result applies only to the first subtask, the phrase 'completed tasks 35% faster' in the abstract overstates the scope of the finding.
  3. [Section 4.3] The higher/lower performer split is defined by total tests passed in both conditions and then used to explain Copilot usage patterns. This makes the RQ3 analysis partly circular: total test performance includes the No Copilot condition and is itself affected by Copilot, so the 'wholesale pasting' behavior may be a marker of poorer outcomes rather than a cause. The natural-gap argument (13 vs 9 tests) is fragile with n=10, and the chi-square result (χ²=56.16, p<0.0001) is not corrected for multiple comparisons or for the post-hoc selection of the split. The manuscript already labels these as trends, but the headline in Section 7 ('Higher-performing students tend to be more selective...') goes beyond what this design can support; please present RQ3 as hypothesis-generating and temper the causal language.
  4. [Section 6.1] The version drift in the Copilot extension is a real threat to the process analysis in RQ2: one participant used Copilot Edits during 40% of their code-writing time, yet the behavioral results aggregate all 10 participants. While the authors disclose this, the potential magnitude of the effect is not assessed. At minimum, re-run the RQ2 code-writing analyses excluding or separately reporting that participant, and state the sensitivity of the conclusions to this single participant.
minor comments (5)
  1. [Abstract and Section 5.2] The abstract states '11% less time manually writing code' and '12% less time conducting web searches,' but Table 4 shows these are percentage-point differences (10.63 and 11.60 percentage points, respectively); the relative reductions are approximately 35% and 86%. Please correct the abstract and Section 5.2 to avoid misleading the reader.
  2. [Section 5.1] The statement '26.2% improvement in test pass rates' does not match the reported means (6.30 vs 4.20 is a 50% relative increase); please reconcile the numbers or explain the basis for the computed percentage.
  3. [Section 3.3] The sentence 'Oregon State University, a large research university with over 4,000 students enrolled its undergraduate computer science programs' is missing a preposition ('in'); please correct the typo.
  4. [Figure 1 and Section 4.1] The figure captions report p-values (e.g., 'p = 0.037' and 'p = 0.012') that differ from the values in the body text (p=0.04 and p=0.01). Please align the reported p-values consistently between the text, figures, and tables.
  5. [Table 4] The 'View Dev Tools (VD)' row reports p=0.250 with r=0.66; this combination is surprising given the large effect size, and the table does not explain how the effect size is computed. Please clarify the effect-size calculation or correct the value.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the performance and process findings are derived from directly measured task outcomes and behavioral observations, not from fitted inputs or self-referential definitions.

full rationale

The paper's central claims (RQ1 and RQ2) rest on directly measured dependent variables: task completion time and number of automated tests passed, plus second-by-second behavioral coding with reported inter-rater reliability. Statistics are non-parametric comparisons of observed distributions (e.g., 'W=7.0, p=0.04', 'W=1.50, p=0.01'), with no fitted parameters, no latent variables, and no equation whose output is defined by its input. The RQ3 higher/lower performer split is defined by total tests passed across both conditions and then compared against Copilot usage; this is a post-hoc descriptive association rather than a prediction or fitted model, and it does not feed back into the RQ1 performance comparison. The acknowledged lack of treatment-order counterbalancing ('the learning effect of the No Copilot treatment went unmitigated,' Section 6.1) is an internal validity threat and confound, not a circularity; it could weaken causal attribution but does not make any measured quantity equivalent to another by construction. No load-bearing self-citations or uniqueness theorems are invoked, and no known result is renamed as a new derivation. Accordingly, the derivation chain is self-contained with respect to the evidence reported in this paper.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about task equivalence, negligible learning effects, valid behavioral coding, and test-suite validity. There are no invented entities and no mathematical free parameters; the only hand-chosen numeric threshold is the post-hoc performance split.

free parameters (1)
  • higher/lower performer split threshold = gap between 13 and 9 tests passed
    Participants were partitioned into groups post hoc based on the gap in total tests passed (Section 4.3), a threshold chosen by inspection rather than predefined.
assumptions (4)
  • domain assumption The 'Add Distance' and 'Add Picture' features are similar enough in difficulty to serve as equivalent within-subjects tasks.
    Section 3.4.2 provides complexity metrics (lines of code, statements, variables) but does not prove equivalence; the comparison across conditions relies on this.
  • domain assumption Completing the No Copilot feature first does not materially change performance on the Copilot feature.
    Section 3.5 and 6.1: treatment order was not counterbalanced, so this assumption is load-bearing for the causal interpretation.
  • domain assumption Screen activity and cursor movement are valid proxies for participants' attention and programming processes.
    Section 6.1: 'categorical shifts were defined by cursor movements to new windows'; no eye tracking.
  • domain assumption The automated test suites are valid measures of whether a task requirement is correctly addressed.
    Section 3.4.3: tests verify UI rendering, behavior, and persistence; passing tests is equated with solution progress.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks." pith.science (2026). https://pith.science/paper/5EPBGEET

@misc{pith2026250610051,
  author       = {Pith},
  title        = {Pith review of: The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EPBGEET}},
  note         = {Machine review of arXiv:2506.10051}
}
read the original abstract

When graduates of computing degree programs enter the software industry, they will most likely join teams working on legacy code bases developed by people other than themselves. In these so-called brownfield software development settings, generative artificial intelligence (GenAI) coding assistants like GitHub Copilot are rapidly transforming software development practices, yet the impact of GenAI on student programmers performing brownfield development tasks remains underexplored. This paper investigates how GitHub Copilot influences undergraduate students' programming performance, behaviors, and understanding when completing brownfield programming tasks in which they add new code to an unfamiliar code base. We conducted a controlled experiment in which 10 undergraduate computer science students completed highly similar brownfield development tasks with and without Copilot in a legacy web application. Using a mixed-methods approach combining performance analysis, behavioral analysis, and exit interviews, we found that students completed tasks 35% faster (p < 0.05) and made 50% more solution progress p (< 0.05) when using Copilot. Moreover, our analysis revealed that, when using Copilot, students spent 11% less time manually writing code (p < 0.05), and 12% less time conducting web searches (p < 0.05), providing evidence of a fundamental shift in how they engaged in programming. In exit interviews, students reported concerns about not understanding how or why Copilot suggestions work. This research suggests the need for computing educators to develop new pedagogical approaches that leverage GenAI assistants' benefits while fostering reflection on how and why GenAI suggestions address brownfield programming tasks. Complete study results and analysis are presented at https://ghcopilot-icer.github.io/.

Figures

Figures reproduced from arXiv: 2506.10051 by the authors.

Figure 1
Figure 1. Box plots of participant task performance with [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Interaction pattern in engaging different activities while solving the problem [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Breakdown of code writing activities, showing the shift from almost exclusively manual code entry without Copilot [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Programming workflow networks comparing ac [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the Copilot interaction strategies of higher and lower performers [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 30 canonical work pages

  1. [1]

    GitHub Copilot

    2021. GitHub Copilot. GitHub. Available at https://github.com/features/copilot

    work page 2021

  2. [2]

    Stack Overflow Developer Survey 2025

    2025. Stack Overflow Developer Survey 2025. https://survey.stackoverflow.co/ 2024/

    work page 2025

  3. [3]

    Fahad H Alshammari. 2022. Trends in Intelligent and AI-Based Software Engineer- ing Processes: A Deep Learning-Based Software Process Model Recommendation Method.Computational intelligence and neuroscience2022, 1 (2022), 1960684

    work page 2022

  4. [4]

    Amazon. [n. d.]. Amazon WorkSpaces. https://aws.amazon.com/pm/workspaces/

    work page

  5. [5]

    Marcellin Atemkeng, Sisipho Hamlomo, Brian Welman, Nicole Oyetunji, Pouya Ataei, and Jean Louis KE Fendji. 2024. Ethics of Software Programming with Generative AI: Is Programming without Generative AI always radical?arXiv preprint arXiv:2408.10554(2024)

    arXiv 2024

  6. [6]

    Yoav Benjamini and Daniel Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency.Annals of Statistics29, 4 (2001), 1165–1188. doi:10.1214/aos/1013699998

    arXiv 2001

  7. [7]

    Aras Bozkurt. 2023. Generative artificial intelligence (AI) powered conversational educational agents: The inevitable paradigm shift.Asian Journal of Distance Education18, 1 (2023)

    work page 2023

  8. [8]

    Allan Collins, John Seely Brown, and Susan E Newman. 1989. Cognitive appren- ticeship: Teaching the crafts of reading, writing, and mathematics.Knowing, learning, and instruction: Essays in honor of Robert Glaser18 (1989), 32–42

    work page 1989

Show all 50 references
  1. [9]

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability?Journal of Systems and Software203 (2023), 111734

    2023

  2. [10]

    Fitsum Deriba, Ismaila Temitayo Sanusi, Oladele O Campbell, and Solomon Sun- day Oyelere. 2024. Computer Programming Education in the Age of Generative AI: Insights from Empirical Research. (2024)

    2024

  3. [11]

    Feathers

    Michael C. Feathers. 2004. Working Effectively with Legacy Code. (2004), 217. doi:10.1007/978-3-540-27777-4_42

    2004 doi

  4. [12]

    James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The robots are coming: Exploring the implications of openai codex on introductory programming. InProceedings of the 24th Australasian computing education conference. 10–19

    2022

  5. [13]

    E Frank and O Godwin. 2024. Enhancing Developer Productivity: A Study on GitHub Copilot’s Code Completion Capabilities. (2024)

    2024

  6. [14]

    Nicholas Gardella, Raymond Pettit, and Sara L Riggs. 2024. Performance, Work- load, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Genera- tion. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. 290–296

    2024

  7. [15]

    Nada Hashmi, Zhi Li, Salvatore Parise, and G Shankaranarayanan. 2024. Gener- ative AI’s impact on programming students: frustration and confidence across learning styles.Issues in Information Systems25, 3 (2024)

    2024

  8. [16]

    Jan H Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden, Cordell Burton Jr, Carson Powers, Fabio Massacci, Akond Rahman, Daniel Votipka, Heather Richter Lipford, et al. 2024. Using AI Assistants in Software Development: A Qualitative Study on Security Practices...

    2024 arXiv

  9. [17]

    Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability

    2011

  10. [18]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161(2023)

    2023 arXiv

  11. [19]

    Liang, Chenyang Yang, and Brad A

    Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2023. Understanding the Usability of AI Programming Assistants.ArXivabs/2303.17125 (2023). https: //api.semanticscholar.org/CorpusID:263870561

    2023 arXiv

  12. [20]

    Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering. 1–13

    2024

  13. [21]

    Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. 2023. Codehelp: Using large language models with guardrails for scalable support in program- ming classes. InProceedings of the 23rd Koli Calling International Conference on Computing Education Research. 1–11

    2023

  14. [22]

    Joe Llerena-Izquierdo, Johan Mendez-Reyes, Raquel Ayala-Carabajo, and Cesar Andrade-Martinez. 2024. Innovations in Introductory Programming Education: The Role of AI with Google Colab and Gemini.Education Sciences14, 12 (2024), 1330

    2024

  15. [23]

    Dastyni Loksa, Andrew J Ko, Will Jernigan, Alannah Oleson, Christopher J Mendez, and Margaret M Burnett. 2016. Programming, problem solving, and self- awareness: effects of explicit guidance. InProceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, ...

    2016

  16. [24]

    Lauren E Margulieux, James Prather, Brent N Reeves, Brett A Becker, Gozde Cetin Uzun, Dastyni Loksa, Juho Leinonen, and Paul Denny. 2024. Self-Regulation, Self-Efficacy, and Fear of Failure Interactions with How Novices Use LLMs to Solve Programming Problems. InProceedings of ...

    2024

  17. [25]

    2015.Visual Studio Code

    Microsoft. 2015.Visual Studio Code. https://code.visualstudio.com/

    2015

  18. [26]

    Kevin KB Ng, Liyana Fauzi, Leon Leow, and Jaren Ng. 2024. Harnessing the Potential of Gen-AI Coding Assistants in Public Sector Software Development. arXiv preprint arXiv:2409.17434(2024)

    2024 arXiv

  19. [27]

    Gabrielle O’Brien. 2025. How Scientists Use Large Language Models to Program. arXiv preprint arXiv:2502.17348(2025)

    2025 arXiv

  20. [28]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590(2023). ICER 2025 Vol. 1, August 3–6, 2025, Charlottesville, VA, USA Shihab et al

    2023 arXiv

  21. [29]

    2010.Permutation Tests for Complex Data: Theory, Applications and Software

    Fortunato Pesarin and Luigi Salmaso. 2010.Permutation Tests for Complex Data: Theory, Applications and Software. John Wiley & Sons, Chichester, UK. doi:10. 1002/9780470689516

    2010

  22. [30]

    Farman Ali Pirzado, Awais Ahmed, Román A Mendoza-Urdiales, and Hugo Terashima-Marin. 2024. Navigating the pitfalls: Analyzing the behavior of LLMs as a coding assistant for computer science students-a systematic review of the literature.IEEE Access(2024)

    2024

  23. [31]

    It’s weird that it knows what i want

    James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. “It’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers.ACM trans...

    2023

  24. [32]

    James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randri- anasolo, Brett A Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The widening gap: The benefits and harms of generative ai for novice programmers. InProceedings of the 2024 ACM Conferenc...

    2024

  25. [33]

    Jaakko Sauvola, Sasu Tarkoma, Mika Klemettinen, Jukka Riekki, and David Doermann. 2024. Future of software development with generative AI.Automated Software Engineering31, 1 (2024), 26

    2024

  26. [34]

    Andreas Scholl and Natalie Kiesler. 2024. How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course.arXiv preprint arXiv:2407.20792(2024)

    2024 arXiv

  27. [35]

    Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, and Iftekhar Ahmed. 2025. Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward.Information and Software Technology178 (2025), 107610

    2025

  28. [36]

    Anshul Shah, Anya Chernova, Elena Tomson, Leo Porter, William G Griswold, and Adalbert Gerald Soosai Raj. 2025. Students’ Use of GitHub Copilot for Working with Large Code Bases. InProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1. 1050–1056

    2025

  29. [37]

    James Skripchuk, John Bacher, and Thomas Price. 2024. An Investigation of the Drivers of Novice Programmers’ Intentions to Use Web Search and GenAI. In Proceedings of the 2024 ACM Conference on International Computing Education Research-Volume 1. 487–501

    2024

  30. [38]

    M Srinivas, G Ramakrishna, K Rajasekhara Rao, and E Suresh Babu. 2016. Analysis of legacy system in software application development: A comparative survey. International Journal of Electrical and Computer Engineering6, 1 (2016), 292

    2016

  31. [39]

    John Sweller. 2011. Cognitive load theory, evolutionary educational psychol- ogy, and instructional design.Evolutionary Psychology and Information Systems Research(2011), 255–269

    2011

  32. [40]

    Benyamin Tabarsi, Heidi Reichert, Ally Limke, Sandeep Kuttal, and Tiffany Barnes

  33. [41]

    Wannita Takerngsaksiri, Cleshan Warusavitarne, Christian Yaacoub, Matthew Hee Keng Hou, and Chakkrit Tantithamthavorn. 2024. Students’ Perspectives on AI Code Completion: Benefits and Challenges. In2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC...

    2024

  34. [42]

    Ben Arie Tanay, Lexy Arinze, Siddhant S Joshi, Kirsten A Davis, and James C Davis. 2024. An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project.arXiv preprint arXiv:2403.18679(2024)

    2024 arXiv

  35. [43]

    Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The metacognitive demands and opportunities of generative AI. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–24

    2024

  36. [44]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)

    2023 arXiv

  37. [45]

    Annapurna Vadaparty, Daniel Zingaro, David H Smith IV, Mounika Padala, Chris- tine Alvarado, Jamie Gorson Benario, and Leo Porter. 2024. Cs1-llm: Integrating llms into cs1 instruction. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. 297–303

    2024

  38. [46]

    Ingmar Visser, Maartje EJ Raijmakers, and Peter CM Molenaar. 2010. A stochas- tic approach for sequential analysis of programming behaviors.Learning and Individual Differences20, 4 (2010), 313–322

    2010

  39. [47]

    Philip Welsby and Bernard MY Cheung. 2023. ChatGPT. 1047–1048 pages

    2023

  40. [48]

    Tianyi Xu, Pranav Vaithilingam, Xinyu Li, Yuntao Tian, Laura Dabbish, and Brad A Myers. 2022. A theory-driven approach to AI-assisted programming.CHI Conference on Human Factors in Computing Systems(2022), 1–15

    2022

  41. [49]

    Hao Yu. 2023. Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching.Frontiers in Psychology14 (2023), 1181712

    2023

  42. [2025]

    LLMs’ Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters.arXiv preprint arXiv:2503.05012(2025)

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.