Pith. sign in

REVIEW 2 major objections 6 minor 28 references

Analysis of Student-LLM Interaction in a Software Engineering Project

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Conversational interaction with ChatGPT produces simpler, better-fitting code than Copilot's auto-completions, and students' prompting visibly improves over one semester.

desk verdict Useful longitudinal dataset on student-LLM interaction, but the ChatGPT-vs-Copilot complexity comparison is confounded by measuring raw ChatGPT output against post-edit Copilot code, so the headline claim should be treated as suggestive, not established. read the letter →

arxiv 2502.01273 v1 pith:KOGR3VTE submitted 2025-02-03 cs.SE cs.AI

Agrawal Naman , Ridwan Shariffdeen , Guanlin Wang , Sanka Rasnayaka , Ganesh Neelakanta Iyer This is my paper · ORCID
classification cs.SEcs.AI
keywords LLMforcodegenerationlearningAIsoftwareengineeringeducationChatGPTGitHubCopilotcomplexityprompt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Across a 13-week software-engineering course, 126 students used ChatGPT and GitHub Copilot to build a static program analyzer, and the researchers studied the conversations, the code that was generated, and the code that was actually integrated. The paper argues that conversational interaction with an LLM produces code that is shorter, less complex, and better matched to what a student needs than the auto-completed code Copilot offers. It also claims that students' prompting improves measurably over the semester, with later prompts producing code that enters the repository almost unchanged. If true, this matters for teaching because it shifts the skill students need from writing code to steering an AI through dialogue, and it suggests curricula should explicitly train prompt refinement.

What carries the argument

The argument runs on three measurement devices working together. The first is a four-metric complexity battery — lines of code, cyclomatic complexity, maximum control-flow depth, and Halstead effort — applied to every tagged LLM-generated snippet. The second is a conversation-to-repository similarity pipeline that matches ChatGPT outputs inside a dialogue to code in the final repository using syntax-tree token sets and Jaccard similarity, with long common subsequences treated as equivalent above 90%. The third is sentiment scoring of prompts using a rule-based lexicon, which yields the positive-to-frustrated-to-resolved arc. The tagging scheme records which tool generated each snippet and how many manual edits were needed (none, under 10% of lines, or over 10%), and this metadata carries the headline comparison.

What would settle it

Record Copilot's verbatim suggestion text before any student edit, run the same four complexity metrics on it, and compare it with ChatGPT's raw responses to identical prompts; if Copilot's raw code is no longer or higher-effort, the reported advantage is a measurement artifact.

Watch

Extended reading notes

Core claim

The central discovery is a workflow-level result: when students interact with ChatGPT through back-and-forth prompts, the generated code's complexity falls as the conversation proceeds, and the final version is more concise and lower in Halstead effort than Copilot's auto-completed output. Copilot-generated code is longer, has higher cyclomatic complexity, and produces many more outliers across all four metrics. Students reported less manual intervention for ChatGPT code, with 26% of ChatGPT snippets requiring no changes versus 15.2% for Copilot. The similarity between ChatGPT output and the final repository code rises across the three milestones as prompts become more specific and context-rich, and sentiment in conversations follows a pattern of initial optimism, mid-conversation frustration, and final resolution.

Load-bearing premise

The headline comparison assumes that ChatGPT's raw outputs and Copilot's accepted outputs are measured on the same footing, but Copilot's snippets were measured after students edited them, so the complexity gap could come from how the tools were tracked rather than from the tools themselves.

Editorial extensions

If this is right

  • ChatGPT's conversational mode yields more concise, lower-effort code than Copilot's auto-completion, so tool choice can measurably affect code maintainability in student projects.
  • Students' prompts become more specific and context-rich over the semester, indicating that prompt engineering is a skill that develops through project experience.
  • AI-generated code is reused with increasing similarity to the repository over milestones, showing that iterative dialogue brings LLM output progressively closer to project needs.
  • Most conversations end with a positive sentiment recovery, suggesting that back-and-forth refinement helps students reach a resolution rather than leaving them stuck.
  • LLM usage is heavy early in the project and declines later, which points to AI assistants being most useful during the initial design and scaffolding phases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit is that the complexity gap between ChatGPT and Copilot may partly reflect a measurement asymmetry: ChatGPT outputs were captured verbatim, while Copilot outputs were measured after student editing, so a controlled study that records Copilot's raw suggestions would be needed to separate workflow effects from model effects.
  • The observed prompt improvement over a single semester suggests prompt engineering is trainable enough to be a formal curriculum component, and this paper's milestone-by-milestone similarity data could serve as a baseline for measuring the effect of such instruction.
  • Because integrated repository code is often more complex than the raw ChatGPT output, students are evidently adding their own logic rather than blindly accepting AI code; that pattern could be used to design exercises that make the critical-editing step explicit and assessable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper reports an observational study of 126 undergraduate students using ChatGPT and GitHub Copilot during a 13-week software engineering project. The authors collected LLM-generated code snippets, student-annotated intervention levels, ChatGPT conversation logs, and repository code, then analyzed tool usage, code complexity (LOC, cyclomatic complexity, CFG depth, Halstead effort), code integration patterns, prompt evolution, and sentiment. The main reported findings are that students prefer ChatGPT over Copilot, ChatGPT-generated code has lower computational complexity than Copilot-generated code, conversational interaction supports iterative code refinement, students' prompting skills improve over the semester, and conversations end on a positive note. The paper also discusses pedagogical implications and design considerations for AI-assisted SE courses.

Significance. If the central claims held, this would be a valuable contribution to the emerging literature on LLM use in software engineering education: it provides a real classroom dataset with 730 tagged code snippets, 62 ChatGPT conversations, and longitudinal data across three milestones, and it explicitly connects code complexity metrics with conversational interaction. The paper is transparent about many of its limitations, including self-report bias and the known difficulty of sentiment analysis on technical text, and it makes its data collection methodology (tagging, intervention levels, conversation linking) reproducible. However, the headline comparison between ChatGPT and Copilot is currently undermined by a measurement asymmetry, and several key claims are made without statistical support. With the load-bearing issues addressed, the dataset and qualitative insights could be of genuine use to educators and researchers.

major comments (2)
  1. [III.B] The comparison of integration effort between ChatGPT and Copilot rests on self-reported intervention levels without any statistical testing. The paper reports that 53.6% of Copilot snippets required level 1 intervention and 15.2% required none, versus 26% of ChatGPT snippets requiring no modification, but it does not test whether these differences are significant or account for per-team clustering. Given that Section IV acknowledges that self-reports may be underreported or selectively disclosed, a formal test (e.g., mixed-effects model or at minimum a chi-square test with team-level clustering) and effect sizes are needed before claiming that ChatGPT code 'generally met project requirements with minimal refinement.'
  2. [III.B] The abstract's claim that 'students prefer ChatGPT over Copilot' is not directly supported by the data presented. Section III.A reports counts of accepted snippets per tool (223 ChatGPT vs. 507 Copilot), which reflects aggregate usage, not preference: higher Copilot usage could indicate convenience or default tool availability, and the paper does not report user surveys or explicit preference ratings. The claim should be softened to 'students used ChatGPT and Copilot in different ways' or supported with a direct preference measure.
minor comments (6)
  1. [III.B] The sentence 'Additionally, the study's analysis of GPT-generated code is more precise due to the ability to track exact model outputs' is important but appears only as an aside; it should be moved into the methodological design and discussed as a limitation with potential mitigation.
  2. [IV] The Threats to Validity section is brief and omits a discussion of how the measurement asymmetry between ChatGPT and Copilot affects every complexity comparison; this should be elevated to a primary threat.
  3. [Throughout] There are several typographical and grammatical errors, including 'Furthre analyzing' (Introduction), 'prevelance' (Summary), and 'the student teams heavily on AI assistants' (Section III.A). A proofreading pass is needed.
  4. [III.C] The reuse analysis reports counts (95 instances used once, 23 twice, etc.) but does not specify the total number of instances considered; the denominator is needed to interpret these numbers.
  5. [III.D] The description of Figure 7's construction is dense and somewhat ambiguous: the 'prioritized the first occurrence' rule for ties should be justified, since it may bias the reported 'mean index' downward.
  6. [References] Several references are incomplete or non-standard, including missing venue details for [11], [13], [14], [17], [19], [20], and [22]; the bibliography should be checked against a consistent style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports observations from a new dataset, and its conclusions are not forced by definition or by self-citation.

full rationale

The paper is an empirical study of student-LLM interactions in a software engineering course. Its central claims—that students prefer ChatGPT, that ChatGPT-generated code has lower computational complexity than Copilot-generated code, that conversational interaction yields code requiring less refinement, and that prompting skills improve over the semester—are supported by measurements on collected data (code snippets, conversation logs, student annotations). No derivation chain is present that would reduce a finding to its own input. The tagging framework and intervention levels are adopted from the authors' prior work [3], but that is a methodological inheritance, not a circular justification of the findings; the conclusions do not rest on [3]'s results. The comparison of ChatGPT and Copilot code complexity is acknowledged by the authors to be asymmetric: ChatGPT outputs are tracked as exact model outputs, while Copilot contributions are assessed through student modifications. This is a validity threat and a potential source of bias, but it is not circularity in the sense of a claim being equivalent to its inputs by construction. Similarly, the improvement in prompting skills is inferred from observed trends in similarity scores and qualitative prompt examples, not from any fitted parameter that presumes the conclusion. The paper is self-contained as an observational study; any weaknesses are matters of internal validity and measurement bias, not circular reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

Empirical study with no fitted parameters or invented entities. The analysis relies on domain assumptions about metric validity and data accuracy.

assumptions (4)
  • domain assumption Complexity metrics (LOC, cyclomatic complexity, CFG depth, Halstead effort) proxy code quality and maintainability.
    Used throughout Section III.B to compare generated code.
  • domain assumption VADER sentiment scores of prompts reflect student sentiment and engagement.
    Used in Section III.E; limitation acknowledged in Threats to Validity.
  • domain assumption Similarity matching between repository code and conversation outputs correctly identifies the code students used.
    Used in Section III.C and III.D to infer integration patterns.
  • domain assumption Self-reported human intervention levels accurately reflect actual modification effort.
    Students tagged code with intervention levels; Threats to Validity notes underreporting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysis of Student-LLM Interaction in a Software Engineering Project." pith.science (2026). https://pith.science/paper/KOGR3VTE

@misc{pith2026250201273,
  author       = {Pith},
  title        = {Pith review of: Analysis of Student-LLM Interaction in a Software Engineering Project},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOGR3VTE}},
  note         = {Machine review of arXiv:2502.01273}
}
read the original abstract

Large Language Models (LLMs) are becoming increasingly competent across various domains, educators are showing a growing interest in integrating these LLMs into the learning process. Especially in software engineering, LLMs have demonstrated qualitatively better capabilities in code summarization, code generation, and debugging. Despite various research on LLMs for software engineering tasks in practice, limited research captures the benefits of LLMs for pedagogical advancements and their impact on the student learning process. To this extent, we analyze 126 undergraduate students' interaction with an AI assistant during a 13-week semester to understand the benefits of AI for software engineering learning. We analyze the conversations, code generated, code utilized, and the human intervention levels to integrate the code into the code base. Our findings suggest that students prefer ChatGPT over CoPilot. Our analysis also finds that ChatGPT generates responses with lower computational complexity compared to CoPilot. Furthermore, conversational-based interaction helps improve the quality of the code generated compared to auto-generated code. Early adoption of LLMs in software engineering is crucial to remain competitive in the rapidly developing landscape. Hence, the next generation of software engineers must acquire the necessary skills to interact with AI to improve productivity.

Figures

Figures reproduced from arXiv: 2502.01273 by the authors.

Figure 1
Figure 1. Density Plot for measured key metrics [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of ChatGPT and Copilot Complexity [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Variation of ChatGPT generated code in a conversation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distribution of Difference Complexity Measures be [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Similarity of generated and integrated code [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Histogram of Conversation Lengths (filtered for con [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 8
Figure 8. Figure 8: Variation of Compound Vader Scores Over a Conver [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 7
Figure 7. Figure 7: Mean Index of the Generated Code Most Similar to [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages

  1. [1]

    Chatgpt for good? on opportunities and challenges of large language models for education,

    E. Kasneci, K. Sessler, S. K ¨uchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. G ¨unnemann, E. H ¨ullermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, “Chatgpt for good? on opportunities and challenges of large language m...

    work page 2023

  2. [2]

    Empowering personalized learning through a conversation-based tutoring system with student modeling,

    M. Park, S. Kim, S. Lee, S. Kwon, and K. Kim, “Empowering personalized learning through a conversation-based tutoring system with student modeling,” in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145...

    arXiv 2024

  3. [3]

    An empirical study on usage and perceptions of llms in a software engineering project,

    S. Rasnayaka, G. Wang, R. Shariffdeen, and G. N. Iyer, “An empirical study on usage and perceptions of llms in a software engineering project,” in Proceedings of the 1st International Workshop on Large Language Models for Code , ser. LLM4Code ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 111–118. [Online]. Available: https://doi.o...

    arXiv 2024

  4. [4]

    Few-shot training llms for project- specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project- specific code-summarization,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’22. New York, NY , USA: Association for Computing Machinery,

    work page

  5. [5]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” ser. FSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 572–576. [Online]. Available: https://doi.org/10.1145/3663529.3663801

    arXiv 2024

  6. [6]

    Detecting code comment inconsistencies using llm and program analysis,

    Y . Zhang, “Detecting code comment inconsistencies using llm and program analysis,” in Companion Proceedings of the 32nd ACM International Conference on the F oundations of Software Engineering, ser. FSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 683–685. [Online]. Available: https://doi.org/10.1145/3663529.3664458

    work page arXiv 2024

  7. [7]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) , 2023, pp. 647–658

    work page 2023

  8. [8]

    Inferfix: End-to-end program repair with llms,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering , ser. ESEC/FSE 2023. New York, NY , USA: Association for Computing Machinery, 2023, p. 1...

    arXiv 2023

Show all 28 references
  1. [9]

    Codeplan: Repository-level coding using llms and planning,

    R. Bairi, A. Sonwane, A. Kanade, V . D. C., A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “Codeplan: Repository-level coding using llms and planning,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643757

    2024 doi

  2. [10]

    Vader: A parsimonious rule-based model for sentiment analysis of social media text,

    C. Hutto and E. Gilbert, “Vader: A parsimonious rule-based model for sentiment analysis of social media text,” Proceedings of the International AAAI Conference on Web and Social Media , vol. 8, no. 1, pp. 216–225, May 2014. [Online]. Available: https://ojs.aaai.org/index.php/I...

    2014

  3. [11]

    Adapting software engineering education to the age of llms: A holistic approach,

    L. Garcia, T. Nguyen, and S. Lee, “Adapting software engineering education to the age of llms: A holistic approach,” Proceedings of the ACM Conference on Software Engineering Education , 2024. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3626252.3630927

    2024

  4. [12]

    Efficient classification of student help requests in programming courses using large language models,

    J. Savelka, P. Denny, M. Liffiton, and B. Sheese, “Efficient classification of student help requests in programming courses using large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.20105

    2023 arXiv

  5. [13]

    Exploring the role of llms as ai tutors in programming education: A case study with gpt-3.5-turbo,

    J. Kim, C. Rivera, and J. Martin, “Exploring the role of llms as ai tutors in programming education: A case study with gpt-3.5-turbo,” Proceedings of the ACM Conference on Learning Technologies ,

  6. [14]

    The impact of generative ai on gamified se education: Insights from code defenders with llm support,

    P. Harrison, X. Liu, and M. Roberts, “The impact of generative ai on gamified se education: Insights from code defenders with llm support,” Proceedings of the ACM Conference on Educational Games ,

  7. [15]

    Studenteval: A benchmark for evaluating llms on novice user prompts in educational settings,

    A. Jackson, M. Green, and R. Patel, “Studenteval: A benchmark for evaluating llms on novice user prompts in educational settings,” LLM4Code Workshop , 2024. [Online]. Available: https://llm4code. github.io/2024/assets/pdf/papers/95.pdf

    2024

  8. [16]

    Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs,

    M. Kazemitabaar, R. Ye, X. Wang, A. Z. Henley, P. Denny, M. Craig, and T. Grossman, “Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs,” in Proceedings of the 2024 CHI Conference on Human Factors in Comput...

    2024

  9. [17]

    Available: https://dl.acm.org/doi/abs/10.1145/3661167

    [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3661167. 3661273

    doi

  10. [18]

    Experimental evaluation of llms for unit test generation: Chatgpt vs. pynguin,

    J. Smith, E. Lee, and M. Davis, “Experimental evaluation of llms for unit test generation: Chatgpt vs. pynguin,” LLM4Code Workshop ,

  11. [19]

    Generating insightful questions with llms: Enhancing data understanding through code alignment,

    A. Williams, J. Brown, and W. Chen, “Generating insightful questions with llms: Enhancing data understanding through code alignment,” LLM4Code Workshop , 2024. [Online]. Available: https: //llm4code.github.io/2024/assets/pdf/papers/21.pdf

    2024

  12. [20]

    Using llms in ide for code understanding: A case study with gpt-3.5-turbo,

    J. Zhang, L. Sun, and X. Qiu, “Using llms in ide for code understanding: A case study with gpt-3.5-turbo,” Proceedings of the ACM Conference on Software Engineering , 2024. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3597503.3639187

    2024

  13. [21]

    A survey on llm-based agents for software engineering: Capabilities, challenges, and future directions,

    S. Miller, P. Garcia, and D. Kim, “A survey on llm-based agents for software engineering: Capabilities, challenges, and future directions,” arXiv preprint , 2024. [Online]. Available: https://arxiv.org/abs/2409. 02977

    2024

  14. [22]

    Available: https://llm4code.github.io/2024/assets/pdf/ papers/38.pdf

    [Online]. Available: https://llm4code.github.io/2024/assets/pdf/ papers/38.pdf

    2024

  15. [23]

    Educational large models (edullms): Transforming digital education through personalized learning and intelligent tutoring,

    L. Chen, M. Zhang, and J. Zhao, “Educational large models (edullms): Transforming digital education through personalized learning and intelligent tutoring,” arXiv preprint , 2023. [Online]. Available: https://arxiv.org/abs/2311.13160

    2023 arXiv

  16. [24]

    Evaluating llms for automated program repair: A comparative study of bug-fixing capabilities,

    Y . Wang, H. Li, and K. Zhang, “Evaluating llms for automated program repair: A comparative study of bug-fixing capabilities,” IEEE Transactions on Software Engineering , 2024. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10172854

    2024

  17. [26]

    Threats to validity in llm-based software engineering research: Challenges and guidelines,

    X. Zhou, Y . Xu, and Z. Jiang, “Threats to validity in llm-based software engineering research: Challenges and guidelines,” Proceedings of the ACM Conference on Software Engineering , 2024. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3639476.3639764

    2024

  18. [28]

    Scaling programming assignment grading with chatgpt: A case study in higher education,

    D. Taylor, A. Singh, and M. Lopez, “Scaling programming assignment grading with chatgpt: A case study in higher education,” LLM4Code Workshop, 2024. [Online]. Available: https://llm4code.github.io/2024/ assets/pdf/papers/5.pdf

    2024

  19. [2023]

    Available: https://doi.org/10.1145/3551349.3559555

    [Online]. Available: https://doi.org/10.1145/3551349.3559555

  20. [2024]

    Available: https://dl.acm.org/doi/abs/10.1145/3639474

    [Online]. Available: https://dl.acm.org/doi/abs/10.1145/3639474. 3640061

    doi

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.