REVIEW 3 major objections 5 minor 3 references
"Give me the code" -- Log Analysis of First-Year CS Students' Interactions With GPT
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Untrained first-year CS students used ChatGPT successfully: 72.2% of logged groups incorporated its solutions, and about half chose between two AI solutions with stated reasons, evidence of developing critical thinking.
desk verdict A useful, well-scoped dataset on first-years using ChatGPT inside a real project, but the critical-thinking conclusion overreaches without a baseline or validated rubric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the five-step exercise template: state the problem to the LLM, analyze the returned solution, ask for an alternative, compare both, and write a conclusion justifying the choice, all recorded in a log file. This template operationalizes 'critical thinking' as a choice between two generated solutions with an explicit decision criterion (simpler, more flexible, or easier to understand), and it is also what makes the interaction logs analyzable. The paper's classification of prompts (abstraction level, nature, context, restrictions, examples, function signature, second-prompt type, conclusion type, and actual code reuse) is the measurement instrument that turns raw chat logs into the reported percentages.
What would settle it
Have a control group of students pick between two ChatGPT solutions by random assignment and write the same style of conclusion; if their 'decided for one solution' rate with a stated criterion approaches the observed 47.2%, the rubric is not distinguishing critical thinking from mere compliance.
Extended reading notes
Core claim
The paper reports that in a Data Structures and Algorithms course project, first-year students asked ChatGPT to help parse a large CSV file, following a supplied five-step template: state the problem, analyze the proposed solution, request an alternative, compare, and decide. Analyzing the interaction logs, the authors found that most initial prompts were direct requests for code in Portuguese with little context or examples, yet 72.2% of groups ultimately used the GPT solution (fully or partially) in their projects, and 47.2% wrote conclusions selecting one of two alternative implementations, with decision criteria distributed among simpler/more compact, more flexible/generic, and easier to understand. The authors interpret this as evidence that students with no prompting training can leverage LLMs successfully and can incorporate the generated solutions, and that the act of choosing between two alternatives with a written rationale is a form of developing critical thinking. The survey adds that 84.6% of respondents said they interacted with GPT, 69.2% would have used it even without the assignment, and 73% found the exercise useful.
Load-bearing premise
The paper's critical-thinking conclusion rests on equating a written choice of one ChatGPT solution over another — with a reason such as 'simpler', 'more flexible', or 'easier to understand' — with demonstrated critical thinking; if that equivalence fails, the educational-judgment claim weakens even though the 72.2% usage-rate finding stands on its own.
Editorial extensions
If this is right
- The five-step template (ask, analyze, ask for an alternative, compare, decide) can be dropped into other programming courses as a graded, low-cost exercise that makes LLM use visible rather than underground.
- Because 72.2% of logged groups used ChatGPT's code, course instructors should assume that unmonitored AI use is already widespread and design assessments around code evaluation skills instead of trying to prevent access.
- The 38.9% of groups who skipped the requested alternative suggests that merely instructing students to compare solutions is not enough; the template would benefit from an explicit sub-question or rubric that forces the comparison step.
- The survey result that 30.8% would not have used GPT without the assignment implies that a mandatory exercise can bring reluctant students into a supervised first interaction with generative AI.
Reading between the lines
- A natural extension the paper does not run is a control condition: have students choose between two AI solutions by coin flip and write the same style of conclusion. If the rate of 'decided for one solution with a criterion' matches the observed 47.2%, the rubric is measuring compliance, not critical thinking.
- The paper's own limitation that GPT behavior changes over time and the model version was uncontrolled means the numerical results are a snapshot; repeating the same log analysis in later terms would test whether the findings are stable or model-era-specific.
- The reported 'simpler, more flexible, easier to understand' criteria could be connected to a learning outcome: a follow-up study could ask students who completed the template to modify or debug their chosen solution, testing whether the act of comparison actually improved code comprehension.
- The template's log files are a ready-made dataset for measuring prompt sophistication; a future analysis could correlate the presence of examples, context, or function signatures with whether the chosen solution passed the automated tests.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports an observational study of 37 groups (69 first-year CS students) who submitted ChatGPT interaction logs for a CSV-parsing exercise embedded in a larger project. The exercise template instructed students to ask for two alternative solutions and justify a choice. The authors manually classify prompts, conclusions, and code usage, and supplement this with a post-hoc survey (N=52). They report that most students used unsophisticated prompts, that 72.2% of logged groups incorporated ChatGPT's solutions (fully or partially) into their projects, and conclude that about half demonstrated judgment in choosing between solutions and thereby developed critical thinking.
Significance. If the descriptive claims are accepted, the paper makes a useful practical contribution: a simple, replicable exercise template for integrating LLMs into introductory CS projects, plus a public dataset of 37 real student-GPT interaction logs. The log analysis provides concrete evidence of novice prompting behavior (e.g., 81.8% of initial prompts lacked context, 92.9% omitted function signatures), which is valuable for educators designing prompt-training interventions. However, the higher-stakes educational claim—that the exercise 'fostered' or 'showcased the development of' critical thinking—is not supported by the evidence as currently operationalized. The paper's value is strongest as a descriptive study of usage and adoption; the critical-thinking inference needs rethinking or substantial additional validation.
major comments (3)
- [Section 6.1, Table 4, and Section 9] The claim that roughly half the students 'demonstrated the ability to exercise judgment' and that this 'showcase[s] the development of their critical thinking' rests entirely on the Section 6.1 coding criterion that classifies a conclusion as 'Decided for one solution' whenever a student selects one of two ChatGPT outputs and gives a reason such as 'simpler', 'more flexible', or 'easier to understand'. This operationalization is not validated, and no inter-rater reliability is reported for this subjective classification. Selecting one option and stating an obvious property of it is not the same as comparing trade-offs, so the evidence does not establish judgment or critical thinking. Please either reframe the claim as 'students chose a solution and provided a rationale' or add a validated rubric and reliability analysis before making the critical-thinking claim.
- [Section 9 and Section 8] The conclusion that the exercise 'fostered' critical thinking requires a baseline or control group. The study only observes students after they have completed the exercise, so it cannot distinguish between critical thinking that was developed by the exercise and judgment that students already possessed. The Limitations section (Section 8) discusses nondeterminism and GPT model version, but it does not address this missing baseline. Please soften the causal language (e.g., 'students exercised judgment') or add a pre/post or comparison-group design.
- [Sections 5, 6, and 9] The paper reports percentages as if they apply to students, but the unit of analysis is groups: 37 groups (69 students) submitted logs, and Table 4's 'used gpt solution' (44.4% + 27.8% = 72.2%) is a per-group percentage. The abstract correctly says 'logged groups', but Section 9 states '72.2% of students incorporating ChatGPT's solutions', which is inaccurate because a group may contain one or two students. Also, the sample is restricted to groups whose projects passed at least half the automated tests and who chose to submit a log; the 72.2% adoption rate is conditional on these criteria. Please report denominators explicitly and avoid student-level generalization from group-level data.
minor comments (5)
- [Table 2 and Section 6.1] There is a numeric inconsistency: Table 2 lists 'ask for code' as 69.4%, but Section 6.1 states 'asked for code (72.2%)'.
- [Table 3 and Section 6.1] Section 6.1 says 38.9% 'failed to request an alternative implementation', but the non-alternative categories in Table 3 (27.8% inexistent + 11.1% clarify initial prompt + 5.6% ask different question) sum to 44.5%; please reconcile.
- [Section 6.2.1] The sentence 'this student was the only one who had taken the Databases course' refers to a group, not necessarily a single student; please clarify whether this means both group members had not taken it.
- [Section 6.1] The phrase '72.2% of participants acknowledged acceptance of one of the provided solutions, with an additional 25% expressing utility' produces a 72.2% figure that is coincidentally the same as the 'used gpt solution' percentage (44.4% + 27.8%); using the same number for two different things is confusing and should be made explicit.
- [Section 5.1.1] The classification example says 'general' for the problem nature, but Section 5.1 defines the categories as 'generic' and 'domain-specific'; use the same terminology.
Circularity Check
No significant circularity: the usage-rate findings rest on external log and code evidence, and the self-citations are motivational rather than load-bearing.
full rationale
The paper's central quantitative claims are grounded in external observational data: student interaction logs, manually inspected project code, and a survey. The claim that 72.2% of analyzed groups incorporated ChatGPT output into their projects is supported by a distinct, non-circular check: 'We verified this by manually analyzing the project's code for each group, searching for code that is similar to one of the GPT solutions' (Section 5.1). This does not reduce to the rubric or to any fitted parameter. Similarly, RQ1 and RQ2 are answered by inspecting prompts and code, not by re-stating an input assumption. The self-citations (Cipriano and Alves 2023, 2024; Alves and Cipriano 2023) appear only in the introduction and lessons-learnt discussion, supporting background motivation and the opinion that evaluating multiple LLM solutions will be important; they are not used to define the outcome variables or to justify the measurement results. The remaining concern about the critical-thinking conclusion is an interpretive/validity issue: the paper equates writing a conclusion that selects one solution with exercising judgment (Section 6.1, 'Decided for one solution'; Section 9), and it lacks a baseline for 'development' of critical thinking. That is a substantive methodological limitation, but it is not circularity under the definitions used here: the observed behavior (choosing and justifying) is measured from the students' own submitted text, not generated by the authors' prior work or by the coding scheme itself in a way that forces the conclusion. Since the paper does not fit parameters, invoke a uniqueness theorem, or rename an earlier result as a prediction, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption Submitted log files faithfully record the students' real ChatGPT interactions, without editing or fabrication.
- domain assumption Manual comparison of project code against GPT solutions correctly identifies whether a solution was used fully or partially.
- ad hoc to paper Choosing one of two generated solutions and stating a criterion is a valid indicator of critical thinking.
- domain assumption The 52 survey respondents are representative of the 154 enrolled students, and self-reports are accurate.
- domain assumption ChatGPT model version and behavior were sufficiently stable across the data collection period.
Cite this review
Pith. "Pith review of "Give me the code" -- Log Analysis of First-Year CS Students' Interactions With GPT." pith.science (2026). https://pith.science/paper/IEAFKYO6
@misc{pith2026241117855,
author = {Pith},
title = {Pith review of: "Give me the code" -- Log Analysis of First-Year CS Students' Interactions With GPT},
year = {2026},
howpublished = {\url{https://pith.science/paper/IEAFKYO6}},
note = {Machine review of arXiv:2411.17855}
}
read the original abstract
The impact of Large Language Models (LLMs) like GPT-3, GPT-4, and Bard in computer science (CS) education is expected to be profound. Students now have the power to generate code solutions for a wide array of programming assignments. For first-year students, this may be particularly problematic since the foundational skills are still in development and an over-reliance on generative AI tools can hinder their ability to grasp essential programming concepts. This paper analyzes the prompts used by 69 freshmen undergraduate students to solve a certain programming problem within a project assignment, without giving them prior prompt training. We also present the rules of the exercise that motivated the prompts, designed to foster critical thinking skills during the interaction. Despite using unsophisticated prompting techniques, our findings suggest that the majority of students successfully leveraged GPT, incorporating the suggested solutions into their projects. Additionally, half of the students demonstrated the ability to exercise judgment in selecting from multiple GPT-generated solutions, showcasing the development of their critical thinking skills in evaluating AI-generated code.
Reference graph
Works this paper leans on
-
[1]
Alves, P. and Cipriano, B. P. (2023). The centaur programmer–How Kasparov’s Advanced Chess spans over to the software development of the future. arXiv preprint arXiv:2304.11172. Babe, H. M., Nguyen, S., Zi, Y ., Guha, A., Feldman, M. Q., and Anderson, C. J. (2023a). Studenteval: A bench- mark of student-written prompts for large language models of code. a...
work page Pith review arXiv 2023
-
[3]
Navigating Complexity in Software Engineering: A Prototype for Comparing GPT-n Solutions
ACM. Sridhar, P., Doyle, A., Agarwal, A., Bogart, C., Savelka, J., and Sakr, M. (2023). Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives. Treude, C. (2023). Navigating Complexity in Software En- gineering: A Prototype for Comparing GPT-n Solu- tions. arXiv:2301.12169 [cs]. Xu, F. F., Alon, U., Neubig, G., and H...
work page Pith review arXiv 2023
-
[2023]
It’s Weird That it Knows What I Want
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., and Reeves, B. N. (2023b). Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Genera- tors. arXiv:2307.16364 [cs]. Destefanis, G., Bartolucci, S., and Ortu, M. (2023). A Pre- liminary Analysis on the Code Generation Capabili- ties of GP...
arXiv 2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.