{"id":"107b0609-b254-4cf8-b049-07f83e2eaa66","arxiv_id":"2411.17855","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"First-year CS students, given no prompt training, mostly obtained usable code from ChatGPT and often incorporated it, though only about half chose between two alternatives in a way the authors count as critical judgment.","lead":"This paper analyzed the chat logs of 69 first-year computer science students who used ChatGPT to write code for part of a semester-long project. Most students managed to get usable code, about three quarters used it in their project, and the authors propose a template for teaching critical evaluation of AI output.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The critical-thinking conclusion rests on an unvalidated rubric and no baseline; the descriptive usage claims are more solid.","rationale":"The reader's weakest assumption correctly identifies the unvalidated operationalization of critical thinking as the central soft spot. I agree with that diagnosis and add that the paper's causal wording ('fostering', 'development') requires a baseline or control that is absent. The descriptive usage-rate findings are more robust because they were verified against project code and the logs are shared, so the paper still has value as an observational account. A CONDITIONAL verdict remains appropriate: the educational contribution should be reframed as evidence of selection behavior, not demonstrated growth in critical thinking, and the rubric needs validation or at least inter-rater reliability. I did not identify a fabrication or internal contradiction; the concern is about interpretive overreach relative to the evidence. The proposed concrete re-coding and transfer test would directly settle whether the critical-thinking claim survives stricter scrutiny.","tokens_in":11510,"tokens_out":2794,"duration_ms":29501,"concrete_test":"Have two independent raters, blind to the paper's hypothesis, re-code the 37 publicly available interaction logs using a stricter rubric that requires (a) explicit comparison of both solutions on at least one shared dimension and (b) a decision criterion that is not merely a restatement of the chosen solution's obvious property. Report Cohen's kappa and the re-estimated proportion of groups meeting the stricter standard. If the proportion drops well below the reported 47.2% or inter-rater agreement is low, the critical-thinking claim should be weakened. To test the 'development' component, administer a transfer task—a new two-solution selection exercise without the template—to this cohort and to a comparable cohort that did not complete the exercise; if transfer performance does not differ, the causal claim about fostering critical thinking is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central educational claim—that roughly half of students demonstrated developing critical thinking—depends on the Section 6.1 coding rubric, which equates choosing one of two ChatGPT solutions and writing a conclusion (e.g., 'simpler', 'more flexible', 'easier to understand') with evidence of judgment. This operationalization is not validated, and the paper reports no inter-rater reliability for the manual classification of conclusions despite the subjective judgment involved. A student can select a solution for a superficial reason or simply restate the selected solution's most obvious property; that is not the same as comparing trade-offs. Moreover, even if the rubric validly measured judgment at one time point, the conclusion in Section 9 that the exercise 'fostered' critical thinking requires a baseline or control group, since an exercise can reveal existing judgment without developing it. The paper's Limitations section discusses nondeterminism and GPT model version, but does not address this missing baseline or the validity of the critical-thinking measure. The narrower descriptive claims—72.2% of analyzed groups incorporated GPT output into their projects and most students could get useful results—are better supported because they were checked against project code, so the load-bearing weakness is specifically the critical-thinking inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an observational study of 37 groups (69 first-year CS students) who submitted ChatGPT interaction logs for a CSV-parsing exercise embedded in a larger project. The exercise template instructed students to ask for two alternative solutions and justify a choice. The authors manually classify prompts, conclusions, and code usage, and supplement this with a post-hoc survey (N=52). They report that most students used unsophisticated prompts, that 72.2% of logged groups incorporated ChatGPT's solutions (fully or partially) into their projects, and conclude that about half demonstrated judgment in choosing between solutions and thereby developed critical thinking.","tokens_in":11711,"tokens_out":4400,"duration_ms":37450,"significance":"If the descriptive claims are accepted, the paper makes a useful practical contribution: a simple, replicable exercise template for integrating LLMs into introductory CS projects, plus a public dataset of 37 real student-GPT interaction logs. The log analysis provides concrete evidence of novice prompting behavior (e.g., 81.8% of initial prompts lacked context, 92.9% omitted function signatures), which is valuable for educators designing prompt-training interventions. However, the higher-stakes educational claim—that the exercise 'fostered' or 'showcased the development of' critical thinking—is not supported by the evidence as currently operationalized. The paper's value is strongest as a descriptive study of usage and adoption; the critical-thinking inference needs rethinking or substantial additional validation.","major_comments":[{"comment":"The claim that roughly half the students 'demonstrated the ability to exercise judgment' and that this 'showcase[s] the development of their critical thinking' rests entirely on the Section 6.1 coding criterion that classifies a conclusion as 'Decided for one solution' whenever a student selects one of two ChatGPT outputs and gives a reason such as 'simpler', 'more flexible', or 'easier to understand'. This operationalization is not validated, and no inter-rater reliability is reported for this subjective classification. Selecting one option and stating an obvious property of it is not the same as comparing trade-offs, so the evidence does not establish judgment or critical thinking. Please either reframe the claim as 'students chose a solution and provided a rationale' or add a validated rubric and reliability analysis before making the critical-thinking claim.","section":"Section 6.1, Table 4, and Section 9"},{"comment":"The conclusion that the exercise 'fostered' critical thinking requires a baseline or control group. The study only observes students after they have completed the exercise, so it cannot distinguish between critical thinking that was developed by the exercise and judgment that students already possessed. The Limitations section (Section 8) discusses nondeterminism and GPT model version, but it does not address this missing baseline. Please soften the causal language (e.g., 'students exercised judgment') or add a pre/post or comparison-group design.","section":"Section 9 and Section 8"},{"comment":"The paper reports percentages as if they apply to students, but the unit of analysis is groups: 37 groups (69 students) submitted logs, and Table 4's 'used gpt solution' (44.4% + 27.8% = 72.2%) is a per-group percentage. The abstract correctly says 'logged groups', but Section 9 states '72.2% of students incorporating ChatGPT's solutions', which is inaccurate because a group may contain one or two students. Also, the sample is restricted to groups whose projects passed at least half the automated tests and who chose to submit a log; the 72.2% adoption rate is conditional on these criteria. Please report denominators explicitly and avoid student-level generalization from group-level data.","section":"Sections 5, 6, and 9"}],"minor_comments":[{"comment":"There is a numeric inconsistency: Table 2 lists 'ask for code' as 69.4%, but Section 6.1 states 'asked for code (72.2%)'.","section":"Table 2 and Section 6.1"},{"comment":"Section 6.1 says 38.9% 'failed to request an alternative implementation', but the non-alternative categories in Table 3 (27.8% inexistent + 11.1% clarify initial prompt + 5.6% ask different question) sum to 44.5%; please reconcile.","section":"Table 3 and Section 6.1"},{"comment":"The sentence 'this student was the only one who had taken the Databases course' refers to a group, not necessarily a single student; please clarify whether this means both group members had not taken it.","section":"Section 6.2.1"},{"comment":"The phrase '72.2% of participants acknowledged acceptance of one of the provided solutions, with an additional 25% expressing utility' produces a 72.2% figure that is coincidentally the same as the 'used gpt solution' percentage (44.4% + 27.8%); using the same number for two different things is confusing and should be made explicit.","section":"Section 6.1"},{"comment":"The classification example says 'general' for the problem nature, but Section 5.1 defines the categories as 'generic' and 'domain-specific'; use the same terminology.","section":"Section 5.1.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is valid and is the core issue: the critical-thinking claim is overinterpreted. Beyond the rubric validity issue, I have flagged the missing baseline and the group-vs-student denominator problem, both of which should be addressed in revision. The descriptive usage-rate findings are solid and worth publishing, but the educational claim needs to be scaled back or properly validated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLMs in CS education. The genuinely new thing is that the log analysis happens inside a real three-month project, not a standalone exercise, and the logs are public. That extends the prior work by Prather, Prasad, Kazemitabaar, and Babe, and the exercise template is simple enough to lift. The paper is clearly written, and the descriptive claims are mostly solid: the authors manually checked the project code to verify whether students actually used the GPT output, and 72.2% of the logged groups did. The survey is small but reported without spin.\n\nThe soft spot is the critical-thinking claim. The rubric in Section 6.1 treats choosing one of two ChatGPT solutions and writing a conclusion like 'simpler' or 'more flexible' as evidence of judgment, and Section 9 then says the template 'fostered' critical thinking. There is no baseline, no control, no inter-rater reliability for the conclusion classification, and the sample is selective: 37 of 65 passing groups submitted logs. A student can pick a solution for a superficial reason or simply restate the most obvious property of the chosen code; that is not the same as comparing trade-offs. The stress-test note is on target here. The usage-rate claims do not depend on the critical-thinking interpretation and stand on their own.\n\nThere is also a numeric consistency issue: several percentage columns in Tables 2 and 3, and the restrictions row, sum well past 100. That should have been caught before submission. The Limitations section is honest about nondeterminism and model version but does not mention the missing baseline or the validity of the critical-thinking measure.\n\nWho is this for? CS educators designing AI-integrated assignments and researchers studying LLM interaction logs. It is a modest but useful data point. The critical-thinking language needs softening — 'some students gave reasons for their choice' rather than 'developed critical thinking' — but the underlying observation is publishable. A serious referee should see it.","headline":"A useful, well-scoped dataset on first-years using ChatGPT inside a real project, but the critical-thinking conclusion overreaches without a baseline or validated rubric.","tokens_in":12219,"tokens_out":2386,"would_cite":false,"duration_ms":20028,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Untrained first-year CS students used ChatGPT successfully: 72.2% of logged groups incorporated its solutions, and about half chose between two AI solutions with stated reasons, evidence of developing critical thinking.","keywords":["large language models","ChatGPT","programming education","interaction log analysis","critical thinking","prompt engineering","first-year students"],"falsifier":"Have a control group of students pick between two ChatGPT solutions by random assignment and write the same style of conclusion; if their 'decided for one solution' rate with a stated criterion approaches the observed 47.2%, the rubric is not distinguishing critical thinking from mere compliance.","tokens_in":11311,"feed_emoji":"🤖","tokens_out":8803,"duration_ms":74414,"temperature":0.7,"pith_summary":"First-year computer science students who received no instruction in prompt engineering still got useful code from ChatGPT: across 37 logged groups (69 students), 72.2% incorporated a ChatGPT-provided solution, fully or partially, into their course project. The paper's central claim is that an exercise template — ask for a solution, ask for an alternative, compare the two, choose one, and justify the choice in a submitted log — fostered critical thinking, with 47.2% of groups explicitly deciding between two AI-generated implementations and explaining their criterion (simpler, more flexible, or easier to understand). The broader point is that structured, low-cost exercise design can channel students' inevitable use of LLMs into a teachable moment about evaluating generated code, without requiring elaborate prompt-training first. The authors also report a post-exercise survey in which 73% of the 52 respondents found the exercise useful.","feed_headline":"72% of first-year students used ChatGPT code in their projects","feed_subtitle":"Even with untrained prompts, most got usable code, and half showed judgment picking between two AI solutions.","key_machinery":"The load-bearing mechanism is the five-step exercise template: state the problem to the LLM, analyze the returned solution, ask for an alternative, compare both, and write a conclusion justifying the choice, all recorded in a log file. This template operationalizes 'critical thinking' as a choice between two generated solutions with an explicit decision criterion (simpler, more flexible, or easier to understand), and it is also what makes the interaction logs analyzable. The paper's classification of prompts (abstraction level, nature, context, restrictions, examples, function signature, second-prompt type, conclusion type, and actual code reuse) is the measurement instrument that turns raw chat logs into the reported percentages.","core_discovery":"The paper reports that in a Data Structures and Algorithms course project, first-year students asked ChatGPT to help parse a large CSV file, following a supplied five-step template: state the problem, analyze the proposed solution, request an alternative, compare, and decide. Analyzing the interaction logs, the authors found that most initial prompts were direct requests for code in Portuguese with little context or examples, yet 72.2% of groups ultimately used the GPT solution (fully or partially) in their projects, and 47.2% wrote conclusions selecting one of two alternative implementations, with decision criteria distributed among simpler/more compact, more flexible/generic, and easier to understand. The authors interpret this as evidence that students with no prompting training can leverage LLMs successfully and can incorporate the generated solutions, and that the act of choosing between two alternatives with a written rationale is a form of developing critical thinking. The survey adds that 84.6% of respondents said they interacted with GPT, 69.2% would have used it even without the assignment, and 73% found the exercise useful.","pith_inferences":["A natural extension the paper does not run is a control condition: have students choose between two AI solutions by coin flip and write the same style of conclusion. If the rate of 'decided for one solution with a criterion' matches the observed 47.2%, the rubric is measuring compliance, not critical thinking.","The paper's own limitation that GPT behavior changes over time and the model version was uncontrolled means the numerical results are a snapshot; repeating the same log analysis in later terms would test whether the findings are stable or model-era-specific.","The reported 'simpler, more flexible, easier to understand' criteria could be connected to a learning outcome: a follow-up study could ask students who completed the template to modify or debug their chosen solution, testing whether the act of comparison actually improved code comprehension.","The template's log files are a ready-made dataset for measuring prompt sophistication; a future analysis could correlate the presence of examples, context, or function signatures with whether the chosen solution passed the automated tests."],"forward_implications":["The five-step template (ask, analyze, ask for an alternative, compare, decide) can be dropped into other programming courses as a graded, low-cost exercise that makes LLM use visible rather than underground.","Because 72.2% of logged groups used ChatGPT's code, course instructors should assume that unmonitored AI use is already widespread and design assessments around code evaluation skills instead of trying to prevent access.","The 38.9% of groups who skipped the requested alternative suggests that merely instructing students to compare solutions is not enough; the template would benefit from an explicit sub-question or rubric that forces the comparison step.","The survey result that 30.8% would not have used GPT without the assignment implies that a mandatory exercise can bring reluctant students into a supervised first interaction with generative AI."],"supporting_citations":[{"why":"Documents how novice programmers struggle with and are wary of code-generation tools, the baseline this study's success rate is contrasted against.","marker":"Prather et al., 2023"},{"why":"Provides the prior methodology for analyzing student interactions with ChatGPT and the prompt-problem exercise format the template builds on.","marker":"Denny et al., 2023b"},{"why":"Analyzes learner interaction logs with a GPT-based code generator and supplies the concepts of self-regulation and over-reliance the conclusion discussion extends to university students.","marker":"Kazemitabaar et al., 2023"},{"why":"Contributes the practice of asking for multiple GPT solutions and comparing them, which is the core of the exercise template.","marker":"Treude, 2023"},{"why":"Supports the recommendation that supplying examples improves LLM responses, grounding the paper's call for prompt training.","marker":"Brown et al., 2020"},{"why":"Shows ChatGPT behavior changes over time, which the paper cites in its limitations to caution against generalizing the results.","marker":"Chen et al., 2023"}],"fun_headline_variants":["72% of first-years used ChatGPT code, even with sloppy prompts","Nearly half of first-years judged between two AI code solutions","Untrained prompts didn't stop 72% of students in AI code test","First-years show critical thinking when choosing ChatGPT code","Most students used GPT code; half actually compared options"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's critical-thinking conclusion rests on equating a written choice of one ChatGPT solution over another — with a reason such as 'simpler', 'more flexible', or 'easier to understand' — with demonstrated critical thinking; if that equivalence fails, the educational-judgment claim weakens even though the 72.2% usage-rate finding stands on its own.","fun_headline_variants_meta":{"raw":{"variants":["72% of first-years used ChatGPT code, even with sloppy prompts","Nearly half of first-years judged between two AI code solutions","Untrained prompts didn't stop 72% of students in AI code test","First-years show critical thinking when choosing ChatGPT code","Most students used GPT code; half actually compared options"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000603,"raw_usage":{"total_tokens":2811,"prompt_tokens":939,"completion_tokens":1872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1787}},"tokens_in":555,"tokens_out":1872,"duration_ms":11594,"temperature":1.0,"reasoning_tokens":1787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:45:46.272441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a control group of students pick between two ChatGPT solutions by random assignment and write the same style of conclusion; if their 'decided for one solution' rate with a stated criterion approaches the observed 47.2%, the rubric is not distinguishing critical thinking from mere compliance.","supporting_citations":[],"review_version":1}