{"id":"b1114536-444d-45c0-a322-18f2d72d9e7b","arxiv_id":"2412.11419","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Computing educators propose redesigning take-home assignments to include and assess student use of generative AI, while shifting educator focus to metacognitive skill development.","lead":"This computing education paper reports how the authors integrated generative AI into university programming courses and proposes two strategies: redesign take-home assignments to assess student process with AI, and shift educator roles toward teaching metacognition. It is an experience and position paper, not a controlled study, useful to educators planning AI-aware assessment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing premise — that process-oriented, GenAI-incorporated assessment deters superficial use and improves learning — is asserted from experience, not demonstrated; a controlled comparison is needed.","rationale":"The paper is an honest experience report, not a deception; the authors explicitly call for future research. The reader's CONDITIONAL verdict is appropriate. My stress-test identifies the same load-bearing premise as the reader's weakest_assumption: the effectiveness and robustness of process-oriented assessment is asserted, not demonstrated. This is the correct central concern because the entire prescriptive claim depends on it. I agree with the reader and see no reason to change the verdict. The proposed concrete test — a randomized comparison against secure exam performance, plus gaming and workload metrics — would directly settle whether the premise holds. No internal inconsistency or methodological failure was found; the concern is purely about the strength of evidence for a practice-oriented recommendation.","tokens_in":12301,"tokens_out":2500,"duration_ms":23721,"concrete_test":"Run a randomized controlled trial in a large CS1 course: one section receives the redesigned process-oriented assignments (GenAI allowed, graded on prompts, reflections, and critiques); the other receives traditional product-based assignments (GenAI prohibited). Compare final secure invigilated exam scores common to both, plus indicators of superficial engagement (e.g., time-on-task, reflection originality, inter-submission code similarity) and grading workload (hours per student, inter-rater reliability of the process rubric). If the process-oriented section does not significantly outperform on the secure exam, or if shallow-but-plausible reflections are indistinguishable from genuine ones, the central recommendation loses its empirical basis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central recommendation (Strategy 1, §3.1) is that take-home assignments should be redesigned to assess students on their process of using GenAI (prompts, reflections, critiques) rather than on the final product. The only support is anecdotal observation (§2.2) and self-cited pilot studies of individual activities (Prompt Problems, BugSpotter, etc.), none of which compare the recommended assessment model against traditional product-based assessment on the outcome that matters: whether students actually learn, as measured by secure assessment. The authors concede in §3.1 that 'concrete methods to implement these strategies must still be developed, evaluated and deployed at scale,' and they acknowledge an increased grading workload, proposing only a speculative 'proxy' tool to automate grading. Without evidence that process artifacts cannot be gamed (e.g., students can prompt GenAI to produce plausible reflections), or that grading such artifacts is feasible at scale with acceptable reliability, the two primary actions remain an unverified prescription. The claim that the redesigned question in Figure 4 'lowers the likelihood of students copying and pasting' is an intuition, not a measured result. If process measures are gameable or grading is impractical, the central claim fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is an experience report from four educators at a large research-intensive institution, describing how they have adapted their computing courses since the widespread adoption of generative AI. It recounts observations of students using GenAI in take-home assignments, summarizes six LLM-based classroom activities developed by the authors and their collaborators (Prompt Problems, BugSpotter, EiPL, student-generated analogies, PuzzleMakerPy, and CodeHelp), and then proposes two primary actions for the computing education community: (1) redesign take-home assignments to incorporate GenAI use and assess the process of using GenAI rather than only the final product, and (2) refocus the educator role on metacognitive skills such as critical thinking and self-evaluation. The paper explicitly frames itself as sharing experiences and calls for further research to evaluate the proposed strategies.","tokens_in":12499,"tokens_out":4273,"duration_ms":41890,"significance":"The paper addresses a timely and important question: how should computing education respond to the ubiquity of GenAI? Its strengths are its concreteness and transparency. Figures 3–7 give a practical, reproducible example of redesigning a CS1 question, and the proposed emphasis on secure invigilated assessments alongside GenAI-permitted assignments is a balanced position that aligns with a growing body of work in the field. The call for empirical evaluation is appropriate. However, the evidence base is largely anecdotal and self-referential: the observations in Section 2.2 are not systematically collected, and the six activities in Section 2.3 are prior works by the same group. If the paper is positioned as a position/experience report, these limitations are acceptable if explicitly acknowledged; if it is positioned as an empirically validated proposal, the evidence is not sufficient.","major_comments":[{"comment":"The observation that take-home assignment scores were much higher than in-person invigilated scores, and that a few students struggled with print statements despite completing prior assignments, is used to motivate the need for redesigned assignments. Yet these are anecdotal observations with no systematic data, no controls for cohort differences, question difficulty, or prior preparation, and the authors themselves concede that 'there are many reasons that students struggle.' The argument that GenAI misuse caused the score gap is therefore not established. The authors should either provide quantitative evidence (e.g., matched pre-/post- comparisons or analysis of the same cohort on GenAI-permitted vs. secure assessments) or explicitly frame this as a motivating hypothesis rather than a demonstrated cause.","section":"Section 2.2"},{"comment":"The claim that the redesigned question 'lowers the likelihood of students copying and pasting without reading and understanding the responses' is asserted as an intuition, not a measured result. A student seeking to bypass learning could prompt a GenAI tool to generate not only the code but also the summaries, comparisons, and critiques requested in the four-step guide. Process artifacts are not inherently harder to fake than final products. The authors should soften this claim or provide evidence from their deployments that students actually engaged with the steps as intended.","section":"Section 3.1, Figure 4"},{"comment":"The feasibility of grading process-based assessments at scale is unresolved. The paper acknowledges the increased grading workload and proposes 'a proxy for students to access GenAI tools that store interactions that may be used to automate grading purposes,' but this is speculative and no design, reliability analysis, or comparison with traditional grading is given. Since Strategy 1 recommends broad adoption of process-oriented assignments, the lack of a validated grading mechanism is a load-bearing gap. The authors should either report on a concrete pilot of such a proxy or present this as an open research problem rather than part of the recommended course of action.","section":"Section 3.1"},{"comment":"The paper cites Prather et al.'s 'Widening Gap' finding that struggling students face persistent metacognitive difficulties when using GenAI, yet it does not address how the proposed process-oriented assignments and metacognition emphasis would help rather than further disadvantage these students. If struggling students are the ones most likely to use GenAI superficially, a redesign that relies on self-reported prompts, reflections, and critiques may compound the gap unless it is scaffolded. The authors should discuss this tension explicitly and explain what course-level support would be needed.","section":"Sections 2.2 and 3.2"},{"comment":"The six activities presented as evidence that GenAI can be integrated into classroom learning (Prompt Problems, BugSpotter, EiPL, analogies, PuzzleMakerPy, CodeHelp) are all prior works by the same group of authors. This is understandable for an experience report, but the manuscript currently reads as if these activities are established effective practices. The authors should clearly distinguish 'we have experience implementing this' from 'this is shown to improve learning outcomes.' Adding an explicit statement about the self-referential nature of the evidence would help readers calibrate the strength of the recommendations.","section":"Section 2.3"}],"minor_comments":[{"comment":"The phrase 'scaffolding rather then simply producing solutions' contains a typo; 'then' should be 'than'.","section":"Section 3.1"},{"comment":"The tool name 'Coderunner' appears with inconsistent capitalization; elsewhere in the paper it is spelled 'CodeRunner' (e.g., Section 2.1). Please use one spelling consistently.","section":"Section 3.1"},{"comment":"The sentence 'We are aware that there are many reasons that students struggle and the difficulties observed are not all necessarily caused by GenAI' is grammatically awkward. Consider rewording to 'We are aware that students struggle for many reasons and that not all observed difficulties are necessarily caused by GenAI.'","section":"Section 2.2"},{"comment":"The redesigned question assumes students have access to ChatGPT. The paper should mention that equivalent LLM tools can be used or that instructors should provide alternatives to address equity and access concerns.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a position/experience paper rather than an empirical study. If the venue accepts such papers, the revision should focus on aligning the strength of the claims with the evidence, adding explicit limitations, and addressing the grading-feasibility and student-equity concerns. The self-citation pattern is consistent with an experience report but should be surfaced transparently. If the venue requires empirical validation, the paper falls short and would be more suited to a call-for-research venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-written practitioner paper from people who have genuinely used GenAI in large CS courses, and it’s worth reading for the concrete activities and the sample redesigned assignment. But the two headline recommendations — assess process rather than product, and teach metacognition — are presented as proposals grounded in experience, not as validated results. The paper says so itself, which I respect.\n\nWhat’s actually new is thin. The six activities (Prompt Problems, BugSpotter, EiPL, analogies, PuzzleMakerPy, CodeHelp) were published in earlier papers by the same group. The two strategies echo Cao & Dede and the ITiCSE 2023 working group report. The genuinely new artifact is Figure 4, the redesigned CS1 question that walks students through using ChatGPT to identify concepts, generate a solution, and critique differences. That’s a useful teaching template, and the surrounding discussion of secure assessment as a complement is sensible.\n\nThe soft spots are exactly where the stress-test lands. The claim that process-oriented assignments 'lower the likelihood of students copying and pasting' is an intuition, not a measured result. The §2.2 evidence — higher take-home scores, students who can’t write print statements — is anecdotal, and the authors acknowledge confounders. They also concede that 'concrete methods to implement these strategies must still be developed, evaluated and deployed at scale,' and the grading workload problem is real, with only a speculative proxy-tool suggestion. None of this is fatal for an experience report, but it means the paper’s central recommendation is a hypothesis, not a finding.\n\nThe self-citation pattern is heavy but not unfair: the activities are theirs, and the original papers contain actual evaluations (BugSpotter had expert review, the analogies study had 350+ students). The paper is honest about its status. I’d send this to peer review — it’s exactly the kind of practice-oriented synthesis that ACE should publish, and it will help other educators. Just don’t cite it as evidence that process assessment works; cite it as a design proposal with concrete examples.","headline":"A clear, honest experience report from a group that has actually deployed GenAI activities, but the two headline strategies are a synthesis of prior work plus anecdote, not an empirical validation.","tokens_in":13024,"tokens_out":1781,"would_cite":true,"duration_ms":15960,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grade how students use AI, not just the code they submit","keywords":["Generative Artificial Intelligence","GenAI","Computing Education","Assignments","Metacognition","Assessments","Prompt Engineering","AI Teaching Assistants"],"falsifier":"A controlled comparison in an introductory programming course: one cohort completes process-oriented assignments that incorporate GenAI (prompts, comparisons, critiques) while a matched cohort completes traditional product-only assignments, both followed by the same secure invigilated exam. If the process-oriented cohort shows no higher exam performance or metacognitive skill, or if students in that cohort can be shown to fabricate convincing process artifacts without understanding the code, the central claim fails.","tokens_in":12106,"feed_emoji":"🎓","tokens_out":5166,"duration_ms":41082,"temperature":0.7,"pith_summary":"This paper argues that the response to generative AI in computing education should be neither prohibition nor passive acceptance, but a deliberate shift in what is assessed and what educators teach. Drawing on several years of teaching first-year programming and upper-level courses, the authors claim that take-home assignments should be redesigned to require students to collaborate with tools like ChatGPT in a visible process — writing prompts, critiquing generated code, and reflecting on differences — rather than submitting only a final solution. They further argue that the educator's central job becomes cultivating metacognitive skills such as critical thinking, self-evaluation, and fact-checking of AI output. The paper's proposals matter because they offer a credible middle path between unreliable AI-detection and the threat of meaningless take-home work.","feed_headline":"Grade how students use AI, not just the code they submit","feed_subtitle":"Two strategies: redesign take-home assignments around visible AI collaboration, and make metacognition the educator's core job.","key_machinery":"The load-bearing mechanism is the process-oriented assignment: a take-home task redesigned so that students must externalize their interaction with GenAI — submitting prompts, intermediate explanations, comparisons of solutions, and self-critiques — and are graded on that record of engagement rather than on the final artifact. This mechanism is supported by a stable of concrete activities the authors deployed, such as Prompt Problems, where students write natural-language prompts that generate working code; BugSpotter, where students design failing test cases for AI-generated buggy code; and CodeHelp, an AI assistant that gives scaffolding feedback without revealing solutions. These activities carry the argument by showing that process can be elicited, observed, and automatically or semi-automatically assessed at scale.","core_discovery":"The paper's central claim is that the learning crisis created by generative AI in computing education is best addressed by redefining both assessment and the educator's role. For assessment, the authors propose that at least some take-home assignments should be redesigned to incorporate GenAI use and be graded on the process — the prompts students write, the comparisons they make between their own and AI-generated code, and their critiques of AI output — rather than on the final product alone. For teaching, they propose that educators should emphasize metacognition: teaching students how to craft effective prompts, how to verify AI-generated information, and how to monitor their own understanding. Secure, invigilated assessments remain necessary for accreditation, but the design of learning tasks should treat GenAI as an allowed, scaffolded tool. The argument is grounded in the authors' classroom experiences, including Prompt Problems, AI-generated debugging exercises, code comprehension through prompting, and a guard-railed AI teaching assistant, which they present as existence proofs that such redesigned activities are feasible in large courses.","pith_inferences":["If process-oriented assessment becomes the norm, a new class of academic-integrity risk appears: students may fabricate plausible process artifacts (prompts, reflections, critiques) without genuinely engaging, so the approach would need its own validation against secure summative measures.","The same redesign logic plausibly extends beyond computing to any discipline where GenAI can produce acceptable products, such as writing, data analysis, and design, though the specific process artifacts would differ.","The paper's proposal implies a research agenda with a concrete test: comparing learning gains and metacognitive growth between students in process-oriented GenAI-integrated courses and those in traditional product-only courses, using secure assessments as the outcome measure.","The emphasis on metacognition suggests that training educators to teach prompting and fact-checking may be as important as training students, and could be studied as a teacher-professional-development problem."],"forward_implications":["Take-home assessments in programming courses will need to be redesigned so that using GenAI is explicit and visible, with the student's prompts, comparisons, and critiques forming part of the graded submission.","Secure, in-person, GenAI-prohibited assessments will remain a necessary component for credentialing, even as assignments openly incorporate GenAI.","Educators will spend more time teaching prompt crafting, fact-checking, and how to evaluate AI-generated code, and less time on delivering routine explanations that students now obtain from GenAI.","Grading process artifacts at scale will require new tools, such as proxies that record student interactions with GenAI for automated or semi-automated assessment.","Courses should teach students to request explanations rather than full solutions from GenAI, and to verify generated information against trusted resources."],"supporting_citations":[{"why":"supplies the survey finding that educators favour shifting to invigilated assessments and process-based grading, the position this paper extends.","marker":"[33]"},{"why":"documents that struggling students face compounded metacognitive difficulties with GenAI tools, the motivation for the metacognition strategy.","marker":"[34]"},{"why":"proposes assessing process rather than product in the GenAI era, which the paper aligns with.","marker":"[3]"},{"why":"defines the Prompt Problems exercise format used to teach prompt-writing as a core instance of process-oriented assessment.","marker":"[6]"},{"why":"provides the automated marking tool used for secure, GenAI-prohibited assessment components.","marker":"[24]"},{"why":"shows that GenAI text detection is unreliable, supporting the case for redesign over prohibition.","marker":"[30]"},{"why":"argues that assignments should be adapted for unrestricted GenAI use, a premise of the redesign proposal.","marker":"[28]"},{"why":"describes the guard-railed AI teaching assistant that gives scaffolding without revealing solutions.","marker":"[7]"}],"fun_headline_variants":["Grade the AI process, not just the final code","Shift grading from product to AI collaboration","Redefine educators as metacognition coaches","Make AI use visible, then grade the thinking","Redesign assignments to assess AI-assisted process"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole proposal rests on the premise that grading the process — the prompts, reflections, and critiques students produce while using GenAI — will actually reflect and encourage genuine learning, and that students cannot cheaply fake that process the way they could copy a final solution.","fun_headline_variants_meta":{"raw":{"variants":["Grade the AI process, not just the final code","Shift grading from product to AI collaboration","Redefine educators as metacognition coaches","Make AI use visible, then grade the thinking","Redesign assignments to assess AI-assisted process"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1225,"prompt_tokens":976,"completion_tokens":249,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":181}},"tokens_in":592,"tokens_out":249,"duration_ms":2894,"temperature":1.0,"reasoning_tokens":181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:56:39.526084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled comparison in an introductory programming course: one cohort completes process-oriented assignments that incorporate GenAI (prompts, comparisons, critiques) while a matched cohort completes traditional product-only assignments, both followed by the same secure invigilated exam. If the process-oriented cohort shows no higher exam performance or metacognitive skill, or if students in that cohort can be shown to fabricate convincing process artifacts without understanding the code, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the survey finding that educators favour shifting to invigilated assessments and process-based grading, the position this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"proposes assessing process rather than product in the GenAI era, which the paper aligns with."},{"cited_title":"Becker, and Brent N","cited_arxiv_id":null,"evidence_quote":"defines the Prompt Problems exercise format used to teach prompt-writing as a core instance of process-oriented assessment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the automated marking tool used for secure, GenAI-prohibited assessment components."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"argues that assignments should be adapted for unrestricted GenAI use, a premise of the redesign proposal."}],"review_version":1}