{"id":"0b8eb9d8-1170-4423-98b7-e96185711c13","arxiv_id":"2607.05034","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Novices solving dialogue-based Prompt Problems omit key specification details and recover mainly by clarifying intent, not by tracing generated code or tests.","lead":"A large CS1 study of 900+ students shows that dialogue-based natural-language Prompt Problems feel easier and more enjoyable than traditional coding, but novices routinely omit return types, argument names, and other key details. When AI code fails, they mostly re-clarify intent and re-read the problem rather than trace code or inspect tests.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged self-report and researcher-coded-mistake premises.","rationale":"The paper's strongest claim is carefully scoped as a large-N descriptive classroom study of perceptions, coded prompt omissions, and self-reported recovery strategies under a specific dialogue-based Prompt Problems interface. The data (N≈900, Tables 1–4, Figures 4–5, Likert Ref3) support those descriptive statements. The reader's weakest_assumption correctly identifies the two places where the interpretation is least secure: researcher-imposed 'missing element' codes and optional self-report strategies. No additional load-bearing flaw (e.g., statistical error, unacknowledged confound that would reverse the ease or omission findings, or contradiction between logs and claims) is present. Therefore the existing CONDITIONAL verdict with HIGH confidence for the descriptive results is already the right calibration; no adjustment is warranted.","tokens_in":24363,"tokens_out":500,"duration_ms":4611,"concrete_test":"Independently re-code a stratified 10% subsample of the 1,286 first-unsuccessful prompts using only the problem I/O specs (no researcher list of 'required elements'); if the top-three omission frequencies in Table 3 shift by more than 15 percentage points or the co-occurrence pattern in Table 4 collapses, the 'omission-dominated mistakes' claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption already isolates the two softest points that actually support the strongest claim: (1) that absence of researcher-defined elements (return type, argument names, expected output, etc.) in first unsuccessful prompts (Table 3, §3.3) are the pedagogically load-bearing mistakes rather than artifacts of the coding scheme or the visual-spec interface, and (2) that optional Ref4 self-reports (n=174) faithfully capture recovery behavior rather than post-hoc rationalization. No stronger internal inconsistency or unstated assumption is required for the descriptive claims to hold as written. The paper is transparent about the lack of A/B comparison, objective learning outcomes, and log-validated strategies (§5.4), so the CONDITIONAL verdict already correctly reflects those limits. The central descriptive findings (ease perceptions, omission frequencies, clarification-heavy reported strategies) are directly supported by the reported N, tables, and themes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper reports a large-scale observational study of dialogue-based Prompt Problems in a CS1 C-programming course (N>900). Students wrote natural-language prompts to GPT-4o mini for six visual-spec tasks across two lab batches, then reflected on experience (Ref1–3) and recovery strategies (Ref4). Thematic analysis of 200 reflections and coding of 1,286 initially unsuccessful first prompts show that students generally perceived prompting as easier, more enjoyable, and better for problem-solving than traditional coding (Likert M=3.80); the most frequent mistakes were omissions of return type, argument names, required inputs, expected output, and functionality details (Table 3); and reported recovery focused more on clarifying intent and re-reading the problem depiction than on tracing generated code or inspecting test cases (Figure 5). Results are interpreted through Cognitive Load Theory as evidence of beneficial offloading of syntax.","tokens_in":24646,"tokens_out":1063,"duration_ms":14474,"significance":"If the descriptive findings hold, the work supplies the first systematic catalog of novice prompt-level omissions and self-reported refinement strategies for dialogue-based Prompt Problems, directly informing curriculum design, scaffolding (e.g., templates, Parsons-style prompt assembly), and tool features as GenAI becomes standard in CS1. Strengths include the large authentic enrollment sample, dual-batch design with success-rate and interaction logs (Tables 1–2), inductive thematic saturation at 200 responses, and exhaustive coding of half of all incorrect first prompts. These baselines are timely and actionable even without causal claims.","major_comments":[{"comment":"§3.3 and Table 3: The mistake taxonomy treats absence of researcher-defined elements (return type, argument names, order, expected output, etc.) as errors even when the tool does not enforce names and the model can often infer them. This coding scheme is load-bearing for the claim that “the most common mistakes are related to the omission of key details,” yet no inter-rater reliability statistic is reported (only consensus discussion) and no ablation shows that these absences, rather than other prompt properties, actually caused the incorrect code. A sensitivity analysis or explicit justification against the visual-spec interface is needed.","section":"§3.3, Table 3"},{"comment":"§4.4 and Figure 5 (RQ3): Strategy themes rest entirely on optional self-reports (n=174 coded) with no triangulation against the logged messages, code executions, or reset events. The central claim that students “focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases” therefore risks post-hoc rationalization or social-desirability bias. At minimum, a sample of log-validated trajectories should be reported or the claim explicitly scoped to “reported strategies.”","section":"§4.4, Figure 5"},{"comment":"§5.1–5.2: Cognitive Load Theory is invoked to interpret ease and offloading, yet no direct CLT measure (subjective or dual-task) is collected. The claim that prompting frees resources for problem-solving therefore remains an untested interpretive overlay rather than an empirical result; either add a brief CLT instrument or soften the causal language linking perceptions to germane load.","section":"§5.1–5.2"}],"minor_comments":[{"comment":"Table 2 reports means for conversations/messages only among students with an initially incorrect prompt; a parallel column for all attempters would clarify selection effects.","section":"Table 2"},{"comment":"Figure 4 stacks Ref1/Ref2 counts but does not report total unique respondents per theme; adding n or percentages would aid interpretation.","section":"Figure 4"},{"comment":"§3.1: The exact system prompt / temperature settings for GPT-4o mini are not stated; reproducibility would benefit from a short appendix note.","section":"§3.1"},{"comment":"Several self-citations to prior Prompt Problems work appear; a brief sentence distinguishing the present dialogue-based contribution from the earlier zero-shot studies would help readers.","section":"§2.3"},{"comment":"Typo/consistency: “Prompt Programming” tool name vs. “Prompt Problems” activity; standardize early.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid, timely empirical contribution that fits ICER well. The two softest points (ad-hoc mistake coding and unvalidated self-reports) are already flagged by the authors in §5.4; requiring them to tighten those claims or add modest validation should be sufficient. No novelty or citation-pattern concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the first solid classroom catalog of what CS1 students actually leave out of dialogue prompts and how they say they recover when the model fails. That is the useful part.\n\nWhat is new is scale plus content coding. Prior Prompt Problems work was mostly zero-shot and engagement-focused. Here they put ~900 students through six dialogue tasks with GPT-4o mini, coded 1,286 unsuccessful first prompts for missing elements (return type, argument names, required inputs, expected output, etc.), and thematically coded reflections on ease and recovery. The descriptive picture is clear: students report the tasks as easier and more problem-solving-oriented than traditional coding (Likert mean 3.80, tested); omissions dominate; and reported recovery leans toward clarifying intent and re-reading the visual spec more than tracing code or inspecting tests. Tables 2–4 and the theme figures make the frequencies and co-occurrences easy to use. Methods are transparent: dual-batch design, saturation at 200 reflections, ethics approval, and an honest limitations section.\n\nSoft spots are real but already flagged by the authors and do not sink the descriptive claims. Mistake codes are researcher-defined absences relative to a complete specification, not independently validated “load-bearing” errors; argument names were not required by the autograder. Ref4 strategies are optional self-report (n=174 coded), not log-validated behavior. No A/B vs zero-shot, no learning-outcome measures, no demographics. Cognitive Load Theory is an interpretive lens, not a tested mechanism. Those limits make the paper a baseline, not a causal or pedagogical-efficacy result.\n\nCitation pattern and data presentation look clean; self-cites to prior Prompt Problems work are appropriate. Who it is for: CS education and AI-in-education people designing prompt scaffolds, templates, or completeness checks. It deserves a serious referee. I would cite the omission frequencies and strategy themes when talking about novice prompting. Engage with it; do not treat it as more than a well-executed descriptive study.","headline":"Large-N CS1 baseline on dialogue Prompt Problems: students like it, omit key details, and mostly clarify rather than read code or tests.","tokens_in":25224,"tokens_out":509,"would_cite":true,"duration_ms":6181,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Novice programmers find natural-language prompting easier than coding, but mostly fail by omitting key details and then refine by clarifying intent rather than tracing code.","keywords":["natural language programming","code-generating AI","Prompt Problems","student perceptions","prompt mistakes","debugging strategies","CS1","cognitive load"],"falsifier":"Re-run the same problems with a condition that forces students to mark or correct the generated code line-by-line (or that withholds the visual problem depiction after the first failure) and measure whether omission rates and reported strategy frequencies reverse.","tokens_in":25296,"feed_emoji":"💬","tokens_out":752,"duration_ms":8028,"temperature":0.7,"pith_summary":"This paper studies how more than 900 CS1 students solve computational tasks by writing natural-language prompts for a code-generating AI instead of writing code themselves. Students generally reported that these dialogue-based Prompt Problems felt easier, more enjoyable, and better aimed at problem-solving than traditional coding exercises, because syntax load was removed. The dominant mistakes in first unsuccessful prompts were omissions of required details such as function return type, argument names, required inputs, and expected output, consistent with over-reliance on the model to fill gaps. When generated code failed, students said they mainly clarified their intent and re-read the problem depiction, far less often tracing the code or inspecting test cases. The work supplies a concrete map of what novices leave out and how they try to recover, so that instructors can decide what scaffolding is still needed.","feed_headline":"Students prompt AI more easily than they code, but omit key details","feed_subtitle":"When code fails they clarify intent more than they trace the program or read tests","key_machinery":"Dialogue-based Prompt Problems: students see a visual input-output specification and iteratively prompt a code model (with no direct code editing) until the generated program passes the hidden tests; unsuccessful first prompts and optional reflections are then coded for missing elements and recovery strategies.","core_discovery":"In a large CS1 deployment of dialogue-based Prompt Problems, students perceived natural-language prompting as easier, more enjoyable, and more focused on problem-solving than traditional coding; their most frequent initial-prompt errors were omissions of key specification details; and their reported recovery strategies centered on clarifying intent and re-examining the problem depiction rather than tracing generated code or examining test cases.","pith_inferences":["The same omission pattern may appear in professional ‘vibe-coding’ workflows, suggesting a shared need for lightweight completeness scaffolding.","If cognitive offloading is the intended benefit, later studies should measure actual germane load and transfer to unaided coding rather than only self-reported ease.","Pairing Prompt Problems with short code-tracing micro-tasks after each failure could convert the dominant recovery strategy into one that also builds code-comprehension skill."],"forward_implications":["Instructors can expect most first-prompt failures to be missing return types, argument names, and expected outputs, so early feedback can target those omissions.","Curriculum designers can treat prompt-writing as a lighter-load entry to problem-solving while still planning explicit practice in code tracing and test-case reading.","Tool builders can add completeness checks or progressive templates before code generation to reduce over-reliance on model inference.","Dialogue-based prompting will not automatically teach traditional debugging habits unless the interface or pedagogy requires attention to generated code and failing tests."],"fun_headline_variants":["Novices find AI prompting easier than coding yet skip crucial details","Students see prompting easier than code but omit key problem details","Prompting feels easier than coding as CS1 students miss key specs","CS1 students omit details in AI prompts, clarify intent over tracing","Students prompt AI easily then recheck problems not code or tests"],"cache_read_input_tokens":14976,"weakest_assumption_plain":"The claim rests on treating coded absences of required elements in first prompts, plus self-reported recovery strategies from optional reflections, as faithful measures of the pedagogically important mistakes and debugging behaviors.","fun_headline_variants_meta":{"raw":{"variants":["Novices find AI prompting easier than coding yet skip crucial details","Students see prompting easier than code but omit key problem details","Prompting feels easier than coding as CS1 students miss key specs","CS1 students omit details in AI prompts, clarify intent over tracing","Students prompt AI easily then recheck problems not code or tests"]},"model":"grok-4.5","effort":"low","cost_usd":0.004488,"raw_usage":{"total_tokens":1264,"prompt_tokens":726,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":44880000,"prompt_tokens_details":{"text_tokens":726,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":464,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":726,"tokens_out":74,"duration_ms":3931,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T09:46:06.856015+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same problems with a condition that forces students to mark or correct the generated code line-by-line (or that withholds the visual problem depiction after the first failure) and measure whether omission rates and reported strategy frequencies reverse.","supporting_citations":[],"review_version":1}