{"id":"885503c6-3ca4-4cf5-b36a-55c185405398","arxiv_id":"2607.16494","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A personalized puzzle generated from a student's own code can reveal when they may not understand what they submitted.","lead":"This paper describes p3, a system that turns a programmer's own code into a short puzzle they must solve to prove they understand it. In a pilot with 111 students, some people could not reassemble their own code, suggesting a new way to catch AI-generated or poorly understood work.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pilot's headline inference relies on an unvalidated proxy: puzzle failure is not yet shown to measure understanding of one's own code, only reconstruction of a short canonical Dockerfile; Section 5 explicitly defers this validation.","rationale":"The reader's weakest_assumption correctly identifies the proxy assumption as the load-bearing point. The pilot is a reasonable proof-of-concept, but the abstract's claim that the study 'shows' p3 can identify gaps in understanding of one's own code requires that puzzle performance measure understanding rather than general reconstruction ability, rote memory, or UI issues. The authors explicitly postpone validation of this assumption ('Future work is needed to correlate puzzle results with code understanding'), so the current evidence is insufficient for an unqualified claim. This is not a fatal flaw; the paper is transparent about the limitation and proposes follow-up work. The CONDITIONAL verdict is appropriate, and my stress-test does not move it. I agree with the reader's identification of the same weakest assumption, and my proposed concrete test would settle whether the concern actually lands by comparing puzzle outcomes against an independent rubric-based assessment of student understanding.","tokens_in":4186,"tokens_out":6492,"duration_ms":76505,"concrete_test":"Use the follow-up TA code-review study the authors mention in Section 5: have TAs independently score each student's line-by-line understanding of their own Dockerfile (blind to puzzle outcome), then compute the association between p3 puzzle failure and low understanding score. Specifically report sensitivity/specificity or a logistic regression of puzzle success on rubric score controlling for a general Dockerfile-knowledge baseline. If puzzle failure does not significantly predict low rubric scores beyond general Dockerfile knowledge, the proxy assumption fails and the pilot's central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference in Section 4 — that the 22.4% of Group-Own students who failed their puzzle 'may not understand their own code' — depends entirely on the Section 5 assumption that 'an SWE's ability to reconstruct or fix their program (as measured by the puzzle results) is good proxy to their understanding of that program.' That assumption is not validated in the paper. The artifact is a 7-line Dockerfile, close to a canonical template; success could come from general Dockerfile knowledge or rote recall of recently written lines, and failure could come from time pressure or trivial ordering mistakes, without any systematic relation to semantic understanding of the submitted artifact. The Group-Ref condition (66.0% solved) shows that a substantial fraction of students can reconstruct the reference Dockerfile without it being their own code, so the own-code advantage of only ~12 points is weak evidence that the instrument is specifically measuring personalized understanding. Because the authors explicitly defer the correlation study to future work, the abstract's 'shows' overstates what the pilot can establish. This does not make the method useless, but it makes the headline claim conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Personalized Probing Puzzles (p3), a method that generates Parsons-puzzle-style reconstruction tasks from a student's own submitted code, and reports a pilot in a graduate cloud computing course (n=111). Students reconstructed either their own Dockerfile (Group-Own, 58 students) or the reference Dockerfile (Group-Ref, 53 students). The headline results are that 77.6% of Group-Own and 66.0% of Group-Ref solved their puzzle; the paper interprets failures as indicating that a substantial fraction of students may not understand their own code. The authors also report post-puzzle survey responses on self-confidence/effort and anecdotal statements from a few students. The paper presents p3 as automatically generated, asynchronously administered, and quick to complete, and it explicitly defers validation of the proxy between puzzle-solving and code understanding to future work.","tokens_in":4488,"tokens_out":4438,"duration_ms":51010,"significance":"If the proxy were validated, p3 would address a timely and important problem in post-AI computing education: assessing whether students understand code they submit as their own. The idea of generating personalized assessments from personal artifacts is novel and practical, and the pilot provides a useful first data point at scale (111 consenting students). The paper is honest about its two key assumptions — the reconstruction-to-understanding proxy and the absence of UI/UX failure — which is a strength. However, the current evidence is only suggestive: the central inference from puzzle failure to an understanding gap is definitionally tied to the unvalidated proxy, and the article itself states that the correlation study is future work. The paper should therefore be read as a feasibility demonstration, not as a validation of the diagnostic claim.","major_comments":[{"comment":"The abstract's claim that the pilot 'shows that p3 can help identify potential gaps' is stronger than the evidence supports, because the inference depends entirely on the Section 5 assumption that reconstruction ability is a good proxy for understanding. The paper explicitly defers validation of that proxy. Moreover, the Group-Ref rate of 66.0% demonstrates that a majority of students who did not author the reference Dockerfile can reconstruct it from general Dockerfile knowledge, so the own-code advantage of only ~11.6 percentage points does little to isolate understanding of one's own artifact. Please either temper the abstract and Section 4 to 'may help identify potential gaps' and frame the pilot as feasibility, or add a validation of the proxy (e.g., correlation with code review scores, think-aloud data, or effort/telemetry).","section":"Abstract and Section 4"},{"comment":"The headline percentages (77.6% vs. 66.0%; 43.1% vs. 35.8%) are purely descriptive: no confidence intervals, effect sizes, or significance tests are reported. More importantly, there is no independent baseline for what failure rate a reconstruction task of this difficulty should produce. The 7-line canonical Dockerfile is close to a tutorial template, and a 22.4% failure rate may reflect time pressure, distractor design, or rote-recall demands rather than a semantic understanding gap. The Group-Ref condition is a natural baseline, but it is not used to calibrate the interpretation. Please report uncertainty and discuss the difficulty/ceiling of the task when interpreting failure rates.","section":"Section 4, bullet 1"},{"comment":"The UI/UX assumption is unresolved and load-bearing. The puzzle includes distractors, possibly removed blocks, a 'No Solution' button, and a 5-minute timer; any of these could cause failure independently of understanding. The paper acknowledges this assumption in Section 5 but does not use the extensive logging described in Section 3 (Puzzle Logger, Browser Logger) to test or even qualitatively inspect whether drag-and-drop errors, resets, or timer expirations preceded failures. The 'handful' of students who admitted not writing their own code or who claimed the puzzle was impossible is anecdotal and unquantified. Please report at least basic telemetry for failed attempts and exact counts for the anecdotal observations.","section":"Sections 3 and 5"}],"minor_comments":[{"comment":"Notation is inconsistent: the title and abstract use p3, while the body uses p^3 and 'p 3'. Use one consistently.","section":"Throughout"},{"comment":"The reference solution is presented as 'listing 1:' with lowercase formatting; format it as a numbered listing or figure for consistency with the rest of the paper.","section":"Listing 1"},{"comment":"'A handful' is used three times without counts. Please report exact numbers of students who admitted not writing their own Dockerfile, who claimed impossibility, and who said the puzzle prompted review.","section":"Section 4"},{"comment":"The survey metric combines 'high self-confidence and low effort' into one percentage (43.1% and 35.8%). Clarify the exact question wording, scale, and how the two dimensions were combined.","section":"Section 4, survey item"},{"comment":"Students could start the puzzle 'anytime the day after the project was due,' so time between artifact creation and puzzle attempt varied. This memory confound is listed in future work but is not controlled; mention it in the pilot limitations.","section":"Section 4, timing"},{"comment":"The paper states that all 111 students received a full task score from the auto-graders, which is useful context, but it is not connected to the puzzle results. Consider reporting whether puzzle failure correlated with autograder score or number of attempts.","section":"Section 1/4"}],"recommendation":"major_revision","confidential_remarks":"This is a work-in-progress pilot paper whose central diagnostic claim rests on an explicitly deferred validation. I do not think rejection is warranted because the method is novel, the pilot is honestly reported, and the fix can be local (tempering the language) or substantive (adding telemetry or baseline comparisons). However, as written, the abstract and Section 4 overstate what the data establish. The authors should be asked to align the claims with the evidence or add the missing validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a novel method paper with a real pilot, but the headline interpretation outruns the data. The thing you should know is that p3 — generating a Parsons puzzle from a student's own submitted Dockerfile and using puzzle failure as evidence of a comprehension gap — is a genuinely new idea, and the authors deployed it in a graduate course with 111 consenting students. The pilot is honest and clearly written. But the central claim in the abstract ('shows that p3 can help identify potential gaps in students' understanding') is not actually established by the data shown here. Section 5 states the load-bearing assumption: that reconstruction ability is a good proxy for understanding. The authors defer validation of that proxy to future work. Without it, a failed puzzle tells you the student couldn't reorder seven lines of a Dockerfile; it doesn't yet tell you why or what it means about understanding. The Group-Ref condition supports that caution: 66% of students solved a reference Dockerfile that wasn't their own, so the own-code advantage is only about 12 points (77.6% vs 66.0%). That's weak support for the claim that the instrument measures personalized understanding specifically, rather than general Dockerfile knowledge and recall. There are also no statistical baselines, and the artifact is a short canonical template, so failure could be time pressure or an ordering slip. None of this sinks the method, and the paper deserves credit for flagging both the proxy and the UI/UX assumption and for running follow-ups. The design is thoughtful: snippet picker, distractors, removed blocks, behavioral logging. I'd want the abstract toned down to 'pilot feasibility' and the analysis presented as exploratory. In peer review, I'd send it out — it's a legitimate first data point for a useful idea — but with a clear expectation that the conclusions be recalibrated to what the evidence actually supports.","headline":"A genuinely new idea with a real pilot, but the headline inference is carried by an unvalidated proxy the authors themselves defer; worth peer review as a feasibility pilot, not yet as a validated instrument.","tokens_in":4895,"tokens_out":2993,"would_cite":true,"duration_ms":31447,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Personalized probing puzzles generated from a student's own code can expose gaps between what is submitted and what is understood, even when the artifact passes automated tests.","keywords":["personalized assessment","program comprehension","Parsons puzzles","code reconstruction","AI-generated code","academic integrity","code review","self-efficacy"],"falsifier":"Give every student a practice playground puzzle first, then run the real puzzle: if the failure rate collapses, UI/UX confusion is the likely explanation. Or, after students fail a puzzle, have an expert interviewer ask them to explain the code's logic, justify its design choices, and predict the effect of specific edits; if a large share of failed-puzzle students can do so correctly, reconstruction success is not measuring understanding.","tokens_in":4135,"feed_emoji":"🧩","tokens_out":6572,"duration_ms":65680,"temperature":0.7,"pith_summary":"The paper argues that with AI coding assistants, a submitted artifact is no longer trustworthy evidence that its author understands it. To close that gap, the authors introduce Personalized Probing Puzzles (p3): a student's own source file is broken into shuffled code blocks, and the student must reconstruct it in a timed, asynchronous assessment. In a graduate cloud-computing pilot, every consenting student had passed the course's autograder, yet a nontrivial share of those shown their own Dockerfile could not reassemble it; the authors read the numbers as suggesting roughly a third may not understand their own code, with more than half lacking confidence. The paper's main claim is that p3 can locate potential gaps in understanding of one's own code, and that this check can be inserted between submission and review at low cost.","feed_headline":"Puzzles built from students' own code catch what autograders miss","feed_subtitle":"p3 turns submitted code into timed reconstruction tests; the pilot shows a gap between passing and understanding.","key_machinery":"The load-bearing mechanism is the Personalized Probing Puzzle (p3), a variant of the Parsons puzzle in which a source artifact is segmented into code blocks, shuffled (with possible distractors or omissions), and presented in a drag-and-drop interface for timed reconstruction asynchronously. The key idea is that reconstructing one's own code exercises understanding of that specific code, as opposed to general programming knowledge or artifact-only grading. Supporting components include a snippet picker that selects and validates code segments, a puzzle maker/assigner that personalizes and distributes puzzles, and loggers that record every drag, drop, click, and window event so reviewers can","core_discovery":"The central discovery is that a Parsons-style puzzle can be generated automatically from a student's own programming artifact, and that performance on that puzzle behaves like a diagnostic of how well the author understands the artifact. The pilot found students who had obtained full credit on the assignment nonetheless failing to reconstruct their own Dockerfile, with a handful explicitly admitting they had not written the code. The authors are careful to frame this not as proof but as evidence: p3 'can help identify potential gaps' in understanding, with a future study planned to formally correlate puzzle results with code understanding.","pith_inferences":["If reconstruction ability tracks understanding, the same mechanism could serve as a lightweight integrity check: students who submit AI-generated code are more likely to fail a puzzle built from it, giving instructors an evidence-based rather than accusatory way to flag potential academic dishonesty.","The paper's snippet picker, currently only for locations the instructor already knows matter, is the critical next step; a reliable algorithm for choosing representative snippets would let p3 scale from fixed assignments to open-ended professional codebases, where any function could be probed.","The logged drag/drop and window-visibility events offer a testable extension: puzzle attempts that show heavy searching, many resets, and guess-like ordering should correlate with low understanding, while fluent reconstructions should correlate with high understanding, turning the puzzle from a binary pass/fail into a graded diagnostic.","A further testable consequence is self-efficacy measurement: solving a puzzle despite low confidence could identify skilled but underconfident engineers who might benefit from mentoring, while failing a puzzle with high confidence would flag overconfidence and risk in critical systems."],"forward_implications":["In a pilot with 111 consenting students, all passed the course's autograder, yet only 77.6% of students who saw a puzzle of their own Dockerfile solved it; about a third could not reconstruct code they had submitted.","p3 puzzles are generated automatically from the artifact, administered asynchronously, and solvable in minutes, so the check can be inserted into an existing course or team workflow before code review begins.","A failed puzzle becomes a concrete signal that a reviewer can use to focus a code review, ask targeted questions, or prompt the author to demonstrate understanding again with a new puzzle.","The study also suggests a confidence gap: more than half of students reported high effort or low confidence, pointing to self-efficacy calibration as a separate benefit."],"fun_headline_variants":["Passing code, failing reconstruction: p3 spots the gap","Your code, your test: puzzles reveal who really wrote it","Autogenerated puzzles expose gaps in code understanding","Full credit but can't rebuild? p3 probes true authorship"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that being able to reconstruct or fix one's own program is a faithful proxy for understanding it; if puzzle failures stem from interface confusion, poor memory, or test anxiety rather than a knowledge gap, the diagnostic signal is invalid.","fun_headline_variants_meta":{"raw":{"variants":["Passing code, failing reconstruction: p3 spots the gap","Your code, your test: puzzles reveal who really wrote it","Autogenerated puzzles expose gaps in code understanding","Full credit but can't rebuild? p3 probes true authorship"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":994,"prompt_tokens":605,"completion_tokens":389,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":349,"completion_tokens_details":{"reasoning_tokens":321}},"tokens_in":349,"tokens_out":389,"duration_ms":4879,"temperature":1.0,"reasoning_tokens":321,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:46:56.162889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give every student a practice playground puzzle first, then run the real puzzle: if the failure rate collapses, UI/UX confusion is the likely explanation. Or, after students fail a puzzle, have an expert interviewer ask them to explain the code's logic, justify its design choices, and predict the effect of specific edits; if a large share of failed-puzzle students can do so correctly, reconstruction success is not measuring understanding.","supporting_citations":[],"review_version":1}