Personalized Assessments from Personal Artifacts

Jaromir Savelka; Majd Sakr; Seth Copen Goldstein; Yufan Zhang

REVIEW 3 major objections 6 minor 12 references

Personalized Assessments from Personal Artifacts

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Personalized probing puzzles generated from a student's own code can expose gaps between what is submitted and what is understood, even when the artifact passes automated tests.

desk verdict A genuinely new idea with a real pilot, but the headline inference is carried by an unvalidated proxy the authors themselves defer; worth peer review as a feasibility pilot, not yet as a validated instrument. read the letter →

arxiv 2607.16494 v1 pith:NTSHUZ5I submitted 2026-07-17 cs.SE

Yufan Zhang , Jaromir Savelka , Seth Copen Goldstein , Majd Sakr This is my paper · ORCID

classification cs.SE

keywords personalizedassessmentprogramcomprehensionParsonspuzzlescodereconstructionAI-generatedacademicintegrityreviewself-efficacy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that with AI coding assistants, a submitted artifact is no longer trustworthy evidence that its author understands it. To close that gap, the authors introduce Personalized Probing Puzzles (p3): a student's own source file is broken into shuffled code blocks, and the student must reconstruct it in a timed, asynchronous assessment. In a graduate cloud-computing pilot, every consenting student had passed the course's autograder, yet a nontrivial share of those shown their own Dockerfile could not reassemble it; the authors read the numbers as suggesting roughly a third may not understand their own code, with more than half lacking confidence. The paper's main claim is that p3 can locate potential gaps in understanding of one's own code, and that this check can be inserted between submission and review at low cost.

What carries the argument

The load-bearing mechanism is the Personalized Probing Puzzle (p3), a variant of the Parsons puzzle in which a source artifact is segmented into code blocks, shuffled (with possible distractors or omissions), and presented in a drag-and-drop interface for timed reconstruction asynchronously. The key idea is that reconstructing one's own code exercises understanding of that specific code, as opposed to general programming knowledge or artifact-only grading. Supporting components include a snippet picker that selects and validates code segments, a puzzle maker/assigner that personalizes and distributes puzzles, and loggers that record every drag, drop, click, and window event so reviewers can

What would settle it

Give every student a practice playground puzzle first, then run the real puzzle: if the failure rate collapses, UI/UX confusion is the likely explanation. Or, after students fail a puzzle, have an expert interviewer ask them to explain the code's logic, justify its design choices, and predict the effect of specific edits; if a large share of failed-puzzle students can do so correctly, reconstruction success is not measuring understanding.

Watch

Extended reading notes

Core claim

The central discovery is that a Parsons-style puzzle can be generated automatically from a student's own programming artifact, and that performance on that puzzle behaves like a diagnostic of how well the author understands the artifact. The pilot found students who had obtained full credit on the assignment nonetheless failing to reconstruct their own Dockerfile, with a handful explicitly admitting they had not written the code. The authors are careful to frame this not as proof but as evidence: p3 'can help identify potential gaps' in understanding, with a future study planned to formally correlate puzzle results with code understanding.

Load-bearing premise

The load-bearing premise is that being able to reconstruct or fix one's own program is a faithful proxy for understanding it; if puzzle failures stem from interface confusion, poor memory, or test anxiety rather than a knowledge gap, the diagnostic signal is invalid.

Editorial extensions

If this is right

In a pilot with 111 consenting students, all passed the course's autograder, yet only 77.6% of students who saw a puzzle of their own Dockerfile solved it; about a third could not reconstruct code they had submitted.
p3 puzzles are generated automatically from the artifact, administered asynchronously, and solvable in minutes, so the check can be inserted into an existing course or team workflow before code review begins.
A failed puzzle becomes a concrete signal that a reviewer can use to focus a code review, ask targeted questions, or prompt the author to demonstrate understanding again with a new puzzle.
The study also suggests a confidence gap: more than half of students reported high effort or low confidence, pointing to self-efficacy calibration as a separate benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If reconstruction ability tracks understanding, the same mechanism could serve as a lightweight integrity check: students who submit AI-generated code are more likely to fail a puzzle built from it, giving instructors an evidence-based rather than accusatory way to flag potential academic dishonesty.
The paper's snippet picker, currently only for locations the instructor already knows matter, is the critical next step; a reliable algorithm for choosing representative snippets would let p3 scale from fixed assignments to open-ended professional codebases, where any function could be probed.
The logged drag/drop and window-visibility events offer a testable extension: puzzle attempts that show heavy searching, many resets, and guess-like ordering should correlate with low understanding, while fluent reconstructions should correlate with high understanding, turning the puzzle from a binary pass/fail into a graded diagnostic.
A further testable consequence is self-efficacy measurement: solving a puzzle despite low confidence could identify skilled but underconfident engineers who might benefit from mentoring, while failing a puzzle with high confidence would flag overconfidence and risk in critical systems.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a novel method paper with a real pilot, but the headline interpretation outruns the data. The thing you should know is that p3 — generating a Parsons puzzle from a student's own submitted Dockerfile and using puzzle failure as evidence of a comprehension gap — is a genuinely new idea, and the authors deployed it in a graduate course with 111 consenting students. The pilot is honest and clearly written. But the central claim in the abstract ('shows that p3 can help identify potential gaps in students' understanding') is not actually established by the data shown here. Section 5 states the load-bearing assumption: that reconstruction ability is a good proxy for understanding. The authors defer validation of that proxy to future work. Without it, a failed puzzle tells you the student couldn't reorder seven lines of a Dockerfile; it doesn't yet tell you why or what it means about understanding. The Group-Ref condition supports that caution: 66% of students solved a reference Dockerfile that wasn't their own, so the own-code advantage is only about 12 points (77.6% vs 66.0%). That's weak support for the claim that the instrument measures personalized understanding specifically, rather than general Dockerfile knowledge and recall. There are also no statistical baselines, and the artifact is a short canonical template, so failure could be time pressure or an ordering slip. None of this sinks the method, and the paper deserves credit for flagging both the proxy and the UI/UX assumption and for running follow-ups. The design is thoughtful: snippet picker, distractors, removed blocks, behavioral logging. I'd want the abstract toned down to 'pilot feasibility' and the analysis presented as exploratory. In peer review, I'd send it out — it's a legitimate first data point for a useful idea — but with a clear expectation that the conclusions be recalibrated to what the evidence actually supports.

Referee Report

3 major / 6 minor

Summary. The paper proposes Personalized Probing Puzzles (p3), a method that generates Parsons-puzzle-style reconstruction tasks from a student's own submitted code, and reports a pilot in a graduate cloud computing course (n=111). Students reconstructed either their own Dockerfile (Group-Own, 58 students) or the reference Dockerfile (Group-Ref, 53 students). The headline results are that 77.6% of Group-Own and 66.0% of Group-Ref solved their puzzle; the paper interprets failures as indicating that a substantial fraction of students may not understand their own code. The authors also report post-puzzle survey responses on self-confidence/effort and anecdotal statements from a few students. The paper presents p3 as automatically generated, asynchronously administered, and quick to complete, and it explicitly defers validation of the proxy between puzzle-solving and code understanding to future work.

Significance. If the proxy were validated, p3 would address a timely and important problem in post-AI computing education: assessing whether students understand code they submit as their own. The idea of generating personalized assessments from personal artifacts is novel and practical, and the pilot provides a useful first data point at scale (111 consenting students). The paper is honest about its two key assumptions — the reconstruction-to-understanding proxy and the absence of UI/UX failure — which is a strength. However, the current evidence is only suggestive: the central inference from puzzle failure to an understanding gap is definitionally tied to the unvalidated proxy, and the article itself states that the correlation study is future work. The paper should therefore be read as a feasibility demonstration, not as a validation of the diagnostic claim.

major comments (3)

[Abstract and Section 4] The abstract's claim that the pilot 'shows that p3 can help identify potential gaps' is stronger than the evidence supports, because the inference depends entirely on the Section 5 assumption that reconstruction ability is a good proxy for understanding. The paper explicitly defers validation of that proxy. Moreover, the Group-Ref rate of 66.0% demonstrates that a majority of students who did not author the reference Dockerfile can reconstruct it from general Dockerfile knowledge, so the own-code advantage of only ~11.6 percentage points does little to isolate understanding of one's own artifact. Please either temper the abstract and Section 4 to 'may help identify potential gaps' and frame the pilot as feasibility, or add a validation of the proxy (e.g., correlation with code review scores, think-aloud data, or effort/telemetry).
[Section 4, bullet 1] The headline percentages (77.6% vs. 66.0%; 43.1% vs. 35.8%) are purely descriptive: no confidence intervals, effect sizes, or significance tests are reported. More importantly, there is no independent baseline for what failure rate a reconstruction task of this difficulty should produce. The 7-line canonical Dockerfile is close to a tutorial template, and a 22.4% failure rate may reflect time pressure, distractor design, or rote-recall demands rather than a semantic understanding gap. The Group-Ref condition is a natural baseline, but it is not used to calibrate the interpretation. Please report uncertainty and discuss the difficulty/ceiling of the task when interpreting failure rates.
[Sections 3 and 5] The UI/UX assumption is unresolved and load-bearing. The puzzle includes distractors, possibly removed blocks, a 'No Solution' button, and a 5-minute timer; any of these could cause failure independently of understanding. The paper acknowledges this assumption in Section 5 but does not use the extensive logging described in Section 3 (Puzzle Logger, Browser Logger) to test or even qualitatively inspect whether drag-and-drop errors, resets, or timer expirations preceded failures. The 'handful' of students who admitted not writing their own code or who claimed the puzzle was impossible is anecdotal and unquantified. Please report at least basic telemetry for failed attempts and exact counts for the anecdotal observations.

minor comments (6)

[Throughout] Notation is inconsistent: the title and abstract use p3, while the body uses p^3 and 'p 3'. Use one consistently.
[Listing 1] The reference solution is presented as 'listing 1:' with lowercase formatting; format it as a numbered listing or figure for consistency with the rest of the paper.
[Section 4] 'A handful' is used three times without counts. Please report exact numbers of students who admitted not writing their own Dockerfile, who claimed impossibility, and who said the puzzle prompted review.
[Section 4, survey item] The survey metric combines 'high self-confidence and low effort' into one percentage (43.1% and 35.8%). Clarify the exact question wording, scale, and how the two dimensions were combined.
[Section 4, timing] Students could start the puzzle 'anytime the day after the project was due,' so time between artifact creation and puzzle attempt varied. This memory confound is listed in future work but is not controlled; mention it in the pilot limitations.
[Section 1/4] The paper states that all 111 students received a full task score from the auto-graders, which is useful context, but it is not connected to the puzzle results. Consider reporting whether puzzle failure correlated with autograder score or number of attempts.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline inference reduces to an unvalidated proxy assumption: puzzle failure is treated as a gap in understanding by definition, not by evidence.

self definitional [Abstract; Section 4 (Pilot Study and Results); Section 5 (Limitations and Future Work)]
"Our pilot study shows that p3 can help identify potential gaps in students' understanding of their own code. ... An assumption made in this study is that an SWE's ability to reconstruct or fix their program (as measured by the puzzle results) is good proxy to their understanding of that program."

The pilot's only evidence of a 'gap in understanding' is failure to solve the puzzle generated from one's own code. But the paper explicitly assumes that puzzle performance is a good proxy for understanding, and defers validation to a future study. Therefore the abstract's 'shows' reduces to the assumption: by stipulation, failing the puzzle means a gap. The empirical observation is only that some students fail, not that they misunderstand their code. The central headline claim is definitional rather than derived.
self definitional [Section 4, results paragraph]
"The numbers suggest about a third of students may not understand their own code, and more than half lack confidence in understanding their own code, which is striking."

This sentence directly converts the failure rate into 'may not understand their own code.' The conversion is exactly the Section 5 proxy assumption, which is unvalidated. Without an independent measure of understanding, the conclusion is not supported by the data; it is the assumption restated in the language of a finding.

full rationale

The paper is candid about its key limitation: Section 5 states that the proxy assumption 'will be validated' in future work, and the abstract itself says 'Future work is needed to correlate puzzle results with code understanding.' This honesty is important, but it confirms the circularity: the headline claim that p3 'can help identify potential gaps in students' understanding' is not an empirical result derived from independent evidence; it is the assumed equivalence between puzzle performance and understanding. The observed failure rates, the Group-Own versus Group-Ref difference, and the self-report survey are real descriptive facts, but their diagnostic meaning depends entirely on an unvalidated proxy. The paper also reports a few students who failed and admitted they did not write their own Dockerfile, which is a small piece of independent support, but it does not validate the proxy for the general population. The Group-Ref condition (66.0% solved) further shows that solving the Dockerfile puzzle does not require ownership of or personal understanding of the specific artifact, weakening the inference that own-code failure specifically indicates a gap in understanding of one's own code. Because the central inference reduces to an explicit definitional assumption, a score of 6 is appropriate: the derivation is partially circular, but the paper does provide useful descriptive pilot data and does not hide the assumption.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claim depends on two domain assumptions (the proxy link between puzzle solving and understanding, and the absence of UI/UX confounds) plus hand-chosen parameters (five-minute timer, binary scoring). No free parameters are fitted to data; the 111-subject pilot produces descriptive percentages only.

free parameters (2)

Puzzle time limit = 5 minutes
Students had under 5 minutes to complete the puzzle; completion rates and the 'one third may not understand' inference depend on this hand-chosen cutoff.
Solution correctness criterion = exact reconstruction (not further specified)
The study likely requires the correct final order to count as 'solved', but the paper does not define how errors are scored; this binary outcome drives all reported percentages.

assumptions (2)

domain assumption An SWE's ability to reconstruct or fix their program (as measured by puzzle results) is a good proxy to their understanding of that program.
Stated in Section 5 as an assumption; the central claim that puzzle failures identify understanding gaps rests entirely on this equivalence, which is not validated in the study.
domain assumption No students failed the puzzle due to confusion with the user interface or UX.
Stated in Section 5; if the UI/UX confound affects failures, the completion rates become uninformative without the playground practice that was added later.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Personalized Assessments from Personal Artifacts." pith.science (2026). https://pith.science/paper/NTSHUZ5I

@misc{pith2026260716494,
  author       = {Pith},
  title        = {Pith review of: Personalized Assessments from Personal Artifacts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NTSHUZ5I}},
  note         = {Machine review of arXiv:2607.16494}
}

abstract

The rapid development and popularization of AI-enabled coding agents have meant software engineering students and professionals cannot be assumed to understand their own code, which risks academic integrity and professional accountability. We developed a method called Personalized Probing Puzzles ($p^3$) to evaluate students' understanding of their own code, and tested $p^3$ in a graduate-level cloud computing course. Our pilot study shows that $p^3$ can help identify potential gaps in students' understanding of their own code. The puzzles are automatically generated, asynchronously administered, and finished in minutes. Future work is needed to correlate puzzle results with code understanding and to embed $p^3$ in a professional code review process.

Figures

Figures reproduced from arXiv: 2607.16494 by the authors.

**Figure 1.** SWE demonstrates understanding before system accepts their code. The current interface of p 3 is a single-page web application ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** The user interface of p 3 , with a parsons puzzle of a Dockerfile be solved with the Available Code Blocks, and a Timer at the bottom. When the time is up, the puzzle is automatically submitted. Not all code blocks need to be used in the solution; some may be distractors [1]. Not all puzzles are solvable with the available code blocks; some code blocks may have been removed. Behind the scenes, p 3 is also equipped w… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

[1]

ITiCSE ’22

Ericson, B., Haynes-Magyar, C.: Adaptive parsons problems as active learning ac- tivities during lecture. ITiCSE ’22. p. 290–296 (2022).10.1145/3502718.3524808

arXiv 2022
[2]

Ericson, B.J., Denny, P., et al.: Parsons problems and beyond: Systematic literature reviewandempiricalstudydesigns.ITiCSE-WGR’22.p.191–234(2022).10.1145/ 3571785.3574127

arXiv 2022
[3]

Koli Calling ’17

Ericson, B.J., Margulieux, L.E., et al.: Solving parsons problems versus fixing and writing code. Koli Calling ’17. p. 20–29 (2017).10.1145/3141880.3141895

arXiv 2017
[4]

Infor- mation and Software Technology164, 107300 (2023).https://doi.org/10.1016/ j.infsof.2023.107300

Heinonen, A., Lehtelä, B., et al.: Synthesizing research on programmers’ mental models of programs, tasks and concepts — a systematic literature review. Infor- mation and Software Technology164, 107300 (2023).https://doi.org/10.1016/ j.infsof.2023.107300

arXiv 2023
[5]

Ihantola, P., Karavirta, V.: Two-dimensional parson’s puzzles: The concept, tools, and first observations. J. ITE: Innovations in Practice10, 1–14 (01 2011).10. 28945/1394

2011
[6]

MacNeil, S., Spurlock, S., et al.: Imagining computing education assessment after generative ai (2024),https://arxiv.org/abs/2401.04601

arXiv 2024
[7]

ACM Transactions on Computing Ed- ucation24(1), 1–43 (Feb 2024).10.1145/3636515

Messer, M., Brown, N.C.C., et al.: Automated grading and feedback tools for pro- gramming education: A systematic review. ACM Transactions on Computing Ed- ucation24(1), 1–43 (Feb 2024).10.1145/3636515

doi:10.1145/3636515 2024
[8]

ESEC/FSE 2021

Patra, J., Pradel, M.: Semantic bug seeding: a learning-based approach for creating realistic bugs. ESEC/FSE 2021. p. 906–918 (2021).10.1145/3468264.3468623

arXiv 2021

Show all 12 references

[9]

CACM69(4), 41–44 (Mar 2026).10.1145/3779312

Russinovich, M., Hanselman, S.: Redefining the software engineering profession for ai. CACM69(4), 41–44 (Mar 2026).10.1145/3779312

2026 doi
[10]

Shen, J.H., Tamkin, A.: How ai impacts skill formation (2026),arxiv.org/abs/ 2601.20245

2026
[11]

Computer science series (1971)

Weinberg, G.M.: The psychology of computer programming. Computer science series (1971)

1971
[12]

CHI ’21 (2021).10.1145/3411764.3445228

Weinman, N., Fox, A., et al.: Improving instruction of programming patterns with faded parsons problems. CHI ’21 (2021).10.1145/3411764.3445228

2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

ITiCSE ’22

Ericson, B., Haynes-Magyar, C.: Adaptive parsons problems as active learning ac- tivities during lecture. ITiCSE ’22. p. 290–296 (2022).10.1145/3502718.3524808

arXiv 2022

[2] [2]

Ericson, B.J., Denny, P., et al.: Parsons problems and beyond: Systematic literature reviewandempiricalstudydesigns.ITiCSE-WGR’22.p.191–234(2022).10.1145/ 3571785.3574127

arXiv 2022

[3] [3]

Koli Calling ’17

Ericson, B.J., Margulieux, L.E., et al.: Solving parsons problems versus fixing and writing code. Koli Calling ’17. p. 20–29 (2017).10.1145/3141880.3141895

arXiv 2017

[4] [4]

Infor- mation and Software Technology164, 107300 (2023).https://doi.org/10.1016/ j.infsof.2023.107300

Heinonen, A., Lehtelä, B., et al.: Synthesizing research on programmers’ mental models of programs, tasks and concepts — a systematic literature review. Infor- mation and Software Technology164, 107300 (2023).https://doi.org/10.1016/ j.infsof.2023.107300

arXiv 2023

[5] [5]

Ihantola, P., Karavirta, V.: Two-dimensional parson’s puzzles: The concept, tools, and first observations. J. ITE: Innovations in Practice10, 1–14 (01 2011).10. 28945/1394

2011

[6] [6]

MacNeil, S., Spurlock, S., et al.: Imagining computing education assessment after generative ai (2024),https://arxiv.org/abs/2401.04601

arXiv 2024

[7] [7]

ACM Transactions on Computing Ed- ucation24(1), 1–43 (Feb 2024).10.1145/3636515

Messer, M., Brown, N.C.C., et al.: Automated grading and feedback tools for pro- gramming education: A systematic review. ACM Transactions on Computing Ed- ucation24(1), 1–43 (Feb 2024).10.1145/3636515

doi:10.1145/3636515 2024

[8] [8]

ESEC/FSE 2021

Patra, J., Pradel, M.: Semantic bug seeding: a learning-based approach for creating realistic bugs. ESEC/FSE 2021. p. 906–918 (2021).10.1145/3468264.3468623

arXiv 2021