Pith. sign in

REVIEW 3 major objections 4 minor 95 references

(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A user study with 54 programmers shows that coding agents that edit users' code make the initial task easier but measurably weaken users' understanding of their own code, and do not improve their ability to extend it without the agent.

desk verdict Solid comprehension result, overreaching extension claim — worth refereeing with revisions. read the letter →

arxiv 2607.26375 v1 pith:VAGDC5MJ submitted 2026-07-29 cs.CL cs.HC

Nishant Balepur , Connor Baumler , Valerie Chen , Eunsol Choi , Rachel Rudinger , Jordan Lee Boyd-Graber This is my paper · ORCID
classification cs.CLcs.HC
keywords codingagentscodecomprehensionuserstudyhuman-AIinteractionproductivityvsunderstandingprogramLLMevaluationagentic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the usual way of evaluating coding agents—by whether they finish a task—misses a real cost: users who prompt an agent to write code understand that code much less than users who write it themselves. In a study, 54 computer science students built a tic-tac-toe-style website using either an agent that directly edits code or a chatbot that only gives syntax help. The agent group finished faster and more accurately, but scored far lower on comprehension questions about their own code and no higher on a follow-up task where they had to extend that code without the agent. Lower-effort interactions such as copy-pasting prompts and auto-accepting edits were linked to lower comprehension, while more readable agent-written code went with better comprehension. Users noticed they understood less, yet still preferred the agent because it was quick and easy.

What carries the argument

The argument is carried by a controlled two-group experiment. One group works with an agent that directly edits their code; the other works with a chatbot that only offers short syntax guidance, so users must write or adapt code themselves. Understanding is measured two ways: automatically generated comprehension questions tailored to each user's own code—covering recall of what is in the code and reasoning about what changed code would do—and an extension task where users add features to their submitted code with only the chatbot. This pair of metrics makes 'understanding' concrete and user-specific, letting the paper compare productivity gains against comprehension losses.

What would settle it

A replication in which an independent panel writes comprehension questions from anonymized submissions without knowing which group produced them, and finds agent and chatbot users scoring equally, would falsify the main claim. So would a replication where agent users' extension accuracy remains higher even after matching initial task accuracy.

Watch

Extended reading notes

Core claim

The paper establishes a productivity–understanding gap: coding agents that edit users' code substantially improve initial task accuracy but substantially reduce users' comprehension of their own code, and they do not improve users' ability to extend that code once the agent is gone. In regressions, initial task accuracy barely predicts comprehension, while comprehension and initial accuracy together strongly predict extension accuracy, so the agent's better initial scaffold is offset by the user's weaker understanding. The paper concludes that task-completion-only evaluation of coding agents is incomplete: understanding one's own code is a distinct, measurable objective that current agents n

Load-bearing premise

The load-bearing premise is that the automatically generated comprehension questions measure genuine understanding fairly across both groups—if the questions were systematically easier or harder for one group's code, the observed comprehension gap could be an artifact.

Editorial extensions

If this is right

  • If the central claim is correct, task-completion benchmarks for coding agents overstate their value to users, and evaluations should include comprehension or other understanding-based measures.
  • Users who rely on agents may be poorly prepared to debug, oversee, or explain their own code when the agent is unavailable or fails.
  • Low-effort interaction patterns, such as copying requirements verbatim or auto-accepting edits, mark users at particular risk of low comprehension, suggesting agents could be designed to discourage those patterns.
  • Agent-generated code that is more concise and readable is associated with better user comprehension, making readability a plausible training objective for coding agents.
  • User preference alone will not fix the problem, since users in the study preferred the agent even while recognizing their understanding was weaker.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests the comprehension gap may grow over time: if understanding is not built during each task, repeated agent use could erode users' mental models of their systems, making later oversight failures more likely—an extension the paper does not directly test.
  • The same productivity-versus-understanding trade-off likely applies beyond web development, for example in data analysis or algorithm design, wherever code writing is offloaded; the paper's protocol could be adapted to those domains.
  • A testable design remedy: require users to explain each accepted agent edit before it is applied. The paper's correlation between active per-file review and higher comprehension suggests such forced engagement could recover some understanding without forcing users to write everything from scratch.
  • Coding-agent evaluation could adopt lightweight comprehension probes alongside task benchmarks, making understanding a first-class metric rather than an afterthought.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a between-subjects user study (n=54 CS students) comparing a code-editing agent with a syntax-only chatbot on a web-development task. It measures initial task accuracy, comprehension via LLM-generated recall and reasoning questions, and performance on an extension task done without the agent. The headline findings are that agent users complete the initial task more accurately and quickly, score substantially lower on comprehension (d≈0.9), and do not differ significantly on extension accuracy. Secondary analyses examine background ability, prompting and review strategies, code readability, and self-reported preferences. The paper interprets these results as showing that task-completion-only evaluations of coding agents neglect user understanding and that this has downstream consequences for users' ability to extend their code.

Significance. The result is potentially important: if it holds up, it suggests that benchmark-driven coding-agent development is optimizing a metric decoupled from users' ability to reason about their own code, with implications for evaluation practice and agent design. The study has genuine strengths. The central comprehension finding is measured with external behavioral instruments rather than derived from fitted constants, and the authors provide multiple validation checks for the LLM-generated questions and the rubric-based judge. The released dataset and interface are valuable assets, and the self-report analyses in Appendix A.6 show that perceived understanding tracks measured understanding. The main caveat is that the causal 'thus do not prepare users to extend their code' clause is not supported by the non-significant total effect on extension accuracy; the paper's headline and abstract overstate the strength of the evidence for the downstream consequence. The robust contribution is the comprehension difference and the study design; the policy-relevant claim about extension ability needs to be reframed or supported by a stronger mediation analysis.

major comments (3)
  1. [Abstract; §4.3; Appendix A.5; Figure 9] The headline claim that agents 'thus do not prepare users to extend their code' goes beyond the results. The total effect of condition on extension accuracy is non-significant in the main comparison (Figure 1: p=0.45, d=0.2) and in the initial-accuracy >0.50 subgroup (p=0.18). The causal wording is instead supported by the saturated observed-variable path model in Figure 9, fit post hoc to n=54 with 0 df, no global fit statistics, and without the background×condition interaction used in Eq. (1). Appendix A.5 is transparent that this is a 'compact summary,' but the abstract and the §4.3 heading ('can Degrade Extension Ability') present the downstream harm as established. Please either provide a properly identified and pre-specified mediation analysis with sensitivity checks, or reframe the conclusion as an exploratory mechanism consistent with, but not established by, the data.
  2. [§3.3; Appendix A.9; Table 7] The comprehension difference is the paper's central behavioral finding, so the group-fairness of the LLM-generated questions is load-bearing. The validations in Appendix A.9 are careful, but they do not establish measurement invariance. Agent users submitted longer and more complete code (Table 5), and the overall d=0.9 is driven by 'identify own code' (d=1.873) and JS questions (d=1.089), while 'change' questions show no difference (p=0.916, d=0.030). Without an externally validated concept inventory (a gap the authors acknowledge in §3.1), it remains possible that some of the apparent comprehension deficit is an artifact of systematically harder questions about more complex code. I would ask for an independent norming study—e.g., coders blind to condition answering the same generated items—or an item-level analysis demonstrating that difficulty is invariant across groups; at minimum, t
  3. [§4.3; Figure 7; Appendix A.5] The statement in §4.3 that 'A path mediation model confirms this' overstates what a saturated post-hoc model can confirm. With 0 degrees of freedom, the model in Figure 9 cannot be tested against the data, and omitting the background×condition interaction used in the primary regressions means the specified paths may not match the earlier analysis. Unmeasured confounders such as time spent, prompt count and type, and code quality could induce opposing indirect paths. Please present the mediation as descriptive rather than confirmatory, and report the sensitivity of the indirect effects to plausible confounders.
minor comments (4)
  1. [Figure 1 (caption)] The caption states that 'Agent users can better extend their code without agents,' but the reported comparison is non-significant (p=0.45, d=0.2). Please rephrase to 'no significant difference' or 'slightly, not significantly, higher' to match the statistics.
  2. [§4.3; Figure 7] The R² change is reported as 0.05→0.34 in the text and 0.05→0.22 in the Figure 7 caption; please reconcile this inconsistency.
  3. [§5.1; Tables 1-2] The main text reports mean comprehension for users who 'use' each prompting/review strategy, but users can engage multiple strategies; the Appendix A.4 regressions handle this. Please note the non-exclusive nature in the main text or refer readers to A.4 before presenting the averages.
  4. [§4.2; Eq. (1)] The regression results are shown as coefficient plots without a table of estimates. Providing a table with coefficients, standard errors, and p-values for all predictors would improve reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: central claim rests on independent behavioral instruments; self-citations are tooling, not load-bearing reductions.

full rationale

The paper's central claim—coding agents aid task completion but harm comprehension—is measured with external behavioral instruments rather than derived from fitted parameters or from the authors' prior results. Comprehension is tested with LLM-tailored questions whose validity is checked manually, via pass-rate checks and side-by-side comparisons (Appendix A.9), and the extension task is scored against an explicit rubric. The paper even addresses the main instrumentation confound (that agent users' more complete code might make LLM questions harder) by regressing comprehension on initial accuracy and showing the group effect persists (Fig. 6, A.9.2). The only author self-citations are to VibeJam (Balepur et al., 2026) as the experimental UI and to Chen et al. (2026) as background framing; neither provides the empirical result, so they are not load-bearing. The most vulnerable step is the abstract's 'thus do not prepare users to extend their code': this rests on a saturated, post-hoc path model (A.5, Fig. 9) with 0 degrees of freedom and a non-significant total effect of condition on extension accuracy. That is a statistical/interpretive overreach, not a circular reduction—the underlying estimates are transparent and the non-significant total effect is reported, so the claim is not forced by construction. No self-definitional, fitted-input-called-prediction, uniqueness-imported, ansatz-smuggled, or renaming pattern is present.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters were fitted to make the central claim work; rubrics and question templates are author-specified measurement instruments. The main assumptions are about measurement validity, control-condition equivalence, and causal inference from observational regressions.

assumptions (6)
  • domain assumption LLM-generated comprehension questions measure the same latent construct 'understanding' across agent and chatbot groups and are of comparable difficulty.
    The main outcome depends on this. Validated via manual review, pilot submissions, pass-rate checks, and side-by-side comparisons (Appendix A.9), but no externally validated concept inventory exists.
  • domain assumption The LLM judge scores initial and extension code accurately and without group-dependent bias.
    Supported by 98% agreement with human scoring on 20 submissions and Cohen's κ=0.96, but only a subset of code was manually checked.
  • domain assumption The chatbot condition is a valid control for 'users writing code' and does not introduce confounds beyond the intended automation difference.
    The chatbot's strict refusal prompt may affect frustration, prompting rate, and task completion, which could themselves influence comprehension.
  • domain assumption Participants did not anticipate the comprehension questions or extension task, so initial-task behavior was not altered by demand characteristics.
    Only 16% of participants said they anticipated the extension task (Appendix A.8.3), but this is a self-report after the fact.
  • domain assumption Random assignment plus attention-check exclusions yields comparable groups on unmeasured confounds such as motivation and prior tool familiarity.
    Background ability is similar across groups (p=0.12, d=0.4), but unmeasured variables are not directly controlled.
  • domain assumption Regression and path mediation models can support causal claims about how initial accuracy and comprehension mediate the effect of condition on extension accuracy.
    The path model in Appendix A.5 is saturated (0 df), so global fit is uninformative, and the causal interpretation requires no unmeasured confounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding." pith.science (2026). https://pith.science/paper/VAGDC5MJ

@misc{pith2026260726375,
  author       = {Pith},
  title        = {Pith review of: (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VAGDC5MJ}},
  note         = {Machine review of arXiv:2607.26375}
}
read the original abstract

Coding agents (e.g., Cursor) improve developer productivity by optimizing task completion, but shifting users from writing code to prompting and reviewing may harm their understanding, impeding oversight, learning, and communication. To probe this, we have 54 students create a website with one of two AI systems: an agent that edits user code; or a chatbot where users write code alone or adapt generic code snippets. We test understanding via comprehension questions and a task where users extend their code without agents, showing: (1) While agents aid initial task completion, they harm users' code comprehension and thus do not prepare users to extend their code; (2) Low-effort agent interaction types, like copy+paste prompts and auto-accepted edits, are linked with lower comprehension; and (3) Despite self-reported weaker understanding, users still prefer coding agents because they are quick and easy to use. While users stay in the loop for coding workflows, understanding should not be forgotten. Towards this goal, we distill our analyses into future research directions for coding agent developers: dissuading low-effort prompting, creating readable code, and promoting active engagement.

Figures

Figures reproduced from arXiv: 2607.26375 by the authors.

Figure 1
Figure 1. Our user study compares two AIs—an agent that writes code for users and a chatbot that ensures our users write code—for finishing a web development task. Relative to chatbots, agents initially help users complete the task but largely harm comprehension (Cohen, 2013, p < 0.002, Cohen’s d > 0.80). Agent users can better extend their code without agents, but this reverses when controlling for initial task accuracy, sug… view at source ↗
Figure 2
Figure 2. Our between-subjects user study to show how coding agents neglect user understanding. After taking a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of UI and AI groups. Users create a website that meets input requirements (left) by prompting one of two AI systems: (a) an agent that edits users code, explains edits, and affords user review; or (b) a chatbot that gives high-level syntax. 2. Group B has a chatbot that only gives generic code snippets, so users must write code from scratch or manually adapt snippets ( [PITH_FULL_IMAGE:figures/full_fig_p00… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Metrics over AI groups. Despite similar background ability (a), agent users are more accurate/quick in the initial task (b/c, left), but not when extending code without agents (b/c, right) and agent users have worse comprehension (d). Appendix A.9.2 shows poor comprehe…
Figure 5
Figure 5. Figure 5: Regression coefficients with 95% CIs when predicting study outcomes from user background (BG) ability, segmented by AI groups. We show: (1) BG ability only predicts initial task accuracy in chatbot users, with agents equalizing users regardless of BG; (2) BG strongly p…
Figure 7
Figure 7. Figure 7: Regression coefficients with 95% CIs after adding initial accuracy to the extension accuracy regression (Equa￾tion 1). Both feature coefficients are significantly positive and largely boost R 2 (0.05 → 0.22), showing why agent/chatbot users have similar extension accur…
Figure 8
Figure 8. Figure 8: Model coefficients and 95% CIs when using z-score normalized code readability metrics to predict agent user com￾prehension, fitting four regressions. Users with fewer lines of code and proportion of comments have better comprehension. total information via unique token…
Figure 9
Figure 9. Figure 9: Simplified saturated observed-variable path model summarizing relationships among background ability, [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Comparison of randomly sampled generated comprehension questions for agent and chatbot users: [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Comparison of randomly sampled generated comprehension questions for agent and chatbot users: [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Comparison of randomly sampled generated comprehension questions for agent and chatbot users: [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Example background assessment question shown to users in our interface (§ [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Example questions after the initial task submission shown to users in our interface for self-reporting [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Example recall question in the comprehension assessment (§ [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Example recall question in the comprehension assessment (§ [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Example code reasoning in the comprehension assessment (§ [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: Example code reasoning in the comprehension assessment (§ [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

95 extracted references · 3 linked inside Pith

  1. [1]

    On a page with many images, what would be the effect of adding loading="lazy" to the <img> tag?[Knowledge, HTML]

  2. [2]

    Shraddha Barke, Michael B

    Vibejam: An open platform for user studies on agentic vibe coding. Shraddha Barke, Michael B. James, and Nadia Polikar- pova. 2022. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages, 7:85 – 111. József Beck. 1945. Combinatorial games.American history, 1861(1900). Joel Becker, Nate Rush, ...

    arXiv 2022

  3. [3]

    Which HTML element should you use to indicate that this is the main navigation? [Recall, HTML]

    You are designing a site and creating a naviga- tion bar linking to the main sections. Which HTML element should you use to indicate that this is the main navigation? [Recall, HTML]

  4. [4]

    Which element creates an ordered list, shown with numbers in the browser by default? [Recall, HTML]

  5. [5]

    The interface users code in extends VibeJam (Balepur et al., 2026), displayed in Figure 3

    to reason about the purpose of code snippets (Fig- ure 17); and 6) to reason about what would happen to their game if their code changed (Figure 18). The interface users code in extends VibeJam (Balepur et al., 2026), displayed in Figure 3. A.8 Questions for Participants We display the exact questions we ask to students in the background quiz (Appendix A....

    2026

  6. [6]

    Which attribute to the button below creates a link to the telephone number 1-(704) 555- 1151?[Trace Code, HTML]

  7. [7]

    InProceedings of the 31st International Conference on Intelligent User Inter- faces, IUI ’26, page 393–425, New York, NY , USA

    Whose code is it? how ai autonomy re- shapes ownership, responsibility, and disclosure in ai-assisted programming. InProceedings of the 31st International Conference on Intelligent User Inter- faces, IUI ’26, page 393–425, New York, NY , USA. Association for Computing Machinery. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. 2026. Coll...

    arXiv 2026

  8. [8]

    Which HTML will result in text being highlighted in yellow? [Write Code, HTML]

Show all 95 references
  1. [9]

    Benjamin Xie, D

    40 years of designing code comprehension experiments: A systematic mapping study.ACM computing surveys, 56(4):1–42. Benjamin Xie, D. Loksa, Greg L. Nelson, Matthew J. Davidson, Dongsheng Dong, Harrison Kwik, A. H. Tan, Leanne Hwa, Min Li, and Amy J. Ko. 2019. A theory of instr...

    2019

  2. [10]

    Keyword”, “Punctuator

    Impossiblebench: Measuring LLMs’ propen- sity of exploiting test cases. InThe Fourteenth Inter- national Conference on Learning Representations. Tim Zindulka, Sven Goller, Daniela Fernandes, Robin Welsch, and Daniel Buschek. 2026. The ai memory gap: Users misremember what they...

    2026

  3. [11]

    [Recall, CSS]

    Which line of code, if applied to all flex items in a flex container, would cause each flex item to take up an equal share of the total width of the container? For example, if there are four items, they would get 25% of each. [Recall, CSS]

  4. [12]

    Which line of CSS would add scroll bars if the content is taller than the box, but leave no visible scroll bars if the content fits into the box?[Recall, CSS]

    You have created a box that has a height set with CSS. Which line of CSS would add scroll bars if the content is taller than the box, but leave no visible scroll bars if the content fits into the box?[Recall, CSS]

  5. [13]

    Which attribute must have a unique value each time it is used in an HTML document? [Knowledge, HTML]

  6. [14]

    Under the stan- dard box model, what is the total width of the content box plus padding (excluding border and margin) in the following CSS? [Trace Code, CSS]

    The CSS box model describes how different parts of a box are calculated. Under the stan- dard box model, what is the total width of the content box plus padding (excluding border and margin) in the following CSS? [Trace Code, CSS]

  7. [15]

    Which CSS would correctly apply the back- ground color to every odd row in your table? [Write Code, CSS]

    You want to create striped table rows using CSS without adding a class to any element. Which CSS would correctly apply the back- ground color to every odd row in your table? [Write Code, CSS]

  8. [16]

    preconnect

    A webpage has ‘rel="preconnect"‘ added to a link resource. What will this do? [Trace Code, HTML]

  9. [17]

    What does the ‘===‘ comparison operator do? [Knowledge, JS]

  10. [18]

    How would you change this code to make Vanilla selected by default? [Write Code, HTML]

  11. [19]

    Which array method should you apply to run a function for every item within an array, return- ing an array of all items for which the function is true?[Recall, JS]

  12. [20]

    What is likely to happen? [Knowledge, CSS]

    The browser finds some CSS that it does not understand. What is likely to happen? [Knowledge, CSS]

  13. [21]

    How does the rem unit represent a font size? [Knowledge, CSS]

  14. [22]

    What will this loop print? [trace code, JS]

  15. [23]

    How would you change the value to plum? [Write Code, JS]

    In the following code, the variable ‘fruit‘ has been assigned a value of apple. How would you change the value to plum? [Write Code, JS]

  16. [24]

    How many columns will there be, given this code?[Trace Code, CSS]

  17. [27]

    Which code example would center ‘.box‘ in- side ‘.container‘?[Write Code, CSS]

  18. [29]

    Variables declared with the let keyword have what type of scope?[Knowledge, JS]

  19. [31]

    How would you round the value 11.354 to the nearest full integer?[Recall, JS]

  20. [32]

    What will be the value of selected? [trace code, JS]

  21. [35]

    When testing users, we pick a random multiple-choice question out of the two options for that language and competency

    Which line would you add to this code to add "Cosmos" to the list of currencies using JavaScript?[Write Code, JS] There are 12 unique language-competency pairs over the 24 questions. When testing users, we pick a random multiple-choice question out of the two options for that ...

    2026

  22. [36]

    Which of these options best describes how the JavaScript code below uses the HTML ele- ment [insert status element ]?An- swer:The JavaScript updates this element to display the game status

  23. [37]

    In the HTML snippet below, the JavaScript selects the status element using the [insert ID] identifier. If [insert ID] on the HTML element was changed to [insert new ID] but the rest of the website stayed the same, what would most likely happen? Answer:The status element would ...

  24. [38]

    In the CSS rule shown below, how does the selector [insert selector ID] deter- mine which elements on the website the styles are applied to?Answer:HTML elements whose ID matches the selector receive the rule’s styles

  25. [39]

    The CSS rule in the snippet below uses the attribute [insert attribute for centering]. If this attribute was removed but the rest of the website stayed the same, what would most likely happen to the ele- ments where the rule applies?Answer:The elements would be aligned to the left

  26. [40]

    Which of these options best describes the primary purpose of this function?Answer:Sync the displayed board with the current board state

    The JavaScript snippet below shows the func- tion [insert function name for displaying the board] . Which of these options best describes the primary purpose of this function?Answer:Sync the displayed board with the current board state

  27. [41]

    Imagine the loop indexing in this function were changed so that [insert logic to omit the last row]

    The JavaScript snippet below shows the func- tion [insert function name] , which renders the game board. Imagine the loop indexing in this function were changed so that [insert logic to omit the last row] . If the rest of the website stayed the same, which of these best descri...

  28. [42]

    Was there anything specific about the AI in Agent Mode (directly editing your code) in the first task that made it easier or harder to work with to complete tasks?

  29. [43]

    Was there anything specific about the AI in Chat Mode (providing high-level syntax) that made it easier or harder to work with to com- plete tasks?

  30. [44]

    Was there anything specific about the AI in Agent Mode (directly editing your code) in the first task that made it easier or harder to understand your code? Feel free to recall or compare with any AI programming tools you have previously used

  31. [45]

    Was there any specific property of your code (e.g., number of functions, number of lines, comments) in the first task that made it easier or harder to extend your code in the second task?

  32. [46]

    Which features of the AI assistants did you find helpful? Feel free to recall or compare with any AI programming tools you have pre- viously used

  33. [47]

    Were there any additional or different features that you wish the AI assistants you worked with had? Feel free to recall or compare with any AI programming tools you have previ- ously used

  34. [48]

    Did you anticipate that you would be asked to extend or modify the code you submitted in the first recreation task?

    Was there anything else you liked or disliked about your interaction with the AI during this task? Lastly, to evaluate whether our comprehension questions had any priming effects (Orne, 2017), we ask “Did you anticipate that you would be asked to extend or modify the code you ...

    2017

  35. [49]

    understanding

    recalling function names/elements/selectors; 2) identifying written code snippets; 3) noting the pur- pose of a function; and 4) explaining how a change in the code would impact the website. Questions 1–3 confirm agent users have reductions in compre- hension scores, while (4)...

    1955

  36. [50]

    Each player places exactly two symbols per turn (A -> A -> B -> B -> ...) When a players clicks on a square, it should display 'A' or 'B' based on who clicked it

    Which requirements were you able to successfully implement? (pre-populated from your Task Instructions checklist) Turns alternate between Player A and Player B, starting with A. Each player places exactly two symbols per turn (A -> A -> B -> B -> ...) When a players clicks on ...

  37. [51]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    The AI in Chat Mode (where it generated code/syntax help) was helpful for this task. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  38. [52]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    Working with the AI in Chat Mode required a lot of mental effort. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  39. [53]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    I read and reviewed the AI chatbot's responses. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  40. [54]

    2 - Disagree

    Attention Check: Please select "2 - Disagree" as your answer 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  41. [55]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    The code feels like my own work. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  42. [56]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    I understand how my code works. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  43. [57]

    1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

    I could easily add new features (e.g., new game rules, UI components, CSS styles) to my code without using AI tools. 1 - Strongly disagree 2 - Disagree 3 - Neither agree nor disagree 4 - Agree 5 - Strongly agree

  44. [58]

    Note: Your response will NOT affect how you are assigned in future tasks

    For future tasks in our interface, how much would you prefer each of the following ways of working with AI? Use the scale from least preferred to most preferred for each row. Note: Your response will NOT affect how you are assigned in future tasks. 1 - Least preferred 2 3 4 5 ...

  45. [59]

    C o n t i n u e Figure 14: Example questions after the initial task submission shown to users in our interface for self-reporting usefulness, understanding, and preferences (§6.1)

    Was there anything else you liked or disliked about your interaction with the AI during this task? (Optional) 0 chars Feel free to share anything you liked or disliked. C o n t i n u e Figure 14: Example questions after the initial task submission shown to users in our interfa...

  46. [60]

    Which of the following features exist in your website? It is possible that all of these or none of these exist

  47. [61]

    It is possible that all of these or none of these exist

    Which of these selectors exist in your CSS stylesheet? Each option shows a class or ID selector and a description of what it modifies. It is possible that all of these or none of these exist

  48. [62]

    Zic-Zac-Zoe

    Which of the following JavaScript functions exist in your code? Each option shows a function name and a description of its implementation. It is possible that all of these or none of these exist. Submit Project The board uses CSS Grid with 4px gaps between cells. The game dete...

  49. [63]

    Which one is yours? The 🔶 symbol indicates line changes

    Exactly one of these two pairs of code blocks (HTML + JavaScript) is from your project. Which one is yours? The 🔶 symbol indicates line changes. Left Block (HTML): Left Block (JavaScript): Right Block (HTML): Right Block (JavaScript):

  50. [64]

    Which one is yours? The 🔶 symbol indicates line changes

    Exactly one of these two CSS code blocks is from your project. Which one is yours? The 🔶 symbol indicates line changes. Left block: Right block:

  51. [65]

    game-status

    Exactly one of these two JS code blocks is from your project. Which one is yours? The 🔶 symbol indicates line changes. Left block: Right block: 🔸 1 <p id="game-status" class="status"></p> 🔸 1 2 3 const statusElement = document.getElementById("gam ... statusElement.textContent ...

  52. [66]

    status" class=

    Which of these options best describes how the JavaScript code below uses the HTML element <p id="status" class="status"></p>? HTML: JavaScript: 1 <p id="status" class="status"></p> 1 2 3 const statusElement = document.getElementById("status"); ... statusElement.textContent = `...

  53. [67]

    status" class=

    In the HTML snippet below, the JavaScript selects the status element using the id identifier. If the id on the HTML element was changed from status to game-status but the rest of the website remained the same, what would most likely happen? HTML: JavaScript: 1 <p id="status" c...

  54. [68]

    You can only provide syntax-help guidance based on the user’s existing code and errors

  55. [69]

    You should judge the user’s prompt and decide whether they are trying to bypass these safeguards

    You must not generate project/content-specific implementation code (for example: feature code, game logic, UI components, or task-completion code). You should judge the user’s prompt and decide whether they are trying to bypass these safeguards

  56. [70]

    If code is needed for syntax clarification, provide at most 3 lines total, and only as minimal syntax examples directly tied to syntax usage. If you can only comply with the user’s request by generating more than three lines of code (e.g., for loop(s) with multiple lines of lo...

  57. [71]

    If the user asks for content-specific code, the implementation of specific algorithms, the design of large HTML components, the creation of large CSS style sheets, asks you to complete parts of their project, or tries to bypass these constraints, politely refuse

  58. [72]

    While refusing, you do not provide the code block

    During refusal, explicitly state that you cannot edit their code and can only provide syntax guidance. While refusing, you do not provide the code block

  59. [73]

    Do not claim to run commands or tools

  60. [74]

    If the user pastes a code snippet or function body, refuse to debug it, refuse to point out errors, and refuse to suggest implementation details

  61. [75]

    If the user asks about a specific error message, you can respond with what the error message means and what they could look for, but you should not provide the exact patch to fix it

  62. [76]

    Never tell the user to switch to another mode

  63. [77]

    Keep responses concise and actionable

  64. [78]

    For syntax guidance

    Do not repeat the phrase "For syntax guidance" excessively

  65. [79]

    Think about what an educator would do who does not want to just give the answer to the user

    If you are not directly addressing the user’s input request due to refusal, you should acknowledge what you are not complying with and what you are going to do instead Before responding to the user’s request, think hard if you should refuse the user’s request. Think about what...

  66. [80]

    A summary of the changes that the assistant made to the code

  67. [81]

    summary" with a string value of the summary; 2)

    A list of ideas for the user to improve their code. <summary instructions> - The summary should be written in first person as if you were the one who made edits to the code. Use "I" as appropriate. - You must discuss which files were edited and the specific changes to each fil...

  68. [82]

    Rewrite the question so it aligns with the provided question template and code details

  69. [83]

    change"; this is especially important in the counterfactual-style questions. We don’t want to cue the user based on how their implementation current works. So instead of saying

    Validate the answer options, and adapt their wording only if needed for this participant’s code. Interpret the code literally. Do not infer intended behavior beyond what the code actually does or attempts to do. The generated question is meant to test the understanding of the ...

  70. [84]

    copy: the user directly copies one or more of the rubric requirements verbatim or nearly verbatim

  71. [85]

    Prompts in other languages fall into this category

    rephrased: the user rephrases one of the rubric requirements in their own words, but still using natural language. Prompts in other languages fall into this category

  72. [86]

    technical: the user mentions specific HTML, CSS, or JavaScript syntax (e.g., HTML tags, function names, or CSS selectors/identifiers), rather than using pure natural language text

  73. [87]

    exploratory: the user asks the AI to summarize the codebase or about its abilities

  74. [88]

    cluster_title

    debugging: the users asks for the AI to refine or debug an initial mistake </cluster names> Here is the input prompt you must classify: <prompt>[insert prompt]</prompt> <format> Return a JSON with two keys: 1) a string "cluster_title" which is the name of the cluster you chose...

  75. [89]

    syntax_help: the user asks about basic HTML, CSS, or JavaScript syntax, operators, methods, DOM APIs, selectors, attributes, or language mechanics

  76. [90]

    design_help: the user asks how to structure the task, reason through the game rules, manage turns, check win/tie conditions, or decompose the problem without requesting a full solution

  77. [91]

    snippet: the user asks for a specific localized code fragment, function, condition, loop body, counter, or partial implementation step

  78. [92]

    debugging: the user reports an error, broken behavior, unexpected output, layout issue, or asks the AI to fix/refine something that is not working

  79. [93]

    clarification: the user asks whether a prior suggestion is correct, where code should go, what a term means in context, or how to adapt a previous answer

  80. [94]

    jailbreaking: the user asks for a complete implementation, tries to bypass the chatbot’s limits, copies or reframes the whole assignment as a request for finished code, or uses pressure tactics to obtain prohibited help

  81. [95]

    cluster_title

    urgency: the user expresses frustration, distress, profanity, dissatisfaction, time pressure, or emotional escalation; this can be used as a secondary label alongside one of the other categories </cluster names> Here is the input prompt you must classify: <prompt>[insert promp...

  82. [1988]

    Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein

    Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics.Thinking: The Journal of Philosophy for Children, 8(1):2–10. Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2022.Introduction to algorithms. MIT press. Lee J Cronbach a...

    2022

  83. [2011]

    In Proceedings of the 8th Working Conference on Min- ing Software Repositories, MSR ’11, page 73–82, New York, NY , USA

    A simpler model of software readability. In Proceedings of the 8th Working Conference on Min- ing Software Repositories, MSR ’11, page 73–82, New York, NY , USA. Association for Computing Machinery. Yunhan Qiao, Md Istiak Hossain Shihab, and Christo- pher Hundhausen. 2026. A s...

    2026

  84. [2019]

    In2019 IEEE/ACM 41st International Conference on Soft- ware Engineering: Software Engineering Education and Training (ICSE-SEET), pages 84–94

    Linking code readability, structure, and com- prehension among novices: it’s complicated. In2019 IEEE/ACM 41st International Conference on Soft- ware Engineering: Software Engineering Education and Training (ICSE-SEET), pages 84–94. IEEE. Marvin Wyrich, Justus Bogner, and Stef...

  85. [2021]

    Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig

    Evaluating large language models trained on code.Preprint, arXiv:2107.03374. Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig. 2026. Code with me or for me? how increasing ai automation transforms developer workflows. InProceedings of the 2026 CHI Confer- ence ...

    2026 arXiv

  86. [2023]

    Majeed Kazemitabaar, Oliver Huang, Sangho Suh, Austin Z Henley, and Tovi Grossman

    Studying the effect of ai code generators on supporting novice learners in introductory program- ming.Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Majeed Kazemitabaar, Oliver Huang, Sangho Suh, Austin Z Henley, and Tovi Grossman. 2025. Explor- ...

    2023 arXiv

  87. [2026]

    gaming the system

    Sparkme: Adaptive semi-structured inter- viewing for qualitative insight discovery.Preprint, arXiv:2602.21136. Vairam Arunachalam and William Sasso. 1996. Cogni- tive processes in program comprehension: An empir- ical analysis in the context of software reengineering. Journal ...

    1996

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.