{"id":"2864ab9c-74ef-41aa-9d22-0290113d5729","arxiv_id":"2607.26375","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using a coding agent improved initial task completion but lowered users' comprehension of their own code, with no net benefit on a later no-agent extension task.","lead":"In a 54-student study, people who used a coding agent that edits their code finished a website task faster and more accurately than people who used a syntax-only chatbot, but they scored much lower on tests of whether they understood their own code. The paper argues that task-completion metrics for coding agents miss this loss of understanding, which matters for oversight, learning, and extending code later.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal 'thus do not prepare users to extend' rests on a saturated, non-pre-registered mediation model; total extension effect is non-significant, so the headline overstates the downstream consequence.","rationale":"The reader's weakest assumption focuses on the validity/group-fairness of LLM-generated comprehension questions. That is a real threat, but the paper defends it with multiple checks (Appendix A.9), and the comprehension deficit is large and consistent across question types. The more load-bearing problem is the causal extension claim: the abstract's 'thus do not prepare users to extend' is the part that would generalize to real-world productivity, yet it rests on a non-significant total effect and a post-hoc saturated mediation model. Even if comprehension were measured perfectly, the downstream claim is not identified. The authors themselves label the path model as a simplified summary with no fit statistics, which is honest but conflicts with the abstract's phrasing. A non-saturated, pre-registered mediation model or a direct replication with a powered total-effect test would settle whether the trade-off interpretation holds. The verdict should remain conditional: the comprehension harm is credible, but the extension-preparation harm needs stronger evidence.","tokens_in":38181,"tokens_out":4057,"duration_ms":51304,"concrete_test":"Re-estimate the Appendix A.5 path model with (1) the background×condition interaction included and (2) at least one additional freely estimated path (e.g., direct Group→Extension) so the model has positive df; report RMSEA/CFI and bootstrapped 95% CIs for the indirect effect through comprehension. If the comprehension-mediated indirect effect is not significant or the model shows poor fit, qualify the abstract's causal claim to a trade-off/correlation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's second clause—'thus do not prepare users to extend their code'—goes beyond the data. In §4.3/Appendix A.5, the authors support it with a saturated observed-variable path model (Fig. 9) that has 0 df, no fit statistics, omits the background×condition interaction used in the primary regressions, and is fit post-hoc to n=54. The total effect of condition on extension accuracy is non-significant (p=0.45, d=0.2; even the initial>0.50 subgroup is p=0.18). The claim therefore depends on an indirect effect through comprehension in a model that is unfalsifiable as specified. Unmeasured confounders (e.g., time spent writing code, number/type of prompts, actual code scaffold quality) could create the opposing indirect paths. Even if the LLM comprehension questions are perfectly group-fair, the policy-relevant downstream harm is not established. The authors are transparent about the model's summary nature, but the abstract states the causal consequence without this caveat. This is load-bearing because the 'harm comprehension' part is robust; the 'thus do not prepare' part is what would change evaluation practice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a between-subjects user study (n=54 CS students) comparing a code-editing agent with a syntax-only chatbot on a web-development task. It measures initial task accuracy, comprehension via LLM-generated recall and reasoning questions, and performance on an extension task done without the agent. The headline findings are that agent users complete the initial task more accurately and quickly, score substantially lower on comprehension (d≈0.9), and do not differ significantly on extension accuracy. Secondary analyses examine background ability, prompting and review strategies, code readability, and self-reported preferences. The paper interprets these results as showing that task-completion-only evaluations of coding agents neglect user understanding and that this has downstream consequences for users' ability to extend their code.","tokens_in":38408,"tokens_out":7651,"duration_ms":83818,"significance":"The result is potentially important: if it holds up, it suggests that benchmark-driven coding-agent development is optimizing a metric decoupled from users' ability to reason about their own code, with implications for evaluation practice and agent design. The study has genuine strengths. The central comprehension finding is measured with external behavioral instruments rather than derived from fitted constants, and the authors provide multiple validation checks for the LLM-generated questions and the rubric-based judge. The released dataset and interface are valuable assets, and the self-report analyses in Appendix A.6 show that perceived understanding tracks measured understanding. The main caveat is that the causal 'thus do not prepare users to extend their code' clause is not supported by the non-significant total effect on extension accuracy; the paper's headline and abstract overstate the strength of the evidence for the downstream consequence. The robust contribution is the comprehension difference and the study design; the policy-relevant claim about extension ability needs to be reframed or supported by a stronger mediation analysis.","major_comments":[{"comment":"The headline claim that agents 'thus do not prepare users to extend their code' goes beyond the results. The total effect of condition on extension accuracy is non-significant in the main comparison (Figure 1: p=0.45, d=0.2) and in the initial-accuracy >0.50 subgroup (p=0.18). The causal wording is instead supported by the saturated observed-variable path model in Figure 9, fit post hoc to n=54 with 0 df, no global fit statistics, and without the background×condition interaction used in Eq. (1). Appendix A.5 is transparent that this is a 'compact summary,' but the abstract and the §4.3 heading ('can Degrade Extension Ability') present the downstream harm as established. Please either provide a properly identified and pre-specified mediation analysis with sensitivity checks, or reframe the conclusion as an exploratory mechanism consistent with, but not established by, the data.","section":"Abstract; §4.3; Appendix A.5; Figure 9"},{"comment":"The comprehension difference is the paper's central behavioral finding, so the group-fairness of the LLM-generated questions is load-bearing. The validations in Appendix A.9 are careful, but they do not establish measurement invariance. Agent users submitted longer and more complete code (Table 5), and the overall d=0.9 is driven by 'identify own code' (d=1.873) and JS questions (d=1.089), while 'change' questions show no difference (p=0.916, d=0.030). Without an externally validated concept inventory (a gap the authors acknowledge in §3.1), it remains possible that some of the apparent comprehension deficit is an artifact of systematically harder questions about more complex code. I would ask for an independent norming study—e.g., coders blind to condition answering the same generated items—or an item-level analysis demonstrating that difficulty is invariant across groups; at minimum, t","section":"§3.3; Appendix A.9; Table 7"},{"comment":"The statement in §4.3 that 'A path mediation model confirms this' overstates what a saturated post-hoc model can confirm. With 0 degrees of freedom, the model in Figure 9 cannot be tested against the data, and omitting the background×condition interaction used in the primary regressions means the specified paths may not match the earlier analysis. Unmeasured confounders such as time spent, prompt count and type, and code quality could induce opposing indirect paths. Please present the mediation as descriptive rather than confirmatory, and report the sensitivity of the indirect effects to plausible confounders.","section":"§4.3; Figure 7; Appendix A.5"}],"minor_comments":[{"comment":"The caption states that 'Agent users can better extend their code without agents,' but the reported comparison is non-significant (p=0.45, d=0.2). Please rephrase to 'no significant difference' or 'slightly, not significantly, higher' to match the statistics.","section":"Figure 1 (caption)"},{"comment":"The R² change is reported as 0.05→0.34 in the text and 0.05→0.22 in the Figure 7 caption; please reconcile this inconsistency.","section":"§4.3; Figure 7"},{"comment":"The main text reports mean comprehension for users who 'use' each prompting/review strategy, but users can engage multiple strategies; the Appendix A.4 regressions handle this. Please note the non-exclusive nature in the main text or refer readers to A.4 before presenting the averages.","section":"§5.1; Tables 1-2"},{"comment":"The regression results are shown as coefficient plots without a table of estimates. Providing a table with coefficients, standard errors, and p-values for all predictors would improve reproducibility.","section":"§4.2; Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper makes a valuable and largely credible empirical contribution. My main reservation is that the abstract and §4.3 overstate the downstream extension claim relative to the non-significant total effect, and the mediation model is saturated and post hoc. This is fixable by reframing and adding sensitivity analyses. The comprehension measurement-invariance concern could change the headline if it materializes, so I would require additional validation or a hedged claim. I recommend major revision, not rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful study with one robust finding and one claim that outruns its evidence. The robust finding — that an agent writing code for people gets the task done but leaves them knowing their own code worse than people who wrote it themselves — is worth taking seriously. The overreach is the second clause of the abstract: \"thus do not prepare users to extend their code.\" That specific downstream consequence is not established by the data.\n\nWhat's new and good: they measure comprehension of users' own code after agent use, not just skill transfer or ownership. The comprehension questions are personalized to each user's code and they put real work into validating them — manual review, LLM-judge checks, pass-rate analyses, side-by-side comparisons across groups. The main between-group comprehension difference is large (d≈0.9) and consistent across question types except change-questions. They also release data and code. The regression showing task accuracy barely predicts comprehension (adj R² change .40→.42) is a useful conceptual point: completion is not understanding.\n\nSoft spots, in proportion. The extension-task claim is the weakest. The total effect of condition on extension accuracy is non-significant (p=0.45, d=0.2), and the \"initial > 0.50\" subgroup is also non-significant (p=0.18). The path model in Fig. 9 is saturated (0 df), fit post-hoc to n=54, omits the background×condition interaction used in the primary analyses, and has no fit statistics. The authors are transparent about this in Appendix A.5, but the abstract states the causal consequence without the caveat. Unmeasured confounders (prompt count, time on task, code scaffold quality) could drive the indirect paths. So the \"harm comprehension\" half holds; the \"prevent extension\" half is a reasonable hypothesis, not a result. The interaction/prompt-strategy analyses in §5 are explicitly exploratory and should be read that way. The comprehension measure itself is well-defended but not externally validated against a concept inventory; that is a minor caveat, not a fatal one.\n\nWho is this for? Anyone evaluating coding agents beyond task-completion benchmarks, and anyone working on human-AI collaboration or computing education. It deserves a serious referee. I'd send it out, and ask for a revision that separates the robust comprehension finding from the speculative extension claim — either by reframing the abstract, or by pre-registering a mediation model with testable fit.","headline":"Solid comprehension result, overreaching extension claim — worth refereeing with revisions.","tokens_in":38926,"tokens_out":2519,"would_cite":true,"duration_ms":27979,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A user study with 54 programmers shows that coding agents that edit users' code make the initial task easier but measurably weaken users' understanding of their own code, and do not improve their ability to extend it without the agent.","keywords":["coding agents","code comprehension","user study","human-AI interaction","productivity vs understanding","program comprehension","LLM evaluation","agentic coding"],"falsifier":"A replication in which an independent panel writes comprehension questions from anonymized submissions without knowing which group produced them, and finds agent and chatbot users scoring equally, would falsify the main claim. So would a replication where agent users' extension accuracy remains higher even after matching initial task accuracy.","tokens_in":38048,"feed_emoji":"🤖","tokens_out":4384,"duration_ms":49405,"temperature":0.7,"pith_summary":"This paper tries to show that the usual way of evaluating coding agents—by whether they finish a task—misses a real cost: users who prompt an agent to write code understand that code much less than users who write it themselves. In a study, 54 computer science students built a tic-tac-toe-style website using either an agent that directly edits code or a chatbot that only gives syntax help. The agent group finished faster and more accurately, but scored far lower on comprehension questions about their own code and no higher on a follow-up task where they had to extend that code without the agent. Lower-effort interactions such as copy-pasting prompts and auto-accepting edits were linked to lower comprehension, while more readable agent-written code went with better comprehension. Users noticed they understood less, yet still preferred the agent because it was quick and easy.","feed_headline":"Coding agents boost output but weaken code comprehension","feed_subtitle":"Users who offload code writing score far lower on understanding tests and gain no edge at extending their code.","key_machinery":"The argument is carried by a controlled two-group experiment. One group works with an agent that directly edits their code; the other works with a chatbot that only offers short syntax guidance, so users must write or adapt code themselves. Understanding is measured two ways: automatically generated comprehension questions tailored to each user's own code—covering recall of what is in the code and reasoning about what changed code would do—and an extension task where users add features to their submitted code with only the chatbot. This pair of metrics makes 'understanding' concrete and user-specific, letting the paper compare productivity gains against comprehension losses.","core_discovery":"The paper establishes a productivity–understanding gap: coding agents that edit users' code substantially improve initial task accuracy but substantially reduce users' comprehension of their own code, and they do not improve users' ability to extend that code once the agent is gone. In regressions, initial task accuracy barely predicts comprehension, while comprehension and initial accuracy together strongly predict extension accuracy, so the agent's better initial scaffold is offset by the user's weaker understanding. The paper concludes that task-completion-only evaluation of coding agents is incomplete: understanding one's own code is a distinct, measurable objective that current agents n","pith_inferences":["This suggests the comprehension gap may grow over time: if understanding is not built during each task, repeated agent use could erode users' mental models of their systems, making later oversight failures more likely—an extension the paper does not directly test.","The same productivity-versus-understanding trade-off likely applies beyond web development, for example in data analysis or algorithm design, wherever code writing is offloaded; the paper's protocol could be adapted to those domains.","A testable design remedy: require users to explain each accepted agent edit before it is applied. The paper's correlation between active per-file review and higher comprehension suggests such forced engagement could recover some understanding without forcing users to write everything from scratch.","Coding-agent evaluation could adopt lightweight comprehension probes alongside task benchmarks, making understanding a first-class metric rather than an afterthought."],"forward_implications":["If the central claim is correct, task-completion benchmarks for coding agents overstate their value to users, and evaluations should include comprehension or other understanding-based measures.","Users who rely on agents may be poorly prepared to debug, oversee, or explain their own code when the agent is unavailable or fails.","Low-effort interaction patterns, such as copying requirements verbatim or auto-accepting edits, mark users at particular risk of low comprehension, suggesting agents could be designed to discourage those patterns.","Agent-generated code that is more concise and readable is associated with better user comprehension, making readability a plausible training objective for coding agents.","User preference alone will not fix the problem, since users in the study preferred the agent even while recognizing their understanding was weaker."],"fun_headline_variants":["AI coding agents: faster builds, weaker understanding","Coding agents save time but cost understanding","Productivity up, comprehension down: the agent trade-off","AI agents: faster delivery, slower learners","Coding agents boost task speed but impair comprehension"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated comprehension questions measure genuine understanding fairly across both groups—if the questions were systematically easier or harder for one group's code, the observed comprehension gap could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["AI coding agents: faster builds, weaker understanding","Coding agents save time but cost understanding","Productivity up, comprehension down: the agent trade-off","AI agents: faster delivery, slower learners","Coding agents boost task speed but impair comprehension"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3154,"prompt_tokens":702,"completion_tokens":2452,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":2391}},"tokens_in":446,"tokens_out":2452,"duration_ms":16236,"temperature":1.0,"reasoning_tokens":2391,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:07:29.055431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication in which an independent panel writes comprehension questions from anonymized submissions without knowing which group produced them, and finds agent and chatbot users scoring equally, would falsify the main claim. So would a replication where agent users' extension accuracy remains higher even after matching initial task accuracy.","supporting_citations":[],"review_version":1}