{"id":"83647e96-26ea-41a8-97c9-e33c5a7dd7a3","arxiv_id":"2501.02092","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In interviews with 20 engineers, AI-assisted code reviews were found to reduce emotional strain but increase cognitive load, with adoption limited by trust and missing context.","lead":"Software engineers were interviewed after receiving both human and ChatGPT reviews of their own code. The study finds that AI reviews feel less emotionally stressful but demand more mental effort, and that trust and lack of context limit how much AI feedback engineers adopt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM-vs-peer comparison is built on one context-free ChatGPT prompt; 'lack of context' and 'excessive detail' may be prompt artifacts, so the abstract's general claims about LLM-assisted review are not yet externally supported.","rationale":"The reader's weakest_assumption correctly identifies the context-free prompt as the load-bearing point. I agree that this is the single most important threat to the central claim. The paper is a well-conducted qualitative study: the interview design is anchored in actual reviews, coding was iterative and checked by a second author, saturation was monitored, member checking was performed with 19 of 20 participants, and a shared package is provided. These features support the internal validity of the descriptive findings. However, the central comparative claims in the abstract and Section IV — specifically that LLM feedback is harder to process because of excessive detail and that adoption is constrained by lack of context — depend on the specific way the LLM review was generated. A generic prompt without codebase context is not representative of many real LLM code-review deployments, so the findings may not generalize beyond this configuration. The proposed test would settle whether the observed constraints are intrinsic to LLM-generated reviews or artifacts of prompt design. Since the reader already assigned a CONDITIONAL verdict with high confidence, and my concern is the same one, no verdict change is needed.","tokens_in":18642,"tokens_out":3363,"duration_ms":37722,"concrete_test":"Re-run a within-subjects follow-up with the same or a comparable sample of engineers, using each author's submitted code. Each participant receives two LLM-generated reviews: (A) the original context-free prompt from Section III.B, and (B) a context-aware prompt that includes a README, related files, tests, and a brief PR description. After each review, participants rate perceived cognitive effort, trust, and willingness to adopt, and list reasons for non-adoption. If condition (B) substantially reduces \"lack of context\" complaints and perceived cognitive load relative to condition (A), the paper's central constraints are prompt artifacts. If (A) and (B) produce similar ratings, the constraints are intrinsic to LLM reviews and the original claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-assisted reviews reduce emotional-regulation needs but raise cognitive load, and that adoption is constrained by trust and lack of context, depends on the study's operationalization of an LLM review. Section III.B describes the prompt: \"You are an expert of [the programming language]. Provide a thorough review of the attached code.\" This gives the model no repository context, no tests, no PR description, and no project conventions. The two headline differences are exactly what such a prompt would be expected to produce: a generic, exhaustive review (hence \"excessive details\" and higher cognitive load) that cannot reference codebase-specific knowledge (hence \"lack of context\"). The paper's own framing in Section IV treats these as properties of LLM-assisted review in general, but they are properties of this particular zero-context configuration. Modern LLM code-review tools can operate on a full repository with tests and PR metadata, so the comparison may understate the usefulness and overstate the cognitive cost of LLM reviews. This is not an internal inconsistency; the qualitative analysis is coherent and well supported by member checking and saturation. But it is a serious external-validity threat to the central comparative claim. A second, related point: the human peer reviewers also lacked full codebase context, yet participants attributed missing context only to the LLM, suggesting the finding may reflect expectations about AI rather than a real difference in the information available.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a qualitative interview study (20 software engineers recruited via Prolific with a two-phase pre-screening) in which each participant authored a code sample, received written peer reviews from two to three other participants, and then, during the interview, viewed a ChatGPT-4.0-generated review of the same code. The authors use these artifacts to investigate how engineers perceive and engage with LLM-assisted code reviews compared with human peer reviews. The central findings are that engagement in code review is multi-dimensional (cognitive, emotional, behavioral), that LLM-assisted reviews reduce the need for emotional regulation because of their consistently polite tone, that LLM-generated feedback sometimes imposes higher cognitive load because of its excessive detail, that engineers apply a similar sense-making process to feedback from both sources, and that adoption of LLM feedback is constrained by trust and by the LLM's lack of codebase context. The paper also derives practical implications about emotional intelligence training, feedback constructiveness, and personalization of AI feedback delivery.","tokens_in":19029,"tokens_out":2307,"duration_ms":24130,"significance":"If the findings hold, the paper makes a useful contribution to the emerging literature on human-AI collaboration in software engineering by moving beyond technical evaluations of LLM code review to the socio-emotional experience of engineers. The qualitative methodology is careful in several respects: iterative pre-screening using task-oriented questions, saturation monitoring, member checking with 19 of 20 participants, peer debriefing, and a shared documents package that includes code, reviews, prompts, and the interview guide. The claims in the findings are generally hedged with terms such as 'sometimes' and 'some engineers,' which is appropriate for an interview study. However, the comparison underpinning the headline claims rests on a single generic, context-free LLM prompt, and this operationalization substantially limits the external validity of the central comparative conclusions.","major_comments":[{"comment":"The LLM-assisted review condition is operationalized with the single prompt 'You are an expert of [the programming language]. Provide a thorough review of the attached code.' This prompt gives the model no repository context, no pull-request description, no tests, and no project conventions. The two headline differences in the findings—higher cognitive load from 'excessive details' (Section IV.A) and adoption constrained by the LLM's 'lack of context' (Section IV.C)—are exactly what such a zero-context prompt would be expected to produce. The abstract and Section IV frame these as properties of LLM-assisted review in general, but they may be properties of this particular prompt configuration. Because this is load-bearing for the central comparative claim, the paper should either temper the generalization or add robustness evidence, for example by varying the prompt to include repository context and showing whether the same cognitive-load and lack-of-context findings persist.","section":"III.B"},{"comment":"The human peer reviewers in the study also lacked access to the author's full codebase, tests, and project history, yet participants are reported as attributing missing context specifically to the LLM ('ChatGPT is not completely aware of what the context of the code you are writing...' per P18). The paper does not address this asymmetry. It is possible that the 'lack of context' finding reflects participants' expectations about AI tools rather than an actual difference between the two review conditions. The limitations section (Section VII) acknowledges the artificial anonymous setting but does not grapple with this specific confound. The authors should discuss whether they probed participants about the peers' equally limited context, or should otherwise present the finding as an expectation-driven perception rather than a verified property of LLM reviews.","section":"IV.A, IV.C, and VII"},{"comment":"The limitations and trade-offs section is candid about anonymity and the artificial research setting, but it omits the most consequential limitation: the LLM review was produced with a single generic prompt and no codebase context, and this choice is what drives the 'lack of context' and 'excessive details' findings. Adding a statement that the results characterize one particular zero-context LLM review configuration, and that modern LLM review tools may operate with repository and PR metadata, is necessary to prevent over-generalization of the abstract's claims.","section":"VII"}],"minor_comments":[{"comment":"In Table I, a few rows list the participant as their own reviewer (for example, P6 lists reviewers 'P6, P11, & P14' and P12 lists 'P4, P12, & P19'). This appears to be a typographical error in participant IDs and should be corrected, since it affects the transparency of the review-assignment description.","section":"Table I"},{"comment":"The phrase 'the cognitive load sometimes is higher' is appropriately hedged, but the only direct supporting quotation in Section IV.A is from P14. Consider adding one or two additional participant excerpts to strengthen the evidence for the cognitive-load finding beyond a single voice.","section":"Abstract and Section IV"},{"comment":"There are several typos and formatting issues: 'hese engagement attributes' in the Introduction should be 'these engagement attributes'; 'whic' in Section VII should be 'which'; the table caption 'CHARACHTERESTICS' should be 'CHARACTERISTICS'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The Introduction states that the authors used ChatGPT 'for its wide accessibility and familiarity among a diverse audience,' which is reasonable, but the choice of ChatGPT 4.0 over other LLM-based review tools should be mentioned in the limitations section, since different models and tools may differ substantially in verbosity, tone, and context handling.","section":"I"}],"recommendation":"major_revision","confidential_remarks":"The core qualitative analysis is coherent and the methods are above the usual bar for interview studies in SE, including member checking and a shared data package. My main concern is that the central comparison is built on a single context-free prompt, and the paper's generalizing language in the abstract and conclusion does not yet reflect that constraint. This is fixable by rephrasing the scope and adding a robustness discussion, so I do not see this as a reject. I would also encourage the authors to verify the reviewer-assignment table for self-review typos before resubmission, as those are easy to fix but undermine trust in the data description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper does something genuinely useful: it gives us a compact framework (cognitive, emotional, behavioral engagement plus sense-making) and a concrete, counterintuitive trade-off from 20 engineers who each got both peer reviews and a ChatGPT-4 review of their own code. The finding that LLM feedback reduces emotional regulation needs but increases cognitive load is clearly supported by the interview data. Methodologically it's careful: prescreening, saturation tracking, member checking with 19 of 20 participants, peer debriefing, and a shared package that includes the actual code and reviews. The claims are hedged appropriately.\n\nThe soft spot is the one the stress-test note points to, and it's real. The LLM condition used a single context-free prompt (\"You are an expert of [language]. Provide a thorough review...\"). So the two headline differences—excessive detail and lack of context—are exactly what that prompt would generate. The participants' human reviewers also had no codebase context, yet the engineers complained about missing context only from the LLM; that asymmetry suggests the finding is partly about expectations of AI rather than a real informational gap. The paper itself even acknowledges (Section III.B, citing [49]) that a prompt without context can generate unrealistic feedback, then proceeds to use one. That's a validity threat to the abstract's general claims about 'LLM-assisted review,' though not to the internal coherence of the qualitative analysis.\n\nThere are two smaller issues: the interview transcripts are not yet available (promised on acceptance), which limits independent audit; and the anonymous Prolific setting is a reasonable trade-off but weakens comparison to real workplace reviews. Neither is disqualifying.\n\nNet: this is an honest, well-executed qualitative study that needs to be scoped. The engagement framework and the emotional/cognitive trade-off deserve a serious referee. I'd push the authors to either add a context-aware LLM condition or clearly frame the findings as about zero-context LLM reviews, and to make transcripts available. I'd accept it with major revision, not desk reject. I'd cite it if I'm writing about human-AI interaction in SE; it's a solid empirical anchor. I'd probably bring it to reading group, mostly because the prompt-artifact issue is a good methodological lesson.","headline":"A well-run interview study with a real finding, but the LLM condition is a single context-free prompt, so the headline 'lack of context / excessive detail' results need scoping.","tokens_in":19391,"tokens_out":2815,"would_cite":true,"duration_ms":26516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-assisted code reviews reduce emotional strain but increase cognitive load for engineers.","keywords":["code review","large language models","human-AI collaboration","engagement","emotional regulation","cognitive load","sense-making","software engineering"],"falsifier":"Run the same study design but feed the LLM the full repository context—surrounding modules, coding standards, and change history—before it reviews the code, and measure engineers' trust, adoption, and cognitive load; if the 'lack of context' complaint and the trust deficit vanish, those constraints are artifacts of the prompt rather than general properties of LLM reviews. A simpler check is to replicate with prompt variants to see whether verbosity, and hence cognitive load, is model- and prompt-dependent.","tokens_in":18450,"feed_emoji":"🤖","tokens_out":6260,"duration_ms":58060,"temperature":0.7,"pith_summary":"The paper tries to establish that code review engagement is multi-dimensional—cognitive, emotional, and behavioral—and that swapping human peers for a large language model changes the balance among those dimensions. It claims that LLM-generated feedback is less emotionally taxing because its tone is consistently polite, so engineers need fewer coping and emotional-regulation strategies, but that the same feedback is more cognitively demanding because it is verbose and overloaded with details. It further claims that engineers evaluate and adopt feedback through the same sense-making process whether the reviewer is a peer or an LLM, yet adoption of LLM suggestions is gated by trust in the model and by the review's lack of codebase-specific context. The study matters because code review is a human-centric socio-technical process; knowing how AI shifts its emotional and cognitive costs guides how to design AI-human workflows that are actually adopted.","feed_headline":"LLM code reviews are kinder to feelings but harder on attention","feed_subtitle":"Twenty engineers report less emotional strain with AI feedback, yet trust gaps and missing context slow adoption.","key_machinery":"The load-bearing framework is the three-dimensional engagement model (cognitive, emotional, and behavioral), coupled with a sense-making process that links feedback to adoption. A review has two components—content and delivery—and delivery style triggers emotional and behavioral responses, while content and clarity determine cognitive effort. The model adds a 'reviewer context' input: for peers it is seniority and familiarity; for LLMs it is the engineer's trust in the model and its known limitations, such as missing codebase context. This framework organizes the interview data and locates the study's central finding, the reversal between emotional ease and cognitive effort when the reviewer is a machine.","core_discovery":"The paper's central claim is that engagement in code review comprises cognitive, emotional, and behavioral dimensions, and that LLM-assisted review alters each dimension in a distinct way. Compared with peer review, LLM review lowers the emotional cost: engineers report less need for emotional regulation, coping mechanisms, and personal-value-guarding behaviors because the machine's delivery is uniformly positive and never 'picky' or 'brutal.' The trade-off is a higher cognitive load, since the LLM's lengthy, detailed output takes more time and mental effort to analyze than the more straightforward comments of human reviewers. Engineers still work through the same reflective sense-making process—going suggestion by suggestion, deciding what makes sense and what to adopt—but for LLM output that process is constrained by two reviewer-context factors: trust in the model's capabilities and the absence of codebase-specific context. The paper concludes that the introduction of LLM review brings an emotional-cognitive trade-off rather than a strict improvement or worsening.","pith_inferences":["The LLM condition in the study used a single generic prompt with no codebase context, so the reported 'lack of context' barrier is at least partly an artifact of prompt design; providing the LLM with repository context, coding standards, and change history would likely weaken or erase that constraint—an untested but direct consequence of the paper's own reviewer-context model.","A natural field experiment follows: hold the engineer and code fixed, vary only the amount of context given to the LLM, and measure trust, adoption rate, and perceived cognitive load; the prediction from the paper's model is that context-rich prompts raise trust and lower the perceived verbosity penalty.","The model may generalize beyond code review to other human-AI writing and analysis feedback loops, where content volume and delivery style trade off against emotional safety and cognitive effort.","Several interviewees already proposed a division of labor—LLM as exhaustive first-pass reviewer, human as contextual and relational judge—which the paper's trade-off structure supports as a likely steady-state arrangement for AI-human collaboration."],"forward_implications":["Organizations adopting LLM-assisted code review can expect fewer emotionally charged interactions and less need for emotional regulation, but engineers may need help managing the heavier cognitive load of verbose AI feedback.","Because engineers use the same sense-making process for both sources, LLM suggestions will not be adopted on authority; they must survive the same line-by-line scrutiny as peer comments.","Trust and missing codebase context are the binding constraints on LLM feedback adoption, so tools that supply repository context or clearly flag their confidence levels could directly raise adoption.","The divergence in delivery-style preferences means a one-size-fits-all LLM review format will leave some engineers dissatisfied; personalization of tone and detail is a design requirement.","The emotional-cognitive trade-off implies that summarization or filtering of LLM output into concise, actionable comments could shift the balance substantially in favor of AI-assisted review."],"supporting_citations":[{"why":"Documents that AI-generated content tends to be overly polite, overly detailed, and lacking in trustworthiness; the paper's findings on tone and verbosity extend this prior observation.","marker":"[5]"},{"why":"Shows that developers prefer customizable bot interactions, which the paper draws on to argue that AI feedback delivery should be personalized to individual preferences.","marker":"[26]"},{"why":"Demonstrates that reporting raw tool output is insufficient for developers to adopt static analysis findings, informing the paper's sense-making and trust interpretation.","marker":"[35]"},{"why":"Reports that developers are annoyed when AI assistants lack local codebase context, directly supporting the paper's finding that missing context constrains adoption of LLM feedback.","marker":"[47]"},{"why":"Argues that code review tasks and context evolve over time, which statically trained LLMs may miss, supporting the paper's context-limitation constraint.","marker":"[64]"},{"why":"The few-shot learning result underlies the paper's expectation that a generic expert prompt would generate relatable, expert-level review feedback for comparison.","marker":"[16]"}],"fun_headline_variants":["LLM code review: Less emotional toll, more mental load","AI review spares feelings but strains focus","LLM feedback: kinder delivery, heavier reading","Code review with AI: Lower emotional cost, higher cognitive price","AI code review: easier on ego, harder on attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on one generic LLM prompt with no codebase background standing in for LLM-assisted code review; if the model had been given the surrounding repository context, the trust and context barriers the paper reports might shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLM code review: Less emotional toll, more mental load","AI review spares feelings but strains focus","LLM feedback: kinder delivery, heavier reading","Code review with AI: Lower emotional cost, higher cognitive price","AI code review: easier on ego, harder on attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1546,"prompt_tokens":981,"completion_tokens":565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":597,"tokens_out":565,"duration_ms":5018,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:30.278389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same study design but feed the LLM the full repository context—surrounding modules, coding standards, and change history—before it reviews the code, and measure engineers' trust, adoption, and cognitive load; if the 'lack of context' complaint and the trust deficit vanish, those constraints are artifacts of the prompt rather than general properties of LLM reviews. A simpler check is to replicate with prompt variants to see whether verbosity, and hence cognitive load, is model- and prompt-dependent.","supporting_citations":[{"cited_title":"Autonomy is an acquired taste: Exploring developer preferences for github bots,","cited_arxiv_id":null,"evidence_quote":"Shows that developers prefer customizable bot interactions, which the paper draws on to argue that AI feedback delivery should be personalized to individual preferences."},{"cited_title":"Why don’t software developers use static analysis tools to find bugs?","cited_arxiv_id":null,"evidence_quote":"Demonstrates that reporting raw tool output is insufficient for developers to adopt static analysis findings, informing the paper's sense-making and trust interpretation."},{"cited_title":"A large-scale survey on the usability of ai programming assistants: Successes and challenges,","cited_arxiv_id":null,"evidence_quote":"Reports that developers are annoyed when AI assistants lack local codebase context, directly supporting the paper's finding that missing context constrains adoption of LLM feedback."},{"cited_title":"An empirical investigation of relevant changes and automation needs in modern code review,","cited_arxiv_id":null,"evidence_quote":"Argues that code review tasks and context evolve over time, which statically trained LLMs may miss, supporting the paper's context-limitation constraint."}],"review_version":1}