{"id":"a5aa763a-e1f1-4df1-87fe-d550b8a907ff","arxiv_id":"2505.16339","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a real-world experiment, developers preferred an AI-led code review mode over an interactive assistant, especially for large or unfamiliar pull requests.","lead":"This paper reports a field study and experiment at a Swedish automotive software company testing two ways for developers to get LLM help while reviewing code. Most developers preferred the AI-led mode, especially for large or unfamiliar pull requests, but only when they trusted the results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'especially for large or unfamiliar pull requests' scope of the central claim rests on hypothetical self-reports, not on controlled variation of PR size or reviewer familiarity; the two actual PRs were moderate and their similarity is unquantified.","rationale":"The paper is a thoughtfully conducted qualitative field study with a real artifact, real developers, and openly available source code. The reader's CONDITIONAL verdict is appropriate. My concern overlaps with the reader's weakest assumption about PR comparability, but I see the more load-bearing issue as the gap between the empirical design and the scope of the central claim: the design does not vary PR size or familiarity as factors, and the 'especially for large or unfamiliar pull requests' conclusion comes substantially from hypothetical self-reports rather than from observed large-PR experiences. This is not an internal inconsistency or a sign of dishonesty; it is a classic scope overreach in qualitative research, where participants' imagined use cases are reported alongside observed behavior without clear separation. The concrete test would quantify PR similarity and re-code the transcripts to distinguish observed from hypothetical preference statements. If the re-analysis confirms that preferences were tied to actual moderate-sized PRs, the paper's contribution remains useful but its headline should be narrowed. Since the reader already assigned CONDITIONAL, my read does not change the verdict; it sharpens the specific revision needed.","tokens_in":11987,"tokens_out":3022,"duration_ms":29940,"concrete_test":"Obtain the two internal pull requests from WirelessCar (or their sanitized diffs) and compute quantitative complexity metrics: lines changed, files touched, number of commits, cyclomatic complexity delta, and number of touched modules/services. Then re-code the Phase 2 interview transcripts and observation notes to separate (a) preferences explicitly anchored in the actual PR the participant reviewed from (b) hypothetical statements about imagined large or unfamiliar PRs. If the PR paired with Mode A is not materially larger or more complex than the other, or if most 'large PR' preference statements are hypothetical, the 'especially for large or unfamiliar PRs' sub-claim should be reworded as an exploratory expectation rather than a finding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the AI-led mode was 'generally preferred, especially for large or unfamiliar pull requests' (Section I), with preferences context-dependent. To support this, the field experiment in Section IV-B2 uses two pull requests that are only asserted to be 'of similar size and complexity,' with no quantitative metrics such as lines changed, files touched, or complexity measures. The pilot to validate suitability involved only two internal developers. More importantly, the 'large or unfamiliar' part of the claim is not actually manipulated in the design: both PRs were moderate in size, so no participant experienced a genuinely large PR in the study. The statements supporting this sub-claim in Section V-B4 are largely hypothetical ('if I were in a new team...', 'especially for large PRs, it's nice...'), which is acceptable as an expressed expectation but not as evidence of observed preference in large-PR contexts. Additionally, the paper does not report a systematic tally of mode preferences across the ten participants; the thematic analysis quotes selected voices, making it difficult to verify that the AI-led mode was 'generally' preferred rather than merely salient in a few vivid quotes. The load-bearing risk is that the headline finding overstates what a small, moderate-sized, convenience-sampled study can establish: the mode comparison may be confounded by unmeasured differences between the two PRs, and the 'large or unfamiliar' condition is speculation rather than observation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a two-phase, qualitative empirical study conducted at WirelessCar Sweden AB. In Phase 1, semi-structured interviews with seven developers (thematic analysis) identify challenges in existing code review practice: context switching, lack of context, delayed and superficial reviews of large PRs, and mixed views on AI assistance. In Phase 2, ten developers used two variants of an LLM-assisted code review tool in a field experiment: an AI-led co-reviewer mode (Mode A) that proactively generates summaries and findings, and an interactive assistant mode (Mode B) that responds only to on-demand queries. Both modes use a RAG pipeline over PR diffs, source code, and Jira tickets. The paper claims that developers generally preferred the AI-led mode, especially for large or unfamiliar pull requests, but that preferences are context-dependent, with familiarity and perceived risk shaping choices. The authors also report design implications: embed AI in existing tools, keep output concise, support both proactive and reactive modes, and consider pre-review use.","tokens_in":12249,"tokens_out":4462,"duration_ms":38898,"significance":"If the findings are accepted as stated, the paper offers practical guidance for building LLM-assisted code review tools and contributes qualitative evidence from a real industrial setting. Notable strengths include the field context with practicing developers, the use of both observation notes and post-session interviews, the rotation of mode assignment to mitigate ordering effects, and the release of the prototype artifacts under GPLv3. The thematic analysis is careful in places and the paper is candid about its qualitative nature. However, the central preference claim is stronger than the evidence: there is no systematic preference tally, the two PRs are only asserted to be comparable, and the 'large or unfamiliar' scope is based on hypothetical self-reports rather than experimental variation. These issues affect the reliability of the headline results and need to be addressed, either by additional evidence or by appropriately qualified wording.","major_comments":[{"comment":"The abstract and Section V-B4 state that 'AI-led reviews are overall more preferred,' but the paper does not report a systematic tally of mode preferences across the ten participants. The results present a thematic selection of quotes (e.g., P7, P11, P12 for Mode A; P8 for Mode B in familiar code) and do not tell the reader how many participants preferred Mode A, Mode B, or a combination. Without such a distribution, 'overall more preferred' is indistinguishable from 'a few vivid quotes.' Please provide a participant-level preference summary (e.g., a table or explicit counts) or weaken the claim to 'several participants expressed a preference for the AI-led mode.'","section":"Section V-B4 and Abstract"},{"comment":"The comparability of the two pull requests is load-bearing for the comparison of Mode A versus Mode B, yet it is only asserted: the paper says the PRs are 'of similar size and complexity' and were validated by a pilot with two internal developers, but gives no metrics such as lines changed, files touched, requirement complexity, or review effort. If the two PRs differ in difficulty, clarity, or domain, the observed mode preferences could be explained by the PRs themselves. Please report the actual PR characteristics and, ideally, a quantitative similarity justification, or explicitly discuss this threat as a limitation of the mode comparison.","section":"Section IV-B2"},{"comment":"The claim that AI-led reviews are especially preferred 'for large or unfamiliar pull requests' is not supported by the experimental design: both PRs are described as moderate in size, and 'unfamiliar' was not manipulated—the paper compares participants from the owning team with those from other teams, but no systematic familiarity-group comparison is reported. The supporting statements in Section V-B4 are hypothetical self-reports ('if I were in a new team'; 'especially for large PRs, it's nice to get the breakdown'), which express expectations rather than observed behavior in large-PR or objectively unfamiliar contexts. Please reframe these as participant expectations, or add a condition that varies PR size and measures familiarity quantitatively.","section":"Section V-B4 and Section VII"},{"comment":"The paper's own footnote 6 states that 'quotes collected from the interview as well as detailed findings are not listed' for Phase 1. This is a significant transparency limitation because the thematic analysis and the link from RQ1 findings to the design of the two modes cannot be independently assessed. Please provide an appendix or supplementary material containing the full coding scheme, theme definitions, and representative quotes. In addition, the paper should discuss whether the five Phase 2 participants who also took part in Phase 1 may have been influenced by the researchers' earlier design discussions; this potential bias is not acknowledged.","section":"Section V-A, footnote 6"},{"comment":"The study has no no-AI baseline condition: participants compared the two AI-assisted modes with each other and with their memory of traditional reviews, not with a contemporaneous manual review of the same PRs. This is acceptable for exploring mode preference, but the paper's language in several places goes further, e.g., 'the AI assistant could speed up the review process' and 'LLMs can meaningfully augment' (Sections V-B2 and I). Because there is no measured comparison against manual review, such statements should be explicitly framed as participant perceptions or hypotheses, not as demonstrated improvements in efficiency or review quality.","section":"Section IV-B2 and Section V-B2"}],"minor_comments":[{"comment":"In the paragraph beginning 'Additionally, interviewers felt that the assistant would be particularly helpful...', the word 'interviewers' should be 'interviewees' or 'participants'.","section":"Section V-B2"},{"comment":"The experiment used two PRs from WirelessCar's codebase, but Figure 2 shows a review of an open-source project (ogen-go/ogen PR 1440). Please clarify in the caption or text whether this screenshot is illustrative and taken from a demo session rather than from one of the experimental PRs.","section":"Section IV-B2 and Figure 2"},{"comment":"The first sentence of Section V-A has typographical issues: 'When reviewing the informal review process at WirelessCar and their practices ,' contains stray spaces and an awkward phrase; please rephrase and check punctuation throughout the manuscript.","section":"Section V-A"},{"comment":"The sentence 'By having all participants conduct the same two code reviews with alternating modes, this approach allowed for a more controlled comparison...' has a dangling modifier; consider rewriting as 'Having all participants conduct... allowed for...'.","section":"Section IV-B2"},{"comment":"Reference [12] contains a typo: 'V . Braun and V . C. and' should be 'V. Braun and V. Clarke'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a small qualitative study with an honest industrial grounding, but the manuscript's abstract and conclusions overstate the evidence. The missing preference tally, the unquantified PR comparability, and the hypothetical basis for the 'large or unfamiliar' scope are all fixable within the manuscript's scope by providing an appendix, reporting participant-level preferences, and softening the claims. The paper's contribution could be a useful design-oriented field study if these issues are addressed. I do not see grounds for rejection, but the current version should not be accepted without revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nThis is a small, honest qualitative study that compares two interaction modes for an LLM-based code review assistant: an AI-led co-reviewer that produces an upfront summary, and an interactive assistant you query on demand. As far as I know, that direct comparison hasn't been done in a field setting before, and the design is a reasonable first cut: seven interviews to identify pain points, then a ten-participant field experiment with a RAG-backed prototype, the code for which is on GitHub. The thematic analysis is standard and the quotes are well chosen.\n\nThe main findings — developers like faster orientation, the AI-led mode helps with unfamiliar code, and preferences are context-dependent — are plausible and broadly consistent with the data. But the paper overstates what the data support in one specific way. The 'especially for large or unfamiliar pull requests' claim is not actually observed. The two PRs used in the experiment were both moderate in size; no participant reviewed a genuinely large PR. The 'large PR' statements in Section V-B4 are hypothetical ('if I were in a new team...', 'especially for large PRs, it's nice...'). Those are useful expectations, but they don't belong in the abstract as a finding. Related, the paper never reports a systematic tally of which mode participants preferred; we get selected quotes, so 'overall more preferred' is hard to verify. The paper's own footnote says detailed findings are not listed due to space limits, which makes this worse.\n\nThe other soft spot is the comparability of the two pull requests, which is asserted ('of similar size and complexity') without any numbers — lines changed, files touched, complexity measures. If the PRs actually differ in difficulty, the mode comparison could be confounded. A pilot with two internal developers does not establish equivalence.\n\nNone of this kills the paper. It's a legitimate exploratory study, and the authors are transparent about most limitations. The fix is straightforward: add a preference count, quantify the PRs, and reframe the 'large PR' claim as an expected benefit or design implication, not an observed effect. With those changes it would be a useful contribution to the LLM code review literature.\n\nI'd send this to peer review — the question is timely and the study is honest — but I'd expect conditional acceptance with those revisions. It's a maybe for reading group and I probably won't cite it directly, but it's worth knowing about.","headline":"A small, honest qualitative study with a genuinely new mode comparison, but the headline 'large PR' preference is speculative and the paper needs a preference tally and PR complexity metrics before the claims can be trusted.","tokens_in":12760,"tokens_out":2241,"would_cite":false,"duration_ms":18858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Developers prefer AI-led code reviews for large or unfamiliar pull requests, but the preference is context-dependent.","keywords":["code review","large language models","retrieval-augmented generation","field experiment","qualitative study","developer experience","AI-assisted software engineering","pull request"],"falsifier":"A controlled experiment that assigns the same developers to review multiple pull requests with measured size, complexity, and clarity (for example, comparable lines changed, files touched, and cyclomatic complexity), and that records both stated preference and actual review performance such as defects found and time spent, could determine whether the observed mode preference is driven by the interaction mode or by differences in the pull requests themselves.","tokens_in":11834,"feed_emoji":"🤖","tokens_out":8409,"duration_ms":51019,"temperature":0.7,"pith_summary":"This paper presents a field study and field experiment at an automotive software company, examining how LLMs can be integrated into code review. The field study, based on seven interviews, identifies common pain points: delayed reviews, context switching, and missing context for large or complex pull requests. The experiment, with ten developers, compares two tool modes: an AI-led co-reviewer that produces an upfront summary, and an on-demand assistant that answers questions only when prompted. The central finding is that developers generally prefer the AI-led mode, especially for large or unfamiliar pull requests, but that preference is context-dependent, with some developers favoring the on-demand mode for familiar code or high-risk changes. The paper concludes that LLMs can augment human reviewers and that tools should support both proactive and reactive interaction modes.","feed_headline":"AI-led code review wins for big, unfamiliar PRs","feed_subtitle":"Developers prefer proactive summaries, but want on-demand help for familiar or risky code.","key_machinery":"The experimental vehicle is a web-based chat interface wrapping a large language model with retrieval-augmented generation (RAG). The assistant is built on three semantic search tools: search_pr (pull-request diffs and metadata), search_code (full source files), and search_requirements (the requirement ticket motivating the change). In Mode A (AI-led co-reviewer), a sub-agent called start_review is added; it receives the entire PR context and produces a structured summary before the reviewer begins, and the reviewer can then ask follow-up questions. In Mode B (on-demand assistant), no upfront summary is generated and the reviewer must prompt the assistant explicitly. This difference in proactivity is the independent variable the study varies.","core_discovery":"The paper's central claim is that the preferred interaction with an LLM code-review assistant is not a single mode but depends on review context. In the field experiment, participants reviewed two pull requests of similar size and complexity, using the AI-led mode for one and the on-demand mode for the other, with mode assignment rotated. Overall, the AI-led mode was preferred, and it was seen as especially valuable for large or unfamiliar pull requests, where it provided quick orientation and surfaced issues a human might miss. Yet several participants said that for a codebase they already know well, or for changes that carry risk, they would rather keep control with an on-demand assistant. The paper interprets this as evidence that LLM assistance should be adaptive and embedded in existing developer workflows, augmenting rather than replacing human review.","pith_inferences":["If mode preference is genuinely context-dependent, an adaptive tool that automatically selects between proactive and reactive modes based on PR size, codebase familiarity, and risk metadata should outperform any fixed-mode tool; this is a testable extension the paper does not evaluate.","The participants' worry about over-reliance in the AI-led mode suggests a measurable anchoring effect: a reviewer might fixate on AI-flagged issues and miss unflagged ones. An experiment comparing defect detection rates with and without an upfront AI summary could quantify this risk.","The paper's RAG pipeline is manually indexed per pull request, so the qualitative benefits it reports may depend on careful indexing rather than on the LLM itself. Automating the indexing step and re-running the experiment would separate these two contributions."],"forward_implications":["Tool builders should offer an AI-led summary as the default for large or unfamiliar pull requests, while keeping an on-demand mode available for reviewers who want full control.","AI review assistance should be embedded directly into the developer's normal platforms (for example, code-hosting platforms, IDEs, or chat tools) rather than a separate interface, since participants cited integration friction as a barrier.","Response latency and output conciseness are decisive for adoption; at least one participant stated they would not use a tool that takes minutes to reply.","The usefulness of the assistant depends on access to diff, source, and requirement-ticket context, so production tools should invest in automated retrieval pipelines for these artifacts.","The same assistant could serve as a pre-review aid, letting authors catch simple issues before submitting a pull request."],"supporting_citations":[{"why":"Provides the finding that automatically highlighted lines can make reviewers miss other areas, which motivated the design of the passive on-demand mode (Mode B).","marker":"[5]"},{"why":"Qualitative study of developers' emotional and cognitive responses to AI-provided feedback, which the paper extends by comparing interaction modes.","marker":"[9]"},{"why":"Early work on automating code review suggestions, used to position the paper's tool-centered interaction focus.","marker":"[10]"},{"why":"Survey of hallucination in LLMs, cited as the reason the study treats LLMs as augmenting rather than replacing reviewers.","marker":"[11]"},{"why":"Supplies the thematic-analysis method used to analyze both interview phases.","marker":"[12]"},{"why":"Provides the data-saturation principle that justifies stopping Phase 1 at seven interviews.","marker":"[15]"}],"fun_headline_variants":["Best LLM review mode? Depends on the PR's newness","AI-led code review preferred for big, unfamiliar PRs","LLM review: proactive for new, on-demand for known","Context matters in LLM code review: AI-led wins for unknown"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison between the two interaction modes assumes that the two selected pull requests are similar in size and complexity, yet the paper provides no quantitative measures of lines changed, files touched, or requirement complexity, and the pilot used only two internal developers.","fun_headline_variants_meta":{"raw":{"variants":["Best LLM review mode? Depends on the PR's newness","AI-led code review preferred for big, unfamiliar PRs","LLM review: proactive for new, on-demand for known","Context matters in LLM code review: AI-led wins for unknown"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1397,"prompt_tokens":881,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":497,"tokens_out":516,"duration_ms":4497,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:02:37.974246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment that assigns the same developers to review multiple pull requests with measured size, complexity, and clarity (for example, comparable lines changed, files touched, and cyclomatic complexity), and that records both stated preference and actual review performance such as defects found and time spent, could determine whether the observed mode preference is driven by the interaction mode or by differences in the pull requests themselves.","supporting_citations":[{"cited_title":"Towards automating code review activities,","cited_arxiv_id":null,"evidence_quote":"Early work on automating code review suggestions, used to position the paper's tool-centered interaction focus."},{"cited_title":"Using thematic analysis in psychology,","cited_arxiv_id":null,"evidence_quote":"Supplies the thematic-analysis method used to analyze both interview phases."},{"cited_title":"How many interviews are enough?: An experiment with data saturation and variability,","cited_arxiv_id":null,"evidence_quote":"Provides the data-saturation principle that justifies stopping Phase 1 at seven interviews."}],"review_version":1}