REVIEW 4 major objections 7 minor
How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs
T0 review · 4 major / 7 minor · reviewed 2026-07-30 · grok-4.5
Pith's one-line read Open-weight LLM code tours help debug unfamiliar codebases only when detail, structure, tone, and trust are right—and LLM judges of those tours are unreliable.
desk verdict Solid empirical DX study on open-weight LLM code tours with real 2025 bugs; the themes are useful and the judge-failure findings land, but every frequency rests on reconstructed notes rather than audio. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
A generation-and-judgment pipeline that authors 26 code tours from real 2025 Java stack traces with three local open-weight models, has the two non-author models produce 52 Likert-plus-rationale annotations on Transparency, Scrutability, and Efficiency, then elicits think-aloud experiences from 26 developers that three coders reduce to 62 first-person experience labels clustered around understanding, acting, navigating, liking, and trusting.
What would settle it
Re-run the same tours and annotations with audio-recorded, independently transcribed sessions (or a second independent interviewer) and check whether the same 62 experience labels, especially the trust and preference clusters, still reach high inter-coder agreement and the same frequency rankings.
Extended reading notes
Core claim
Properties of open-weight LLM-authored code-tour components systematically shape developer experience when debugging unfamiliar Java codebases: developers generally prefer detail scaled to segment length, non-restating scannable structure, and a guiding tone; some preferences (such as imperative mood) conflict; stack traces are often insufficient for step selection; perceived human versus AI authorship shifts trust toward misuse or disuse; and LLM judges of tour quality exhibit pervasive sycophancy, confabulation, and incoherence.
Load-bearing premise
That interviewer notes taken on the fly, later translated from French and rewritten as first-person excerpts, faithfully capture what participants experienced rather than what the interviewer reconstructed.
Editorial extensions
If this is right
- Fine-tuning open-weight models on the shared preferences (scaled detail, scannable structure, guiding tone, no pure restatement) should raise baseline tour usefulness.
- Personalization is required for mutually exclusive tastes such as imperative mood and deliberately incomplete descriptions.
- Step selection must reach beyond the raw stack trace to constructors, concrete implementations, and other missing frames while staying under roughly five steps.
- Trust calibration needs explicit disclosure of AI authorship and gradual exposure so users neither over-trust human-sounding text nor under-trust capable AI text.
- LLM-as-judge pipelines for code tours cannot be used as quality gates until sycophancy, confabulation, and cross-criterion incoherence are reduced.
Reading between the lines
- The same preference map (scannable structure, task-scoped suggestions, calibrated detail) likely transfers to other forms of in-IDE onboarding documentation, not only debugging tours.
- If stack-trace-only selection systematically omits constructors and implementations, hybrid static-analysis plus LLM step pickers become a natural next experiment.
- The observed misuse/disuse split suggests that any production onboarding tool should surface provenance and AI-literacy cues rather than hide them.
- Because judges praised duplicate lambda frames that humans rejected, automated evaluators may need an explicit structural checklist before free-form rationale generation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies how properties of code-tour components authored and judged by open-weight LLMs shape developer experience when debugging unfamiliar Java codebases. The authors mine non-flaky 2025 GitHub bugs via Gitbug-Actions, generate 26 stack-trace-based tours with three local SE models (Qwen2.5-Coder 14B, DeepSeek-Coder-V2, Devstral Small 1.1), obtain 52 author≠judge annotations on Transparency/Scrutability/Efficiency, and run a BIBD user study with 26 developers who think aloud while rating tours and annotations. Three coders produce 62 first-person experience labels (α≥0.8) clustered around understanding, acting, navigating, liking, and trusting. Main findings: developers prefer detail scaled to segment length, non-restating scannable structure, and a guiding tone (with some mutually exclusive preferences); stack traces miss steps developers want; perceived human vs AI authorship shifts trust toward misuse or disuse; and LLM judges show sycophancy, confabulation, and incoherence.
Significance. The work fills a clear gap: prior code-tour work has not jointly examined fully AI-generated tours with open-weight models, developer experience, trust calibration, a debugging focus, and LLM-as-a-judge in one pipeline (Table 1). Strengths include use of post-cutoff 2025 reproducible bugs (mitigating leakage), stratified sampling, BIBD assignment, author≠judge separation, multi-coder coding with reported Krippendorff α, an explicit threats section (§5.2), and a public replication package. The trust-calibration and LLM-judge unreliability results are timely for SE tooling. If the qualitative substrate is accepted as adequate, the preference and step-selection findings give concrete, actionable directions for fine-tuning, personalization, and tour design.
major comments (4)
- [§3.7, §5.2.1, Table 4, Figures 6 and 8] §3.7 and §5.2.1: All 62 experience labels (Table 4) and the edge weights in Figures 6 and 8 rest on interviewer on-the-fly notes with no audio, then French→English translation and reformulation into first-person Lejeune-style experiences. Krippendorff α≥0.8 after iterative discussion measures coder agreement on these reconstructions, not fidelity to participants’ utterances. Concurrent think-aloud plus interviewer presence is also flagged (P19). This is load-bearing for the central claim that tour-component properties systematically shape goals. The manuscript should either (a) strengthen the chain of evidence (e.g., second-pass member checks, dual note-takers on a subset, or limited audio where consent allows) or (b) systematically bound every frequency claim as “reported in interviewer reconstructions,” and avoid language that treats n’s as direct counts of participant utterances witho
- [§4.5.2, abstract, Conclusion] §4.5.2 and abstract: The claim that sycophancy, confabulation, and incoherence are “pervasive” in LLM judges is supported by participant reports (e.g., E23 n=13, E27 n=9, E26 n=3) and vivid examples, but there is no systematic audit of the 52 annotations (error rates per criterion/model, inter-judge agreement, rate of self-contradiction, factual checks against tour content). A compact quantitative table over the full annotation set would make the judge-unreliability result falsifiable and proportionate to how strongly it is stated in the abstract and conclusion.
- [§3.4] §3.4: Generation and judging use temperature=1.0 and top-p=1.0 “to explore freely.” For a study that treats specific tour texts and judge rationales as the objects of human evaluation, this maximizes sample variance and weakens reproducibility of the exact artifacts participants saw. The paper should justify this choice against lower-temperature or multi-sample settings, report whether tours/annotations were regenerated or fixed once, and discuss sensitivity: would the same preference and judge-failure themes be expected under more deterministic decoding?
- [Abstract, §4 synthesis, §5.2.3, Conclusion] §3.6 and §5.2.3: The study is scoped to Java, mostly junior developers (13 junior / 9 medior / 4 senior), a web interface rather than an IDE with full codebase access, and three mid-size 2025 open-weight models. That scope is acceptable for an exploratory qualitative study, but several result statements (e.g., “developers generally preferred…”) read as broader than the sample. Tighten claim scope in the abstract, §4 synthesis box, and conclusion to “in this Java debugging onboarding setting with predominantly junior participants,” and separate design implications that are sample-bound from those offered as general SE guidance.
minor comments (7)
- [Figure 2] Figure 2 / Graphical Abstract: several labels render with missing characters (e.g., “bug-xes”, “Ofine”), which hurts readability of an otherwise helpful pipeline figure.
- [Table 4, E17] Table 4 E17 name text says “I can locate the fault when it is not highlighted” while valence is Negative; this appears inconsistent with E16/E15 and with the prose in §4.2.1 (nine could not identify the fault). Please correct the label wording.
- [§2.1, Table 1] §2 and related work: Kara et al. is cited as 2026 in places and Lacy interchangeably; normalize naming and year for the reader.
- [Figure 2] Criteria spelling: “Scrutatibility” appears in Figure 2 caption while “Scrutability” is used elsewhere; standardize.
- [§3.2, §3.3] §3.2–3.3: The path from 110 bugs → 243 traces → 26 stratified samples is clear, but a one-row summary table (projects, error types, tour lengths) would help readers assess diversity without mining Table 5.
- [§3.5.2] Rokeman [34] is cited for the 7-point scale; the more common reference is typically Rokeman/Likert scale methodology literature—double-check the bibliographic key and spelling (Rokeman vs common variants).
- [Throughout] Minor prose/typos: “mediors” (Table 3), “prematurely” misspelled as “prematuraly” (§4.1.5), “strack trace” in Figure 3 caption, “forgetAllTotalMethodsPath” spacing in Figure 9.
Circularity Check
No significant circularity: inductive qualitative themes from a user study, not definitional or fitted predictions.
full rationale
This is an empirical software-engineering user study. The load-bearing chain is: mine 2025 reproducible Java bugs → generate code tours with open-weight LLMs from stack traces → have other LLMs annotate tours on Transparency/Scrutability/Efficiency → 26 developers think aloud while exploring tours and rating annotations → three coders produce 62 first-person experience labels (α≥0.8) clustered around understand/act/navigate/like/trust. Those labels and frequencies (Table 4, Figs. 6/8) are inductive reports of participant reactions, not quantities derived from parameters fitted to the same outcomes. The three evaluation criteria are adapted from Balfroid et al. [4] and used as discussion prompts and LLM-judge axes; they do not define or force the qualitative preferences (detail scaling, scannability, guiding tone, authorship-trust effects, judge sycophancy). Self-positioning in Table 1 and Background against the authors’ prior code-tour papers is normal related-work framing, not a uniqueness theorem or ansatz that closes the result. There is no fitted-input-called-prediction, no self-definitional X⇔Y, and no renaming of a known law as a first-principles derivation. Methodological threats (reconstructed notes, no audio) affect construct fidelity, not circularity of the claim chain. Score 0 is the proportionate finding.
Assumptions & free parameters
free parameters (3)
- Temperature=1.0 and top-p=1.0 for generation and judging =
temperature 1.0, top-p 1.0
- BIBD block size r=6 annotations per developer =
r=6, k=3, λ=1, v=13 per round
- Stratified sample of 26 stack traces from 243 =
26 tours / 52 annotations
assumptions (6)
- domain assumption Think-aloud concurrent verbalization plus interviewer note-taking yields usable evidence of developer experience after translation and first-person reformulation.
- domain assumption Stack-trace frames are a reasonable primary skeleton for debugging-focused code-tour steps.
- domain assumption Transparency, Scrutability, and Efficiency (adapted from Balfroid et al. 2024) are adequate discussion anchors for tour quality without subcriteria.
- standard math Krippendorff's α≥0.8 after discussion indicates stable shared meaning of experience labels.
- domain assumption Open-weight models with pre-2025 cutoffs on 2025 bugs sufficiently mitigate training-data leakage for this study.
- standard math Balanced incomplete block design equalizes annotation load and pairwise overlap enough for fair qualitative coverage.
invented entities (2)
-
62 first-person 'Experience' labels (E1–E62) decomposed into goal/valence/component/property
-
Author–judge multi-LLM code-tour pipeline with three fixed open-weight SE models
independent evidence
Cite this review
Pith. "Pith review of How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs." pith.science (2026). https://pith.science/paper/K7PBZEV2
@misc{pith2026260726987,
author = {Pith},
title = {Pith review of: How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/K7PBZEV2}},
note = {Machine review of arXiv:2607.26987}
}
read the original abstract
Code tours are interactive, onboarding documentation to guide developers through a codebase. Large Language Models (LLMs) can automatically synthesize code tours. Prior work on code tour generation has not studied developer experience or trust calibration when debugging unfamiliar codebases with code tours generated and evaluated by open-weight LLMs. This study surveys how the properties of components in open-weight LLM-authored code tours influence developers' experiences when debugging unfamiliar codebases. We built a pipeline that generated and evaluated code tours from real reproducible bugs. 26 developers with varying backgrounds participated in a user study. In total, 26 code tours were authored from real Java bugs mined from 2025 GitHub commits, with each tour independently judged by two different LLMs, resulting in 52 evaluated configurations. Participants thought aloud as they explored each tour. Three authors qualitatively coded the interviews to identify recurring themes. Developers generally preferred tours that scaled detail with the code length, avoided merely restating code, were easily scannable, and adopted a guiding tone. However, some preferences were mutually exclusive, such as the use of imperative mood. Stack traces were often insufficient to identify all steps developers found relevant. Developers also trusted descriptions they perceived as human-written more than those they believed were AI-generated. Finally, LLM-generated annotations of tour quality were unreliable: sycophancy, confabulation, and incoherence were pervasive. This work lays a basis for future research on fine-tuning open-weight models for code tour generation, personalizing generation to accommodate diverging preferences, selecting relevant steps beyond stack traces, calibrating users' trust to avoid both disuse and misuse, and improving open-weight LLMs' ability to be more trustworthy evaluators
Figures
Figures from the paper (6 more)
Reviewed July 30, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.