{"id":"4348fa6d-6855-474b-8b08-ace8c8600c53","arxiv_id":"2607.26987","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Developers prefer scannable, length-scaled, guiding LLM code tours for unfamiliar Java bugs, distrust AI-looking text, and find open-weight LLM judges unreliable due to sycophancy and confabulation.","lead":"A 26-developer think-aloud study finds that open-weight LLM code tours help debugging only when detail scales with code length, text is scannable, and tone guides rather than restates code—while LLM judges of those tours are often sycophantic and incoherent. It matters for anyone shipping AI onboarding docs or trusting LLM-as-judge in software engineering.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Inter-coder α≥0.8 validates agreement on reconstructed excerpts, not fidelity of the note→translation→first-person chain that supplies every experience label.","rationale":"The reader correctly located the load-bearing soft spot in the reconstruction pipeline (§3.7/§5.2.1). The stress-test only sharpens the mechanism: post-hoc coder agreement cannot audit upstream note-taking and translation, which are the sole empirical base for the property→experience claim and for the trust and LLM-as-judge themes. Other threats (sample, language, model vintage, session fatigue, stack-trace-only step selection) mainly bound external validity and are already priced into CONDITIONAL. Judge sycophancy/confabulation quotes are somewhat more checkable against the written annotations themselves, but still pass through the same notes. No circular math, shipped artifacts help, and disclosed threats do not invert the main themes—so no move to REJECT or ACCEPT. Verdict stays CONDITIONAL; confidence in that call remains high.","tokens_in":36055,"tokens_out":627,"duration_ms":57165,"concrete_test":"If original French notes are retained, have two bilingual coders blind to published English excerpts and E-labels independently map a stratified sample covering the ≥10 highest-frequency labels (E34, E51, E48, E37, E58, E57, E4, E23, E27, E24) onto the published scheme; report per-label recall vs the paper’s coding. If recall for any n≥10 label falls below 0.70, reconstruction has materially reshaped that theme’s evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that tour-component properties systematically shape developer goals (understand/act/navigate/like/trust)—rests on 62 first-person labels (Table 4) counted into frequencies and Figures 6/8. Those labels were built from interviewer on-the-fly notes with no audio, French→English translation, and reformulation into Lejeune-style experiences (§3.7; threat disclosed in §5.2.1). Three coders reached α≥0.8 only after iterative discussion on this already-processed text. High α therefore measures consistency of reading the interviewer’s reconstructions, not correspondence to participants’ utterances. Systematic note-taking filters (what was written, how it was phrased, what the concurrent think-aloud and interviewer presence primed—cf. P19) can inflate themes the interviewer was attuned to (detail, scannability, AI-sounding tone, judge sycophancy) and thin others. Every n in Table 4 and every edge weight is a count of these reconstructions; the property→experience mapping is only as secure as that unvalidated upstream step. Generalizability limits (Java, juniors, three 2025 models) matter less for the internal claim than this substrate problem.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This paper studies how properties of code-tour components authored and judged by open-weight LLMs shape developer experience when debugging unfamiliar Java codebases. The authors mine non-flaky 2025 GitHub bugs via Gitbug-Actions, generate 26 stack-trace-based tours with three local SE models (Qwen2.5-Coder 14B, DeepSeek-Coder-V2, Devstral Small 1.1), obtain 52 author≠judge annotations on Transparency/Scrutability/Efficiency, and run a BIBD user study with 26 developers who think aloud while rating tours and annotations. Three coders produce 62 first-person experience labels (α≥0.8) clustered around understanding, acting, navigating, liking, and trusting. Main findings: developers prefer detail scaled to segment length, non-restating scannable structure, and a guiding tone (with some mutually exclusive preferences); stack traces miss steps developers want; perceived human vs AI authorship shifts trust toward misuse or disuse; and LLM judges show sycophancy, confabulation, and incoherence.","tokens_in":36406,"tokens_out":1702,"duration_ms":45107,"significance":"The work fills a clear gap: prior code-tour work has not jointly examined fully AI-generated tours with open-weight models, developer experience, trust calibration, a debugging focus, and LLM-as-a-judge in one pipeline (Table 1). Strengths include use of post-cutoff 2025 reproducible bugs (mitigating leakage), stratified sampling, BIBD assignment, author≠judge separation, multi-coder coding with reported Krippendorff α, an explicit threats section (§5.2), and a public replication package. The trust-calibration and LLM-judge unreliability results are timely for SE tooling. If the qualitative substrate is accepted as adequate, the preference and step-selection findings give concrete, actionable directions for fine-tuning, personalization, and tour design.","major_comments":[{"comment":"§3.7 and §5.2.1: All 62 experience labels (Table 4) and the edge weights in Figures 6 and 8 rest on interviewer on-the-fly notes with no audio, then French→English translation and reformulation into first-person Lejeune-style experiences. Krippendorff α≥0.8 after iterative discussion measures coder agreement on these reconstructions, not fidelity to participants’ utterances. Concurrent think-aloud plus interviewer presence is also flagged (P19). This is load-bearing for the central claim that tour-component properties systematically shape goals. The manuscript should either (a) strengthen the chain of evidence (e.g., second-pass member checks, dual note-takers on a subset, or limited audio where consent allows) or (b) systematically bound every frequency claim as “reported in interviewer reconstructions,” and avoid language that treats n’s as direct counts of participant utterances witho","section":"§3.7, §5.2.1, Table 4, Figures 6 and 8"},{"comment":"§4.5.2 and abstract: The claim that sycophancy, confabulation, and incoherence are “pervasive” in LLM judges is supported by participant reports (e.g., E23 n=13, E27 n=9, E26 n=3) and vivid examples, but there is no systematic audit of the 52 annotations (error rates per criterion/model, inter-judge agreement, rate of self-contradiction, factual checks against tour content). A compact quantitative table over the full annotation set would make the judge-unreliability result falsifiable and proportionate to how strongly it is stated in the abstract and conclusion.","section":"§4.5.2, abstract, Conclusion"},{"comment":"§3.4: Generation and judging use temperature=1.0 and top-p=1.0 “to explore freely.” For a study that treats specific tour texts and judge rationales as the objects of human evaluation, this maximizes sample variance and weakens reproducibility of the exact artifacts participants saw. The paper should justify this choice against lower-temperature or multi-sample settings, report whether tours/annotations were regenerated or fixed once, and discuss sensitivity: would the same preference and judge-failure themes be expected under more deterministic decoding?","section":"§3.4"},{"comment":"§3.6 and §5.2.3: The study is scoped to Java, mostly junior developers (13 junior / 9 medior / 4 senior), a web interface rather than an IDE with full codebase access, and three mid-size 2025 open-weight models. That scope is acceptable for an exploratory qualitative study, but several result statements (e.g., “developers generally preferred…”) read as broader than the sample. Tighten claim scope in the abstract, §4 synthesis box, and conclusion to “in this Java debugging onboarding setting with predominantly junior participants,” and separate design implications that are sample-bound from those offered as general SE guidance.","section":"Abstract, §4 synthesis, §5.2.3, Conclusion"}],"minor_comments":[{"comment":"Figure 2 / Graphical Abstract: several labels render with missing characters (e.g., “bug-xes”, “Ofine”), which hurts readability of an otherwise helpful pipeline figure.","section":"Figure 2"},{"comment":"Table 4 E17 name text says “I can locate the fault when it is not highlighted” while valence is Negative; this appears inconsistent with E16/E15 and with the prose in §4.2.1 (nine could not identify the fault). Please correct the label wording.","section":"Table 4, E17"},{"comment":"§2 and related work: Kara et al. is cited as 2026 in places and Lacy interchangeably; normalize naming and year for the reader.","section":"§2.1, Table 1"},{"comment":"Criteria spelling: “Scrutatibility” appears in Figure 2 caption while “Scrutability” is used elsewhere; standardize.","section":"Figure 2"},{"comment":"§3.2–3.3: The path from 110 bugs → 243 traces → 26 stratified samples is clear, but a one-row summary table (projects, error types, tour lengths) would help readers assess diversity without mining Table 5.","section":"§3.2, §3.3"},{"comment":"Rokeman [34] is cited for the 7-point scale; the more common reference is typically Rokeman/Likert scale methodology literature—double-check the bibliographic key and spelling (Rokeman vs common variants).","section":"§3.5.2"},{"comment":"Minor prose/typos: “mediors” (Table 3), “prematurely” misspelled as “prematuraly” (§4.1.5), “strack trace” in Figure 3 caption, “forgetAllTotalMethodsPath” spacing in Figure 9.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"Fit for a software engineering / human factors venue is good. Novelty is real relative to prior code-tour papers, but the contribution is primarily empirical-qualitative; acceptance should hinge on fixing the evidence-chain presentation and quantifying judge failures rather than on expanding theory. The note-taking limitation is disclosed in good faith—authors are not hiding it—but Results currently over-speak relative to that disclosure. No integrity red flags; replication package is a plus."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is straightforward: this is the first pipeline that jointly runs fully AI-authored debugging code tours on recent non-leaky Java bugs, open-weight local models, systematic experience coding, trust calibration, and LLM-as-judge. Table 1 is honest about the gap, and the design matches the claim type—Gitbug-Actions 2025 commits, stratified traces, BIBD assignment, author ≠ judge, three coders to α ≥ 0.8 after discussion, replication package on Zenodo.\n\nWhat is actually new is the qualitative map itself. Developers want detail scaled to segment length, scannable structure, guiding tone, and no mere restatement; stack traces miss constructors and concrete implementations; perceived human vs AI authorship pushes trust toward misuse or disuse; and the open-weight judges show sycophancy, confabulation, and cross-criterion incoherence. Those last points are concrete and useful for anyone building or evaluating AI onboarding docs. The threats section is unusually transparent.\n\nThe soft spot that matters is upstream of the α. There is no audio. Excerpts are on-the-fly notes, French-to-English, then reformulated into first-person Lejeune-style experiences. High agreement measures consistency of reading the interviewer’s reconstructions, not fidelity to what participants said. The authors flag this; it does not invent the themes, but every edge weight in Figures 6/8 and every n in Table 4 is a count of that chain. Generalizability (Java, juniors, three 2025-era models, temp=1) is secondary and already disclosed. No circular math; criteria are discussion prompts, not self-proving scores.\n\nThis is for people working on developer onboarding, code summarization, trust in AI docs, or LLM-as-judge reliability in SE. Not theory-reorganizing, but actionable and carefully run. I would send it to referees. Engage if that is your area; the artifacts and the judge-failure evidence are worth the time.","headline":"Solid empirical DX study on open-weight LLM code tours with real 2025 bugs; the themes are useful and the judge-failure findings land, but every frequency rests on reconstructed notes rather than audio.","tokens_in":37059,"tokens_out":522,"would_cite":true,"duration_ms":20445,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Open-weight LLM code tours help debug unfamiliar codebases only when detail, structure, tone, and trust are right—and LLM judges of those tours are unreliable.","keywords":["code tours","onboarding","open-weight LLMs","developer experience","trust calibration","LLM-as-a-judge","debugging","code summarization"],"falsifier":"Re-run the same tours and annotations with audio-recorded, independently transcribed sessions (or a second independent interviewer) and check whether the same 62 experience labels, especially the trust and preference clusters, still reach high inter-coder agreement and the same frequency rankings.","tokens_in":36939,"feed_emoji":"🧭","tokens_out":1032,"duration_ms":24483,"temperature":0.7,"pith_summary":"This paper asks how the properties of code-tour components authored and judged by local open-weight LLMs shape what developers experience when they try to fix real bugs in unfamiliar Java codebases. The authors built a pipeline that turns 2025 GitHub bug-fix commits into stack-trace-driven tours, has one model write each tour and two others score it, then sat 26 developers down to think aloud while rating the tours and the machine judgments. What emerges is a concrete preference map: detail should scale with segment length and not merely restate the code; text should be scannable and guiding rather than lazy or theatrical; stack traces alone miss steps people want (constructors, concrete implementations); and people trust prose they read as human-written while discounting prose they read as AI-written. The same study shows that the LLM judges themselves are poor critics—sycophantic, confabulatory, and incoherent across criteria—so automated quality labels cannot yet be trusted. The result is a practical agenda for fine-tuning, personalization, better step selection, trust calibration, and more honest machine evaluators.","feed_headline":"LLM code tours help only when detail, tone, and trust align","feed_subtitle":"Developers want scannable guiding tours; stack traces miss steps; AI judges praise too freely","key_machinery":"A generation-and-judgment pipeline that authors 26 code tours from real 2025 Java stack traces with three local open-weight models, has the two non-author models produce 52 Likert-plus-rationale annotations on Transparency, Scrutability, and Efficiency, then elicits think-aloud experiences from 26 developers that three coders reduce to 62 first-person experience labels clustered around understanding, acting, navigating, liking, and trusting.","core_discovery":"Properties of open-weight LLM-authored code-tour components systematically shape developer experience when debugging unfamiliar Java codebases: developers generally prefer detail scaled to segment length, non-restating scannable structure, and a guiding tone; some preferences (such as imperative mood) conflict; stack traces are often insufficient for step selection; perceived human versus AI authorship shifts trust toward misuse or disuse; and LLM judges of tour quality exhibit pervasive sycophancy, confabulation, and incoherence.","pith_inferences":["The same preference map (scannable structure, task-scoped suggestions, calibrated detail) likely transfers to other forms of in-IDE onboarding documentation, not only debugging tours.","If stack-trace-only selection systematically omits constructors and implementations, hybrid static-analysis plus LLM step pickers become a natural next experiment.","The observed misuse/disuse split suggests that any production onboarding tool should surface provenance and AI-literacy cues rather than hide them.","Because judges praised duplicate lambda frames that humans rejected, automated evaluators may need an explicit structural checklist before free-form rationale generation."],"forward_implications":["Fine-tuning open-weight models on the shared preferences (scaled detail, scannable structure, guiding tone, no pure restatement) should raise baseline tour usefulness.","Personalization is required for mutually exclusive tastes such as imperative mood and deliberately incomplete descriptions.","Step selection must reach beyond the raw stack trace to constructors, concrete implementations, and other missing frames while staying under roughly five steps.","Trust calibration needs explicit disclosure of AI authorship and gradual exposure so users neither over-trust human-sounding text nor under-trust capable AI text.","LLM-as-judge pipelines for code tours cannot be used as quality gates until sycophancy, confabulation, and cross-criterion incoherence are reduced."],"fun_headline_variants":["LLM code tours aid debugging only when detail tone and trust align","Devs prefer scannable guiding tours; AI judges overpraise freely","Stack traces miss steps; perceived AI authorship shifts trust","Open-weight LLM tours shape debug experience via detail and tone","Tour preferences conflict; LLM quality scores show sycophancy"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"That interviewer notes taken on the fly, later translated from French and rewritten as first-person excerpts, faithfully capture what participants experienced rather than what the interviewer reconstructed.","fun_headline_variants_meta":{"raw":{"variants":["LLM code tours aid debugging only when detail tone and trust align","Devs prefer scannable guiding tours; AI judges overpraise freely","Stack traces miss steps; perceived AI authorship shifts trust","Open-weight LLM tours shape debug experience via detail and tone","Tour preferences conflict; LLM quality scores show sycophancy"]},"model":"grok-4.5","effort":"low","cost_usd":0.002594,"raw_usage":{"total_tokens":1068,"prompt_tokens":899,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":25944000,"prompt_tokens_details":{"text_tokens":899,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":100,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":899,"tokens_out":69,"duration_ms":3288,"temperature":1.0,"reasoning_tokens":100,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T14:51:08.561469+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same tours and annotations with audio-recorded, independently transcribed sessions (or a second independent interviewer) and check whether the same 62 experience labels, especially the trust and preference clusters, still reach high inter-coder agreement and the same frequency rankings.","supporting_citations":[],"review_version":1}