{"id":"8899fc55-72f6-4f86-9e28-f5e005cefae5","arxiv_id":"2412.03815","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM plus a repository knowledge graph answers software repository questions with 84% accuracy when few-shot chain-of-thought prompting is added, outperforming an intent-based bot and web-search GPT-4o.","lead":"This paper combines a knowledge graph of Git repository data with GPT-4o to answer natural-language questions about commits, issues, and files. On a curated set of repository questions, accuracy rose from 65% to 84% after adding few-shot chain-of-thought prompting, and a 20-person study found users were faster and more accurate with the tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy is not an unbiased estimate: the same 20 questions were used for LLM selection, prompt development, and the reported 84%, and §5.4/Table 15 conflates this with a 150-question baseline evaluation.","rationale":"I read this as a legitimate empirical engineering study with real assets: a published replication package, a detailed failure analysis, and a user study that, despite its limitations, shows practical promise. The load-bearing weakness is not that the system is obviously worse than claimed, but that the paper's evaluation design cannot rule out two natural explanations for the 84% number: overfitting to the 20-question set through model selection and prompt iteration, and a correctness standard that depends on the authors' interpretations. The disclosed 84%-to-94% sensitivity partially cuts against the reader's ground-truth concern as a standalone objection, because it shows the strict reading is conservative; however, it also demonstrates how much the score depends on evaluator judgment. The decisive issue is the absence of any held-out set, which is compounded by the unreconciled 20-versus-150 question reporting. Because these threats are addressable with a standard held-out evaluation and independent annotation, I would keep the reader's CONDITIONAL verdict rather than moving to REJECT; the paper should not be accepted as-is until the quantitative claim is validated on data not used for design choices.","tokens_in":33116,"tokens_out":9139,"duration_ms":95407,"concrete_test":"Use the shared replication package to split the 150 templates into a development set (used for model selection and CoT prompt design) and a held-out test set of at least 50 templates. Freeze the GPT-4o configuration and the final few-shot CoT prompt, then have two independent annotators, blind to the authors' oracle queries, score each final answer as correct or incorrect using a pre-specified rubric. Report held-out accuracy with inter-annotator agreement and a 95% confidence interval, and compare the approach against MSRBot and GPT-4o-search on the same held-out questions. If held-out accuracy falls below 84% or no longer exceeds the baselines, the headline is an artifact of the current evaluation design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the KG+LLM+CoT pipeline answers repository questions at 84% accuracy and beats MSRBot. For that claim to be credible, accuracy must be measured on a set that was not used to make design choices, and correctness must not rest solely on the authors' interpretation. Neither condition holds. RQ0 selects GPT-4o using the same 20 templates used in RQ1/RQ3, and the few-shot CoT prompt was developed after inspecting failures on those same questions; there is no held-out split. The reporting is internally inconsistent in a way that masks this: §4.2 defines the evaluation set as 20 templates (100 Q-A pairs), yet §5.4/Table 15 reports the approach's accuracy over 750 questions (150 per project), and the abstract attributes 84% to 150 questions. Correctness is also scored manually against author-written oracle Cypher, with the paper itself reporting that accepting alternative reasonable interpretations moves accuracy from 84% to 94%. The 84% figure is therefore not a stable property of the system; it is a function of test-set choices and scoring judgments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a repository question-answering pipeline that combines a knowledge graph of Git metadata (users, commits, issues, files) with an LLM that translates natural-language questions into Cypher queries, executes them against a Neo4j graph, and generates natural-language answers. The authors evaluate on five GitHub projects using 20 question templates derived from the MSRBot dataset, report a baseline accuracy of 65%, identify LLM reasoning errors through manual analysis, add few-shot chain-of-thought prompting and report 84% accuracy, compare against MSRBot (70%) and GPT-4o-search-preview (19%), and supplement this with a 20-participant task-based user study. The paper also provides a failure taxonomy, an ad-hoc example outside the test set, and a replication package.","tokens_in":33358,"tokens_out":4961,"duration_ms":51004,"significance":"If the accuracy claims held, this would be a useful empirical contribution: it is among the first studies to apply a knowledge-graph-plus-LLM architecture to repository-metadata question answering, it provides a detailed error taxonomy, it includes a user study, and it releases a replication package. However, the central quantitative claims are not currently supported by the evaluation design. The same 20 question templates were used for LLM selection, for prompt development, and for the reported RQ1/RQ3 results; the relationship between the 20-question evaluation and the 150-question set referenced in the abstract and in Table 15 is unclear; and correctness is judged against author-written Cypher queries whose interpretation changes the headline result from 84% to 94%. These are load-bearing issues, so the paper needs a major revision with a held-out evaluation and a transparent, reproducible correctness protocol.","major_comments":[{"comment":"The evaluation is contaminated by test-set leakage: RQ0 selects GPT-4o by execution accuracy on the same 20 templates (Table 4) that are later used to report the 65% and 84% results (Tables 5–6 and 13–14), and the few-shot chain-of-thought prompt was designed after inspecting failures on those same questions. Because no held-out split exists, the headline accuracy and the comparison with MSRBot are estimates of in-sample fit, not of generalization. Please re-run the evaluation on a disjoint held-out set, or otherwise account for the model-selection and prompt-development steps in the reported numbers.","section":"§5.1 and §4.2"},{"comment":"The manuscript is internally inconsistent about the evaluation size. Section 4.2 defines the evaluation set as 20 templates (100 question-project pairs), and RQ1/RQ3 report results over 100 executed question-project pairs. Section 5.4 and Table 15 then report 750 questions (150 per project) for the comparison with MSRBot and GPT-4o-search-preview, and the abstract attributes the 84% figure to 150 questions. Please clarify which set produced each number; if the full 150-question set was used for the chain-of-thought evaluation, report its per-project and per-difficulty results separately. As written, the abstract's '150 questions' claim does not match the main RQ3 evaluation, and the baseline comparison is not directly comparable if the two methods were evaluated on different sets.","section":"§5.4, Table 15, and §4.2"},{"comment":"The correctness measure is subjective in a way that directly affects the headline result. The ground truth consists of Cypher queries written by the first author and reviewed by the co-authors, and Section 6 reports that accepting alternative reasonable interpretations of ambiguous questions raises accuracy from 84% to 94%. The 84% figure is therefore not a stable property of the system but a function of the authors' query semantics. Please pre-specify the intended semantics for each question template, use multiple independent annotators to judge whether a generated answer matches the ground truth, report inter-annotator agreement, and present both the strict and the lenient scores.","section":"§4.2 and §6"}],"minor_comments":[{"comment":"The abstract and conclusion say the approach was evaluated on '150 questions,' while RQ1 and RQ3 report results on 20 question templates; align the abstract with the actual evaluation described in Sections 4.2 and 5.2–5.4.","section":"Abstract and §8"},{"comment":"The RQ2 summary and the conclusion state that faulty reasoning accounts for 80.5% of errors, but Table 7 reports 75.0% for incorrect relationship modeling and lists other reasoning-related categories separately; please explain how the 80.5% figure is computed.","section":"§5.3 and §8"},{"comment":"The discussion claims that if the LLM generated the same Cypher query for each question, accuracy would be constant across repositories; this ignores differences in repository data and in which questions are answered, so please qualify the claim.","section":"§6"},{"comment":"Figure 6 would be easier to interpret if the cumulative number of questions and the majority-vote threshold (3/5) were marked directly on the plot.","section":"Figure 6"},{"comment":"Section 4.4 explains that a RAG baseline was excluded because it performed poorly, and GPT-4o-search-preview is introduced as a replacement; please add one sentence explaining how web search is expected to serve the same evaluation purpose as RAG.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of an empirical software engineering venue, and the approach plus failure taxonomy are worth publishing if the evaluation can be made sound. The evaluation-leakage issue is serious: the same 20 questions drive model selection, prompt development, and the headline accuracy, so the 84% figure cannot be taken at face value. I would not reject outright because the problem is fixable by re-running on a held-out set and by making the ground-truth protocol explicit and independently verifiable, but I would require those changes before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: the 84% accuracy figure is not a stable property of this system. The same 20 question templates were used to choose the LLM (RQ0), to inspect failures and design the few-shot CoT prompt (RQ2/RQ3), and to report the 84% (RQ1/RQ3). There is no held-out split. The paper even admits that accepting 'alternative reasonable interpretations' moves accuracy to 94%—which tells you how much the scoring depends on the authors' own ground-truth Cypher.\n\nWhat the paper does well: the pipeline is clearly described (KG construction from Git metadata, LLM-to-Cypher, execution, response generation), and the error taxonomy from 164 failed executions is the most valuable part—incorrect relationship modeling dominates, followed by arithmetic, attribute filtering, and date formatting. That taxonomy is reusable for anyone building similar systems. The user study, while small (20 participants), gives a plausible signal that the approach saves time and improves task completion. The baseline comparison on the full 150-question set (Table 15) is a good instinct, even though it inherits the same oracle ground truth. They also share data and scripts on Zenodo, which is good practice.\n\nSoft spots, in proportion:\n1. Evaluation leakage is real and load-bearing. Model selection, prompt tuning, and the headline number come from the same 20 questions. This is textbook overfitting to the test set, and the threats-to-validity section does not acknowledge it.\n2. The ground truth is the first author's hand-written Cypher, reviewed by the team. The 84%-to-94% ambiguity analysis shows the metric is sensitive to interpretive choices. That makes the head-to-head with MSRBot (0.70 vs 0.84 on different sets) harder to interpret.\n3. The abstract's \"150 questions / 84%\" is misleading: 84% comes from a 20-template subset (100 Q-A pairs); the 150-question run gives 0.82. Numbers need to be reported per evaluation set.\n\nThis is a reasonable engineering paper with a useful failure analysis, not a solid empirical claim yet. It deserves a serious referee, but needs major revision: a held-out test set, blinded or rubric-based correctness scoring, and transparent reporting of which set produced which number. I wouldn't cite the 84% as evidence, but I'd send it out and ask for the evaluation to be rebuilt.","headline":"Useful pipeline and error taxonomy, but the 84% headline is not trustworthy as reported because the same 20 questions were used for model selection, prompt tuning, and the final evaluation.","tokens_in":33870,"tokens_out":3752,"would_cite":false,"duration_ms":37307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge-graph-grounded LLM with chain-of-thought prompting answered repository questions correctly 84% of the time in this study's evaluation.","keywords":["knowledge graph question answering","software repository mining","LLM query generation","Cypher query language","chain-of-thought prompting","software engineering chatbots","empirical evaluation"],"falsifier":"Have an independent team write ground-truth graph queries for the same 20 questions without seeing the LLM outputs, then re-run the five-per-project evaluation; the claim is confirmed only if the few-shot chain-of-thought pipeline again beats the 65% no-CoT baseline and the intent-based baseline.","tokens_in":32921,"feed_emoji":"🤖","tokens_out":8833,"duration_ms":79059,"temperature":0.7,"pith_summary":"This paper tries to establish that a chatbot grounded in a knowledge graph can answer questions about a software repository's history more reliably than either intent-based chatbots or an LLM with web search. The proposed pipeline models Git metadata—users, commits, issues, and files—as a graph, uses an LLM to translate each natural-language question into a graph query, executes the query, and has a second LLM call turn the results into an answer. In an evaluation over 150 questions on five popular open-source projects, the pipeline answered 65% correctly on its own and 84% when the query generator was prompted with few-shot chain-of-thought examples, with the biggest gains on multi-hop questions. A 20-participant user study reported more tasks completed correctly and in roughly half the time compared with the participants' usual tools. If these results hold, repository data become accessible to non-specialists without writing Git commands or reading logs.","feed_headline":"Knowledge graphs lift repository Q&A chatbots to 84% accuracy","feed_subtitle":"Grounding LLMs in a repository knowledge graph plus chain-of-thought prompting beats intent-based bots and web search","key_machinery":"The load-bearing object is a repository knowledge graph whose schema has four node types—User, Commit, Issue, and File—connected by relationships such as author, fixed, introduced, impacted, and changed, with evolving file attributes stored on edges. The LLM-based query generator receives this schema plus the question and must emit Cypher, a declarative query language for property graphs, with randomness reduced to zero; a regular-expression extractor isolates the Cypher from stray text, the query runs against a graph database, and the response generator, another LLM call, produces the answer from the query results. The decisive mechanism is the few-shot chain-of-thought prompt, which embeds two worked examples that model how to list possible interpretations, choose the most probable one, identify relevant nodes and relationships, and assemble the query step by step; this is what lifts accuracy from 65% to 84%.","core_discovery":"The central claim is that the bottleneck in LLM-based repository question answering is not retrieving facts but reasoning over the graph structure, and that few-shot chain-of-thought prompting directly addresses that bottleneck. The paper reports that overall accuracy rises from 65% to 84% when two worked reasoning examples are added to the query-generation prompt; for level-3 questions requiring two or more graph relationships, accuracy rises from 50% to 90%. Against baselines on the full 150-question set, the approach scored 82% versus 70% for an intent-based chatbot and 19% for an LLM with web search. In a user study with 20 participants, 84% of tasks were completed correctly with the chatbot versus 36% manually, and median time per session fell from about 20.7 minutes to 10.3 minutes. The paper also documents that 75% of failures trace to incorrect relationship modeling, supporting the reasoning-centric diagnosis.","pith_inferences":["If ambiguous questions were scored as correct whenever the LLM's interpretation is defensible, the paper's own re-analysis puts accuracy near 94%; a benchmark that pre-registers interpretation rules would make the headline number less dependent on the authors' judgment.","The same schema-plus-few-shot-reasoning recipe is a natural template for other structured domains—enterprise data, scientific metadata, or CI/CD logs—where a graph schema and a query language already exist and the open question is how reliably an LLM can translate natural language into queries.","The paper's error taxonomy suggests a testable extension: adding a verification pass that executes a generated query, checks the result for empty or anomalous output, and re-prompts the LLM could recover a share of the remaining 16% errors without changing the schema.","Because the user study compared chatbot use against whatever methods participants chose, part of the time saving may reflect tool familiarity; a controlled comparison against a chatbot without knowledge-graph grounding would isolate the contribution of the graph itself."],"forward_implications":["A query generator with few-shot chain-of-thought can answer questions requiring two or more relationships at 90% accuracy, up from 50% without it, so the hardest part of repository Q&A is reasoning rather than retrieval.","The approach beats an intent-based chatbot (70%) and an LLM with web search (19%) on the same 150 questions, suggesting structured grounding outperforms both fixed intents and unstructured retrieval for repository metadata questions.","Non-specialists can get correct answers to repository questions without command-line skill: in the user study, 84% of chatbot-aided answers were correct versus 36% manually, in about half the median time.","Because the schema is identical across projects, the 80–90% per-project accuracy range suggests the method transfers across repositories once the graph is built.","The documented failure modes—incorrect relationship modeling, faulty arithmetic, misapplied filters and dates, and hallucination—provide a concrete checklist for improving LLM-to-graph query generation."],"supporting_citations":[{"why":"Supplies the 150-question evaluation set, the ten intents, and the MSRBot baseline that the approach is measured against.","marker":"[2]"},{"why":"Documents that retrieval-augmented LLM chatbots fail on repository questions, which motivates grounding the LLM in a knowledge graph.","marker":"[4]"},{"why":"Provides the chain-of-thought prompting technique that, in few-shot form, raises accuracy from 65% to 84%.","marker":"[69]"},{"why":"Defines the execution-accuracy metric and the zero-shot protocol used to compare LLMs for query generation.","marker":"[43]"},{"why":"Defines Cypher, the graph query language that the query generator must produce and the executor runs.","marker":"[23]"},{"why":"Supplies the R-SZZ variant used to identify bug-introducing commits when constructing the knowledge graph.","marker":"[18]"},{"why":"Provides the original SZZ algorithm underlying bug-introducing-change identification in the graph construction.","marker":"[62]"}],"fun_headline_variants":["Reasoning is the real bottleneck in LLM repo Q&A","KG-grounded LLM hits 84% with few-shot reasoning prompts","Repo Q&A: chain-of-thought beats web search and intent bots","Why LLM repo bots fail: reasoning, not retrieval","84% accuracy, 2x faster: KG+LLM for repository questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy depends on the authors' hand-written graph queries being the uniquely correct interpretation of each question, and on the authors' manual judgment of which answers count as correct.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning is the real bottleneck in LLM repo Q&A","KG-grounded LLM hits 84% with few-shot reasoning prompts","Repo Q&A: chain-of-thought beats web search and intent bots","Why LLM repo bots fail: reasoning, not retrieval","84% accuracy, 2x faster: KG+LLM for repository questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1964,"prompt_tokens":964,"completion_tokens":1000,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":580,"tokens_out":1000,"duration_ms":9569,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:04:05.575848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent team write ground-truth graph queries for the same 20 questions without seeing the LLM outputs, then re-run the five-per-project evaluation; the claim is confirmed only if the few-shot chain-of-thought pipeline again beats the 65% no-CoT baseline and the intent-based baseline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 150-question evaluation set, the ten intents, and the MSRBot baseline that the approach is measured against."},{"cited_title":"Le, and Denny Zhou","cited_arxiv_id":null,"evidence_quote":"Provides the chain-of-thought prompting technique that, in few-shot form, raises accuracy from 65% to 84%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the execution-accuracy metric and the zero-shot protocol used to compare LLMs for query generation."}],"review_version":1}