Pith. sign in

REVIEW 3 major objections 5 minor 83 references

Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A knowledge-graph-grounded LLM with chain-of-thought prompting answered repository questions correctly 84% of the time in this study's evaluation.

desk verdict Useful pipeline and error taxonomy, but the 84% headline is not trustworthy as reported because the same 20 questions were used for model selection, prompt tuning, and the final evaluation. read the letter →

arxiv 2412.03815 v2 pith:TGZP5MLO submitted 2024-12-05 cs.SE cs.AIcs.CLcs.LG

Samuel Abedu , SayedHassan Khatoonabadi , Emad Shihab This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords knowledgegraphquestionansweringsoftwarerepositoryminingLLMquerygenerationCypherlanguagechain-of-thoughtpromptingengineeringchatbotsempiricalevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a chatbot grounded in a knowledge graph can answer questions about a software repository's history more reliably than either intent-based chatbots or an LLM with web search. The proposed pipeline models Git metadata—users, commits, issues, and files—as a graph, uses an LLM to translate each natural-language question into a graph query, executes the query, and has a second LLM call turn the results into an answer. In an evaluation over 150 questions on five popular open-source projects, the pipeline answered 65% correctly on its own and 84% when the query generator was prompted with few-shot chain-of-thought examples, with the biggest gains on multi-hop questions. A 20-participant user study reported more tasks completed correctly and in roughly half the time compared with the participants' usual tools. If these results hold, repository data become accessible to non-specialists without writing Git commands or reading logs.

What carries the argument

The load-bearing object is a repository knowledge graph whose schema has four node types—User, Commit, Issue, and File—connected by relationships such as author, fixed, introduced, impacted, and changed, with evolving file attributes stored on edges. The LLM-based query generator receives this schema plus the question and must emit Cypher, a declarative query language for property graphs, with randomness reduced to zero; a regular-expression extractor isolates the Cypher from stray text, the query runs against a graph database, and the response generator, another LLM call, produces the answer from the query results. The decisive mechanism is the few-shot chain-of-thought prompt, which embeds two worked examples that model how to list possible interpretations, choose the most probable one, identify relevant nodes and relationships, and assemble the query step by step; this is what lifts accuracy from 65% to 84%.

What would settle it

Have an independent team write ground-truth graph queries for the same 20 questions without seeing the LLM outputs, then re-run the five-per-project evaluation; the claim is confirmed only if the few-shot chain-of-thought pipeline again beats the 65% no-CoT baseline and the intent-based baseline.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in LLM-based repository question answering is not retrieving facts but reasoning over the graph structure, and that few-shot chain-of-thought prompting directly addresses that bottleneck. The paper reports that overall accuracy rises from 65% to 84% when two worked reasoning examples are added to the query-generation prompt; for level-3 questions requiring two or more graph relationships, accuracy rises from 50% to 90%. Against baselines on the full 150-question set, the approach scored 82% versus 70% for an intent-based chatbot and 19% for an LLM with web search. In a user study with 20 participants, 84% of tasks were completed correctly with the chatbot versus 36% manually, and median time per session fell from about 20.7 minutes to 10.3 minutes. The paper also documents that 75% of failures trace to incorrect relationship modeling, supporting the reasoning-centric diagnosis.

Load-bearing premise

The headline accuracy depends on the authors' hand-written graph queries being the uniquely correct interpretation of each question, and on the authors' manual judgment of which answers count as correct.

Editorial extensions

If this is right

  • A query generator with few-shot chain-of-thought can answer questions requiring two or more relationships at 90% accuracy, up from 50% without it, so the hardest part of repository Q&A is reasoning rather than retrieval.
  • The approach beats an intent-based chatbot (70%) and an LLM with web search (19%) on the same 150 questions, suggesting structured grounding outperforms both fixed intents and unstructured retrieval for repository metadata questions.
  • Non-specialists can get correct answers to repository questions without command-line skill: in the user study, 84% of chatbot-aided answers were correct versus 36% manually, in about half the median time.
  • Because the schema is identical across projects, the 80–90% per-project accuracy range suggests the method transfers across repositories once the graph is built.
  • The documented failure modes—incorrect relationship modeling, faulty arithmetic, misapplied filters and dates, and hallucination—provide a concrete checklist for improving LLM-to-graph query generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ambiguous questions were scored as correct whenever the LLM's interpretation is defensible, the paper's own re-analysis puts accuracy near 94%; a benchmark that pre-registers interpretation rules would make the headline number less dependent on the authors' judgment.
  • The same schema-plus-few-shot-reasoning recipe is a natural template for other structured domains—enterprise data, scientific metadata, or CI/CD logs—where a graph schema and a query language already exist and the open question is how reliably an LLM can translate natural language into queries.
  • The paper's error taxonomy suggests a testable extension: adding a verification pass that executes a generated query, checks the result for empty or anomalous output, and re-prompts the LLM could recover a share of the remaining 16% errors without changing the schema.
  • Because the user study compared chatbot use against whatever methods participants chose, part of the time saving may reflect tool familiarity; a controlled comparison against a chatbot without knowledge-graph grounding would isolate the contribution of the graph itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes a repository question-answering pipeline that combines a knowledge graph of Git metadata (users, commits, issues, files) with an LLM that translates natural-language questions into Cypher queries, executes them against a Neo4j graph, and generates natural-language answers. The authors evaluate on five GitHub projects using 20 question templates derived from the MSRBot dataset, report a baseline accuracy of 65%, identify LLM reasoning errors through manual analysis, add few-shot chain-of-thought prompting and report 84% accuracy, compare against MSRBot (70%) and GPT-4o-search-preview (19%), and supplement this with a 20-participant task-based user study. The paper also provides a failure taxonomy, an ad-hoc example outside the test set, and a replication package.

Significance. If the accuracy claims held, this would be a useful empirical contribution: it is among the first studies to apply a knowledge-graph-plus-LLM architecture to repository-metadata question answering, it provides a detailed error taxonomy, it includes a user study, and it releases a replication package. However, the central quantitative claims are not currently supported by the evaluation design. The same 20 question templates were used for LLM selection, for prompt development, and for the reported RQ1/RQ3 results; the relationship between the 20-question evaluation and the 150-question set referenced in the abstract and in Table 15 is unclear; and correctness is judged against author-written Cypher queries whose interpretation changes the headline result from 84% to 94%. These are load-bearing issues, so the paper needs a major revision with a held-out evaluation and a transparent, reproducible correctness protocol.

major comments (3)
  1. [§5.1 and §4.2] The evaluation is contaminated by test-set leakage: RQ0 selects GPT-4o by execution accuracy on the same 20 templates (Table 4) that are later used to report the 65% and 84% results (Tables 5–6 and 13–14), and the few-shot chain-of-thought prompt was designed after inspecting failures on those same questions. Because no held-out split exists, the headline accuracy and the comparison with MSRBot are estimates of in-sample fit, not of generalization. Please re-run the evaluation on a disjoint held-out set, or otherwise account for the model-selection and prompt-development steps in the reported numbers.
  2. [§5.4, Table 15, and §4.2] The manuscript is internally inconsistent about the evaluation size. Section 4.2 defines the evaluation set as 20 templates (100 question-project pairs), and RQ1/RQ3 report results over 100 executed question-project pairs. Section 5.4 and Table 15 then report 750 questions (150 per project) for the comparison with MSRBot and GPT-4o-search-preview, and the abstract attributes the 84% figure to 150 questions. Please clarify which set produced each number; if the full 150-question set was used for the chain-of-thought evaluation, report its per-project and per-difficulty results separately. As written, the abstract's '150 questions' claim does not match the main RQ3 evaluation, and the baseline comparison is not directly comparable if the two methods were evaluated on different sets.
  3. [§4.2 and §6] The correctness measure is subjective in a way that directly affects the headline result. The ground truth consists of Cypher queries written by the first author and reviewed by the co-authors, and Section 6 reports that accepting alternative reasonable interpretations of ambiguous questions raises accuracy from 84% to 94%. The 84% figure is therefore not a stable property of the system but a function of the authors' query semantics. Please pre-specify the intended semantics for each question template, use multiple independent annotators to judge whether a generated answer matches the ground truth, report inter-annotator agreement, and present both the strict and the lenient scores.
minor comments (5)
  1. [Abstract and §8] The abstract and conclusion say the approach was evaluated on '150 questions,' while RQ1 and RQ3 report results on 20 question templates; align the abstract with the actual evaluation described in Sections 4.2 and 5.2–5.4.
  2. [§5.3 and §8] The RQ2 summary and the conclusion state that faulty reasoning accounts for 80.5% of errors, but Table 7 reports 75.0% for incorrect relationship modeling and lists other reasoning-related categories separately; please explain how the 80.5% figure is computed.
  3. [§6] The discussion claims that if the LLM generated the same Cypher query for each question, accuracy would be constant across repositories; this ignores differences in repository data and in which questions are answered, so please qualify the claim.
  4. [Figure 6] Figure 6 would be easier to interpret if the cumulative number of questions and the majority-vote threshold (3/5) were marked directly on the plot.
  5. [§4.4] Section 4.4 explains that a RAG baseline was excluded because it performed poorly, and GPT-4o-search-preview is introduced as a replacement; please add one sentence explaining how web search is expected to serve the same evaluation purpose as RAG.

Circularity Check

2 steps flagged · score 5.0 of 10

Headline 84% is measured on the same 20 questions used to select the model and develop the CoT prompt, and correctness is scored against the authors' own hand-written oracle; the claim is partly circular, though the 150-question baseline comparison retains independent content.

  1. fitted input called prediction [Section 5.1 (RQ0, model selection) and Section 5.4 (RQ3, few-shot CoT evaluation); Tables 4 and 13]
    "We evaluate the models on the 20 questions described in Section 4.2 using the prompt template shown in Figure 3 to generate the Cypher query. [...] The selection of the GPT-4o model is based on its performance in our exploratory question (discussed in Section 5.1). [...] To evaluate the gains, we use the same set of 20 questions described in Section 4 and used in RQ1, executing the experiments five times for each question to account for the stochastic nature of the LLM's generation."

    GPT-4o is chosen because it scored highest (EX 0.65) on the same 20-question set that later defines the RQ1/RQ3 accuracy, and the few-shot chain-of-thought configuration was introduced after failure analysis on those same questions. The reported 84% is therefore the score of a configuration selected and refined on the very test set from which it is reported, so the headline accuracy is not an unbiased estimate for unseen questions. Table 15's full 150-question run reports 0.82 (not 0.84), confirming that the abstract's 84% is the leaked-set number rather than a stable system property.

  2. self definitional [Section 4.2 (ground-truth construction) and Section 6 (ambiguity discussion); Appendix E]
    "To establish the ground truth for our evaluation, the first author manually wrote Cypher queries corresponding to all 20 questions for each of the selected repositories. To ensure the correctness of these queries and eliminate potential bias, the authors collaboratively reviewed and discussed the logic employed in each query. [...] In the evaluation in our RQ3, we considered these scenarios as incorrectly answered questions. However, if we had considered such scenarios in our evaluation as correct, the accuracy in RQ3 increased from an average of 84% to 94%."

    The oracle answers are authored by the first author and reviewed by the same team, so 'correct' is defined by the authors' interpretation of each question rather than by an independent standard. The paper itself quantifies the consequence: reclassifying 'reasonable interpretations' of ambiguous questions as correct moves the headline from 84% to 94%. The measured accuracy is thus a function of the scoring convention adopted by the authors, and the 84% figure cannot be treated as an intrinsic, interpretation-independent property of the approach.

full rationale

The paper makes no first-principles derivation; it is an empirical system evaluation, so circularity arises only in how the headline number is produced and scored. Two intertwined issues make the headline 84% partially circular. First, RQ0 selects GPT-4o on the same 20-question set used for RQ1/RQ3, and the CoT prompting decision was made after inspecting failures on that same set, so the 84% is measured on the set used to make the design choices (test-set leakage). Second, the oracle answers are hand-written and author-reviewed, and Section 6 shows that accepting alternative reasonable interpretations raises the same result to 94%, so the 'correctness' label is an author-defined convention rather than an objective ground truth. These issues do not void the paper: Table 15 reports a full 150-question-per-project comparison (0.82 vs MSRBot's 0.70 and GPT-4o-search's 0.19), and the user study is a separate, externally anchored evaluation; those parts retain independent content. The paper also contains a reporting inconsistency: Section 4.2 defines the evaluation set as 20 templates, while Section 5.4/Table 15 and Section 7.3 refer to 150 questions per project, and the abstract conflates the two by attributing 84% to the 150-question evaluation. The self-citation to Abedu et al. [4] for RAG's failure justifies omitting a RAG baseline but is not load-bearing for the central comparison against MSRBot and GPT-4o-search, so it does not raise the score further. Overall: partial circularity in the headline accuracy, but the approach has independent empirical content beyond the leaked set.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical systems paper, so the ledger lists design choices and domain assumptions rather than mathematical axioms. The main burden is that the evaluation pipeline is built and judged by the authors: model selection, CoT prompt content, ground-truth queries, and correctness judgment all come from the same team.

free parameters (3)
  • LLM choice = GPT-4o
    Selected because it achieved the best execution accuracy (0.65) on the same 20 evaluation questions used in the main study (Section 5.1); this is a hyperparameter chosen on the evaluation set.
  • Few-shot CoT examples = 2 hand-crafted examples
    The two chain-of-thought examples in Figure 5 were written by the authors to demonstrate reasoning that matches the ground-truth query logic; this prompt content is a tuned parameter of the system.
  • Majority-vote threshold = 3 out of 5
    A question is counted correct if at least 3 of 5 runs produce the correct answer, a lenient threshold that raises reported accuracy; the 5/5 consistency is reported as 76.4% in Section 5.4.
assumptions (4)
  • domain assumption R-SZZ correctly identifies bug-introducing commits
    The knowledge graph and ground-truth answers for bug-related questions assume the R-SZZ bug-introducing commit detection (Davies et al. 2014) is reliable; errors propagate into the KG and the oracle answers. See Sections 3.1 and 7.2.
  • domain assumption The 150 questions from Abdellatif et al. cover the space of repository-related questions
    The evaluation set is taken from a single prior study; the paper does not sample or validate the breadth of real user questions. See Section 4.2.
  • domain assumption Git commit logs reference issue IDs for fixing commits
    The KG constructor identifies fixing commits by searching commit logs for bug IDs; missing or inconsistent references would create incomplete knowledge graphs. See Section 7.2.
  • ad hoc to paper Manual judgment of LLM responses is a valid correctness measure
    No code or rubric for judging answer correctness is provided, and the paper shows that correctness varies from 84% to 94% depending on whether ambiguous interpretations are accepted (Section 6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering." pith.science (2026). https://pith.science/paper/TGZP5MLO

@misc{pith2026241203815,
  author       = {Pith},
  title        = {Pith review of: Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TGZP5MLO}},
  note         = {Machine review of arXiv:2412.03815}
}
read the original abstract

Software repositories contain valuable information for understanding the development process. However, extracting insights from repository data is time-consuming and requires technical expertise. While software engineering chatbots support natural language interactions with repositories, chatbots struggle to understand questions beyond their trained intents and to accurately retrieve the relevant data. This study aims to improve the accuracy of LLM-based chatbots in answering repository-related questions by augmenting them with knowledge graphs. We use a two-step approach: constructing a knowledge graph from repository data, and synergizing the knowledge graph with an LLM to handle natural language questions and answers. We curated 150 questions of varying complexity and evaluated the approach on five popular open-source projects. Our initial results revealed the limitations of the approach, with most errors due to the reasoning ability of the LLM. We therefore applied few-shot chain-of-thought prompting, which improved accuracy to 84%. We also compared against baselines (MSRBot and GPT-4o-search-preview), and our approach performed significantly better. In a task-based user study with 20 participants, users completed more tasks correctly and in less time with our approach, and they reported that it was useful. Our findings demonstrate that LLMs and knowledge graphs are a viable solution for making repository data accessible.

Figures

Figures reproduced from arXiv: 2412.03815 by the authors.

Figure 1
Figure 1. Overview of our approach in answering software repository-related questions by synergizing LLMs [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Overview of the schema of the knowledge graph used in this study. The circles represent the entities [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Prompt template used by the Query Generator LLM. The prompt includes the current date and time, [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Prompt template used by the Response Generator LLM. The prompt includes the schema of the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for the few-shot chain-of-thought. The prompt includes the current date and time, [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Distribution of correct responses across multiple runs for each project. The blue bar represents when [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Boxplot of task completion times: participants consistently completed tasks faster with the chatbot [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Demonstration of the approach answering a question not in the evaluation set. To protect privacy, we [PITH_FULL_IMAGE:figures/full_fig_p040_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 25 canonical work pages

  1. [1]

    Ahmad Abdellatif, Khaled Badran, Diego Elias Costa, and Emad Shihab. 2022. A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering.IEEE Transactions on Software Engineering48, 8 (Aug. 2022), 3087–3102. https://doi.org/10.1109/TSE.2021.3078384

    arXiv 2022

  2. [2]

    Ahmad Abdellatif, Khaled Badran, and Emad Shihab. 2020. MSRBot: Using Bots to Answer Questions from Software Repositories.Empirical Software Engineering25, 3 (May 2020), 1834–1863. https://doi.org/10.1007/s10664-019-09788-5

    work page doi:10.1007/s10664-019-09788-5 2020

  3. [3]

    Ahmad Abdellatif, Diego Costa, Khaled Badran, Rabe Abdalkareem, and Emad Shihab. 2020. Challenges in Chatbot Development: A Study of Stack Overflow Posts. InProceedings of the 17th International Conference on Mining Software Repositories (MSR ’20). Association for Computing Machinery, New York, NY, USA, 174–185. https://doi.org/10.1145/ 3379597.3387472

    arXiv 2020

  4. [4]

    Samuel Abedu, Ahmad Abdellatif, and Emad Shihab. 2024. LLM-Based Chatbots for Mining Software Repositories: Challenges and Opportunities. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE ’24). Association for Computing Machinery, New York, NY, USA, 201–210. https: //doi.org/10.1145/3661167.3661218

    arXiv 2024

  5. [5]

    Samuel Abedu, SayedHassan Khatoonabadi, and Emad Shihab. 2024. Sabedu/Knowledge_graph_llm_synergy: Replica- tion Package Release. Zenodo. https://doi.org/10.5281/zenodo.14271490

    work page doi:10.5281/zenodo.14271490 2024

  6. [6]

    Hasan Abu-Rasheed, Christian Weber, and Madjid Fathi. 2024. Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations. https://doi.org/10.48550/arXiv.2403.03008 arXiv:2403.03008

    arXiv doi:10.48550/arxiv.2403.03008 2024

  7. [7]

    Eleni Adamopoulou and Lefteris Moussiades. 2020. An Overview of Chatbot Technology. InArtificial Intelligence Applications and Innovations, Ilias Maglogiannis, Lazaros Iliadis, and Elias Pimenidis (Eds.). Springer International Publishing, Cham, 373–383. https://doi.org/10.1007/978-3-030-49186-4_31

    doi:10.1007/978-3-030-49186-4_31 2020

  8. [8]

    Anthropic. 2024. Introducing Claude 3.5 Sonnet \ Anthropic. https://www.anthropic.com/news/claude-3-5-sonnet

    2024

Show all 83 references
  1. [9]

    Sean Banerjee and Bojan Cukic. 2015. On the Cost of Mining Very Large Open Source Repositories. In2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering. IEEE Press, 37–43. https://doi.org/10.1109/BIGDSE.2015.16

    2015 doi

  2. [10]

    Andrew Begel and Thomas Zimmermann. 2014. Analyze This! 145 Questions for Data Scientists in Software Engineering. InProceedings of the 36th International Conference on Software Engineering (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 12–23. https://doi...

    2014

  3. [11]

    Hudson Borges and Marco Tulio Valente. 2018. What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform.Journal of Systems and Software146 (Dec. 2018), 112–129. https://doi.org/10.1016/j.jss. 2018.09.016

    2018 doi

  4. [12]

    Bradley, Thomas Fritz, and Reid Holmes

    Nick C. Bradley, Thomas Fritz, and Reid Holmes. 2018. Context-Aware Conversational Developer Assistants. InPro- ceedings of the 40th International Conference on Software Engineering (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 993–1003. https://doi.org/1...

    2018

  5. [13]

    Gleison Brito, Thais Mombach, and Marco Tulio Valente. 2019. Migrating to GraphQL: A Practical Assessment. In2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE Computer Society, 140–150. https://doi.org/10.1109/SANER.2019.8667986

    2019

  6. [14]

    Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, and Dongsheng Li. 2024. Improving Large Language Models in Event Relation Logical Prediction. InProceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), Lun-Wei Ku...

    2024 doi

  7. [15]

    Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A Review: Knowledge Reasoning over Knowledge Graph.Expert Systems with Applications141 (March 2020), 112948. https://doi.org/10.1016/j.eswa.2019.112948

    2020

  8. [16]

    Lan Cheng, Emerson Murphy-Hill, Mark Canning, Ciera Jaspan, Collin Green, Andrea Knight, Nan Zhang, and Elizabeth Kammer. 2022. What Improves Developer Productivity at Google? Code Quality. InProceedings of the 30th ACM Joint European Software Engineering Conference and Sympos...

    2022

  9. [17]

    Daniel Alencar da Costa, Shane McIntosh, Weiyi Shang, Uirá Kulesza, Roberta Coelho, and Ahmed E. Hassan. 2017. A Framework for Evaluating the Results of the SZZ Approach for Identifying Bug-Introducing Changes.IEEE Transactions on Software Engineering43, 7 (July 2017), 641–657...

    2017

  10. [18]

    Steven Davies, Marc Roper, and Murray Wood. 2014. Comparing Text-Based and Dependence-Based Approaches for Determining the Origins of Bugs.Journal of Software: Evolution and Process26, 1 (2014), 107–139. https: //doi.org/10.1002/smr.1619 , Vol. 1, No. 1, Article . Publication ...

    2014 doi

  11. [19]

    Malinda Dilhara, Ameya Ketkar, and Danny Dig. 2021. Understanding Software-2.0: A Study of Machine Learning Library Usage and Evolution.ACM Transactions on Software Engineering and Methodology30, 4 (July 2021), 1–42. https://doi.org/10.1145/3453478

    2021 doi

  12. [20]

    James Dominic, Jada Houser, Igor Steinmacher, Charles Ritter, and Paige Rodeghero. 2020. Conversational Bot for Newcomers Onboarding to Open Source Projects. InProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW’20). Association f...

    2020

  13. [21]

    EbookFoundation. 2024. EbookFoundation/Free-Programming-Books. Free Ebook Foundation

    2024

  14. [22]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson

  15. [23]

    Nadime Francis, Alastair Green, Paolo Guagliardo, Leonid Libkin, Tobias Lindaaker, Victor Marsault, Stefan Plantikow, Mats Rydberg, Petra Selmer, and Andrés Taylor. 2018. Cypher: An Evolving Query Language for Property Graphs. In Proceedings of the 2018 International Conferenc...

    2018

  16. [24]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao

  17. [25]

    GitHub. 2024. GitHub GraphQL API Documentation. https://docs.github.com/en/graphql

    2024

  18. [26]

    InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(New York, NY, USA, 2024-02-06) (ICSE ’24)

    Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(New York, NY, USA, 2024-02-06) (ICSE ’24). Association for Computing Machinery, 1–13. ht...

    2024

  19. [27]

    GitHub. 2024. Public Schema. https://docs.github.com/en/graphql/overview/public-schema

    2024

  20. [28]

    GitHub. 2024. GitHub REST API Documentation. https://docs.github.com/en/rest

    2024

  21. [29]

    Hideaki Hata, Nicole Novielli, Sebastian Baltes, Raula Gaikovina Kula, and Christoph Treude. 2021. GitHub Discussions: An Exploratory Study of Early Adoption.Empirical Software Engineering27, 1 (Oct. 2021), 3. https://doi.org/10.1007/ s10664-021-10058-6

    2021

  22. [30]

    Ahmed E. Hassan. 2008. The Road Ahead for Mining Software Repositories. In2008 Frontiers of Software Maintenance. IEEE, 48–57. https://doi.org/10.1109/FOSM.2008.4659248

    2008

  23. [31]

    Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann

    Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia D’amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequed...

    2021 doi

  24. [32]

    Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. 2024. GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements. https://doi.org/10.48550/arXiv.2402.10963 arXiv:2402.10963

    2024 doi arXiv

  25. [33]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang

  26. [34]

    Aidan Hogan, Xin Luna Dong, Denny Vrandečić, and Gerhard Weikum. 2025. Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users’ Questions. https://doi.org/10.48550/arXiv.2501.06699 arXiv:2501.06699 [cs]

    2025 doi arXiv

  27. [35]

    Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2022. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications.IEEE Transactions on Neural Networks and Learning Systems33, 2 (Feb. 2022), 494–514. https://doi.org/10.1109/TNNLS.2021.3070843

    2022

  28. [36]

    33, 8 (2024), 220:1–220:79

    Large Language Models for Software Engineering: A Systematic Literature Review. 33, 8 (2024), 220:1–220:79. https://doi.org/10.1145/3695988

    2024 doi

  29. [37]

    izkeros. 2022. How Can I Calculate the Number of Lines Changed since Last Commit in Git?

    2022

  30. [38]

    Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice. 1 (2024), 81:1819–81:1840. Issue FSE. https://doi.org/10.1145/3660788

    2024 doi

  31. [39]

    Aditya Kalyanpur, Kailash Karthik Saravanakumar, Victor Barres, Jennifer Chu-Carroll, David Melville, and David Ferrucci. 2024. LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic. https://doi.org/10.48550/arXiv.2406. 17663 arXiv:2406.17663

    2024 doi arXiv

  32. [40]

    SayedHassan Khatoonabadi, Ahmad Abdellatif, Diego Elias Costa, and Emad Shihab. 2024. Predicting the First Response Latency of Maintainers and Contributors in Pull Requests.IEEE Transactions on Software Engineering50, 10 (Oct. 2024), 2529–2543. https://doi.org/10.1109/TSE.2024.3443741

    2024

  33. [41]

    Jasmine Latendresse, Samuel Abedu, Ahmad Abdellatif, and Emad Shihab. 2024. An Exploratory Study on Machine Learning Model Management.ACM Trans. Softw. Eng. Methodol.(Aug. 2024). https://doi.org/10.1145/3688841

    2024 doi

  34. [42]

    Ko, Robert DeLine, and Gina Venolia

    Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collocated Software Development Teams. In 29th International Conference on Software Engineering (ICSE’07)(2007-05). 344–353. https://doi.org/10.1109/ICSE.2007.45 ISSN: 1558-1225. , Vol. 1, No. 1, Article . ...

    2007 doi

  35. [43]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models Are Zero-Shot Reasoners.Advances in Neural Information Processing Systems35 (Dec. 2022), 22199–22213

    2022

  36. [44]

    Qianlong Li, Chen Huang, Shuai Li, Yuanxin Xiang, Deng Xiong, and Wenqiang Lei. 2024. GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering. https://doi.org/10.48550/arXiv.2412.01230 arXiv:2412.01230 [cs]

    2024 doi arXiv

  37. [45]

    Ernests Lavrinovics, Russa Biswas, Johannes Bjerva, and Katja Hose. 2025. Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective. 85 (2025), 100844. https://doi.org/10.1016/j.websem.2024.100844

    2025

  38. [46]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Ma Chenhao, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Lar...

    2023

  39. [47]

    Akshat Malik, Bram Adams, and Ahmed Hassan. 2024. Towards Graph-Anonymization of Software Analytics Data: Empirical Study on JIT Defect Prediction.Empirical Software Engineering29, 4 (June 2024), 76. https://doi.org/10.1007/ s10664-024-10464-6

    2024

  40. [48]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to Understand Whole Software Repository? https://doi.org/10.48550/arXiv.2406.01422 arXiv:2406.01422 [cs]

    2024 doi arXiv

  41. [49]

    Yacine Majdoub and Eya Ben Charrada. 2024. Debugging with Open-Source Large Language Models: An Evaluation. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’24). Association for Computing Machinery, New York, ...

    2024

  42. [50]

    Dušan Okanović, Samuel Beck, Lasse Merz, Christoph Zorn, Leonel Merino, André van Hoorn, and Fabian Beck

  43. [51]

    Maxwell, L

    K.D. Maxwell, L. Van Wassenhove, and S. Dutta. 1996. Software Development Productivity of European Space, Military, and Industrial Applications.IEEE Transactions on Software Engineering22, 10 (Oct. 1996), 706–718. https: //doi.org/10.1109/32.544349

    1996 doi

  44. [52]

    Meta. 2024. Meta-Llama/Meta-Llama-3-8B·Hugging Face. https://huggingface.co/meta-llama/Meta-Llama-3-8B

    2024

  45. [53]

    Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying Large Language Models and Knowledge Graphs: A Roadmap.IEEE Transactions on Knowledge and Data Engineering36, 7 (July 2024), 3580–3599. https://doi.org/10.1109/TKDE.2024.3352100

    2024

  46. [54]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. https://doi.org/10.48550/arXiv.2302.06590 arXiv:2302.06590 [cs]

    2023 doi arXiv

  47. [55]

    OpenAI. 2024. Best Practices for Prompt Engineering with the OpenAI API. https://help.openai.com/en/articles/6654000- best-practices-for-prompt-engineering-with-the-openai-api

    2024

  48. [56]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/

    2024

  49. [57]

    Juan Sequeda, Dean Allemang, and Bryon Jacob. 2025. Knowledge Graphs as a source of trust for LLM-powered enterprise question answering. (2025), 100858. https://doi.org/10.1016/j.websem.2024.100858

    2025

  50. [58]

    Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, Nagu Rangan, Tara Safavi, Siddharth Suri, Mengting Wan, Leijie Wang, and Longqi Yang. 2024. Using Large Languag...

    2024 doi arXiv

  51. [59]

    Chandrasekaran

    Kiran Ramesh, Surya Ravishankaran, Abhishek Joshi, and K. Chandrasekaran. 2017. A Survey of Design Techniques for Conversational Agents. InInformation, Communication and Computing Technology, Saroj Kaushik, Daya Gupta, Latika Kharb, and Deepak Chahal (Eds.). Springer, Singapor...

    2017 doi

  52. [60]

    Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015. Are Students Representatives of Professionals in Software Engineering Experiments?. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. 666–676. https://doi.org/10.1109/ICSE.2015.82 I...

    2015 doi

  53. [61]

    Amit Singhal. 2012. Introducing the Knowledge Graph: Things, Not Strings. https://blog.google/products/search/introducing-knowledge-graph-things-not/. , Vol. 1, No. 1, Article . Publication date: October 2025. 34 Abedu et al

    2012

  54. [62]

    Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller. 2005. When Do Changes Induce Fixes?ACM SIGSOFT Software Engineering Notes30, 4 (May 2005), 1–5. https://doi.org/10.1145/1082983.1083147

    2005

  55. [63]

    Vibhu Saujanya Sharma, Rohit Mehra, and Vikrant Kaulgud. 2017. What Do Developers Want? An Advisor Approach for Developer Priorities. In2017 IEEE/ACM 10th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE Press, 78–81. https://doi.or...

    2017 doi

  56. [64]

    Jiho Shin, Clark Tang, Tahmineh Mohati, Maleknaz Nayebi, Song Wang, and Hadi Hemmati. 2025. Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR)(2025-04). 490–502. https...

    2025

  57. [65]

    M. Vidoni. 2022. A Systematic Process for Mining Software Repositories: Results from a Systematic Literature Review. Information and Software Technology144 (April 2022), 106791. https://doi.org/10.1016/j.infsof.2021.106791

    2022

  58. [66]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui

  59. [67]

    2024.AI | 2024 Stack Overflow Developer Survey

    Stackoverflow. 2024.AI | 2024 Stack Overflow Developer Survey. https://survey.stackoverflow.co/2024/ai

    2024

  60. [68]

    Linus Torvalds. 2024. Torvalds/Linux

    2024

  61. [69]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou

  62. [70]

    Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. 2023. How Effective Are Neural Networks for Fixing Security Vulnerabilities. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysi...

    2023

  63. [71]

    Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation Learning of Knowledge Graphs with Entity Descriptions.Proceedings of the AAAI Conference on Artificial Intelligence30, 1 (March 2016). https://doi.org/10.1609/aaai.v30i1.10329

    2016 doi

  64. [72]

    Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. 2023. Making Large Language Models Better Reasoners with Alignment. https://doi.org/10.48550/arXiv.2309.02144 arXiv:2309.02144

    2023 doi arXiv

  65. [73]

    Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. https://doi.org/10.48550/arXiv.2305.07922 arXiv:2305.07922 [cs]

    2023 doi arXiv

  66. [74]

    Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. 2024. Can LLM Graph Reasoning Generalize beyond Pattern Memorization? https://doi.org/10.48550/arXiv.2406.15992 arXiv:2406.15992

    2024 doi arXiv

  67. [75]

    Yanjie Zhao, Haoyu Wang, Lei Ma, Yuxin Liu, Li Li, and John Grundy. 2019. Knowledge Graphing Git Repositories: A Preliminary Study. In2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 599–603. https://doi.org/10.1109/SANER...

    2019

  68. [76]

    Ting Zhou, Yanjie Zhao, Xinyi Hou, Xiaoyu Sun, Kai Chen, and Haoyu Wang. 2024. Bridging Design and Development with Automated Declarative UI Code Generation. https://doi.org/10.48550/arXiv.2409.11667 arXiv:2409.11667

    2024 doi arXiv

  69. [77]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Ya...

    2025 doi arXiv

  70. [78]

    Derong Xu, Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, and Enhong Chen. 2025. Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation. https://doi.org/10.48550/ar...

    2025 doi arXiv

  71. [79]

    Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conferenc...

    2024

  72. [2020]

    InProceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE ’20)

    Can a Chatbot Support Software Engineers with Load Testing? Approach and Experiences. InProceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE ’20). Association for Computing Machinery, New York, NY, USA, 120–129. https://doi.org/10.1145/3358960.3375792

  73. [2022]

    2022), 24824–24837

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.Advances in Neural Information Processing Systems35 (Dec. 2022), 24824–24837

    2022

  74. [2023]

    https://doi.org/10.48550/arXiv.2305.17926 arXiv:2305.17926

    Large Language Models Are Not Fair Evaluators. https://doi.org/10.48550/arXiv.2305.17926 arXiv:2305.17926

    doi arXiv

  75. [2024]

    https://doi.org/10.48550/ arXiv.2404.16130 arXiv:2404.16130 [cs]

    From Local to Global: A Graph RAG Approach to Query-Focused Summarization. https://doi.org/10.48550/ arXiv.2404.16130 arXiv:2404.16130 [cs]

    doi arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.