Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt knowledge gaps in developer-ChatGPT conversations are closely tied to unresolved GitHub issues, and three measurable heuristics could flag them automatically.

desk verdict A genuinely useful annotated dataset and taxonomy, but the headline 44.6% vs. 12.6% claim rests on an unvalidated proxy and the model barely beats the majority-class baseline. read the letter →

arxiv 2501.11709 v3 pith:CVHSYKCP submitted 2025-01-20 cs.SE

Ramtin Ehsani , Sakshi Pathak , Preetha Chatterjee This is my paper · ORCID
classification cs.SE
keywords promptknowledgegapslargelanguagemodelsissueresolutionGitHubissuesdeveloper-ChatGPTconversationsheuristicsMissingContextbrowserextension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that vague or incomplete developer prompts are a major reason LLM-assisted issue resolution fails, and that the gaps can be detected automatically. Analyzing 433 developer-ChatGPT conversations shared inside GitHub issue threads, the authors find that conversations linked to open issues contain prompt knowledge gaps in 44.6% of prompts, versus 12.6% in conversations linked to closed issues. Four gap types carry the analysis: Missing Context, Missing Specification, Multiple Context, and Unclear Instructions, with Missing Context the most frequent in both settings. The practical payoff is a set of three measurable heuristic families, Specificity, Contextual Richness, and Clarity, which the authors embed in a lightweight browser extension that flags weak prompts before they derail a conversation.

What carries the argument

The load-bearing mechanism is a two-layer scheme: a qualitative annotation taxonomy that names the gap, and a computable feature set that measures it. The taxonomy has four gap categories and seven conversation styles, built by refining prior category sets through iterative dual annotation. The feature set is grouped into Specificity, Contextual Richness, and Clarity, and scored by a logistic regression model that predicts whether a conversation belongs to a closed issue. The paper's outcome proxy, open versus closed GitHub issue status, is what connects the annotations and the heuristics to effectiveness.

What would settle it

A decisive check would be an issue-status-blind re-annotation of a random sample of the same 433 conversations: if knowledge-gap rates in closed-issue and open-issue conversations become statistically indistinguishable, or if the difference disappears after stratifying by issue difficulty, the claimed association fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt content quality, not just prompting technique, shapes whether LLM-assisted issue resolution succeeds. On 433 developer-ChatGPT conversations drawn from GitHub issue threads, the authors define four prompt knowledge gaps and seven conversation styles, then compare conversations linked to closed issues against those linked to open issues. They report that ineffective conversations carry knowledge gaps in 44.6% of prompts versus 12.6% in effective ones, with Missing Context the single most frequent gap in both and present in every conversation style. The authors conclude that prompt knowledge gaps are a measurable, actionable impediment to resolution, and they propose three heuristic families whose component metrics associate with issue closure and can be computed automatically.

Load-bearing premise

The paper's entire comparison leans on the assumption that a conversation was effective exactly when its GitHub issue is closed and ineffective when the issue remains open, so unseen external help or arbitrary issue abandonment would muddy the measured link between prompt gaps and outcomes.

Editorial extensions

If this is right

  • Automated prompt-gap detection is feasible without subjective review: the heuristics are computable from text and code and feed directly into a logistic regression model.
  • Prompt templates that pre-request context, prior attempts, error logs, and technical specifications should reduce the most common gap, Missing Context.
  • Since knowledge gaps appear in all seven conversation styles, tools should nudge developers on content rather than on prompting technique.
  • Developers can be guided toward short code snippets plus error messages and references, since large snippet size and missing error logs are associated with open issues.
  • A browser-extension prototype demonstrates a practical deployment path inside normal developer workflows, not just a retrospective analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same three heuristic families could be retrained as a general prompt-quality scorer for other LLM applications, such as documentation Q&A or code review, by swapping the outcome label from issue closure to task success.
  • Editorial inference: a randomized field test, where half the developers receive the extension's gap warnings and half do not, could convert the observed correlation into a direct test of whether flagging Missing Context changes closure rates.
  • Editorial inference: because the paper does not control for issue difficulty, the gap metrics might partly track how hard an issue is to describe; scoring prompts on deliberately easy versus hard issues would clarify this.
  • Editorial inference: the Missing Context finding suggests that a minimal intervention, such as a structured issue template with fields for goal, attempts, code, and expected behavior, could be tested even before building any machine-learning detector.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper analyzes 433 developer-ChatGPT conversations extracted from GitHub issue threads (DevGPT dataset) to investigate how prompt knowledge gaps and conversation styles relate to issue resolution. The authors manually annotate the conversations with four prompt knowledge-gap categories (Missing Context, Missing Specification, Unclear Instructions, Multiple Context) and seven conversation styles, using an iterative coding process with reported Cohen's Kappa values. They define conversation effectiveness by the status of the linked GitHub issue (closed = effective, open = ineffective), and report that 44.6% of prompts in ineffective conversations contain knowledge gaps versus 12.6% in effective ones. They then design three heuristic families (Specificity, Contextual Richness, Clarity) and use logistic regression with SHAP explanations to associate these heuristics with issue closure. Finally, they present a browser-extension prototype that scores prompts against these heuristics. The paper argues that prompt knowledge gaps, especially Missing Context, are a central obstacle to LLM-guided issue resolution and that automated gap detection is feasible.

Significance. The paper addresses a timely and practical problem: understanding why developer-LLM interactions fail during issue resolution. Its main strengths are the manually annotated dataset (433 conversations, with iterative coding and inter-rater reliability checks), the public replication package, and the concrete prototype demonstrating a possible tool direction. If the central association were robust, the finding that knowledge gaps (especially Missing Context) are far more prevalent in conversations linked to open issues would be of practical value to developers and tool builders. However, the current evidence is weakened by the acknowledged open/closed proxy, the absence of validation of the heuristics against the manual gap labels, and the incomplete statistical reporting of the regression analysis. The paper is a solid empirical starting point, but its claims presently outrun the evidence.

major comments (5)
  1. [III-A, VI, IV-A] The outcome proxy is load-bearing: the headline 44.6% vs. 12.6% difference and the RQ2 regression both use GitHub issue status (closed/open) as a direct measure of conversation effectiveness. The assumption stated in Section III-A ('conversations within closed issues likely contributed to successful resolutions, while those within open issues did not') is acknowledged in Section VI but never subjected to sensitivity analysis. Closed issues may be resolved by other means despite an unhelpful conversation, and open issues may be abandoned despite helpful exchanges. In addition, issue difficulty is a plausible confound: difficult issues are more likely to remain open and also more likely to be described incompletely. Please add a robustness analysis (e.g., a subset of conversations explicitly referenced as helpful, or controlling for indicators of issue complexity) and report gap rates at the conversation level, since the current per-prompt percentages ignore the differing numbers of prompts per conversation.
  2. [IV-B, Table IV] The logistic regression evidence for the heuristics is incomplete. The model was selected on the same data ('Among various configurations and parameters tested, our best-performing regression model...'), yet no baseline accuracy is given; the reported 62% CV accuracy is close to the majority-class rate (262/433 ≈ 60.5%). Coefficients in Table IV are presented without confidence intervals or standard errors, and the only significance information comes from an undefined t-test in the text. Please report the majority baseline, coefficient confidence intervals, and a proper holdout or nested cross-validation evaluation.
  3. [III-B, IV-B] The heuristics are partly self-referential. Section III-B states that the heuristics were 'derived using NLP and code-related metrics to directly correspond to the knowledge gaps identified in prompts' (Contextual Richness for Missing/Multiple Context, Specificity for Missing Specification, Clarity for Unclear Instructions), and they are evaluated on the same dataset from which the gap categories were induced. This does not invalidate the descriptive analysis, but it does not constitute validation. Please test whether the heuristic scores actually discriminate the manually annotated gap labels (e.g., correlation or classification against the annotations), or explicitly re-frame RQ2 as a descriptive feasibility analysis rather than a demonstration of association.
  4. [IV-A] The text states 'We found a significant difference in the number of prompts with knowledge gaps: 334 in open issues compared to only 107 in closed ones,' but no statistical test is reported for this comparison. The p-values mentioned in that section refer to conversation styles, not to gap counts. Please provide a formal test (e.g., chi-square or a mixed-effects model with conversation as a random effect) and adjust for the differing number of prompts per conversation.
  5. [V] The feasibility study does not define how the displayed heuristic scores are computed. The text says the scores are 'the mean average of the features included in each heuristic,' but the features in Table II have entirely different scales (e.g., #Code Snippets vs. Flesch Reading Ease), so an unweighted mean is not meaningful unless features are normalized. The claim that the tool 'correctly identifies' the gaps in the open conversation is not accompanied by any evaluation. Please specify the normalization, describe the score formulas, and provide at least a small reliability check against manual assessments.
minor comments (5)
  1. [III-A] The use of the Shapiro-Wilk test to 'confirm' normality is confusing: the variables compared (conversation style counts) are counts, and the text does not say which variable is tested. Please clarify the statistical procedure.
  2. [III-A] The threats-to-validity section reports 'the final average Cohen's Kappa agreement between them was 78%,' without noting that this is the average of the gap (0.84) and style (0.72) Kappas. Please state this explicitly to avoid inconsistency with the earlier numbers.
  3. [Table II] The notation '0<8<289' for min-median-max is easy to misread; consider separate columns for min, median, and max, or a clearer format.
  4. [IV-B, Figure 5] The SHAP figure is difficult to interpret from the text; please ensure axis labels and feature names are legible, and describe the 'expanded limits' analysis in the main text rather than only in the replication package.
  5. [Abstract, III-A] The abstract uses 'effective'/'ineffective' for closed/open issues without the caveat stated in Section III-A; consider adding a brief qualifier in the abstract to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline gap/status association is computed from manual annotations and an external outcome label, not from fitted parameters or self-citations.

full rationale

The paper's central quantitative claim, that prompts in conversations linked to open issues contain knowledge gaps in 44.6% of cases versus 12.6% for closed issues, is a direct empirical calculation from manually annotated prompt gaps and GitHub issue status. The outcome label (open vs. closed) is external to the gap definitions and is not a fitted quantity. The authors explicitly state the assumption that closed issues indicate effective conversations in Section III-A and acknowledge its limitations in Section VI; this is a validity threat, not circularity, because the gap annotations and the outcome label are separate measurements. RQ2 defines heuristics that correspond to the manually identified gap categories, which creates a mild self-referential flavor, but the logistic regression evaluates these heuristics against issue closure status rather than against the manual gap labels, so the reported association is not forced by construction. The self-citations in the paper, such as [31] and [32], are methodological references for specific text metrics and are not load-bearing for the central conclusion. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The acknowledged open/closed proxy is best treated as a correctness risk, not as circular reasoning.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The quantitative claims rest on a fitted logistic regression, a VIF threshold, and several domain assumptions about what GitHub issue status and shared conversations mean. No physical or mathematical entities are introduced. The heuristic families are measurement constructs built from the annotated gaps, which creates a mild self-referential loop for RQ2.

free parameters (2)
  • Logistic regression feature coefficients (e.g., #Misspellings = -0.1806, Flesch Reading Ease = 0.0697) = 14 coefficients reported in Table IV
    The coefficients are fitted to the same 433 conversations and interpreted via SHAP as evidence that specific heuristics matter. No external validation set is reported.
  • VIF threshold for feature exclusion = 5
    Features with VIF greater than 5 were removed before regression (Section IV-B). This hand-set threshold changes which heuristics enter the model and therefore which associations are reported.
assumptions (4)
  • domain assumption Conversations in closed issues contributed to successful resolutions; those in open issues did not.
    Stated in Section III-A and revisited in Section VI. This is the key validity premise behind every open-versus-closed comparison in the paper.
  • domain assumption Developers share ChatGPT conversations that are relevant to the linked GitHub issue.
    Used to justify the DevGPT sample in Section II. Shared conversations may be self-selected and unrepresentative of all ChatGPT issue interactions.
  • domain assumption GPT-4 correctly identifies unstructured code and error messages in prompts.
    The preprocessing step in Section II relies on GPT-4 following prior work [19]. Errors here propagate into #Code Snippets, #Error Message, and related metrics.
  • domain assumption The refined annotation categories capture the same construct in open and closed conversations.
    A blinded re-annotation of 50 conversations found 4 disagreements, so invariance is partially checked but not fully established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution." pith.science (2026). https://pith.science/paper/CVHSYKCP

@misc{pith2026250111709,
  author       = {Pith},
  title        = {Pith review of: Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVHSYKCP}},
  note         = {Machine review of arXiv:2501.11709}
}
read the original abstract

Large language models (LLMs) have become essential in software development, especially for issue resolution. However, despite their widespread use, significant challenges persist in the quality of LLM responses to issue resolution queries. LLM interactions often yield incorrect, incomplete, or ambiguous information, largely due to knowledge gaps in prompt design, which can lead to unproductive exchanges and reduced developer productivity. In this paper, we analyze 433 developer-ChatGPT conversations within GitHub issue threads to examine the impact of prompt knowledge gaps and conversation styles on issue resolution. We identify four main knowledge gaps in developer prompts: Missing Context, Missing Specifications, Multiple Context, and Unclear Instructions. Assuming that conversations within closed issues contributed to successful resolutions while those in open issues did not, we find that ineffective conversations contain knowledge gaps in 44.6% of prompts, compared to only 12.6% in effective ones. Additionally, we observe seven distinct conversational styles, with Directive Prompting, Chain of Thought, and Responsive Feedback being the most prevalent. We find that knowledge gaps are present in all styles of conversations, with Missing Context being the most repeated challenge developers face in issue-resolution conversations. Based on our analysis, we identify key textual and code-related heuristics (Specificity, Contextual Richness, and Clarity) that are associated with successful issue closure and help assess prompt quality. These heuristics lay the foundation for an automated tool that can dynamically flag unclear prompts and suggest structured improvements. To test feasibility, we developed a lightweight browser extension prototype for detecting prompt gaps, that can be easily adapted to other tools within developer workflows.

Figures

Figures reproduced from arXiv: 2501.11709 by the authors.

Figure 1
Figure 1. Example of Open vs. Closed Conversations: Closed conversation provides Context and Specifications to ChatGPT vs. the missing Context in open conversation lead ChatGPT to hallucinate. We calculate the frequency of #Constraints, #Modifiers, and #Subordinate Clauses using Python’s NLTK. For instance, Subordinate Clauses introduce additional details or conditions that refine the request (e.g., “if the library is compati… view at source ↗
Figure 2
Figure 2. shows the frequency of the seven conversation styles across open and closed issues. The predominant styles of conversation in both open and closed issue threads were Direc￾tive Prompting, Chain of Thought, and Responsive Feedback. Given the large sample size, we applied the independent t￾test [38] (p-value<0.05), which is robust to minor deviations from normality. The t-test showed no significant difference in the s… view at source ↗
Figure 3
Figure 3. Multiple Context in a Conversation Linked to an Open Issue conversation style for both open and closed issues. Since closed issues have a higher total prompt count, the higher number of knowledge gaps within each category for open issues result in higher percentages. Almost all conversation styles in open issues exhibit at least one type of gap, with Chain of Thought showing the highest number of gaps across all ide… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Progression of Conversations with Prompt Knowledge Gaps B. RQ2: What heuristics can be used to automatically mea￾sure the prompt knowledge gaps? Table II presents the set of textual and code-related heuris￾tics that we investigate to automatically measure prompt knowle…
Figure 5
Figure 5. Figure 5: Impact of Features on Model’s Outcome Based on SHAP V. FEASIBILITY STUDY To explore whether the heuristics from RQ2 can be lever￾aged to develop a tool for detecting prompt knowledge gaps, we conduct a feasibility study. Our goal is to develop a lightweight tool that c…
Figure 6
Figure 6. Figure 6: Tool for Automatic Prompt Knowledge Gap Detection adoption and refinement, we have made the code and usage instructions available in our replication package [43]. VI. THREATS TO VALIDITY Construct Validity. To reduce subjectivity in our annotations, we conducted multip…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution

    cs.MA 2026-08 conditional novelty 6.0 of 10

    An eight-agent question-asking system that front-loads intent clarification produced more complete prompts, higher-rated outputs, and single-turn task completion in a four-person pilot, with unstable effect sizes.

  2. Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement

    cs.SE 2025-05 conditional novelty 6.0 of 10

    AlignMind, a multi-agent system with theory-of-mind helpers, refines software requirements through multi-round dialogue, outperforming a direct-prompt baseline in LLM-judged quality and lexical richness, at high token...

  3. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

62 extracted references · 45 canonical work pages · cited by 3 Pith papers

  1. [1]

    The state of developer ecosystem 2023

    “The state of developer ecosystem 2023.” [Online]. Available: https://www.jetbrains.com/lp/devecosystem-2023/

    work page 2023

  2. [2]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” 2024

    work page 2024

  3. [3]

    Large language models in fault localisation,

    Y . Wu, Z. Li, J. M. Zhang, M. Papadakis, M. Harman, and Y . Liu, “Large language models in fault localisation,” 2023

    work page 2023

  4. [4]

    How conversational programming will democra- tize computing

    “How conversational programming will democra- tize computing.” [Online]. Available: https://thenewstack.io/ how-conversational-programming-will-democratize-computing/

    work page

  5. [5]

    The programmer’s assistant: Conversational interaction with a large language model for software development,

    S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, “The programmer’s assistant: Conversational interaction with a large language model for software development,” in Proceedings of the 28th International Conference on Intelligent User Interfaces , ser. IUI ’23. ACM, Mar. 2023

    work page 2023

  6. [6]

    ”always nice and confident, sometimes wrong

    J. Li, E. D. Mynatt, V . Mishra, and J. Bell, “”always nice and confident, sometimes wrong”: Developer’s experiences engaging generative ai chat- bots versus human-powered q&a platforms,”ArXiv, vol. abs/2309.13684, 2023

    arXiv 2023

  7. [7]

    Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution,

    S. Mondal, S. D. Bappon, and C. K. Roy, “Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution,” 2024

    work page 2024

  8. [8]

    Exploring the problems, their causes and solutions of ai pair programming: A study with practitioners of github copilot,

    X. Zhou, P. Liang, B. Zhang, Z. Li, A. Ahmad, M. Shahin, and M. Waseem, “Exploring the problems, their causes and solutions of ai pair programming: A study with practitioners of github copilot,” 2024

    work page 2024

Show all 62 references
  1. [9]

    Exploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wang, Z. Yang, L. Zhang, Z. Li, and Y . Ma, “Exploring and evaluating hallucinations in llm-powered code generation,” 2024

    2024

  2. [10]

    The unreasonable effectiveness of eccentric automatic prompts,

    R. Battle and T. Gollapudi, “The unreasonable effectiveness of eccentric automatic prompts,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.10949

    2024 arXiv

  3. [11]

    Why and when llm- based assistants can go wrong: Investigating the effectiveness of prompt- based interactions for software help-seeking,

    A. Khurana, H. Subramonyam, and P. K. Chilana, “Why and when llm- based assistants can go wrong: Investigating the effectiveness of prompt- based interactions for software help-seeking,” in Proceedings of the 29th International Conference on Intelligent User Interfaces , ser. ...

    2024

  4. [12]

    An empirical study on developers shared conversations with chatgpt in github pull requests and issues,

    H. Hao, K. A. Hasan, H. Qin, M. Macedo, Y . Tian, S. H. H. Ding, and A. E. Hassan, “An empirical study on developers shared conversations with chatgpt in github pull requests and issues,” 2024

    2024

  5. [13]

    What you say = what you want? teaching humans to articulate requirements for llms,

    Q. Ma, W. Peng, H. Shen, K. Koedinger, and T. Wu, “What you say = what you want? teaching humans to articulate requirements for llms,”

  6. [14]

    Which is better? exploring prompting strategy for llm-based metrics,

    J. Kim, S. Park, K. Jeong, S. Lee, S. H. Han, J. Lee, and P. Kang, “Which is better? exploring prompting strategy for llm-based metrics,”

  7. [15]

    DevGPT: Studying Developer-ChatGPT Conversations,

    T. Xiao, C. Treude, H. Hata, and K. Matsumoto, “DevGPT: Studying Developer-ChatGPT Conversations,” Feb. 2024, arXiv:2309.03914 [cs]. [Online]. Available: http://arxiv.org/abs/2309.03914

    2024 arXiv

  8. [16]

    Chatting with ai: Deciphering developer conversations with chatgpt,

    S. Mohamed, A. Parvin, and E. Parra, “Chatting with ai: Deciphering developer conversations with chatgpt,” 2024

    2024

  9. [17]

    On the taxonomy of develop- ers’ discussion topics with chatgpt,

    M. R. I. Ertugrul Sagdic, Arda Bayram, “On the taxonomy of develop- ers’ discussion topics with chatgpt,” 2024

    2024

  10. [18]

    pemistahl/lingua-py,

    P. M. Stahl, “pemistahl/lingua-py,” 2024. [Online]. Available: https: //github.com/pemistahl/lingua-py

    2024

  11. [19]

    Chatgpt code detection: Techniques for uncovering the source of code,

    M. Oedingen, R. C. Engelhardt, R. Denz, M. Hammer, and W. Konen, “Chatgpt code detection: Techniques for uncovering the source of code,” AI, vol. 5, no. 3, 2024. [Online]. Available: http://dx.doi.org/10.3390/ai5030053

    2024 doi

  12. [20]

    Qualitative research: deductive and induc- tive approaches to data analysis,

    T. Azungah and R. Kasmad, “Qualitative research: deductive and induc- tive approaches to data analysis,” 08 2020

    2020

  13. [21]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” 2023. [Online]. Available: https://arxiv.org/abs/2302.11382

    2023 arXiv

  14. [22]

    An empirical categorization of prompting techniques for large language models: A practitioner’s guide,

    O. Fagbohun, R. M. Harrison, and A. Dereventsov, “An empirical categorization of prompting techniques for large language models: A practitioner’s guide,” 2024. [Online]. Available: https://arxiv.org/abs/ 2402.14837

    2024 arXiv

  15. [23]

    Prompt Engineering Guide – Nextra,

    “Prompt Engineering Guide – Nextra,” Sep. 2024. [Online]. Available: https://www.promptingguide.ai/

    2024

  16. [24]

    Interrater reliability: The kappa statistic,

    M. McHugh, “Interrater reliability: The kappa statistic,” Biochemia medica : ˇcasopis Hrvatskoga druˇstva medicinskih biokemiˇcara / HDMB, vol. 22, pp. 276–82, 10 2012

    2012

  17. [25]

    Corbin and A

    J. Corbin and A. Strauss, Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory . SAGE Publications, Inc., 2008

    2008

  18. [26]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions...

    2022 arXiv

  19. [27]

    Unsupervised software-specific morphological forms inference from informal discussions,

    C. Chen, Z. Xing, and X. Wang, “Unsupervised software-specific morphological forms inference from informal discussions,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 2017, pp. 450–461

    2017

  20. [28]

    Dictionary of Software Terms

    “Dictionary of Software Terms.” [Online]. Available: https://techterms. com/category/software

  21. [29]

    NLTK : Natural Language Toolkit

    “NLTK : Natural Language Toolkit.” [Online]. Available: https: //www.nltk.org/

  22. [30]

    Exploring context- aware conversational agents in software development,

    G. Melo, E. Law, P. Alencar, and D. Cowan, “Exploring context- aware conversational agents in software development,” 2020. [Online]. Available: https://arxiv.org/abs/2006.02370

    2020 arXiv

  23. [31]

    Automatically identifying the quality of developer chats for post hoc use,

    P. Chatterjee, K. Damevski, N. A. Kraft, and L. Pollock, “Automatically identifying the quality of developer chats for post hoc use,” ACM Trans. Softw. Eng. Methodol. , vol. 30, no. 4, jul 2021. [Online]. Available: https://doi.org/10.1145/3450503

    2021 doi

  24. [32]

    Extracting code segments and their descriptions from research articles,

    P. Chatterjee, B. Gause, H. Hedinger, and L. Pollock, “Extracting code segments and their descriptions from research articles,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) , 2017, pp. 91–101. [Online]. Available: https: //doi.org/10.110...

    2017 doi

  25. [33]

    spacy · PyPI

    “spacy · PyPI.” [Online]. Available: https://pypi.org/project/spacy/

  26. [34]

    pyspellchecker: Pure python spell checker based on work by Peter Norvig

    T. Barrus, “pyspellchecker: Pure python spell checker based on work by Peter Norvig.”

  27. [35]

    Flesch Reading Ease and the Flesch Kincaid Grade Level

    “Flesch Reading Ease and the Flesch Kincaid Grade Level.” [Online]. Available: https://readable.com/readability/ flesch-reading-ease-flesch-kincaid-grade-level/

  28. [36]

    The SMOG Readability Formula, a Simple Measure of Gobbledygook,

    B. Scott, “The SMOG Readability Formula, a Simple Measure of Gobbledygook,” Oct. 2023, section: Readability Formulas Help. [Online]. Available: https://readabilityformulas.com/ the-smog-readability-formula/

    2023

  29. [37]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019

    2019

  30. [38]

    T test as a parametric statistic,

    T. K. Kim, “T test as a parametric statistic,” Korean Journal of Anesthesiology, vol. 68, no. 6, p. 540, Nov. 2015. [Online]. Available: https://pmc.ncbi.nlm.nih.gov/articles/PMC4667138/

    2015

  31. [39]

    Normality Tests for Statistical Analysis: A Guide for Non-Statisticians,

    A. Ghasemi and S. Zahediasl, “Normality Tests for Statistical Analysis: A Guide for Non-Statisticians,” International Journal of Endocrinology and Metabolism, vol. 10, no. 2, pp. 486–489, 2012. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3693611/

    2012

  32. [40]

    The mann-whitney u: A test for assessing whether two independent samples come from the same distribution,

    N. Nachar, “The mann-whitney u: A test for assessing whether two independent samples come from the same distribution,” Tutorials in Quantitative Methods for Psychology , vol. 4, 03 2008

    2008

  33. [41]

    Collinearity diagnostics of binary logistic regression model,

    S. S. Habshah Midi and S. Rana, “Collinearity diagnostics of binary logistic regression model,” Journal of Interdisciplinary Mathematics , vol. 13, no. 3, pp. 253–267, 2010. [Online]. Available: https: //doi.org/10.1080/09720502.2010.10700699

    2010

  34. [42]

    A unified approach to interpreting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, p. 4768–4777

    2017

  35. [43]

    Replication package

    “Replication package.” [Online]. Available: https://github.com/ SOAR-Lab/prompt-knowledge-gap

  36. [44]

    An empirical study of developer behaviors for validating and repairing ai-generated code

    N. Tang, M. Chen, Z. Ning, A. Bansal, Y . Huang, C. McMillan, and T. J.-J. Li, “An empirical study of developer behaviors for validating and repairing ai-generated code.” Plateau Workshop

  37. [45]

    A deep dive into large language models for automated bug localization and repair,

    S. B. Hossain, N. Jiang, Q. Zhou, X. Li, W.-H. Chiang, Y . Lyu, H. Nguyen, and O. Tripp, “A deep dive into large language models for automated bug localization and repair,” vol. 1, no. FSE, 2024. [Online]. Available: https://doi.org/10.1145/3660773

    2024 doi

  38. [47]

    Investigating the utility of chatgpt in the issue tracking system: An exploratory study,

    J. K. Das, S. Mondal, and C. K. Roy, “Investigating the utility of chatgpt in the issue tracking system: An exploratory study,” 2024

    2024

  39. [48]

    Chatgpt vs llama: Impact, reliability, and challenges in stack overflow discussions,

    L. D. Silva, J. Samhi, and F. Khomh, “Chatgpt vs llama: Impact, reliability, and challenges in stack overflow discussions,” 2024

    2024

  40. [49]

    Evaluating privacy questions from stack overflow: Can chatgpt compete?

    Z. Delile, S. Radel, J. Godinez, G. Engstrom, T. Brucker, K. Young, and S. Ghanavati, “Evaluating privacy questions from stack overflow: Can chatgpt compete?” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) , 2023, pp. 239–244

    2023

  41. [50]

    Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,

    S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,” 2024

    2024

  42. [51]

    Are we ready to embrace generative ai for software q&a?

    B. Xu, T.-D. Nguyen, T. Le-Cong, T. Hoang, J. Liu, K. Kim, C. Gong, C. Niu, C. Wang, B. Le, and D. Lo, “Are we ready to embrace generative ai for software q&a?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2023, pp. 1713–1717

    2023

  43. [52]

    Which is a better programming assistant? a comparative study between chatgpt and stack overflow,

    J. Liu, X. Tang, L. Li, P. Chen, and Y . Liu, “Which is a better programming assistant? a comparative study between chatgpt and stack overflow,” 2023

    2023

  44. [53]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 1482–1494. [Online]. Available: https://doi.org/10.11...

    2023

  45. [54]

    Using large language models for bug localization and fixing,

    T. D. Viet and K. Markov, “Using large language models for bug localization and fixing,” in 2023 12th International Conference on Awareness Science and Technology (iCAST) , 2023, pp. 192–197

    2023

  46. [55]

    Enhancing automated program repair with solution design,

    J. Zhao, D. Yang, L. Zhang, X. Lian, Z. Yang, and F. Liu, “Enhancing automated program repair with solution design,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY , USA: Association for Computing Mach...

    2024

  47. [56]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” 2024. [Online]. Available: https://arxiv.org/abs/2310. 06770

    2024

  48. [57]

    Magis: Llm-based multi-agent framework for github issue resolution,

    W. Tao, Y . Zhou, Y . Wang, W. Zhang, H. Zhang, and Y . Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,”

  49. [58]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY , USA: Association for Computing Machinery, 2024, p....

    2024

  50. [59]

    Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation,

    K. Jin, C.-Y . Wang, H. V . Pham, and H. Hemmati, “Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation,” Mar. 2024, arXiv:2402.11702 [cs]. [Online]. Available: http://arxiv.org/abs/2402.11702

    2024 arXiv

  51. [60]

    Available: https://arxiv.org/abs/2403.17927

    [Online]. Available: https://arxiv.org/abs/2403.17927

    arXiv

  52. [63]

    Chatgpt in action: Analyzing its use in software development,

    A. I. Champa, M. F. Rabbi, C. Nachuma, and M. F. Zibran, “Chatgpt in action: Analyzing its use in software development,” 2024

    2024

  53. [2023]

    Available: https://arxiv.org/abs/2311.03754

    [Online]. Available: https://arxiv.org/abs/2311.03754

    arXiv

  54. [2024]

    Available: https://arxiv.org/abs/2409.08775

    [Online]. Available: https://arxiv.org/abs/2409.08775

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.