{"id":"428a958b-dbeb-4ade-903c-36f9d7c65c89","arxiv_id":"2501.11709","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In conversations linked to open GitHub issues, 44.6% of developer prompts contain knowledge gaps, versus 12.6% in conversations linked to closed issues; Missing Context is the most common gap.","lead":"This paper studies 433 developer-ChatGPT conversations attached to GitHub issues and finds that prompts in conversations linked to still-open issues contain knowledge gaps (missing context, unclear instructions, etc.) much more often than prompts in conversations linked to closed issues. It also proposes three automated prompt-quality heuristics and a browser extension prototype that scores prompts for context, specificity, and clarity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 44.6% vs. 12.6% gap rests on equating closed issues with effective conversations; a direct contribution-based re-labeling test is needed.","rationale":"The reader's weakest_assumption is exactly the open/closed proxy, and I agree this is the most load-bearing concern. The central claim is the strong association between prompt knowledge gaps and unresolved issues; without a valid outcome label, that association loses its meaning. The paper explicitly acknowledges the proxy but does not test its robustness. My proposed check directly re-labels a sample with a more accurate contribution signal and compares the result. This is a feasible empirical sensitivity analysis that would either confirm the finding or reveal it as a proxy artifact. Since the reader's verdict is already CONDITIONAL and this concern is addressable rather than fatal, no verdict change is needed.","tokens_in":17235,"tokens_out":4032,"duration_ms":40741,"concrete_test":"Take a stratified random sample of 100 of the 433 conversations (50 open, 50 closed). For each, use the GitHub issue thread and the ChatGPT conversation to independently determine whether the conversation materially contributed to the issue's resolution (e.g., the final fix uses code or advice from the conversation, or the thread explicitly credits it). Recompute the knowledge-gap rate per prompt and refit the RQ2 logistic regression using this 'contributed' label as the outcome. If the 44.6% vs. 12.6% gap collapses or the regression coefficients change materially, the headline finding depends on the proxy. If the gap is similar under direct outcome labeling, the proxy concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ1's outcome measure is the GitHub issue status (open/closed), as stated in Section III-A: 'We assume that conversations within closed issues likely contributed to successful resolutions, while those within open issues did not.' This proxy is load-bearing because the headline result—44.6% of prompts in ineffective conversations have knowledge gaps vs. 12.6% in effective ones—is computed with this label, and the RQ2 logistic regression predicts this same label. The proxy can distort the association in both directions: closed issues may be resolved by other contributors or external resources despite an ineffective ChatGPT exchange, and open issues may be abandoned or blocked even when the ChatGPT conversation was helpful. The paper acknowledges this in Section VI but provides no sensitivity analysis. A related confound, issue difficulty, is also set aside: complex issues may be both more likely to remain open and harder for developers to describe fully, which would produce a gap/status association without a causal link from gaps to ineffectiveness. Because the conclusion is phrased as prompt gaps 'leading to unsuccessful outcomes,' the reliance on this coarse proxy is the weakest point in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes 433 developer-ChatGPT conversations extracted from GitHub issue threads (DevGPT dataset) to investigate how prompt knowledge gaps and conversation styles relate to issue resolution. The authors manually annotate the conversations with four prompt knowledge-gap categories (Missing Context, Missing Specification, Unclear Instructions, Multiple Context) and seven conversation styles, using an iterative coding process with reported Cohen's Kappa values. They define conversation effectiveness by the status of the linked GitHub issue (closed = effective, open = ineffective), and report that 44.6% of prompts in ineffective conversations contain knowledge gaps versus 12.6% in effective ones. They then design three heuristic families (Specificity, Contextual Richness, Clarity) and use logistic regression with SHAP explanations to associate these heuristics with issue closure. Finally, they present a browser-extension prototype that scores prompts against these heuristics. The paper argues that prompt knowledge gaps, especially Missing Context, are a central obstacle to LLM-guided issue resolution and that automated gap detection is feasible.","tokens_in":17403,"tokens_out":8075,"duration_ms":71586,"significance":"The paper addresses a timely and practical problem: understanding why developer-LLM interactions fail during issue resolution. Its main strengths are the manually annotated dataset (433 conversations, with iterative coding and inter-rater reliability checks), the public replication package, and the concrete prototype demonstrating a possible tool direction. If the central association were robust, the finding that knowledge gaps (especially Missing Context) are far more prevalent in conversations linked to open issues would be of practical value to developers and tool builders. However, the current evidence is weakened by the acknowledged open/closed proxy, the absence of validation of the heuristics against the manual gap labels, and the incomplete statistical reporting of the regression analysis. The paper is a solid empirical starting point, but its claims presently outrun the evidence.","major_comments":[{"comment":"The outcome proxy is load-bearing: the headline 44.6% vs. 12.6% difference and the RQ2 regression both use GitHub issue status (closed/open) as a direct measure of conversation effectiveness. The assumption stated in Section III-A ('conversations within closed issues likely contributed to successful resolutions, while those within open issues did not') is acknowledged in Section VI but never subjected to sensitivity analysis. Closed issues may be resolved by other means despite an unhelpful conversation, and open issues may be abandoned despite helpful exchanges. In addition, issue difficulty is a plausible confound: difficult issues are more likely to remain open and also more likely to be described incompletely. Please add a robustness analysis (e.g., a subset of conversations explicitly referenced as helpful, or controlling for indicators of issue complexity) and report gap rates at the conversation level, since the current per-prompt percentages ignore the differing numbers of prompts per conversation.","section":"III-A, VI, IV-A"},{"comment":"The logistic regression evidence for the heuristics is incomplete. The model was selected on the same data ('Among various configurations and parameters tested, our best-performing regression model...'), yet no baseline accuracy is given; the reported 62% CV accuracy is close to the majority-class rate (262/433 ≈ 60.5%). Coefficients in Table IV are presented without confidence intervals or standard errors, and the only significance information comes from an undefined t-test in the text. Please report the majority baseline, coefficient confidence intervals, and a proper holdout or nested cross-validation evaluation.","section":"IV-B, Table IV"},{"comment":"The heuristics are partly self-referential. Section III-B states that the heuristics were 'derived using NLP and code-related metrics to directly correspond to the knowledge gaps identified in prompts' (Contextual Richness for Missing/Multiple Context, Specificity for Missing Specification, Clarity for Unclear Instructions), and they are evaluated on the same dataset from which the gap categories were induced. This does not invalidate the descriptive analysis, but it does not constitute validation. Please test whether the heuristic scores actually discriminate the manually annotated gap labels (e.g., correlation or classification against the annotations), or explicitly re-frame RQ2 as a descriptive feasibility analysis rather than a demonstration of association.","section":"III-B, IV-B"},{"comment":"The text states 'We found a significant difference in the number of prompts with knowledge gaps: 334 in open issues compared to only 107 in closed ones,' but no statistical test is reported for this comparison. The p-values mentioned in that section refer to conversation styles, not to gap counts. Please provide a formal test (e.g., chi-square or a mixed-effects model with conversation as a random effect) and adjust for the differing number of prompts per conversation.","section":"IV-A"},{"comment":"The feasibility study does not define how the displayed heuristic scores are computed. The text says the scores are 'the mean average of the features included in each heuristic,' but the features in Table II have entirely different scales (e.g., #Code Snippets vs. Flesch Reading Ease), so an unweighted mean is not meaningful unless features are normalized. The claim that the tool 'correctly identifies' the gaps in the open conversation is not accompanied by any evaluation. Please specify the normalization, describe the score formulas, and provide at least a small reliability check against manual assessments.","section":"V"}],"minor_comments":[{"comment":"The use of the Shapiro-Wilk test to 'confirm' normality is confusing: the variables compared (conversation style counts) are counts, and the text does not say which variable is tested. Please clarify the statistical procedure.","section":"III-A"},{"comment":"The threats-to-validity section reports 'the final average Cohen's Kappa agreement between them was 78%,' without noting that this is the average of the gap (0.84) and style (0.72) Kappas. Please state this explicitly to avoid inconsistency with the earlier numbers.","section":"III-A"},{"comment":"The notation '0<8<289' for min-median-max is easy to misread; consider separate columns for min, median, and max, or a clearer format.","section":"Table II"},{"comment":"The SHAP figure is difficult to interpret from the text; please ensure axis labels and feature names are legible, and describe the 'expanded limits' analysis in the main text rather than only in the replication package.","section":"IV-B, Figure 5"},{"comment":"The abstract uses 'effective'/'ineffective' for closed/open issues without the caveat stated in Section III-A; consider adding a brief qualifier in the abstract to avoid overclaiming.","section":"Abstract, III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution, and the replication package is a plus. The main risk is that the headline claim is tied to a proxy that the authors themselves acknowledge; the requested robustness analyses should be feasible within the scope of the current dataset. I would encourage the editor to send the revised version back to the same reviewers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Adam, here's my take on arXiv:2501.11709.\n\nThe paper is worth reading for its dataset, not for its headline numbers. The 433 developer-ChatGPT conversations, annotated for four knowledge gaps and seven conversation styles with Kappa 0.84 and 0.72, is a real contribution. The taxonomy sharpens Mondal et al.'s eleven gaps into issue-resolution-relevant categories, and the public replication package is a plus. The paper is also admirably transparent: it states the open/closed proxy assumption in Section III-A and returns to it in the threats section. That honesty earns credit.\n\nNow the soft spots, in proportion.\n\nFirst, the central quantitative claim—44.6% vs. 12.6% gap rates—is entirely dependent on the proxy that 'closed' means 'effective.' The stress-test concern is on target: no sensitivity analysis, and issue difficulty is acknowledged but set aside. This makes the causal-sounding conclusion in the abstract too strong for what is actually a correlation under an assumed label.\n\nSecond, the RQ2 model is weaker than it looks. The 62% cross-validated accuracy needs to be benchmarked against the trivial majority-class baseline of 60.5% (262/433 closed issues). Without that baseline, the model's contribution is essentially nil. The logistic regression is selected on the same data, coefficients come without confidence intervals, and the heuristics are defined to mirror the gap categories, so the association is partly built in.\n\nThird, the feasibility prototype is a two-example demo, which is fine as a proof of concept but shouldn't be interpreted as a validated tool.\n\nThe reader's CONDITIONAL verdict is fair. I'd add the baseline issue to the list. None of these flaws are fatal to the paper's real value: the annotated data and the taxonomy will be useful to anyone studying developer-LLM interactions. The fix is straightforward: add a majority-class baseline, report CIs, do a proxy sensitivity check (e.g., re-label ambiguous issues or use contribution-based labels), and maybe test the heuristics against actual resolution contribution.\n\nRecommendation: yes, send this to peer review. A serious referee can push for those additions without rejecting the core dataset work. It's a modest but honest step forward.","headline":"A genuinely useful annotated dataset and taxonomy, but the headline 44.6% vs. 12.6% claim rests on an unvalidated proxy and the model barely beats the majority-class baseline.","tokens_in":17976,"tokens_out":3195,"would_cite":true,"duration_ms":32150,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt knowledge gaps in developer-ChatGPT conversations are closely tied to unresolved GitHub issues, and three measurable heuristics could flag them automatically.","keywords":["prompt knowledge gaps","large language models","issue resolution","GitHub issues","developer-ChatGPT conversations","prompt heuristics","Missing Context","browser extension"],"falsifier":"A decisive check would be an issue-status-blind re-annotation of a random sample of the same 433 conversations: if knowledge-gap rates in closed-issue and open-issue conversations become statistically indistinguishable, or if the difference disappears after stratifying by issue difficulty, the claimed association fails.","tokens_in":16986,"feed_emoji":"💬","tokens_out":11162,"duration_ms":102944,"temperature":0.7,"pith_summary":"The paper tries to establish that vague or incomplete developer prompts are a major reason LLM-assisted issue resolution fails, and that the gaps can be detected automatically. Analyzing 433 developer-ChatGPT conversations shared inside GitHub issue threads, the authors find that conversations linked to open issues contain prompt knowledge gaps in 44.6% of prompts, versus 12.6% in conversations linked to closed issues. Four gap types carry the analysis: Missing Context, Missing Specification, Multiple Context, and Unclear Instructions, with Missing Context the most frequent in both settings. The practical payoff is a set of three measurable heuristic families, Specificity, Contextual Richness, and Clarity, which the authors embed in a lightweight browser extension that flags weak prompts before they derail a conversation.","feed_headline":"Knowledge gaps mark 44.6% of ineffective prompts, vs 12.6% effective","feed_subtitle":"Missing context shows up in all conversation styles; Specificity, Contextual Richness, and Clarity make gaps detectable.","key_machinery":"The load-bearing mechanism is a two-layer scheme: a qualitative annotation taxonomy that names the gap, and a computable feature set that measures it. The taxonomy has four gap categories and seven conversation styles, built by refining prior category sets through iterative dual annotation. The feature set is grouped into Specificity, Contextual Richness, and Clarity, and scored by a logistic regression model that predicts whether a conversation belongs to a closed issue. The paper's outcome proxy, open versus closed GitHub issue status, is what connects the annotations and the heuristics to effectiveness.","core_discovery":"The paper's central claim is that prompt content quality, not just prompting technique, shapes whether LLM-assisted issue resolution succeeds. On 433 developer-ChatGPT conversations drawn from GitHub issue threads, the authors define four prompt knowledge gaps and seven conversation styles, then compare conversations linked to closed issues against those linked to open issues. They report that ineffective conversations carry knowledge gaps in 44.6% of prompts versus 12.6% in effective ones, with Missing Context the single most frequent gap in both and present in every conversation style. The authors conclude that prompt knowledge gaps are a measurable, actionable impediment to resolution, and they propose three heuristic families whose component metrics associate with issue closure and can be computed automatically.","pith_inferences":["Editorial inference: the same three heuristic families could be retrained as a general prompt-quality scorer for other LLM applications, such as documentation Q&A or code review, by swapping the outcome label from issue closure to task success.","Editorial inference: a randomized field test, where half the developers receive the extension's gap warnings and half do not, could convert the observed correlation into a direct test of whether flagging Missing Context changes closure rates.","Editorial inference: because the paper does not control for issue difficulty, the gap metrics might partly track how hard an issue is to describe; scoring prompts on deliberately easy versus hard issues would clarify this.","Editorial inference: the Missing Context finding suggests that a minimal intervention, such as a structured issue template with fields for goal, attempts, code, and expected behavior, could be tested even before building any machine-learning detector."],"forward_implications":["Automated prompt-gap detection is feasible without subjective review: the heuristics are computable from text and code and feed directly into a logistic regression model.","Prompt templates that pre-request context, prior attempts, error logs, and technical specifications should reduce the most common gap, Missing Context.","Since knowledge gaps appear in all seven conversation styles, tools should nudge developers on content rather than on prompting technique.","Developers can be guided toward short code snippets plus error messages and references, since large snippet size and missing error logs are associated with open issues.","A browser-extension prototype demonstrates a practical deployment path inside normal developer workflows, not just a retrospective analysis."],"supporting_citations":[{"why":"Supplies the corpus from which the 433 developer-ChatGPT conversations are drawn.","marker":"[15]"},{"why":"Provides the initial prompt-gap taxonomy that is refined into the paper's four gap types.","marker":"[7]"},{"why":"Establishes that developer-ChatGPT conversations shared in GitHub issues are relevant to issue resolution, justifying the dataset selection.","marker":"[12]"},{"why":"Supplies the initial conversation-style categories from which the final seven styles are derived.","marker":"[21]"},{"why":"Provides empirical grounding for the Chain of Thought, Tree of Thought, and related prompting styles used in the style taxonomy.","marker":"[22]"}],"fun_headline_variants":["44.6% of failed prompt chats show knowledge gaps, vs 12.6% for wins","Missing context is the top prompt gap in every LLM conversation style","Specificity, context, and clarity heuristics flag weak prompts in GitHub","Prompt gaps: 44.6% in unresolved chats vs 12.6% resolved","Knowledge gaps in prompts predict failed issue resolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's entire comparison leans on the assumption that a conversation was effective exactly when its GitHub issue is closed and ineffective when the issue remains open, so unseen external help or arbitrary issue abandonment would muddy the measured link between prompt gaps and outcomes.","fun_headline_variants_meta":{"raw":{"variants":["44.6% of failed prompt chats show knowledge gaps, vs 12.6% for wins","Missing context is the top prompt gap in every LLM conversation style","Specificity, context, and clarity heuristics flag weak prompts in GitHub","Prompt gaps: 44.6% in unresolved chats vs 12.6% resolved","Knowledge gaps in prompts predict failed issue resolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3037,"prompt_tokens":987,"completion_tokens":2050,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1951}},"tokens_in":603,"tokens_out":2050,"duration_ms":15069,"temperature":1.0,"reasoning_tokens":1951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:56:34.895912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be an issue-status-blind re-annotation of a random sample of the same 433 conversations: if knowledge-gap rates in closed-issue and open-issue conversations become statistically indistinguishable, or if the difference disappears after stratifying by issue difficulty, the claimed association fails.","supporting_citations":[{"cited_title":"Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution,","cited_arxiv_id":null,"evidence_quote":"Provides the initial prompt-gap taxonomy that is refined into the paper's four gap types."},{"cited_title":"An empirical study on developers shared conversations with chatgpt in github pull requests and issues,","cited_arxiv_id":null,"evidence_quote":"Establishes that developer-ChatGPT conversations shared in GitHub issues are relevant to issue resolution, justifying the dataset selection."}],"review_version":1}