{"id":"84d07ada-9b50-4215-a07e-cf1b6da10f62","arxiv_id":"2506.01604","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Using keyword matching on DevGPT conversations, the authors rank seven prompt patterns by a hand-weighted effectiveness score and claim 'Context and Instruction' and 'Recipe' reduce iterations.","lead":"This paper mines ChatGPT conversations from the DevGPT dataset to see which prompt patterns let developers reach satisfactory code in the fewest back-and-forth turns. It claims 'Context and Instruction' and 'Recipe' patterns are most efficient, but the analysis uses a self-defined effectiveness metric and suffers from several biases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing flaw: Eq. 2's 'effectiveness score' is never validated as a proxy for output quality, and Table V's own score ratios undercut the abstract (Recipe > C&I on PR; Persona > C&I on Issues).","rationale":"Good-faith reading: the paper aims to identify prompt patterns that reduce iterations while maintaining quality. The reader's weakest assumption is the same one I find most load-bearing: Eq. 2 is an unvalidated proxy. Even if the keyword classifier and the 'exploded' multi-pattern rows were accepted, the claim that Context and Instruction is 'most efficient across both datasets' fails against Table V: Recipe's PR score ratio is 14.51 and Persona's Issues ratio is 52.88, both higher than C&I's 12.10 and 13.58. RQ1 also lists different winners (Output Automator, Instructions-Based, Question). So the paper is internally inconsistent, and the external validity of Eq. 2 is unsupported. The suggested annotator study is minimal and decisive: if Eq. 2 does not track human or compiler judgment, all ranking conclusions are vacuous. I therefore do not change the reader's REJECT verdict. Credit where due: the paper is transparent about its dataset and formula, and the code links are a useful starting point; but transparency does not repair an unvalidated target metric and a contradictory headline.","tokens_in":8969,"tokens_out":3804,"duration_ms":41796,"concrete_test":"Take a stratified sample of 100 closed PR and 100 closed Issue conversations that contain detected patterns. For each, have two independent annotators judge whether the final assistant response actually resolves the request and, where code is present, whether the code compiles or passes the associated tests. Compute the Spearman rank correlation between the Eq. 2 effectiveness score and these ground-truth labels, reporting inter-annotator agreement. If the correlation is weak, zero, or negative, the Table V rankings cannot support the claim that particular prompt patterns produce high-quality outputs with minimal iterations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 2 defines effectiveness as 0.5 × answer word count + 0.3 × answer/prompt token ratio + 0.2 × TextBlob sentiment polarity. Nothing in the paper establishes that these components correlate with code correctness, task success, or a human developer's notion of a satisfactory output. In fact, the formula rewards long answers and positive-sounding language, so a verbose but wrong answer can outscore a short correct one. Because every ranking in Table V, the ANOVA, and the abstract's recommendation are computed from this score, the central claim that 'Context and Instruction' and 'Recipe' are 'particularly effective' has no validity anchor. The internal evidence also argues against the abstract: in the PR data, Recipe has a score ratio of 14.51 versus 12.10 for Context and Instruction; in the Issues data, Persona has a ratio of 52.88 versus 13.58 for Context and Instruction. Thus even granting the metric, the stated winner is not supported by the table.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes developer-ChatGPT conversations from the DevGPT dataset to study seven prompt patterns. The authors detect patterns using keyword matching, count interaction numbers, and compute a hand-weighted 'effectiveness score' from answer length, prompt-answer token ratio, and TextBlob sentiment. They then rank patterns by this score, run ANOVA, and conclude that structured patterns, especially 'Context and Instruction' and 'Recipe', achieve satisfactory code generation in fewer interactions. The paper presents three research questions addressing which patterns work within five interactions, which patterns are most efficient, and how the best pattern compares with simple question prompts.","tokens_in":9178,"tokens_out":4549,"duration_ms":49079,"significance":"If the claims were valid, the paper would offer practical, easily actionable guidance for developer-AI collaboration, and its use of a real-world dataset (DevGPT) with publicly available analysis code is a strength. The authors also ask a relevant research question about reducing iterative interaction. However, the central conclusion rests entirely on an unvalidated effectiveness metric, and the paper's own tables contradict the stated findings. The keyword-based pattern detection is overlapping and multi-label handling is inconsistent, so the pattern groups themselves are not well defined. The ANOVA results are computed on non-independent exploded rows, making the reported significance meaningless. As it stands, the paper does not provide a sound empirical basis for its recommendations.","major_comments":[{"comment":"The effectiveness score is never validated as a proxy for output quality. The weights 0.5, 0.3, and 0.2 are arbitrary, and the components (answer word count, prompt-answer token ratio, and TextBlob sentiment polarity) have no established relationship to code correctness or task success. A verbose but incorrect answer can therefore outscore a short, correct one. Because the pattern rankings, the ANOVA results, and the abstract's recommendations are all computed from this score, the paper's central claim has no validity anchor. The authors would need to calibrate the score against human judgments or actual code correctness before using it to rank patterns.","section":"V-C, Eq. (2)"},{"comment":"The abstract and Section V-C conclude that 'Context and Instruction' and 'Recipe' are particularly effective and that Context and Instruction is the most efficient pattern across both datasets. However, Table V shows that in the PR dataset, Recipe has the highest score ratio (14.51 vs. 12.10 for Context and Instruction), and in the Issues dataset, Persona has a score ratio of 52.88 vs. 13.58 for Context and Instruction. Even under the paper's own metric, the stated winners are not supported. Furthermore, Section V-B (RQ1) identifies Output Automator, Simple Instruction, and Question as most efficient for the sub-five-interaction subset, which is not reconciled with the RQ2 ranking.","section":"Table V and Section V-C"},{"comment":"The keyword sets used to detect prompt patterns are overlapping and non-exclusive. For example, 'code' is listed under Output Automator, 'explain' and 'describe' are listed under Instructions-Based, and 'based on' is listed under Context and Instructions, so a single prompt can match multiple patterns. The analysis then 'explodes' multi-label rows into separate observations, meaning the pattern groups are not disjoint. This is evident in Table V: the PR frequencies sum to 8,661 across 3,515 closed PR records, and the Issues frequencies also exceed the record count after explosion. The pattern-group definitions and the ANOVA comparisons are therefore ill-defined.","section":"Table II and Section V-A"},{"comment":"The ANOVA is run after exploding multi-label rows, so the observations are not independent and the reported F-statistics and p-values are not statistically valid. The text also defines 'Normalized Score' as the effectiveness score divided by pattern frequency, yet Table V reports 'Score Ratio' as average effectiveness divided by average number of prompts; the relationship between these metrics is unclear. These inconsistencies directly affect the ranking logic and the conclusions drawn from it.","section":"V-C, ANOVA and score normalization"},{"comment":"The choice of the five-prompt threshold is circular. The authors state that the mean number of interactions for closed outcomes was around four, and then use that observation to justify a threshold of five for identifying 'successful' patterns. Selecting a threshold based on the outcome variable, without an independent criterion, biases RQ1 toward patterns that happen to have average interaction counts near the threshold. A pre-registered or independently justified threshold would be needed for this analysis to be meaningful.","section":"V-B, RQ1 threshold"}],"minor_comments":[{"comment":"The final term is truncated as 'Sentiment polarity of Conversation Ans'; it should read 'Answer'.","section":"Equation (2)"},{"comment":"The dataset statistics are internally inconsistent: Total Pull Requests is 413 while RQ1 states 412, and for Issues the total is 250 while the open and closed counts are 253 and 219. Table IV also reports vastly different record counts (7,976 PR and 12,618 Issues), so the relationship between the sample and the full dataset is unclear.","section":"Table III"},{"comment":"References [10] and [11] are the same paper (Reynolds and McDonell), and some entries are incomplete or malformed (for example, 'T. B. B. et al.' and 'X. W. et al.'), which prevents readers from locating the cited works.","section":"References"},{"comment":"Figures 1-3 are referenced in the text but do not appear in the manuscript, making the heatmaps and score distributions unavailable for verification.","section":"Figures"},{"comment":"While the GitHub repository link in [13] is useful, the manuscript should specify the exact version of the analysis code and the complete preprocessing pipeline, including how multi-label patterns were handled, to allow independent reproduction.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The manuscript has the structure of an exploratory student project, and it currently does not meet the methodological bar for a software engineering venue. The core effectiveness metric is unvalidated, the reported results contradict the paper's own conclusions, and the statistical analysis is invalid due to non-independent exploded rows. A full reanalysis with a validated outcome measure, non-overlapping pattern labels, and an independence-respecting statistical approach could potentially form the basis of a new submission, but the present version is beyond minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a reasonably transparent empirical study of prompt patterns in DevGPT, and the authors are honest about their method. But the central claim does not survive contact with their own Table V, and the effectiveness score is never validated. So the abstract's recommendation is not supported.\n\nWhat's new: they take a public dataset (DevGPT) and apply keyword-based detection for seven known prompt patterns, then compare average number of prompts and a constructed effectiveness score. The idea of mining real developer-ChatGPT conversations to see which prompt structures correlate with shorter interactions is a good one, and the paper makes its data and scripts available. Credit where due: the dataset is real, the preprocessing is described, and the keyword sets are specified.\n\nSoft spots: Equation (2) is load-bearing. Effectiveness = 0.5×response word count + 0.3×token ratio + 0.2×TextBlob sentiment. Nothing ties this to code correctness or task success. A verbose wrong answer scores higher than a terse correct one. Every ranking in Table V is derived from this metric, so the 'which pattern is best' conclusion is a parameter scan, not an empirical finding. The internal evidence actually cuts the other way: in the PR data Recipe has a score ratio of 14.51 vs 12.10 for Context and Instruction; in Issues Persona has 52.88 vs 13.58. The abstract names Context and Instruction as 'particularly effective' but that is not what the table says. The keyword sets overlap ('code' appears in Output Automator; 'explain' in Instructions-Based; 'what' in Question) and multi-pattern conversations are exploded into duplicate rows, which inflates the sample and invalidates the ANOVA's independence assumption. The 5-prompt threshold for RQ1 is also chosen after looking at the data.\n\nThe paper's central argument does not hold up. But the flaws are identifiable and fixable: validate the effectiveness score against at least a sample of human judgments or code compilation; fix the overlapping keyword sets; keep conversation-level analysis. The paper reads like a solid course project.\n\nFor whom: practitioners might still get the intuition that structured prompts reduce iterations, but they won't get a reliable ranking here. Researchers studying human-AI interaction could use this as a cautionary example of metric construction. I would not cite it. It deserves a serious referee because the question is real and the transparency makes a revision possible, but it needs major changes.","headline":"A transparent but methodologically weak study: the effectiveness score is unvalidated and Table V contradicts the abstract, so the central claim does not survive contact with the paper's own data.","tokens_in":9711,"tokens_out":1567,"would_cite":false,"duration_ms":17412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that structured prompt patterns—especially 'Context and Instruction' and 'Recipe'—cut the number of ChatGPT turns needed to get acceptable code, grounding the claim in a large dataset of real developer conversations.","keywords":["prompt engineering","AI-assisted code generation","prompt patterns","developer-AI collaboration","effectiveness score","conversation iterations","large language models","code generation"],"falsifier":"For a random sample of the analyzed conversations, have developers or automated tests judge whether the final AI response truly satisfies the request, then rank the patterns by that ground truth; if Context and Instruction and Recipe do not top the resulting ranking, the paper's central recommendation fails.","tokens_in":8775,"feed_emoji":"🤖","tokens_out":6194,"duration_ms":66093,"temperature":0.7,"pith_summary":"This paper tries to show that the way a developer writes their first prompt to an AI coding assistant determines how many back-and-forth turns are needed before the output is acceptable. Using a public corpus of real ChatGPT-assisted pull requests and issues, it sorts each conversation into one of seven prompt patterns by keyword matching and ranks the patterns by a composite effectiveness score divided by the number of prompts. The findings point to 'Context and Instruction' and 'Recipe' as the most efficient patterns, meaning a prompt that embeds background information with explicit instructions, or spells out steps, tends to reach closure fastest. If this holds, developers can lower the cost of AI-assisted coding by choosing a structured prompt pattern rather than starting with a bare question.","feed_headline":"Context-rich prompts beat bare questions in AI coding, data shows","feed_subtitle":"Analyzing thousands of ChatGPT coding sessions, the paper finds context-plus-instruction prompts reach closure in fewer turns.","key_machinery":"The mechanism is a keyword classifier combined with a composite effectiveness score. Seven named prompt patterns—Persona, Recipe, Template, Output Automator, Instructions-Based, Context and Instructions, and Question—are detected by looking for pattern-specific phrases such as 'you are', 'step-by-step', or 'based on'. Each conversation is scored as 0.5 times the answer's word count, plus 0.3 times the ratio of answer tokens to prompt tokens, plus 0.2 times the answer's sentiment polarity; patterns are then ranked by this average effectiveness score divided by the average number of prompts, a combined metric the paper calls the Score Ratio. This Score Ratio is the load-bearing device that turns raw conversation data into a ranking of which prompt patterns are most efficient.","core_discovery":"The paper's central claim is that structured prompt patterns measurably reduce the number of developer–AI exchanges required for satisfactory code generation, and that two patterns in particular—'Context and Instruction' and 'Recipe'—perform best. On closed pull-request conversations, Recipe achieved the highest score ratio (average effectiveness per prompt count), and Context and Instruction was the most consistent pattern across both pull-request and issue datasets. The authors also report that under a stricter threshold of fewer than five prompts, the most frequent successful patterns were Output Automator, Simple Instruction, and Question, so the overall recommendation is not that every task needs the same pattern but that deliberately structured prompts beat unstructured ones. A statistical test across patterns supports the claim that effectiveness scores differ by pattern rather than varying by chance.","pith_inferences":["An implication the authors leave implicit is that prompt-pattern advice is task-dependent: Recipe for algorithmic tasks, Template for recurring formats, and Question for quick factual lookups.","The effectiveness score rewards long answers and positive sentiment, which may favor verbose or agreeable responses over concise correct code; if that bias is real, the true best pattern could differ from the paper's ranking.","A direct test of the paper's implicit promise would be a controlled experiment where the same coding task is given to a model with a bare question versus a context-and-instruction prompt, measuring turns to the first correct solution.","Because pattern detection relies on fixed keywords, pattern frequencies and rankings should be read as approximate until labels are manually validated or re-done with a more flexible classifier."],"forward_implications":["Developers who write context-plus-instruction prompts can expect shorter ChatGPT sessions than those who start with a bare question.","For multi-step or algorithmic requests, spelling out a step-by-step 'Recipe' gives the highest quality-per-prompt in pull-request data.","The most common human pattern, plain questioning, is not the most efficient; matching the pattern to the task type matters.","Because only closed conversations are ranked, the recommendations apply to tasks that reached completion, not to abandoned or still-open sessions.","Pattern choice has a statistically detectable effect on the paper's effectiveness score, so prompt structure is not noise in developer-AI collaboration."],"supporting_citations":[{"why":"Supplies the corpus of closed developer–ChatGPT pull-request and issue conversations that all pattern counts and effectiveness scores are computed from.","marker":"[12]"},{"why":"Motivates structured prompt design and contributes the idea that well-crafted prompts improve LLM output quality.","marker":"[2]"},{"why":"Inspires the Recipe pattern through step-by-step instruction induction claims.","marker":"[9]"},{"why":"Inspires the Persona pattern through prompt-programming experiments with role-based prompts.","marker":"[10]"},{"why":"Inspires the Output Automator pattern by showing that structured outputs improve reasoning in large language models.","marker":"[7]"},{"why":"Inspires the Template and Context-and-Instruction patterns through prompt-engineering techniques that standardize input formats.","marker":"[8]"}],"fun_headline_variants":["Structured prompts cut AI coding exchanges, study finds","Context and Recipe prompts lead to fewer AI coding rounds","Prompt patterns reduce back-and-forth in AI-assisted coding","Effective AI coding: use context, not just a bare question"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking depends on a made-up quality score that adds response length, a token ratio, and sentiment; no one checked whether that score matches code that actually works or tasks that are actually solved.","fun_headline_variants_meta":{"raw":{"variants":["Structured prompts cut AI coding exchanges, study finds","Context and Recipe prompts lead to fewer AI coding rounds","Prompt patterns reduce back-and-forth in AI-assisted coding","Effective AI coding: use context, not just a bare question"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1451,"prompt_tokens":832,"completion_tokens":619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":448,"tokens_out":619,"duration_ms":6711,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:37:03.145163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a random sample of the analyzed conversations, have developers or automated tests judge whether the final AI response truly satisfies the request, then rank the patterns by that ground truth; if Context and Instruction and Recipe do not top the resulting ranking, the paper's central recommendation fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the corpus of closed developer–ChatGPT pull-request and issue conversations that all pattern counts and effectiveness scores are computed from."},{"cited_title":"Reynolds and K","cited_arxiv_id":null,"evidence_quote":"Inspires the Persona pattern through prompt-programming experiments with role-based prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the Output Automator pattern by showing that structured outputs improve reasoning in large language models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the Template and Context-and-Instruction patterns through prompt-engineering techniques that standardize input formats."}],"review_version":1}