Exploring Prompt Patterns in AI-Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration

Amanda Zambrana; Eman Abdullah AlOmar; Khushi Suman; Priyanshi Yadav; Sashidhar Madiraju; Sophia DiCuffa

REVIEW 5 major objections 5 minor 13 references

Exploring Prompt Patterns in AI-Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that structured prompt patterns—especially 'Context and Instruction' and 'Recipe'—cut the number of ChatGPT turns needed to get acceptable code, grounding the claim in a large dataset of real developer conversations.

desk verdict A transparent but methodologically weak study: the effectiveness score is unvalidated and Table V contradicts the abstract, so the central claim does not survive contact with the paper's own data. read the letter →

arxiv 2506.01604 v1 pith:FFRNXYTZ submitted 2025-06-02 cs.SE

Sophia DiCuffa , Amanda Zambrana , Priyanshi Yadav , Sashidhar Madiraju , Khushi Suman , Eman Abdullah AlOmar This is my paper

classification cs.SE

keywords promptengineeringAI-assistedcodegenerationpatternsdeveloper-AIcollaborationeffectivenessscoreconversationiterationslargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the way a developer writes their first prompt to an AI coding assistant determines how many back-and-forth turns are needed before the output is acceptable. Using a public corpus of real ChatGPT-assisted pull requests and issues, it sorts each conversation into one of seven prompt patterns by keyword matching and ranks the patterns by a composite effectiveness score divided by the number of prompts. The findings point to 'Context and Instruction' and 'Recipe' as the most efficient patterns, meaning a prompt that embeds background information with explicit instructions, or spells out steps, tends to reach closure fastest. If this holds, developers can lower the cost of AI-assisted coding by choosing a structured prompt pattern rather than starting with a bare question.

What carries the argument

The mechanism is a keyword classifier combined with a composite effectiveness score. Seven named prompt patterns—Persona, Recipe, Template, Output Automator, Instructions-Based, Context and Instructions, and Question—are detected by looking for pattern-specific phrases such as 'you are', 'step-by-step', or 'based on'. Each conversation is scored as 0.5 times the answer's word count, plus 0.3 times the ratio of answer tokens to prompt tokens, plus 0.2 times the answer's sentiment polarity; patterns are then ranked by this average effectiveness score divided by the average number of prompts, a combined metric the paper calls the Score Ratio. This Score Ratio is the load-bearing device that turns raw conversation data into a ranking of which prompt patterns are most efficient.

What would settle it

For a random sample of the analyzed conversations, have developers or automated tests judge whether the final AI response truly satisfies the request, then rank the patterns by that ground truth; if Context and Instruction and Recipe do not top the resulting ranking, the paper's central recommendation fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured prompt patterns measurably reduce the number of developer–AI exchanges required for satisfactory code generation, and that two patterns in particular—'Context and Instruction' and 'Recipe'—perform best. On closed pull-request conversations, Recipe achieved the highest score ratio (average effectiveness per prompt count), and Context and Instruction was the most consistent pattern across both pull-request and issue datasets. The authors also report that under a stricter threshold of fewer than five prompts, the most frequent successful patterns were Output Automator, Simple Instruction, and Question, so the overall recommendation is not that every task needs the same pattern but that deliberately structured prompts beat unstructured ones. A statistical test across patterns supports the claim that effectiveness scores differ by pattern rather than varying by chance.

Load-bearing premise

The whole ranking depends on a made-up quality score that adds response length, a token ratio, and sentiment; no one checked whether that score matches code that actually works or tasks that are actually solved.

Editorial extensions

If this is right

Developers who write context-plus-instruction prompts can expect shorter ChatGPT sessions than those who start with a bare question.
For multi-step or algorithmic requests, spelling out a step-by-step 'Recipe' gives the highest quality-per-prompt in pull-request data.
The most common human pattern, plain questioning, is not the most efficient; matching the pattern to the task type matters.
Because only closed conversations are ranked, the recommendations apply to tasks that reached completion, not to abandoned or still-open sessions.
Pattern choice has a statistically detectable effect on the paper's effectiveness score, so prompt structure is not noise in developer-AI collaboration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit is that prompt-pattern advice is task-dependent: Recipe for algorithmic tasks, Template for recurring formats, and Question for quick factual lookups.
The effectiveness score rewards long answers and positive sentiment, which may favor verbose or agreeable responses over concise correct code; if that bias is real, the true best pattern could differ from the paper's ranking.
A direct test of the paper's implicit promise would be a controlled experiment where the same coding task is given to a model with a bare question versus a context-and-instruction prompt, measuring turns to the first correct solution.
Because pattern detection relies on fixed keywords, pattern frequencies and rankings should be read as approximate until labels are manually validated or re-done with a more flexible classifier.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper is a reasonably transparent empirical study of prompt patterns in DevGPT, and the authors are honest about their method. But the central claim does not survive contact with their own Table V, and the effectiveness score is never validated. So the abstract's recommendation is not supported.

What's new: they take a public dataset (DevGPT) and apply keyword-based detection for seven known prompt patterns, then compare average number of prompts and a constructed effectiveness score. The idea of mining real developer-ChatGPT conversations to see which prompt structures correlate with shorter interactions is a good one, and the paper makes its data and scripts available. Credit where due: the dataset is real, the preprocessing is described, and the keyword sets are specified.

Soft spots: Equation (2) is load-bearing. Effectiveness = 0.5×response word count + 0.3×token ratio + 0.2×TextBlob sentiment. Nothing ties this to code correctness or task success. A verbose wrong answer scores higher than a terse correct one. Every ranking in Table V is derived from this metric, so the 'which pattern is best' conclusion is a parameter scan, not an empirical finding. The internal evidence actually cuts the other way: in the PR data Recipe has a score ratio of 14.51 vs 12.10 for Context and Instruction; in Issues Persona has 52.88 vs 13.58. The abstract names Context and Instruction as 'particularly effective' but that is not what the table says. The keyword sets overlap ('code' appears in Output Automator; 'explain' in Instructions-Based; 'what' in Question) and multi-pattern conversations are exploded into duplicate rows, which inflates the sample and invalidates the ANOVA's independence assumption. The 5-prompt threshold for RQ1 is also chosen after looking at the data.

The paper's central argument does not hold up. But the flaws are identifiable and fixable: validate the effectiveness score against at least a sample of human judgments or code compilation; fix the overlapping keyword sets; keep conversation-level analysis. The paper reads like a solid course project.

For whom: practitioners might still get the intuition that structured prompts reduce iterations, but they won't get a reliable ranking here. Researchers studying human-AI interaction could use this as a cautionary example of metric construction. I would not cite it. It deserves a serious referee because the question is real and the transparency makes a revision possible, but it needs major changes.

Referee Report

5 major / 5 minor

Summary. The paper analyzes developer-ChatGPT conversations from the DevGPT dataset to study seven prompt patterns. The authors detect patterns using keyword matching, count interaction numbers, and compute a hand-weighted 'effectiveness score' from answer length, prompt-answer token ratio, and TextBlob sentiment. They then rank patterns by this score, run ANOVA, and conclude that structured patterns, especially 'Context and Instruction' and 'Recipe', achieve satisfactory code generation in fewer interactions. The paper presents three research questions addressing which patterns work within five interactions, which patterns are most efficient, and how the best pattern compares with simple question prompts.

Significance. If the claims were valid, the paper would offer practical, easily actionable guidance for developer-AI collaboration, and its use of a real-world dataset (DevGPT) with publicly available analysis code is a strength. The authors also ask a relevant research question about reducing iterative interaction. However, the central conclusion rests entirely on an unvalidated effectiveness metric, and the paper's own tables contradict the stated findings. The keyword-based pattern detection is overlapping and multi-label handling is inconsistent, so the pattern groups themselves are not well defined. The ANOVA results are computed on non-independent exploded rows, making the reported significance meaningless. As it stands, the paper does not provide a sound empirical basis for its recommendations.

major comments (5)

[V-C, Eq. (2)] The effectiveness score is never validated as a proxy for output quality. The weights 0.5, 0.3, and 0.2 are arbitrary, and the components (answer word count, prompt-answer token ratio, and TextBlob sentiment polarity) have no established relationship to code correctness or task success. A verbose but incorrect answer can therefore outscore a short, correct one. Because the pattern rankings, the ANOVA results, and the abstract's recommendations are all computed from this score, the paper's central claim has no validity anchor. The authors would need to calibrate the score against human judgments or actual code correctness before using it to rank patterns.
[Table V and Section V-C] The abstract and Section V-C conclude that 'Context and Instruction' and 'Recipe' are particularly effective and that Context and Instruction is the most efficient pattern across both datasets. However, Table V shows that in the PR dataset, Recipe has the highest score ratio (14.51 vs. 12.10 for Context and Instruction), and in the Issues dataset, Persona has a score ratio of 52.88 vs. 13.58 for Context and Instruction. Even under the paper's own metric, the stated winners are not supported. Furthermore, Section V-B (RQ1) identifies Output Automator, Simple Instruction, and Question as most efficient for the sub-five-interaction subset, which is not reconciled with the RQ2 ranking.
[Table II and Section V-A] The keyword sets used to detect prompt patterns are overlapping and non-exclusive. For example, 'code' is listed under Output Automator, 'explain' and 'describe' are listed under Instructions-Based, and 'based on' is listed under Context and Instructions, so a single prompt can match multiple patterns. The analysis then 'explodes' multi-label rows into separate observations, meaning the pattern groups are not disjoint. This is evident in Table V: the PR frequencies sum to 8,661 across 3,515 closed PR records, and the Issues frequencies also exceed the record count after explosion. The pattern-group definitions and the ANOVA comparisons are therefore ill-defined.
[V-C, ANOVA and score normalization] The ANOVA is run after exploding multi-label rows, so the observations are not independent and the reported F-statistics and p-values are not statistically valid. The text also defines 'Normalized Score' as the effectiveness score divided by pattern frequency, yet Table V reports 'Score Ratio' as average effectiveness divided by average number of prompts; the relationship between these metrics is unclear. These inconsistencies directly affect the ranking logic and the conclusions drawn from it.
[V-B, RQ1 threshold] The choice of the five-prompt threshold is circular. The authors state that the mean number of interactions for closed outcomes was around four, and then use that observation to justify a threshold of five for identifying 'successful' patterns. Selecting a threshold based on the outcome variable, without an independent criterion, biases RQ1 toward patterns that happen to have average interaction counts near the threshold. A pre-registered or independently justified threshold would be needed for this analysis to be meaningful.

minor comments (5)

[Equation (2)] The final term is truncated as 'Sentiment polarity of Conversation Ans'; it should read 'Answer'.
[Table III] The dataset statistics are internally inconsistent: Total Pull Requests is 413 while RQ1 states 412, and for Issues the total is 250 while the open and closed counts are 253 and 219. Table IV also reports vastly different record counts (7,976 PR and 12,618 Issues), so the relationship between the sample and the full dataset is unclear.
[References] References [10] and [11] are the same paper (Reynolds and McDonell), and some entries are incomplete or malformed (for example, 'T. B. B. et al.' and 'X. W. et al.'), which prevents readers from locating the cited works.
[Figures] Figures 1-3 are referenced in the text but do not appear in the manuscript, making the heatmaps and score distributions unavailable for verification.
[Reproducibility] While the GitHub repository link in [13] is useful, the manuscript should specify the exact version of the analysis code and the complete preprocessing pipeline, including how multi-label patterns were handled, to allow independent reproduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ranking is derived from a hand-weighted but independently defined effectiveness metric, so the central claim does not reduce to its inputs by construction.

full rationale

I walked the derivation chain from DevGPT conversation data through keyword-based pattern detection (Table II), Eq. (2)'s effectiveness score, and the mean/ratio rankings in Table V. No step exhibits a definitional reduction: the pattern labels are assigned from keyword lists that do not reference the effectiveness score, and the effectiveness score is a hand-weighted combination of response length, token ratio, and sentiment that does not include the pattern label or the predicted conclusion. Thus the finding that some patterns score higher is an empirical consequence of the chosen metric, not a tautology. The paper does contain self-citations (e.g., refs. [5] and [13] are author-maintained data/code repositories, and refs. [1], [3], [4] include co-authors), but these are used as data/code pointers or background literature, not as load-bearing evidence for the central claim. The serious weaknesses here are construct validity and internal consistency: Eq. (2) is never validated as a proxy for code quality or task success, and Table V's own score ratios do not clearly support the abstract's choice of 'Context and Instruction' as most efficient (Recipe has a higher PR score ratio, and Persona has a higher Issues score ratio). These are correctness risks rather than circularity. Under the requirement to exhibit a specific reduction, no circular step can be quoted, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a custom, unvalidated effectiveness metric and several untested domain assumptions about the DevGPT data. These are not supported by external evidence, and the headline finding is contradicted by the paper's own Table V.

free parameters (3)

Effectiveness score weights = 0.5, 0.3, 0.2
Chosen by hand without justification. The weights directly determine the effectiveness score and thus the pattern ranking in Table V.
RQ1 interaction threshold = 5 prompts
Set based on the mean number of prompts in closed conversations. Used to filter 'efficient' conversations in RQ1, reflecting a data-driven threshold.
Pattern detection keyword sets = Table II keywords
Hand-selected keywords with heavy overlap among patterns. The classification of conversations depends entirely on these lists.

assumptions (5)

domain assumption DevGPT conversations are representative of real developer-AI interactions.
Used throughout the study without validation; the dataset may be skewed or incomplete.
domain assumption A closed PR or issue implies a satisfactory outcome.
The authors treat closed states as successful, but closed items may also be abandoned or closed without resolution.
domain assumption The number of prompts measures interaction efficiency.
Used as the primary efficiency metric without controlling for task complexity or developer style.
domain assumption Keyword presence accurately identifies the prompt pattern.
Keyword sets overlap and can match multiple patterns, yet the analysis treats them as reliable classifiers.
ad hoc to paper Sentiment polarity of the answer is a meaningful component of output quality.
No evidence supports the use of TextBlob sentiment as a proxy for code quality or task success.

invented entities (1)

Effectiveness Score
purpose: To quantify prompt effectiveness by combining response length, token ratio, and sentiment.
This is a newly defined metric in the paper. It is not validated against any external measure of code correctness or developer satisfaction, and the pattern ranking depends entirely on it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Prompt Patterns in AI-Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration." pith.science (2026). https://pith.science/paper/FFRNXYTZ

@misc{pith2026250601604,
  author       = {Pith},
  title        = {Pith review of: Exploring Prompt Patterns in AI-Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFRNXYTZ}},
  note         = {Machine review of arXiv:2506.01604}
}

read the original abstract

The growing integration of AI tools in software development, particularly Large Language Models (LLMs) such as ChatGPT, has revolutionized how developers approach coding tasks. However, achieving high-quality code often requires iterative interactions, which can be time-consuming and inefficient. This paper explores the application of structured prompt patterns to minimize the number of interactions required for satisfactory AI-assisted code generation. Using the DevGPT dataset, we analyzed seven distinct prompt patterns to evaluate their effectiveness in reducing back-and-forth communication between developers and AI. Our findings highlight patterns such as ''Context and Instruction'' and ''Recipe'' as particularly effective in achieving high-quality outputs with minimal iterations. The study emphasizes the potential for prompt engineering to streamline developer-AI collaboration, providing practical insights into crafting prompts that balance precision, efficiency, and clarity.

Figures

Figures reproduced from arXiv: 2506.01604 by the authors.

**Figure 2.** Prompt Patterns and the associated Effectiveness Score for PR [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Prompt Patterns and the associated Effectiveness Score for Issues effectiveness with fewer prompts, indicating more efficient interaction patterns. The patterns are then sorted based on this combined score, providing a ranking of patterns that are both highly effective and efficient in terms of prompt usage. This analysis helps identify the most impactful patterns and their relationship with the number of prompts ne… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. Newman, and A. Ouni. How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations. In Proceedings of the 21st International Conference on Mining Software Repositories , pages 202– 206, 2024

work page 2024
[2]

Amatriain

X. Amatriain. Prompt design and engineering: Introduction and ad- vanced methods. arXiv preprint arXiv:2401.14423 , 2024

arXiv 2024
[3]

Chouchen, N

M. Chouchen, N. Bessghaier, M. Begoug, A. Ouni, E. Alomar, and M. W. Mkaouer. How do software developers use chatgpt? an ex- ploratory study on github pull requests. In Proceedings of the 21st International Conference on Mining Software Repositories , pages 212– 216, 2024

work page 2024
[4]

DePalma, I

K. DePalma, I. Miminoshvili, C. Henselder, K. Moss, and E. A. AlOmar. Exploring chatgpt’s code refactoring capabilities: An empirical study. Expert Systems with Applications , 249:123602, 2024

work page 2024
[5]

S. Dicuffa. Ssw625. https://github.com/sophiadicuffa/SSW625, 2024. GitHub Repository

work page 2024
[6]

T. B. B. et al. Language models are few-shot learners. page 75, 2020

work page 2020
[7]

X. W. et al. Chain-of-thought prompting elicits reasoning in large language models. page 43, 2022

work page 2022
[8]

Y . Z. et al. Large language models are human-level prompt engineers. page 43, 2024

work page 2024

Show all 13 references

[9]

Honovich, U

O. Honovich, U. Shaham, S. R. Bowman, and O. Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782 , 2022

2022 arXiv
[10]

Reynolds and K

L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. page 10, 2021

2021
[11]

Reynolds and K

L. Reynolds and K. McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems , pages 1–7, 2021

2021
[12]

T. Xiao, C. Treude, H. Hata, and K. Matsumoto. Devgpt: Studying developer-chatgpt conversations, 2024

2024
[13]

P. Yadav. Prompt patterns analysis. https://github.com/pyadav6/ SSW-627, 2024. GitHub Repository. 7

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

E. A. AlOmar, A. Venkatakrishnan, M. W. Mkaouer, C. Newman, and A. Ouni. How to refactor this code? an exploratory study on developer-chatgpt refactoring conversations. In Proceedings of the 21st International Conference on Mining Software Repositories , pages 202– 206, 2024

work page 2024

[2] [2]

Amatriain

X. Amatriain. Prompt design and engineering: Introduction and ad- vanced methods. arXiv preprint arXiv:2401.14423 , 2024

arXiv 2024

[3] [3]

Chouchen, N

M. Chouchen, N. Bessghaier, M. Begoug, A. Ouni, E. Alomar, and M. W. Mkaouer. How do software developers use chatgpt? an ex- ploratory study on github pull requests. In Proceedings of the 21st International Conference on Mining Software Repositories , pages 212– 216, 2024

work page 2024

[4] [4]

DePalma, I

K. DePalma, I. Miminoshvili, C. Henselder, K. Moss, and E. A. AlOmar. Exploring chatgpt’s code refactoring capabilities: An empirical study. Expert Systems with Applications , 249:123602, 2024

work page 2024

[5] [5]

S. Dicuffa. Ssw625. https://github.com/sophiadicuffa/SSW625, 2024. GitHub Repository

work page 2024

[6] [6]

T. B. B. et al. Language models are few-shot learners. page 75, 2020

work page 2020

[7] [7]

X. W. et al. Chain-of-thought prompting elicits reasoning in large language models. page 43, 2022

work page 2022

[8] [8]

Y . Z. et al. Large language models are human-level prompt engineers. page 43, 2024

work page 2024