Pith. sign in

REVIEW 2 major objections 6 minor 40 references

IDE-embedded AI tutor lifts prompt quality 14% in one hour

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 17:19 UTC pith:G36HPTJ6

load-bearing objection In-IDE Socratic tutor for prompt engineering shows promise, but the evaluation is circular: same model teaches and grades. the 2 major comments →

arxiv 2607.06074 v1 pith:G36HPTJ6 submitted 2026-07-07 cs.SE cs.AIcs.CYcs.HC

Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development

Rohit Mehra , Kapil Singi , Vikrant Kaulgud , Vibhu Saujanya Sharma , Swapnajeet Gon Choudhury , Swati Sharma , Adam P. Burden , Majd Sakr This is my paper
classification cs.SE cs.AIcs.CYcs.HC
keywords prompt engineeringagentic tutorSocratic methodIDE integrationLLM-as-judgesoftware engineering educationcode generationmulti-agent system
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Prompt Coach (PC), a multi-agent system that lives inside a developer's IDE and teaches prompt engineering through Socratic questioning rather than prescriptive fixes. PC evaluates each code-generation prompt across eight quality dimensions—clarity, specificity, context awareness, adaptability, inclusion of constraints, error handling, output requirements, and testability—then surfaces targeted questions that nudge the developer toward self-correction. The system grounds its guidance in two sources: the developer's own codebase (indexed into a vector store) and a hidden preview of what code the target LLM would actually generate from the current prompt, which lets PC identify concrete failure modes and ask about them specifically. A study with 15 professional developers found that a single 60-minute session with PC produced statistically significant improvements in prompt quality scores, with the largest gains in the dimensions developers most often neglect: specifying constraints, handling errors, and providing context.

Core claim

The central finding is that developers' prompt-writing proficiency does not scale with programming experience (r = 0.11), and that a brief, Socratic, in-IDE tutoring session can produce measurable improvements in the specific dimensions where developers are weakest. Developers naturally write clear prompts that communicate intent but systematically underspecify constraints, error handling, and contextual detail. PC's dimensional evaluation combined with consequence-grounded Socratic questioning targets exactly these blind spots, yielding a mean quality improvement from 63.04 to 71.69 (p < .001) after one session, with 13 of 15 participants improving and none declining meaningfully.

What carries the argument

The Socratic Guidance Agent, which transforms dimensional prompt-quality scores into context-aware pedagogical questions by combining two information sources: a project-specific knowledge store built from the developer's codebase, and a Consequence Preview Agent that silently runs the developer's prompt through the target LLM to identify what would go wrong, then asks the developer about those specific failure modes rather than telling them the answer.

Load-bearing premise

The paper uses the same model (GPT-4.1) as both the tutoring system and the evaluation judge, and the eight quality dimensions that PC teaches are the same dimensions the evaluator scores. This means the measured improvements partly reflect developers learning to include what PC's rubric rewards, as assessed by PC's own rubric—an independent human grader or a different evaluation model would be needed to confirm the gains are genuine rather than circular.

What would settle it

If an independent human expert or a different LLM family scored the pre- and post-session prompts using a different rubric, and the improvements disappeared or substantially shrank, the paper's core claim of measurable learning impact would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If Socratic in-flow tutoring generalizes beyond prompt engineering, the same architecture—dimensional evaluation plus consequence preview plus adaptive questioning—could be applied to other emerging software engineering skills that are interactive, context-dependent, and evolving too fast for static curricula.
  • The finding that prompt quality does not correlate with programming experience suggests that organizations cannot assume senior developers are better at AI collaboration; targeted skill development may be needed across all experience levels.
  • The dimensional scoring framework itself (eight quality dimensions with operational definitions) could serve as a shared vocabulary for organizations to assess and benchmark prompt-engineering proficiency independently of the tutoring tool.
  • The Consequence Preview Agent pattern—silently executing a user's input to identify failure modes, then asking about them rather than showing the result—may be a generalizable pedagogical technique for any skill where the gap between intent and outcome is best learned through guided discovery.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The measured gains may partly reflect convergence between what PC teaches and what PC measures: the same eight dimensions define both the tutoring guidance and the evaluation rubric, so improvement on those dimensions is expected even if the underlying skill transfer is partial. An independent rubric or human grader would be needed to confirm that the gains reflect genuine prompt-quality improveme
  • The 60-minute single-session design captures immediate learning effects but cannot distinguish between durable skill acquisition and short-term priming. Retention testing days or weeks later would determine whether the Socratic approach builds lasting intuition or temporary pattern-matching.
  • The study was conducted without real project context (using APPS benchmark tasks), which is precisely the context PC is designed to leverage. In real development settings with actual codebases, the contextual grounding could either amplify the learning effect (richer, more relevant nudges) or dilute it (noise from large, complex codebases).
  • If the Consequence Preview Agent's hidden execution reveals failure modes the developer would not otherwise discover, this raises the question of whether the learning transfers to scenarios where no such preview is available—i.e., whether developers internalize the habit of anticipating failure modes or become dependent on the system surfacing them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces Prompt Coach (PC), a multi-agent tutoring system embedded in the VSCode IDE that helps developers improve code-generation prompts through Socratic guidance. PC evaluates prompts across eight quality dimensions (Clarity, Specificity, Context Awareness, Adaptability, Inclusion of Constraints, Error Handling, Output Requirements, Testability), generates context-aware questions targeting weak dimensions, and previews LLM consequences internally to ground its nudges. An empirical study with 15 professional developers uses a single-arm, within-subjects pre/post design: participants write prompts for three APPS benchmark tasks, receive 60 minutes of PC tutoring, then write prompts for three new tasks. Results show a statistically significant overall improvement (mean 63.04 to 71.69, p<.001, paired Wilcoxon), with the largest gains in Inclusion of Constraints (+31.63%), Error Handling (+30.66%), and Context Awareness (+23.61%). Qualitative questionnaire data show strong perceived learning effectiveness, trust, and adoption readiness.

Significance. The paper addresses a timely and practical problem: teaching prompt engineering as an in-flow skill rather than through static materials. The system architecture is well-motivated, combining dimensional scoring, consequence preview, and developer modeling in a coherent multi-agent design. The empirical study provides both quantitative and qualitative measures with appropriate statistical tests (paired Wilcoxon, Holm-Bonferroni correction). The dimension-wise analysis identifying cognitive blind spots (constraints, error handling) is a useful finding for the community. The work is positioned appropriately for an Industry Showcase venue, where the system design and preliminary evidence are the primary contributions.

major comments (2)
  1. Section 4.1 states that post-study prompts were scored 'using an LLM-as-judge evaluation with the same model and prompting setup as the Prompt Evaluator Agent.' This creates a shared-bias loop: PC's Socratic Guidance Agent teaches developers to improve on the eight dimensions defined in the Dimension Registry (Section 3.1), and the post-study evaluation confirms improvement using the same model (GPT-4.1) and rubric. The largest gains occurring on the lowest-baseline dimensions (Inclusion of Constraints +31.63%, Error Handling +30.66%) are consistent with both genuine learning and a rubric-alignment artifact. The paper should explicitly acknowledge this circularity risk in Section 4.3 (Threats to Validity) and, at minimum, discuss how an independent human grader or a different model as judge would strengthen the claim.
  2. The single-arm pre/post design cannot attribute the 13.7% improvement specifically to PC versus the effect of spending 60 minutes reflecting on prompts. The abstract frames the result causally ('demonstrated statistically significant improvements'), but without a control condition (e.g., a group that receives generic prompt-engineering advice or simply reflects on prompts for 60 minutes), the design establishes pre/post change but not PC-specific causal impact. The authors acknowledge in Section 5 that 'comparative evaluations against traditional learning approaches' are planned for future work; the abstract and RQ2 framing should be softened to match what the design can establish.
minor comments (6)
  1. Section 3.1: The eight quality dimensions are described as 'representative' and 'commonly discussed in the literature.' A brief note on how these were selected or whether other dimensions were considered would strengthen the rubric's coverage.
  2. Section 4.1: The study mentions five phases with an average duration of approximately 2 hours 30 minutes, but the learning phase is 60 minutes and the three baselining tasks plus three post-learning tasks are 10 minutes each (60 minutes total). The remaining ~30 minutes presumably covers questionnaires and setup, but this should be made explicit.
  3. Table 1: The 'AI-Refined Description' column header could be misread as PC's output; a note clarifying that task descriptions were pre-processed by an LLM to remove extraneous detail (as described in Section 4.1) would help.
  4. Section 4.2, RQ1: The Pearson correlation (r=0.11) between experience and baseline score is reported without a p-value or confidence interval; given n=15, the statistical significance (or lack thereof) should be made explicit.
  5. Section 4.3: The threats to validity section does not mention the shared-model circularity or the absence of a control condition. These are the two most important threats and should be acknowledged.
  6. Data Availability Statement: The implementation and study data cannot be released due to organizational constraints. Consider releasing at minimum the evaluation rubric and scoring prompt, which would allow independent replication of the LLM-as-judge setup.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive review. Both major comments identify legitimate methodological limitations of our study design. We agree with both points and will revise the manuscript accordingly: (1) we will explicitly acknowledge the shared-bias/circularity risk in Threats to Validity and discuss how an independent human grader or alternative model-as-judge would strengthen the evaluation, and (2) we will soften the causal framing in the abstract and RQ2 to match what a single-arm pre/post design can establish, making clear that the study demonstrates pre/post change attributable to the overall intervention but cannot isolate PC-specific effects without a control condition. We provide detailed responses below.

read point-by-point responses

  1. Referee: Shared-bias loop: PC's Socratic Guidance Agent teaches developers to improve on the eight dimensions in the Dimension Registry, and post-study evaluation uses the same model (GPT-4.1) and rubric to confirm improvement. The largest gains on lowest-baseline dimensions (Constraints +31.63%, Error Handling +30.66%) are consistent with both genuine learning and a rubric-alignment artifact. The paper should acknowledge this circularity risk in Section 4.3 and discuss how an independent human grader or different model as judge would strengthen the claim.

    Authors: The referee is correct. There is a genuine circularity risk in our design: because the Prompt Evaluator Agent defines the eight quality dimensions and generates the Socratic guidance that targets weak dimensions, and the post-study evaluation uses the same model (GPT-4.1) and the same rubric to score improvement, the observed gains could partly reflect alignment to PC's own evaluation criteria rather than purely independent quality improvement. We agree this should be explicitly acknowledged. In the revised manuscript, we will add a dedicated paragraph in Section 4.3 (Threats to Validity) that names this shared-bias risk, explains why the pattern of largest gains on lowest-baseline dimensions is consistent with both genuine learning and a rubric-alignment artifact, and discusses how an independent human grader or a different LLM-as-judge (e.g., Claude or Gemini with an independently constructed rubric) would provide a stronger, decoupled evaluation. We will also note this as a priority for our planned future work. We want to be transparent that we cannot fully resolve this concern with the current study data, as no independent grading was conducted. However, we would note two points that partially mitigate (though do not eliminate) the concern: (1) the eight dimensions are drawn from established prompt-engineering literature (cited in Section 3.1), not invented ad hoc by PC, so the rubric reflects community-recognized quality criteria rather than an arbitrary internal standard; and (2) the qualitative questionnaire results (Section 4.2, RQ3) provide independent evidence that participants perceived genuine cognitive gains, including increased elaboration and confidence, though we acknowledge these are self-reported and also not immune to bias. These mitigations will be纳入 revision: yes

  2. Referee: Single-arm pre/post design cannot attribute the 13.7% improvement specifically to PC versus the effect of 60 minutes of reflecting on prompts. The abstract frames the result causally ('demonstrated statistically significant improvements'), but without a control condition the design establishes pre/post change but not PC-specific causal impact. The abstract and RQ2 framing should be softened to match what the design can establish.

    Authors: The referee is correct. Our single-arm, within-subjects pre/post design can establish that participants improved from pre to post, but it cannot isolate the causal contribution of PC specifically versus the effect of spending 60 minutes reflecting on prompt construction more generally. A control condition (e.g., a group receiving generic prompt-engineering advice or simply reflecting on prompts for 60 minutes without PC) would be needed to attribute the improvement to PC's tutoring mechanism. We will revise the manuscript accordingly. Specifically: (1) In the abstract, we will reframe the result from 'demonstrated statistically significant improvements' to language that makes clear we observed statistically significant pre/post improvements following a single PC session, without implying PC-specific causal attribution. (2) In RQ2, we will adjust the framing from 'measured learning impact of PC' to 'measured pre/post change in prompt-engineering proficiency following a PC learning session,' and we will add an explicit note in the RQ2 results discussion that the design cannot distinguish PC-specific effects from the general effect of structured reflection time. (3) In Section 4.3 (Threats to Validity), we will add the absence of a control condition as a named internal validity threat. (4) In Section 5, we already note that comparative evaluations against traditional learning approaches are planned for future work; we will strengthen this to explicitly state that such comparisons are necessary to establish PC-specific causal impact. We agree with the referee that the current framing overstates what the design can support, and we appreciate the precise guidance on where and how to adjust it. revision: yes

Circularity Check

2 steps flagged

PC teaches to its own rubric and then grades with the same model and rubric, making the measured gains partly self-confirming.

specific steps
  1. fitted input called prediction [Section 4.1 (evaluation methodology) and Section 3.1 (Dimension Registry / Socratic Guidance Agent)]
    "Prompts collected during the baselining and post-learning phases were subsequently scored across the eight quality dimensions defined in Section 3, using an LLM-as-judge evaluation with the same model and prompting setup as the Prompt Evaluator Agent."

    The Prompt Evaluator Agent defines eight quality dimensions and scores prompts. The Socratic Guidance Agent then nudges developers to improve on those exact dimensions. The post-study evaluation reuses 'the same model and prompting setup as the Prompt Evaluator Agent' to score the improved prompts. This creates a closed loop: PC trains developers toward its own rubric, then measures success against that same rubric with the same model. The largest gains occurring on the lowest-baseline dimensions (Inclusion of Constraints +31.63%, Error Handling +30.66%) is consistent with both genuine learning and a rubric-alignment artifact, and the study design cannot distinguish these. The 'prediction' (improved scores) is partly forced by construction because the tutoring target and the evaluation rub

  2. self definitional [Section 3.1 (Socratic Guidance Agent) and Section 3.2 (Implementation)]
    "Prompt evaluation and guidance are powered by GPT-4.1 from OpenAI, which serves both as the target LLM and as the underlying model for LLM-as-a-judge and consequence preview mechanisms."

    GPT-4.1 serves simultaneously as (1) the Prompt Evaluator Agent that defines and scores the eight quality dimensions, (2) the Socratic Guidance Agent that nudges developers toward improving those same dimensions, (3) the Consequence Preview Agent, and (4) the post-study evaluation judge. The tutoring system's scoring rubric directly shapes what PC teaches; the post-study scoring reuses that same rubric and model. PC trains developers to include constraints, error handling, and context awareness (its rubric dimensions), then confirms improvement by scoring on those exact dimensions with the same model. The measured gains are partly self-confirming by construction.

full rationale

The paper's central quantitative claim — statistically significant improvement in prompt quality — is measured by an evaluation instrument (GPT-4.1 as LLM-as-judge, eight-dimension rubric) that is identical to the tutoring instrument (GPT-4.1 as Prompt Evaluator Agent and Socratic Guidance Agent, same eight dimensions). PC teaches developers to improve on the dimensions its evaluator scores, then the evaluator confirms improvement on those same dimensions. This is a genuine circularity: the measurement tool is the teaching tool. The paper does not use an independent human grader or a different model as judge. However, the circularity is partial rather than total: developers wrote new prompts on different tasks for the post-test, PC was revoked during post-testing, and two participants showed no improvement — so the gains are not entirely forced by construction. The absence of a control condition (no generic-advice or reflection-only group) means practice effects are also confounded with PC's specific tutoring. The paper acknowledges limitations in Section 4.3 but does not acknowledge the shared-bias circularity. Score 6 reflects that one or more 'predictions' (the dimension-wise gains) reduce partly by construction to the tutoring rubric, while some independent signal (new tasks, PC revoked, non-improvers) prevents a score of 8+.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper introduces a system whose evaluation depends on four key fitted/chosen parameters (dimension set, judge prompt, guidance prompt, task selection) and relies on three domain assumptions that are not independently validated. The most critical is the LLM-as-judge axiom: the same model tutors and grades, and no human validation of scores is provided. The invented entities (PC system, learner profile) have no independent evidence beyond the study itself.

free parameters (4)
  • Eight quality dimensions (Clarity, Specificity, Context Awareness, Adaptability, Inclusion of Constraints, ErrorHandling = 8 dimensions selected by authors
    Chosen by the authors as 'commonly discussed in the literature' (Section 3.1). No empirical justification for this specific set over alternatives; the dimensions directly determine what PC teaches and what the evaluator scores.
  • LLM-as-judge scoring prompt and rubric = Not specified
    The exact prompting setup used by the Prompt Evaluator Agent to assign 0-100 scores is not provided. This prompt determines all quantitative results but is treated as a fixed black box.
  • Socratic guidance generation prompt = Not specified
    The prompt template that transforms dimensional scores into Socratic questions is not provided. This is the core tutoring mechanism.
  • Task selection (APPS IDs 4987, 1961, 2207, 2815, 1924, 2016) = 6 specific APPS tasks
    Selected by the authors across three complexity levels. The specific tasks chosen affect baseline and post-learning scores. No randomization or justification for these particular tasks is given beyond complexity level.
axioms (4)
  • domain assumption LLM-as-judge evaluation with GPT-4.1 produces reliable and valid prompt-quality scores.
    Section 4.1 states prompts were scored 'using an LLM-as-judge evaluation with the same model and prompting setup as the Prompt Evaluator Agent.' No human validation of the judge's scores is provided. All quantitative results depend on this assumption.
  • domain assumption The eight selected quality dimensions collectively capture prompt quality for code generation.
    Section 3.1 states these dimensions were 'chosen because they are commonly discussed in the literature.' No factor analysis or empirical validation of completeness is provided.
  • domain assumption Improvement on prompt-quality scores reflects genuine skill acquisition rather than rubric-specific optimization.
    The paper claims PC 'improved their prompt-writing skills' (abstract), but the evidence shows improvement on PC's own rubric. The assumption that this transfers to independent quality measures is unstated.
  • domain assumption A 60-minute session is sufficient to produce measurable learning effects.
    Section 4.1 fixes the learning phase at 60 minutes due to 'limited participant availability.' The paper does not test whether shorter or longer sessions produce different patterns.
invented entities (2)
  • Prompt Coach multi-agent system no independent evidence
    purpose: Agentic tutor for in-IDE prompt engineering learning
    PC is a new system built for this paper. The evaluation is the only evidence of its effectiveness, and the evaluation has the shared-bias problem described above. No external deployment or independent replication exists.
  • Developer Modeling and Tracking Agent's learner profile no independent evidence
    purpose: Personalizes guidance based on observed developer behavior
    The paper describes this agent's function (Section 3.1) but provides no evaluation of whether the learner profile actually improves guidance quality or learning outcomes. No ablation comparing personalized vs. non-personalized guidance is reported.

pith-pipeline@v1.1.0-glm · 14642 in / 3246 out tokens · 504930 ms · 2026-07-08T17:19:45.499438+00:00 · methodology

0 comments
read the original abstract

Prompt engineering has emerged as a critical yet undertaught skill for software developers, one that traditional learning approaches are ill-equipped to support given its evolving, interactive, and context-dependent nature. In this paper, we introduce Prompt Coach (PC), an agentic tutor that helps developers learn how to craft high-quality code-generation prompts through Socratic guidance embedded in-flow within their IDE. PC evaluates prompt quality across multiple dimensions and surfaces targeted questions to guide self-correction, grounded in the developer's codebase and the behavior of the target LLM. We present an early empirical study with 15 professional developers combining quantitative prompt quality scoring with qualitative perception measures. Participants showed statistically significant improvements after a single 60-minute session, with the largest gains across dimensions commonly overlooked by developers. They also reported strong trust, high adoption readiness, and unanimous agreement that PC improved their prompt-writing skills.

Figures

Figures reproduced from arXiv: 2607.06074 by Adam P. Burden, Kapil Singi, Majd Sakr, Rohit Mehra, Swapnajeet Gon Choudhury, Swati Sharma, Vibhu Saujanya Sharma, Vikrant Kaulgud.

Figure 1
Figure 1. Figure 1: Prompt Coach architecture: An agentic tutor for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Prompt Coach in action inside the VSCode IDE for an example prompt. (a) Dimensional assessment of the developer’s [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · 3 internal anchors

  1. [1]

    Erfan Al-Hossami, Razvan Bunescu, Justin Smith, and Ryan Teehan. 2024. Can Language Models Employ the Socratic Method? Experiments with Code Debug- ging. InProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024). Association for Computing Machinery, New York, NY, USA, 53–59. doi:10.1145/3626252.3630799

    work page doi:10.1145/3626252.3630799 2024

  2. [2]

    Mamdouh Alenezi and Mohammed Akour. 2025. AI-Driven Innovations in Software Engineering: A Review of Current Practices and Future Directions. Applied Sciences15, 3 (2025). doi:10.3390/app15031344

    work page doi:10.3390/app15031344 2025

  3. [3]

    Anthropic. 2024. Prompt Engineering Interactive Tutorial. https://github.com/ anthropics/prompt-eng-interactive-tutorial Accessed: 2026-04-08

    work page 2024

  4. [4]

    Anthropic. 2026. Prompt Improver. https://console.anthropic.com Accessed: 2026-04-08

    work page 2026

  5. [5]

    Ian Arawjo, Chelse Wang, Andrew McNutt, Ryo Suzuki, Erik Argyle, Caleb Singh, and Bjoern Hartmann. 2024. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. InProceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA. doi:10.1145/3613904.3642016

    work page doi:10.1145/3613904.3642016 2024

  6. [6]

    James, and Nadia Polikarpova

    Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models.Proceedings of the ACM on Programming Languages7, OOPSLA (2023), 85–111. doi:10.1145/ 3586030

    work page 2023

  7. [7]

    Zhang, and Yang Liu

    Zhenpeng Chen, Chong Wang, Weisong Sun, Xuanzhe Liu, Jie M. Zhang, and Yang Liu. 2026. Promptware Engineering: Software Engineering for Prompt-Enabled Systems.ACM Trans. Softw. Eng. Methodol.(Feb. 2026). doi:10.1145/3796535

    work page doi:10.1145/3796535 2026

  8. [8]

    Chroma. 2026. Chroma: The Open-Source Embedding Database. https://www. trychroma.com Accessed: 2026-04-08

    work page 2026

  9. [9]

    CrewAI. 2026. CrewAI: Fast and Flexible Multi-Agent Automation Framework. https://github.com/crewAIInc/crewAI Accessed: 2026-04-08

    work page 2026

  10. [10]

    Sophia DiCuffa, Amanda Zambrana, Priyanshi Yadav, Sashidhar Madiraju, Khushi Suman, and Eman Abdullah AlOmar. 2025. Exploring Prompt Patterns in AI- Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration. In2025 IEEE 4th International Conference on Computing and Ma- chine Intelligence (ICMI). 1–7. doi:10.1109/ICMI65310.2025...

    work page doi:10.1109/icmi65310.2025.11141320 2025

  11. [11]

    "My toxic trait is thinking I'll remember this": gaps in the learner experience of video tutorials for feature-rich software

    Ian Drosos, Advait Sarkar, and Andrew D. Gordon. 2024. "My toxic trait is thinking I’ll remember this": gaps in the learner experience of video tutorials for feature-rich software. arXiv:2404.07114 [cs.HC] https://arxiv.org/abs/2404.07114

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Denis Federiakin, Dimitri Molerov, Olga Zlatkin-Troitschanskaia, and Andreas Maur. 2024. Prompt engineering as a new 21st century skill.Frontiers in Education Volume 9 - 2024 (2024). doi:10.3389/feduc.2024.1366434

    work page doi:10.3389/feduc.2024.1366434 2024

  13. [13]

    Gartner. 2024. Gartner Says Generative AI will Require 80% of Engineering Work- force to Upskill Through 2027. https://www.gartner.com/en/newsroom/press- releases/2024-10-03-gartner-says-generative-ai-will-require-80-percent-of- engineering-workforce-to-upskill-through-2027 Accessed: 2026-04-08

    work page 2024

  14. [14]

    GitHub. 2025. The new identity of a developer: What changes and what doesn’t in the AI era. https://github.blog/news-insights/octoverse/the-new-identity-of-a- developer-what-changes-and-what-doesnt-in-the-ai-era/ Accessed: 2026-04-08

    work page 2025

  15. [15]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1. https:/...

    work page 2021

  16. [16]

    Lee, Joyce Malyn-Smith, Beatriz Perret, Vikram Tiwari, Joshua Kenitzer, Andrew Macvean, and Erin Barrar

    Matthew Kam, Cody Miller, Miaoxin Wang, Abey Tidwell, Irene A. Lee, Joyce Malyn-Smith, Beatriz Perret, Vikram Tiwari, Joshua Kenitzer, Andrew Macvean, and Erin Barrar. 2025. What do professional software developers need to know to succeed in an age of Artificial Intelligence?. InProceedings of the 33rd ACM International Conference on the Foundations of So...

    work page doi:10.1145/3696630.3727251 2025

  17. [17]

    Priyanka Kargupta, Ishika Agarwal, Dilek Hakkani Tur, and Jiawei Han. 2024. Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Question- ing for Socratic Code Debugging. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguist...

    work page 2024

  18. [18]

    Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, and Gregorio Ponti

    work page

  19. [19]

    doi:10.1038/s41598-025-97652-6

    AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting.Scientific Reports 15, 1 (3 6 2025), 17458. doi:10.1038/s41598-025-97652-6

    work page doi:10.1038/s41598-025-97652-6 2025

  20. [20]

    Georgios Kostopoulos, Vasileios Gkamas, Maria Rigou, and Sotiris Kotsiantis

    work page

  21. [21]

    doi:10.1109/ACCESS.2025.3620473

    Agentic AI in Education: State of the Art and Future Directions.IEEE Access13 (2025), 177467–177491. doi:10.1109/ACCESS.2025.3620473

    work page doi:10.1109/access.2025.3620473 2025

  22. [22]

    Stefan Kuchemann, Karina E. Avila, Yavuz Dinc, Chiara Hortmann, Natalia Revenga, Verena Ruf, Niklas Stausberg, Steffen Steinert, Frank Fischer, Mar- tin Fischer, Enkelejda Kasneci, Gjergji Kasneci, Thomas Kuhr, Gitta Kutyniok, Sarah Malone, Michael Sailer, Albrecht Schmidt, Matthias Stadler, Jochen Weller, and Jochen Kuhn. 2025. On opportunities and chall...

    work page doi:10.1038/s41539-025-00301-w 2025

  23. [23]

    Angélique Létourneau, Marion Deslandes Martineau, Patrick Charland, John Alexander Karran, Jared Boasen, and Pierre Majorique Léger. 2025. A systematic review of AI-driven intelligent tutoring systems (ITS) in K-12 educa- tion.npj Science of Learning10, 1 (14 5 2025), 29. doi:10.1038/s41539-025-00320-7

    work page doi:10.1038/s41539-025-00320-7 2025

  24. [24]

    Ziyou Li, Agnia Sergeyuk, and Maliheh Izadi. 2025. Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3346–3356. doi:10.1109/ASE63991.2025.00276

    work page doi:10.1109/ase63991.2025.00276 2025

  25. [25]

    Ben Liu, Jihai Zhang, Fangquan Lin, Xu Jia, and Min Peng. 2025. One Size doesn’t Fit All: A Personalized Conversational Tutoring Agent for Mathematics Instruction. InCompanion Proceedings of the ACM on Web Conference 2025(Sydney NSW, Australia)(WWW ’25). Association for Computing Machinery, New York, NY, USA, 2401–2410. doi:10.1145/3701716.3717527

    work page doi:10.1145/3701716.3717527 2025

  26. [26]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Development and LLM-based Code Generation. InProceedings of the 39th IEEE/ACM Interna- tional Conference on Automated Software Engineering(Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 1583–1594. doi:10.1145/3691620.3695527

    work page doi:10.1145/3691620.3695527 2024

  27. [27]

    Microsoft. 2026. Microsoft Azure Cloud Computing Platform. https://azure. microsoft.com Accessed: 2026-04-08

    work page 2026

  28. [28]

    Microsoft. 2026. Visual Studio Code. https://code.visualstudio.com Accessed: 2026-04-08

    work page 2026

  29. [29]

    Alessandro Midolo, Alessandro Giagnorio, Fiorella Zampetti, Rosalia Tufano, Gabriele Bavota, and Massimiliano Di Penta. 2026. Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization. arXiv:2601.13118 [cs.SE] https://arxiv.org/abs/2601.13118

    work page arXiv 2026

  30. [30]

    OpenAI. 2024. Prompt Engineering Guide. https://platform.openai.com/docs/ guides/prompt-engineering Accessed: 2026-04-08

    work page 2024

  31. [31]

    OpenAI. 2026. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/ Accessed: 2026-04-08

    work page 2026

  32. [32]

    OpenAI. 2026. Prompt Optimizer. https://platform.openai.com Accessed: 2026- 04-08

    work page 2026

  33. [33]

    Seetaram Rao Rayarao, Nilesh Dnyaneshwar Bhandarwar, Rahul Dattangire, and Sudhakar Vuriti. 2025. An Autonomous Agentic Framework for Interactive Code Mentoring and Error Resolution using Large Language Models. In2025 Modern Electronics Devices and Intelligent Communication Systems (MEDCOM). 45–51. doi:10.1109/MEDCOM67532.2025.11404910

    work page doi:10.1109/medcom67532.2025.11404910 2025

  34. [34]

    Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, and Xiaohu Yang. 2024. From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering(Echternach, Luxembourg)(ASE ’23). IEEE Press, 976–987. doi:10.1109/ASE56229.2023.00143

    work page doi:10.1109/ase56229.2023.00143 2024

  35. [35]

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, et al. 2024. The Prompt Report: A Systematic Survey of Prompting Techniques. arXiv:2406.06608 [cs.AI] https://arxiv.org/abs/2406.06608

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Díaz Ferreyra, Markus Mutas, Salem Dhif, and Ric- cardo Scandariato

    Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhif, and Ric- cardo Scandariato. 2025. Prompting Techniques for Secure Code Generation: A Systematic Investigation.ACM Trans. Softw. Eng. Methodol.34, 8, Article 225 (Oct. 2025), 53 pages. doi:10.1145/3722108 Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Enginee...

    work page doi:10.1145/3722108 2025

  37. [37]

    Christoph Treude and Margaret-Anne Storey. 2025. Generative AI and Empirical Software Engineering: A Paradigm Shift . In2025 2nd IEEE/ACM International Con- ference on AI-powered Software (AIware). IEEE Computer Society, Los Alamitos, CA, USA, 233–239. doi:10.1109/AIware69974.2025.00033

    work page doi:10.1109/aiware69974.2025.00033 2025

  38. [38]

    An Vu and Jonas Oppenlaender. 2026. Prompt Engineer: Analyzing Hard and Soft Skill Requirements in the AI Job Market. arXiv:2506.00058 [cs.CY] https: //arxiv.org/abs/2506.00058

    work page arXiv 2026

  39. [39]

    Tianjia Wang, Matthew Trimble, and Chris Brown. 2025. DevCoach: Support- ing Students Learning the Software Development Life Cycle with a Generative AI powered Multi-Agent System. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering(Clarion Hotel Trondheim, Trondheim, Norway)(FSE Companion ’25). Association for...

    work page doi:10.1145/3696630.3727255 2025

  40. [40]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] https://arxiv.org/abs/2302.11382

    work page internal anchor Pith review Pith/arXiv arXiv 2023