Pith. sign in

REVIEW 2 major objections 7 minor 1 cited by

Understanding Prompt Programming Tasks and Questions

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt programming is still a manual craft: 16 of 51 key questions have no tool support.

desk verdict Real taxonomy, shaky tool-gap numbers: the 16-questions claim outruns the reliability of the documentation-only coding. read the letter →

arxiv 2507.17264 v1 pith:ALHYN7WT submitted 2025-07-23 cs.SE cs.AIcs.HC

Jenny T. Liang , Chenyang Yang , Agnia Sergeyuk , Travis D. Breaux , Brad A. Myers This is my paper
classification cs.SEcs.AIcs.HC
keywords promptprogrammingdeveloperinformationneedsengineeringtoolstaxonomyempiricalsoftwarelargelanguagemodelstoolevaluationdebugging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt programming, building software that embeds natural-language prompts to foundation models, is a trial-and-error craft, yet nobody had catalogued what developers actually need to know while iterating on a prompt. This paper builds that catalogue: 25 tasks and 51 questions, gathered from interviews, direct observation of prompt changes, and a survey, then ranked by how frequent and how helpful each would be to answer. The paper then checks how many of those questions 48 research and commercial tools answer. Its finding is the central claim: every one of the 25 tasks still required manual work in the study, and 16 of the 51 questions, including four of the six highest-ranked ones, are answered by no tool at all. If this is right, tool builders now have a concrete, ranked checklist of the information needs that matter most to prompt programmers.

What carries the argument

The central instrument is the taxonomy of 25 prompt programming tasks and 51 questions, organized into eight activity themes. It is built by coding 16 interviews and 8 observational sessions of developers making real prompt changes, refined with survey feedback, and then validated and quantified with a survey of 50 prompt programmers. The importance measure pairs frequency, how often a developer encounters the need, with helpfulness, how much answering it would help, and a question counts as unsupported only if no tool in the 48-tool sample addresses it.

What would settle it

The central claim would be overturned by a direct audit of a larger, independently sampled set of prompt programming tools: if more than two of the six highest-ranked questions turned out to be answered by tools outside the 48 examined, or if a representative sample of prompt programmers rated the 16 supposedly unanswered questions as rarely needed, then the conclusion that the most-valued needs are systematically unsupported would collapse.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that prompt programming today is an unsupported manual craft, and that the structural reason is visible in its taxonomy: the questions developers ask when changing a prompt cluster into eight themes, from understanding a version's content and behavior, through debugging, to comparing, retrieving, and remembering versions. Existing tools cover only the surface questions within each theme. The two highest-ranked questions are which parts of a prompt are logically related to the part being examined, and how representative the test examples are; neither is answered by any of the 48 tools examined. The paper therefore positions the taxonomy itself as the contribution: a reusable, validated instrument that tool builders can use to decide which prompt-programming needs to address, and a baseline for evaluating future tools.

Load-bearing premise

The load-bearing premise is that the importance rankings from the 42 survey respondents who passed attention checks, recruited partly through one developer-tools company's panel and snowball sampling, represent prompt programmers in general; if that sample is unrepresentative, the claim that the highest-ranked questions are unsupported may not hold for the broader population.

Editorial extensions

If this is right

  • Tool builders now have a ranked list of 51 questions to prioritize, starting with prompt-component relationships, example representativeness, and fault localization by content and by change.
  • The unanswered questions among the top-ranked ones, such as whether test examples are representative and where in the code a prompt depends, mark concrete openings for new prompt-development tools.
  • Research tools that emphasize visualization and editing should be complemented by support for code-dependency tracking and dataset-quality assessment.
  • Evaluation of prompt-programming tools can be structured around coverage of the taxonomy instead of ad hoc feature checklists.
  • The taxonomy provides a baseline for comparing future prompt tools against current coverage, since the same questions can be re-audited after new tools appear.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the rankings generalize, the most underserved needs are the ones that cross boundaries between prompt text, code, and data, so the highest-leverage tools may be those that link these three artifacts together.
  • A testable extension would be to turn the 51-question taxonomy into a benchmark: for a given prompt-change scenario, measure whether a tool can answer each question, with human-verified ground truth rather than documentation review.
  • The taxonomy could also serve as a prompt-programming curriculum checklist, since it exposes why trial-and-error dominates: the questions programmers most want answered are exactly the ones tools and training do not address.
  • Because several of the unanswered questions involve comparing prompt versions and recalling why changes were made, prompt tools may benefit from borrowing version-history and design-rationale mechanisms from traditional software development environments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper develops a taxonomy of 25 prompt programming tasks and 51 questions through interviews with 16 developers, observations of 8 developers, and a survey of 50 developers. It applies the taxonomy to 48 research and commercial tools, reporting that 16 of the 51 questions—including four of the six most important—are unsupported. The paper concludes with five opportunities for tool builders. The central claim is that prompt programming is not well supported and that current tools systematically miss the most important information needs.

Significance. If the findings hold, the taxonomy is a valuable instrument for designing and evaluating prompt programming tools, and the gap analysis provides actionable direction. The study is well-motivated and uses a sensible mixed-method design with triangulation across interviews, observation, survey, and tool review; the taxonomy is grounded in new participant data rather than derived from the tool evaluation, so the gap analysis is not circular. The authors also achieve code saturation and provide supplemental materials. However, the strength of the headline quantitative claims depends on two links—the reliability of the tool-coverage coding and the representativeness of the importance ratings—both of which need strengthening before the claims can be taken at face value.

major comments (2)
  1. [Section 3.4 Analysis; Section 4.3; Table 1] The headline claim that 16 of the 51 questions, including four of the six most important, are unanswered (Section 4.3 Key Findings) is computed from the Phase 4 tool-coverage codes. The inter-rater reliability check reported in Section 3.4 is limited to a task-level Cohen's kappa of 0.80 on five research tools; the 19 commercial tools were coded by a single author from documentation, with no reliability check, and question-level agreement is never reported. Documentation review can understate tool capabilities, especially for commercial tools with rapidly evolving features (e.g., LangSmith, PromptFlow, MLflow). Since the count of 16 unsupported questions is the paper's central quantitative result, the authors should provide question-level reliability evidence covering both research and commercial tools, or at minimum hands-on verification of a subset of the unsupported high-importance questions (Q3, Q13, Q22, Q10); otherwise the claim should be reworded as a documentation-based assessment.
  2. [Section 3.3 Analysis; Table 1; Table 2] The paper states that items with over 15 responses were considered to reduce noise (Section 3.3 Analysis), but it does not report the number of responses underlying any of the frequency/helpfulness percentages in Table 1 or Table 2. With a total of 50 survey responses and skip logic that only asks questions for tasks rated frequent/helpful, some questions may have N near the 15 threshold; the 95% confidence intervals for a percentage based on 15–25 responses are very wide. The 'most important' rankings in Table 2, and the derived claim that a majority of the most important questions are unsupported, are thus more fragile than the presentation suggests. Please report N per question and, ideally, confidence intervals or a sensitivity analysis.
minor comments (7)
  1. [Abstract; Section 1] The abstract's claim that 'all tasks are done manually' is imprecise: Table 1 shows tool coverage for many of these tasks (e.g., T14, T2), and the intended meaning is that participants in the study performed the tasks by hand even where tools exist. This phrasing overstates the gap and invites misreading; please rephrase to 'participants performed all tasks manually in our observations, despite some tool coverage.'
  2. [Table 1 caption] The caption appears to swap the two constructs: it says 'Frequency bars show the percent of Extremely/Very helpful (left) vs. Slightly/Not helpful (right) task ratings' and 'Helpfulness bars show % of Always/Often (left) vs. Rarely/Never (right) ratings,' but the columns are labeled Frequency and Helpfulness; the descriptions should be reversed.
  3. [Section 4.2] The text refers to 'Knowing what was tried before (Q25, f=66%, h=74%)' and 'tracking progress (Q24, f=67%, h=69%)' when discussing tasks; these should reference tasks T25 and T24, respectively, since Q24 and Q25 are different questions in the taxonomy.
  4. [Section 4.3.4] The sentence 'Only fault localization via prompt content (T10, Q10, 2×)' should cite Q20, not Q10; Q10 is the code dependency question under T5, whereas Q20 is the prompt-content fault-localization question under T10.
  5. [Section 4.3.5] The paragraph on change understanding cites T17 and T18 for content changes and behavior changes, but the correct task numbers are T15 and T16; the paragraph later correctly refers to T17 for design rationale, so this appears to be an off-by-one typo.
  6. [Figure 1] The opportunity label 'Fault localization to a part of the code (Q24) or external artifacts (Q24)' lists Q24 twice; the second should be Q25, which is the question about examples, context, or model input.
  7. [ACM Reference Format] The reference format block still contains the placeholder conference year '2018' and placeholder titles; update these to the actual submission venue and year.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the 'helpfulness' construct is defined as importance for tool-unaddressed questions, which inflates the headline that most-important questions are unsupported.

  1. self definitional [Section 3.2 (construct definition) and Section 4.3 Key Findings (tool coverage claim)]
    "The importance of a taxonomy task or question was measured with Likert scales for two constructs from Myers et al. [47]: frequency to represent importance in current practices, and helpfulness to represent importance for tasks and questions not addressed by existing tools. ... Key Findings (RQ3): Research and commercial tools cover 22 of the 25 tasks. Yet, 16 of the 51 questions, including four of the six most important ones, are not covered by existing tools."

    The paper defines the helpfulness dimension as 'importance for tasks and questions not addressed by existing tools.' It then ranks questions as 'most important' by averaging frequency and helpfulness (Table 2) and uses this ranking to report that a majority of the most important questions are unanswered by tools. Because the helpfulness ratings are explicitly framed as capturing importance for tool-unaddressed needs, the finding that top-helpfulness questions tend to be unsupported is partly built into the construct definition. The circularity is partial: frequency ratings and the independent tool-coverage coding provide non-tautological content, and not all top-ranked questions are unsupported (e.g., Q20 and Q7 have some tool support).

full rationale

This paper is an empirical taxonomy study with no fitted parameters or equations, so most circularity patterns do not apply. The taxonomy is derived from interviews, observations, and surveys, and the tool-coverage analysis is a separate coding exercise. The main circularity concern is the definition of the helpfulness construct in Section 3.2: participants are told that helpfulness means importance for tasks and questions not addressed by existing tools, and the paper subsequently uses 'most important' (frequency plus helpfulness) to claim that the most important questions remain unsupported. This is a partial self-definitional overlap, though not a full reduction because importance also includes frequency and because the tool-coverage coding is empirically independent and does not perfectly align with helpfulness ratings. The paper's reliance on the authors' prior definition of prompt programs (Liang et al. [39]) and on that prior work's curated tool list is disclosed and is not load-bearing for the central claim: the taxonomy, importance ratings, and tool coverage are all newly collected or independently coded. Overall, the central contribution is not circular by construction, but the 'most important unsupported questions' headline is partly shaped by the way helpfulness was defined, warranting a score of 4.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical paper with no free parameters, fitted values, or invented entities. The central claim rests on domain assumptions about the validity of self-report, sample representativeness, and documentation-based tool coding, which are acknowledged in the paper's threats to validity.

assumptions (4)
  • domain assumption Prompt programs are correctly defined by Liang et al. [39] and this definition is appropriate for participant inclusion.
    All inclusion criteria use this definition; if the definition is too narrow or broad, the taxonomy may miss or overinclude tasks. Invoked in Section 3.1 Sampling Strategy and Section 3.3 Sampling Strategy.
  • domain assumption Self-reported frequency and helpfulness ratings reflect actual importance of tasks and questions.
    The importance rankings (RQ2) rely on Likert ratings; memory bias and hypothetical 'magical system' prompts may not match real behavior. Invoked in Section 3.2 and Section 3.3 Protocol.
  • domain assumption The samples (16 interviews, 8 observations, 50 surveys) are sufficiently representative to generalize the taxonomy and importance rankings.
    The paper acknowledges small sample sizes and convenience/snowball sampling; this is the main load-bearing assumption for generalizability. Invoked in Section 5 External Validity.
  • domain assumption Coding tool capabilities from documentation accurately captures which questions a tool answers.
    Commercial tools were coded from documentation, and research tools from papers; actual features may differ from documented ones. Invoked in Section 3.4 Analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Prompt Programming Tasks and Questions." pith.science (2026). https://pith.science/paper/ALHYN7WT

@misc{pith2026250717264,
  author       = {Pith},
  title        = {Pith review of: Understanding Prompt Programming Tasks and Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ALHYN7WT}},
  note         = {Machine review of arXiv:2507.17264}
}
read the original abstract

Prompting foundation models (FMs) like large language models (LLMs) have enabled new AI-powered software features (e.g., text summarization) that previously were only possible by fine-tuning FMs. Now, developers are embedding prompts in software, known as prompt programs. The process of prompt programming requires the developer to make many changes to their prompt. Yet, the questions developers ask to update their prompt is unknown, despite the answers to these questions affecting how developers plan their changes. With the growing number of research and commercial prompt programming tools, it is unclear whether prompt programmers' needs are being adequately addressed. We address these challenges by developing a taxonomy of 25 tasks prompt programmers do and 51 questions they ask, measuring the importance of each task and question. We interview 16 prompt programmers, observe 8 developers make prompt changes, and survey 50 developers. We then compare the taxonomy with 48 research and commercial tools. We find that prompt programming is not well-supported: all tasks are done manually, and 16 of the 51 questions -- including a majority of the most important ones -- remain unanswered. Based on this, we outline important opportunities for prompt programming tools.

Figures

Figures reproduced from arXiv: 2507.17264 by the authors.

Figure 1
Figure 1. An overview of opportunities (­) for prompt pro￾gramming tools, based on taxonomy themes (#), tasks (T#), and questions (Q#) that lack support in existing tools. 1 Introduction Foundation models (FMs), such as large language models (LLMs) like GPT-4 [5] and vision models like DALL-E [9], have enabled new forms of intelligent software. AI capabilities once limited to laborious fine-tuning (e.g., summarization, questi… view at source ↗
Figure 2
Figure 2. A subset of the interview questions. The complete [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

    cs.HC 2025-10 conditional novelty 6.0 of 10

    Agent prototyping for non-experts requires scaffolds for scoping the agent, designing its chat/UI display, defining user interactions, running it, and debugging its runtime behavior.

Reference graph

Works this paper leans on

70 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    ChatGPT - ScholarGPT

    2025. ChatGPT - ScholarGPT. Retrieved July 4, 2025 from https://chatgpt.com/g/g- kZ0eYXlJe-scholar-gpt

    work page 2025

  2. [2]

    Cursor - The AI Code Editor

    2025. Cursor - The AI Code Editor. Retrieved July 4, 2025 from https://cursor. com/en

    work page 2025

  3. [3]

    Introducing the GPT Store | OpenAI

    2025. Introducing the GPT Store | OpenAI. Retrieved July 4, 2025 from https: //openai.com/index/introducing-the-gpt-store/

    work page 2025

  4. [4]

    Welcome to the State of Developer Ecosystem Report 2024

    2025. Welcome to the State of Developer Ecosystem Report 2024. Retrieved July 4, 2025 from https://www.jetbrains.com/lp/devecosystem-2024/

    work page 2025

  5. [5]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  6. [6]

    Rifat Mehreen Amin, Oliver Hans Kühle, Daniel Buschek, and Andreas Butz

  7. [7]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L Glassman. 2024. Chainforge: A visual toolkit for prompt engineer- ing and llm hypothesis testing. InCHI Conference on Human Factors in Computing Systems (CHI). 1–18. doi:10.1145/3613904.3642016

    arXiv 2024

  8. [8]

    Understanding Prompt Programming Tasks and Questions

    Anonymous Authors. 2025. Supplemental Materials to "Understanding Prompt Programming Tasks and Questions". The supplemental materials are available on HotCRP and Figshare at https://figshare.com/s/c1d893b8f025cbd429ec. We will make the supplemental materials publicly available upon acceptance

    work page 2025

Show all 70 references
  1. [9]

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving image generation with better captions. 2, 3 (2023), 8. doi:papers/dall-e-3.pdf

    2023

  2. [10]

    Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman

  3. [11]

    Silvia Breu, Rahul Premraj, Jonathan Sillito, and Thomas Zimmermann. 2010. Information needs in bug reports: Improving cooperation between developers and users. In ACM Conference on Computer Supported Cooperative Work (CSCW) . 301–310. doi:10.1145/1718918.1718973

    2010

  4. [12]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), ...

    2020

  5. [13]

    Juliet Corbin and Anselm Strauss. 2015. Basics of qualitative research . Vol. 14. Sage. doi:10.1177/1094428108324514

    2015 doi

  6. [14]

    Hai Dang, Frederik Brudy, George Fitzmaurice, and Fraser Anderson. 2023. World- smith: Iterative and expressive prompting for world building with a generative AI. In ACM Symposium on User Interface Software and Technology (UIST) . 1–17. doi:10.1145/3586183.3606772

    2023

  7. [15]

    Mateusz Dolata, Norbert Lange, and Gerhard Schwabe. 2024. Development in times of hype: How freelancers explore Generative AI?. InIEEE/ACM International Conference on Software Engineering (ICSE) . 1–13. doi:10.1145/3597503.3639111

    2024

  8. [16]

    Ekwa Duala-Ekoko and Martin P Robillard. 2012. Asking and answering questions about unfamiliar APIs: An exploratory study. In International Conference on Software Engineering (ICSE). IEEE, 266–276. doi:10.1109/ICSE.2012.6227187

    2012

  9. [17]

    Will Epperson, Gagan Bansal, Victor C Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive debugging and steering of multi- agent ai systems. In CHI Conference on Human Factors in Computing Systems . 1–15. doi:10.1145/3706598.3713581

    2025

  10. [19]

    Alexander J Fiannaca, Chinmay Kulkarni, Carrie J Cai, and Michael Terry. 2023. Programming without a programming language: Challenges and opportunities for designing developer tools for prompt programming. In CHI Conference on Human Factors in Computing Systems Extended Abstra...

    2023

  11. [20]

    Katy Ilonka Gero, Jonathan K Kummerfeld, and Elena L Glassman. 2022. Sense- making interfaces for human evaluation of language model outputs. In NeurIPS Workshop on Human Evaluation of Generative Models

    2022

  12. [21]

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K Kummerfeld, and Elena L Glassman. 2024. Supporting sensemaking of large language model outputs at scale. In CHI Conference on Human Factors in Computing Systems (CHI) . 1–21. doi:10.1145/3613904.3642139

    2024

  13. [22]

    Görkem Giray. 2021. A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software (JSS) 180 (2021), 111031. doi:10.1016/j.jss.2021.111031

    2021

  14. [23]

    Nicole Haenni, Mircea Lungu, Niko Schwarz, and Oscar Nierstrasz. 2013. Cate- gorizing developer information needs in software ecosystems. In International Workshop on Ecosystem Architectures (WEA). 1–5. doi:10.1145/2501585.2501586

    2013

  15. [24]

    David Hammer and Leema K Berland. 2014. Confusing claims for data: A critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences 23, 1 (2014), 37–46. doi:10.1080/10508406.2013.802652

    2014

  16. [25]

    Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defining customer- centered systems. Elsevier

    1997

  17. [26]

    Yu Huang, Denae Ford, and Thomas Zimmermann. 2021. Leaving my fingerprints: Motivations and challenges of contributing to OSS for social good. In IEEE/ACM International Conference on Software Engineering (ICSE) . IEEE, 1020–1032. doi:10. 1109/ICSE43902.2021.0009

    2021

  18. [27]

    Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. Promptmaker: Prompt-based prototyping with large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA) . 1–8. doi:10.1...

    2022

  19. [28]

    Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue: Exploring large language model responses with interactive diagrams. In ACM Symposium on User Interface Software and Technology . 1–20. doi:10.1145/3586183. 3606737

    2023 doi

  20. [30]

    Mary Beth Kery. 2021. Designing effective history support for exploratory pro- gramming data work. Ph. D. Dissertation. Ph. D. Dissertation. Carnegie Mellon University, Pittsburgh, Pennsylvania

    2021

  21. [31]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In CHI Conference on Human Factors in Computing Systems (CHI) . 1–21. doi:10.1145/3613904.3642216

    2024

  22. [32]

    Barbara A Kitchenham and Shari L Pfleeger. 2008. Personal opinion surveys. In Guide to Advanced Empirical Software Engineering . Springer, 63–92. doi:10.1007/ 978-1-84800-044-5_3

    2008

  23. [33]

    Amy J Ko, Robert DeLine, and Gina Venolia. 2007. Information needs in collocated software development teams. In International Conference on Software Engineering (ICSE). IEEE, 344–353. doi:10.1109/ICSE.2007.45

    2007 doi

  24. [34]

    Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering (ESE) 20 (2015), 110–141. doi:10.1007/s10664-013- 9279-3

    2015 doi

  25. [35]

    Amy J Ko and Brad A Myers. 2008. Debugging reinvented: Asking and answering why and why not questions about program behavior. In International Conference on Software Engineering (ICSE) . 301–310. doi:10.1145/1368088.1368130

    2008

  26. [36]

    Amy J Ko and Brad A Myers. 2010. Extracting and answering why and why not questions about Java program output. ACM Transactions on Software Engineering and Methodology (TOSEM) 20, 2 (2010), 1–36. doi:10.1145/1824760.1824761

    2010

  27. [37]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. doi:10.2307/2529310

    1977 doi

  28. [38]

    Thomas D LaToza and Brad A Myers. 2010. Hard-to-answer questions about code. In Evaluation and Usability of Programming Languages and Tools (PLATEAU) . 1–6. doi:10.1145/1937117.1937125

    2010

  29. [39]

    Jenny T Liang, Melissa Lin, Nikitha Rao, and Brad A Myers. 2025. Prompts are programs too! Understanding how developers build software containing prompts. Proceedings of the ACM on Software Engineering 2, FSE (2025), 1591–

    2025

  30. [40]

    Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of AI programming assistants: Successes and challenges. In IEEE/ACM International Conference on Software Engineering (ICSE) . 1–13. doi:10. 1145/3597503.3608128

    2024

  31. [41]

    Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022. Understanding skills for OSS communities on github. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . 170–182. doi:10.1145/3540250.3549082

    2022

  32. [42]

    We need structured output

    Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024. "We need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors ...

    2024

  33. [43]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR) 55, 9 (2023), 1–35. doi:10.1145/3560815

    2023 doi

  34. [44]

    Yuetian Mao, Junjie He, and Chunyang Chen. 2025. From prompts to templates: A systematic prompt template analysis for real-world LLMapps. arXiv preprint arXiv:2504.02052 (2025)

    2025 arXiv

  35. [45]

    Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23. doi:10.1145/3359174

    2019 doi

  36. [46]

    Aditi Mishra, Bretho Danzy, Utkarsh Soni, Anjana Arunkumar, Jinbin Huang, Bum Chul Kwon, and Chris Bryan. 2025. PromptAid: Visual prompt exploration, perturbation, testing and iteration for large language models. IEEE Transactions on Visualization and Computer Graphics (TVCG) ...

    2025 doi

  37. [47]

    Brad A Myers, Amy J Ko, Thomas D LaToza, and YoungSeok Yoon. 2016. Pro- grammers are users too: Human-centered methods for improving programming tools. Computer 49, 7 (2016), 44–52. doi:10.1109/MC.2016.200

    2016 doi

  38. [48]

    Nadim Nachar et al. 2008. The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in Quantitative Methods for Psychology 4, 1 (2008), 13–20. doi:10.20982/tqmp.04.1.p013

    2008 doi

  39. [49]

    Nadia Nahar, Christian Kästner, Jenna Butler, Chris Parnin, Thomas Zimmermann, and Christian Bird. 2024. Beyond the comfort zone: Emerging solutions to overcome challenges in integrating LLMs into software products. arXiv preprint arXiv:2410.12071 (2024)

    2024 arXiv

  40. [50]

    Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner

  41. [51]

    Richard E Nisbett and Timothy D Wilson. 1977. Telling more than we can know: Verbal reports on mental processes. Psychological review 84, 3 (1977), 231. doi:10.1037/0033-295X.84.3.231

    1977 doi

  42. [52]

    Chris Parnin, Gustavo Soares, Rahul Pandita, Sumit Gulwani, Jessica Rich, and Austin Z Henley. 2025. Building your own product copilot: Challenges, opportu- nities, and needs. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 338...

    2025

  43. [53]

    In IEEE/ACM International Conference on AI Engineering–Software Engineering for AI (CAIN)

    A meta-summary of challenges in building products with ML components– Collecting experiences from 4758+ practitioners. In IEEE/ACM International Conference on AI Engineering–Software Engineering for AI (CAIN) . 171–183. doi:10.1109/CAIN58948.2023.00034

  44. [54]

    Crystal Qian, Emily Reif, and Minsuk Kahng. 2024. Understanding the dataset practitioners behind large language models. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI) . 1–7. doi:10.1145/3613905. 3651007

    2024 doi

  45. [55]

    Mohi Reza, Ioannis Anastasopoulos, Shreya Bhandari, and Zachary A Pardos

  46. [56]

    Kanchanjot Kaur Phokela, Samarth Sikand, Kapil Singi, Kuntal Dey, Vibhu Sau- janya Sharma, and Vikrant Kaulgud. 2023. Smart prompt advisor: Multi- objective prompt framework for consistency and best practices. In IEEE/ACM International Conference on Automated Software Engineer...

    2023

  47. [57]

    Vagner Figueredo de Santana, Sara Berger, Tiago Machado, Maysa Malfiza Garcia de Macedo, Cassia Sampaio Sanctos, Lemara Williams, and Zhaoqing Wu. 2025. Can LLMs recommend more responsible prompts?. In International Conference on Intelligent User Interfaces (IUI) . 298–313. do...

    2025

  48. [58]

    Jonathan Sillito, Gail C Murphy, and Kris De Volder. 2006. Questions program- mers ask during software evolution tasks. In ACM International Symposium on Foundations of Software Engineering (FSE) . 23–34. doi:10.1145/1181775.1181779

    2006

  49. [59]

    InCHI Conference on Human Factors in Computing Systems (CHI)

    PromptHive: Bringing subject matter experts back to the forefront with col- laborative prompt engineering for educational content creation. InCHI Conference on Human Factors in Computing Systems (CHI). 1–22. doi:10.1145/3706598.3714051

  50. [60]

    Johnny Saldaña. 2009. The Coding Manual for Qualitative Researchers . SAGE Publications

    2009

  51. [61]

    Harsh Suri. 2011. Purposeful sampling in qualitative research synthesis. Qualita- tive research journal 11, 2 (2011), 63–75. doi:10.3316/QRJ1102063

    2011 doi

  52. [62]

    Mahan Tafreshipour, Aaron Imani, Eric Huang, Eduardo Santana de Almeida, Thomas Zimmermann, and Iftekhar Ahmed. 2025. Prompting in the wild: An empirical study of prompt evolution in software repositories. In IEEE/ACM 22nd International Conference on Mining Software Repositori...

    2025

  53. [63]

    Jonathan Sillito, Gail C Murphy, and Kris De Volder. 2008. Asking and answering questions during a programming change task. IEEE Transactions on Software Engineering (TSE) 34, 4 (2008), 434–451. doi:10.1109/TSE.2008.26

    2008 doi

  54. [64]

    Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE Trans- actions on Visualization and Computer Graph...

    2022

  55. [65]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large lan- guage model prompts through visual programming. In Extended Abstracts of CHI Conference on Human Factors in Computing Systems (CHI...

    2022

  56. [66]

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In CHI Conference on Human Factors in Computing Systems (CHI) . 1–22. doi:10. 1145/3491102.3517582

    2022

  57. [67]

    Zhiyuan Wan, Xin Xia, David Lo, and Gail C Murphy. 2019. How does machine learning change software development practices? IEEE Transactions on Software Engineering (TSE) 47, 9 (2019), 1857–1871. doi:10.1109/TSE.2019.2937083

    2019

  58. [68]

    Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2024. Promptcharm: Text-to-image generation through multi-modal prompting and refinement. In CHI Conference on Human Factors in Computing Systems (CHI) . 1–21. doi:10.1145/3613904.3642803

    2024

  59. [69]

    J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: How non-AI experts try (and fail) to design LLM prompts. In CHI Conference on Human Factors in Computing Systems (CHI). 1–21. doi:10.1145/3544548.3581388 Received 20 Feb...

    2023

  60. [71]

    Qinyuan Ye, Mohamed Ahmed, Reid Pryzant, and Fereshte Khani. 2024. Prompt engineering a prompt engineer. In Findings of the Association for Computational Linguistics, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). 355–385. doi:10.18653/v1/2024.findings-acl.21

    2024 doi

  61. [72]

    JD Zamfirescu-Pereira, Bjoern Hartmann, and Qian Yang. 2023. Conversation regression testing: A design technique for prototyping generalizable prompt strategies for pre-trained language models.arXiv preprint arXiv:2302.03154 (2023)

    2023 arXiv

  62. [2025]

    In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA)

    Composable prompting workspaces for creative writing: Exploration and iteration using dynamic widgets. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA). 1–11. doi:10.1145/3706599.3720243

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.