REVIEW 8 cited by
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.
Forward citations
Cited by 8 Pith papers
-
ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts
ViseGPT automatically converts user prompts into test cases and visualizes which steps of an LLM-generated data wrangling script pass or fail.
-
Designing Conversational AI to Support Think-Aloud Practice in Technical Interview Preparation for CS Students
A 17-participant study finds CS students value LLM-based mock interviews for think-aloud practice and want AI that feels more present, gives feedback beyond spoken words, and mixes human examples with AI generation.
-
Who's the Leader? Analyzing Novice Workflows in LLM-Assisted Debugging of Machine Learning Code
In an eight-person formative study, novice ML engineers who actively led the ChatGPT debugging conversation outperformed those who followed it, with patterns of over- and under-reliance.
-
Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steering
Preference-based activation steering lets users control chatbot personalization with a linear factor, and a small user study found that steered chatbots align better with self-reported preferences than prompting alone.
-
"I Would Have Written My Code Differently'': Beginners Struggle to Understand LLM-Generated Code
CS1 students correctly predicted the output of LLM-generated Python code in 32.5% of tasks, versus 59.4% for natural-language prompts, and still failed code prediction 58% of the time when they understood the prompt.
-
Steering Semantic Data Processing With DocWrangler
An IDE for LLM-powered text data processing, with user studies showing that people convert open-ended operations into structured classifiers and use vague prompts to explore their data.
-
Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution
In conversations linked to open GitHub issues, 44.6% of developer prompts contain knowledge gaps, versus 12.6% in conversations linked to closed issues; Missing Context is the most common gap.
-
From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law
An LLM pipeline that flags and fixes non-compliant consent use cases works imperfectly: it catches about two-thirds of relevant cases with reasoning prompts, and most of its fixes are legally sound but logically inconsistent.
Discussion (0). Continue with ORCID to comment.