REVIEW 13 cited by
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
read the original abstract
Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language models can be incorporated into existing and future benchmarks and practical applications.
Forward citations
Cited by 13 Pith papers
-
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Chain-of-thought prompting, by including intermediate reasoning steps in few-shot examples, elicits strong reasoning abilities in large language models on arithmetic, commonsense, and symbolic tasks.
-
Teaching Models to Express Their Uncertainty in Words
GPT-3 can learn to express well-calibrated uncertainty about its answers using natural language phrases rather than logits.
-
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
The authors present a catalog of prompt patterns that provide reusable solutions to common problems in generating and interacting with outputs from LLMs.
-
Large Language Models are Zero-Shot Reasoners
Adding the fixed prompt 'Let's think step by step' enables large language models to achieve substantial zero-shot gains on arithmetic, symbolic, and logical reasoning benchmarks without any task-specific examples.
-
Multitask Prompted Training Enables Zero-Shot Task Generalization
Multitask fine-tuning of an encoder-decoder model on prompted datasets produces zero-shot generalization that often beats models up to 16 times larger on standard benchmarks.
-
Agentic Synthesis against Counterexample-Supplemented Sketches
Operator-approved counterexamples are folded into an evolving sketch that carries the learned policy; in one CatSynth run, rebuilding from that sketch passed 19/21 withheld cases versus 15/21 when replaying all exampl...
-
Empirical Prompt Engineering for Construct Identification with Large Language Models
For LLM classification of psychological constructs, selecting the best prompt from many variants improves human-model agreement more than personas, chain-of-thought, or explanations.
-
Emergent Abilities of Large Language Models
Emergent abilities are capabilities present in large language models but absent in smaller ones and cannot be predicted by extrapolating smaller model performance.
-
Agentic Synthesis against Counterexample-Supplemented Sketches
This paper proposes counterexample-supplemented sketches: a repository workflow where human operators approve policy changes and a dual gate (deterministic replay plus sketch review) controls agent-driven code synthesis.
-
Augment Engineering: A Methodology for Multi-Tool AI Orchestration Across Professional Domains
Introduces Augment Engineering as a six-phase multi-tool orchestration methodology, supported by exploratory statistics from a single-practitioner case study across seven domains.
-
Large-Scale ChatBot Validation Through Customer Digital Twin Simulations
Synthetic customer agents built from real bank data can mimic customer semantics and personality well enough to serve as scalable chatbot validation proxies.
-
Mechanistic Personality Analysis of LLMs Steering Personality via Latent Feature Interventions
Applies sparse autoencoders to locate and steer latent features for OCEAN personality traits in LLMs while preserving benchmark performance.
-
Sustainable Code Generation Using Large Language Models: A Systematic Literature Review
A systematic review finds research on the sustainability of LLM-generated code to be limited, fragmented, and without accepted frameworks for measurement or benchmarking.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.