REVIEW 8 cited by
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.
Forward citations
Cited by 8 Pith papers
-
Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
RG-PT learns a reliability DAG over hyperparameters and combines it with FDR-controlled testing, yielding shorter reliable LLM prompts than LTT and Pareto testing.
-
Evaluating Language Models as Synthetic Data Generators
AgoraBench shows that an LM's ability to solve problems does not predict its ability to generate useful synthetic training data.
-
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization
SymDPO replaces answer text in multimodal demonstrations with meaningless symbols during preference training, forcing models to use image context and improving in-context learning performance on five benchmarks.
-
A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models
A Bayesian 'knowledge gradient' policy for sequentially choosing which prompts to evaluate finds better language-model prompts within 30 evaluations than evolutionary, bandit, and greedy baselines on instruction-induc...
-
On the Role of Model Prior in Real-World Inductive Reasoning
LLMs' hypotheses for real-world classification tasks are driven mostly by task priors, and in-context demonstrations, even with flipped labels, do little to change them.
-
SI-Agent: An Agentic Framework for Feedback-Driven Generation and Tuning of Human-Readable System Instructions for Large Language Models
The paper proposes a multi-agent loop (instructor, follower, feedback) to auto-generate human-readable system prompts, claiming good benchmark performance and readability, but the supporting experiments are not reprod...
-
Exploring Prompt Patterns in AI-Assisted Code Generation: Towards Faster and More Effective Developer-AI Collaboration
Using keyword matching on DevGPT conversations, the authors rank seven prompt patterns by a hand-weighted effectiveness score and claim 'Context and Instruction' and 'Recipe' reduce iterations.
-
A Survey on Large Language Models with some Insights on their Capabilities and Limitations
A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.
Discussion (0). Continue with ORCID to comment.