REVIEW 14 cited by
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
read the original abstract
Language Model Programs, i.e. sophisticated pipelines of modular language model (LM) calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly effective for all modules. We study prompt optimization for LM programs, i.e. how to update these prompts to maximize a downstream metric without access to module-level labels or gradients. To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules. Our strategies include (i) program- and data-aware techniques for proposing effective instructions, (ii) a stochastic mini-batch evaluation function for learning a surrogate model of our objective, and (iii) a meta-optimization procedure in which we refine how LMs construct proposals over time. Using these insights we develop MIPRO, a novel algorithm for optimizing LM programs. MIPRO outperforms baseline optimizers on five of seven diverse multi-stage LM programs using a best-in-class open-source model (Llama-3-8B), by as high as 13% accuracy. We have released our new optimizers and benchmark in DSPy at http://dspy.ai
Forward citations
Cited by 14 Pith papers
-
Learning, Fast and Slow: Towards LLMs That Adapt Continually
Fast-Slow Training uses context optimization as fast weights alongside parameter updates as slow weights to achieve up to 3x better sample efficiency, higher performance, and less catastrophic forgetting than standard...
-
FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients
FlowBot automatically induces LLM workflows through bilevel optimization with textual gradients, achieving competitive performance against human-crafted baselines.
-
IUU+DB: Tracking Illegal, Unreported, and Unregulated Fishing, Seafood Fraud, and Labor Abuse through LLM-driven Information Extraction
An LLM-driven extraction system builds a global IUU+ incident database of 8,435 events from 2,472 sources across 143 countries, with validation and case studies.
-
Evolving and Detecting Multi-Turn Deception using Geometric Signatures
Multi-objective genetic prompt optimization creates multi-turn deceptive datasets validated by humans, then detected with 0.89 recall using angular coverage, distance ratio, and linearity features in embeddings.
-
LLMs Are Already Good Tutors: Training-Free Prompt Optimization for Pedagogical Math Tutoring
Training-free prompt optimization methods, including five new education-focused ones, surpass the strongest RL-trained baseline across five conditions on two OOD suites while showing distinct teaching behavior patterns.
-
optimize_anything: A Universal API for Optimizing any Text Parameter
A universal LLM optimizer for text artifacts achieves SOTA results on six tasks including tripling ARC-AGI accuracy and cutting cloud costs by 40% via cross-task transfer and side information.
-
Contexting as Recommendation: Evolutionary Collaborative Filtering for Context Engineering
NCCE reframes context engineering as instance-level recommendation via bootstrapped anchor contexts and a co-evolving neural collaborative filtering router that assigns specialized contexts per input.
-
Learning, Fast and Slow: Towards LLMs That Adapt Continually
Fast-Slow Training combines slow parameter updates with fast context optimization to achieve up to 3x better sample efficiency, higher performance, less forgetting, and preserved plasticity in continual LLM learning.
-
DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection
Detection Prompt Optimization (DetPO) improves few-shot object detection with black-box MLLMs by iteratively refining text prompts from TP/FP/FN errors on few-shot examples, gaining up to 9.7 mAP over prior black-box methods.
-
EditFlow: Benchmarking and Optimizing Code Edit Recommendation Systems via Reconstruction of Developer Flows
EditFlow reconstructs temporal developer editing flows from code changes to benchmark and optimize AI code edit recommenders so they align with natural incremental reasoning rather than static snapshots.
-
Contrastive Reflection for Iterative Prompt Optimization
Contrastive Reflection identifies error-anchored slices in agent traces, adds contrastive successes, and uses a Teacher LLM to generate prompt edits that are accepted only if they improve validation performance, raisi...
-
IUU+DB: Tracking Illegal, Unreported, and Unregulated Fishing, Seafood Fraud, and Labor Abuse through LLM-driven Information Extraction
IUU+DB is an LLM-driven pipeline that classifies documents, extracts structured incident data on IUU+ activities, and supports deduplication and trend analysis.
-
Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents
Trace2Policy's EISR iteratively refines expert-derived rules into compiled Python code reaching 79.6% accuracy on skewed compliance tasks, outperforming one-shot LLM distillation and a deployed LLM baseline.
-
Automated Instruction Revision (AIR): A Structured Comparison of Task Adaptation Strategies for LLM
AIR excels on label-remapping classification tasks while KNN retrieval leads on closed-book QA and fine-tuning leads on structured extraction and event-order reasoning, showing task-dependent adaptation performance.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.