REVIEW 7 cited by
Dr.ICL: Demonstration-Retrieved In-context Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In-context learning (ICL), teaching a large language model (LLM) to perform a task with few-shot demonstrations rather than adjusting the model parameters, has emerged as a strong paradigm for using LLMs. While early studies primarily used a fixed or random set of demonstrations for all test queries, recent research suggests that retrieving semantically similar demonstrations to the input from a pool of available demonstrations results in better performance. This work expands the applicability of retrieval-based ICL approaches by demonstrating that even simple word-overlap similarity measures such as BM25 outperform randomly selected demonstrations. Furthermore, we extend the success of retrieval-based ICL to instruction-finetuned LLMs as well as Chain-of-Thought (CoT) prompting. For instruction-finetuned LLMs, we find that although a model has already seen the training data at training time, retrieving demonstrations from the training data at test time yields better results compared to using no demonstrations or random demonstrations. Last but not least, we train a task-specific demonstration retriever that outperforms off-the-shelf retrievers.
Forward citations
Cited by 7 Pith papers
-
PowerDAG: Supervisory Agentic AI System for Automating Distribution Grid Analysis
PowerDAG achieves 94-100% success on unseen distribution grid analysis queries by combining adaptive retrieval with similarity-decay cutoff and just-in-time supervision, outperforming ReAct, LangChain, and CrewAI baselines.
-
Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrieval
DOPA approximates an inaccessible target domain via source-tuned vs. base LLM perplexity ratios, then retrieves diverse source demonstrations to improve OOD in-context learning.
-
Failures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samples
A retrieval method that uses semantically similar wrong answers as anchors to find corrective correct demonstrations improves few-shot in-context learning accuracy on seven reasoning datasets.
-
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
CLG selects many-shot demonstrations by matching fine-tuning gradients of a small language model to the full training set, improving accuracy over random selection by 2-4%.
-
Learning to Select In-Context Demonstration Preferred by Large Language Model
A generative preference-learning method trains a latent demonstration selector from LLM feedback and improves few-shot in-context learning performance on most of 19 benchmark datasets.
-
The Role of Diversity in In-Context Learning for Large Language Models
Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.
-
ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval
ALLabel combines diversity, similarity, and uncertainty sampling to pick the most useful texts for human annotation, reaching near-full-data NER accuracy with 5-10% of labels.
Discussion (0). Sign in to comment.