REVIEW 8 cited by
In-Context Learning with Iterative Demonstration Selection
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Spurred by advancements in scale, large language models (LLMs) have demonstrated strong few-shot learning ability via in-context learning (ICL). However, the performance of ICL has been shown to be highly sensitive to the selection of few-shot demonstrations. Selecting the most suitable examples as context remains an ongoing challenge and an open problem. Existing literature has highlighted the importance of selecting examples that are diverse or semantically similar to the test sample while ignoring the fact that the optimal selection dimension, i.e., diversity or similarity, is task-specific. Based on how the test sample is answered, we propose Iterative Demonstration Selection (IDS) to leverage the merits of both dimensions. Using zero-shot chain-of-thought reasoning (Zero-shot-CoT), IDS iteratively selects examples that are diverse but still strongly correlated with the test sample as ICL demonstrations. Specifically, IDS applies Zero-shot-CoT to the test sample before demonstration selection. The output reasoning path is then used to choose demonstrations that are prepended to the test sample for inference. The generated answer is followed by its corresponding reasoning path for extracting a new set of demonstrations in the next iteration. After several iterations, IDS adopts majority voting to obtain the final result. Through extensive experiments on tasks including reasoning, question answering, and topic classification, we demonstrate that IDS can consistently outperform existing ICL demonstration selection methods.
Forward citations
Cited by 8 Pith papers
-
Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
LongGuide automatically learns task-specific quality and length guidelines from small training sets, significantly improving LLM long-form generation.
-
Surprise Calibration for Better In-Context Learning
Surprise Calibration uses the model's own surprise at each demonstration's label to dynamically correct class priors in in-context learning, improving accuracy on eight NLP benchmarks.
-
Mobile Traffic Prediction using LLMs with Efficient In-context Demonstration Selection
A two-step in-context demonstration selection method, based on distance similarity and zero-shot prediction error, improves LLM-based mobile traffic prediction on a real 5G dataset compared with zero-shot and simple b...
-
The Role of Diversity in In-Context Learning for Large Language Models
Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.
-
Learning to Select Visual In-Context Demonstrations
A Dueling-DQN agent selects visual in-context demonstrations and outperforms kNN retrieval on objective regression benchmarks but not on subjective preference tasks, per the paper's main table.
-
DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer
DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...
-
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...
-
MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning
MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.
Discussion (0). Continue with ORCID to comment.