REVIEW 8 cited by
Active Example Selection for In-Context Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
With a handful of demonstration examples, large-scale language models show strong capability to perform various tasks by in-context learning from these examples, without any fine-tuning. We demonstrate that in-context learning performance can be highly unstable across samples of examples, indicating the idiosyncrasies of how language models acquire information. We formulate example selection for in-context learning as a sequential decision problem, and propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples. For GPT-2, our learned policies demonstrate strong abilities of generalizing to unseen tasks in training, with a $5.8\%$ improvement on average. Examples selected from our learned policies can even achieve a small improvement on GPT-3 Ada. However, the improvement diminishes on larger GPT-3 models, suggesting emerging capabilities of large language models.
Forward citations
Cited by 8 Pith papers
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
Towards Compute-Optimal Many-Shot In-Context Learning
Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.
-
Mobile Traffic Prediction using LLMs with Efficient In-context Demonstration Selection
A two-step in-context demonstration selection method, based on distance similarity and zero-shot prediction error, improves LLM-based mobile traffic prediction on a real 5G dataset compared with zero-shot and simple b...
-
Retrieval Augmented Generation based Large Language Models for Causality Mining
Dynamic retrieval of few-shot examples by causal connective and semantic similarity improves LLM causality detection and extraction over static prompting in most tested settings.
-
Learning to Select Visual In-Context Demonstrations
A Dueling-DQN agent selects visual in-context demonstrations and outperforms kNN retrieval on objective regression benchmarks but not on subjective preference tasks, per the paper's main table.
-
InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity
A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.
-
Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
Discussion (0). Sign in to comment.