REVIEW 7 cited by
In-context Example Selection with Influences
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use $\textit{in-context influences}$ to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a $16.3\%$ performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.
Forward citations
Cited by 7 Pith papers
-
From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program Repair
Backward reasoning distilled from a repository's already-fixed issues, injected as prompts, lifts LLM repair success on SWE-Bench Lite by up to 10.4 Pass@1 points.
-
Exploring Imbalanced Annotations for Effective In-Context Learning
Class-imbalanced annotation sets degrade in-context learning, and reweighting demonstration scores by class weights plus a validation-fitted conditional bias term (RCB) recovers most of the loss.
-
DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer
DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...
-
Demystifying Feature Requests: Leveraging LLMs to Refine Feature Requests in Open-Source Software
GPT-4o with in-context learning can flag ambiguity and incompleteness in GitHub feature requests and draft clarification questions, though moderate annotator agreement and a small sample limit the strength of the evidence.
-
Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples
A combinatorial scoring method for retriever training data improves few-shot dialogue state tracking by 20x in data efficiency and by 12% in oracle upper-bound JGA over prior methods.
-
NER4all or Context is All You Need: Using LLMs for low-effort, high-performance NER on historical texts. A humanities informed approach
With context-rich prompts and persona modeling, ChatGPT-4o outperformed off-the-shelf spaCy and flair on named entity recognition for a 1921 German travel guide.
-
OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models
The paper advocates protecting and leveraging OpenReview's peer review corpus as a community asset for LLM-based review assistance, benchmarks, and alignment.
Discussion (0). Continue with ORCID to comment.