Pith. sign in

REVIEW 47 cited by

What Makes Good In-Context Examples for GPT-$3$?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2101.06804 v1 pith:7JQWOXS7 submitted 2021-01-17 cs.CL

Jiachang Liu , Dinghan Shen , Yizhe Zhang , Bill Dolan , Lawrence Carin , Weizhu Chen This is my paper · ORCID
classification cs.CL
keywords gpt-examplesin-contextfew-shotapproachcapabilitiesdatasetgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

GPT-$3$ has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT-$3$ depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-$3$'s few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT-$3$'s extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT-$3$ and large-scale pre-trained LMs in general and enhance their few-shot capabilities.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 47 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 159 citations worldwide. Full citation record

  1. Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

    cs.LG 2025-06 conditional novelty 7.0 of 10

    RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.

  2. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  3. Symbol and Footprint Database for Electronic Components by Agentic Recognition and Generation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    An MLLM-driven agentic pipeline generates PCB component symbols and footprints from datasheets with reported 86%/80% accuracy and builds a 1,000-component library.

  4. KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning

    cs.LG 2025-09 reject novelty 6.0 of 10

    KITE greedily selects in-context examples by maximizing a kernelized query-specific relevance score plus a D-optimal design diversity term, and reports accuracy gains over KATE, BM25, and DPP baselines on five classif...

  5. Fluent but Unfeeling: The Emotional Blind Spots of Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    On EXPRESS, best LLMs reach only ~31-36% lexical accuracy and ~39-44% basic-emotion-vector accuracy in predicting self-disclosed emotions, with chain-of-thought prompting hurting.

  6. Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    Human-in-Context is a single in-context-learning model that handles 10 pose- and mesh-based 3D human motion tasks across 4 datasets and outperforms prior unified and task-specific baselines.

  7. Towards Compute-Optimal Many-Shot In-Context Learning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.

  8. LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra

    cs.MA 2025-07 reject novelty 6.0 of 10

    The LLM Economist framework couples persona-conditioned worker agents with an in-context RL planner to search US-bracket tax schedules, yet its Saez benchmark is derived from the planner's own solution and its headlin...

  9. DanceChat: Large Language Model-Guided Music-to-Dance Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    An LLM-generated text choreography, fused with music and beat features, guides a diffusion model to produce more diverse and physically plausible dance motion, with a multi-modal alignment loss intended to bridge musi...

  10. Adaptive Task Vectors for Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...

  11. Retrieval Augmented Generation based Large Language Models for Causality Mining

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Dynamic retrieval of few-shot examples by causal connective and semantic similarity improves LLM causality detection and extraction over static prompting in most tested settings.

  12. SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Large language models perform poorly on a new C-code vulnerability benchmark, indicating they rely on pattern matching rather than genuine reasoning.

  13. Learning to Select In-Context Demonstration Preferred by Large Language Model

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A generative preference-learning method trains a latent demonstration selector from LLM feedback and improves few-shot in-context learning performance on most of 19 benchmark datasets.

  14. The Role of Diversity in In-Context Learning for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.

  15. Investigating the Zone of Proximal Development of Language Models for In-Context Learning

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A framework that predicts, per query, whether an LLM can solve it directly, only with demonstrations, or not at all, and uses those predictions for selective in-context learning and curriculum fine-tuning.

  16. OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    OptiSeq selects the in-context example ordering whose output gets the highest zero-shot log-likelihood, improving few-shot accuracy by up to 10.5 points in tests on API sequencing and classification.

  17. Online Prompt Selection for Program Synthesis

    cs.AI 2025-01 conditional novelty 6.0 of 10

    An online multi-armed bandit that selects among symbolic solvers and LLM-prompt combinations for program synthesis solves 37.2% more queries than the best single solver and reaches 96% of the virtual best solver's per...

  18. Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    On randomly sampled 3-state DFA language tasks, foundation LLMs underperform n-gram baselines under pure in-context-learning prompts.

  19. Language Model as Visual Explainer

    cs.CV 2024-12 reject novelty 6.0 of 10

    LVX builds LLM-generated attribute trees to explain any trained image classifier without training the explainer, but its faithfulness metric is directly optimized by the method.

  20. PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks

    cs.CL 2024-12 conditional novelty 6.0 of 10

    PromptRefine uses alternating minimization over language-specific retrievers plus diversity-aware DPP fine-tuning to select cross-lingual in-context examples, improving few-shot generation in low-resource Indic languages.

  21. Does Few-Shot Learning Help LLM Performance in Code Synthesis?

    cs.SE 2024-12 conditional novelty 6.0 of 10

    Few-shot example choice measurably affects LLM code output, and two proposed selectors (a perplexity ranker and a trained MLP ranker) each improve CodeLlama's Pass@1 on HumanEval+ by about five points.

  22. On the Privacy Risk of In-context Learning

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A confidence-based membership inference attack identifies prompt demonstration data with AUC 0.69-0.86, more than fine-tuned models leak at matched utility, and ensembling reduces this to near random.

  23. Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models

    cs.AI 2026-08 reject novelty 5.0 of 10

    Across five LLMs, a sharp decrease in embedding isotropy at a critical layer predicts multiple-choice accuracy, with Spearman correlations up to -0.92.

  24. When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness

    cs.IR 2026-07 conditional novelty 5.0 of 10

    On AG News (n=200), five LLMs show four distinct few-shot regimes (monotonic gain, one-shot rescue, zero-shot-only, and worse-then-partial-recovery), and a substring parser under-scored Llama 3.3 70B by up to 206%.

  25. Beyond Exact Match: How Evaluation Methodology Dominates Model Choice in LLM-Based Product Attribute Extraction

    cs.IR 2026-07 conditional novelty 5.0 of 10

    On the MAVE benchmark, switching from exact to fuzzy matching changes reported F1 by ~0.12, dwarfing model choice (~0.005) and prompt choice (~0.024), and fuzzy-match auditing labels 23.2% of exact-match failures as s...

  26. Quantifying Political Partisanship for Cross-Platform Analyses

    cs.SI 2026-07 reject novelty 5.0 of 10

    Partisanship of individual posts can be scored on a common embedding axis anchored by AllSides news-bias labels, yielding cross-platform scores that transfer from Bluesky/Truth Social to X.

  27. Prompt as a Data Type: In-Database LLM Prompt Management and Rewriting

    cs.DB 2026-07 conditional novelty 5.0 of 10

    Treating LLM prompts as tuple-level database values lets a DBMS rewrite prompts using schema, constraints, and statistics, improving validity and cost-quality trade-offs.

  28. Learning to Select Visual In-Context Demonstrations

    cs.LG 2026-03 reject novelty 5.0 of 10

    A Dueling-DQN agent selects visual in-context demonstrations and outperforms kNN retrieval on objective regression benchmarks but not on subjective preference tasks, per the paper's main table.

  29. Robust Knowledge Editing via Explicit Reasoning Chains for Distractor-Resilient Multi-Hop QA

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Teaching an LLM to emit a fixed four-stage reasoning chain during fine-tuning makes single-pass multi-hop knowledge editing robust to distractor facts.

  30. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...

  31. Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

    cs.CL 2025-07 conditional novelty 5.0 of 10

    For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...

  32. How to Retrieve Examples in In-context Learning to Improve Conversational Emotion Recognition using Large Language Models?

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Retrieving a semantically similar example and voting over paraphrased versions of it improves conversational emotion recognition macro F1 over random in-context examples.

  33. Federated In-Context Learning: Iterative Refinement for Improved Answer Quality

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Fed-ICL iteratively refines QA answers via federated in-context learning with only label transmission, showing convergence on a linear attention model and gains on MMLU and TruthfulQA.

  34. MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning

    cs.AI 2025-05 conditional novelty 5.0 of 10

    MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.

  35. NER4all or Context is All You Need: Using LLMs for low-effort, high-performance NER on historical texts. A humanities informed approach

    cs.CL 2025-02 conditional novelty 5.0 of 10

    With context-rich prompts and persona modeling, ChatGPT-4o outperformed off-the-shelf spaCy and flair on named entity recognition for a 1921 German travel guide.

  36. Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL

    cs.CL 2024-11 reject novelty 5.0 of 10

    LPE-SQL improves text-to-SQL accuracy on BIRD by retrieving from dynamically grown correct and mistake notebooks, but its main gains come from feeding ground-truth answers into those notebooks during evaluation.

  37. GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

    cs.CL 2024-11 conditional novelty 5.0 of 10

    A knowledge-graph-based reinforcement learning policy selects and orders in-context examples for LLM prompts, reporting modest ROUGE and BLEU improvements.

  38. PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations using LLMs with Enhanced Safety, Fairness, and Controllability

    cs.CV 2025-09 reject novelty 4.0 of 10

    A framework paper that claims its VulnGuard prompt technique cuts harmful LLM outputs by 25-30% via theoretical bounds, without a real proof or empirical test.

  39. MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers

    cs.CL 2025-02 conditional novelty 4.0 of 10

    MultiQ&A automatically rephrases questions and crowdsources answers from LLM agents, showing gpt-3.5-turbo is fairly stable on extractive and multiple-choice QA but more variable on free-form answers.

  40. GeneSUM: Large Language Model-based Gene Summary Extraction

    q-bio.GN 2024-12 reject novelty 4.0 of 10

    A two-stage LLM pipeline that selects key sentences from gene literature via GO annotations and fine-tunes Gemma-7B to generate gene summaries, reporting large ROUGE gains that may be inflated by training/evaluation overlap.

  41. The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Sequentially choosing the most uncertain training question given previously chosen exemplars improves few-shot chain-of-thought accuracy by about 0.7 points on average over non-adaptive active prompting.

  42. Memory-Augmented Agent Training for Business Document Understanding

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Matrix iteratively refines an LLM agent's memory from training trajectories and outperforms prompting and vanilla agent baselines on private UBL invoice transport-reference extraction, though gains on the released ano...

  43. Labeling NIDS Rules with MITRE ATT&CK Techniques: Machine Learning vs. Large Language Models

    cs.CR 2024-12 conditional novelty 4.0 of 10

    On a dataset of 973 Snort rules, traditional ML models (SVM, F1 up to 0.87) outperformed ChatGPT, Claude, and Gemini (best F1 0.62) at labeling rules with MITRE ATT&CK techniques.

  44. Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop

    cs.CV 2024-11 conditional novelty 4.0 of 10

    Scene Copilot is a training-free pipeline that combines an LLM with retrieval over Infinigen's codebase and human-in-the-loop Blender editing to generate customized 3D scenes and videos from text prompts.

  45. From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

    cs.CL 2026-07 conditional novelty 3.0 of 10

    A question-type-specific LLM ensemble and multi-agent pipeline achieved competitive results on BioASQ 14b Task B, including first place in the factoid subtask of Batch 4.

  46. LLM-based event log analysis techniques: A survey

    cs.AI 2025-02 conditional novelty 3.0 of 10

    The paper organizes existing LLM event-log analysis research into a task taxonomy, identifies common limitations, and lists future research directions.

  47. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools