Pith. sign in

REVIEW 29 cited by

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2104.08786 v2 pith:FPECB4L2 submitted 2021-04-18 cs.CL cs.AI

Yao Lu , Max Bartolo , Alastair Moore , Sebastian Riedel , Pontus Stenetorp This is my paper · ORCID
classification cs.CLcs.AI
keywords modelslanguagepermutationssamplesacrossdevelopmentfew-shotlarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are "fantastic" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 120 citations worldwide. Full citation record

  1. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  2. Visual Grounding in Zero-Shot Vision-Language Control

    cs.RO 2026-08 conditional novelty 6.0 of 10

    Input-ablation tests show most current VLMs are not visually grounded controllers, though a small symmetry-consensus ensemble works as a hazard monitor.

  3. Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy

    cs.SE 2026-03 conditional novelty 6.0 of 10

    MCP server faults form five empirical categories—server setting, server/tool configuration, server/host configuration, documentation, and general programming—confirmed by a 41-practitioner survey.

  4. Failures Are the Stepping Stones to Success: Enhancing Few-Shot In-Context Learning by Leveraging Negative Samples

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A retrieval method that uses semantically similar wrong answers as anchors to find corrective correct demonstrations improves few-shot in-context learning accuracy on seven reasoning datasets.

  5. Towards Compute-Optimal Many-Shot In-Context Learning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.

  6. LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra

    cs.MA 2025-07 reject novelty 6.0 of 10

    The LLM Economist framework couples persona-conditioned worker agents with an in-context RL planner to search US-bracket tax schedules, yet its Saez benchmark is derived from the planner's own solution and its headlin...

  7. Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Representation smoothness can be used to regularize training, stop early without validation labels, and guide active learning combined with parameter-efficient fine-tuning, reducing data and compute.

  8. LLMs are Bayesian, In Expectation, Not in Realization

    stat.ML 2025-07 conditional novelty 6.0 of 10

    Transformers can be Bayes-competitive in prequential log loss even when their predictive distributions are not invariant to example order, provided the cumulative predictive KL to the Bayesian reference stays small.

  9. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  10. Adaptive Task Vectors for Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Adaptive Task Vectors use a small model to generate query-specific steering vectors for frozen LLMs, reporting strong accuracy and generalization, though the theoretical equivalences to LoRA and Prefix-Tuning are not ...

  11. SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Large language models perform poorly on a new C-code vulnerability benchmark, indicating they rely on pattern matching rather than genuine reasoning.

  12. Learning to Select In-Context Demonstration Preferred by Large Language Model

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A generative preference-learning method trains a latent demonstration selector from LLM feedback and improves few-shot in-context learning performance on most of 19 benchmark datasets.

  13. The Role of Diversity in In-Context Learning for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Diversity-aware selection of in-context examples improves performance on complex and out-of-distribution tasks, though effect sizes are often modest.

  14. OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    OptiSeq selects the in-context example ordering whose output gets the highest zero-shot log-likelihood, improving few-shot accuracy by up to 10.5 points in tests on API sequencing and classification.

  15. UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    UGMathBench provides a 5,062-problem dynamic benchmark for undergraduate math reasoning and shows that leading LLMs solve all versions of only about half the problems.

  16. Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

    cs.CR 2026-07 conditional novelty 5.0 of 10

    A new benchmark of 2,742 real scam messages shows top LLMs reach about 64-65% micro-F1 and generalize to an unseen 59,991-sample proprietary set better than a fine-tuned BERT.

  17. Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

    cs.AI 2026-07 accept novelty 5.0 of 10

    Across 56 tasks, 9 model configurations, and 10,584 runs, LLM-generated skill files provided no reliable performance improvement over task-only prompting for data-science workflows.

  18. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...

  19. Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

    cs.CL 2025-07 conditional novelty 5.0 of 10

    For Flamingo-style models, increasing the number of in-context examples improves language coherence but degrades visual-text alignment, and similarity-based image retrieval inflates CIDEr scores by encouraging caption...

  20. Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Vision-language models improve little, often not at all, when given demonstrations, even when demonstrations contain explicit reasoning steps.

  21. MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning

    cs.AI 2025-05 conditional novelty 5.0 of 10

    MAPLE uses graph-influence scores to select and pseudo-label the most useful unlabeled examples, then adaptively chooses demonstrations per query, improving many-shot in-context learning with few human labels.

  22. StaICC: Standardized Evaluation for Classification Task in In-context Learning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    StaICC standardizes in-context classification evaluation with fixed prompts and splits, then measures 29 LMs and 10 inference methods under those fixed settings.

  23. DTKG: Dual-Track Knowledge Graph-Verified Reasoning Framework for Multi-Hop QA

    cs.AI 2025-10 conditional novelty 4.0 of 10

    DTKG routes multi-hop questions into either a fact-verification or knowledge-graph chain-reasoning branch, reporting modest accuracy gains on four QA benchmarks.

  24. Fine-tuning on simulated data outperforms prompting for agent tone of voice

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Fine-tuning a 1B-parameter LLM on as few as 100 synthetically generated, readability-filtered samples achieved conversational tone more reliably than a verbose system prompt.

  25. RAILS: Retrieval-Augmented Intelligence for Learning Software Development

    cs.SE 2025-06 reject novelty 4.0 of 10

    RAILS, a RAG pipeline with FAISS retrieval and a compile-error feedback loop, is reported to outperform baseline GPT-3.5-Turbo prompting on 78 Java import error cases.

  26. Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.

  27. Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

    cs.AI 2025-05 accept novelty 4.0 of 10

    LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.

  28. Label Anything: An Interpretable, High-Fidelity and Prompt-Free Annotator

    cs.RO 2025-02 reject novelty 4.0 of 10

    The paper claims a single-seed, prompt-free annotator with 99.99% mIoU, but the described inference pipeline bakes the seed's ground-truth label into the output for every image.

  29. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools