Pith. sign in

REVIEW 21 cited by

Calibrate Before Use: Improving Few-Shot Performance of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2102.09690 v2 pith:MIUIVO5A submitted 2021-02-19 cs.CL cs.LG

Tony Z. Zhao , Eric Wallace , Shi Feng , Dan Klein , Sameer Singh This is my paper · ORCID
classification cs.CLcs.LG
keywords prompttrainingexampleslanguagenearaccuracyacrossanswers
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 72 citations worldwide. Full citation record

  1. EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering

    cs.CL 2025-07 conditional novelty 7.0 of 10

    EsBBQ and CaBBQ are new Spanish and Catalan bias benchmarks for multiple-choice QA, built with survey-validated stereotypes from Spain and evaluated on 17 language models.

  2. Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting

    cs.AI 2025-10 conditional novelty 6.0 of 10

    An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.

  3. MedBench-IT: A Comprehensive Benchmark for Evaluating Large Language Models on Italian Medical Entrance Examinations

    cs.CL 2025-09 conditional novelty 6.0 of 10

    MedBench-IT collects 17,410 Italian medical entrance exam questions and reports model accuracy, response consistency, ordering bias, reasoning-prompt effects, and readability correlations.

  4. Characterizing Fitness Landscape Structures in Prompt Engineering

    cs.AI 2025-09 reject novelty 6.0 of 10

    Prompt fitness autocorrelation appears smooth under systematic enumeration but rugged with an intermediate-distance peak under novelty-driven sampling, yet the two analyses cover non-overlapping distance ranges.

  5. MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers

    cs.IR 2025-06 conditional novelty 6.0 of 10

    A zero-shot mixture of sparse, dense, and simulated human retrievers, weighted by pre- and post-retrieval geometry signals, beats individual small retrievers and 7B LLM retrievers on four scientific retrieval benchmarks.

  6. MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools

    cs.CL 2025-04 conditional novelty 6.0 of 10

    A classifier trained on layerwise decodes and BERTScore similarities to the final output gives better-calibrated confidence for tool calls and improves expected utility at medium and high risk levels.

  7. Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Output token count, observable through response timing, can reveal a user's target language or classification result with 70-87% accuracy in the authors' experiments.

  8. Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A translated temporal reasoning dataset plus a cross-lingual example retriever that outperforms semantic-alignment baselines on low-resource language temporal questions.

  9. AURA: Affordance-Understanding and Risk-aware Alignment Technique for Large Language Models

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    AURA uses step-level reward models, self-critique, and safety-aware decoding to reduce affordance-based safety failures in LLM outputs.

  10. StaICC: Standardized Evaluation for Classification Task in In-context Learning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    StaICC standardizes in-context classification evaluation with fixed prompts and splits, then measures 29 LMs and 10 inference methods under those fixed settings.

  11. From Words to Workflows: Automating Business Processes

    cs.AI 2024-12 conditional novelty 5.0 of 10

    Text2Workflow is a multi-prompt LLM system with human feedback that generates JSON workflows from natural language, scoring 71.3% average semantic accuracy on the authors' 60-request Process2JSON dataset, versus 64.2%...

  12. Does Prompt Formatting Have Any Impact on LLM Performance?

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Prompt format alone can shift GPT model scores by double-digit percentage points, with GPT-3.5 more affected than GPT-4 and no universally best template.

  13. (Towards) Scalable Reliable Automated Evaluation with Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.

  14. Fine-tuning on simulated data outperforms prompting for agent tone of voice

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Fine-tuning a 1B-parameter LLM on as few as 100 synthetically generated, readability-filtered samples achieved conversational tone more reliably than a verbose system prompt.

  15. Dynamic Context-Aware Prompt Recommendation for Domain-Specific AI Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A dynamic prompt recommendation system for skill-based security copilots combines retrieval, hierarchical skill selection, and telemetry-based ranking, reporting high usefulness in internal evaluations.

  16. Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.

  17. Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

    cs.AI 2025-05 accept novelty 4.0 of 10

    LLMs extend, rather than replace, classical social science methods, with a proposed three-tier bias framework for LLM-augmented surveys.

  18. Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection

    cs.CL 2025-05 reject novelty 4.0 of 10

    A hybrid Euclidean-distance and LLM-relevance example selector for few-shot sensor classification reports a small, statistically fragile gain over distance-only selection on a fatigue detection dataset.

  19. Leveraging Large Language Models for enzymatic reaction prediction and characterization

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Fine-tuned Llama-3.1 models can perform enzymatic reaction prediction tasks, and multitask learning improves forward and retrosynthesis over single-task training.

  20. LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models

    cs.CY 2025-05 reject novelty 4.0 of 10

    A proposed three-dimensional benchmark for LLM moral reasoning that combines MFQ, WVS, and moral dilemmas, but the reported model scores are not reproducible from the paper.

  21. Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture

    cs.CL 2025-02 reject novelty 3.0 of 10

    A retrieval-augmented encoder-decoder that injects 'in-context vectors' into latent states is presented, with claims of competing with much larger RAG models on three QA benchmarks.

Pith tools