Pith. sign in

REVIEW 11 cited by

Prompting GPT-3 To Be Reliable

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.09150 v2 pith:RD6JRUX4 submitted 2022-10-17 cs.CL

Chenglei Si , Zhe Gan , Zhengyuan Yang , Shuohang Wang , Jianfeng Wang , Jordan Boyd-Graber , Lijuan Wang This is my paper · ORCID
classification cs.CL
keywords gpt-3reliabilitypromptinglanguagellmsbiasesfacetsimprove
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) show impressive abilities via few-shot prompting. Commercialized APIs such as OpenAI GPT-3 further increase their use in real-world language applications. However, the crucial problem of how to improve the reliability of GPT-3 is still under-explored. While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality. Our core contribution is to establish simple and effective prompts that improve GPT-3's reliability as it: 1) generalizes out-of-distribution, 2) balances demographic distribution and uses natural language instructions to reduce social biases, 3) calibrates output probabilities, and 4) updates the LLM's factual knowledge and reasoning chains. With appropriate prompts, GPT-3 is more reliable than smaller-scale supervised models on all these facets. We release all processed datasets, evaluation scripts, and model predictions. Our systematic empirical study not only sheds new insights on the reliability of prompting LLMs, but more importantly, our prompting strategies can help practitioners more reliably use LLMs like GPT-3.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations

    cs.CL 2026-05 conditional novelty 6.0 of 10

    EAACD reduces hallucination in MoE LLMs by contrasting predictions of high-reliability expert groups against hallucination-amplified low-reliability expert groups.

  2. Novobo: Supporting Teachers' Peer Learning of Instructional Gestures by Teaching a Mentee AI-Agent Together

    cs.HC 2025-05 conditional novelty 6.0 of 10

    Novobo, a teachable AI agent that acts as an apprentice teacher, helped 30 teachers in 10 sessions externalize and co-construct knowledge about instructional gestures through group discussion and embodied demonstration.

  3. Context-DPO: Aligning Language Models for Context-Faithfulness

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Context-DPO fine-tunes LLMs with direct preference optimization on counterfactual passages, yielding 35-280% context-faithfulness gains on its new ConFiQA benchmark.

  4. Biased or Flawed? Mitigating Stereotypes in Generative Language Models by Addressing Task-Specific Flaws

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Instruction tuning on general-purpose reading comprehension data reduces stereotype-consistent answers in ambiguous contexts by over 60%, suggesting much observed bias is task-specific comprehension failure.

  5. S2LPP: Small-to-Large Prompt Prediction across LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Small and large LLMs often share the same optimal prompt, so a small model can select effective prompts for a larger model at lower cost.

  6. FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering

    cs.CL 2025-04 conditional novelty 5.0 of 10

    FairSteer uses a linear probe to detect biased activations and adds a contrastively computed steering vector to shift generation toward unbiased answers, cutting bias across six LLMs without retraining.

  7. Generative AI Literacy: Twelve Defining Competencies

    cs.HC 2024-11 conditional novelty 5.0 of 10

    A literature-based proposal of twelve competencies for generative AI literacy, intended as a framework for education, policy, and assessment.

  8. Advertising in AI systems: Society must be vigilant

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Generative AI outputs will likely carry embedded commercial content, and the paper proposes design principles, provenance tracking, and two debiasing strategies to preserve transparency.

  9. How Knowledge Popularity Influences and Enhances LLM Knowledge Boundary Perception

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Entity popularity and entity co-occurrence in Wikipedia correlate with LLM QA accuracy, confidence, and calibration, and combining them with confidence improves answer-correctness prediction by 5.24% on average.

  10. Relative Bias: A Comparative Framework for Quantifying Bias in LLMs

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A model is 'relatively biased' when its responses deviate from the consensus of a baseline LLM set, and this deviation can be scored by embedding distances or LLM judges plus equivalence tests.

  11. The Generative AI Ethics Playbook

    cs.CY 2024-12 conditional novelty 4.0 of 10

    A structured playbook that collects existing guidance, checklists, and case studies to help generative AI practitioners identify and mitigate ethical harms across six lifecycle stages.

Pith tools