{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:BFPBRD4FKNT44DNQQDW5NDPNQJ","short_pith_number":"pith:BFPBRD4F","schema_version":"1.0","canonical_sha256":"095e188f855367ce0db080edd68ded82449f590a6aed7fe6047344d1874fd1e7","source":{"kind":"arxiv","id":"2205.14334","version":2},"attestation_state":"computed","paper":{"title":"Teaching Models to Express Their Uncertainty in Words","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"GPT-3 can learn to state its own uncertainty in natural language, and those statements map to well-calibrated probabilities.","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Jacob Hilton, Owain Evans, Stephanie Lin","submitted_at":"2022-05-28T05:02:31Z","abstract_excerpt":"We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g. \"90% confidence\" or \"high confidence\"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own ans"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2205.14334","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2022-05-28T05:02:31Z","cross_cats_sorted":["cs.AI","cs.LG"],"title_canon_sha256":"c0ee3606ce5cf484b933e412df2861c63b35d8078aa38ec6fbdc9c3ed3ce0a0e","abstract_canon_sha256":"77c7c1f6c18f4d94025f67e8ed0712faa7b963a1b015c5b72a76f2e1043da9fa"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:47.109448Z","signature_b64":"4C5vAgT8mYqXprLp1pl8QU8Bo3TpukYXRfKme85m6vIzRXAi+AahHTaaGdS6c05wnVfzEMVpAk1N5k6HiWPcCw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"095e188f855367ce0db080edd68ded82449f590a6aed7fe6047344d1874fd1e7","last_reissued_at":"2026-05-17T23:38:47.109011Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:47.109011Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Teaching Models to Express Their Uncertainty in Words","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"GPT-3 can learn to state its own uncertainty in natural language, and those statements map to well-calibrated probabilities.","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Jacob Hilton, Owain Evans, Stephanie Lin","submitted_at":"2022-05-28T05:02:31Z","abstract_excerpt":"We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g. \"90% confidence\" or \"high confidence\"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own ans"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. These levels map to probabilities that are well calibrated.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the verbalized confidence levels reflect the model's actual epistemic uncertainty rather than surface-level imitation of training examples or prompt patterns.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"GPT-3 can learn to express well-calibrated uncertainty about its answers using natural language phrases rather than logits.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"GPT-3 can learn to state its own uncertainty in natural language, and those statements map to well-calibrated probabilities.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"4c87f58e0363e4d908e834f650a08065ddeea34a46d0fa6f545f5840ae73d128"},"source":{"id":"2205.14334","kind":"arxiv","version":2},"verdict":{"id":"fe36c0ab-3d73-403c-bdcf-470acbce34fc","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T17:31:22.258256Z","strongest_claim":"a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. These levels map to probabilities that are well calibrated.","one_line_summary":"GPT-3 can learn to express well-calibrated uncertainty about its answers using natural language phrases rather than logits.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the verbalized confidence levels reflect the model's actual epistemic uncertainty rather than surface-level imitation of training examples or prompt patterns.","pith_extraction_headline":"GPT-3 can learn to state its own uncertainty in natural language, and those statements map to well-calibrated probabilities."},"references":{"count":24,"sample":[{"doi":"","year":null,"title":"A General Language Assistant as a Laboratory for Alignment","work_id":"a43f9ea0-01be-47d5-b8ee-a1a9f73381c5","ref_index":1,"cited_arxiv_id":"2112.00861","is_internal_anchor":true},{"doi":"","year":2022,"title":"https://www.gwern.net/GPT-3-nonfiction# calibration, Last accessed on 2022-04-24. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav ","work_id":"1491d240-ec27-49af-8cc5-dee31d7f13a4","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"PaLM: Scaling Language Modeling with Pathways","work_id":"a94f3ef7-2c49-4445-93fe-6ec16aafd966","ref_index":3,"cited_arxiv_id":"2204.02311","is_internal_anchor":true},{"doi":"","year":2022,"title":"Gabriela Csurka","work_id":"37812da7-1273-4dca-9164-57d9360ab2d1","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2020,"title":"arXiv preprint arXiv:1702.05374 (2017) https://doi.org/10.1007/ 978-3-319-58347-1 1","work_id":"c5eee634-1361-4941-9ae4-4ab25ce09055","ref_index":5,"cited_arxiv_id":"1702.05374","is_internal_anchor":true}],"resolved_work":24,"snapshot_sha256":"f18db649e4fff15d4787a811d4b61750caf6d961111d22bdb91863bd3c0c0e9c","internal_anchors":7},"formal_canon":{"evidence_count":1,"snapshot_sha256":"ee5d5e3b1d89aa2b97ef6f7462c695e3be1b375c2522df17f8df98d4378c2992"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2205.14334","created_at":"2026-05-17T23:38:47.109079+00:00"},{"alias_kind":"arxiv_version","alias_value":"2205.14334v2","created_at":"2026-05-17T23:38:47.109079+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2205.14334","created_at":"2026-05-17T23:38:47.109079+00:00"},{"alias_kind":"pith_short_12","alias_value":"BFPBRD4FKNT4","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"BFPBRD4FKNT44DNQ","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"BFPBRD4F","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":29,"internal_anchor_count":29,"sample":[{"citing_arxiv_id":"2305.09617","citing_title":"Towards Expert-Level Medical Question Answering with Large Language Models","ref_index":51,"is_internal_anchor":true},{"citing_arxiv_id":"2409.00084","citing_title":"Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18792","citing_title":"Trust or Abstain? A Self-Aware RAG Approach","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.14038","citing_title":"Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19220","citing_title":"Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering","ref_index":131,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19344","citing_title":"Retrieval-Augmented Linguistic Calibration","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17324","citing_title":"ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents","ref_index":101,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16824","citing_title":"Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2505.23912","citing_title":"LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2506.00886","citing_title":"Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary","ref_index":74,"is_internal_anchor":true},{"citing_arxiv_id":"2508.16846","citing_title":"BASIL: Bayesian Assessment of Sycophancy in LLMs","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2308.05374","citing_title":"Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment","ref_index":100,"is_internal_anchor":true},{"citing_arxiv_id":"2309.14525","citing_title":"Aligning Large Multimodal Models with Factually Augmented RLHF","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2411.04368","citing_title":"Measuring short-form factuality in large language models","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.14038","citing_title":"Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03216","citing_title":"BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11436","citing_title":"Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11235","citing_title":"Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2302.09664","citing_title":"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2604.25591","citing_title":"Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models","ref_index":72,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06308","citing_title":"Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.01428","citing_title":"Hallucinations Undermine Trust; Metacognition is a Way Forward","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2605.01853","citing_title":"Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2502.18864","citing_title":"Towards an AI co-scientist","ref_index":133,"is_internal_anchor":true},{"citing_arxiv_id":"2604.10718","citing_title":"SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?","ref_index":28,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":1,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ","json":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ.json","graph_json":"https://pith.science/api/pith-number/BFPBRD4FKNT44DNQQDW5NDPNQJ/graph.json","events_json":"https://pith.science/api/pith-number/BFPBRD4FKNT44DNQQDW5NDPNQJ/events.json","paper":"https://pith.science/paper/BFPBRD4F"},"agent_actions":{"view_html":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ","download_json":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ.json","view_paper":"https://pith.science/paper/BFPBRD4F","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2205.14334&json=true","fetch_graph":"https://pith.science/api/pith-number/BFPBRD4FKNT44DNQQDW5NDPNQJ/graph.json","fetch_events":"https://pith.science/api/pith-number/BFPBRD4FKNT44DNQQDW5NDPNQJ/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ/action/timestamp_anchor","attest_storage":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ/action/storage_attestation","attest_author":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ/action/author_attestation","sign_citation":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ/action/citation_signature","submit_replication":"https://pith.science/pith/BFPBRD4FKNT44DNQQDW5NDPNQJ/action/replication_record"}},"created_at":"2026-05-17T23:38:47.109079+00:00","updated_at":"2026-05-17T23:38:47.109079+00:00"}