{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:C3UQ73X5FVMUCFFM2WFYUB7YGX","short_pith_number":"pith:C3UQ73X5","schema_version":"1.0","canonical_sha256":"16e90feefd2d594114acd58b8a07f835c9f30bd15dbc46ea45010449fea68f94","source":{"kind":"arxiv","id":"2302.09664","version":3},"attestation_state":"computed","paper":{"title":"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks.","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Lorenz Kuhn, Sebastian Farquhar, Yarin Gal","submitted_at":"2023-02-19T20:10:07Z","abstract_excerpt":"We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of \"semantic equivalence\" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf langu"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2302.09664","kind":"arxiv","version":3},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2023-02-19T20:10:07Z","cross_cats_sorted":["cs.AI","cs.LG"],"title_canon_sha256":"1b7a83c9d7816ddb61a493456a1a6422aebd2049c325545d3170787ad7c60bc9","abstract_canon_sha256":"bcf064bd4a2af44b0f301eea4b175098bd5e06adf2709724ad80d6592cab7b9d"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T06:01:17.352535Z","signature_b64":"MwOoyYmzQ4I9TD8VVmUQ4WL12q2OvnLXoP2OO+/HBxCj6NJJ7/yk+BAW/klH5aEq5kAO4dPd9y+wVfVbQtb3BQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"16e90feefd2d594114acd58b8a07f835c9f30bd15dbc46ea45010449fea68f94","last_reissued_at":"2026-07-05T06:01:17.352036Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T06:01:17.352036Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks.","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Lorenz Kuhn, Sebastian Farquhar, Yarin Gal","submitted_at":"2023-02-19T20:10:07Z","abstract_excerpt":"We introduce a method to measure uncertainty in large language models. For tasks like question answering, it is essential to know when we can trust the natural language outputs of foundation models. We show that measuring uncertainty in natural language is challenging because of \"semantic equivalence\" -- different sentences can mean the same thing. To overcome these challenges we introduce semantic entropy -- an entropy which incorporates linguistic invariances created by shared meanings. Our method is unsupervised, uses only a single model, and requires no modifications to off-the-shelf langu"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That semantic equivalence classes among generated sentences can be reliably identified in an unsupervised manner using the language model itself.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Semantic entropy improves uncertainty estimation in natural language generation by incorporating semantic equivalences, outperforming standard entropy baselines on predicting model accuracy for question answering.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"506d00e4b09b163fdbd56de459c33e6be089f4fbaddafc2620ba11600132d55c"},"source":{"id":"2302.09664","kind":"arxiv","version":3},"verdict":{"id":"ac1eb26f-3ff7-4881-90d9-4ffd392feff1","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-12T17:55:04.481439Z","strongest_claim":"In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.","one_line_summary":"Semantic entropy improves uncertainty estimation in natural language generation by incorporating semantic equivalences, outperforming standard entropy baselines on predicting model accuracy for question answering.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That semantic equivalence classes among generated sentences can be reliably identified in an unsupervised manner using the language model itself.","pith_extraction_headline":"Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2302.09664/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":27,"sample":[{"doi":"","year":1901,"title":"Language models are few-shot learners","work_id":"1bb0b0e2-1e9d-41e2-a9ab-62e7dab5daba","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"PaLM: Scaling Language Modeling with Pathways","work_id":"a94f3ef7-2c49-4445-93fe-6ec16aafd966","ref_index":2,"cited_arxiv_id":"2204.02311","is_internal_anchor":true},{"doi":"","year":2020,"title":"Calibration of pre-trained transformers","work_id":"68eb5e52-0b70-43bd-a1ed-f236780d6499","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"Unsupervised quality estimation for neural machine translation","work_id":"374906e2-d226-42b8-b266-192ea85010d1","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Uncertainty-aware ma- chine translation evaluation","work_id":"543c3044-cf4f-4d1e-b2a3-c41297ecbeba","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":27,"snapshot_sha256":"d14af88314b1d60ccc24f4f328603577f9eacf3868167a3d853b430c4a1da642","internal_anchors":7},"formal_canon":{"evidence_count":2,"snapshot_sha256":"46e24691b67ebd8ef7ec640ba4799c0fe60319e094aade8aa179ee1fbc69ad76"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2302.09664","created_at":"2026-07-05T06:01:17.352101+00:00"},{"alias_kind":"arxiv_version","alias_value":"2302.09664v3","created_at":"2026-07-05T06:01:17.352101+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2302.09664","created_at":"2026-07-05T06:01:17.352101+00:00"},{"alias_kind":"pith_short_12","alias_value":"C3UQ73X5FVMU","created_at":"2026-07-05T06:01:17.352101+00:00"},{"alias_kind":"pith_short_16","alias_value":"C3UQ73X5FVMUCFFM","created_at":"2026-07-05T06:01:17.352101+00:00"},{"alias_kind":"pith_short_8","alias_value":"C3UQ73X5","created_at":"2026-07-05T06:01:17.352101+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":85,"internal_anchor_count":85,"sample":[{"citing_arxiv_id":"2607.05721","citing_title":"SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2607.07670","citing_title":"Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale","ref_index":56,"is_internal_anchor":true},{"citing_arxiv_id":"2607.08065","citing_title":"When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2607.06327","citing_title":"Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24281","citing_title":"CALIBER: Calibrating Confidence Before and After Reasoning in Language Models","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24937","citing_title":"The Hitchhiker's Guide to Agentic AI: From Foundations to Systems","ref_index":163,"is_internal_anchor":true},{"citing_arxiv_id":"2606.20959","citing_title":"Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models","ref_index":59,"is_internal_anchor":true},{"citing_arxiv_id":"2606.20244","citing_title":"SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2606.19416","citing_title":"MortarBench: Evaluating Mortgage Loan Origination Agents","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2606.18068","citing_title":"Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2607.02182","citing_title":"Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2607.02423","citing_title":"Neuron-Aware Active Few-Shot Learning for LLMs","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2607.01612","citing_title":"Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2607.02186","citing_title":"UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2606.20662","citing_title":"Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2606.09800","citing_title":"FASE: Fast Adaptive Semantic Entropy for Code Quality","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2606.08508","citing_title":"ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies","ref_index":39,"is_internal_anchor":true},{"citing_arxiv_id":"2606.08277","citing_title":"Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2606.08088","citing_title":"ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learning","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2607.00972","citing_title":"Bayesian Uncertainty Propagation for Agentic RAG Pipelines: A Proof-of-Concept Study on Multi-Hop Question Answering","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2606.04727","citing_title":"EviRank: Evidence-Based Confidence Estimation for LLM-Based Ranking","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2606.04661","citing_title":"CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2606.03087","citing_title":"Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2606.01581","citing_title":"Agent System Operations: Categorization, Challenges, and Future Directions","ref_index":97,"is_internal_anchor":true},{"citing_arxiv_id":"2606.01850","citing_title":"Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction","ref_index":46,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX","json":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX.json","graph_json":"https://pith.science/api/pith-number/C3UQ73X5FVMUCFFM2WFYUB7YGX/graph.json","events_json":"https://pith.science/api/pith-number/C3UQ73X5FVMUCFFM2WFYUB7YGX/events.json","paper":"https://pith.science/paper/C3UQ73X5"},"agent_actions":{"view_html":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX","download_json":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX.json","view_paper":"https://pith.science/paper/C3UQ73X5","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2302.09664&json=true","fetch_graph":"https://pith.science/api/pith-number/C3UQ73X5FVMUCFFM2WFYUB7YGX/graph.json","fetch_events":"https://pith.science/api/pith-number/C3UQ73X5FVMUCFFM2WFYUB7YGX/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX/action/timestamp_anchor","attest_storage":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX/action/storage_attestation","attest_author":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX/action/author_attestation","sign_citation":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX/action/citation_signature","submit_replication":"https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX/action/replication_record"}},"created_at":"2026-07-05T06:01:17.352101+00:00","updated_at":"2026-07-05T06:01:17.352101+00:00"}