Pith Number

pith:C3UQ73X5

pith:2023:C3UQ73X5FVMUCFFM2WFYUB7YGX

not attested not anchored not stored refs resolved

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

Lorenz Kuhn, Sebastian Farquhar, Yarin Gal

Semantic entropy, which groups model outputs by shared meaning before measuring uncertainty, predicts answer accuracy more reliably than token-level entropy on question answering tasks.

arxiv:2302.09664 v3 · 2023-02-19 · cs.CL · cs.AI · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{C3UQ73X5FVMUCFFM2WFYUB7YGX}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

In comprehensive ablation studies we show that the semantic entropy is more predictive of model accuracy on question answering data sets than comparable baselines.

C2weakest assumption

That semantic equivalence classes among generated sentences can be reliably identified in an unsupervised manner using the language model itself.

C3one line summary

Semantic entropy improves uncertainty estimation in natural language generation by incorporating semantic equivalences, outperforming standard entropy baselines on predicting model accuracy for question answering.

References

27 extracted · 27 resolved · 7 Pith anchors

[1] Language models are few-shot learners 1901

[2] PaLM: Scaling Language Modeling with Pathways · arXiv:2204.02311

[3] Calibration of pre-trained transformers 2020

[4] Unsupervised quality estimation for neural machine translation 2023

[5] Uncertainty-aware ma- chine translation evaluation

Formal links

2 machine-checked theorem links

Cited by

85 papers in Pith

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

Receipt and verification

First computed	2026-07-05T06:01:17.352036Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

16e90feefd2d594114acd58b8a07f835c9f30bd15dbc46ea45010449fea68f94

Aliases

arxiv: 2302.09664 · arxiv_version: 2302.09664v3 · doi: 10.48550/arxiv.2302.09664 · pith_short_12: C3UQ73X5FVMU · pith_short_16: C3UQ73X5FVMUCFFM · pith_short_8: C3UQ73X5

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/C3UQ73X5FVMUCFFM2WFYUB7YGX \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 16e90feefd2d594114acd58b8a07f835c9f30bd15dbc46ea45010449fea68f94

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "bcf064bd4a2af44b0f301eea4b175098bd5e06adf2709724ad80d6592cab7b9d",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.LG"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2023-02-19T20:10:07Z",
    "title_canon_sha256": "1b7a83c9d7816ddb61a493456a1a6422aebd2049c325545d3170787ad7c60bc9"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2302.09664",
    "kind": "arxiv",
    "version": 3
  }
}