Pith Number

pith:IFPYM52X

pith:2021:IFPYM52XEP34DMTCINOO2PGOIV

not attested not anchored not stored refs resolved

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Jacob Hilton, Owain Evans, Stephanie Lin

Language models repeat human misconceptions more as they get larger, according to a new benchmark of 817 questions.

arxiv:2109.07958 v2 · 2021-09-08 · cs.CL · cs.AI · cs.CY · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{IFPYM52XEP34DMTCINOO2PGOIV}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution.

C2weakest assumption

That the 817 questions accurately capture misconceptions that models learn from training data rather than other factors, and that avoiding these specific false answers measures general truthfulness.

C3one line summary

A new benchmark reveals that language models including GPT-3 are truthful on only 58% of questions designed to elicit popular misconceptions, far below human performance of 94%, with larger models performing worse.

References

16 extracted · 16 resolved · 3 Pith anchors

[1] A General Language Assistant as a Laboratory for Alignment 2018 · arXiv:2112.00861

[2] Evaluating Large Language Models Trained on Code 2018 · arXiv:2107.03374

[3] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks 2021 · arXiv:2005.11401

[4] arXiv preprint arXiv:2105.11447 , year= 2019

[5] Retrieval augmentation reduces hallucination in conversation 2021

Formal links

2 machine-checked theorem links

Cited by

100 papers in Pith

A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

BetXplain: An Explanation-Annotated Dataset for Detecting Manipulative Betting Advertisements on Social Media

Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime

Exposing the Illusion of Erasure in Knowledge Editing for LLMs

Receipt and verification

First computed	2026-07-05T04:21:05.805534Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

415f86775723f7c1b262435ced3cce45520faf3c6b360f778301d450786f6caf

Aliases

arxiv: 2109.07958 · arxiv_version: 2109.07958v2 · doi: 10.48550/arxiv.2109.07958 · pith_short_12: IFPYM52XEP34 · pith_short_16: IFPYM52XEP34DMTC · pith_short_8: IFPYM52X

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 415f86775723f7c1b262435ced3cce45520faf3c6b360f778301d450786f6caf

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "d02dd95148184519ea9626e9d10f61710729a05170315c500810c89935071097",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.CY",
      "cs.LG"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2021-09-08T17:15:27Z",
    "title_canon_sha256": "2d11b4bac2c042daee54b5f544ebd32f698aef6baff5a03635cd35a7dc8c1516"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2109.07958",
    "kind": "arxiv",
    "version": 2
  }
}