{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2021:IFPYM52XEP34DMTCINOO2PGOIV","short_pith_number":"pith:IFPYM52X","schema_version":"1.0","canonical_sha256":"415f86775723f7c1b262435ced3cce45520faf3c6b360f778301d450786f6caf","source":{"kind":"arxiv","id":"2109.07958","version":2},"attestation_state":"computed","paper":{"title":"TruthfulQA: Measuring How Models Mimic Human Falsehoods","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Language models repeat human misconceptions more as they get larger, according to a new benchmark of 817 questions.","cross_cats":["cs.AI","cs.CY","cs.LG"],"primary_cat":"cs.CL","authors_text":"Jacob Hilton, Owain Evans, Stephanie Lin","submitted_at":"2021-09-08T17:15:27Z","abstract_excerpt":"We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popul"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2109.07958","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2021-09-08T17:15:27Z","cross_cats_sorted":["cs.AI","cs.CY","cs.LG"],"title_canon_sha256":"2d11b4bac2c042daee54b5f544ebd32f698aef6baff5a03635cd35a7dc8c1516","abstract_canon_sha256":"d02dd95148184519ea9626e9d10f61710729a05170315c500810c89935071097"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T04:21:05.806044Z","signature_b64":"f2cVhGQHmROrxhFzXkP3GnApX/4Pg7lDNbi6398bMzg072iWriwfwvZopbP22FWLzcQrM+Zc1t97Cw4Ln1fBDg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"415f86775723f7c1b262435ced3cce45520faf3c6b360f778301d450786f6caf","last_reissued_at":"2026-07-05T04:21:05.805534Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T04:21:05.805534Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"TruthfulQA: Measuring How Models Mimic Human Falsehoods","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Language models repeat human misconceptions more as they get larger, according to a new benchmark of 817 questions.","cross_cats":["cs.AI","cs.CY","cs.LG"],"primary_cat":"cs.CL","authors_text":"Jacob Hilton, Owain Evans, Stephanie Lin","submitted_at":"2021-09-08T17:15:27Z","abstract_excerpt":"We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popul"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the 817 questions accurately capture misconceptions that models learn from training data rather than other factors, and that avoiding these specific false answers measures general truthfulness.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"A new benchmark reveals that language models including GPT-3 are truthful on only 58% of questions designed to elicit popular misconceptions, far below human performance of 94%, with larger models performing worse.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Language models repeat human misconceptions more as they get larger, according to a new benchmark of 817 questions.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"44c3daac860fc63f14fd075fa42f59abe2c8112143195c7d66688e69251d4043"},"source":{"id":"2109.07958","kind":"arxiv","version":2},"verdict":{"id":"1d13e202-a3d0-40b4-a310-18592e0aa91d","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-11T21:44:11.773355Z","strongest_claim":"The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution.","one_line_summary":"A new benchmark reveals that language models including GPT-3 are truthful on only 58% of questions designed to elicit popular misconceptions, far below human performance of 94%, with larger models performing worse.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the 817 questions accurately capture misconceptions that models learn from training data rather than other factors, and that avoiding these specific false answers measures general truthfulness.","pith_extraction_headline":"Language models repeat human misconceptions more as they get larger, according to a new benchmark of 817 questions."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2109.07958/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":16,"sample":[{"doi":"","year":2018,"title":"A General Language Assistant as a Laboratory for Alignment","work_id":"a43f9ea0-01be-47d5-b8ee-a1a9f73381c5","ref_index":1,"cited_arxiv_id":"2112.00861","is_internal_anchor":true},{"doi":"","year":2018,"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","ref_index":2,"cited_arxiv_id":"2107.03374","is_internal_anchor":true},{"doi":"","year":2021,"title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","work_id":"27eaec54-c105-4969-8188-da5f0fca3688","ref_index":3,"cited_arxiv_id":"2005.11401","is_internal_anchor":true},{"doi":"","year":2019,"title":"arXiv preprint arXiv:2105.11447 , year=","work_id":"4b0122d4-63a1-4e63-8f4b-e70d000c195c","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"Retrieval augmentation reduces hallucination in conversation","work_id":"764f4422-9533-439b-857b-07eab22d3719","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":16,"snapshot_sha256":"cab272164c1387d5866cf3ebe96b7e7e83746d422db67f5ed6a337f3cdbdf45f","internal_anchors":3},"formal_canon":{"evidence_count":2,"snapshot_sha256":"2351ebcf543ca2ee775a538c259060b93769847ed4fc11e41b67d852b26b44d7"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2109.07958","created_at":"2026-07-05T04:21:05.805603+00:00"},{"alias_kind":"arxiv_version","alias_value":"2109.07958v2","created_at":"2026-07-05T04:21:05.805603+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2109.07958","created_at":"2026-07-05T04:21:05.805603+00:00"},{"alias_kind":"pith_short_12","alias_value":"IFPYM52XEP34","created_at":"2026-07-05T04:21:05.805603+00:00"},{"alias_kind":"pith_short_16","alias_value":"IFPYM52XEP34DMTC","created_at":"2026-07-05T04:21:05.805603+00:00"},{"alias_kind":"pith_short_8","alias_value":"IFPYM52X","created_at":"2026-07-05T04:21:05.805603+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":100,"internal_anchor_count":100,"sample":[{"citing_arxiv_id":"2606.25476","citing_title":"A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation","ref_index":84,"is_internal_anchor":true},{"citing_arxiv_id":"2606.25097","citing_title":"Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2606.27274","citing_title":"BetXplain: An Explanation-Annotated Dataset for Detecting Manipulative Betting Advertisements on Social Media","ref_index":173,"is_internal_anchor":true},{"citing_arxiv_id":"2606.23364","citing_title":"Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2606.23276","citing_title":"Exposing the Illusion of Erasure in Knowledge Editing for LLMs","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2606.13111","citing_title":"M\\\"OVE: A Holistic LLM Benchmark for the German Public Sector","ref_index":81,"is_internal_anchor":true},{"citing_arxiv_id":"2606.12234","citing_title":"On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study","ref_index":114,"is_internal_anchor":true},{"citing_arxiv_id":"2606.11712","citing_title":"Substrate Asymmetry in User-Side Memory: A Diagnostic Framework","ref_index":57,"is_internal_anchor":true},{"citing_arxiv_id":"2606.07783","citing_title":"Evaluating RAG Reliability under Clean, Misleading, and Mixed Retrieval","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2606.03022","citing_title":"Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2606.02430","citing_title":"Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference","ref_index":47,"is_internal_anchor":true},{"citing_arxiv_id":"2605.28860","citing_title":"Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2606.28962","citing_title":"FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2606.29239","citing_title":"Breaking the Rounding Trap: Securing LLMs against Quantization-Conditioned Backdoors","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2605.25492","citing_title":"SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2605.25603","citing_title":"Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancy","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2605.27763","citing_title":"A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2605.26738","citing_title":"KARMA: Karma-Aligned Reward Model Adaptation","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2605.30076","citing_title":"UniSteer: Text-Guided Flow Matching in Activation Space for Versatile LLM Steering","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.29295","citing_title":"EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2606.00819","citing_title":"Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2606.24790","citing_title":"Grad Detect: Gradient-Based Hallucination Detection in LLMs","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22672","citing_title":"Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23189","citing_title":"Empirical Bayes Conformal Prediction for Vision and Language Models","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23262","citing_title":"Design and Report Benchmarks for Knowledge Work","ref_index":63,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV","json":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV.json","graph_json":"https://pith.science/api/pith-number/IFPYM52XEP34DMTCINOO2PGOIV/graph.json","events_json":"https://pith.science/api/pith-number/IFPYM52XEP34DMTCINOO2PGOIV/events.json","paper":"https://pith.science/paper/IFPYM52X"},"agent_actions":{"view_html":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV","download_json":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV.json","view_paper":"https://pith.science/paper/IFPYM52X","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2109.07958&json=true","fetch_graph":"https://pith.science/api/pith-number/IFPYM52XEP34DMTCINOO2PGOIV/graph.json","fetch_events":"https://pith.science/api/pith-number/IFPYM52XEP34DMTCINOO2PGOIV/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV/action/timestamp_anchor","attest_storage":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV/action/storage_attestation","attest_author":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV/action/author_attestation","sign_citation":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV/action/citation_signature","submit_replication":"https://pith.science/pith/IFPYM52XEP34DMTCINOO2PGOIV/action/replication_record"}},"created_at":"2026-07-05T04:21:05.805603+00:00","updated_at":"2026-07-05T04:21:05.805603+00:00"}