{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:33NV57HYMIFM5GWDBTWSEYIN2F","short_pith_number":"pith:33NV57HY","schema_version":"1.0","canonical_sha256":"dedb5efcf8620ace9ac30ced22610dd1616a0f2592cb05ab0854df3c2d44b3c6","source":{"kind":"arxiv","id":"2311.17035","version":1},"attestation_state":"computed","paper":{"title":"Scalable Extraction of Training Data from (Production) Language Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Adversaries can extract gigabytes of training data from language models including ChatGPT by querying them without prior knowledge of the data.","cross_cats":["cs.CL","cs.CR"],"primary_cat":"cs.LG","authors_text":"A. Feder Cooper, Christopher A. Choquette-Choo, Daphne Ippolito, Eric Wallace, Florian Tram\\`er, Jonathan Hayase, Katherine Lee, Matthew Jagielski, Milad Nasr, Nicholas Carlini","submitted_at":"2023-11-28T18:47:03Z","abstract_excerpt":"This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and em"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2311.17035","kind":"arxiv","version":1},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2023-11-28T18:47:03Z","cross_cats_sorted":["cs.CL","cs.CR"],"title_canon_sha256":"b92f16cf18c2856205cecdb2cb789e5f9b1896bee9511d819789558b5381838d","abstract_canon_sha256":"78c268ca3f6d957e3e6106181db95edd8ea82003d47c3b317c03666251909969"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:50.501823Z","signature_b64":"/npr72BKYo24VhkXrN1UhZrL8NbOkovXJqZSYiIpR/UV+NMxliW8mA9PkqwRWkCHXkIzpwnIGW9/TJaZbpftAw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"dedb5efcf8620ace9ac30ced22610dd1616a0f2592cb05ab0854df3c2d44b3c6","last_reissued_at":"2026-05-17T23:38:50.501353Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:50.501353Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Scalable Extraction of Training Data from (Production) Language Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Adversaries can extract gigabytes of training data from language models including ChatGPT by querying them without prior knowledge of the data.","cross_cats":["cs.CL","cs.CR"],"primary_cat":"cs.LG","authors_text":"A. Feder Cooper, Christopher A. Choquette-Choo, Daphne Ippolito, Eric Wallace, Florian Tram\\`er, Jonathan Hayase, Katherine Lee, Matthew Jagielski, Milad Nasr, Nicholas Carlini","submitted_at":"2023-11-28T18:47:03Z","abstract_excerpt":"This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and em"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the strings emitted by the models are verifiably present in the original training datasets rather than plausible generations, and that the divergence attack requires no prior knowledge of the training data.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Adversaries can scalably extract gigabytes of training data from open, semi-open, and closed language models via querying attacks, including a divergence method that increases extraction rates 150x on aligned models like ChatGPT.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Adversaries can extract gigabytes of training data from language models including ChatGPT by querying them without prior knowledge of the data.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"48ef0d7e0dee3ffea16d9c21860e9fffcd3b41cac47e054ce94eec5809dae039"},"source":{"id":"2311.17035","kind":"arxiv","version":1},"verdict":{"id":"8d2af25f-7bc9-44be-8b60-de34beec6922","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T18:56:53.144454Z","strongest_claim":"Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.","one_line_summary":"Adversaries can scalably extract gigabytes of training data from open, semi-open, and closed language models via querying attacks, including a divergence method that increases extraction rates 150x on aligned models like ChatGPT.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the strings emitted by the models are verifiably present in the original training datasets rather than plausible generations, and that the divergence attack requires no prior knowledge of the training data.","pith_extraction_headline":"Adversaries can extract gigabytes of training data from language models including ChatGPT by querying them without prior knowledge of the data."},"references":{"count":64,"sample":[{"doi":"","year":null,"title":"Sequential Good-Turing and the miss- ing species problem","work_id":"112d6175-296d-4c8d-84a9-d1d13c2fda86","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"M., F IRAT, O., ET AL","work_id":"54530589-394d-4124-899c-d420caebff29","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback","work_id":"a1f2574b-a899-4713-be60-c87ba332656c","ref_index":3,"cited_arxiv_id":"2204.05862","is_internal_anchor":true},{"doi":"","year":2022,"title":"Recon- structing training data with informed adversaries","work_id":"7050bd3e-e69c-4319-ad19-b233440b2051","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"A., P UROHIT , S., P RASHANTH , U","work_id":"df0775cf-cb9c-4447-80bd-c56c1598d731","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":64,"snapshot_sha256":"7a35e5421e9cd5535705fc85ce79224627a32da4d29cad60020d418f84950d4e","internal_anchors":5},"formal_canon":{"evidence_count":1,"snapshot_sha256":"739684a15a5a08e0c264aa4e7dd31b80c77e84fc23719c532cb7b031c76c2316"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2311.17035","created_at":"2026-05-17T23:38:50.501422+00:00"},{"alias_kind":"arxiv_version","alias_value":"2311.17035v1","created_at":"2026-05-17T23:38:50.501422+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2311.17035","created_at":"2026-05-17T23:38:50.501422+00:00"},{"alias_kind":"pith_short_12","alias_value":"33NV57HYMIFM","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"33NV57HYMIFM5GWD","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"33NV57HY","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":38,"internal_anchor_count":38,"sample":[{"citing_arxiv_id":"2406.11354","citing_title":"Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2409.18169","citing_title":"Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey","ref_index":110,"is_internal_anchor":true},{"citing_arxiv_id":"2501.02407","citing_title":"Towards the Anonymization of the Language Modeling","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2502.05206","citing_title":"Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety","ref_index":208,"is_internal_anchor":true},{"citing_arxiv_id":"2503.19786","citing_title":"Gemma 3 Technical Report","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2505.12546","citing_title":"Extracting memorized pieces of (copyrighted) books from open-weight language models","ref_index":192,"is_internal_anchor":true},{"citing_arxiv_id":"2505.16831","citing_title":"Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18879","citing_title":"ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20279","citing_title":"The Economics of Model Collapse: Equilibrium, Welfare, and Optimal Provenance Subsidies in Synthetic Data Markets","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16471","citing_title":"From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI","ref_index":84,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18879","citing_title":"ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18133","citing_title":"An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16776","citing_title":"Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17034","citing_title":"Privacy Policy Enforcement Guardrails for Data-Sensitive Retrieval-Augmented Generation","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2506.17185","citing_title":"A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset","ref_index":95,"is_internal_anchor":true},{"citing_arxiv_id":"2507.02974","citing_title":"InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2507.06261","citing_title":"Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities","ref_index":58,"is_internal_anchor":true},{"citing_arxiv_id":"2406.08464","citing_title":"Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing","ref_index":132,"is_internal_anchor":true},{"citing_arxiv_id":"2603.01059","citing_title":"GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant","ref_index":52,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03121","citing_title":"An Independent Safety Evaluation of Kimi K2.5","ref_index":96,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11170","citing_title":"Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data","ref_index":211,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27132","citing_title":"TRUST: A Framework for Decentralized AI Service v.0.1","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09990","citing_title":"Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09611","citing_title":"Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2604.16838","citing_title":"enclawed: A Configurable, Sector-Neutral Hardening Framework for Single-User AI Assistant Gateways","ref_index":32,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":1,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F","json":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F.json","graph_json":"https://pith.science/api/pith-number/33NV57HYMIFM5GWDBTWSEYIN2F/graph.json","events_json":"https://pith.science/api/pith-number/33NV57HYMIFM5GWDBTWSEYIN2F/events.json","paper":"https://pith.science/paper/33NV57HY"},"agent_actions":{"view_html":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F","download_json":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F.json","view_paper":"https://pith.science/paper/33NV57HY","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2311.17035&json=true","fetch_graph":"https://pith.science/api/pith-number/33NV57HYMIFM5GWDBTWSEYIN2F/graph.json","fetch_events":"https://pith.science/api/pith-number/33NV57HYMIFM5GWDBTWSEYIN2F/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F/action/timestamp_anchor","attest_storage":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F/action/storage_attestation","attest_author":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F/action/author_attestation","sign_citation":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F/action/citation_signature","submit_replication":"https://pith.science/pith/33NV57HYMIFM5GWDBTWSEYIN2F/action/replication_record"}},"created_at":"2026-05-17T23:38:50.501422+00:00","updated_at":"2026-05-17T23:38:50.501422+00:00"}