{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:ZMRRB3PSO4E7GT4ORV74FBG2RU","short_pith_number":"pith:ZMRRB3PS","schema_version":"1.0","canonical_sha256":"cb2310edf27709f34f8e8d7fc284da8d1f3d3e20fd073e24bb6c12bae24c5af3","source":{"kind":"arxiv","id":"2402.07841","version":2},"attestation_state":"computed","paper":{"title":"Do Membership Inference Attacks Work on Large Language Models?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Anshuman Suri, David Evans, Hannaneh Hajishirzi, Luke Zettlemoyer, Michael Duan, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Yejin Choi, Yulia Tsvetkov","submitted_at":"2024-02-12T17:52:05Z","abstract_excerpt":"Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Our further analyses reveal that this poor performanc"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2402.07841","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-02-12T17:52:05Z","cross_cats_sorted":[],"title_canon_sha256":"861a9e1d1df72f4dfb4c933796dc19938f23ec6d03d167c264ab01bac017dfc7","abstract_canon_sha256":"951e7462aa263816d13b11d319eb0993aa9df36a2c7473d0ca36bb53fcf923b2"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:07:26.285627Z","signature_b64":"2G6qYv25OT5UVC1XPB/MH2PXauyGoCBW4+6dB8aml7qrlPJ9nRnFzVWRc5BslIKBSxGzqvx6VFj/XO4PUCfEAA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"cb2310edf27709f34f8e8d7fc284da8d1f3d3e20fd073e24bb6c12bae24c5af3","last_reissued_at":"2026-07-05T09:07:26.285162Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:07:26.285162Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Do Membership Inference Attacks Work on Large Language Models?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Anshuman Suri, David Evans, Hannaneh Hajishirzi, Luke Zettlemoyer, Michael Duan, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Yejin Choi, Yulia Tsvetkov","submitted_at":"2024-02-12T17:52:05Z","abstract_excerpt":"Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Our further analyses reveal that this poor performanc"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2402.07841","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2402.07841/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2402.07841","created_at":"2026-07-05T09:07:26.285235+00:00"},{"alias_kind":"arxiv_version","alias_value":"2402.07841v2","created_at":"2026-07-05T09:07:26.285235+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2402.07841","created_at":"2026-07-05T09:07:26.285235+00:00"},{"alias_kind":"pith_short_12","alias_value":"ZMRRB3PSO4E7","created_at":"2026-07-05T09:07:26.285235+00:00"},{"alias_kind":"pith_short_16","alias_value":"ZMRRB3PSO4E7GT4O","created_at":"2026-07-05T09:07:26.285235+00:00"},{"alias_kind":"pith_short_8","alias_value":"ZMRRB3PS","created_at":"2026-07-05T09:07:26.285235+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":17,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.23030","citing_title":"Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models","ref_index":21,"is_internal_anchor":false},{"citing_arxiv_id":"2607.01686","citing_title":"WARP: Weight-Space Analysis for Recovering Training Data Portfolios","ref_index":15,"is_internal_anchor":false},{"citing_arxiv_id":"2606.10091","citing_title":"SoK: Colluding Adversaries in Machine Learning Pipelines","ref_index":28,"is_internal_anchor":false},{"citing_arxiv_id":"2606.06946","citing_title":"Auditing Training Data in Domain-adapted LLMs: LoRA-MINT","ref_index":36,"is_internal_anchor":false},{"citing_arxiv_id":"2607.00325","citing_title":"Watermarking for Proprietary Dataset Protection","ref_index":3,"is_internal_anchor":false},{"citing_arxiv_id":"2606.28479","citing_title":"Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTs","ref_index":15,"is_internal_anchor":false},{"citing_arxiv_id":"2605.24079","citing_title":"TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs","ref_index":14,"is_internal_anchor":false},{"citing_arxiv_id":"2605.27825","citing_title":"MRMMIA: Membership Inference Attacks on Memory in Chat Agents","ref_index":7,"is_internal_anchor":false},{"citing_arxiv_id":"2605.29202","citing_title":"Auditing Training Data in Generative Music Models via Black-Box Membership Inference","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2605.16776","citing_title":"Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning","ref_index":88,"is_internal_anchor":false},{"citing_arxiv_id":"2506.06057","citing_title":"Hey, That's My Data! Token-Only Dataset Inference in Large Language Models","ref_index":11,"is_internal_anchor":false},{"citing_arxiv_id":"2512.22753","citing_title":"From Rookie to Expert: Manipulating LLMs for Automated Vulnerability Exploitation in Enterprise Software","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2605.12574","citing_title":"DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction","ref_index":21,"is_internal_anchor":false},{"citing_arxiv_id":"2604.03199","citing_title":"Learning the Signature of Memorization in Autoregressive Language Models","ref_index":7,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06423","citing_title":"Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models","ref_index":12,"is_internal_anchor":false},{"citing_arxiv_id":"2604.12342","citing_title":"CoLA: A Choice Leakage Attack Framework to Expose Privacy Risks in Subset Training","ref_index":1,"is_internal_anchor":false},{"citing_arxiv_id":"2605.07878","citing_title":"Black-box model classification under the discriminative factorization","ref_index":9,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU","json":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU.json","graph_json":"https://pith.science/api/pith-number/ZMRRB3PSO4E7GT4ORV74FBG2RU/graph.json","events_json":"https://pith.science/api/pith-number/ZMRRB3PSO4E7GT4ORV74FBG2RU/events.json","paper":"https://pith.science/paper/ZMRRB3PS"},"agent_actions":{"view_html":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU","download_json":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU.json","view_paper":"https://pith.science/paper/ZMRRB3PS","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2402.07841&json=true","fetch_graph":"https://pith.science/api/pith-number/ZMRRB3PSO4E7GT4ORV74FBG2RU/graph.json","fetch_events":"https://pith.science/api/pith-number/ZMRRB3PSO4E7GT4ORV74FBG2RU/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU/action/timestamp_anchor","attest_storage":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU/action/storage_attestation","attest_author":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU/action/author_attestation","sign_citation":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU/action/citation_signature","submit_replication":"https://pith.science/pith/ZMRRB3PSO4E7GT4ORV74FBG2RU/action/replication_record"}},"created_at":"2026-07-05T09:07:26.285235+00:00","updated_at":"2026-07-05T09:07:26.285235+00:00"}