{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:CIHWLDKTKGCTCOIPMPEHC3WEPE","short_pith_number":"pith:CIHWLDKT","schema_version":"1.0","canonical_sha256":"120f658d53518531390f63c8716ec479191aa9c9eb53d212b17f5f5f4bd0e183","source":{"kind":"arxiv","id":"2304.01373","version":2},"attestation_state":"computed","paper":{"title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","license":"http://creativecommons.org/licenses/by-sa/4.0/","headline":"A suite of 16 language models trained on identical public data in the same order from 70M to 12B parameters enables direct tracking of how abilities emerge during training and across scales.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Aviya Skowron, Edward Raff, Eric Hallahan, Hailey Schoelkopf, Herbie Bradley, Kyle O'Brien, Lintang Sutawika, Mohammad Aflah Khan, Oskar van der Wal, Quentin Anthony, Shivanshu Purohit, Stella Biderman, USVSN Sai Prashanth","submitted_at":"2023-04-03T20:58:15Z","abstract_excerpt":"How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \\textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \\textit{Pythia} to facilitate research in many areas, and we present several case studies including novel res"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2304.01373","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by-sa/4.0/","primary_cat":"cs.CL","submitted_at":"2023-04-03T20:58:15Z","cross_cats_sorted":[],"title_canon_sha256":"0ace9f335158bb7146489e81651673b6809aa20162fa00ad6de66d9a408ff3db","abstract_canon_sha256":"cb96737b1f74b472e0e4c6f7b934e1843aa0112ab33accf7b0435c9c26ebe4ce"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:50.687412Z","signature_b64":"JGwXWmKkcLD1ujlKb7aBarfbY+JQ9rRuTVbmjxR/awVfc8Jb5/u8zbOYM4AXZk4UXhpwz2bXyB3s42IhaPdcCQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"120f658d53518531390f63c8716ec479191aa9c9eb53d212b17f5f5f4bd0e183","last_reissued_at":"2026-05-17T23:38:50.686760Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:50.686760Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","license":"http://creativecommons.org/licenses/by-sa/4.0/","headline":"A suite of 16 language models trained on identical public data in the same order from 70M to 12B parameters enables direct tracking of how abilities emerge during training and across scales.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Aviya Skowron, Edward Raff, Eric Hallahan, Hailey Schoelkopf, Herbie Bradley, Kyle O'Brien, Lintang Sutawika, Mohammad Aflah Khan, Oskar van der Wal, Quentin Anthony, Shivanshu Purohit, Stella Biderman, USVSN Sai Prashanth","submitted_at":"2023-04-03T20:58:15Z","abstract_excerpt":"How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \\textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \\textit{Pythia} to facilitate research in many areas, and we present several case studies including novel res"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We introduce Pythia, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That training all models on the exact same public data in identical order, combined with released checkpoints, will produce reproducible and generalizable insights into training dynamics without major unaccounted confounding from data selection or implementation details.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A suite of 16 language models trained on identical public data in the same order from 70M to 12B parameters enables direct tracking of how abilities emerge during training and across scales.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"83547dcd4e8a958e939ffff8dedf386c524724f13ea93f2bcbe1c828782a876b"},"source":{"id":"2304.01373","kind":"arxiv","version":2},"verdict":{"id":"e26e5881-5d37-4723-b9fd-accbad77771b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T17:39:56.487514Z","strongest_claim":"We introduce Pythia, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics.","one_line_summary":"Pythia releases 16 identically trained LLMs with full checkpoints and data tools to study training dynamics, scaling, memorization, and bias in language models.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That training all models on the exact same public data in identical order, combined with released checkpoints, will produce reproducible and generalizable insights into training dynamics without major unaccounted confounding from data selection or implementation details.","pith_extraction_headline":"A suite of 16 language models trained on identical public data in the same order from 70M to 12B parameters enables direct tracking of how abilities emerge during training and across scales."},"references":{"count":198,"sample":[{"doi":"","year":2022,"title":"J., Berenberg, D., Fisk, I., Zanichelli, N., Zhang, B., et al","work_id":"1662457e-d86b-4d61-b9b2-b0e7f45a1fde","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"GPT-NeoX : Large scale autoregressive language modeling in PyTorch , 8 2021","work_id":"b3387a01-83d4-450f-a38c-e5da600946f1","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"H., Sanh, V., Yong, Z.-X., Webson, A., Raffel, C., Nayak, N","work_id":"1ecb7ba1-5877-47bf-a9ba-6fb99718e7a5","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"S., Sutawika, L., Purohit, S., Schoelkopf, H., Anthony, Q., and Raff, E","work_id":"82be10dc-2bc8-48d8-b059-ef2354881ed7","ref_index":6,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"GPT-Neo : Large scale autoregressive language modeling with Mesh-TensorFlow","work_id":"b60b3a29-5cfa-4f5c-ae16-d8eddcde0dfa","ref_index":8,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":198,"snapshot_sha256":"9258ef03a0a53c74a34a5cf88a359acc06333b3edbed47318973f1e9136ac0f8","internal_anchors":27},"formal_canon":{"evidence_count":2,"snapshot_sha256":"e9be58d49d4d7cea3017eaac2aa9a5fe1520d4e23dfccb19040ab8b6bbe401e3"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2304.01373","created_at":"2026-05-17T23:38:50.686862+00:00"},{"alias_kind":"arxiv_version","alias_value":"2304.01373v2","created_at":"2026-05-17T23:38:50.686862+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2304.01373","created_at":"2026-05-17T23:38:50.686862+00:00"},{"alias_kind":"pith_short_12","alias_value":"CIHWLDKTKGCT","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"CIHWLDKTKGCTCOIP","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"CIHWLDKT","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":29,"internal_anchor_count":29,"sample":[{"citing_arxiv_id":"2605.23901","citing_title":"LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2309.10305","citing_title":"Baichuan 2: Open Large-scale Language Models","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2502.12120","citing_title":"LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18789","citing_title":"Features have life history. And we should care","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2509.18218","citing_title":"Similarity Field Theory: A Mathematical Framework for Intelligence","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2509.20265","citing_title":"Failure Modes of Maximum Entropy RLHF","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2305.16264","citing_title":"Scaling Data-Constrained Language Models","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2602.04572","citing_title":"From Competition to Collaboration: Designing Sustainable Mechanisms Between LLMs and Online Forums","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2305.14233","citing_title":"Enhancing Chat Language Models by Scaling High-quality Instructional Conversations","ref_index":231,"is_internal_anchor":true},{"citing_arxiv_id":"2306.01116","citing_title":"The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2306.08543","citing_title":"MiniLLM: On-Policy Distillation of Large Language Models","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2502.05171","citing_title":"Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10504","citing_title":"Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2407.21787","citing_title":"Large Language Monkeys: Scaling Inference Compute with Repeated Sampling","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08740","citing_title":"Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06366","citing_title":"Layer Collapse in Diffusion Language Models","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2605.00195","citing_title":"Diversity in Large Language Models under Supervised Fine-Tuning","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2305.14314","citing_title":"QLoRA: Efficient Finetuning of Quantized LLMs","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2605.02968","citing_title":"Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiency","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2604.11119","citing_title":"DDO-RM: Distribution-Level Policy Improvement after Reward Learning","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07105","citing_title":"Theoretical Limits of Language Model Alignment","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2305.06161","citing_title":"StarCoder: may the source be with you!","ref_index":213,"is_internal_anchor":true},{"citing_arxiv_id":"2303.18223","citing_title":"A Survey of Large Language Models","ref_index":98,"is_internal_anchor":true},{"citing_arxiv_id":"2604.13950","citing_title":"Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2604.18239","citing_title":"Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner","ref_index":1,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE","json":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE.json","graph_json":"https://pith.science/api/pith-number/CIHWLDKTKGCTCOIPMPEHC3WEPE/graph.json","events_json":"https://pith.science/api/pith-number/CIHWLDKTKGCTCOIPMPEHC3WEPE/events.json","paper":"https://pith.science/paper/CIHWLDKT"},"agent_actions":{"view_html":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE","download_json":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE.json","view_paper":"https://pith.science/paper/CIHWLDKT","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2304.01373&json=true","fetch_graph":"https://pith.science/api/pith-number/CIHWLDKTKGCTCOIPMPEHC3WEPE/graph.json","fetch_events":"https://pith.science/api/pith-number/CIHWLDKTKGCTCOIPMPEHC3WEPE/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE/action/timestamp_anchor","attest_storage":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE/action/storage_attestation","attest_author":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE/action/author_attestation","sign_citation":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE/action/citation_signature","submit_replication":"https://pith.science/pith/CIHWLDKTKGCTCOIPMPEHC3WEPE/action/replication_record"}},"created_at":"2026-05-17T23:38:50.686862+00:00","updated_at":"2026-05-17T23:38:50.686862+00:00"}