{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:25UDD36X2UP2PB7GW72NPMPWYQ","short_pith_number":"pith:25UDD36X","schema_version":"1.0","canonical_sha256":"d76831efd7d51fa787e6b7f4d7b1f6c40832234cfdc3703561f95d2113407665","source":{"kind":"arxiv","id":"2408.11796","version":4},"attestation_state":"computed","paper":{"title":"LLM Pruning and Distillation in Practice: The Minitron Approach","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Ameya Sunil Mahabaleshwarkar, Ashwath Aithal, Bryan Catanzaro, Chenhan Yu, Daniel Korzekwa, Gerald Shen, Hayley Ross, Jan Kautz, Jiaqi Zeng, Marcin Chochowski, Mohammad Shoeybi, Mostofa Patwary, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pavlo Molchanov, Raviraj Joshi, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Shizhe Diao, Wei-Chun Chen, Yoshi Suhara, Zijia Chen","submitted_at":"2024-08-21T17:38:48Z","abstract_excerpt":"We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2408.11796","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-08-21T17:38:48Z","cross_cats_sorted":["cs.AI","cs.LG"],"title_canon_sha256":"b835fe9fdb3eb7bfb2261ed4fd9972426a6f5c6e906a9f830a4a4c8256b446f2","abstract_canon_sha256":"b1581486d401edb19353b81fc0f17e19f7e0eb064802a4ca050e3263ca723db5"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:46:29.805799Z","signature_b64":"0ZjiHf9QM+5us32lMgS4+RyPFcV73bG7aF7ED7Pe6vPYKK5y64QKMzM1vtkBCCNcq+tj9NFc+7q7YUZdqMVjBg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"d76831efd7d51fa787e6b7f4d7b1f6c40832234cfdc3703561f95d2113407665","last_reissued_at":"2026-07-05T09:46:29.805196Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:46:29.805196Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"LLM Pruning and Distillation in Practice: The Minitron Approach","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Ameya Sunil Mahabaleshwarkar, Ashwath Aithal, Bryan Catanzaro, Chenhan Yu, Daniel Korzekwa, Gerald Shen, Hayley Ross, Jan Kautz, Jiaqi Zeng, Marcin Chochowski, Mohammad Shoeybi, Mostofa Patwary, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pavlo Molchanov, Raviraj Joshi, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Shizhe Diao, Wei-Chun Chen, Yoshi Suhara, Zijia Chen","submitted_at":"2024-08-21T17:38:48Z","abstract_excerpt":"We present a comprehensive report on compressing the Llama 3.1 8B and Mistral NeMo 12B models to 4B and 8B parameters, respectively, using pruning and distillation. We explore two distinct pruning strategies: (1) depth pruning and (2) joint hidden/attention/MLP (width) pruning, and evaluate the results on common benchmarks from the LM Evaluation Harness. The models are then aligned with NeMo Aligner and tested in instruct-tuned versions. This approach produces a compelling 4B model from Llama 3.1 8B and a state-of-the-art Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2408.11796","kind":"arxiv","version":4},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2408.11796/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2408.11796","created_at":"2026-07-05T09:46:29.805260+00:00"},{"alias_kind":"arxiv_version","alias_value":"2408.11796v4","created_at":"2026-07-05T09:46:29.805260+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2408.11796","created_at":"2026-07-05T09:46:29.805260+00:00"},{"alias_kind":"pith_short_12","alias_value":"25UDD36X2UP2","created_at":"2026-07-05T09:46:29.805260+00:00"},{"alias_kind":"pith_short_16","alias_value":"25UDD36X2UP2PB7G","created_at":"2026-07-05T09:46:29.805260+00:00"},{"alias_kind":"pith_short_8","alias_value":"25UDD36X","created_at":"2026-07-05T09:46:29.805260+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":20,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.26530","citing_title":"DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models","ref_index":35,"is_internal_anchor":false},{"citing_arxiv_id":"2606.14150","citing_title":"Small LLMs: Pruning vs. Training from Scratch","ref_index":8,"is_internal_anchor":false},{"citing_arxiv_id":"2606.07819","citing_title":"Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression","ref_index":24,"is_internal_anchor":false},{"citing_arxiv_id":"2606.04063","citing_title":"LLM Compression with Jointly Optimizing Architectural and Quantization choices","ref_index":23,"is_internal_anchor":false},{"citing_arxiv_id":"2606.14150","citing_title":"Small LLMs: Pruning vs. Training from Scratch","ref_index":8,"is_internal_anchor":false},{"citing_arxiv_id":"2606.26530","citing_title":"DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models","ref_index":35,"is_internal_anchor":false},{"citing_arxiv_id":"2605.28207","citing_title":"Pruning and Distilling Mixture-of-Experts into Dense Language Models","ref_index":30,"is_internal_anchor":false},{"citing_arxiv_id":"2606.07604","citing_title":"Contribution Weights: A Geometrical Analysis of Self-Attention Transformers","ref_index":91,"is_internal_anchor":false},{"citing_arxiv_id":"2512.12744","citing_title":"Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity","ref_index":70,"is_internal_anchor":false},{"citing_arxiv_id":"2510.08008","citing_title":"Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts","ref_index":19,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21171","citing_title":"FTerViT: Fully Ternary Vision Transformer","ref_index":36,"is_internal_anchor":false},{"citing_arxiv_id":"2506.03610","citing_title":"Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games","ref_index":69,"is_internal_anchor":false},{"citing_arxiv_id":"2506.12876","citing_title":"MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs","ref_index":21,"is_internal_anchor":false},{"citing_arxiv_id":"2512.20856","citing_title":"NVIDIA Nemotron 3: Efficient and Open Intelligence","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2512.12744","citing_title":"Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity","ref_index":70,"is_internal_anchor":false},{"citing_arxiv_id":"2602.01997","citing_title":"On the Limits of Layer Pruning for Generative Reasoning in Large Language Models","ref_index":25,"is_internal_anchor":false},{"citing_arxiv_id":"2406.08464","citing_title":"Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing","ref_index":137,"is_internal_anchor":false},{"citing_arxiv_id":"2601.08584","citing_title":"Ministral 3","ref_index":24,"is_internal_anchor":false},{"citing_arxiv_id":"2604.24380","citing_title":"Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency","ref_index":66,"is_internal_anchor":false},{"citing_arxiv_id":"2604.10627","citing_title":"Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment","ref_index":28,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ","json":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ.json","graph_json":"https://pith.science/api/pith-number/25UDD36X2UP2PB7GW72NPMPWYQ/graph.json","events_json":"https://pith.science/api/pith-number/25UDD36X2UP2PB7GW72NPMPWYQ/events.json","paper":"https://pith.science/paper/25UDD36X"},"agent_actions":{"view_html":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ","download_json":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ.json","view_paper":"https://pith.science/paper/25UDD36X","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2408.11796&json=true","fetch_graph":"https://pith.science/api/pith-number/25UDD36X2UP2PB7GW72NPMPWYQ/graph.json","fetch_events":"https://pith.science/api/pith-number/25UDD36X2UP2PB7GW72NPMPWYQ/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ/action/timestamp_anchor","attest_storage":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ/action/storage_attestation","attest_author":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ/action/author_attestation","sign_citation":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ/action/citation_signature","submit_replication":"https://pith.science/pith/25UDD36X2UP2PB7GW72NPMPWYQ/action/replication_record"}},"created_at":"2026-07-05T09:46:29.805260+00:00","updated_at":"2026-07-05T09:46:29.805260+00:00"}