{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2019:J655K2KRI7DJHF4SOFYZCQRRE5","short_pith_number":"pith:J655K2KR","schema_version":"1.0","canonical_sha256":"4fbbd5695147c693979271719142312751c3535b842a03f4c428753a52087fea","source":{"kind":"arxiv","id":"1904.00962","version":5},"attestation_state":"computed","paper":{"title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":["cs.AI","cs.CL","stat.ML"],"primary_cat":"cs.LG","authors_text":"Cho-Jui Hsieh, James Demmel, Jing Li, Jonathan Hseu, Kurt Keutzer, Sanjiv Kumar, Sashank Reddi, Srinadh Bhojanapalli, Xiaodan Song, Yang You","submitted_at":"2019-04-01T16:53:35Z","abstract_excerpt":"Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"1904.00962","kind":"arxiv","version":5},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2019-04-01T16:53:35Z","cross_cats_sorted":["cs.AI","cs.CL","stat.ML"],"title_canon_sha256":"a5caf378a4c2dd1dd9eac4879099bbb0eb0311ed2a0c2f9f2217d21148ad731e","abstract_canon_sha256":"85b43dc17abea7dc1561a9470fd799224beb337b037b437a99d04548d290b798"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-21T21:29:48.722412Z","signature_b64":"45ODB9eYA7bNRGE8R4hsxXAdjIog3sHqwSeuWHnMvjOcbJKDujSvvZAQ4G7MCxFcajuANexXZ55d5WTsgDOpBw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"4fbbd5695147c693979271719142312751c3535b842a03f4c428753a52087fea","last_reissued_at":"2026-05-21T21:29:48.720470Z","signature_status":"signed_v1","first_computed_at":"2026-05-21T21:29:48.720470Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":["cs.AI","cs.CL","stat.ML"],"primary_cat":"cs.LG","authors_text":"Cho-Jui Hsieh, James Demmel, Jing Li, Jonathan Hseu, Kurt Keutzer, Sanjiv Kumar, Sashank Reddi, Srinadh Bhojanapalli, Xiaodan Song, Yang You","submitted_at":"2019-04-01T16:53:35Z","abstract_excerpt":"Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes. However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks. In this paper, we first study a principled layerwise adaptation strategy to accelerate training of "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"1904.00962","kind":"arxiv","version":5},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/1904.00962/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"1904.00962","created_at":"2026-05-21T21:29:48.720571+00:00"},{"alias_kind":"arxiv_version","alias_value":"1904.00962v5","created_at":"2026-05-21T21:29:48.720571+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.1904.00962","created_at":"2026-05-21T21:29:48.720571+00:00"},{"alias_kind":"pith_short_12","alias_value":"J655K2KRI7DJ","created_at":"2026-05-21T21:29:48.720571+00:00"},{"alias_kind":"pith_short_16","alias_value":"J655K2KRI7DJHF4S","created_at":"2026-05-21T21:29:48.720571+00:00"},{"alias_kind":"pith_short_8","alias_value":"J655K2KR","created_at":"2026-05-21T21:29:48.720571+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":27,"internal_anchor_count":27,"sample":[{"citing_arxiv_id":"2605.22997","citing_title":"Scene Reconstruction as Mapping Priors for 3D Detection","ref_index":60,"is_internal_anchor":true},{"citing_arxiv_id":"2410.21316","citing_title":"Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2503.23947","citing_title":"Spectral-Adaptive Modulation Networks for Visual Perception","ref_index":104,"is_internal_anchor":true},{"citing_arxiv_id":"2505.13196","citing_title":"A Physics-Inspired Optimizer: Velocity Regularized Adam","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22297","citing_title":"One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22098","citing_title":"TextTeacher: What Can Language Teach About Images?","ref_index":71,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21557","citing_title":"Scalable On-Policy Reinforcement Learning via Adaptive Batch Scaling","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2502.07529","citing_title":"Training Deep Learning Models with Norm-Constrained LMOs","ref_index":216,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20390","citing_title":"STELLAR: Scaling 3D Perception Large Models for Autonomous Driving","ref_index":56,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16600","citing_title":"Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space","ref_index":57,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15435","citing_title":"On the Stability of Growth in Structural Plasticity","ref_index":45,"is_internal_anchor":true},{"citing_arxiv_id":"2505.23737","citing_title":"On the Convergence Analysis of Muon","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2506.12542","citing_title":"PLD: A Choice-Theoretic List-Wise Knowledge Distillation","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2510.04988","citing_title":"Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization","ref_index":70,"is_internal_anchor":true},{"citing_arxiv_id":"2510.18900","citing_title":"Foundation Models for Discovery and Exploration in Chemical Space","ref_index":128,"is_internal_anchor":true},{"citing_arxiv_id":"1910.02054","citing_title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2309.16671","citing_title":"Demystifying CLIP Data","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2602.21545","citing_title":"MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03346","citing_title":"Learning PDEs for Portfolio Optimization with Quantum Physics-Informed Neural Networks","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"1909.11942","citing_title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11111","citing_title":"ShardTensor: Domain Parallelism for Scientific Machine Learning","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06654","citing_title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","ref_index":39,"is_internal_anchor":true},{"citing_arxiv_id":"2605.00650","citing_title":"AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07815","citing_title":"OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2205.01068","citing_title":"OPT: Open Pre-trained Transformer Language Models","ref_index":46,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5","json":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5.json","graph_json":"https://pith.science/api/pith-number/J655K2KRI7DJHF4SOFYZCQRRE5/graph.json","events_json":"https://pith.science/api/pith-number/J655K2KRI7DJHF4SOFYZCQRRE5/events.json","paper":"https://pith.science/paper/J655K2KR"},"agent_actions":{"view_html":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5","download_json":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5.json","view_paper":"https://pith.science/paper/J655K2KR","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=1904.00962&json=true","fetch_graph":"https://pith.science/api/pith-number/J655K2KRI7DJHF4SOFYZCQRRE5/graph.json","fetch_events":"https://pith.science/api/pith-number/J655K2KRI7DJHF4SOFYZCQRRE5/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5/action/timestamp_anchor","attest_storage":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5/action/storage_attestation","attest_author":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5/action/author_attestation","sign_citation":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5/action/citation_signature","submit_replication":"https://pith.science/pith/J655K2KRI7DJHF4SOFYZCQRRE5/action/replication_record"}},"created_at":"2026-05-21T21:29:48.720571+00:00","updated_at":"2026-05-21T21:29:48.720571+00:00"}