Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Towards joint scaling laws with optimal batch size schedules

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Any learning-rate schedule has a closed-form optimal batch-size schedule, independent of model size and optimizer.

desk verdict A genuinely useful closed-form optimal batch-size schedule, with an honest but significant caveat: the optimality claim lives and dies with the loss-equality approximation, which fails under pure bf16. read the letter →

arxiv 2607.27731 v1 pith:RUKBOT7P submitted 2026-07-30 cs.LG cs.AImath.OC

Jiaxiang Li , Zhiqi Bu , Shiyun Xu This is my paper
classification cs.LGcs.AImath.OC
keywords batchsizeschedulelearningratescalinglawslargelanguagemodeltrainingconvexoptimizationlosspredictiongradientvariancehyperparametertransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the batch size during deep learning training should not be static: for any prescribed learning-rate schedule there is a closed-form optimal batch-size schedule that minimizes the predicted final loss under a fixed token budget. The derivation starts from a convex-optimization bound on the loss that couples learning rate and batch size, treats that bound as an equality in practice, and solves a variational problem to find B_t^opt = K b_t / ∫_0^T b_s ds with b_t = η_t / sqrt(∫_t^T η_s ds). Because the schedule depends only on the shape of the learning-rate curve, not on its peak value, weight decay, model size, architecture, or optimizer, the batch size becomes a transferable scaling dimension rather than a separately tuned hyperparameter. The authors report 6-15% compute-efficiency gains over static batch size with zero overhead, on dense and mixture-of-experts language models up to 7B parameters, and show the schedule preserves universal (supercollapsed) training dynamics.

What carries the argument

The load-bearing object is the loss predictor (2.5), derived from a convex bound with the gradient moment model E||g_t||^2 ≤ G^2 + X/B_t, where G is the norm of the mean gradient and X bounds per-example gradient variance. Treating that bound as an equality, the paper solves: minimize the final-loss bound over batch sizes B_t subject to ∫_0^T B_t dt = K (data or compute budget). The solution, Theorem 3, is B_t^opt = K b_t / ∫_0^T b_s ds with b_t := η_t / sqrt(∫_t^T η_s ds), which is the central identity of the paper. The same variational machinery yields, via Lemma 4, an optimal schedule for any cost function f(B), and via Theorem 5, joint scaling laws that scale peak learning rate and weigh

What would settle it

A controlled full-precision run, at a fixed compute budget and a base batch size small enough that the variance term matters, in which the static schedule ties or beats the dynamic schedule on final loss across several model sizes — or, more directly, a measurement of E||g_t||^2 versus 1/B_t that shows the slope X drifting non-negligibly over training or failing to fit G^2 + X/B_t.

Watch

Extended reading notes

Core claim

The central claim is a characterization, in (2.5), of the loss sequence as an explicit three-term function of the learning-rate sequence and batch-size sequence — an initial distance term, a gradient-bias term, and a gradient-noise term proportional to X/B_t — together with the closed-form optimal batch schedule that minimizes the last term under a total budget K: B_t^opt = K * η_t/sqrt(∫_t^T η_s ds) / (∫_0^T η_k/sqrt(∫_k^T η_s ds) dk). The authors argue, through linear-regression fits on Llama-3 and Qwen-3 training runs and controlled ablations, that the bound is tight enough to act as an equality for non-convex networks, and that the resulting schedule is independent of peak learning rate

Load-bearing premise

That the convex bound (2.5) is actually an equality for real non-convex LLMs, with a single time-independent gradient-variance constant X and a strict G^2 + X/B_t form; if that fails, Theorem 3's schedule is not guaranteed optimal for the true loss.

Editorial extensions

If this is right

  • Fixed batch size is strictly suboptimal under this model: the dynamic schedule is always no worse, with the gap equal to the variance of the ratio b_t over training (Jensen's inequality).
  • Batch size becomes a free transferable dimension — the same schedule curve applies across model sizes, optimizers, and architectures, and does not require tuning the peak learning rate.
  • Learning rate and batch size are not interchangeable: matching η_t/B_t trajectories does not match dynamics, so batch-size scheduling is an independent axis, not a reparameterization of learning-rate decay.
  • Joint scaling laws with 1/√T scaling of peak learning rate and weight decay yield O(1/√T) loss convergence and universal (supercollapsed) loss curves that support prediction of larger runs.
  • The terminal behavior of the optimal batch schedule is governed by how the learning rate approaches zero: linear and WSD cooldowns (p=1) end at a constant batch size, while cosine (p=2) drives batch size to zero.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the schedule's independence from model size survives at frontier scale, batch-size curves could be shipped as a universal config shared across a whole training fleet, and compute-optimal scaling laws would need a third axis for the batch shape rather than a single optimal batch size.
  • The theory predicts a symmetry: two learning-rate schedules with the same normalized b_t curve (same shape, different peak) should admit the same normalized optimal batch schedule; a cheap small-scale test across schedules would validate or falsify this directly.
  • The reported bf16 ablation (§C.3) suggests the advantage can vanish under low precision, implying that the effective variance constant X or the equality in (2.5) shifts with numeric precision; quantifying that shift could extend the framework to quantized training.
  • The same variational recipe — bound a loss term by a variance model, then minimize under a budget — could be reused for other scheduled quantities with known variance structure, such as context length, data mix, or expert routing ratios; these are not claims of the paper but natural extensions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a joint characterization of LLM training loss as a function of both the learning-rate schedule and the batch-size schedule, starting from a convex-SGD upper bound (Theorem 1) and the gradient-noise assumption E||g_t||^2 ≤ G^2 + X/B_t (Eq. 2.2). After replacing the bound by an approximate equality (Eq. 2.5) and passing to a continuous-time limit (Eq. 3.1), it derives a closed-form optimal batch-size schedule for a fixed compute/data budget (Theorem 3): B_t^opt = K b_t / ∫ b_t, with b_t = η_t / sqrt(∫_t^T η_k dk). The schedule depends only on the shape of the learning-rate schedule, not on peak learning rate, model size, or optimizer. The authors validate the loss characterization by fitting L*, D, G, X per loss curve, and report consistent gains of dynamic over static batch size across Llama3, Qwen3 MoE, and VLM/fine-tuning settings, with compute-efficiency improvements of 6–15%. They also propose joint scaling laws for learning rate, weight decay, and batch size, and show a universal-dynamics (supercollapse) property.

Significance. If the central equality (2.5) holds for real LLM training, the result is practically significant: a closed-form, tuning-free optimal batch-size schedule that yields several percent compute savings with zero overhead is a genuinely useful contribution. The mathematical core is sound: Theorem 3 correctly minimizes the X/B_t term in the stated model, and Corollary 3.3 correctly identifies a Jensen improvement of dynamic over static schedules. The paper is also strong empirically in breadth — dense and MoE models, multiple optimizers, pre-training and fine-tuning, and honest ablations including the bf16 case where the advantage vanishes. However, the optimality claim is only as strong as the unvalidated equality (2.5), and the paper does not directly test optimality against a continuum of alternative batch-size schedules, nor does it provide out-of-sample validation of the loss predictor.

major comments (4)
  1. [§2.3, Eq. (2.4)–(2.5)] The loss characterization is validated only by fitting L*, D, G, X separately to each loss trajectory (Eq. 2.4). This is an in-sample fit: a flexible four-parameter model can track a smooth curve without establishing that Eq. (2.5) predicts the loss for a different learning-rate/batch-size schedule. Since Theorem 3 optimizes the X/B_t term of exactly this fitted model, the central claim requires out-of-sample validation: fit (L*, D, G, X) on one schedule and use them to predict (or at least rank) another schedule, or compare the predicted optimal schedule against several plausible alternatives (e.g., linear ramp, step, inverse-B). Without such a test, 'optimal' is an assertion about the fitted curve, not about true LLM training loss.
  2. [§3.1, Eq. (2.6) and Table 2/Figure 4] Equation (2.6) is stated for η_T = 0, i.e., a learning rate that decays to zero at the terminal iterate. Yet the paper applies the resulting continuous formula to a constant learning-rate schedule in Table 2, Figure 4, and Table 3, where η_T = η > 0. In the discrete bound (2.3), the final term (η_τ/2)(G^2 + X/B_τ) is not negligible in that case, so the continuous objective (3.1) is not the limit of (2.6) for constant LR. The constant-LR experiment is one of the three main schedule comparisons, so this inconsistency needs to be addressed — either by deriving the appropriate endpoint term and showing the optimal schedule is unchanged, or by restricting the claim to decaying schedules and moving the constant-LR result to a heuristic.
  3. [§C.3, Figure 12] The bf16 ablation shows the dynamic-vs-static advantage essentially vanishes (perplexity delta 0.1%), while fp32 and mixed-precision show 2.5–2.6% improvements. This is the paper's own evidence that the gradient-noise term X/B_t is not the sole determinant of final loss under low precision, so the loss equality (2.5) fails in that regime. This is not just a minor caveat: it bounds the scope of the universal claim in §1 that the schedule is independent of 'optimizer' and 'model architecture'. The conclusion should be narrowed to full/mixed precision, and the paper should discuss why the schedule ceases to be beneficial under bf16 (e.g., quantization noise dominating gradient noise).
  4. [§3.1, Theorem 3 and §4.2] The empirical comparisons are always dynamic vs. static batch size. That demonstrates improvement over one baseline, but not that the specific closed-form schedule is optimal among all schedules with the same token budget. A direct test would compare the predicted B_t^opt against, for example, a linearly increasing schedule, a step schedule, and the inverse of the table's static baseline, all with the same learning-rate schedule and total tokens. Without such a test, the phrase 'optimal batch size schedule' in the title and abstract is stronger than what the experiments establish; the results support 'an effective dynamic batch size schedule'.
minor comments (5)
  1. [Corollaries 3.2–3.3] The definition of Z_t is inconsistent between the displayed formulas: Corollary 3.2 appears to use Z_t = η_t sqrt(∫_t^T η_k dk), while the derivation and Jensen comparison require Z_t = η_t / sqrt(∫_t^T η_k dk). Please clarify the notation and check the displayed expression for L_dynamic(T).
  2. [§2.1, Theorem 1 and abstract] The abstract says the characterization is 'applicable to general optimizers and model architectures', but Theorem 1 is proven for convex loss and SGD; the generalization to Adam/Muon and non-convex LLMs is empirical, not proven. Consider softening the wording or citing the precise empirical evidence earlier.
  3. [§4.1, Theorem 5] The term 'qualified schedule' is used without a definition. Since Theorem 5 relies on it, the condition should be stated explicitly (e.g., decay rate of the LR tail, or the order p in §3.1).
  4. [§2.3, footnote 1] The footnote correctly notes that with constant B_t the features x_2 and x_3 are collinear, but the top two rows of Figure 2 use constant B_t and still report fitted G and X. Please state how the regression is made identifiable in that case, or clarify that only the sum G^2 + X/B is identified.
  5. [Throughout] Minor typographical issues: in Table 2, the WSD B_t^opt expressions mix Bstatic and BT; the caption of Figure 1 says 'same BS' but the dynamic panel uses the scheduled BS; and the reference list contains several entries with inconsistent formatting (e.g., arXiv IDs and page ranges).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Theorem 3 is a genuine minimization of the stated loss model; the optimal schedule is independent of the fitted constants, so the central result is not fitted to the outcome it predicts.

full rationale

The derivation chain is: Theorem 1 (convex SGD bound from Defazio et al.) + (2.2) variance decomposition -> Theorem 2 -> continuous version (3.1) -> Theorem 3 minimizes the X/B_t term. Theorem 3 is a standard calculus-of-variations solution (equalizing b_t^2/B_t^2), and the resulting B_t^opt depends only on the learning-rate schedule shape, T, and total budget K; it is independent of the fitted parameters L*, D, G, X. Consequently, the central claim is not a case of fitting and re-predicting the same quantity. The dynamic-vs-static advantage (Corollary 3.3) is a Jensen/Cauchy-Schwarz consequence for any X>0, and the empirical comparisons (Figures 4-8, Tables 1/6) test real training runs, not the fitted curve itself. Two adjacent elements deserve mention but do not amount to circularity: (i) the generalization of the convex-SGD bound to non-convex networks and adaptive optimizers is partly justified by citing Bu et al. (2026), co-authored by two of the present authors (§2.1); the paper also fits (2.4) to its own Llama3/Qwen3 runs (Figures 2 and 10), so the citation is supportive rather than the sole evidence, although those fits are in-sample. (ii) The loss characterization (2.5) is validated by in-sample regression, so it establishes that the model can interpolate a single loss curve, not that it predicts a different schedule's loss out-of-sample; the bf16 ablation (§C.3) and the application of (2.6) with η_T > 0 in constant-LR experiments are correctness/validation limitations, not circular reductions. Under the stated model, Theorem 3 is derived, not assumed.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the convex-optimization loss model, the mean-covariance gradient noise bound, and two regularity assumptions (eta_T = 0, time-independent X). The batch-size formula itself is derived from the assumed model rather than fitted; the fitted constants appear only in the validation of the loss model and in the power-law quantification of the efficiency gain.

free parameters (3)
  • L*, D, G, X = not reported
    Fitted in Eq. (2.4) by linear regression on the full loss trajectory to justify the equality form (2.5). These do not enter the B_t^opt formula, but they support the 'precise characterization' claim.
  • Power-law coefficients (a, b) in Figure 5 = e.g., Llama3 dynamic: a=36.933, b=-0.058; static: a=34.184, b=-0.056
    Fitted to validation loss vs compute in Figure 5 and used to quantify the 1.15x/1.06x compute-efficiency improvement.
  • eta_ref (peak LR anchor) = tuned on small-scale runs (values not given)
    Anchor for the 1/sqrt(T) LR and weight-decay scaling in Eq. (4.1), inherited from Bu et al. (2026).
assumptions (7)
  • domain assumption Convex loss and SGD are a valid model for deep learning dynamics ('convex dominance')
    Theorem 1 is proven for convex loss/SGD (Defazio et al. 2023), but applied to nonconvex LLMs and adaptive optimizers; the generalization is attributed to Bu et al. (2026), a self-citation.
  • domain assumption Gradient second moment decomposes as E||g_t||^2 <= G^2 + X/B_t with time-independent X
    Eq. (2.2). Section A.2 derives it from mean-covariance decomposition assuming uniform upper bounds G and X on ||Eg_t|| and trace(Sigma_t); for nonconvex training these bounds are not established.
  • ad hoc to paper The inequality in Theorem 2 can be replaced by equality in practice
    Eq. (2.5) is presented as an approximation after fitting L*, D, G, X in (2.4); no out-of-sample validation is reported. The central schedule is derived from this equality.
  • domain assumption Learning rate decays to zero at the final iteration (eta_T = 0)
    Eq. (2.6) states this condition; however, constant-LR experiments in Figure 4 and Table 2 use the schedule even though eta_T = eta > 0.
  • standard math Continuous replacement of sums by integrals is accurate
    Section 3 replaces discrete summations with Riemann integrals, valid for slowly varying schedules and large T; the endpoint error is not quantified.
  • domain assumption Weight decay acts as equivalent regularization with bounded gradient (G + kappa W)
    Proof of Theorem 5 assumes bounded parameters W and translates the gradient bound to G + kappa W; this is not an established property for trained neural networks.
  • domain assumption The critical batch size X/G^2 is time-independent
    Footnote 2 treats X/G^2 as time-independent in line with Merrill et al. (2025) and Golmant et al. (2018), avoiding a per-iteration noise-model fit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards joint scaling laws with optimal batch size schedules." pith.science (2026). https://pith.science/paper/RUKBOT7P

@misc{pith2026260727731,
  author       = {Pith},
  title        = {Pith review of: Towards joint scaling laws with optimal batch size schedules},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUKBOT7P}},
  note         = {Machine review of arXiv:2607.27731}
}
read the original abstract

Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 1 linked inside Pith

  1. [1]

    Scaling laws for neural language models , year =

    Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario , journal =. Scaling laws for neural language models , year =

  2. [2]

    Training compute-optimal large language models , year =

    Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and Casas, Diego de Las and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and others , journal =. Training compute-optimal large language models , year =

  3. [3]

    Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer , volume =

    Yang, Ge and Hu, Edward and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , journal =. Tensor Programs V: Tuning large neural networks via zero-shot hyperparameter transfer , volume =

  4. [4]

    Predictable Scale: Part I--Optimal Hyperparameter Scaling Law in Large Language Model Pretraining , year =

    Li, Houyi and Zheng, Wenzhen and Hu, Jingcheng and Wang, Qiufeng and Zhang, Hanshan and Wang, Zili and Xuyang, Shijie and Fan, Yuantao and Zhou, Shuigeng and Zhang, Xiangyu and others , journal =. Predictable Scale: Part I--Optimal Hyperparameter Scaling Law in Large Language Model Pretraining , year =

  5. [5]

    Measuring the effects of data parallelism on neural network training , volume =

    Shallue, Christopher J and Lee, Jaehoon and Antognini, Joseph and Sohl-Dickstein, Jascha and Frostig, Roy and Dahl, George E , journal =. Measuring the effects of data parallelism on neural network training , volume =

  6. [6]

    arXiv preprint arXiv:1706.02677 , title =

    Goyal, Priya and Doll. arXiv preprint arXiv:1706.02677 , title =

    arXiv

  7. [7]

    ZeRO: memory optimizations toward training trillion parameter models , year =

    Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong , booktitle =. ZeRO: memory optimizations toward training trillion parameter models , year =

  8. [8]

    An empirical model of large-batch training , year =

    McCandlish, Sam and Kaplan, Jared and Amodei, Dario and Team, OpenAI Dota , journal =. An empirical model of large-batch training , year =

Show all 43 references
  1. [9]

    On the computational inefficiency of large batch sizes for stochastic gradient descent , year =

    Golmant, Noah and Vemuri, Nikita and Yao, Zhewei and Feinberg, Vladimir and Gholami, Amir and Rothauge, Kai and Mahoney, Michael W and Gonzalez, Joseph , journal =. On the computational inefficiency of large batch sizes for stochastic gradient descent , year =

  2. [10]

    On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima , year =

    Keskar, Nitish Shirish and Mudigere, Dheevatsa and Nocedal, Jorge and Smelyanskiy, Mikhail and Tang, Ping Tak Peter , booktitle =. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima , year =

  3. [11]

    Minicpm: Unveiling the potential of small language models with scalable training strategies , year =

    Hu, Shengding and Tu, Yuge and Han, Xu and He, Chaoqun and Cui, Ganqu and Long, Xiang and Zheng, Zhi and Fang, Yewei and Huang, Yuxiang and Zhao, Weilin and others , journal =. Minicpm: Unveiling the potential of small language models with scalable training strategies , year =

  4. [12]

    How Does Critical Batch Size Scale in Pre-training? , year =

    Zhang, Hanlin and Morwani, Depen and Vyas, Nikhil and Wu, Jingfeng and Zou, Difan and Ghai, Udaya and Foster, Dean and Kakade, Sham M , booktitle =. How Does Critical Batch Size Scale in Pre-training? , year =

  5. [13]

    Power Lines: Scaling laws for weight decay and batch size in

    Shane Bergsma and Nolan Simran Dey and Gurpreet Gosal and Gavia Gray and Daria Soboleva and Joel Hestness , booktitle =. Power Lines: Scaling laws for weight decay and batch size in. 2025 , url =

    2025

  6. [14]

    Scaling Law for Language Models Training Considering Batch Size , year =

    Shuai, Xian and Wang, Yiding and Wu, Yimeng and Jiang, Xin and Ren, Xiaozhe , journal =. Scaling Law for Language Models Training Considering Batch Size , year =

  7. [15]

    Deepseek llm: Scaling open-source language models with longtermism , year =

    Bi, Xiao and Chen, Deli and Chen, Guanting and Chen, Shanhuang and Dai, Damai and Deng, Chengqi and Ding, Honghui and Dong, Kai and Du, Qiushi and Fu, Zhe and others , journal =. Deepseek llm: Scaling open-source language models with longtermism , year =

  8. [16]

    Advances in neural information processing systems , title =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser,. Advances in neural information processing systems , title =

  9. [17]

    Dahl and Justin Gilmer and Christopher J

    Varun Godbole and George E. Dahl and Justin Gilmer and Christopher J. Shallue and Zachary Nado , note =. Deep Learning Tuning Playbook , year =

  10. [18]

    Optimal linear decay learning rate schedules and further refinements , year =

    Defazio, Aaron and Cutkosky, Ashok and Mehta, Harsh and Mishchenko, Konstantin , journal =. Optimal linear decay learning rate schedules and further refinements , year =

  11. [19]

    International Conference on Machine Learning , title =

    Schaipp, Fabian and H. International Conference on Machine Learning , title =

  12. [20]

    Tensor programs VI: Feature learning in infinite depth neural networks , volume =

    Yang, Greg and Yu, Dingli and Zhu, Chen and Hayou, Soufiane , booktitle =. Tensor programs VI: Feature learning in infinite depth neural networks , volume =

  13. [21]

    Don't be lazy: CompleteP enables compute-efficient deep transformers , volume =

    Dey, Nolan and Zhang, Bin and Noci, Lorenzo and Li, Mufan and Bordelon, Blake and Bergsma, Shane and Pehlevan, Cengiz and Hanin, Boris and Hestness, Joel , journal =. Don't be lazy: CompleteP enables compute-efficient deep transformers , volume =

  14. [22]

    Scaling optimal lr across token horizons , volume =

    Bjorck, Johan and Benhaim, Alon and Chaudhary, Vishrav and Wei, Furu and Song, Xia , booktitle =. Scaling optimal lr across token horizons , volume =

  15. [23]

    Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate , year =

    Zhiqi Bu and Shiyun Xu and Jialin Mao , booktitle =. Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate , year =

  16. [24]

    Smith and Pieter-Jan Kindermans and Quoc V

    Samuel L. Smith and Pieter-Jan Kindermans and Quoc V. Le , booktitle =. Don't Decay the Learning Rate, Increase the Batch Size , year =

  17. [25]

    Coupling Adaptive Batch Sizes with Learning Rates , year =

    Lucas Balles and Javier Romero and Philipp Hennig , booktitle =. Coupling Adaptive Batch Sizes with Learning Rates , year =

  18. [26]

    Kakade , title =

    Alexandru Meterez and Depen Morwani and Jingfeng Wu and Costin-Andrei Oncescu and Cengiz Pehlevan and Sham M. Kakade , title =

  19. [27]

    Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training , year =

    Merrill, William and Arora, Shane and Groeneveld, Dirk and Hajishirzi, Hannaneh , booktitle =. Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training , year =

  20. [28]

    and Vinyals, Oriol and Sifre, Laurent , booktitle =

    Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and Hennigan, Tom and Noland, Eric and Millican, Katie and van den Dri...

    2022

  21. [29]

    Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks , year =

    Qiu, Shikai and Xiao, Lechao and Wilson, Andrew Gordon and Pennington, Jeffrey and Agarwala, Atish , booktitle =. Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks , year =

  22. [30]

    Scaling with collapse: Efficient and predictable training of llm families , year =

    Bergsma, Shane and Zhang, Bin Claire and Dey, Nolan and Muhammad, Shaheer and Gosal, Gurpreet and Hestness, Joel , journal =. Scaling with collapse: Efficient and predictable training of llm families , year =

  23. [31]

    nanoVLM , year =

    Luis Wiedmann and Aritra Roy Gosthipaty and Andrés Marafioti , howpublished =. nanoVLM , year =. GitHub repository , publisher =

  24. [32]

    Hugo Laurençon and Léo Tronchon and Matthieu Cord and Victor Sanh , title =

  25. [33]

    2025 , url =

    Qwen Team , title =. 2025 , url =

    2025

  26. [34]

    Arindam Mitra and Hamed Khanpour and Corby Rosset and Ahmed Awadallah , title =

  27. [35]

    Towards understanding of orthogonalization in muon , year =

    Boreiko, Valentyn and Bu, Zhiqi and Zha, Sheng , booktitle =. Towards understanding of orthogonalization in muon , year =

  28. [36]

    Decoupled Weight Decay Regularization , year =

    Loshchilov, Ilya and Hutter, Frank , booktitle =. Decoupled Weight Decay Regularization , year =

  29. [37]

    The llama 3 herd of models , year =

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal =. The llama 3 herd of models , year =

  30. [38]

    Qwen3 technical report , year =

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and others , journal =. Qwen3 technical report , year =

  31. [39]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom ...

    2020

  32. [40]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model , year =

    Smith, Shaden and Patwary, Mostofa and Norick, Brandon and LeGresley, Patrick and Rajbhandari, Samyam and Casper, Jared and Liu, Zhun and Prabhumoye, Shrimai and Zerveas, George and Korthikanti, Vijay and others , journal =. Using deepspeed and megatron to train megatron-turin...

  33. [41]

    Chowdhery, Aakanksha and Narang, Sharan and Devlin, Jacob and Bosma, Maarten and Mishra, Gaurav and Roberts, Adam and Barham, Paul and Chung, Hyung Won and Sutton, Charles and Gehrmann, Sebastian and Schuh, Parker and Shi, Kensen and Tsvyashchenko, Sasha and Maynez, Joshua and...

    2023

  34. [42]

    Rae, Jack W. and Borgeaud, Sebastian and Cai, Trevor and Millican, Katie and Hoffmann, Jordan and Song, Francis and Aslanides, John and Henderson, Sarah and Ring, Roman and Young, Susannah and others , title =. 2021 , doi =

    2021

  35. [43]

    2023 , url =

    Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and Tam, Weng Lam and Ma, Zixuan and Xue, Yufei and Zhai, Jidong and Chen, Wenguang and Zhang, Peng and Dong, Yuxiao and Tang...

    2023

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.