Pith. sign in

REVIEW 4 major objections 5 minor 37 references

On Using Large-Batches in Federated Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In federated learning, multiplying large-batch gradients by a fixed $X$—outside critical phases—raises test accuracy over small-batch training by up to 32.33% at the same iteration count.

desk verdict The headline accuracy claim is confounded by unequal sample counts, and the evaluated method is a tuned learning-rate scaling trick; the actual teacher model is never implemented. read the letter →

arxiv 2509.10537 v1 pith:XXKDHO2D submitted 2025-09-05 cs.LG cs.AIcs.DC

Sahil Tyagi This is my paper · ORCID
classification cs.LGcs.AIcs.DC
keywords federatedlearninglarge-batchtraininggeneralizationgapgradientscalingcriticalphasesnoisecompressionbatch-sizeestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the test-accuracy penalty normally associated with large-batch federated learning can be removed by a very simple operation: multiply the large-batch gradient by a fixed factor $X$ except during critical phases of training. Under the same iteration budget, this scaled large-batch training is reported to beat ordinary small-batch training by up to 32.33% on ResNet50 (batch 1024 at $X=16$ versus batch 32) and by 3.74% on VGG11 (batch 1024 at $X=8$ versus batch 64). The practical stakes are that federated systems could keep the parallel speed and communication savings of large global batches while recovering small-batch generalization. The paper also proposes, but does not implement, a trained teacher model that would map large-batch gradients to small-batch equivalents, with the step function serving as a naive substitute.

What carries the argument

The load-bearing object is the step-function surrogate $U(X)$ defined by $U(X)=1.0$ in critical training phases and $U(X)=X$ otherwise; the update becomes $\tilde{G}=X\odot G(b_{\text{large}})$ outside critical phases and $\tilde{G}=G(b_{\text{large}})$ inside them. Critical phases are detected from the normalized gradient-change metric $\triangle(G^{(b)}_{(i)})=\left|\,|G^{(b)}_{(i)}|^2-|G^{(b)}_{(i-1)}|^2\,\right|/|G^{(b)}_{(i-1)}|^2$, with thresholds 0.5 and 0.8 tested. This machinery implements the paper's assumed identity $G(b_{\text{small}})\approx X\odot G(b_{\text{large}})$, i.e., the added term $(X-1)G(b_{\text{large}})$ plays the role of small-batch gradient noise. The paper pairs this statistical mechanism with two linear prediction models—one for activation and batch memory, one for compute time—so that the largest permissible and fastest batch size can be estimated for a given device.

What would settle it

Train ResNet50 on CIFAR10 with batch 1024, $X=16$, and threshold 0.5 for the same iteration count as batch 32; if the reported 32.33% test-accuracy advantage does not reproduce, the claim fails. A more direct check would record true batch-32 gradients and scaled batch-1024 gradients at identical weights and compute their cosine similarity or Euclidean distance.

Watch

Extended reading notes

Core claim

The central claim, stated in the paper's own terms, is that 'naively scaling up the gradient updates improves model quality and diminishes generalization gap even at larger batches.' Concretely, for the same number of training iterations, the step-function surrogate $U(X)$—which multiplies gradients by $X$ when the normalized gradient change is above threshold and leaves them at $1\times$ in critical phases—yields up to 32.33% higher test accuracy than small-batch training for ResNet50 on CIFAR10 and 3.74% higher for VGG11 on CIFAR100. The paper presents this as preliminary evidence that a large-batch update scaled by $X$ approximates a small-batch update, and that switching back to unscaled updates during sensitive periods prevents divergence. It then conjectures that replacing the step function with a learned teacher model would improve the estimate further.

Load-bearing premise

The load-bearing premise is that multiplying a large-batch gradient by a fixed constant $X$ gives a valid stand-in for a small-batch gradient (Equation (7)), and that the gradient-change threshold reliably marks the phases where the model must not be scaled; the paper never directly compares scaled large-batch gradients with true small-batch gradients, and the best $X$ and threshold are chosen per model.

Editorial extensions

If this is right

  • Synchronous federated training can use large global batches without automatically accepting worse test accuracy, because outside critical phases the update is simply re-scaled rather than recomputed at a smaller batch.
  • A client can predict, before training, both the largest batch that fits in memory and the batch that minimizes per-epoch time, making large-batch FL practical on heterogeneous devices.
  • Large-batch gradients are already close to compressed sparse updates, so scaled large-batch FL can tolerate aggressive gradient compression and save communication without proportional loss in accuracy.
  • The choice of $X$ and of the gradient-change threshold changes the outcome; at threshold 0.5, VGG11 degrades at $X=8,16$, while raising the threshold to 0.8 markedly improves $X=8$ (though not above $X=1$), so critical-phase detection carries the method.
  • If the step function is a valid stand-in, a teacher model trained to output small-batch gradients from large-batch inputs should extend the benefit to settings where a single fixed $X$ is not optimal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If $X\odot G(b_{\text{large}})\approx G(b_{\text{small}})$ is a real relationship, the same idea could apply to local-SGD and asynchronous federated variants, with each client's scaling factor chosen from its own batch size rather than one global $X$.
  • Because the best $X$ and threshold are found per model, a natural extension is an online calibration rule that estimates gradient noise during training and sets $X$ adaptively, eliminating grid search.
  • The success of the naive step function suggests the small-batch advantage may come mainly from injecting update noise of the right scale; if so, any cheap noise-injection scheme with similar magnitude might work at large batches.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses large-batch training in federated learning. It introduces a memory-estimation model and a compute-time model to predict batch-size bounds, and proposes a "gradient mapping" approach in which a teacher model maps large-batch gradients to small-batch-like gradients. As a naive substitute for the teacher model, it evaluates a step function that scales gradients by a factor X except in "critical training phases" detected via a gradient-change metric. Experiments on ResNet50/CIFAR10 and VGG11/CIFAR100 report that with this scaling, large-batch training (B=1024) achieves higher test accuracy than small-batch training for the same number of iterations, up to 32.33% higher for ResNet50 and 3.74% higher for VGG11, with respect to small-batch baselines at B=32 and B=64 respectively. The proposed teacher model itself is not implemented; only the step-function surrogate is evaluated.

Significance. If the headline result were valid, it would be significant for federated learning: a simple gradient-scaling heuristic with critical-phase detection could reduce the large-batch generalization gap while preserving the parallel efficiency of large batches. The paper also provides useful empirical timing and memory measurements across several models. However, the central empirical claim is confounded by the number of training samples seen per iteration, and the reported improvements depend on per-model, post hoc selection of the scaling factor and threshold. Without an equal-sample/equal-epoch comparison, multiple seeds, or any validation of the underlying gradient-equivalence assumption, the evidence does not establish the claimed effect.

major comments (4)
  1. [Section III.B, Figs. 7-8] The headline comparison is confounded by sample count. Reporting test accuracy after the same number of iterations with B=1024 versus B=32 does not hold the number of training samples constant: after T iterations the large-batch run has processed 32T samples while the small-batch run has processed only T samples. The claimed 32.33% improvement for ResNet50 could therefore reflect the larger number of epochs over the training data rather than any benefit of the proposed step-function scaling. An equal-epoch or equal-sample-count comparison, or a baseline that trains the small-batch model for 32 times more iterations, is necessary to isolate the effect of gradient scaling.
  2. [Section III.B, Eq. (7)] The modeling assumption G(bsmall) ≈ X·G(blarge) is unvalidated. No experiment compares scaled large-batch gradients to true small-batch gradients, and the gradient noise term γ in Eq. (4) is never measured. The step-function method is presented as a naive substitute for the teacher model A(·) of Eq. (5), but A(·) is never implemented or tested. Thus the claimed mechanism for improving large-batch generalization — that scaling by X reproduces small-batch updates outside critical phases — has no direct empirical support.
  3. [Section III.B, Figs. 7-8 and Summary] The reported successes depend on post hoc selection of the scaling factor X and the gradient-change threshold per model: X=16 with threshold 0.5 for ResNet50, and X=8 with threshold 0.8 for VGG11. The paper itself states that the ideal threshold varies with model and configuration, and it reports degraded accuracy for VGG11 at X=8 and X=16 under threshold 0.5. No seeds, error bars, or confidence intervals are reported. Consequently, the statement that "naively scaling up the gradient updates improves model quality and diminishes generalization gap even at larger batches" is not supported as a general claim.
  4. [Section III.A, Figs. 4-5] The accuracy claims for the linear compute-time and memory models are based on in-sample evaluation. The linear model is fitted to compute times at batch sizes such as 32, 128, 512, and 1024, and the reported error percentages are computed at those same fitted points. This does not demonstrate predictive accuracy for unseen batch sizes. A held-out evaluation or a cross-validation procedure is required to substantiate the claim that the models can "accurately predict" batch-memory or compute time for a given configuration.
minor comments (5)
  1. [Figure 3 caption] The caption says "When B increases, generalization improves as models converge over flatter minima," which contradicts the figure and the surrounding text (where larger batches achieve lower test accuracy). This should read "worsens" or "degrades."
  2. [Equations (6a)-(6c)] The notation for the step function is inconsistent: Eq. (6a) defines U(X), while Eqs. (6b) and (6c) use U(X, G). Please use a consistent function signature, e.g., U(G; X).
  3. [Section III.B] There is a typo in "hetereogeneous" (should be "heterogeneous") in the paragraph on heterogeneity under federated learning.
  4. [References] Reference [25] is incomplete: it lists only a title, without authors, venue, or year. Please provide full bibliographic information.
  5. [Abstract] The phrase "explore new directions" in the abstract should be "explores new directions" to agree with the subject "this work."

Circularity Check

2 steps flagged · score 4.0 of 10

Supporting batch-memory and compute-time 'predictions' are in-sample linear fits; headline accuracy claim is confounded but not circular.

  1. fitted input called prediction [Section III-A, Figure 4b and surrounding text]
    "we model the relationship between M_batch and batch-size as a linear regression problem. For CIFAR100, CalTech101 and Food101 datasets, the dashed lines show how the linear model predicts M_batch for different batches. CIFAR100 is predicted within 4.56-20.45% error-rate via linear fitting."

    The batch-memory model is a linear regression fitted to the same measured batch-memory values it is then said to 'predict.' The quoted error ranges are in-sample fit residuals, not out-of-sample prediction errors, so the claimed predictive accuracy reduces to the fit by construction.

  2. fitted input called prediction [Section III-A, Figure 5 and surrounding text]
    "we model compute time with respect to batch-size, i.e., t_c ∝ |b|. The dashed line shows the predicted time with a simplistic linear model fitted over the compute times logged for different batches ... Linear interpolation predicts the compute time for a given batch-size with an error margin of 0.6-9% in ResNet50."

    The compute-time model is also a linear model fitted to the logged compute times for the same batch sizes used in the evaluation. The reported error margins are residuals from the fitted curve, so the 'prediction' is equivalent to the fitting data rather than an independent check.

full rationale

The central gradient-scaling result is an empirical observation, not a derived prediction: the authors sweep step-up factors X and thresholds, report configurations that improve over the X=1 baseline, and compare to small-batch training at equal iteration counts. That claim is not circular, although it is weakened by unequal sample counts across batch sizes and by post-hoc selection of X and the threshold. The genuinely circular pieces are the supporting prediction models in Section III-A: both the batch-memory and compute-time models are linear regressions fitted to the same data points they are said to predict, so the quoted error margins are in-sample residuals rather than out-of-sample predictions. The critical-phase detector and threshold are imported from the author's own prior work [2], but the paper also varies the threshold and the detector is a published heuristic, so I do not treat that as load-bearing circularity. Overall, the side 'predictions' reduce to their fits by construction, while the main accuracy claim has independent empirical content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central accuracy claim rests on two tuned hyperparameters (X and threshold) and on an unvalidated equivalence between scaled large-batch gradients and small-batch gradients. The proposed teacher model is an unimplemented invented component. The auxiliary compute-time and memory models are linear fits evaluated in-sample.

free parameters (3)
  • Gradient step-up factor X = 16 (ResNet50 best), 8 (VGG11 best with threshold 0.8); scanned over 2,4,8,16
    Chosen post hoc per model to maximize reported test accuracy; the paper shows X=8 and X=16 degrade VGG11 at threshold 0.5.
  • Gradient-change threshold = 0.5 for most runs, 0.8 for VGG11 improved results
    Tuned per model; the paper states 'the ideal threshold can vary with model and configuration, and thus requires carefully choosing the optimal value.'
  • Linear model parameters for M_batch and t_c = Not reported numerically
    Least-squares fits evaluated on the same data points used to fit them; these support the parallel-performance estimation, not the central accuracy claim.
assumptions (3)
  • domain assumption Large-batch training converges to sharp minima with worse generalization, while small-batch training explores flat minima (Keskar et al. 2017, Yao et al. 2018).
    The entire motivation for the step-function relies on this prior characterization of the generalization gap.
  • domain assumption The gradient-change metric Δ from Tyagi and Swany [2] detects critical training phases.
    The step function switches between 1× and X based on Δ; no independent validation is provided here.
  • ad hoc to paper Equation (7): G(bsmall) ≈ X ⊙ G(blarge) is a valid approximation.
    Introduced without derivation or direct comparison to true small-batch gradients; it is the load-bearing equivalence for the naive method.
invented entities (1)
  • Teacher model A(·)
    purpose: Maps large-batch gradients to approximated small-batch gradients (Eq. 5) to improve large-batch generalization.
    Never implemented or tested; the paper substitutes a step function U(·) and only conjectures that A would improve results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Using Large-Batches in Federated Learning." pith.science (2026). https://pith.science/paper/XXKDHO2D

@misc{pith2026250910537,
  author       = {Pith},
  title        = {Pith review of: On Using Large-Batches in Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXKDHO2D}},
  note         = {Machine review of arXiv:2509.10537}
}
read the original abstract

Efficient Federated learning (FL) is crucial for training deep networks over devices with limited compute resources and bounded networks. With the advent of big data, devices either generate or collect multimodal data to train either generic or local-context aware networks, particularly when data privacy and locality is vital. FL algorithms generally trade-off between parallel and statistical performance, improving model quality at the cost of higher communication frequency, or vice versa. Under frequent synchronization settings, FL over a large cluster of devices may perform more work per-training iteration by processing a larger global batch-size, thus attaining considerable training speedup. However, this may result in poor test performance (i.e., low test loss or accuracy) due to generalization degradation issues associated with large-batch training. To address these challenges with large-batches, this work proposes our vision of exploiting the trade-offs between small and large-batch training, and explore new directions to enjoy both the parallel scaling of large-batches and good generalizability of small-batch training. For the same number of iterations, we observe that our proposed large-batch training technique attains about 32.33% and 3.74% higher test accuracy than small-batch training in ResNet50 and VGG11 models respectively.

Figures

Figures reproduced from arXiv: 2509.10537 by the authors.

Figure 1
Figure 1. Computation time and data-movement overhead rises as batch-size [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of time taken to complete one training epoch over [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Test accuracy is worse over larger batch-sizes due to convergence over sharper minima. When [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Activation memory is proportional to the batch-size. (b) Batch [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Compute time of different models over varying batch-sizes. Compared [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Gradient norm in early training stages over different batches. As [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: VGG11 convergence at various step-up factors [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 7
Figure 7. Figure 7: Convergence quality and iteration kernel density estimates (KDE) [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages

  1. [1]

    Communication-Efficient Learning of Deep Net- works from Decentralized Data

    McMahan, H. B. et al. “Communication-Efficient Learning of Deep Net- works from Decentralized Data.” International Conference on Artificial Intelligence and Statistics (2016)

    work page 2016

  2. [2]

    Accelerating Distributed ML Training via Selective Synchronization

    Tyagi, Sahil and Martin Swany. “Accelerating Distributed ML Training via Selective Synchronization.” 2023 IEEE International Conference on Cluster Computing (CLUSTER) (2023): 1-12

    work page 2023

  3. [3]

    Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

    Tyagi, Sahil and Martin Swany. “Flexible Communication for Optimal Distributed Learning over Unpredictable Networks.” 2023 IEEE Inter- national Conference on Big Data (BigData) (2023): 925-935

    work page 2023

  4. [4]

    ImageNet Training in Minutes

    You, Yang et al. “ImageNet Training in Minutes.” Proceedings of the 47th International Conference on Parallel Processing (2017)

    work page 2017

  5. [5]

    An Empirical Model of Large-Batch Training

    McCandlish, Sam et al. “An Empirical Model of Large-Batch Training.” ArXiv abs/1812.06162 (2018)

    arXiv 2018

  6. [6]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

    Goyal, Priya et al. “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.” ArXiv abs/1706.02677 (2017)

    arXiv 2017

  7. [7]

    An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise

    Wen, Yeming et al. “An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise.” 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) (2020)

    work page 2020

  8. [8]

    Distributed Deep Learning Using Synchronous Stochastic Gradient Descent

    Das, Dipankar et al. “Distributed Deep Learning Using Synchronous Stochastic Gradient Descent.” ArXiv abs/1602.06709 (2016)

    arXiv 2016

Show all 37 references
  1. [9]

    Train longer, generalize better: closing the general- ization gap in large batch training of neural networks

    Hoffer, Elad et al. “Train longer, generalize better: closing the general- ization gap in large batch training of neural networks.” 31st International Conference on Neural Information Processing Systems (NIPS) (2017)

    2017

  2. [10]

    Extrapolation for Large-batch Training in Deep Learn- ing

    Lin, Tao et al. “Extrapolation for Large-batch Training in Deep Learn- ing.” 37th Proceedings of Machine Learning Research (PMLR) (2020)

    2020

  3. [11]

    On Large-Batch Training for Deep Learn- ing: Generalization Gap and Sharp Minima

    Keskar, Nitish Shirish et al. “On Large-Batch Training for Deep Learn- ing: Generalization Gap and Sharp Minima.” Proceedings of Interna- tional Conference on Learning Representations (ICLR) (2017)

    2017

  4. [12]

    Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

    Yao, Zhewei et al. “Hessian-based Analysis of Large Batch Training and Robustness to Adversaries.” 32nd International Conference on Neural Information Processing Systems (NeurIPS) (2018)

    2018

  5. [13]

    Learned Gradient Compression for Dis- tributed Deep Learning

    Abrahamyan, Lusine et al. “Learned Gradient Compression for Dis- tributed Deep Learning.” IEEE Transactions on Neural Networks and Learning Systems 33 (2021): 7330-7344

    2021

  6. [14]

    Learned Parameter Compression for Efficient and Privacy-Preserving Federated Learning

    Chen, Yiming et al. “Learned Parameter Compression for Efficient and Privacy-Preserving Federated Learning.” IEEE Open Journal of the Communications Society 5 (2024): 3503-3516

    2024

  7. [15]

    Scavenger: A Cloud Service For Optimizing Cost and Performance of ML Training

    Tyagi, Sahil and Prateek Sharma. “Scavenger: A Cloud Service For Optimizing Cost and Performance of ML Training.” 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid): 403-413

    2023

  8. [16]

    Large batch size training of neural networks with ad- versarial training and second-order information

    Yao, Zhewei et al. “Large batch size training of neural networks with ad- versarial training and second-order information.” ArXiv abs/1810.01021

    arXiv

  9. [17]

    Large Batch Training of Convolutional Networks

    You, Yang et al. “Large Batch Training of Convolutional Networks.” arXiv: Computer Vision and Pattern Recognition (CVPR) (2017)

    2017

  10. [18]

    Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

    Jia, Xianyan et al. “Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes.” Systems for ML workshop (NeurIPS) (2018)

    2018

  11. [19]

    Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

    You, Yang et al. “Large Batch Optimization for Deep Learning: Training BERT in 76 minutes.” Proceedings of the 8th International Conference on Learning Representations (ICLR) (2020)

    2020

  12. [20]

    Don’t Decay the Learning Rate, Increase the Batch Size

    Smith, Samuel L. et al. “Don’t Decay the Learning Rate, Increase the Batch Size.” Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018)

    2018

  13. [21]

    Don’t Use Large Mini-Batches, Use Local SGD

    Lin, Tao et al. “Don’t Use Large Mini-Batches, Use Local SGD.” Proceedings of the 8th International Conference on Learning Repre- sentations (ICLR) (2020)

    2020

  14. [22]

    ScaDLES: Scalable Deep Learning over Streaming data at the Edge

    Tyagi, Sahil and Martin Swany. “ScaDLES: Scalable Deep Learning over Streaming data at the Edge.” IEEE International Conference on Big Data (Big Data) (2022): 2113-2122

    2022

  15. [23]

    Adaptive Federated Learning in Resource Con- strained Edge Computing Systems

    Wang, Shiqiang et al. “Adaptive Federated Learning in Resource Con- strained Edge Computing Systems.” IEEE Journal on Selected Areas in Communications 37 (2018): 1205-1221

    2018

  16. [24]

    Federated Learning: Challenges, Methods, and Future Directions

    Li, Tian et al. “Federated Learning: Challenges, Methods, and Future Directions.” IEEE Signal Processing Magazine 37 (2019): 50-60

    2019

  17. [25]

    Apple Intelligence Foundation Language Models

    Gunter, Tom et al. “Apple Intelligence Foundation Language Models”

  18. [26]

    Decentralized Federated Learning: A Survey and Perspective

    Yuan, Liangqi et al. “Decentralized Federated Learning: A Survey and Perspective.” ArXiv abs/2306.01603 (2023)

    2023 arXiv

  19. [27]

    GraV AC: Adaptive Compression for Communication-Efficient Distributed DL Training

    Tyagi, Sahil and Martin Swany. “GraV AC: Adaptive Compression for Communication-Efficient Distributed DL Training.” 2023 IEEE 16th International Conference on Cloud Computing (CLOUD): 319-329

    2023

  20. [28]

    ZeRO: Memory Optimization Towards Training A Trillion Parameter Models

    Rajbhandari, Samyam et al. “ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.” ArXiv abs/1910.02054 (2019)

    2019 arXiv

  21. [29]

    On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent

    Golmant, Noah et al. “On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent.” ArXiv abs/1811.12941 (2018)

    2018 arXiv

  22. [30]

    Revisiting LARS for Large Batch Training Generalization of Neural Networks

    Do, Khoi Duy et al. “Revisiting LARS for Large Batch Training Generalization of Neural Networks.” ArXiv abs/2309.14053 (2023)

    2023 arXiv

  23. [31]

    A Bayesian Perspective on Generalization and Stochastic Gradient Descent

    Smith, Samuel L. et al. “A Bayesian Perspective on Generalization and Stochastic Gradient Descent.” ICLR 2018

    2018

  24. [32]

    Critical Learning Periods in Deep Neural Networks

    Achille, Alessandro et al. “Critical Learning Periods in Deep Neural Networks.” ArXiv abs/1711.08856 (2017)

    2017 arXiv

  25. [33]

    The Early Phase of Neural Network Training

    Frankle, Jonathan et al. “The Early Phase of Neural Network Training.” ArXiv abs/2002.10365 (2020)

    2020 arXiv

  26. [34]

    Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification

    Agarwal, Saurabh et al. “Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification.” 4th MLSys (2021)

    2021

  27. [35]

    Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

    Tyagi, Sahil and Prateek Sharma. “Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching.” 2020 IEEE ACSOS

    2020

  28. [36]

    OmniLearn: A Framework for Distributed Deep Learning Over Heterogeneous Clusters

    Tyagi, Sahil and Prateek Sharma. “OmniLearn: A Framework for Distributed Deep Learning Over Heterogeneous Clusters.” IEEE Trans- actions on Parallel and Distributed Systems 36 (2025): 1253-1267

    2025

  29. [37]

    ”An Overview of Computational and Communica- tion Mechanisms for Scalable AI Systems”

    Tyagi, Sahil. ”An Overview of Computational and Communica- tion Mechanisms for Scalable AI Systems”. 2025 ResearchGate DOI:10.13140/RG.2.2.21481.61283

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.