Revisiting Small Batch Training for Deep Neural Networks
read the original abstract
Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size $m$. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between $m = 2$ and $m = 32$, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech
Bilingual fine-tuning on a new parallel Filipino-English dementia dataset yields Macro-F1 scores of 0.969-0.973 and eliminates cross-lingual degradation for all tested transformers.
-
Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
Momentum SGD exhibits two distinct EoSS regimes for batch sharpness, stabilizing at 2(1-β)/η for small batches and 2(1+β)/η for large batches, aligning with linear stability thresholds.
-
Stochastic Gradient Optimization with Model-Assisted Sampling
Model-assisted sampling applies survey sampling variance reduction with auxiliary predictors to stochastic gradients, yielding empirical gains on benchmarks and faster generalization with AdamW.
-
Behavior Score Prediction in Resting-State Functional MRI by Deep State Space Modeling
A deep state space model on rs-fMRI time series predicts Alzheimer's behavior scores better than functional connectivity approaches and identifies key predictive brain regions.
-
Deep Multi-View Learning via Task-Optimal CCA
End-to-end deep optimization of CCA plus task loss produces discriminative shared representations that outperform prior multi-view methods on classification and semi-supervised tasks.
-
A Residual-Subspace Constraint Framework for Fourier Ptychographic Microscopy
RSCF decouples low-rank systematic residuals from noise via subspace constraints to enable high-fidelity phase and amplitude recovery in Fourier ptychographic microscopy under aberrations and misalignments.
-
Algorithmic Advantage on a Gate-Based Photonic Quantum Neural Network
Photonic QNNs with two trainable parameters solve nonlinear tasks like XOR at 100% accuracy where parameter-matched ANNs fail, with hardware deployment confirming the result.
-
Algorithmic Advantage on a Gate-Based Photonic Quantum Neural Network
A two-parameter photonic QNN achieves 100% accuracy on nonlinear tasks where a matched classical ANN saturates at random guessing, suggesting algorithmic advantage on current photonic hardware.
-
Simulation-based inference for rapid Bayesian parameter estimation in epidemiological models: a comparison with MCMC
SBI matches MCMC posterior accuracy on a SECIR model but runs 15-120 times faster on GPU for 31-day and 201-day inference windows.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.