REVIEW 4 major objections 4 minor 20 references
Collaborative Batch Size Optimization for Federated Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A federated randomized binary search can determine the largest safe shared batch size in a few rounds, speeding up training by up to 7.8x versus a default batch size of 4.
desk verdict A useful but overclaimed batch-size search: the hard cap at 64 means the paper can't support its 'physical maximum' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the joint randomized binary-search protocol: the server broadcasts the current batch-size interval, each client samples one value from it, trains one local step, and reports success or failure, and the server shrinks the interval by taking the maximum of the clients' lower bounds and the minimum of their upper bounds. A fraction $f$ of clients train with the current safe minimum during the search, so no round is entirely wasted while the interval is being narrowed. This parallelizes a binary search over batch sizes across the federation and makes the search a side effect of normal federated training.
What would settle it
Run the same federations on the same GPU with $b_{\max}$ initialized to 256 instead of 64 and record whether the search ever converges to a shared batch size above 64 with faster wall-clock completion; if it does, the capped configuration is not finding the physical maximum the method claims to find.
Extended reading notes
Core claim
RASBA identifies the maximum batch size that every client in a federation can physically support, without clients disclosing their hardware and without the federation failing on an out-of-memory error. In each round, every client samples a batch size uniformly from the current shared interval $[b_{\min}, b_{\max}]$; a successful local update makes the sampled value the client's new lower bound, while an out-of-memory error makes it a new upper bound. The server then sets $b_{\min}$ to the maximum of the reported lower bounds and $b_{\max}$ to the minimum of the reported upper bounds, narrowing the interval until all clients converge on one common value. In the reported experiments the adaptive run finishes about 7.8x faster than batch size 4 on MNIST and about 4.8x faster on CIFAR10, with final test accuracies of 96.09% and 73.42%, close to the best fixed batch sizes.
Load-bearing premise
The search is capped by a hand-chosen initial upper bound $b_{\max}=64$, so the claim that it finds the maximum physically supported shared batch size holds only if the true maximum is at or below that cap.
Editorial extensions
If this is right
- A federation can start with a conservative default and automatically enlarge its shared batch size until some client's hardware objects, which removes the need for a manual hardware survey.
- Because the search lives entirely in the local training step and the server-side bound aggregation, it can be combined with any aggregation strategy or client-side training procedure.
- Researchers running federated simulations on a shared GPU can skip the common pre-experiment step of testing each possible batch size by hand.
- The reported 15x speedup on MNIST between batch size 4 and 256 shows that the batch size is a first-order factor in total training time, so finding the feasible maximum is worth several search rounds.
- The shared batch size equals the lowest common denominator of all clients, which keeps every client in the federation but can leave faster hardware partially idle.
Reading between the lines
- The initial cap $b_{\max}=64$ means the search only explores a preselected range; if the true hardware limit lies above 64, the reported 'maximal' batch size is a property of the cap, not of the hardware.
- On heterogeneous hardware the lowest-common-denominator choice underutilizes faster clients, so grouping clients by their reported limits and running one search per tier would likely recover additional speedup; the paper notes this direction as future work.
- The experiments run on a single GPU model, so the search's convergence time under genuinely distributed or memory-diverse clients remains untested; a multi-GPU test with varying VRAM sizes would show how the protocol scales.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RASBA, a greedy randomized binary-search procedure run collaboratively across federated-learning clients, to choose a shared local mini-batch size. The server sends b_min and b_max to clients, each client samples a batch size in that interval and reports success or an out-of-memory failure, and the server aggregates by taking the maximum of the reported minima and the minimum of the reported maxima. Experiments on MNIST with MobileNet-v3 and CIFAR10 with ResNet18 on a single GTX 1080 GPU compare RASBA against fixed batch sizes in terms of wall-clock time and final accuracy. The paper reports that RASBA reaches near-optimal accuracy much faster than the default batch size 4 and that it finds the maximal possible shared batch size within a few rounds.
Significance. If the central claims held, the method would be a simple, practical tool for FL simulations, where batch-size limits are often chosen manually and conservatively. The paper has concrete strengths: the code is stated to be available, the experimental setup uses standard frameworks (Flower, PyTorch), and Table I provides a clear wall-clock comparison across fixed batch sizes. The qualitative result that larger batches reduce training time on this GPU is plausible and consistent with Figure 1. However, the paper's main contribution, discovering the maximum physically supported shared batch size, is not established by the current evidence because the search is hard-capped at 64 even though batches 128 and 256 run successfully on the same hardware. The accuracy discrepancy between RASBA and fixed batch 64 is also unexplained, and there are no repeated runs or error bars. The contribution is therefore only conditionally supported; it is a reasonable idea that needs a corrected evaluation and a re-scoped or strengthened claim.
major comments (4)
- [Section IV, Algorithm 1 and Table I] The initial cap b_max=64 contradicts the paper's maximum-discovery claim. Table I shows that batch sizes 128 and 256 complete successfully on the same GTX 1080 (e.g., 416s and 342s on MNIST), and Figure 1 includes batch=128 and batch=256. Since Algorithm 1 only samples b in [b_min, b_max] and the server sets b_max = min_i(b_i_max), no client ever samples above 64 and the shared b_max can never exceed 64. Consequently, RASBA cannot find the maximum physically supported shared batch size; it only finds a safe value inside the hand-chosen interval [b_min, 64]. The abstract, introduction, and conclusion state that the method determines 'the maximum achievable shared batch size' or 'maximal possible shared batch size,' and those statements are not supported by the experiments. Either re-scope the claim to 'largest safe batch size within a user-specified interval' or modify the algorithm to expand the upper bound (e.g., doubling) until failure is observed.
- [Section V, Table I] The accuracy of 'our method' is not consistent with the method's own search behavior. On MNIST, RASBA reaches 96.09% accuracy, whereas fixed batch 64 reaches 91.21%; on CIFAR10, the corresponding values are 73.42% versus 70.62%. Because the search is capped at 64 and is claimed to converge to the shared batch size, one would expect RASBA's accuracy to be close to that of fixed batch 64 rather than substantially better. The discrepancy is not explained; it may be due to the fraction f=0.5 of clients training at b_min during the search, to random intermediate batch sizes, or to unmeasured run-to-run variance. No repeated runs or error bars are reported, so the 7.76x and 4.78x speedup figures and the accuracy numbers could be affected by noise. The paper needs either an explanation of the mechanism behind the accuracy difference or experiments with multiple seeds to establish that the advantage is real.
- [Section V, Figures 2/3] The paper claims that 'with f=0.5 we observe that our method finds the maximal possible shared batch size in at most 3 rounds,' but no evidence for this is presented. There is no plot or table showing the batch size selected by RASBA over rounds, no record of how many clients sampled which values, no count of out-of-memory failures, and no measurement of the round at which b_min and b_max converge. Without this information, the central convergence-time claim is not verifiable. The authors should report the search trajectory (sampled batch sizes and b_min/b_max progress) for at least one representative run on each dataset.
- [Section V, Setup] The evaluation uses a single homogeneous GPU (GTX 1080) with 10 simulated clients, so the federated, collaborative aspect of the method is never tested under hardware heterogeneity. The paper's motivation is that the central server is unaware of client hardware capabilities, but all clients in the experiment have identical capabilities. Consequently, the claim that RASBA helps when clients have different hardware limits is not demonstrated. A small experiment with two or three client hardware tiers (or simulated memory limits) would directly test the server's min-aggregation rule and the method's robustness to heterogeneous b_i_max.
minor comments (4)
- [Section IV] The sentence 'Due to a higher sampling rate compared to the generic randomized binary search, our method reduces the time to convergence by ((1-f)×m)^{-1} compared to the binary search' is unclear and is not derived or verified in the experiments; please clarify the definition of 'sampling rate' and the derivation of this factor.
- [Section IV, Algorithm 1] The text says clients 'randomly sample a batch size bi > bmin,' while Algorithm 1 samples b in [b_min, b_max]; these formulations are inconsistent, and the algorithm should state the exact sampling distribution (e.g., uniform over integers in the interval).
- [Section V, Setup] There are several small presentation issues: 'Federations on MNIST run for 25, while federations on CIFAR10 run for 20 global rounds' is missing the word 'rounds'; the anonymous GitHub link is not included in the text; and Figure 1 would benefit from explicit values or error bars for memory and GPU utilization.
- [Section V, Table I] The fixed batch 128 result on CIFAR10 (936s) is slower than fixed batch 64 (607s) and even fixed batch 32 (820s), which is not discussed; this non-monotonicity is worth a brief explanation because it affects the claim that larger batches always reduce time.
Circularity Check
The claimed 'maximal shared batch size' is imposed by the hand-set cap b_max=64, since RASBA never samples above 64; the result is the input cap by construction.
-
self definitional
[Section IV (Algorithm 1) and Section V.B; Table I]
"We further restrict bmax following the observations in Section II-B and set bmax = 64 in the initial step. ... With f=0.5 we observe that our method finds the maximal possible shared batch size in at most 3 rounds."
Algorithm 1 samples b in [b_min, b_max] and the server aggregates b_max = min_i(b_i^max). With the initial cap b_max=64, no client ever proposes a batch above 64, so the search result cannot exceed 64. When every client successfully trains at 64, the algorithm outputs 64 and the paper calls this the 'maximal possible shared batch size.' However, Table I shows that batches 128 and 256 execute on the same GTX 1080 and 256 is the fastest row (342 s on MNIST). The claimed maximum is therefore not discovered but is the initial cap by construction; the method only certifies the largest safe batch within the hand-chosen interval [b_min, 64].
full rationale
Aside from the b_max=64 cap, the paper contains no fitted equations, no parameters calibrated to the evaluation data, and no self-citations; RASBA's convergence-time comparison to fixed batch sizes is an honest empirical benchmark. The central overclaim is that the method determines the physically maximal shared batch size, when the output is bounded by a hand-set cap that is below what the paper's own hardware supports. This is a partial circularity: the reported 'maximum' reduces to the input cap whenever all clients succeed, so the headline result does not have independent content beyond the initial bound. The speedup relative to batch=4 is a real consequence of using a larger batch, not a circular artifact. Score 6 reflects that the main claimed prediction (the maximum) is imposed by an input, while the rest of the empirical work is self-contained.
Assumptions & free parameters
free parameters (2)
- initial upper bound b_max =
64
- fraction f of clients that keep training at b_min during search =
0.5
assumptions (4)
- domain assumption Batch-size feasibility is monotone in batch size: if a batch fits in GPU memory, all smaller batches fit; if it fails with OutOfMemoryError, all larger batches fail.
- domain assumption A single random batch-size probe per client per round yields a reliable lower or upper bound for that client.
- domain assumption Clients report their sampled bounds honestly and the server aggregates them with min/max.
- domain assumption Larger batch sizes reduce computation time per epoch on the evaluated hardware.
Cite this review
Pith. "Pith review of Collaborative Batch Size Optimization for Federated Learning." pith.science (2026). https://pith.science/paper/GUOXPJ3W
@misc{pith2026250620511,
author = {Pith},
title = {Pith review of: Collaborative Batch Size Optimization for Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GUOXPJ3W}},
note = {Machine review of arXiv:2506.20511}
}
read the original abstract
Federated Learning (FL) is a decentralized collaborative Machine Learning framework for training models without collecting data in a centralized location. It has seen application across various disciplines, from helping medical diagnoses in hospitals to detecting fraud in financial transactions. In this paper, we focus on improving the local training process through hardware usage optimization. While participants in a federation might share the hardware they are training on, since there is no information exchange between them, their training process can be hindered by an improper training configuration. Taking advantage of the parallel processing inherent to Federated Learning, we use a greedy randomized search to optimize local batch sizes for the best training settings across all participants. Our results show that against default parameter settings, our method improves convergence speed while staying nearly on par with the case where local parameters are optimized.
Figures
Reference graph
Works this paper leans on
-
[1]
Communication-efficient learning of deep networks from decentralized data,
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics. PMLR, 2017, pp. 1273– 1282
2017
-
[2]
A review of applications in federated learning,
L. Li, Y . Fan, M. Tse, and K.-Y . Lin, “A review of applications in federated learning,”Computers & Industrial Engineering, vol. 149, p. 106854, 2020
2020
-
[3]
Flowertune: A cross- domain benchmark for federated fine-tuning of large language models,
Y . Gao, M. R. Scamarcia, J. Fernandez-Marques, M. Naseri, C. S. Ng, D. Stripelis, Z. Li, T. Shen, J. Bai, D. Chenet al., “Flowertune: A cross- domain benchmark for federated fine-tuning of large language models,” arXiv preprint arXiv:2506.02961, 2025
arXiv 2025
-
[4]
Don’t decay the learning rate, increase the batch size,
S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V . Le, “Don’t decay the learning rate, increase the batch size,”arXiv preprint arXiv:1711.00489, 2017
arXiv 2017
-
[5]
Y . You, Y . Wang, H. Zhang, Z. Zhang, J. Demmel, and C.-J. Hsieh, “The limit of the batch size,”arXiv preprint arXiv:2006.08517, 2020
work page Pith review arXiv 2006
-
[6]
Control batch size and learning rate to generalize well: Theoretical and empirical evidence,
F. He, T. Liu, and D. Tao, “Control batch size and learning rate to generalize well: Theoretical and empirical evidence,”Advances in neural information processing systems, vol. 32, 2019
work page 2019
-
[7]
Revisiting small batch training for deep neural networks. arxiv 2018,
D. Masters and C. Luschi, “Revisiting small batch training for deep neural networks. arxiv 2018,”arXiv preprint arXiv:1804.07612, 1804
arXiv 2018
-
[8]
Adaptive batch size for federated learning in resource-constrained edge computing,
Z. Ma, Y . Xu, H. Xu, Z. Meng, L. Huang, and Y . Xue, “Adaptive batch size for federated learning in resource-constrained edge computing,” IEEE Transactions on Mobile Computing, vol. 22, no. 1, pp. 37–53, 2021
work page 2021
Show all 20 references
-
[9]
Mergesfl: Split federated learning with feature merging and batch size regulation,
Y . Liao, Y . Xu, H. Xu, L. Wang, Z. Yao, and C. Qiao, “Mergesfl: Split federated learning with feature merging and batch size regulation,” in 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024, pp. 2054–2067
2024
-
[10]
Dynamite: Dynamic interplay of mini-batch size and aggregation frequency for federated learning with static and streaming datasets,
W. Liu, X. Zhang, J. Duan, C. Joe-Wong, Z. Zhou, and X. Chen, “Dynamite: Dynamic interplay of mini-batch size and aggregation frequency for federated learning with static and streaming datasets,” IEEE Transactions on Mobile Computing, vol. 23, no. 7, pp. 7664–7679, 2023
2023
-
[11]
Adaptive batchsize selection and gradient compression for wireless federated learning,
S. Liu, G. Yu, R. Yin, J. Yuan, and F. Qu, “Adaptive batchsize selection and gradient compression for wireless federated learning,” inGLOBECOM 2020-2020 IEEE Global Communications Conference. IEEE, 2020, pp. 1–6
2020
-
[12]
Ada- coopt: Leverage the interplay of batch size and aggregation frequency for federated learning,
W. Liu, X. Zhang, J. Duan, C. Joe-Wong, Z. Zhou, and X. Chen, “Ada- coopt: Leverage the interplay of batch size and aggregation frequency for federated learning,” in2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS). IEEE, 2023, pp. 1–10
2023
-
[13]
To talk or to work: Dynamic batch sizes assisted time efficient federated learn- ing over future mobile edge devices,
D. Shi, L. Li, M. Wu, M. Shu, R. Yu, M. Pan, and Z. Han, “To talk or to work: Dynamic batch sizes assisted time efficient federated learn- ing over future mobile edge devices,”IEEE Transactions on Wireless Communications, vol. 21, no. 12, pp. 11 038–11 050, 2022
2022
-
[14]
Amble: Adjusting mini-batch and local epoch for federated learning with heterogeneous devices,
J. Park, D. Yoon, S. Yeo, and S. Oh, “Amble: Adjusting mini-batch and local epoch for federated learning with heterogeneous devices,”Journal of Parallel and Distributed Computing, vol. 170, pp. 13–23, 2022
2022
-
[15]
Flower: A friendly federated learning research framework,
D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y . Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusm ˜aoet al., “Flower: A friendly federated learning research framework,”arXiv preprint arXiv:2007.14390, 2020
2007 arXiv
-
[16]
Mobilenets: Efficient convo- lutional neural networks for mobile vision applications,
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convo- lutional neural networks for mobile vision applications,”arXiv preprint arXiv:1704.04861, 2017
2017 arXiv
-
[17]
The mnist database of handwritten digit images for machine learning research [best of the web],
L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],”IEEE signal processing magazine, vol. 29, no. 6, 2012
2012
-
[18]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778
2016
-
[19]
Learning multiple layers of features from tiny images,
A. Krizhevsky, G. Hintonet al., “Learning multiple layers of features from tiny images,” 2009
2009
-
[20]
Federated learning with matched averaging,
H. Wang, M. Yurochkin, Y . Sun, D. Papailiopoulos, and Y . Khaz- aeni, “Federated learning with matched averaging,”arXiv preprint arXiv:2002.06440, 2020
2002 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.