REVIEW 2 major objections 6 minor 13 references
Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A two-step schedule lets federated clients too resource-poor for backpropagation train a model from random initialization using only forward passes, recovering accuracy lost when those clients are excluded.
desk verdict Warm-up plus zeroth-order is a real new recipe for federated pre-training and the accuracy results are mostly solid, but the memory-savings claim is overstated by roughly 4x and the low-resource simulation is only label-based. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the two-step training regime of Algorithm 1: a first-order warm-up on high-resource clients, then a zeroth-order phase open to all clients. The zeroth-order engine is SPSA (Eq. 2), which needs two forward passes per perturbation direction and no backpropagation. The protocol exploits this: the server broadcasts S seeds, each client regenerates its S perturbation directions locally, computes S scalars ΔL = L(w + εz, B) − L(w − εz, B), and returns only those scalars, making uplink cost negligible. Two variance-reduction choices complete the mechanism: perturbations are drawn from a scaled Rademacher distribution (τ·Rad) instead of a Gaussian, and clients take one grad
What would settle it
Run ZOWarmUp on clients whose peak memory is hard-capped at the zeroth-order footprint of Eq. 5—weights plus the single largest activation buffer, with no gradient or activation storage—and whose uplink is capped at S 32-bit scalars per round; if the warm-up phase fails to produce a usable model, or the accuracy gain over the High-Res-Only baseline vanishes under the real cap, the feasibility claim is refuted.
Extended reading notes
Core claim
Zeroth-order pre-training fails on variance, not memory, and a two-phase schedule removes that barrier. Clients that can run backpropagation first train the model from random initialization (FedAvg or FedAdam); after a tuned 'pivot' round, all clients switch to zeroth-order updates: the server sends S seeds, each client computes S loss differences at positive and negative weight perturbations (SPSA, Eq. 2), and returns only those S scalars. This schedule beats the high-resource-only baseline in every tested split and HeteroFL in most, and even lets FedKSeed converge from random initialization when FedKSeed is used as the second-phase optimizer—the warm start itself unlocks from-scratch zerot
Load-bearing premise
The central claim assumes that a 'low-resource' client can still hold the full model weights, run forward passes over its local data, and return S scalar loss values reliably—if even that is beyond the device, ZOWarmUp cannot involve it and the recovered accuracy is unreachable; a second fragility is that the warm-up/pivot round must be tuned, since a mistimed switch erases the gains (Figure 4).
Editorial extensions
If this is right
- Federated pre-training no longer requires every participant to run backpropagation: any client that can hold the weights, run forward passes, and return a few scalars can contribute its data, shrinking system-induced bias.
- Zeroth-order federated methods gain a generic path from random initialization: the warm-up phase is agnostic to the choice of high-resource algorithm (FedAvg or FedAdam) and to the choice of second-phase zeroth-order optimizer (the paper's variant or FedKSeed).
- Per-client communication drops to S floating-point scalars per round and on-device memory falls roughly six-fold versus FedAvg for a ResNet18 (89.4 MB vs 533.2 MB), widening the set of devices that can participate.
- Single-step zeroth-order updates dominate multi-step updates at equal data (final Rouge-L 0.2015 vs 0.1723 in the FedKSeed comparison), so future federated zeroth-order algorithms should favor one step per round.
- Accuracy improves whenever low-resource data is admitted—even at a 90/10 split where high-resource clients already see most of the data—so excluding any reachable data is measurably harmful.
Reading between the lines
- The low-resource label is assigned in simulation, not enforced as a hardware ceiling; testing the method on devices with memory hard-capped at the zeroth-order footprint (Eq. 5) and with uplink limited to S scalars would tell whether the gains survive real constraints.
- The paper's critical-learning-period explanation implies a testable, hyperparameter-free alternative: choose the pivot online, at the round where step-one accuracy gains saturate or the zeroth-order gradient signal-to-noise ratio becomes usable, instead of tuning the pivot per task.
- The Rademacher-over-Gaussian result points to perturbation variance, not estimator unbiasedness, as the binding constraint in from-scratch zeroth-order training—so other low-variance perturbation schemes deserve explicit tests that the paper does not run.
- A system-design consequence the paper states but does not pursue: capable clients should downgrade to zeroth-order updates once the pivot passes, accepting noisier updates for update homogeneity; whether that homogeneity trades off against the wasted capability of strong clients is an open question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ZOWarmUp, a two-phase federated learning method for training neural networks from random initialization when a large fraction of clients are too resource-constrained to run backpropagation or full-bandwidth communication. In phase 1, high-resource clients train the model with a standard federated optimizer for N rounds; in phase 2, all clients switch to a zeroth-order (SPSA/MeZO-style) update that requires only scalar loss values per random seed, combined with variance-reduction choices (Rademacher perturbations scaled by tau, S perturbations per step, and a single local gradient step). Experiments on CIFAR-10 and ImageNet32 with ResNet18 and ViT-B/16, under simulated high/low resource splits, show that ZOWarmUp improves over a high-resource-only baseline and generally outperforms HeteroFL and a FedKSeed-based warm-start baseline, while also showing that the pivot round is a sensitive hyperparameter.
Significance. If the resource-feasibility premise were established, ZOWarmUp would be a useful step: it is, to my knowledge, the first federated zeroth-order method demonstrated from random initialization, and the two-phase idea is simple and compatible with other ZO optimizers. The experimental package is reasonably strong: results are averaged over five seeds, several resource splits are tested, and the paper includes ablations of the perturbation distribution, tau, S, gradient steps, pivot point, server optimizer, and model architecture. However, the central claim that low-resource clients can participate under realistic memory ceilings is not currently supported, because the reported memory numbers omit a term in the paper's own formula. This tempers the significance but does not make the algorithmic idea unsound.
major comments (2)
- [Section 3.1, Table 1, Eq. 5, Section A.5] Table 1 reports 89.4 MB for the ZO client, which is exactly the 2P term (2 x 11.2M x 4 bytes) of Eq. 5. Eq. 5 also contains BS * max(N_l * W_l * H_l); for the experimental setting of Section 4 and A.5, where the ZO batch size is the whole client dataset, the largest ResNet18 activation map is 64 x 32 x 32 floats. With roughly 1000 CIFAR-10 samples per client, the activation term is about 262 MB, giving about 352 MB total rather than 89.4 MB. The advertised ~6x saving over FedAvg (533 MB) becomes about 1.5x at this batch size. Since the low-resource split is assigned without imposing a measured memory or bandwidth ceiling, the paper does not yet demonstrate that the clients it claims to include actually fit within the resource envelope. Please recompute the table with the actual batch sizes and state the client memory threshold used in the simulation.
- [Section 4, Section A.5] The empirical evaluation labels clients as low-resource purely by sampling a ratio; no resource ceiling is measured or enforced. The central motivation is that such clients are below a critical memory/communication threshold, so the simulation should verify that the ZO procedure with its chosen batch size, S perturbations, and communication of S*K scalars fits under a stated per-client budget for each resource class. Without this, the reported accuracy gains could reflect a data-abundance effect rather than the inclusion of clients under the claimed constraints. A concrete fix is to report peak memory using Eq. 5 with the real batch size and uplink/downlink bytes for each split, and to ablate around the threshold.
minor comments (6)
- [Abstract] Typo: 'can can be applied' should be 'can be applied'.
- [Section 4.1 and Conclusion] The claim that ZOWarmUp 'consistently outperforms other resource efficient baselines' is stronger than Table 2 supports: for ImageNet32 at the 10/90 split, HeteroFL achieves 16.3 vs. ZOWarmUp's 12.3, and the text itself acknowledges this exception. Suggest rewording to 'generally outperforms' or explicitly qualifying the exception.
- [Algorithm 1] Clarify whether high-resource clients also execute ZOUPDATE in the second phase. The pseudocode updates only clients j in L (line 18), while the text says all clients switch to low-resource training; the discrepancy matters for model synchronization in subsequent rounds.
- [Table 7] The labels 'ZOWarmUp(hi+lo)' and 'ZOWarmUp(lo only)' are confusing given the prose. Rename to something like 'mixed high/low updates' vs. 'all-ZO updates'.
- [Table 3 and Section 4.2] The column header 'NO. GRAD STEPS' should be 'NUM. GRAD STEPS'. Also clarify whether the compared configurations use the same total sample budget per round and how 'effective BS' relates to the full-client-dataset batch size used elsewhere.
- [Section 4, Appendix A.5] State the actual number of training samples per client for ImageNet32; the text only says data is partitioned equally among 50 clients, but this number is needed to evaluate Eq. 5 and the feasibility claim.
Circularity Check
No significant circularity: empirical algorithm paper; minor self-citations are not load-bearing.
full rationale
The paper's central contribution is an empirical algorithm (ZOWarmUp) and its evaluation; it does not derive a prediction from first principles in a way that reduces to its inputs. The two-step warm-start procedure is a design choice, not a quantity fitted to the result. Hyperparameters (τ, S, pivot round, learning rates) are tuned via grid search and openly ablated in Figures 6–7, Table 3, and Figure 4, so they are reported as fitted choices rather than disguised predictions. The choice of the Rademacher distribution is justified by an empirical comparison (Table 6) on the same benchmark, which is hyperparameter selection, not circular reasoning. No uniqueness theorem, ansatz, or fitted parameter from the authors' prior work is invoked to force the central claim. Self-citations (Legate et al., 2023) appear in the introduction and in a peripheral explanation of logit masking; they are not load-bearing for the method's validity or the main experimental results. The only notable quantitative concern is the on-device memory figure in Table 1, which appears to omit the BS·max(Nℓ·Wℓ·Hℓ) term of Eq. 5 under the batch-size setting described in §A.5. That is an internal-consistency or feasibility issue, not circularity, because the memory claim is not used to define or construct the measured outcomes. Overall, the paper's derivation chain is self-contained against external benchmarks, and no step equates an output with an input by construction.
Assumptions & free parameters
free parameters (5)
- epsilon (SPSA perturbation scale) =
1e-4
- tau (Rademacher scale) =
0.75
- S (number of perturbation seeds) =
3
- pivot point (warm-up rounds) =
200
- learning rates (server/client, step1/step2) =
not reported
assumptions (4)
- domain assumption SPSA with Rademacher perturbations provides a usable gradient estimate for nonconvex deep networks.
- domain assumption Low-resource clients can hold full model weights, run forward passes, and communicate S scalar loss differences.
- domain assumption A warm-up phase using high-resource clients stabilizes weights enough for ZO training from random initialization.
- domain assumption Taking a single gradient step per round avoids client drift for noisy ZO updates.
Cite this review
Pith. "Pith review of Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients." pith.science (2026). https://pith.science/paper/JM6GKF6J
@misc{pith2026250903503,
author = {Pith},
title = {Pith review of: Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients},
year = {2026},
howpublished = {\url{https://pith.science/paper/JM6GKF6J}},
note = {Machine review of arXiv:2509.03503}
}
read the original abstract
Federated learning enables collaborative model training across numerous edge devices without requiring participants to share data; however, memory and communication constraints on these edge devices may preclude their participation in training. We consider a setting in which a subset of edge devices are below a critical memory or communication threshold required to conduct model updates. Under typical federated optimization algorithms, these devices are excluded from training which renders their data inaccessible and increases system induced bias. We are inspired by MeZO, a zeroth-order method used for memory-efficient fine-tuning. The increased variance inherent to zeroth-order gradient approximations has relegated previous zeroth-order optimizers exclusively to the domain of fine tuning; a limitation we seek to correct. We devise a federated, memory-efficient zeroth-order optimizer, ZOWarmUp that permits zeroth-order training from a random initialization. ZOWarmUp leverages differing client capabilities and careful variance reduction techniques to facilitate participation of under-represented, low-resource clients in model training. Like other federated zeroth-order methods, ZOWarmUp eliminates the need for edge devices to transmit their full gradients to the server and instead relies on only a small set of random seeds, rendering the up-link communication cost negligible. We present experiments using various datasets and model architectures to show that ZOWarmUp is a robust algorithm that can can be applied under a wide variety of circumstances. For systems with a high proportion of edge devices that would otherwise be excluded from training, this algorithm provides access to a greater volume and diversity of data, thus improving training outcomes.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Optimization without backpropagation
Gabriel Belouze. Optimization without backpropagation. arXiv preprint arXiv:2209.06302,
-
[4]
doi: 10.1109/TSP.2022.3214122. Cristiano Gratton, Naveen K. D. Venkategowda, Reza Arablouei, and Stefan Werner. Privacy-preserved distributed learning with zeroth-order optimization. IEEE Transactions on Information Forensics and Security , 17:265–279,
-
[5]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
doi: 10.1109/TIFS.2021.3139267. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 770–778,
-
[10]
Critical Learning Periods in Federated Learning
Gang Yan, Hao Wang, and Jian Li. Critical learning periods in federated learning. arXiv preprint arXiv:2109.05613,
-
[11]
Federated learning with non-iid data
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582,
-
[13]
49.4 7.7 11.9 2.9 Rademacher 65.5 5.2 9.3 1.4 Table 6: A comparison of variance in the results obtained using both Gaussian and Rademacher distributions to compute the zeroth-order gradient approximations. The Rademacher distribution exhibits considerably lower variance and better overall accuracy than the Gaussian distribution. A A PPENDIX A.1 SPSA W ITH...
work page 2020
-
[1992]
doi: 10.1109/9.119632. A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
-
[2009]
cs.toronto.edu/˜kriz/learning-features-2009-TR.pdf
URL https://www. cs.toronto.edu/˜kriz/learning-features-2009-TR.pdf . Gwen Legate, Lucas Caccia, and Eugene Belilovsky. Re-weighted softmax cross-entropy to control forgetting in federated learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 ofProceedings of Machine Learning Research, pp. 764–780. PMLR, 22–25 Aug
work page 2009
Show all 13 references
-
[2015]
Model-heterogeneous federated learning for internet of things: Enabling technologies and future directions
Boyu Fan, Siyang Jiang, Xiang Su, and Pan Hui. Model-heterogeneous federated learning for internet of things: Enabling technologies and future directions. arXiv preprint arXiv:2312.12091,
-
[2018]
Understanding why vit trains badly on small datasets: An intuitive perspective
Haoran Zhu, Boyuan Chen, and Carter Yang. Understanding why vit trains badly on small datasets: An intuitive perspective. arXiv preprint arXiv:2302.03751,
-
[2020]
Zhong Long, Yuling Chen, Hui Dou, Yun Luo, Chaoyue Tan, and Yancheng Sun
11 Preprint. Zhong Long, Yuling Chen, Hui Dou, Yun Luo, Chaoyue Tan, and Yancheng Sun. Communication-efficient federated learning with sparsity and quantization. In 2023 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Compu...
2023
-
[2022]
A downsampled variant of imagenet as an alternative to the cifar datasets
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819,
-
[2024]
Zerofl: Efficient on-device training for federated learning with local sparsity
Xinchi Qiu, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507 ,
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.