REVIEW 3 major objections 5 minor 24 references
Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A one-shot federated learning method, FedOL, claims to build a larger server model from heterogeneous clients by sharing only predictions on an unlabeled public dataset, outperforming baselines in a single round.
desk verdict Useful one-shot FL recipe with strong CIFAR-100 numbers, but the derivation of its central objective replaces private-data membership with an unvalidated confidence heuristic, and the missing ablations leave the actual mechanism unclear. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The objective in Eq (7): a per-client distillation loss $L_d(w_s)$ weighted by $\lambda_k(x) = \exp(-H(\sigma(w_k^c(x)))) / \sum_j \exp(-H(\sigma(w_j^c(x))))$ plus a pseudo-label loss $L_u(w_s, \hat{y})$, optimized by alternating pseudo-label generation and server updates. The pseudo-labels come from Algorithm 1's weighted vote over client and server predictions, using class-wise confidence scores $C^2_k$ and negative-learning label vectors.
What would settle it
Train a client on a single class and measure its average softmax confidence on public samples from classes it never saw; if that confidence is not clearly lower than on its own class, the $\lambda_k(x)$ weighting in Eq (6) is not tracking membership. A sharper test: replace $\lambda_k(x)$ with the true indicator $1[x \in D_{l,k}]$ (available in simulation) and compare final accuracy; if accuracy does not drop, the confidence surrogate is not the mechanism carrying the result.
Extended reading notes
Core claim
The paper claims that one-shot federated learning via prediction sharing on an unlabeled public dataset can build a larger server model than any client's, even when clients hold heterogeneous architectures and skewed label distributions. FedOL's server trains by minimizing Eq (7), which combines per-client distillation with iteratively refined pseudo-labels. On CIFAR-100, FedOL outperforms all baselines under one communication round in every partition setting, e.g., 37.2% vs 29.8% best baseline under Dir(1) and 29.8% vs 18.8% under Dir(0.05), and uses only 0.38 MB of communication per client compared with 65.14 MB for parameter sharing.
Load-bearing premise
The server's training loss assumes that a client's prediction confidence on a public sample reflects whether that sample's class appears in the client's private data, and if confidence does not track membership the central objective is optimizing the wrong surrogate.
Editorial extensions
If this is right
- With one communication round, FedOL beats the best baseline by more than 5.8% absolute accuracy under every tested partition of CIFAR-100.
- Communication per client drops to 0.38 MB for prediction sharing versus 65.14 MB for sending ResNet20 parameters.
- FedOL matches or exceeds knowledge-distillation baselines that are allowed 15 communication rounds, while itself using a single round.
- Clients are free to use different model architectures, since only predictions are shared, not weights.
- The method requires no labeled public data, only an unlabeled dataset shared by all participants.
Reading between the lines
- The confidence-as-membership surrogate in Eq (14) is untested; if confidence on a public sample does not track whether that sample's class appeared in the client's private data, the distillation loss optimizes an incorrect objective. A direct test would be replacing $\lambda_k(x)$ with the oracle membership indicator and comparing accuracy.
- The one-shot prediction-sharing protocol suggests a 'model market' where clients contribute pre-trained models or their predictions without iterative coordination; the paper's cost numbers (0.38 MB vs 65.14 MB) quantify the saving.
- The paper evaluates on CIFAR-100 only; the pseudo-label voting and entropy-threshold schedule could behave differently on larger or noisier public datasets, where confident predictions are systematically wrong.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. FedOL is a one-shot federated learning method in which clients train local models on private data and share only their predictions over an unlabeled public dataset. The server then trains a larger model by optimizing an objective (Eq. (7)) that combines a per-client distillation loss with a pseudo-label loss, using an entropy-based weighting lambda_k(x) (Eq. (14)) and a class-wise confidence-weighted voting scheme (Algorithm 1). Experiments on CIFAR-100 under four heterogeneous partitions report that FedOL outperforms existing one-shot and multi-round baselines by large margins, with lower communication cost. The central technical claim is that the server objective is derived from a semi-supervised learning loss by replacing private-data membership with a confidence-based surrogate.
Significance. The problem addressed is practically important: federated learning with heterogeneous client models and a single communication round is an attractive scenario for mobile networks, and the paper proposes a plausible system design with a favorable communication cost. If the empirical results hold, FedOL would be a useful contribution to one-shot federated distillation. The paper also makes a specific, falsifiable design choice—using prediction confidence as a proxy for membership and weighting votes by class-wise confidence—that could be a meaningful advance if validated. However, the central derivation in Eqs. (5)-(6) rests on an unvalidated surrogate, and the evaluation does not isolate which component of the method drives the gains. The significance is therefore conditional on the authors providing direct validation of the lambda_k(x) approximation and an ablation study.
major comments (3)
- [§III-B, Eqs. (5)-(6); §IV-A, Eq. (14)] The derivation of the server objective is not mathematically grounded as written. Eq. (5) defines L_d(w_s) as an expectation over private labeled data D_l with the membership indicator 1[x in D_l,k]. Eq. (6) replaces this with an expectation over the public dataset D_u using weights lambda_k(x). Under the experimental setup in §V-A, D_u and D_l are disjoint random splits, so for every x in D_u the exact indicator in Eq. (5) is zero; lambda_k(x) in Eq. (14) is therefore not an approximation of that indicator but a new heuristic. No theorem or experiment in the paper shows that softmax confidence on a public sample correlates with that sample's class being present in the client's private training set. Under severe label skew (Dir(0.05), #I_c=20), a client model can be highly confident on public samples from classes it never trained on, so the proposed lambda could be negatively correlated
- [§V-B, Table I] The evaluation does not isolate the contribution of the proposed objective. FedOL differs from knowledge-distillation baselines by several algorithmic choices: the per-client distillation loss with lambda_k(x), the pseudo-label loss L_u, the entropy-gated class-wise confidence-weighted voting in Algorithm 1, and the iterative refinement schedule. Table I only reports the full system against baselines. Without ablations (e.g., uniform lambda versus Eq. (14), removing L_u, replacing the weighted vote with a simple majority vote, or using a non-iterative single-pass pseudo-label generation), the reader cannot determine whether the reported gains come from the specialized objective in Eq. (7) or from the pseudo-label voting mechanism. The paper's formal contribution is the objective function, so this missing ablation is essential.
- [§III-B, Eq. (4)] A second surrogate step is introduced without analysis: the true label y in D_l,k is replaced by the client model's soft prediction sigma(w_k^c(x)) in Eq. (4). For private samples on which the client is not perfectly confident, this changes the loss being minimized, and error propagation is not discussed. This is related to the lambda_k(x) issue but is a separate approximation that should be acknowledged and, if possible, bounded or tested.
minor comments (5)
- [Table II] The cost table is difficult to read. The row 'Comm/MB 0.38 65.14 0.38' does not align with the column headers, and the entry 'Comp standard supervised-learning procedure' appears truncated. Please clarify which entries correspond to communication and which to computation, and specify the values for each method.
- [Eq. (14)] The denominator notation is malformed: 'Pj=K j=1' should be sum_{j=1}^K. Please fix the typesetting.
- [§V-A] Hyperparameters rho, tau, and the number of server iterations are given only as default values (rho starts at 0.1 and increases by 0.05, tau=0.2, 10 iterations). A sensitivity analysis over these parameters would improve reproducibility and help assess robustness.
- [Fig. 2] In the multi-round comparison, FedOL is shown as a single value over rounds. The caption should explicitly state that FedOL uses one communication round and is plotted as a horizontal line for comparison, to avoid ambiguity.
- [§V-A, Baselines] The one-shot adaptation of baselines such as FedDF, FedHKT, FedET, and FedKT is not specified in detail. For a fair comparison, please state the number of local epochs, server-side training epochs, and public dataset size used for each baseline in the one-shot setting.
Circularity Check
No significant circularity: the method is a heuristic KD/self-training scheme whose empirical claims are tested on held-out labels.
full rationale
The paper's derivation chain is not circular. Equations (4)-(6) replace the inaccessible labeled-data loss with a distillation loss using client soft predictions; this is a standard approximation, not an assumption of the conclusion. The weight λ_k(x) in Eq. (14) is introduced as a confidence-based heuristic and is not constructed so that Eq. (6) equals Eq. (5) by definition; the mismatch between the indicator over private data and the public-data surrogate is a correctness/validity concern, not circularity. Algorithm 1's pseudo-labels are generated from the same client (and server) predictions that appear in the distillation term, so the pseudo-label loss is self-referential in the sense of self-training/KD; however, the paper never claims these pseudo-labels are independent ground truth, and the final evaluation is against held-out CIFAR-100 labels. Thus the empirical claim is not pre-supplied by the construction. Self-citations [8] and [9] are background only and are not load-bearing. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors. The central mechanism is a heuristic aggregation scheme whose value is an empirical question, not a circular one.
Assumptions & free parameters
free parameters (3)
- rho (participation ratio) =
starts 0.1, increases by 0.05 per server iteration
- tau (pseudo-label loss weight) =
0.2
- server iteration rounds =
10
assumptions (5)
- domain assumption Client softmax predictions approximate true labels for samples in the client's private data, used to replace y with sigma(w_k^c(x)) in Eq (4).
- ad hoc to paper lambda_k(x) = exp(-H(sigma(w_k^c(x)))) / sum_j exp(-H(sigma(w_j^c(x)))) approximates the private-data membership indicator 1[x in D_l,k].
- domain assumption Prediction entropy is a valid reliability measure for selecting pseudo-label voters in Algorithm 1.
- domain assumption The public unlabeled set is representative enough of private data that knowledge transferred via D_u also improves D_l.
- standard math CE(u,v) = DKL(u||v) + H(u), the standard cross-entropy and KL divergence identity.
Cite this review
Pith. "Pith review of Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models." pith.science (2026). https://pith.science/paper/FTDWRNRK
@misc{pith2026250813625,
author = {Pith},
title = {Pith review of: Towards a Larger Model via One-Shot Federated Learning on Heterogeneous Client Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/FTDWRNRK}},
note = {Machine review of arXiv:2508.13625}
}
read the original abstract
Large models, renowned for superior performance, outperform smaller ones even without billion-parameter scales. While mobile network servers have ample computational resources to support larger models than client devices, privacy constraints prevent clients from directly sharing their raw data. Federated Learning (FL) enables decentralized clients to collaboratively train a shared model by exchanging model parameters instead of transmitting raw data. Yet, it requires a uniform model architecture and multiple communication rounds, which neglect resource heterogeneity, impose heavy computational demands on clients, and increase communication overhead. To address these challenges, we propose FedOL, to construct a larger and more comprehensive server model in one-shot settings (i.e., in a single communication round). Instead of model parameter sharing, FedOL employs knowledge distillation, where clients only exchange model prediction outputs on an unlabeled public dataset. This reduces communication overhead by transmitting compact predictions instead of full model weights and enables model customization by allowing heterogeneous model architectures. A key challenge in this setting is that client predictions may be biased due to skewed local data distributions, and the lack of ground-truth labels in the public dataset further complicates reliable learning. To mitigate these issues, FedOL introduces a specialized objective function that iteratively refines pseudo-labels and the server model, improving learning reliability. To complement this, FedOL incorporates a tailored pseudo-label generation and knowledge distillation strategy that effectively integrates diverse knowledge. Simulation results show that FedOL significantly outperforms existing baselines, offering a cost-effective solution for mobile networks where clients possess valuable private data but limited computational resources.
Figures
Reference graph
Works this paper leans on
-
[1]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, et al., “Language models are few-shot learners,” in NeurIPS, 2020
work page 2020
-
[2]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
D. Guo, et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.” arXiv preprint arXiv:2501.12948, 2025
arXiv 2025
-
[3]
Why do larger models generalize better? A theoretical perspective via the XOR problem,
A. Brutzkus, and A. Globerson, “Why do larger models generalize better? A theoretical perspective via the XOR problem,” in ICML, 2019
work page 2019
-
[4]
C. Qian, G. Rattan, F. Geerts, M. Niepert, C. Morris, ”Ordered subgraph aggregation networks,” in NeurIPS, 2022
work page 2022
-
[5]
Advances and open problems in federated learning,
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, et al., “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning, 2021
work page 2021
-
[6]
Communication-efficient learning of deep networks from decentralized data,
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “ Communication-efficient learning of deep networks from decentralized data,” in AISTATS, 2017
work page 2017
-
[7]
Federated optimization in heterogeneous networks,
T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” in MLSys, 2020
work page 2020
-
[8]
FLaaS6G: Federated learning as a service in 6G using distributed data management architec- ture,
W. Ye, X. An, X. Yan, M. Hamad, S. Steinhorst, “FLaaS6G: Federated learning as a service in 6G using distributed data management architec- ture,” in Globecom, 2022
work page 2022
Show all 24 references
-
[9]
Advancing federated learning in 6G: A trusted architecture with graph-based analysis,
W. Ye, C. Qian, X. An, X. Yan, G. Carle, “Advancing federated learning in 6G: A trusted architecture with graph-based analysis,” in Globecom, 2023
2023
-
[10]
Distilling the knowledge in a neural network,
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015
2015 arXiv
-
[11]
Het- erogeneous ensemble knowledge transfer for training large models in federated learning,
Y . J. Cho, A. Manoel, G. Joshi, R. Sim, and D. Dimitriadis, “Het- erogeneous ensemble knowledge transfer for training large models in federated learning,” in IJCAI, 2022
2022
-
[12]
Fedgems: Federated learning of larger server models via selective knowledge fusion,
S. Cheng, J. Wu, Y . Xiao, and Y . Liu, “Fedgems: Federated learning of larger server models via selective knowledge fusion,” arXiv preprint arXiv:2110.11027, 2021
2021
-
[13]
Ensemble distillation for robust model fusion in federated learning,
T. Lin, L. Kong, S. U. Stich, M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in NeurIPS, 2020
2020
-
[14]
Not all knowledge is created equal: Mutual distillation of confident knowledge,
Z. Li, X. Wang, D. Hu, N. M. Robertson, D. A. Clifton, C. Meinel, and H. Yang, “Not all knowledge is created equal: Mutual distillation of confident knowledge,” in NeurIPS workshop, 2022
2022
-
[15]
Personalized federated learning for heterogeneous clients with clustered knowledge transfer,
Y . J. Cho, J. Wang, T. Chiruvolu, and G. Joshi, “Personalized federated learning for heterogeneous clients with clustered knowledge transfer,” arXiv preprint arXiv:2109.08119, 2021
2021 arXiv
-
[16]
A hierarchical knowledge transfer framework for heterogeneous federated learning,
Y . Deng, J. Ren, C. Tang, F. Lyu, Y . Liu, and Y . Zhang, “A hierarchical knowledge transfer framework for heterogeneous federated learning,” in INFOCOM, 2023
2023
-
[17]
One-shot federated learning,
N. Guha, A. Talwalkar, and V . Smith, “One-shot federated learning,” arXiv preprint arXiv:1902.11175, 2019
1902 arXiv
-
[18]
Dense: Data-free one-shot federated learning,
J. Zhang, C. Chen, B. Li, L. Lyu, S. Wu, S. Ding et al, “Dense: Data-free one-shot federated learning,” in NeurIPS, 2022
2022
-
[19]
Towards addressing label skews in one-shot federated learning,
Y . Diao, Q. Li, and B. He, “Towards addressing label skews in one-shot federated learning,” in ICLR, 2023
2023
-
[20]
Semi-supervised learning by entropy minimization,
Y . Grandvalet and Y . Bengio, “Semi-supervised learning by entropy minimization,” in NeurIPS, 2004
2004
-
[21]
Self-paced curriculum learning,
L. Jiang, D. Meng, Q. Zhao, S. Shan, and A. Hauptmann, “Self-paced curriculum learning,” in AAAI, 2015
2015
-
[22]
Nlnl: Negative learning for noisy labels,
Y . Kim, J. Yim, J. Yun, and J. Kim, “Nlnl: Negative learning for noisy labels,” in ICCV , 2019
2019
-
[23]
Federated learning on non-iid data silos: An experimental study,
Q. Li, Y . Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in ICDE, 2022
2022
-
[24]
Practical one-shot federated learning for cross-silo setting,
Q. Li, B. He, and D. Song, “Practical one-shot federated learning for cross-silo setting,” in IJCAI, 2021
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.