Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead

Hyeon-Seo Park; Si-Hyeon Lee; Won-Jun Jang

REVIEW 2 major objections 4 minor 76 references

Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead

T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that per-point pseudo-label weights computed from the odds of client discriminators make a federated ensemble provably no worse, in expected loss, than the best single model on the average client distribution.

desk verdict The core weighting idea is clean and the experiments are honest, but the 'provably' claim does not cover the implemented discriminator because of the capped-odds output. read the letter →

arxiv 2502.06349 v2 pith:M42XPOAV submitted 2025-02-10 cs.LG

Won-Jun Jang , Hyeon-Seo Park , Si-Hyeon Lee This is my paper · ORCID

classification cs.LG

keywords federatedlearningensembledistillationclientheterogeneityGANdiscriminatorspseudo-labelingoptimalmodelknowledgenon-IIDdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated ensemble distillation lets a server label unlabeled data by combining client model outputs, and the quality of those pseudo-labels depends on how each client is weighted per data point. This paper aims to establish that the optimal weight for client $k$ at point $x$ is that client's share of the local data density, $w^*_k(x)=\pi_k p_k(x)/\sum_i \pi_i p_i(x)$, and that this weight can be computed from the odds of a client discriminator trained against a server-distributed generator. For convex losses the resulting ensemble is an optimal model ensemble: its expected loss on the average client distribution is no larger than the best single model's loss on that distribution. A server model distilled from those pseudo-labels remains near-optimal up to an explicit distillation gap and a distribution-discrepancy term. If correct, this turns the weight-selection problem in federated ensemble distillation into a density-ratio estimation problem that can be solved once per client, with negligible extra communication and client computation.

What carries the argument

The central object is the per-point client weight $w^*_k(x)=\pi_k p_k(x)/\sum_i \pi_i p_i(x)$, which is the posterior probability that a point $x$ came from client $k$ under the mixture of client distributions. What makes it computable is the GAN discriminator identity: an optimal discriminator against a fixed generator with density $p_g$ outputs $D_k(x)=p_k(x)/(p_k(x)+p_g(x))$, so its odds $\Phi(\phi)=\phi/(1-\phi)$ equal the density ratio $p_k(x)/p_g(x)$; normalizing these odds by client sample sizes gives $w^*_k(x)$. The argument is carried by the optimal-model-ensemble definition together with a domain-adaptation decomposition, $L_p(h)\le L_p(\sum_k w_k h_k)+L_{p_s}(h,\sum_k w_k h_k)+\frac{1}{2}d_{\bar H\triangle\bar H}(p,p_s)$, which converts the ensemble guarantee into the near-optimality bound for the distilled single model.

What would settle it

Train two clients on disjoint halves of a two-mode Gaussian mixture and give one client an extra third mode; use a generator whose support omits that third mode. After training discriminators to convergence, compute the odds-based pseudo-label weights and measure the ensemble loss on the mixture. If that loss exceeds the best single model's loss on the mixture, the guarantee has failed because the support-coverage or discriminator-optimality condition is violated; if it stays below, then approximate discriminators and partial support coverage are not the binding constraint in this setting.

Watch

Extended reading notes

Core claim

Under a convex loss, the paper proves (Theorem 3.4) that the ensemble $\{h^*_{p_k}, w^*_k\}$ with $w^*_k(x)=\pi_k p_k(x)/\sum_i \pi_i p_i(x)$ satisfies $L_p(\sum_k w^*_k h^*_{p_k}) \le L_p(h^*_p)$, so the weighted client models beat or match the best single model on the average client distribution. Theorem 3.6 then shows these weights are realizable: for an optimal discriminator $D_k$ against a fixed generator with density $p_g$, the odds $\Phi_k(x)=p_k(x)/p_g(x)$ give $n_k\Phi_k(x)/\sum_i n_i\Phi_i(x)=w^*_k(x)$ on the generator's support. Theorems 3.2 and Corollary 3.3 convert ensemble optimality into a server-model guarantee, bounding the distilled model's loss by the optimal single-model loss plus the distillation loss and half the discrepancy between the average client distribution and the server's distillation distribution.

Load-bearing premise

The proof assumes each client's discriminator is optimal against the fixed generator and that the generator's distribution covers the support of the average client distribution, so that discriminator odds exactly equal the client density divided by the generator density; the paper itself notes that a pretrained ImageNet generator misses several CIFAR-100 classes, meaning the implemented method relies on this condition only approximately.

Editorial extensions

If this is right

For convex losses, the odds-weighted client ensemble is guaranteed to be at least as good as the best single model trained on the pooled average client distribution, eliminating the need to hand-tune per-client aggregation weights.
The server model distilled from those pseudo-labels carries a provable bound: its loss is the optimal single-model loss plus a distillation gap and a distribution-discrepancy term (Corollary 3.3).
Because clients train their discriminators once in a pre-FL stage, the extra communication, client-side privacy leakage, and client-side computation do not grow with the number of federated rounds.
The same weight construction applies both when the server has an unlabeled dataset and when it must generate a distillation dataset, so the method covers data-free federated ensemble distillation as well.
The reported experiments show the largest gains on the most heterogeneous splits, with the server model approaching central-training accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the only role of the discriminator is to estimate the density ratio $p_k/p_g$, any other density-ratio estimator could be dropped into the same weighting recipe; that would decouple the method from GAN training and might reduce its dependence on discriminator optimality.
The optimality proof is for convex losses and, in the formal bound, binary $\ell^1$ losses; extending the result to nonconvex deep-network losses or to clients with different model architectures would need a new argument, and the paper states that model-heterogeneous ensembles are currently out of reach.
A testable prediction of the theory is that the method's advantage over uniform weighting should grow with heterogeneity and shrink as the generator's support covers less of the client distribution; measuring ensemble loss while varying generator coverage would separate the density-ratio error from the distillation gap.
The one-shot discriminator transmission is itself a client-side disclosure; under local differential privacy, its privacy cost should be composed with the per-round model updates, and the paper's analysis already accounts for this as a one-time additive term.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful part of this paper is the weighting scheme: use client discriminators trained once against a server-distributed generator to estimate density ratios, then weight client predictions by odds. The convexity argument in Theorem 3.4 is simple and correct, and the generalization bound in Theorem C.1 is tighter than prior work. Experiments are thorough — multiple datasets, heterogeneity levels, data-free variants, client counts, and an honest cost/privacy analysis. The toy example is illustrative. Code is released.

The soft spot is real and internal. The theory says the optimal discriminator odds equal p_k(x)/p_g(x), and Theorem 3.6 converts that into the weighting w*_k. But the implemented discriminator in Appendix E.2 uses two sigmoids to clamp the odds to [1, e]. That means even with perfectly trained discriminators and full generator support, the actual weights are not w*_k. The paper never quantifies the gap between the capped weights and the oracle weights. The support-coverage caveat in Section 4.2 is secondary; this cap breaks the identity on the support of p_g as well. The title and abstract promise "provably near-optimal," and the proof is for an algorithm nobody ran.

That said, the empirical method may still be fine — the cap is a heuristic to prevent discriminator overconfidence, and the gains on CIFAR-10 are large. The modest improvements on CIFAR-100 and ImageNet100 are acknowledged and plausibly explained by distillation loss. The binary-class ℓ1 theory vs multi-class experiments is a standard gap, worth noting but not disqualifying.

The paper is worth engaging seriously. A referee should ask for a perturbation analysis or an adjusted theorem that covers the capped odds, and for the "provably" language to be scoped to the ideal setting. I would accept it conditionally, and I would cite it for the weighting idea while flagging the implementation caveat.

Referee Report

2 major / 4 minor

Summary. The paper proposes FedGO, a federated ensemble distillation algorithm that assigns per-sample weights to client predictions using client discriminators trained against a server-side generator. The authors define an optimal model ensemble, prove that density-ratio weights w*_k(x) = π_k p_k(x) / Σ_i π_i p_i(x) achieve the optimal ensemble loss under a convex loss (Theorem 3.4), and show that these weights can be obtained from the odds of optimal client discriminators (Theorem 3.6). They also derive a generalization bound for the ensemble with these weights (Theorem C.1). Experiments on CIFAR-10/100 and ImageNet100 with Dirichlet heterogeneity show that FedGO improves server test accuracy and convergence speed over several federated ensemble distillation baselines, with a claimed negligible overhead in communication and client-side computation.

Significance. If the theoretical optimality claim applied to the implemented algorithm, this would be a valuable contribution: it provides a principled, theory-motivated weighting scheme for federated ensemble distillation, with a tight generalization bound and a careful overhead analysis. The paper's main strengths are the clean proof of Theorem 3.4, the explicit use of the GAN density-ratio identity, the open-source code for reproducibility, and the extensive experimental evaluation including data-free scenarios. However, the central guarantee is established only for an idealized version of the algorithm, and the evaluated implementation uses a discriminator output that provably does not produce the required odds. This gap currently prevents the paper from substantiating the 'provably near-optimal' claim for the method that is actually run.

major comments (2)

[Appendix E.2, Eq. (8)-(9), Theorem 3.6] The implemented FedGO weighting does not satisfy the assumption of Theorem 3.6. Appendix E.2 states that the discriminator output is 'a composition of two sigmoid activations' that constrains the odds value Φ_k between 1 and e. Therefore, even with optimal discriminators and full generator support, the computed weights are n_k · g(p_k(x)/p_g(x)) / Σ_i n_i · g(p_i(x)/p_g(x)) for a monotone saturating map g, which is not equal to w*_k(x) = n_k p_k(x) / Σ_i n_i p_i(x) from Eq. (8) whenever the density ratios differ across clients—precisely the heterogeneous settings the paper targets. Consequently, the optimality results in Corollary 3.3, Theorem C.1, and Table 1 do not apply to the weights used in the experiments, and no bound quantifies the gap introduced by the saturation. This is a load-bearing mismatch because the paper's central claim is that the proposed weighting is provably near-optimal.
[Section 3.2 / Appendix E.2, Algorithm 2-3, Theorem 3.4] Even if the saturation issue were removed, the theoretical guarantee assumes ideal objects that the algorithm does not provide. Theorem 3.4 uses true risk minimizers h*_{p_k}, and Theorem 3.6 uses optimal discriminators, while Algorithm 2/3 use empirical risk minimizers trained for a small number of epochs and discriminators trained for 10-30 epochs (Appendix E.2). The paper provides no perturbation or robustness analysis showing how the ensemble loss L_p(Σ_k w_k h_k) degrades with suboptimal discriminators or with h*_{\hat{p}_k} in place of h*_{p_k}. The support-coverage caveat noted in Section 4.2 is related but secondary: Theorem 3.6 only holds on supp(p_g), and for the pretrained generator (G2) the paper itself notes that several CIFAR-100 classes are not covered by ImageNet. Without a quantified gap or additional assumptions, the provable near-optimality does not transfer to the evaluated algorithm.

minor comments (4)

[Section 3.1.1, Theorem 3.2] The main-text statement of Theorem 3.2 is labeled 'Informal' and does not mention that the formal version in Appendix A is restricted to binary classification with ℓ1 loss; please state these restrictions in the main text to prevent over-generalization.
[Appendix F.3, Table 7] The DaFKD ensemble test accuracy for α = 0.05 is reported as '56.06 ±207', which appears to be a typo for '56.06 ±2.07'.
[Section 4.2] The statement that ImageNet 'includes the classes of CIFAR-10 except deer' is imprecise; ImageNet has deer in multiple synsets, so please clarify what is meant by class inclusion.
[Appendix C, Theorem C.1] The term 4 + sqrt(log(τ_H(2n_k))) / ((δ/K)√(2n_k)) places the confidence parameter in the denominator, which makes the bound increasingly loose as δ decreases; please verify this expression against the cited Theorem 6.11 and clarify the notation in Table 1.

Circularity Check

1 steps flagged · score 2.0 of 10

No substantive circularity: the weight-optimality proof is self-contained; only a minor definitional circularity arises from defining 'optimal model ensemble' as exactly the inequality the construction proves.

self definitional [Definition 3.1 and Theorem 3.4 (Section 3.1)]
"Definition 3.1: ... is said to be an optimal model ensemble if ... Lp(Σ_k w_k·h_k) ≤ min_{h∈H} Lp(h) = Lp(h*_p). ... Theorem 3.4: Define w*_k(x) ≜ n_k·p_k(x)/Σ_i n_i·p_i(x). Then, the ensemble {h*_{p_k}, w*_k} is an optimal model ensemble, i.e., Lp(Σ_k w*_k·h*_{p_k}) ≤ Lp(h*_p)."

The predicate 'optimal model ensemble' is defined as the exact inequality that Theorem 3.4 proves for the proposed weights. In that sense the theorem verifies that the constructed weights satisfy the definition; it is not an optimization over all possible weighting rules. The convexity argument itself is mathematically valid and self-contained, so this is a semantic/definitional issue in the phrase 'provably near-optimal' rather than a circular computation.

full rationale

The main derivation chain is not circular. Theorem 3.4 is proved from convexity and the definitions of w*_k and h*_{p_k}; Theorem 3.6 uses the external GAN result of Goodfellow et al. (Proposition 1) plus the odds transformation; and Theorem C.1 is a standard empirical-risk/union-bound argument. No fitted parameter is renamed as a prediction, and no load-bearing self-citation chain is used: the authors' prior work (Park et al. 2024) is cited only as an entropy-weighting baseline. The one definitional circularity is flagged above: 'optimal model ensemble' in Definition 3.1 is stipulated to be exactly the inequality that the proposed w*_k can prove, so the word 'optimal' carries less independent content than the paper's 'provably near-optimal' phrasing suggests. There is also a stated scoping gap that I do not count as circularity: Appendix E.2 says the discriminator output uses 'a composition of two sigmoid activations ... ensures that the odds value Φ_k ... is constrained between 1 and e,' so the implemented Φ_k is not p_k/p_g and Theorem 3.6 does not strictly cover the executed algorithm. That is a correctness/scope gap acknowledged in the paper, not a reduction of the result to its inputs, and it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The provable near-optimality is conditional on idealized conditions that the implemented algorithm does not guarantee: true client risk minimizers, optimal discriminators, full support coverage, and (for the main distillation theorem) binary ℓ1 loss. Several hyperparameters that materially affect empirical performance are chosen by hand or grid search and are not part of the theoretical claim.

free parameters (3)

Discriminator training epochs = 30 (CIFAR-10/100), 10 (ImageNet100)
Chosen by hand; Table 13 shows accuracy varies from 63.96 to 76.92 as this changes. Theorem 3.6 assumes optimal discriminators, which finite epochs violate.
Generator training steps (G1/G3) = 100,000 WGAN-GP steps; 5 FedGAN rounds for data-free
Controls pg support and discriminator input quality; Table 12 shows sensitivity to this value.
Server distillation epochs E_s = 10 (CIFAR-10/100), 3 (ImageNet100)
Grid-searched on CIFAR-100 α=0.05 and reused elsewhere; Table 10 shows moderate sensitivity.

assumptions (7)

domain assumption Loss function l is convex (Theorem 3.4, Lemma B.1).
The optimality proof relies on Jensen's inequality. Cross-entropy is convex on the probability simplex, but the theorem is stated for generic convex l and ignores softmax parameterization effects.
domain assumption Client models are true risk minimizers h*_{p_k} of L_{p_k} (Theorem 3.4).
Algorithm 2 obtains client models with a few epochs of SGD, not exact minimizers; the proof does not include approximation error.
domain assumption Discriminators are optimal for the fixed generator (Theorem 3.6).
D_k(x)=p_k/(p_k+p_g) holds only at the GAN optimum; clients train discriminators for 10 to 30 epochs, so the odds are approximate.
domain assumption Support condition supp(p) ⊆ supp(p_g) for the weight identity (Theorem 3.6).
The density-ratio identity is stated for x in supp(p_g); Section 4.2 acknowledges that off-the-shelf generators may miss classes, so part of p is uncovered.
domain assumption Binary classification and ℓ1 loss for Theorem 3.2, Corollary 3.3, and Theorem C.1 (Appendix A).
The formal statements in Appendix A assume binary tasks with ℓ1 loss, while experiments are multi-class CIFAR/ImageNet.
standard math VC/growth-function generalization bounds (Shalev-Shwartz & Ben-David Theorem 6.11).
Used in the proof of Theorem C.1 to derive finite-sample uniform deviation bounds for empirical risk minimizers.
standard math Optimal discriminator characterization for GANs (Goodfellow et al. Proposition 1).
Basis for the density ratio p_k/p_g in Theorem 3.6; cited external result.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead." pith.science (2026). https://pith.science/paper/M42XPOAV

@misc{pith2026250206349,
  author       = {Pith},
  title        = {Pith review of: Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M42XPOAV}},
  note         = {Machine review of arXiv:2502.06349}
}

read the original abstract

Federated ensemble distillation addresses client heterogeneity by generating pseudo-labels for an unlabeled server dataset based on client predictions and training the server model using the pseudo-labeled dataset. The unlabeled server dataset can either be pre-existing or generated through a data-free approach. The effectiveness of this approach critically depends on the method of assigning weights to client predictions when creating pseudo-labels, especially in highly heterogeneous settings. Inspired by theoretical results from GANs, we propose a provably near-optimal weighting method that leverages client discriminators trained with a server-distributed generator and local datasets. Our experiments on various image classification tasks demonstrate that the proposed method significantly outperforms baselines. Furthermore, we show that the additional communication cost, client-side privacy leakage, and client-side computational overhead introduced by our method are negligible, both in scenarios with and without a pre-existing server dataset.

Figures

Figures reproduced from arXiv: 2502.06349 by the authors.

**Figure 1.** A toy example of decision boundaries of aggregated models. Each point represents data, and its color represents the label. The background color represents the decision boundary of each model in the RGB channels. The oracle decision boundary, shown by the black lines, corresponds to the x-axis and y-axis. For aggregated models, we consider the parameter-averaged model (McMahan et al., 2017) and ensemble-distilled mod… view at source ↗

**Figure 2.** Ensemble test accuracy (%) of FedGO and other baseline weighting methods over communication rounds on CIFAR-10 with α = 0.1 and α = 0.05. to performance enhancement seems to be the larger model structure of the pretrained generator and its training with a richer dataset. This enhances the generalization performance of client discriminators, enabling optimal weighting even for test data. However, since the assumptio… view at source ↗

**Figure 3.** Illustration of our FedGO algorithm. data, with its color indicating the class label: data from Gaussians with means at (4, 4) and (-4, -4) are labeled as Red, data from the Gaussian with mean at (-4, 4) as Blue, and data from the Gaussian with mean at (4, -4) as Green. Each Gaussian provides 300 data samples. Each client holds 90% of data from the Gaussian whose mean is in a certain quadrant (the 3rd, 4th, 2nd, 1st… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 5.** Figure 5: Client data split for CIFAR-10 with α = 0.1, 0.05. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Client 0 10 20 30 40 50 60 70 80 90 Class (a) α = 0.1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Client 0 10 20 30 40 50 60 70 80 90 Class (b) α = 0.05 [PITH_FU…

**Figure 6.** Figure 6: Client data split for CIFAR-100 with α = 0.1, 0.05. two measures were evaluated only for algorithms incorporating ensemble distillation. FedGO achieves the test accuracy of 69.52%, which is slightly lower than 72.35% with 20 clients ( [PITH_FULL_IMAGE:figures/full_fig…

**Figure 7.** Figure 7: Server test accuracy (%), test accuracy of the ensemble model (%), and test loss of the ensemble model of our FedGO and baselines for 100 clients on CIFAR-10 dataset with α = 0.05. To support our hypothesis, we show the minimum of mean distillation loss for five differ…

**Figure 8.** Figure 8: Ensemble test accuracy (%) of FedGO and baselines over communication rounds on three image datasets with α = 0.1, 0.05. hyperparameters are set to the same as the main paper [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

**Figure 9.** Figure 9: and [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

**Figure 10.** Figure 10: shows the test accuracy of the server model and the test accuracy of the ensemble model during the training process for our FedGO algorithm. We conducted experiments by reducing the server dataset size to 50% and 20% of the size assumed in our main CIFAR-10 experiment…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 52 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Investigating under and overfitting in wasserstein generative adversarial networks

Adlam, B., Weill, C., and Kapoor, A. Investigating under and overfitting in wasserstein generative adversarial networks. arXiv preprint arXiv:1910.14137, 2019

arXiv 1910
[3]

R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., and Lee, K

Behera, M. R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., and Lee, K. F. Fedsyn: Synthetic data generation using federated learning. arXiv preprint arXiv:2203.05931, 2022

arXiv 2022
[4]

Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. https://www.alexkulesza.com/pubs/adapt_mlj10.pdf

work page
[5]

Analysis of representations for domain adaptation

Ben-David, S., Blitzer, J., Crammer, K., Pereira, and Fernando. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006

work page 2006
[6]

Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer

Chang, H., Shejwalkar, V., Shokri, R., and Houmansadr, A. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279, 2019

arXiv 1912
[7]

Infogan: Interpretable representation learning by information maximizing generative adversarial nets

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016

2016
[8]

J., Manoel, A., Joshi, G., Sim, R., and Dimitriadis, D

Cho, Y. J., Manoel, A., Joshi, G., Sim, R., and Dimitriadis, D. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI) Main Track, 2022

work page 2022

Show all 76 references

[9]

Stargan: Unified generative adversarial networks for multi-domain image-to-image translation

Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 8789--8797, 2018

2018
[10]

A downsampled variant of ImageNet as an alternative to the CIFAR datasets

Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. CoRR, 2017

2017
[11]

Learning from multiple sources

Crammer, K., Kearns, M., and Wortman, J. Learning from multiple sources. Journal of Machine Learning Research, 9 0 (8), 2008

2008
[12]

Tackling data heterogeneity in federated learning with class prototypes

Dai, Y., Chen, Z., Li, J., Heinecke, S., Sun, L., and Xu, R. Tackling data heterogeneity in federated learning with class prototypes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 7314--7322, 2023

2023
[13]

A hierarchical knowledge transfer framework for heterogeneous federated learning

Deng, Y., Ren, J., Tang, C., Lyu, F., Liu, Y., and Zhang, Y. A hierarchical knowledge transfer framework for heterogeneous federated learning. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications, pp.\ 1--10. IEEE, 2023

2023
[14]

Q., Li, A., and Kung, H

Dong, X., Zhang, S. Q., Li, A., and Kung, H. Spherefed: Hyperspherical federated learning. In European Conference on Computer Vision, pp.\ 165--184. Springer, 2022

2022
[15]

Calibrating noise to sensitivity in private data analysis

Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.\ 265--284. Springer, 2006

2006
[16]

The algorithmic foundations of differential privacy

Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9 0 (3--4): 0 211--407, 2014

2014
[17]

and Liu, P

Fan, C. and Liu, P. Federated generative adversarial learning. In Pattern Recognition and Computer Vision: Third Chinese Conference, PRCV 2020, Nanjing, China, October 16--18, 2020, Proceedings, Part III 3, pp.\ 3--15. Springer, 2020

2020
[18]

Ensemble attention distillation for privacy-preserving federated learning

Gong, X., Sharma, A., Karanam, S., Wu, Z., Chen, T., Doermann, D., and Innanje, A. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15076--15086, 2021

2021
[19]

Generative adversarial nets

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

2014
[20]

Guerraoui, R., Guirguis, A., Kermarrec, A.-M., and Merrer, E. L. Fegan: Scaling distributed gans. In Proceedings of the 21st International Middleware Conference, pp.\ 193--206, 2020

2020
[21]

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017

2017
[22]

Md-gan: Multi-discriminator generative adversarial networks for distributed datasets

Hardy, C., Le Merrer, E., and Sericola, B. Md-gan: Multi-discriminator generative adversarial networks for distributed datasets. In 2019 IEEE international parallel and distributed processing symposium (IPDPS), pp.\ 866--877. IEEE, 2019

2019
[23]

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

2016
[24]

Distilling the knowledge in a neural network

Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[25]

Neural networks for machine learning

Hinton, G., sh Srivastava, N., and Swersky, K. Neural networks for machine learning. https://www.cs.toronto.edu/ tijmen/csc321/slides/lecture_slides_lec6.pdf
[26]

Understanding convergence and generalization in federated learning through feature learning theory

Huang, W., Shi, Y., Cai, Z., and Suzuki, T. Understanding convergence and generalization in federated learning through feature learning theory. In The Twelfth International Conference on Learning Representations, 2023

2023
[27]

https://www.kaggle.com/datasets/ambityga/imagenet100

ImageNet 100 dataset. https://www.kaggle.com/datasets/ambityga/imagenet100
[28]

and Szegedy, C

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp.\ 448--456. pmlr, 2015

2015
[29]

B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and trends in machine learning , 14 0 (1--2): 0 1--210, 2021

2021
[30]

P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: S tochastic controlled averaging for federated learning. In International conference on machine learning, pp.\ 5132--5143. PMLR, 2020

2020
[31]

A style-based generator architecture for generative adversarial networks

Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 4401--4410, 2019

2019
[32]

Detecting change in data streams

Kifer, D., Ben-David, S., and Gehrke, J. Detecting change in data streams. In VLDB, volume 4, pp.\ 180--191. Toronto, Canada, 2004

2004
[33]

Fed FN: F eature normalization for alleviating data heterogeneity problem in federated learning

Kim, S., Lee, G., Oh, J., and Yun, S.-Y. Fed FN: F eature normalization for alleviating data heterogeneity problem in federated learning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023

2023
[34]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

2015 arXiv
[35]

Learning multiple layers of features from tiny images

Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https://api.semanticscholar.org/CorpusID:18268744

2009
[36]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

2012
[37]

Fedtp: Federated learning by transformer personalization

Li, H., Cai, Z., Wang, J., Tang, J., Ding, W., Lin, C.-T., and Shi, Y. Fedtp: Federated learning by transformer personalization. IEEE transactions on neural networks and learning systems, 2023

2023
[38]

K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V

Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 0 429--450, 2020 a

2020
[39]

Ifl-gan: Improved federated learning generative adversarial network with maximum mean discrepancy model aggregation

Li, W., Chen, J., Wang, Z., Shen, Z., Ma, C., and Cui, X. Ifl-gan: Improved federated learning generative adversarial network with maximum mean discrepancy model aggregation. IEEE Transactions on Neural Networks and Learning Systems, 2022

2022
[40]

On the convergence of fed A vg on Non-IID data

Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fed A vg on Non-IID data. In International Conference on Learning Representations, 2020 b

2020
[41]

and Zhan, D.-C

Li, X.-C. and Zhan, D.-C. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.\ 995--1005, 2021

2021
[42]

Variance reduced local sgd with lower communication complexity

Liang, X., Shen, S., Liu, J., Pan, Z., Chen, E., and Cheng, Y. Variance reduced local sgd with lower communication complexity. arXiv preprint arXiv:1912.12844, 2019

1912 arXiv
[43]

U., and Jaggi, M

Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 0 2351--2363, 2020

2020
[44]

and Hutter, F

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2022

2022
[45]

Personalized federated learning through local memorization

Marfoq, O., Neglia, G., Vidal, R., and Kameni, L. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp.\ 15070--15092. PMLR, 2022

2022
[46]

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.\ 1273--1282. PMLR, 2017

2017
[47]

Local learning matters: Rethinking data heterogeneity in federated learning

Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., and Chen, C. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8397--8406, 2022

2022
[48]

Overcoming client data deficiency in federated learning by exploiting unlabeled data on the server

Park, J.-M., Jang, W.-J., Oh, T.-H., and Lee, S.-H. Overcoming client data deficiency in federated learning by exploiting unlabeled data on the server. IEEE Access, 2024

2024
[49]

Unsupervised representation learning with deep convolutional generative adversarial networks

Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, ...

2016 arXiv
[50]

Fedgan: Federated generative adversarial networks for distributed data

Rasouli, M., Sun, T., and Rajagopal, R. Fedgan: Federated generative adversarial networks for distributed data. arXiv preprint arXiv:2006.07228, 2020

2006 arXiv
[51]

Communication-efficient federated distillation

Sattler, F., Marban, A., Rischke, R., and Samek, W. Communication-efficient federated distillation. arXiv preprint arXiv:2012.00632, 2020

2012 arXiv
[52]

Stylegan-xl: Scaling stylegan to large diverse datasets

Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp.\ 1--10, 2022

2022
[53]

and Ben-David, S

Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms. https://www.cs.huji.ac.il/ shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf https://www.cs.huji.ac.il/ shais/UnderstandingMachineLearning/unde...
[54]

and Zisserman, A

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. U...

2015 arXiv
[55]

M., Kim, M.-H., Chung, T.-M., Huang, C., and Liu, X

Son, H. M., Kim, M.-H., Chung, T.-M., Huang, C., and Liu, X. Feduv: uniformity and variance for heterogeneous federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 5863--5872, 2024

2024
[56]

Virtual homogeneity learning: D efending against data heterogeneity in federated learning

Tang, Z., Zhang, Y., Shi, S., He, X., Han, B., and Chu, X. Virtual homogeneity learning: D efending against data heterogeneity in federated learning. In International Conference on Machine Learning, pp.\ 21111--21132. PMLR, 2022

2022
[57]

Fed Impro : M easuring and improving client update in federated learning

Tang, Z., Zhang, Y., Shi, S., Tian, X., Liu, T., Han, B., and Chu, X. Fed Impro : M easuring and improving client update in federated learning. In The Twelfth International Conference on Learning Representations, 2024

2024
[58]

Federated learning with matched averaging

Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. In International Conference on Learning Representations, 2020 a

2020
[59]

Dafkd: Domain-aware federated knowledge distillation

Wang, H., Li, Y., Xu, W., Li, R., Zhan, Y., and Zeng, Z. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp.\ 20412--20421, 2023 a

2023
[60]

Fed CDA: F ederated learning with cross-rounds divergence-aware aggregation

Wang, H., Xu, H., Li, Y., Xu, Y., Li, R., and Zhang, T. Fed CDA: F ederated learning with cross-rounds divergence-aware aggregation. In The Twelfth International Conference on Learning Representations, 2023 b

2023
[61]

Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 0 7611--7623, 2020 b

2020
[62]

Fed Med-GAN : Fed erated domain translation on unsupervised cross-modality brain image synthesis

Wang, J., Xie, G., Huang, Y., Lyu, J., Zheng, F., Zheng, Y., and Jin, Y. Fed Med-GAN : Fed erated domain translation on unsupervised cross-modality brain image synthesis. Neurocomputing, 546: 0 126282, 2023 c

2023
[63]

Do generated data always help contrastive learning? In The Twelfth International Conference on Learning Representations, 2024

Wang, Y., Zhang, J., and Wang, Y. Do generated data always help contrastive learning? In The Twelfth International Conference on Learning Representations, 2024

2024
[64]

An efficient federated distillation learning system for multitask time series classification

Xing, H., Xiao, Z., Qu, R., Zhu, Z., and Zhao, B. An efficient federated distillation learning system for multitask time series classification. IEEE Transactions on Instrumentation and Measurement, 71: 0 1--12, 2022

2022
[65]

Federated generative model on multi-source heterogeneous data in iot

Xiong, Z., Li, W., and Cai, Z. Federated generative model on multi-source heterogeneous data in iot. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 10537--10545, 2023

2023
[66]

Improving gans with a dynamic discriminator

Yang, C., Shen, Y., Xu, Y., Zhao, D., Dai, B., and Zhou, B. Improving gans with a dynamic discriminator. Advances in Neural Information Processing Systems, 35: 0 15093--15104, 2022

2022
[67]

Fed Fed: F eature distillation against data heterogeneity in federated learning

Yang, Z., Zhang, Y., Zheng, Y., Tian, X., Peng, H., Liu, T., and Han, B. Fed Fed: F eature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024

2024
[68]

Local-global knowledge distillation in heterogeneous federated learning with non-iid data

Yao, D., Pan, W., Dai, Y., Wan, Y., Ding, X., Jin, H., Xu, Z., and Sun, L. Local-global knowledge distillation in heterogeneous federated learning with non-iid data. arXiv preprint arXiv:2107.00051, 2021

2021 arXiv
[69]

Fed Disco : Federated learning with discrepancy-aware collaboration

Ye, R., Xu, M., Wang, J., Xu, C., Chen, S., and Wang, Y. Fed Disco : Federated learning with discrepancy-aware collaboration. In International Conference on Machine Learning, pp.\ 39879--39902. PMLR, 2023

2023
[70]

Redifine: Reusable diffusion finetuning for mitigating degradation in the chain of diffusion

Yoon, Y., Hu, D., Weissburg, I., Qin, Y., and Jeong, H. Redifine: Reusable diffusion finetuning for mitigating degradation in the chain of diffusion. arXiv preprint arXiv:2407.17493, 2024

2024 arXiv
[71]

Zhang, J., Guo, S., Guo, J., Zeng, D., Zhou, J., and Zomaya, A. Y. Towards data-independent knowledge transfer in model-heterogeneous federated learning. IEEE Transactions on Computers, 72 0 (10): 0 2888--2901, 2023 a

2023
[72]

Y., and Zomaya, A

Zhang, J., Zhao, L., Yu, K., Min, G., Al-Dubai, A. Y., and Zomaya, A. Y. A novel federated learning scheme for generative adversarial networks. IEEE Transactions on Mobile Computing, 2023 b

2023
[73]

Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning

Zhang, J., Liu, Y., Hua, Y., and Cao, J. Fedtgp: Trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp.\ 16768--16776, 2024

2024
[74]

Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models

Zhang, L., Wu, D., and Yuan, X. Fedzkt: Zero-shot knowledge transfer towards resource-constrained federated learning with heterogeneous on-device models. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS), pp.\ 928--938. IEEE, 2022

2022
[75]

Zhang, X., Zhu, X., Wang, J., Bao, W., and Yang, L. T. Dance: Distributed generative adversarial networks with communication compression. ACM Transactions on Internet Technology (TOIT), 22 0 (2): 0 1--32, 2021

2021
[76]

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.\ 2223--2232, 2017

2017

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Investigating under and overfitting in wasserstein generative adversarial networks

Adlam, B., Weill, C., and Kapoor, A. Investigating under and overfitting in wasserstein generative adversarial networks. arXiv preprint arXiv:1910.14137, 2019

arXiv 1910

[3] [3]

R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., and Lee, K

Behera, M. R., Upadhyay, S., Shetty, S., Priyadarshini, S., Patel, P., and Lee, K. F. Fedsyn: Synthetic data generation using federated learning. arXiv preprint arXiv:2203.05931, 2022

arXiv 2022

[4] [4]

Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. https://www.alexkulesza.com/pubs/adapt_mlj10.pdf

work page

[5] [5]

Analysis of representations for domain adaptation

Ben-David, S., Blitzer, J., Crammer, K., Pereira, and Fernando. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006

work page 2006

[6] [6]

Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer

Chang, H., Shejwalkar, V., Shokri, R., and Houmansadr, A. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279, 2019

arXiv 1912

[7] [7]

Infogan: Interpretable representation learning by information maximizing generative adversarial nets

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016

2016

[8] [8]

J., Manoel, A., Joshi, G., Sim, R., and Dimitriadis, D

Cho, Y. J., Manoel, A., Joshi, G., Sim, R., and Dimitriadis, D. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI) Main Track, 2022

work page 2022