Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fusing global and local features on clients cuts federated-learning communication rounds by over 60 percent, and an MMD constraint cuts them by over 20 percent.

desk verdict Incremental combination of two prior conference papers; the headline 60% round-reduction claim rests on post hoc operator selection and missing error bars. read the letter →

arxiv 1908.05891 v2 pith:6LNX5H4B submitted 2019-08-16 cs.LG cs.DCstat.ML

Xin Yao , Tianchi Huang , Chenglei Wu , Rui-Xiao Zhang , Lifeng Sun This is my paper · ORCID
classification cs.LGcs.DCstat.ML
keywords FederatedlearningCommunicationefficiencyNon-IIDdataMaximumMeanDiscrepancyFeaturefusionOn-devicetrainingFedAvgPermutedMNIST
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes two on-device additions to federated averaging that reduce the number of communication rounds needed to reach target accuracy. FedMMD keeps the global model fixed during local training and adds a Maximum Mean Discrepancy term that pulls local model outputs toward global outputs. FedFusion freezes the global feature extractor, fuses its feature maps with local features through a learned fusion operator, and trains the fusion module and classifier locally. On user-specific non-IID Permuted MNIST, FedFusion with the convolution operator reaches 94 percent accuracy in 34 rounds compared with FedAvg's 100, a 66 percent reduction, and reaches 95 percent in 92 rounds versus 256. If right, this means communication cost, the main bottleneck in federated learning, can be cut without sacrificing accuracy.

What carries the argument

The key machinery is the frozen global model retained on the client during local training. In FedMMD this frozen model supplies output distributions for an MMD penalty; in FedFusion it supplies a feature stream combined with local features by a fusion module $F$ with three variants: $F_{\mathrm{conv}} = W_{\mathrm{conv}}(E_g(x) \| E_l(x))$, $F_{\mathrm{multi}} = \lambda \odot E_g(x) + (1-\lambda)\odot E_l(x)$, and $F_{\mathrm{single}} = \lambda E_g(x) + (1-\lambda)E_l(x)$. The fusion module is trained by backpropagation on the client and returned to the server, so each client learns the blend of global and local knowledge that suits its own data distribution. The global model itself stays frozen during local updates, which decouples the cross-client knowledge source from the local fitting process.

What would settle it

A concrete test: train FedFusion with the conv fusion operator and FedAvg on Permuted MNIST, but replace the global feature extractor with a freshly initialized network every round instead of the aggregated model; if FedFusion no longer reaches 94 percent accuracy in fewer rounds than FedAvg, the benefit comes from the frozen global reference being informative rather than from the fusion mechanism alone.

Watch

Extended reading notes

Core claim

The central claim is that communication cost in federated learning can be reduced by changing what happens on the client rather than by compressing messages. In FedMMD, each client trains a local model initialized from the global model while a frozen copy of the global model serves as a reference; the loss is classification loss plus $\lambda\operatorname{MMD}^2(\theta_G(X^t), \theta_L(X^t))$, which encourages local features to stay close to global features. In FedFusion, the global feature extractor is kept frozen, and a learned fusion module combines its feature maps with the local extractor's maps before classification; three operators are tested: a $1\times1$ convolution over concatenated maps, a per-channel learned weighted sum, and a single learned scalar weighted sum. The experimental claim is that the conv operator in FedFusion reduces required communication rounds by more than 60 percent on user-specific non-IID data, with final accuracy 95.79 versus FedAvg's 95.20, while FedMMD reduces rounds by about 20 percent on non-IID CIFAR-10 and by 23.4 percent on 100-client non-IID MNIST.

Load-bearing premise

The load-bearing premise is that the global model, frozen during local training, remains a useful source of features at every communication round; if the global model is poorly trained early on, fusing its features with local features could slow convergence rather than speed it up.

Editorial extensions

If this is right

  • On user-specific non-IID data, FedFusion with the conv operator reaches 94 percent accuracy in 34 rounds instead of FedAvg's 100, so a federated system can cut its communication budget by about two thirds at the same accuracy milestone.
  • FedMMD reduces required communication rounds by 20.2 percent on non-IID CIFAR-10 and by 23.4 percent on 100-client non-IID MNIST without lowering final accuracy, indicating the MMD constraint accelerates convergence as a regularizer.
  • New clients joining an existing system converge in fewer local epochs under FedFusion with the conv operator (about 65 local epochs versus 92 for FedAvg), so the learned fusion modules act as a better initialization for future participants.
  • The best fusion operator depends on the data partition: the multi operator suits artificial non-IID class gaps, the conv operator suits user-specific non-IID distributions, and the single operator gives little benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the two mechanisms address different non-IID failure modes, a natural extension is to combine them: apply the MMD penalty to the fused representation rather than the local output alone, and test whether the round reductions compound.
  • The frozen-global design can be read as client-side knowledge distillation from a slowly changing teacher; an untested corollary is that annealing the fusion weight as the global model matures, or unfreezing the global extractor late in training, could improve final accuracy.
  • If the fusion weights learned by each client are diagnostic of that client's data shift, they could be used as a lightweight personalization signal or as input to client clustering, though the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two client-side mechanisms to reduce communication rounds in federated learning. The first, FedMMD, replaces the single local model in FedAvg with a two-stream setup in which a frozen global model and a trainable local model are trained jointly under a classification loss plus an MMD penalty on their outputs. The second, FedFusion, keeps the global feature extractor frozen on each client and trains a feature fusion module that combines local and global feature maps via one of three operators (conv, multi, single), with the updated local model sent back to the server. Experiments on MNIST and CIFAR-10 under artificial non-IID, user-specific non-IID, and IID partitions report that FedMMD reduces communication rounds by more than 20% and FedFusion by more than 60% compared with FedAvg, and that FedFusion provides better initialization for newly incoming clients.

Significance. If the claimed reductions hold, the two mechanisms would be simple and broadly applicable additions to FedAvg, and the feature-fusion idea is a plausible direction for client-side personalization in federated learning. The paper clearly specifies three fusion operators and gives useful convergence curves for several partitions. However, the headline quantitative claims rest on a small set of single-run experiments, one post-hoc-selected operator, and at least one arithmetic inconsistency in the reported FedMMD reduction. The central idea is defensible but the current evidence does not establish the 'more than 60%' claim as a general property of FedFusion. The paper does not provide code, proofs, or pre-registered protocols; its value depends on whether the reported results are robust to seed variance and operator selection.

major comments (4)
  1. [§4.2.2 (Figure 4d)] The text reports that FedMMD reaches 98% accuracy in 72 rounds versus FedAvg's 128 rounds, 'which means a reduction of 23.4%.' The arithmetic contradicts the stated numbers: (128−72)/128 = 43.75%, not 23.4%. Since the abstract and conclusion explicitly claim a reduction of more than 20% for FedMMD, this inconsistency must be corrected and all other reported reductions rechecked against the underlying round counts.
  2. [§4.3.2 and Table 2] The headline claim of a more than 60% reduction in communication rounds is based entirely on FedFusion+Conv on Permuted MNIST (34 vs 100 rounds at 94% and 92 vs 256 at 95%). This operator was selected post hoc: §4.3.1 reports that the multi operator is best under artificial non-IID, while §4.3.2 states that conv 'leads in a large margin, which is different from that in artificial non-IID partition.' No standard deviations, multiple seeds, or holdout operator-selection rule are reported. As written, the 66.0%/64.1% figures are the maximum over three operator configurations on one benchmark, so the abstract's 'more than 60%' claim is not established. Repetition with multiple seeds and a fixed selection protocol (or reporting all operators without selecting) is necessary.
  3. [Algorithm 1, line 7] The server update in FedMMD is written as Θ_G^{r+1} ← Σ_{t∈S_r} Θ_t^{r+1}, an unweighted sum without normalization, whereas FedAvg and Algorithm 2 both use an averaging operation. Unless every client has identical sample sizes and the normalizing constant is absorbed into the learning rate, this update will make the global model scale with the number of participating clients, which is not a valid model averaging step. This needs to be corrected and the experiments rerun or justified.
  4. [§5 and Abstract] The conclusion and abstract generalize the 'more than 60%' reduction to 'popular FL scenarios,' but the only quantitative support is the single Permuted MNIST experiment in Table 2. The artificial non-IID and IID experiments in Figure 5 show convergence curves and final accuracies but do not report round reductions to milestones; the artificial non-IID results indicate that multi is the better operator and likely give a smaller reduction. The claims should be scoped to the specific benchmark and operator, or additional experiments on more than one dataset with the selected operator are needed.
minor comments (5)
  1. [§3.1, Eq. (5) and Figure 1] Equation (5) defines the MMD loss on the outputs θ_G(X_t) and θ_L(X_t) of the full models, while Figure 1 and its caption refer to 'MMD of High-level Features.' If the implementation uses feature maps rather than logits or softmax outputs, the equation and figure should be aligned.
  2. [§3.3, last paragraph] The statement 'For multi and single operators, we use an exponential moving average strategy to smooth the update' is underspecified; the momentum parameter and the variables to which it is applied should be given.
  3. [§4.1.1] There is a typo: 'we use the the same model as FedAvg' should read 'we use the same model as FedAvg.'
  4. [§4.2.1] The phrase 'For convenience but without loss of generality' is not accurate in this context, since the chosen hyperparameters and client count directly affect the reported reductions.
  5. [Figure 4 caption] The caption says 'upper row' and 'left column' but the panels are labeled (a)–(d); using the panel labels consistently would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's communication-round reductions are empirical measurements, not consequences of fitted definitions or self-citation chains.

full rationale

The paper does not derive its headline claims from definitions that already contain those claims. FedMMD and FedFusion are described in full within the manuscript, and the reported round reductions (20% for FedMMD, 60% for FedFusion with the conv operator) are measured from training curves against FedAvg on specific benchmarks. No fitted parameter is renamed as a prediction: the MMD weight and fusion operators are evaluated, not inferred from the target accuracy milestones. The paper transparently states that it combines the authors' prior VCIP 2018 and ICIP 2019 work, but the methods are not justified by appealing to an unverified self-citation; MMD is cited to Gretton et al. and FedAvg to McMahan et al. The fact that the conv operator is reported as best in the user-specific non-IID setting while multi is best in the artificial non-IID setting reflects post-hoc selection among three operators, which is a robustness limitation (selection bias, no error bars) rather than a circularity: the paper never claims to have predicted which operator would win before running the experiments. No equation in the paper reduces to another by construction, and no self-citation is load-bearing for the empirical results. A non-finding is therefore appropriate: score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on two hand-tuned hyperparameters (MMD lambda and L2 penalties) and on the domain assumption that a frozen global model is a useful feature source during local training. No fundamentally new theoretical entities are introduced; the feature fusion modules are neural network components.

free parameters (2)
  • MMD penalty coefficient lambda = 0.1
    Chosen by hand for both CIFAR-10 and MNIST FedMMD experiments (Sections 4.2.1 and 4.2.2); no sensitivity analysis beyond trying two values for the L2 baseline.
  • L2 penalty coefficient (baseline) = 0.1, 0.01, 0.001
    Tuned per dataset for the L2-constrained baseline; the paper notes a larger value can cause non-convergence.
assumptions (3)
  • domain assumption MMD computed on model outputs in an RKHS approximates the divergence between local and global feature distributions
    Invoked in Section 3.1, Eq. (5): the MMD loss between theta_G(X_t) and theta_L(X_t) is assumed to align local and global knowledge.
  • domain assumption The frozen global feature extractor provides useful supervision for local training throughout all rounds
    Assumed in Section 3.3 and Figure 3, where E_g is fixed and fused with E_l; if the global model is poor in early rounds, fusing its features could hurt.
  • standard math Standard SGD with cross-entropy loss suffices to train the fusion modules and local model
    Standard optimization assumption, Section 4.1.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs." pith.science (2026). https://pith.science/paper/6LNX5H4B

@misc{pith2026190805891,
  author       = {Pith},
  title        = {Pith review of: Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LNX5H4B}},
  note         = {Machine review of arXiv:1908.05891}
}
read the original abstract

Federated learning (FL) enables on-device training over distributed networks consisting of a massive amount of modern smart devices, such as smartphones and IoT (Internet of Things) devices. However, the leading optimization algorithm in such settings, i.e., federated averaging (FedAvg), suffers from heavy communication costs and the inevitable performance drop, especially when the local data is distributed in a non-IID way. To alleviate this problem, we propose two potential solutions by introducing additional mechanisms to the on-device training. The first (FedMMD) is adopting a two-stream model with the MMD (Maximum Mean Discrepancy) constraint instead of a single model in vanilla FedAvg to be trained on devices. Experiments show that the proposed method outperforms baselines, especially in non-IID FL settings, with a reduction of more than 20% in required communication rounds. The second is FL with feature fusion (FedFusion). By aggregating the features from both the local and global models, we achieve higher accuracy at fewer communication costs. Furthermore, the feature fusion modules offer better initialization for newly incoming clients and thus speed up the process of convergence. Experiments in popular FL scenarios show that our FedFusion outperforms baselines in both accuracy and generalization ability while reducing the number of required communication rounds by more than 60%.

Figures

Figures reproduced from arXiv: 1908.05891 by the authors.

Figure 1
Figure 1. Two-stream model with MMD: the global model is fixed while the local model is trained [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Three types of feature fusion modules. The fusion operator is actually a mapping function, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training Iteration of FedFusion. During the training procedure on clients, the local and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Test accuracy over communication rounds for FedMMD on CIFAR-10 (upper row) and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Test accuracy vs. communication rounds under different settings. (a) and (b): the artificial [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Number of local epochs to reach convergence for newly incoming clients. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models and Social Media Information Integrity: Opportunities, Challenges, and Research Directions

    cs.CR 2026-08 conditional novelty 4.0 of 10

    A systematic review of 215 studies concludes that large language models both enable and counter misinformation, social bots, and privacy threats on social media, and maps open research gaps.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Caldas, J

    S. Caldas, J. Koneˇcny, H. B. McMahan, and A. Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210, 2018

    arXiv 2018

  2. [2]

    Caldas, P

    S. Caldas, P. Wu, T. Li, J. Koneˇcn`y, H. B. McMahan, V . Smith, and A. Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018

    arXiv 2018

  3. [3]

    F. Chen, Z. Dong, Z. Li, and X. He. Federated meta-learning for recommendation. arXiv preprint arXiv:1802.07876, 2018

    arXiv 2018

  4. [4]

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y . Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013

    arXiv 2013

  5. [5]

    Gretton, D

    A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in neural information processing systems, pages 1205–1213, 2012

    work page 2012

  6. [6]

    Jeong, S

    E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018

    arXiv 2018

  7. [7]

    Koneˇcn`y, B

    J. Koneˇcn`y, B. McMahan, and D. Ramage. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575, 2015

    arXiv 2015

  8. [8]

    Koneˇcn`y, H

    J. Koneˇcn`y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016

    arXiv 2016

Show all 22 references
  1. [9]

    Krizhevsky and G

    A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  2. [10]

    LeCun, L

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  3. [11]

    M. Long, Y . Cao, J. Wang, and M. I. Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:1502.02791, 2015

    2015 arXiv

  4. [12]

    M. Long, J. Wang, G. Ding, J. Sun, and S. Y . Philip. Transfer feature learning with joint distribution adaptation. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 2200–2207. IEEE, 2013

    2013

  5. [13]

    McMahan, E

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pages 1273–1282, 2017

    2017

  6. [14]

    Smith, C.-K

    V . Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems, pages 4424–4434, 2017

    2017

  7. [15]

    Sun and K

    B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, pages 443–450. Springer, 2016

    2016

  8. [16]

    A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3329–3337. JMLR. org, 2017

    2017

  9. [17]

    Tzeng, J

    E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximiz- ing for domain invariance. arXiv preprint arXiv:1412.3474, 2014

    2014 arXiv

  10. [18]

    X. Yao, C. Huang, and L. Sun. Two-stream federated learning: Reduce the communication costs. In Visual Communications and Image Processing (VCIP), 2018, pages 1–4. IEEE, 2018

    2018

  11. [19]

    X. Yao, T. Huang, C. Wu, R. Zhang, and L. Sun. Towards faster and better federated learning: A feature fusion approach. In IEEE International Conference on Image Processing, 2019. 11

    2019

  12. [20]

    Zenke, B

    F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995, 2017

    2017

  13. [21]

    Y . Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018

    2018 arXiv

  14. [22]

    J. Zhuo, S. Wang, W. Zhang, and Q. Huang. Deep unsupervised convolutional domain adaptation. In Proceedings of the 2017 ACM on Multimedia Conference, pages 261–269. ACM, 2017. 12

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.