Pith. sign in

REVIEW 4 major objections 6 minor 32 references

FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read FedADP claims that heterogeneous client models can be morphed into a unified global structure each round, aggregated with FedAvg, then morphed back to each client's own architecture, allowing every client to contribute its full model…

desk verdict Real idea, honest experiments, but the paper's load-bearing morphing algorithm as written does not preserve function, so the reported gains are currently unexplained. read the letter →

arxiv 2505.06497 v1 pith:OKGO5U6W submitted 2025-05-10 cs.LG

classification cs.LG
keywords federatedlearningmodelheterogeneityarchitectureaggregationNetChangeVGGpersonalizedresource-constraineddevices
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FedADP is a federated learning framework that lets clients with different model architectures train together without discarding any layer or neuron. It works by temporarily reshaping every local model into a common global architecture before aggregation, then returning each client its own structure after the round. The paper reports that this approach outperforms existing heterogeneous-FL methods on standard image benchmarks, with accuracy gains up to 23.30% over FlexiFed and up to 46.25% over Clustered-FL. The significance is that even weak clients with small models can contribute fully to a shared global model, rather than being limited to overlapping substructures.

What carries the argument

The central mechanism is NetChange, a set of four structure-morphing operations that extend Net2Net. To-Wider duplicates neurons and divides their incoming weights to keep layer outputs unchanged; To-Deeper inserts layers initialized as identity maps; To-Narrower removes excess neurons and redistributes their summed values to remaining neurons; To-Shallower deletes layers. These operations are applied in opposite directions before aggregation and before distribution, so all client models share one structure while retaining their original functional behavior.

What would settle it

A direct test would take a trained local model, apply NetChange to the global shape and then back to the local shape, and compare predictions on a fixed validation batch; if the output distribution changes substantially (for example, classification accuracy drops more than a small tolerance), the functional-preservation assumption fails and FedADP's reported gains would not hold under repeated round-trip morphing. A second check is to re-run the CIFAR-100 experiment with the same hyperparameters and verify that the 23.30% relative improvement over FlexiFed reproduces.

Watch

Extended reading notes

Core claim

The paper's central claim is that model heterogeneity in federated learning can be handled by modifying model structures rather than by searching for common substructures. FedADP constructs a global model as the union of all client architectures, then applies NetChange operations: To-Wider and To-Deeper expand each local model to the global shape before aggregation, while To-Narrower and To-Shallower trim the global model back to each client's shape before distribution. After local training, the expanded models are aggregated with standard FedAvg. Experiments with VGG variants (VGG-13 through VGG-19-Wider) on MNIST, F-MNIST, CIFAR-10, and CIFAR-100 show FedADP achieving higher accuracy than FlexiFed, Clustered-FL, and Standalone, with the largest gains on CIFAR-100.

Load-bearing premise

The load-bearing premise is that the NetChange operations (To-Wider, To-Deeper, To-Narrower, To-Shallower) preserve each model's functional behavior when morphing it to and from the global architecture, so the aggregated model is a faithful combination of client knowledge rather than a distorted blend.

Editorial extensions

If this is right

  • Every client, regardless of model size or depth, can participate in global aggregation without losing the knowledge encoded in its unique layers.
  • The global model is not tied to any single client architecture; it can be the union of all participating architectures, enabling full utilization of every client's capacity.
  • FedADP achieves higher accuracy than common-layer-only aggregation methods like FlexiFed while maintaining comparable convergence speed.
  • The approach applies to standard convolutional families such as VGG and could be extended to other architectures where layer-wise morphing is well-defined.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to measure the round-trip distortion of NetChange: apply To-Wider and To-Narrower in sequence to a trained model and compare its outputs on a fixed batch; if the outputs shift materially, the functional-preservation assumption is violated and cumulative drift would degrade long-horizon training.
  • The paper reports results on 20 clients with full participation; a likely stress test is partial participation and non-IID data, where the morphing operations may interact with client drift and require adaptive weighting.
  • Because the global model grows to the union of all architectures, communication and server-side memory costs scale with the largest client model; a practical variant might cap the global structure to control resource overhead.
  • The reported accuracy gains are on VGG-style networks; establishing similar gains on ResNet-style or transformer architectures would test whether the method generalizes beyond the evaluated family.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FedADP, a federated learning framework that aims to allow clients with heterogeneous model architectures to participate in a single global aggregation process. The key idea is to define a global architecture as the union of client architectures and then transform each client model to this global form before aggregation (To-Wider, To-Deeper) and transform the global model back to each client architecture before distribution (To-Narrower, To-Shallower). The transformations are collectively called NetChange and are asserted to preserve the original functionality of the models. The authors evaluate FedADP on MNIST, F-MNIST, CIFAR-10, and CIFAR-100 using VGG-family variants, reporting accuracy improvements over FlexiFed, Clustered-FL, and Standalone, with claimed improvements of up to 23.30% over FlexiFed.

Significance. If the NetChange transformations truly preserved model functionality, FedADP would be a valuable contribution to model-heterogeneous federated learning: it avoids discarding client-specific layers and allows all clients to contribute to a unified global model. The experimental results in Table I are internally consistent and show clear gains over the compared baselines across four datasets. However, the central mechanism of FedADP, the function-preservation property of NetChange, is asserted rather than proved, and the algorithmic descriptions are incomplete in ways that make the asserted property doubtful. Because the aggregation procedure in FedADP relies entirely on this property, the validity of the reported accuracy improvements is not established. The paper does not provide code, proofs, or sufficiently detailed experimental specifications to reproduce the results, which further weakens the contribution in its current form.

major comments (4)
  1. [Section III-B, Algorithm 3] Algorithm 3 (To-Narrower) does not preserve the function of a trained network as claimed. Removing neurons r after Ntar changes the input dimension of the next layer, so the next layer's weight matrix and bias are no longer compatible. The algorithm only adds the sum of the removed neurons' values divided by Ntar to the remaining neurons; it specifies no update to the downstream layer's weights. For a generic trained network, the layer output z_r = W_r h_{r-1} + b_r changes for every input once rows of W_r and b_r are silently removed. To preserve the composed function, one must also delete the corresponding columns of the next layer's weight matrix or otherwise compensate for the removed contributions. The paper does not supply such a construction, so the load-bearing claim that the narrowed model 'retains its original functionality and performance' is unsupported.
  2. [Section III-B, To-Shallower] To-Shallower is described only as the 'removal of unnecessary layers to match the client's model structure' with no algorithmic detail. Removing a layer changes the function class of the network; preserving behavior requires composing the removed layer with its adjacent layers (for example, by multiplying the relevant weight matrices and folding biases). The manuscript does not describe or justify such a composition, nor does it explain how the operation is applied to the convolutional VGG models used in the experiments. Without a precise, function-preserving shallowing operation, the distribution step of FedADP (Algorithm 1, line 6) cannot be verified and the experimental results cannot be attributed to a faithful aggregation of client knowledge.
  3. [Section III-B, Algorithm 2] Algorithm 2 (To-Wider) is also underspecified with respect to the next layer's weights. The text states that new neurons are created by duplicating existing neurons and their incoming connections, but the pseudocode only copies scalar neuron values and scales values by |Mi|. It does not state how the outgoing connections of the new neuron are initialized or how the downstream weight matrix is adjusted to keep the layer output unchanged. For feedforward and convolutional networks, function preservation under widening requires duplicating the corresponding rows/columns of the downstream weight matrix and, in convolutional layers, handling filters and channels; none of this is specified. The paper must provide exact update rules for all affected weight tensors and a formal statement of the conditions under which the output function is preserved.
  4. [Section IV-A] The evaluation only covers VGG-family architectures, where the notion of the 'union' of layer structures is straightforward and where the global model is a specific wider/deeper VGG variant. This does not demonstrate the general claim that FedADP can handle arbitrary heterogeneous model architectures. In addition, the experimental setup omits key reproducibility details: the data partitioning strategy (IID vs. non-IID) is not described, the exact set of eight architectures and their hyperparameters are not fully enumerated, and the implementation of the FlexiFed and Clustered-FL baselines in this heterogeneous setting is not specified. These omissions make the strong quantitative claims in Table I difficult to verify.
minor comments (6)
  1. [Algorithm 1] The argument order of NetChange is inconsistent: line 6 uses NetChange(ωt, ωk) while line 10 uses NetChange(ωk, ωt), but the text says NetChange(a, b) modifies a to conform to b. Clarify which argument is the source and which is the target in both calls.
  2. [Section IV-A] The sentence 'the number of types of model architectures is set to 8, in this case, 6 clients will be trained using VGG-19, and the other 7 models will be adopted by two clients each' is ambiguous. With K=20, the arithmetic is 6 + 2·7 = 20, but it is unclear which of the eight architecture types are assigned to which clients. Please list the architectures explicitly.
  3. [Section IV-A] The phrase 'Clients will use 20% of their datasets in each round of training' is unclear: does this mean a randomly sampled 20% subset per round, the same 20% throughout, or a fraction of the local data used for each local epoch?
  4. [Throughout] There are typographical inconsistencies, e.g., 'exsiting' in the Introduction, and the inconsistent use of 'Cluster-FL' versus 'Clustered-FL' in the text and Figure 4.
  5. [Table I and Figure 4] Table I reports means and standard deviations but no statistical significance tests. Given the small differences on MNIST and F-MNIST, a simple significance test would help support the claimed improvements.
  6. [References] Reference [32] for CIFAR-100 is listed as 'not found'; provide the full citation. Several references use informal arXiv identifiers without venue or version information.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FedADP's reported gains are external benchmark measurements and the NetChange claims, while under-specified, are not reductions of the output to the input.

full rationale

The claimed contribution is an algorithm (FedADP) that morphs heterogeneous client models to a common global structure via NetChange and then applies FedAvg. The global model is chosen as the union of client architectures (Section III-B: 'the system first constructs a global model by taking the union of the structures of all the client models'). This is a design choice, not a fitted parameter. The reported numbers in Table I are experimental accuracies on MNIST, F-MNIST, CIFAR-10, and CIFAR-100, compared with external baselines (FlexiFed, Clustered-FL, Standalone); they are not derived from the paper's own formulas, and no quantity in the paper is fitted to these test accuracies. The load-bearing step of the method is the assertion that NetChange preserves functionality ('ensures that the model retains its original functionality and performance'), but this is an unverified and, as written, under-specified correctness assumption: Algorithm 3 deletes neurons and redistributes their values without specifying the required downstream-layer updates, so for a generic trained network it likely does not preserve the composed function. A false or unsupported technical assumption is a validity problem, not circularity: the paper does not define the claimed accuracy in terms of the assumption, nor import the assumption from a self-citation; Net2Net [21] is external prior work by different authors, and no uniqueness theorem or privileged self-citation is load-bearing. Thus no circular step can be exhibited, and the paper should be scored at the bottom of the scale.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the assumption that the NetChange morphing operations preserve model behavior, and that all client models can be aligned to a common union architecture. No numerical constants are fitted to data.

assumptions (2)
  • ad hoc to paper The NetChange transformations preserve each model's learned function under widening, narrowing, deepening, and shallowing.
    Stated in Section III-B without proof; essential for FedAvg on morphed models to be meaningful.
  • domain assumption All client architectures can be aligned to a common 'union' architecture via layer-wise correspondences.
    Section III-B constructs the global model as the union of client structures, which requires that layers are comparable across models; true for VGG variants, not shown for arbitrary architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures." pith.science (2026). https://pith.science/paper/OKGO5U6W

@misc{pith2026250506497,
  author       = {Pith},
  title        = {Pith review of: FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKGO5U6W}},
  note         = {Machine review of arXiv:2505.06497}
}
read the original abstract

Traditional Federated Learning (FL) faces significant challenges in terms of efficiency and accuracy, particularly in heterogeneous environments where clients employ diverse model architectures and have varying computational resources. Such heterogeneity complicates the aggregation process, leading to performance bottlenecks and reduced model generalizability. To address these issues, we propose FedADP, a federated learning framework designed to adapt to client heterogeneity by dynamically adjusting model architectures during aggregation. FedADP enables effective collaboration among clients with differing capabilities, maximizing resource utilization and ensuring model quality. Our experimental results demonstrate that FedADP significantly outperforms existing methods, such as FlexiFed, achieving an accuracy improvement of up to 23.30%, thereby enhancing model adaptability and training efficiency in heterogeneous real-world settings.

Figures

Figures reproduced from arXiv: 2505.06497 by the authors.

Figure 1
Figure 1. Some variations of the VGG model. each client’s specific needs. Another approach, FedMN [14], uses a pool of submodels to construct client-specific architectures adaptively, ensuring effective personalization while addressing computational limitations. In addition, many methods with other strategies, such as Ditto [15], FedMoE [16], in which each client participates in global model training while maintaining and upd… view at source ↗
Figure 2
Figure 2. Difference between traditional PFL and FedADP in the workflow [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Other forms of the VGG model: VGG-14, VGG-15, VGG-17, and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The experimental results compare the performance of FedADP, FlexiFed, Cluster-FL, and Standalone on the MNIST, F-MNIST, CIFAR-10, and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages

  1. [1]

    Federated learning: Challenges, methods, and future directions

    Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine , 37(3):50–60, 2020

  2. [2]

    Adaptive federated learning in resource constrained edge computing systems

    Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications , 37(6):1205–1221, 2019

  3. [3]

    Personalized federated learning with theoretical guarantees: A model-agnostic meta- learning approach

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta- learning approach. Advances in neural information processing systems , 33:3557–3568, 2020

  4. [4]

    When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5g ultradense network

    Shuai Yu, Xu Chen, Zhi Zhou, Xiaowen Gong, and Di Wu. When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5g ultradense network. IEEE Internet of Things Journal , 8(4):2238– 2251, 2020

  5. [5]

    Heterofl: Computation and communication efficient federated learning for heterogeneous clients

    Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264 , 2020

  6. [6]

    Efficient federated learning on resource-constrained edge devices based on model pruning

    Tingting Wu, Chunhe Song, and Peng Zeng. Efficient federated learning on resource-constrained edge devices based on model pruning. Complex & Intelligent Systems , 9(6):6999–7013, 2023

  7. [7]

    Federated learning for resource- constrained iot devices: Panoramas and state of the art

    Ahmed Imteaj, Khandaker Mamun Ahmed, Urmish Thakker, Shiqiang Wang, Jian Li, and M Hadi Amini. Federated learning for resource- constrained iot devices: Panoramas and state of the art. Federated and Transfer Learning, pages 7–27, 2022

  8. [8]

    Federated multi-task learning

    Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems , 30, 2017

Show all 32 references
  1. [9]

    Flexifed: Personalized federated learning for edge clients with heterogeneous model architectures

    Kaibin Wang, Qiang He, Feifei Chen, Chunyang Chen, Faliang Huang, Hai Jin, and Yun Yang. Flexifed: Personalized federated learning for edge clients with heterogeneous model architectures. In Proceedings of the ACM Web Conference 2023 , pages 2979–2990, 2023

  2. [10]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

  3. [11]

    Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints

    Felix Sattler, Klaus-Robert M ¨uller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems , 32(8):3710–3722, 2020

  4. [12]

    Neural architecture search with reinforcement learning

    Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016

  5. [13]

    Spider: Searching personalized neural architecture for federated learning

    Erum Mushtaq, Chaoyang He, Jie Ding, and Salman Avestimehr. Spider: Searching personalized neural architecture for federated learning. arXiv preprint arXiv:2112.13939, 2021

  6. [14]

    Personalized federated learning via heterogeneous modular networks

    Tianchun Wang, Wei Cheng, Dongsheng Luo, Wenchao Yu, Jingchao Ni, Liang Tong, Haifeng Chen, and Xiang Zhang. Personalized federated learning via heterogeneous modular networks. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 1197–1202. IEEE, 2022

  7. [15]

    Ditto: Fair and robust federated learning through personalization

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning , 2020

  8. [16]

    Fedmoe: Personalized federated learning via heterogeneous mixture of experts

    Hanzi Mei, Dongqi Cai, Ao Zhou, Shangguang Wang, and Mengwei Xu. Fedmoe: Personalized federated learning via heterogeneous mixture of experts. arXiv preprint arXiv:2408.11304 , 2024

  9. [17]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429–450, 2020

  10. [18]

    Federated learning on non-iid data silos: An experimental study

    Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE) , pages 965–978. IEEE, 2022

  11. [19]

    Speed up federated learning in heterogeneous environment: A dynamic tiering approach

    Seyed Mahmoud Sajjadi Mohammadabadi, Syed Zawad, Feng Yan, and Lei Yang. Speed up federated learning in heterogeneous environment: A dynamic tiering approach. arXiv preprint arXiv:2312.05642 , 2023

  12. [20]

    Fedada: Fast-convergent adaptive federated learning in heterogeneous mobile edge computing environment

    Jinghui Zhang, Xinyu Cheng, Cheng Wang, Yuchen Wang, Zhan Shi, Jiahui Jin, Aibo Song, Wei Zhao, Liangsheng Wen, and Tingting Zhang. Fedada: Fast-convergent adaptive federated learning in heterogeneous mobile edge computing environment. World Wide Web , 25(5):1971– 1998, 2022

  13. [21]

    Net2net: Accelerating learning via knowledge transfer

    Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015

  14. [22]

    {ModelKeeper}: Accelerating {DNN} training via automated training warmup

    Fan Lai, Yinwei Dai, Harsha V Madhyastha, and Mosharaf Chowdhury. {ModelKeeper}: Accelerating {DNN} training via automated training warmup. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) , pages 769–785, 2023

  15. [23]

    Hetefedrec: Federated recommender systems with model heterogeneity

    Wei Yuan, Liang Qu, Lizhen Cui, Yongxin Tong, Xiaofang Zhou, and Hongzhi Yin. Hetefedrec: Federated recommender systems with model heterogeneity. In 2024 IEEE 40th International Conference on Data Engineering (ICDE) , pages 1324–1337. IEEE, 2024

  16. [24]

    Hdhrfl: A hierarchical robust federated learning framework for dual-heterogeneous and noisy clients

    Yalan Jiang, Dan Wang, Bin Song, and Shengyang Luo. Hdhrfl: A hierarchical robust federated learning framework for dual-heterogeneous and noisy clients. Future Generation Computer Systems , 2024

  17. [25]

    Adapterfl: Adaptive heterogeneous federated learning for resource-constrained mobile computing systems

    Ruixuan Liu, Ming Hu, Zeke Xia, Jun Xia, Pengyu Zhang, Yihao Huang, Yang Liu, and Mingsong Chen. Adapterfl: Adaptive heterogeneous federated learning for resource-constrained mobile computing systems. arXiv preprint arXiv:2311.14037 , 2023

  18. [26]

    Fiarse: Model-heterogeneous federated learning via importance- aware submodel extraction

    Feijie Wu, Xingchen Wang, Yaqing Wang, Tianci Liu, Lu Su, and Jing Gao. Fiarse: Model-heterogeneous federated learning via importance- aware submodel extraction. arXiv preprint arXiv:2407.19389 , 2024

  19. [27]

    Vfedmh: Vertical federated learning for training multi-party heterogeneous models

    Shuo Wang, Keke Gai, Jing Yu, and Liehuang Zhu. Vfedmh: Vertical federated learning for training multi-party heterogeneous models. arXiv preprint arXiv:2310.13367, 2023

  20. [28]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400–407, 1951

  21. [29]

    Survey of personalization techniques for federated learning

    Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 fourth world conference on smart trends in systems, security and sustainability (WorldS4), pages 794–797. IEEE, 2020

  22. [30]

    Gradient-based learning applied to document recognition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998

  23. [31]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    H Xiao. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017

  24. [32]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. not found , 2009

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.