REVIEW 6 major objections 5 minor 27 references
FLsim: A Modular and Library-Agnostic Simulation Framework for Federated Learning
T0 review · 6 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read FLsim claims to be a comprehensive, modular, library-agnostic simulation framework that executes the full range of federated learning workflows, from client-server to decentralized, with reproducible and scalable experiments.
desk verdict A genuinely modular FL simulator with a reproducible evaluation, but the built-in algorithm implementations are never checked against reference results, so the broad versatility claim runs ahead of the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the FLsim Strategy class, an object-oriented wrapper that bundles the dataset, model, training loop, testing loop, and aggregation function into one definition users implement, alongside the YAML job configuration that declares dataset, consensus, cluster, strategy, and node settings. This bundle is what makes the framework library-agnostic: the framework only sees the strategy interface, not the underlying ML library. The second mechanism is FLsim's node synchronization protocol, based on ProcessPhase and NodeStage signals managed by the Logic Controller, which keeps all clients and workers in lockstep across local training, aggregation, and global-model selection. The framework also exposes a pluggable consensus function and a blockchain API through which aggregated models are voted on and the next global parameter is chosen.
What would settle it
Run the same FL job definition on FLsim and on the original published codebases for the seven algorithms with identical data, model, seeds, and hyperparameters; if final-round accuracy or loss diverges substantially beyond hardware-level floating-point differences, the claim that FLsim can faithfully simulate state-of-the-art FL workflows is refuted. A second check would be to add a new ML library through the strategy interface and verify that no framework-internal change is needed, testing the complete library-agnosticism claim.
Extended reading notes
Core claim
The paper's central discovery is that a workflow abstraction built on a job-configuration file (dataset parameters, consensus, topology/clusters, FL strategy, node defaults, and per-node settings) plus a single object-oriented FLsim Strategy class can express diverse FL proposals without touching framework internals. FLsim implements this abstraction with six components: a Job Orchestrator, a Logic Controller, a Dataset Distributor, client/worker nodes, a key-value store, and a performance logger, communicating through REST APIs and a pub-sub key-value store. On this basis the paper claims complete ML library agnosticism, support for client-server, hierarchical, and decentralized topologies, pluggable blockchain integration, deterministic reproducibility across runs, and scalability to 1,000 clients.
Load-bearing premise
The capacity claims rest on the assumption that the seven built-in implementations (FedAvg, FedAvgM, SCAFFOLD, MOON, Geyer et al., Briggs et al., and Fedstellar) faithfully reproduce the behavior of the original algorithms, since the paper validates them only by plausible accuracy and loss curves, not by comparison to published reference results.
Editorial extensions
If this is right
- A researcher can implement the same FL strategy once and run it under any major ML library, switching between client-server, hierarchical, and decentralized topologies by editing a configuration file.
- Multi-worker and blockchain-aided aggregation becomes testable in simulation, including scenarios with malicious workers trying to poison the global model.
- Seed-synchronized runs on different hardware (x86 CPU, distributed CPU, GPU, and ARM) produce near-identical accuracy and loss, so experiments can be compared across environments.
- The framework scales from tens to 1,000 clients, making large-scale FL research feasible on a single machine.
Reading between the lines
- If FLsim is adopted as a common harness, the FL community could move toward benchmark results that differ only in algorithm and configuration, reducing the fragmentation caused by each project using its own simulator.
- The strategy-class abstraction suggests that algorithms written for one ML library could be ported to another with minimal changes, though the paper does not demonstrate such a port directly.
- The reproducibility mechanism could be pressed further into a standard, such as requiring seed-synchronized deterministic mode plus hardware-specific floating-point notes in every FL paper, an extension the authors do not claim.
- The blockchain integration opens a testable path for studying consensus and poisoning defenses at scale, but simulation fidelity depends on how faithfully the smart contracts and network behavior match a real deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FLsim, a Python-based simulation framework for federated learning that is modular, configurable through YAML job files, and agnostic to ML libraries by delegating model/training/aggregation logic to user-defined strategy classes. The framework is described as supporting diverse FL algorithms, multiple network topologies, multi-worker consensus, pluggable blockchain integration, deterministic reproducibility, and large-scale simulations. The evaluation in Section 4 attempts to answer seven research questions covering support for diverse FL proposals, ML-library agnosticism, multi-worker aggregation, topologies, reproducibility, scalability, and (via description only) blockchain support. The paper also includes an architecture overview, a synchronization algorithm, and a qualitative comparison table against existing FL frameworks.
Significance. If the reported capabilities are validated, FLsim could be a useful contribution to the FL simulation landscape, particularly because it is released as open source and its design separates orchestration from user-defined strategy logic. The paper gives explicit credit to the framework's modular architecture, its support for multiple topologies, and its reproducibility experiments across x86, GPU, and ARM hardware. However, the central claim that FLsim can faithfully execute the full range of FL workflows is not yet established by the evaluation, because the built-in algorithm implementations are not checked against reference results and some experiments do not isolate the claimed capability. The architectural contribution is real, but the empirical validation needs substantial strengthening before the versatility and library-agnosticism claims can be accepted.
major comments (6)
- [Section 4.1, Figure 8] The central RQ1 claim that FLsim can faithfully execute a diverse range of FL algorithms rests entirely on the authors' own implementations of FedAvg, FedAvgM, SCAFFOLD, MOON, Geyer et al., Briggs et al., and Fedstellar. No comparison is made to the original codebases, to published reference accuracy/loss numbers, or to independent reimplementations. Because all seven algorithms are run on the same CIFAR-10 Dirichlet(0.5) setup with the same three-CNN model and the same learning rate, plausible monotone curves cannot distinguish a faithful implementation from a lookalike. To support the versatility claim, the paper should include at least one reference point per algorithm, for example a comparison to results from the original papers or to an existing benchmark, and an ablation of the algorithm-specific mechanism (control variates for SCAFFOLD, contrastive loss for MOON, clustering criterion for Briggs et al., peer-to-peer exchange for Fedstellar).
- [Section 4.2, Figure 9] The experiment for RQ2 does not test ML-library agnosticism. The three implementations differ in model architecture (a three-CNN model for PyTorch and TensorFlow versus a flattened four-hidden-layer MLP for Scikit-Learn), so the observed differences in accuracy, memory, and bandwidth are confounded with architecture and cannot be attributed to the ML library. Since FLsim delegates all model and training logic to user-defined strategy classes, library agnosticism may hold by design, but it is not validated by this experiment. A valid test would keep the model architecture and training procedure identical across libraries where possible, or at least use the same architecture in PyTorch and TensorFlow and verify that the resulting global-model trajectories match.
- [Section 2.3, Algorithm 1] Algorithm 1 as written has a synchronization deadlock in the first round. The loop begins with ProcessPhase = 0, and lines 17--22 call waitForProcessPhase(1) on all clients and waitForProcessPhase(2) on all workers before ProcessPhase is set to 1 at line 23. If these waits are blocking, no node's condition is ever satisfied and the assignments on lines 23 and 38 are never reached. If the actual implementation uses polling with timeouts or event-driven signals, the pseudocode should make that explicit, because the framework's core synchronization mechanism is described incorrectly as presented.
- [Section 4.5, Tables 1 and 2] The reproducibility evidence is ambiguous as presented. The tables list the same accuracy and loss vectors three times under 'Trial 1', 'Trial 2', and 'Trial 3' for each hardware configuration, which may be intended to demonstrate exact determinism, but the table structure does not make clear whether the three trials were independently executed with different random seeds, data orders, or environment states. The paper should state the exact seed-initialization procedure (including the DETERMINISTIC and RANDOM_SEED environment variables described in Section 5) and report trial-level variation explicitly, for example as mean ± std over runs, so that the claim of controlled reproducibility is verifiable.
- [Section 4.3, Figure 10] The multi-worker consensus experiment relies solely on an implementation of the FedRLChain consensus algorithm [13], which has two of the current authors as co-authors, and is evaluated without a benign-worker baseline or a comparison to the original FedRLChain results. The observed recovery when honest workers outnumber malicious workers is consistent with the algorithm's intent, but it does not demonstrate that FLsim's consensus mechanism is correctly integrated. Adding a 0M-4H baseline (no malicious workers) and optionally an independent reimplementation of the consensus logic would make the support for RQ3 falsifiable.
- [Section 4 (RQ4)] The paper states in Section 2.4 that FLsim provides pluggable blockchain support for Ethereum and Hyperledger Fabric, yet Section 4 contains no experimental evaluation for RQ4 (platform-agnostic blockchain-based traceability and verifiability of the learning process). The only evidence is the architectural description and the API outline. To substantiate the claim, the paper should include at least a demonstration of a BCFL experiment on one of the supported blockchains, showing that the smart contract executes, the consensus outcome is recorded on-chain, and the global model selection is traceable.
minor comments (5)
- [Section 4.6, Figure 12] The legend in Figure 12(b) says '500 Client' and '1000 Client' (singular), and the x-axis of that panel is labeled 'Time (min)' while earlier bandwidth plots use seconds; please unify the units and names.
- [Section 5, third paragraph] The statement that 'upgrading the Flask server to use waitress, which uses CPython under the hood' is misleading: waitress is a pure-Python WSGI server and is not tied to CPython in the way the sentence suggests.
- [Table 3] The column 'ML Framework Agnostic' is undefined; if it means the framework does not hard-code a particular ML library, define it in the caption and verify the entries against the cited capabilities of each framework.
- [Section 2.2, Figure 2] The YAML example in Figure 2 uses 'concensus_def' and 'consensus_alias' with inconsistent spelling ('concensus' versus 'consensus'), and the caption 'depicted n Figure 2b' should read 'in Figure 2b'.
- [Section 2.3, Algorithm 1] The comment 'Model Aggergation' in the ProcessPhase definition contains a typo; it should read 'Model Aggregation'.
Circularity Check
No significant circularity: FLsim is a systems paper whose capability claims are supported by demonstrations, not by a derivation chain that reduces to its own inputs.
full rationale
This is a systems/engineering paper, not a theoretical derivation, so the classic circularity patterns (self-definitional prediction, fitted input called prediction, uniqueness theorems, ansatz smuggling) do not apply. The central claim is that FLsim is a modular, library-agnostic simulation framework, and the evidence is a set of experiments showing that seven external FL algorithms, three ML libraries, multiple topologies, and a multi-worker consensus scenario can be run through the framework. The only self-citation is the use of Chowdhury et al. [13] in Section 4.3 as the consensus algorithm for the multi-worker poisoning experiment. That citation is not load-bearing for the framework's generality: [13] supplies an example algorithm to be hosted, not evidence that FLsim works, and no prediction is derived from it. The paper makes no claim that the consensus outcome is forced by the framework; it reports an observed behavior of that particular algorithm. Concerns that the authors' implementations of FedAvgM, SCAFFOLD, MOON, etc. might not faithfully reproduce the originals are legitimate correctness/validation risks, but they are not circularity: the framework is not defined in terms of those implementations' outputs, and the experimental curves are demonstrations, not predictions fitted from the same data. The comparison against external algorithms (FedAvg, FedAvgM, SCAFFOLD, MOON, Geyer et al., Briggs et al., Fedstellar) gives the versatility claim independent content. Overall, no circular step can be exhibited from the paper's own equations or definitions.
Assumptions & free parameters
assumptions (4)
- domain assumption The evaluated algorithms (FedAvg, FedAvgM, SCAFFOLD, MOON, DP-FL, hierarchical clustering, Fedstellar) are faithfully implemented in FLsim and behave like the original methods.
- domain assumption Setting random seeds and library-level deterministic flags (DETERMINISTIC, RANDOM_SEED) is sufficient to make experiments exactly reproducible.
- ad hoc to paper The REST-based coordination and Key-Value Store do not change the semantics of the user-supplied FL algorithm.
- ad hoc to paper ML library agnosticism is achieved by delegating all model and training logic to user-implemented strategy classes.
Cite this review
Pith. "Pith review of FLsim: A Modular and Library-Agnostic Simulation Framework for Federated Learning." pith.science (2026). https://pith.science/paper/WUL6IRS7
@misc{pith2026250711430,
author = {Pith},
title = {Pith review of: FLsim: A Modular and Library-Agnostic Simulation Framework for Federated Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/WUL6IRS7}},
note = {Machine review of arXiv:2507.11430}
}
read the original abstract
Federated Learning (FL) has undergone significant development since its inception in 2016, advancing from basic algorithms to complex methodologies tailored to address diverse challenges and use cases. However, research and benchmarking of novel FL techniques against a plethora of established state-of-the-art solutions remain challenging. To streamline this process, we introduce FLsim, a comprehensive FL simulation framework designed to meet the diverse requirements of FL workflows in the literature. FLsim is characterized by its modularity, scalability, resource efficiency, and controlled reproducibility of experimental outcomes. Its easy to use interface allows users to specify customized FL requirements through job configuration, which supports: (a) customized data distributions, ranging from non-independent and identically distributed (non-iid) data to independent and identically distributed (iid) data, (b) selection of local learning algorithms according to user preferences, with complete agnosticism to ML libraries, (c) choice of network topology illustrating communication patterns among nodes, (d) definition of model aggregation and consensus algorithms, and (e) pluggable blockchain support for enhanced robustness. Through a series of experimental evaluations, we demonstrate the effectiveness and versatility of FLsim in simulating a diverse range of state-of-the-art FL experiments. We envisage that FLsim would mark a significant advancement in FL simulation frameworks, offering unprecedented flexibility and functionality for researchers and practitioners alike.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[13]
Fedrlchain: Secure federated deep reinforcement learning with blockchain
Sujit Chowdhury, Arnab Mukherjee, and Raju Halder. Fedrlchain: Secure federated deep reinforcement learning with blockchain. IEEE Transactions on Services Computing, 2023
work page 2023
-
[1]
Communication- efficient learning of deep networks from decentralized data
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–
-
[2]
Measuring the effects of non-identical data distribution for federated visual classification
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019
arXiv 1909
-
[3]
Federated optimization in heterogeneous networks
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020
2020
-
[4]
Model-contrastive federated learning
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713–10722, 2021
work page 2021
-
[5]
Scaffold: Stochastic controlled averaging for federated learning
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132–5143. PMLR, 2020
2020
-
[6]
Adaptive federated optimization
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone ˇcn`y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020
arXiv 2003
-
[7]
Differentially private federated learning: A client level perspec- tive
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspec- tive. arXiv preprint arXiv:1712.07557, 2017
arXiv 2017
Show all 27 references
-
[8]
Pcfed: Privacy-enhanced and communication-efficient federated learning for industrial iots
Qing Han, Shusen Yang, Xuebin Ren, Peng Zhao, Cong Zhao, and Yimeng Wang. Pcfed: Privacy-enhanced and communication-efficient federated learning for industrial iots. IEEE Transactions on Industrial Informatics, 18(9):6181–6191, 2022
2022
-
[9]
Federated learning with matched averaging
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020
2002 arXiv
-
[10]
Federated learning with personalization layers
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019. 15 FL SIM : A M ODULAR AND LIBRARY -AGNOSTIC SIMULATION FRAMEWORK FOR FEDERATED LEARNING
1912 arXiv
-
[11]
Grace: A generalized and personalized federated learning method for medical imaging
Ruipeng Zhang, Ziqing Fan, Qinwei Xu, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Grace: A generalized and personalized federated learning method for medical imaging. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 14–24. Springer, 2023
2023
-
[12]
A greedy agglomerative framework for clustered federated learning
Manan Mehta and Chenhui Shao. A greedy agglomerative framework for clustered federated learning. IEEE Transactions on Industrial Informatics, 2023
2023
-
[14]
Blockchain-based federated learning with secure aggregation in trusted execution envi- ronment for internet-of-things
Aditya Pribadi Kalapaaking, Ibrahim Khalil, Mohammad Saidur Rahman, Mohammed Atiquzzaman, Xun Yi, and Mahathir Almashor. Blockchain-based federated learning with secure aggregation in trusted execution envi- ronment for internet-of-things. IEEE Transactions on Industrial Infor...
2022
-
[15]
Federated active semi-supervised learning with communication efficiency
Chen Zhang, Yu Xie, Hang Bai, Xiongwei Hu, Bin Yu, and Yuan Gao. Federated active semi-supervised learning with communication efficiency. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023
2023
-
[16]
Fedstream: Prototype-based federated learning on distributed concept-drifting data streams
Cobbinah B Mawuli, Liwei Che, Jay Kumar, Salah Ud Din, Zhili Qin, Qinli Yang, and Junming Shao. Fedstream: Prototype-based federated learning on distributed concept-drifting data streams. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023
2023
-
[17]
https://www.tensorflow.org/federated
TensorFlow Federated — tensorflow.org. https://www.tensorflow.org/federated. [Accessed 06-02- 2024]
2024
-
[18]
Pysyft: A library for easy federated learning
Alexander Ziller, Andrew Trask, Antonio Lopardo, Benjamin Szymkow, Bobby Wagner, Emma Bluemke, Jean- Mickael Nounahon, Jonathan Passerat-Palmbach, Kritika Prakash, Nick Rose, et al. Pysyft: A library for easy federated learning. Federated Learning Systems: Towards Next-Generat...
2021
-
[19]
Fate: An industrial grade platform for collaborative learning with data protection
Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, and Qiang Yang. Fate: An industrial grade platform for collaborative learning with data protection. The Journal of Machine Learning Research, 22(1):10320–10325, 2021
2021
-
[20]
Fed-biomed: Open, transparent and trusted federated learning for real-world healthcare applications
Francesco Cremonesi, Marc Vesin, Sergen Cansiz, Yannick Bouillard, Irene Balelli, Lucia Innocenti, Santiago Silva, Samy-Safwan Ayed, Riccardo Taiello, Laetita Kameni, et al. Fed-biomed: Open, transparent and trusted federated learning for real-world healthcare applications. ar...
2023 arXiv
-
[21]
Flower: A friendly federated learning research framework
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusm ˜ao, et al. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390, 2020
2007 arXiv
-
[22]
Fedlab: A flexible federated learning frame- work
Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning frame- work. Journal of Machine Learning Research, 24(100):1–7, 2023
2023
-
[23]
Fedml: A research library and benchmark for federated ma- chine learning
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated ma- chine learning. arXiv preprint arXiv:2007.13518, 2020
2007 arXiv
-
[24]
Fedstellar: A platform for decentralized federated learning
Enrique Tom ´as Mart´ınez Beltr´an, ´Angel Luis Perales G ´omez, Chao Feng, Pedro Miguel S ´anchez S´anchez, Ser- gio L ´opez Bernal, G ´erˆome Bovet, Manuel Gil P ´erez, Gregorio Mart ´ınez P ´erez, and Alberto Huertas Celdr ´an. Fedstellar: A platform for decentralized feder...
2024
-
[25]
Deepchain: Auditable and privacy- preserving deep learning with blockchain-based incentive
Jiasi Weng, Jian Weng, Jilian Zhang, Ming Li, Yue Zhang, and Weiqi Luo. Deepchain: Auditable and privacy- preserving deep learning with blockchain-based incentive. IEEE Transactions on Dependable and Secure Com- puting, 18(5):2438–2455, 2019
2019
-
[26]
Federated learning with hierarchical clustering of local up- dates to improve training on non-iid data
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local up- dates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2020
2020
-
[27]
1,500 scientists lift the lid on reproducibility
Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 2016. 16
2016
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.