Pith. sign in

REVIEW 2 major objections 2 minor 19 references

ByzFL: Research Framework for Robust Federated Learning

T0 review · 2 major / 2 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ByzFL is an open-source library that bundles robust aggregators, Byzantine attacks, and a federated learning simulator behind a single JSON configuration, aiming to make robustness benchmarks in federated learning reproducible and…

desk verdict A genuinely useful benchmark toolkit for robust FL, but the authors have not shown their implementations are faithful, and that missing validation is the load-bearing gap. read the letter →

arxiv 2505.24802 v1 pith:2JSTOA6X submitted 2025-05-30 cs.LG

Marc González , Rachid Guerraoui , Rafael Pinot , Geovani Rizk , John Stephan , François Taïani This is my paper
classification cs.LG
keywords federatedlearningByzantinerobustnessrobustaggregationbenchmarkingadversarialattacksdataheterogeneityJSONconfiguration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents ByzFL, an open-source Python library for benchmarking robust federated learning. The authors argue that the field lacks a standardized testbed, with each paper using custom implementations and threat models, making fair comparison difficult. ByzFL is their answer: a modular framework that implements state-of-the-art robust aggregators and pre-aggregators, a suite of Byzantine attack strategies, and a full federated training simulator, all driven by a single JSON configuration file. A user can define the number of clients, data heterogeneity, attack, aggregator, and training algorithm in one config, run the benchmark with one function call, and get accuracy curves and worst-case heatmaps. The library also works as a standalone robust mean estimator on PyTorch tensors or NumPy arrays, which the authors say makes it useful beyond federated learning.

What carries the argument

The machinery that carries the argument is the modular architecture centered on a RobustAggregator module, which implements aggregators such as MultiKrum, geometric median, trimmed mean, centered clipping, and others, plus pre-aggregators such as NNM, bucketing, static clipping, and ARC. The Client, ByzantineClient, Server, and DataDistributor classes compose robust aggregation into a complete federated training loop, and the Benchmark class reads a single config.json file that declaratively specifies the entire experiment. The JSON configuration is the core reproducibility device: it encodes the threat model, data partition, training algorithm, and evaluation metrics, so that launching a benchmark is a one-function-call operation and the configuration file itself documents the experiment setup.

What would settle it

Take a published experiment from one of the original papers, such as the trimmed mean accuracy trajectory under ALIE from Yin et al. 2018, and run the same configuration through ByzFL; if the resulting curve differs substantially from the published one, the library's implementations cannot be treated as faithful and every benchmark built on them inherits that doubt.

Watch

Extended reading notes

Core claim

The central claim is that ByzFL provides a unified and extensible framework for designing, testing, and benchmarking robust federated learning algorithms. The library implements a range of state-of-the-art robust aggregators and pre-aggregators, along with canonical Byzantine attack strategies, and composes them into a full simulation through Client, ByzantineClient, Server, and DataDistributor classes. The paper introduces a worst-case maximal accuracy metric for the built-in heatmaps: for each configuration of Byzantine client count and heterogeneity level, the benchmark runs every attack in the user-specified suite and reports the minimum of the maximum test accuracies across attacks, so an aggregator is scored by its performance under the most damaging threat it faces. The authors also claim that the modular design, in which aggregators operate directly on PyTorch tensors or NumPy arrays, lets the robust aggregation components be used independently of any training pipeline, extending the library's use to robust statistics.

Load-bearing premise

The library's benchmarks are only as trustworthy as its reimplementations of the published aggregators and attacks, because the paper does not validate its code against results from the original papers.

Editorial extensions

If this is right

  • If ByzFL is adopted, robustness papers can report results on a common platform, making it possible to compare aggregators under identical threat models and heterogeneity settings.
  • A single JSON configuration makes experiments reproducible by construction, since the config captures the entire experimental pipeline.
  • The worst-case maximal accuracy heatmaps give an attack-agnostic performance score, so a defense cannot look strong by ignoring attacks it was not designed for.
  • Because aggregators and attacks are decoupled from the training loop, they can be reused as standalone robust mean estimators outside federated learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the community trusts the library's reimplementations, ByzFL could become a de facto standard benchmark, but that would require the paper or its maintainers to publish validation against original-paper results; the paper itself does not do this.
  • The Opt-IPM and Opt-ALIE adaptive attacks, which grid-search the attack factor to maximize the gap between the honest mean and the aggregator output, could serve as a tool to estimate an upper bound on the damage a tuned adversary can inflict on a given defense.
  • The JSON-driven orchestration pattern could extend to other dimensions, such as privacy attacks, communication constraints, or cross-silo settings, since the same declarative structure is generic.
  • A concrete validation study, in which ByzFL reproduces the numbers from several original algorithm papers, would either establish the library as authoritative or reveal which reimplementations need fixing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper presents ByzFL, an open-source Python library for benchmarking and prototyping robust federated learning. The library provides implementations of robust aggregators and pre-aggregators (e.g., MultiKrum, geometric median, trimmed mean, mean around median, NNM, bucketing, ARC), a set of Byzantine attacks (sign flipping, IPM, ALIE, and adaptive variants), data heterogeneity simulation, and a JSON-based configuration system that launches complete FL experiments. The paper describes the modular architecture, gives code listings and one example experiment with the trimmed mean aggregator, and introduces a worst-case maximal accuracy metric for comparing defenses across attacks. The central claim is that ByzFL standardizes and simplifies the empirical evaluation of robust FL methods.

Significance. If the implementations are faithful to the cited algorithms, ByzFL would fill a genuine need in the robust FL community, which currently lacks a standardized, extensible benchmarking platform. The modular design that decouples robust aggregators from the training pipeline is a strength, as is the config-driven experimentation model. The paper also makes a useful contribution by proposing the worst-case maximal accuracy metric, which encourages attack-agnostic evaluation. However, the paper's value rests entirely on the correctness and reliability of its reimplementations of published algorithms. The manuscript does not currently demonstrate that reliability, and without a test suite or reproducibility checks against original papers, the benchmark outputs cannot be trusted. The significance is therefore conditional on adding such validation.

major comments (2)
  1. [Section 2, 'Robust (pre-)aggregators' and 'Simulating adversarial behavior'] The central claim that ByzFL is a reliable benchmarking framework depends on the faithful reimplementation of the cited robust aggregators and attacks. The paper provides no validation evidence for these implementations: there are no unit tests, no reference vectors, no hand-computed examples beyond trivial cases, and no comparison to published numerical results from Blanchard et al. 2017, Yin et al. 2018, Baruch et al. 2019, or any other source. A subtle bug in any aggregator or attack (e.g., self-distance handling in MultiKrum, the attack-factor scaling in ALIE, or the definition of f for the trimmed mean) would silently invalidate every benchmark produced by the library. I ask the authors to add a proper test suite and at least one reproduction experiment whose numbers can be checked against a published table or figure.
  2. [Section 3, 'Result interpretation and visualization', Figure 3 and surrounding text] The example experiment reports test-accuracy heatmaps and accuracy-over-round curves without any measure of variance. The config.json in Listing 7 includes 'nb_training_seeds': 3, suggesting that multiple seeds are supported, but Figure 3 appears to show point estimates with no error bars, confidence intervals, or mention of how many seeds were averaged. Since robust FL training is stochastic and can be noisy, a benchmarking tool must communicate run-to-run variability; otherwise, differences between aggregators or attacks in the heatmap cannot be distinguished from noise. Please report the number of seeds and show the variance (e.g., standard deviation or percentile bands) for each reported value.
minor comments (2)
  1. [Listing 1] Line 14 has unbalanced parentheses: 'agg(pre_agg(x)' is missing a closing parenthesis, and line 15 has an extra closing parenthesis in the print statement. This should be corrected so that the code snippet is executable as printed.
  2. [Listing 7] The sample config.json uses '//' comments, which are not valid JSON. If the ByzFL parser supports comments, this should be documented; otherwise the example must be changed to valid JSON so that users can copy it without errors.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ByzFL is an implementation/benchmarking paper whose cited algorithms are external attributions; the missing cross-validation is a fidelity risk, not a circular reduction.

full rationale

ByzFL makes no derived theoretical predictions; its contributions are an open-source library, a JSON configuration system, modular aggregators/attacks, and a worst-case metric. The robust aggregators and attacks are attributed to external peer-reviewed sources (e.g., MultiKrum [Blanchard et al., 2017], ALIE [Baruch et al., 2019], IPM [Xie et al., 2019]), and the many self-citations (Allouah et al., Farhadkhani et al., Guerraoui et al., Stephan, Pinot, Rizk) are attributions of prior algorithms and theory, not load-bearing premises that make the framework's value true by definition. The adaptive Opt-IPM/Opt-ALIE variants optimize an explicit attack objective via grid search; this is an attack-construction procedure, not a fitted parameter later renamed a prediction. The worst-case maximal accuracy metric is a definition (min over attacks of max accuracy), not a derivation. The absence of cross-validation against original papers' numerical results is a real benchmark-fidelity risk, but it is an empirical validation gap, not a circular step: no equation reduces to another by construction, and no claimed prediction is forced by a fit. Therefore no significant circularity.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is a software paper with no mathematical derivation. It introduces no fitted parameters and no new physical or mathematical entities. The load-bearing elements are implementation correctness and framework design choices, which are not captured by free parameters or formal axioms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ByzFL: Research Framework for Robust Federated Learning." pith.science (2026). https://pith.science/paper/2JSTOA6X

@misc{pith2026250524802,
  author       = {Pith},
  title        = {Pith review of: ByzFL: Research Framework for Robust Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2JSTOA6X}},
  note         = {Machine review of arXiv:2505.24802}
}
read the original abstract

We present ByzFL, an open-source Python library for developing and benchmarking robust federated learning (FL) algorithms. ByzFL provides a unified and extensible framework that includes implementations of state-of-the-art robust aggregators, a suite of configurable attacks, and tools for simulating a variety of FL scenarios, including heterogeneous data distributions, multiple training algorithms, and adversarial threat models. The library enables systematic experimentation via a single JSON-based configuration file and includes built-in utilities for result visualization. Compatible with PyTorch tensors and NumPy arrays, ByzFL is designed to facilitate reproducible research and rapid prototyping of robust FL solutions. ByzFL is available at https://byzfl.epfl.ch/, with source code hosted on GitHub: https://github.com/LPD-EPFL/byzfl.

Figures

Figures reproduced from arXiv: 2505.24802 by the authors.

Figure 1
Figure 1. FL setup with server-side robust aggregation of client updates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of a robust FL pipeline using [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance of the trimmed mean aggregator [Yin et al., 2018] under adversarial conditions [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 7 canonical work pages

  1. [9]

    doi: 10.1561/2200000083

    ISSN 1935-8237. doi: 10.1561/2200000083. URL http://dx.doi.org/10.1561/2200000083. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132–5143. PMLR,

    doi:10.1561/2200000083 1935

  2. [12]

    Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed

    URL https://proceedings.mlsys.org/paper_files/paper/2020/file/ 1f5fe83998a09396ebe6477d9475ba0c-Paper.pdf. Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Universal multi-party poi- soning attacks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedi...

    work page 2020

  3. [13]

    v13i1.625

    doi: 10.29379/jedem. v13i1.625. URL https://jedem.org/index.php/jedem/article/view/625. Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn´ y, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In Interna- tional Conference on Learning Representations ,

    work page doi:10.29379/jedem

  4. [14]

    URL https://doi.org/10.1177/2053951716649398

    doi: 10.1177/ 2053951716649398. URL https://doi.org/10.1177/2053951716649398. Th´ eo Ryffel, Andrew Trask, Michael Dahl, Jason Wagner, Bobby Mancuso, Daniel Rueckert, and Jonathan Passerat-Palmbach. A generic framework for privacy-preserving deep learning. arXiv preprint arXiv:1811.04017,

    arXiv

  5. [15]

    TensorFlow Federated Contributors

    URL https: //arxiv.org/abs/2201.11990. TensorFlow Federated Contributors. Tensorflow federated: Machine learning on decentralized data. https://www.tensorflow.org/federated,

    arXiv

  6. [16]

    Colin van Noordt and Gianluca Misuraca

    Accessed: 2024-05-24. Colin van Noordt and Gianluca Misuraca. Exploratory insights on artificial intelligence for gov- ernment in europe. Social Science Computer Review , 40(2):426–444,

    work page 2024

  7. [17]

    URL https://doi.org/10.1177/0894439320980449

    doi: 10.1177/ 0894439320980449. URL https://doi.org/10.1177/0894439320980449. Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Generalized byzantine-tolerant sgd,

    doi:10.1177/0894439320980449

  8. [19]

    Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Peter Civin, and Vikas Chandra

    URL https: //proceedings.mlr.press/v80/yin18a.html. Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Peter Civin, and Vikas Chandra. Federated learning with non-iid data. In arXiv preprint arXiv:1806.00582 ,

    arXiv

Show all 19 references
  1. [1982]

    doi: 10.1145/357172.357176

    ISSN 0164-0925. doi: 10.1145/357172.357176. Yann LeCun and Corinna Cortes. MNIST handwritten digit database. Unpublished dataset,

  2. [2010]

    Shenghui Li, Edith Ngai, Fanghua Ye, Li Ju, Tianru Zhang, and Thiemo Voigt

    URL http://yann.lecun.com/exdb/mnist/. Shenghui Li, Edith Ngai, Fanghua Ye, Li Ju, Tianru Zhang, and Thiemo Voigt. Blades: A unified benchmark suite for byzantine attacks and defenses in federated learning. In 2024 IEEE/ACM Ninth International Conference on Internet-of-Things ...

    2024

  3. [2012]

    URL https://proceedings.neurips.cc/paper_files/paper/2012/file/ 6aca97005c68f1206823815f66102863-Paper.pdf. Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muze...

    2012

  4. [2015]

    Daniel Beutel, Tilo Topal, Akhil Mathur, Xiang Qiu, Titouan Parcollet, and Nicholas D. Lane. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390,

    2007 arXiv

  5. [2016]

    Zeyuan Allen-Zhu, Faeze Ebrahimianghazani, Jerry Li, and Dan Alistarh

    URL https://arxiv.org/abs/1603.04467. Zeyuan Allen-Zhu, Faeze Ebrahimianghazani, Jerry Li, and Dan Alistarh. Byzantine-resilient non- convex stochastic gradient descent. In International Conference on Learning Representations ,

    arXiv

  6. [2018]

    Fall of empires: Breaking byzantine-tolerant SGD by inner product manipulation

    Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine-tolerant SGD by inner product manipulation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019 , page 83,

    2019

  7. [2019]

    Peter Kairouz, H

    URL https://arxiv.org/abs/1909.06335. Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aur´ elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Hubert Eichner, Salim El Rouayheb, David ...

    1909 arXiv

  8. [2020]

    doi: 10.1038/s41586-020-2649-2

    ISSN 1476-4687. doi: 10.1038/s41586-020-2649-2. URL http://dx.doi.org/10.1038/s41586-020-2649-2 . Chaoyang He, Murali Annavaram, and Salman Avestimehr. Fedml: A research library and bench- mark for federated machine learning. arXiv preprint arXiv:2007.13518 ,

    arXiv

  9. [2021]

    doi: 10.1162/daed a 01861

    ISSN 0011-5266. doi: 10.1162/daed a 01861. URL https://doi.org/10.1162/daed_a_ 01861. 10 Dimitri Bertsekas and John Tsitsiklis. Parallel and distributed computation: numerical methods . Athena Scientific,

    doi

  10. [2022]

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc ' au- relio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng

    URL https://arxiv.org/abs/2204.02311. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc ' au- relio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng. Large scale distributed deep networks. In F. Pereira, C.J. Burges, L. Botto...

    arXiv

  11. [2024]

    Adaptive gradient clipping for robust federated learning

    Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Ahmed Jellouli, Geovani Rizk, and John Stephan. Adaptive gradient clipping for robust federated learning. In The Thirteenth Interna- tional Conference on Learning Representations , 2025a. URL https://openreview.net/forum? id=03...

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.