Demo: A Practical Testbed for Decentralized Federated Learning on Physical Edge Devices

Alberto Huertas Celdran; Burkhard Stiller; Chao Feng; Gerome Bovet; Nicolas Huber

REVIEW 3 major objections 5 minor 1 cited by

Demo: A Practical Testbed for Decentralized Federated Learning on Physical Edge Devices

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Denser communication topologies improve decentralized federated learning accuracy, and a four-device physical testbed can measure the real energy cost of training.

desk verdict A useful physical testbed demo whose topology claim is not supported by un-replicated single runs; worth reviewing for the artifact, not for the conclusion. read the letter →

arxiv 2505.08033 v1 pith:FDXF557N submitted 2025-05-12 cs.LG cs.DC

Chao Feng , Nicolas Huber , Alberto Huertas Celdran , Gerome Bovet , Burkhard Stiller This is my paper · ORCID

classification cs.LGcs.DC

keywords decentralizedfederatedlearningphysicaltestbededgedevicesenergyconsumptionmonitoringcommunicationtopologyRaspberryPiJetsonNanoresource-constrainedtraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a physical testbed of four off-the-shelf edge devices—three Raspberry Pi 4 boards and an NVIDIA Jetson Nano—can serve as a realistic platform for decentralized federated learning (DFL), reproducing the model accuracy of Docker-based simulation while adding trustworthy measurements of CPU, memory, bandwidth, and energy use. Its central empirical claim is that communication topology affects model performance: in experiments on MNIST and FashionMNIST, the fully connected topology reached the highest F1 scores (around 81–82%), while ring, star, and random topologies scored lower. The authors also show that the physical testbed's F1 scores match a virtualized baseline on the fully connected topology, despite the edge devices completing training much more slowly. If correct, this gives DFL researchers a practical, energy-aware testbed for studying realistic deployments on resource-constrained hardware.

What carries the argument

The load-bearing object is the physical testbed itself: three Raspberry Pi 4 Model B units and one NVIDIA Jetson Nano connected over local Ethernet, running NEBULA, a decentralized federated learning platform, with three modifications—lightweight HTTP servers for configuration, periodic metric streaming, and JT-TC66C USB multimeters for real-time power and energy logging. This hardware arrangement turns the abstract variable of 'communication topology' into concrete overlay networks (fully connected, star, ring, random), and it produces the resource and energy numbers that a virtualized Docker setup cannot. The testbed's role in the argument is to provide the experimental conditions under which the F1 differences and resource measurements are observed, so the entire topology claim stands on the fidelity of this device configuration.

What would settle it

Repeat the MNIST experiments on the same testbed with at least five independent runs per topology, using the same datasets and model, and compare the spread of F1 scores; if the within-topology variation overlaps the between-topology gaps reported in Table II (e.g., 82.0 vs 76.0 vs 78.5 vs 74.7), then the topology ordering is not a stable effect. A second check would re-run under non-IID partitions and with more rounds (e.g., 50) to see whether the ordering persists.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a physical testbed can stand in for virtualized decentralized federated learning without losing model quality, and that the testbed exposes a topology effect that a simulation baseline alone would not quantify. Using three Raspberry Pi 4 devices and one Jetson Nano running the NEBULA platform with power monitoring, the authors ran 10 rounds of FedAvg with one local epoch on IID partitions of MNIST and FashionMNIST across four topologies. Fully connected training achieved average F1 scores of 82.0 (MNIST) and 81.0 (FashionMNIST), matching the virtualized Docker baseline exactly on those numbers, while star (76.0/75.0), ring (78.5/68.8), and random (74.7/64.9) topologies scored lower on average. Measured resource use stayed between 25.7% and 29.1% CPU, about 33% RAM, 3.0–3.5 W power, and 1249–1404 J total energy, leading the authors to conclude that the testbed replicates learning behavior under realistic conditions and is feasible for DFL research.

Load-bearing premise

The result that denser topologies improve accuracy rests on one run per setting with only ten training rounds and evenly distributed data; if the few-point F1 gaps between topologies come from run-to-run noise, or disappear when data is unevenly spread across devices, that conclusion collapses.

Editorial extensions

If this is right

Model performance in DFL is not fixed by the aggregation algorithm alone; the overlay topology is a first-order factor, with denser connections yielding better average F1 in these experiments.
Physical testbeds of four inexpensive edge devices can reproduce the accuracy of Docker-based DFL experiments, so future accuracy studies can trust virtualized baselines while using physical runs to obtain energy and resource measurements.
Energy monitoring on real hardware is now possible for DFL, enabling quantitative comparison of total joules consumed per topology or per round.
The measured headroom (roughly 25–29% CPU, about 3–3.5 W per device) suggests that current commodity edge devices can sustain DFL training without exhausting memory or power budgets, supporting on-device training feasibility.
The star topology's behavior—central nodes bearing higher communication loads and showing higher accuracy than edge nodes—indicates that the distribution of communication load within a topology matters, not just the number of edges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper uses only IID data partitions and a short 10-round horizon; a natural next test is whether the topology ordering persists under non-IID partitions, which are the norm in federated learning.
Because the power-monitoring module reports joules per run, one could directly compare energy efficiency per accuracy point across topologies, turning topology selection into a sustainability decision rather than an accuracy-only one.
The fact that the virtualized baseline matched physical F1 with a much faster run is itself informative: it suggests the simple MLP model is not bottlenecked by the edge hardware, so more complex models or larger datasets might expose differences the testbed is designed to capture.
A controlled experiment that varies node degree while holding edge count constant could isolate whether the star topology's central-node advantage comes from centrality or simply from having one node aggregate more updates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: the useful thing here is the physical testbed plus energy metering; the headline claim about topology should be ignored until they rerun with seeds.

The real contribution is engineering. They take NEBULA/Fedstellar, which normally runs Docker containers, and deploy it on three Raspberry Pi 4s and a Jetson Nano, adding HTTP config push, periodic metric reporting, and JT-TC66C USB power meters. That closes a genuine gap: most DFL evaluations are simulated or containerized, and energy data from actual constrained devices is rare. The fact that the fully-connected physical run matched the virtualized baseline exactly (82.0/81.0 F1 on both datasets) is a good sanity check and suggests the harness is not broken.

The soft spots are in the experimental claims, not the testbed. The abstract says "denser topologies lead to better outcomes," but Table II is one run per topology, ten rounds, one local epoch, IID partitions, and no error bars. The F1 spread on MNIST (74.7–82.0) could plausibly be seed noise with a short-horizon MLP. FashionMNIST is even less clean: ring (4 edges) scored 68.8 while star (3 edges) scored 75.0, so the ordering is not monotonic in edge count. "Denser" is doing work the data do not support. The virtual baseline only covers fully connected, so the "replicated learning behavior" conclusion is narrower than the text implies.

Other limitations are real but not fatal: no code or configuration artifacts released, only two small datasets, IID only, and the "random" topology is not precisely specified. For a demo paper that is acceptable if claims stay local; it becomes a problem when the abstract generalizes. I would also want the power data analyzed more carefully, since energy per round across topologies is one of the genuinely novel measurements here and the paper only reports totals.

Bottom line: the testbed is worth having, and the energy-measurement angle is a legitimate contribution for DFL sustainability work. But the paper needs repeated runs with seeds, multi-topology virtual baselines, and a softened topology statement. I would send it to peer review with expectation of heavy revision, mainly because the artifact could help the community and the current flaws are fixable. I would not cite the topology result as it stands, but I would cite the testbed if they release code.

Referee Report

3 major / 5 minor

Summary. The paper presents a physical testbed for decentralized federated learning (DFL) built from three Raspberry Pi 4 boards and a Jetson Nano, extending the NEBULA platform with REST-based configuration delivery, real-time metric reporting, and power/energy monitoring via JT-TC66C USB multimeters. The authors compare physical and virtualized (Docker) runs on MNIST and FashionMNIST using a simple MLP with FedAvg for 10 rounds and one local epoch, and they test four physical topologies (fully connected, star, ring, random). The abstract claims that communication topology influences model performance, with denser topologies yielding better outcomes, and Section III-B concludes that the physical testbed effectively replicated learning behavior, validating its feasibility for DFL research under realistic conditions.

Significance. If the topology claim held, the paper would provide a useful demonstration that small physical DFL testbeds can be built and instrumented for realistic resource and energy measurement. The engineering contribution is concrete: the paper reports a working integration of NEBULA with physical nodes, per-device heterogeneity, and power metering, and it shows one matched fully-connected physical/virtualized comparison (82.0 vs 82.0 F1 on MNIST; 81.0 vs 81.0 on FashionMNIST). However, the headline scientific claim about topology effects is not supported by the reported single-run measurements, and the virtualized baseline does not cover the topologies that are the subject of that claim. The contribution is therefore currently at the level of a testbed demonstration rather than an established empirical finding.

major comments (3)

[Table II / Section III-B] The central claim that 'denser topologies lead to better outcomes' rests entirely on Table II, which reports one training run per topology per dataset. No repeats, no standard deviations, no confidence intervals, and no significance tests are provided. With 10 federation rounds and one local epoch, the F1 differences (e.g., MNIST fully-connected 82.0 vs ring 78.5 vs star 76.0 vs random 74.7) may be within run-to-run noise. Moreover, the ordering is not monotonic in edge count on FashionMNIST: the ring topology (4 edges) scores 68.8, below the star topology (3 edges) at 75.0. The phrase 'denser' is therefore doing unstated work, and the reported evidence does not establish the abstract's general claim.
[Section III-A / Table I] The virtualized baseline is tested only with the fully connected topology, as stated in Section III-A ('due to simplicity and consistency constraints'). Consequently, the claim in Section III-B that 'the physical testbed effectively replicated learning behavior' is supported only for the fully connected topology, and the physical-vs-virtual comparison cannot validate the testbed's behavior for star, ring, or random topologies. Since the topology claim is the paper's main scientific contribution, this missing multi-topology baseline is a load-bearing gap.
[Section III-B / Table II] The paper does not specify how the average F1-scores in Table II are computed across nodes, nor how node-level variability is distributed. For the star topology, the text says central nodes benefited from higher communication loads, but no per-node F1 values or variance measures are given to verify this. Without this information, readers cannot assess whether the reported topology differences reflect consistent performance across participants or are driven by a single node's behavior.

minor comments (5)

[References / Section II] Reference [5] is titled 'Fedstellar' while the text calls the platform NEBULA; clarify whether NEBULA is the same system or a different one, and cite accordingly.
[General] No code, configuration files, or random graph seeds are made available; providing at least the topology definitions and hyperparameters would improve reproducibility.
[Table II] The star topology's network traffic is listed as 'P0: 30, Oth: 10'; define P0 and explain why the central node transmits more data than the other nodes.
[Section III-A / Table II] The power and energy measurements should specify the sampling interval and measurement accuracy of the JT-TC66C multimeters; Table II reports single values without error bounds.
[General] The paper uses only IID data partitions; adding at least one non-IID configuration would make the topology findings more relevant to DFL practice, though this is not required for a demo.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's empirical comparisons use external benchmarks and a virtualized baseline, and the only self-citations are background/infrastructure rather than load-bearing support.

full rationale

This paper contains no derivation chain whose output is equivalent to its input. The central empirical comparisons use external benchmarks (MNIST, FashionMNIST) and an independently implemented virtualized baseline, so the physical-versus-virtualized replication claim is a direct experimental comparison rather than a restatement of an assumption. NEBULA/Fedstellar (Ref. [5]) and the DFL survey (Ref. [2]) are self-citations with overlapping authorship, but they are cited for background and as infrastructure; neither is invoked to establish the paper's conclusions about topology or resource consumption. The topology-dependence claim rests on Table II single-run comparisons, which is a statistical robustness concern, not a circularity concern. No fitted parameter is relabeled as a prediction, no quantity is defined in terms of the quantity it is said to predict, and no uniqueness theorem from the authors' prior work is used to force a choice. Score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is empirical and introduces no derived constants, so the ledger records the main hand-chosen experimental settings (10 rounds, 1 local epoch) and the domain assumptions that the central comparison silently relies on: IID data, FedAvg on a decentralized graph, uncalibrated power meters, and benign network conditions.

free parameters (2)

Federation rounds = 10
Chosen by hand for the demo; no convergence analysis, and the observed topology ordering may depend on this short horizon.
Local epochs per round = 1
Chosen by hand; with one local epoch, client drift is small, which may affect how topology influences performance.

assumptions (4)

domain assumption All datasets are partitioned IID across devices.
Section III-A states datasets were partitioned independently and identically distributed, which is unrealistic for DFL and may suppress topology effects that emerge under non-IID data.
domain assumption Decentralized execution of FedAvg over the overlay preserves the expected convergence properties.
The paper uses FedAvg on a decentralized topology without analyzing or proving convergence for the modified gossip-style aggregation.
domain assumption JT-TC66C USB multimeter readings are accurate ground truth for power and energy.
Section II introduces the power monitoring module but provides no calibration or comparison against a reference instrument.
domain assumption The local Ethernet network is representative of real edge deployment conditions.
The testbed uses a local Ethernet network with no packet loss, bandwidth limits, or wireless interference, which could affect the topology comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Demo: A Practical Testbed for Decentralized Federated Learning on Physical Edge Devices." pith.science (2026). https://pith.science/paper/FDXF557N

@misc{pith2026250508033,
  author       = {Pith},
  title        = {Pith review of: Demo: A Practical Testbed for Decentralized Federated Learning on Physical Edge Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FDXF557N}},
  note         = {Machine review of arXiv:2505.08033}
}

read the original abstract

Federated Learning (FL) enables collaborative model training without sharing raw data, preserving participant privacy. Decentralized FL (DFL) eliminates reliance on a central server, mitigating the single point of failure inherent in the traditional FL paradigm, while introducing deployment challenges on resource-constrained devices. To evaluate real-world applicability, this work designs and deploys a physical testbed using edge devices such as Raspberry Pi and Jetson Nano. The testbed is built upon a DFL training platform, NEBULA, and extends it with a power monitoring module to measure energy consumption during training. Experiments across multiple datasets show that model performance is influenced by the communication topology, with denser topologies leading to better outcomes in DFL settings.

Figures

Figures reproduced from arXiv: 2505.08033 by the authors.

**Figure 2.** Sequence diagram of configuration distribution, model training, and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Chained Attacks on Drone-Based Federated Learning: From Network Disruption to Device Impersonation
cs.CR 2026-07 conditional novelty 5.0 of 10

A chained Wi-Fi deauthentication and credential-impersonation attack degrades drone federated learning and lets an attacker replace a disconnected client without detection.

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages · cited by 1 Pith paper

[1]

Federated Learning for Resource-Constrained IoT Devices: Panoramas and State-of-the-art

A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “Federated learning for resource-constrained iot devices: Panoramas and state-of- the-art,” CoRR, vol. abs/2002.10610, 2020

work page Pith review arXiv 2002
[2]

Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,

E. T. Mart ´ınez Beltr ´an, M. Q. P ´erez, P. M. S. S ´anchez, S. L. Bernal, G. Bovet, M. G. P ´erez, G. M. P ´erez, and A. H. Celdr ´an, “Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,” IEEE Communications Surveys & Tutorials, vol. 25, no. 4, p. 2983–3013, 2023

work page 2023
[3]

Federated optimization: Distributed machine learning for on-device intelligence,

J. Kone ˇcn`y, H. B. McMahan, D. Ramage, and P. Richt ´arik, “Federated optimization: Distributed machine learning for on-device intelligence,” arXiv preprint arXiv:1610.02527 , 2016

arXiv 2016
[4]

Privacy-preserving and syscall-based intrusion detection system for iot spectrum sensors affected by data falsification attacks,

A. Huertas Celdr ´an, P. M. S ´anchez S ´anchez, C. Feng, G. Bovet, G. M. P´erez, and B. Stiller, “Privacy-preserving and syscall-based intrusion detection system for iot spectrum sensors affected by data falsification attacks,” IEEE Internet of Things Journal , vol. 10, no. 10, pp. 8408– 8415, 2023

work page 2023
[5]

Fedstellar: A platform for decentralized federated learning,

E. T. M. Beltr ´an, ´A. L. P. G´omez, C. Feng, P. M. S. S´anchez, S. L. Bernal, G. Bovet, M. G. P ´erez, G. M. P ´erez, and A. H. Celdr ´an, “Fedstellar: A platform for decentralized federated learning,” Expert Systems with Applications, vol. 242, p. 122861, 2024

work page 2024
[6]

The mnist database of handwritten digit images for machine learning research [best of the web],

L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE Signal Processing Magazine , vol. 29, no. 6, pp. 141–142, 2012

2012
[7]

H. Xiao, K. Rasul, and R. V ollgraf. (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Federated Learning for Resource-Constrained IoT Devices: Panoramas and State-of-the-art

A. Imteaj, U. Thakker, S. Wang, J. Li, and M. H. Amini, “Federated learning for resource-constrained iot devices: Panoramas and state-of- the-art,” CoRR, vol. abs/2002.10610, 2020

work page Pith review arXiv 2002

[2] [2]

Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,

E. T. Mart ´ınez Beltr ´an, M. Q. P ´erez, P. M. S. S ´anchez, S. L. Bernal, G. Bovet, M. G. P ´erez, G. M. P ´erez, and A. H. Celdr ´an, “Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,” IEEE Communications Surveys & Tutorials, vol. 25, no. 4, p. 2983–3013, 2023

work page 2023

[3] [3]

Federated optimization: Distributed machine learning for on-device intelligence,

J. Kone ˇcn`y, H. B. McMahan, D. Ramage, and P. Richt ´arik, “Federated optimization: Distributed machine learning for on-device intelligence,” arXiv preprint arXiv:1610.02527 , 2016

arXiv 2016

[4] [4]

Privacy-preserving and syscall-based intrusion detection system for iot spectrum sensors affected by data falsification attacks,

A. Huertas Celdr ´an, P. M. S ´anchez S ´anchez, C. Feng, G. Bovet, G. M. P´erez, and B. Stiller, “Privacy-preserving and syscall-based intrusion detection system for iot spectrum sensors affected by data falsification attacks,” IEEE Internet of Things Journal , vol. 10, no. 10, pp. 8408– 8415, 2023

work page 2023

[5] [5]

Fedstellar: A platform for decentralized federated learning,

E. T. M. Beltr ´an, ´A. L. P. G´omez, C. Feng, P. M. S. S´anchez, S. L. Bernal, G. Bovet, M. G. P ´erez, G. M. P ´erez, and A. H. Celdr ´an, “Fedstellar: A platform for decentralized federated learning,” Expert Systems with Applications, vol. 242, p. 122861, 2024

work page 2024

[6] [6]

The mnist database of handwritten digit images for machine learning research [best of the web],

L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE Signal Processing Magazine , vol. 29, no. 6, pp. 141–142, 2012

2012

[7] [7]

H. Xiao, K. Rasul, and R. V ollgraf. (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

2017