{"id":"d873ff44-e4b3-4398-a628-a19c2889ed6b","arxiv_id":"2505.08033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A four-device physical testbed for decentralized federated learning trained MLPs to 81-82% F1 on MNIST and Fashion-MNIST, matching a virtualized baseline and showing denser topologies generally outperform sparser ones.","lead":"This paper describes a physical testbed of Raspberry Pi and Jetson Nano devices for decentralized federated learning, with power monitoring added to the NEBULA platform. It reports that denser communication topologies gave higher F1 scores on MNIST and Fashion-MNIST, and that physical results matched a virtualized baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Topology conclusion rests on single un-replicated runs; a multi-seed replication test would determine whether the claimed density effect is real or noise.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing flaw: the topology comparison is a single-run, no-variance result, and the short IID protocol makes the headline claim fragile. I agree that this is not a reason to reject the testbed; the physical setup and energy monitoring are concrete, demonstrable contributions, and the exact F1 match between physical and virtualized fully connected runs (82.0/81.0) lends some support to the feasibility claim. However, the abstract overstates the topology result, and the paper should either present repeated runs with variance or qualify the finding as preliminary. A multi-seed replication test is the most direct way to settle the concern, and until such data are provided, a conditional verdict is appropriate. I see no additional internal inconsistency that would change the reader's assessment, so the verdict remains conditional.","tokens_in":4224,"tokens_out":2474,"duration_ms":27016,"concrete_test":"Run K independent repetitions (e.g., 10 different random seeds) of each of the four topologies on both MNIST and FashionMNIST, using the exact Section III-A protocol (MLP, FedAvg, 10 rounds, 1 local epoch, IID split), and report mean F1 with 95% confidence intervals or bootstrap intervals. Then perform a pairwise comparison (e.g., Mann-Whitney U or bootstrap test) between fully connected and each other topology, and also check whether F1 is monotonically ordered by edge count. If the confidence intervals overlap or the ordering is non-monotonic, the claim that denser topologies improve outcomes should be explicitly softened in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that \"denser topologies lead to better outcomes\" is based entirely on Table II: one training run per topology per dataset, with 10 federation rounds, one local epoch, IID partitions, and no error bars. The observed F1 differences are small (MNIST: fully connected 82.0, star 76.0, ring 78.5, random 74.7; FashionMNIST: 81.0, 75.0, 68.8, 64.9). With no repeated seeds or confidence intervals, these differences could easily lie within run-to-run noise, especially given the short training horizon and the MLP model. Furthermore, the ordering is not even monotonic in edge count: on FashionMNIST, the ring topology (4 edges) scores lower than the star topology (3 edges), so the phrase \"denser topologies\" is doing unstated work. The paper itself limits the virtualized baseline to the fully connected topology \"due to simplicity and consistency constraints\" (Section III-A), so the virtual/physical replication claim also lacks multi-topology support. The testbed is a plausible engineering contribution, but the abstract's general statement about topology effects is not established by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a physical testbed for decentralized federated learning (DFL) built from three Raspberry Pi 4 boards and a Jetson Nano, extending the NEBULA platform with REST-based configuration delivery, real-time metric reporting, and power/energy monitoring via JT-TC66C USB multimeters. The authors compare physical and virtualized (Docker) runs on MNIST and FashionMNIST using a simple MLP with FedAvg for 10 rounds and one local epoch, and they test four physical topologies (fully connected, star, ring, random). The abstract claims that communication topology influences model performance, with denser topologies yielding better outcomes, and Section III-B concludes that the physical testbed effectively replicated learning behavior, validating its feasibility for DFL research under realistic conditions.","tokens_in":4387,"tokens_out":4410,"duration_ms":41458,"significance":"If the topology claim held, the paper would provide a useful demonstration that small physical DFL testbeds can be built and instrumented for realistic resource and energy measurement. The engineering contribution is concrete: the paper reports a working integration of NEBULA with physical nodes, per-device heterogeneity, and power metering, and it shows one matched fully-connected physical/virtualized comparison (82.0 vs 82.0 F1 on MNIST; 81.0 vs 81.0 on FashionMNIST). However, the headline scientific claim about topology effects is not supported by the reported single-run measurements, and the virtualized baseline does not cover the topologies that are the subject of that claim. The contribution is therefore currently at the level of a testbed demonstration rather than an established empirical finding.","major_comments":[{"comment":"The central claim that 'denser topologies lead to better outcomes' rests entirely on Table II, which reports one training run per topology per dataset. No repeats, no standard deviations, no confidence intervals, and no significance tests are provided. With 10 federation rounds and one local epoch, the F1 differences (e.g., MNIST fully-connected 82.0 vs ring 78.5 vs star 76.0 vs random 74.7) may be within run-to-run noise. Moreover, the ordering is not monotonic in edge count on FashionMNIST: the ring topology (4 edges) scores 68.8, below the star topology (3 edges) at 75.0. The phrase 'denser' is therefore doing unstated work, and the reported evidence does not establish the abstract's general claim.","section":"Table II / Section III-B"},{"comment":"The virtualized baseline is tested only with the fully connected topology, as stated in Section III-A ('due to simplicity and consistency constraints'). Consequently, the claim in Section III-B that 'the physical testbed effectively replicated learning behavior' is supported only for the fully connected topology, and the physical-vs-virtual comparison cannot validate the testbed's behavior for star, ring, or random topologies. Since the topology claim is the paper's main scientific contribution, this missing multi-topology baseline is a load-bearing gap.","section":"Section III-A / Table I"},{"comment":"The paper does not specify how the average F1-scores in Table II are computed across nodes, nor how node-level variability is distributed. For the star topology, the text says central nodes benefited from higher communication loads, but no per-node F1 values or variance measures are given to verify this. Without this information, readers cannot assess whether the reported topology differences reflect consistent performance across participants or are driven by a single node's behavior.","section":"Section III-B / Table II"}],"minor_comments":[{"comment":"Reference [5] is titled 'Fedstellar' while the text calls the platform NEBULA; clarify whether NEBULA is the same system or a different one, and cite accordingly.","section":"References / Section II"},{"comment":"No code, configuration files, or random graph seeds are made available; providing at least the topology definitions and hyperparameters would improve reproducibility.","section":"General"},{"comment":"The star topology's network traffic is listed as 'P0: 30, Oth: 10'; define P0 and explain why the central node transmits more data than the other nodes.","section":"Table II"},{"comment":"The power and energy measurements should specify the sampling interval and measurement accuracy of the JT-TC66C multimeters; Table II reports single values without error bounds.","section":"Section III-A / Table II"},{"comment":"The paper uses only IID data partitions; adding at least one non-IID configuration would make the topology findings more relevant to DFL practice, though this is not required for a demo.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system/demo paper; its engineering is plausible and the fully-connected physical/virtualized replication is nice. However, the abstract generalizes beyond the evidence. I recommend a major revision requiring either (a) deleting the topology-effect claim from the abstract and framing the paper strictly as a testbed demo, or (b) adding replicated runs with variance and significance testing, plus a virtualized baseline for at least one sparse topology. The lack of a code release also limits reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the useful thing here is the physical testbed plus energy metering; the headline claim about topology should be ignored until they rerun with seeds.\n\nThe real contribution is engineering. They take NEBULA/Fedstellar, which normally runs Docker containers, and deploy it on three Raspberry Pi 4s and a Jetson Nano, adding HTTP config push, periodic metric reporting, and JT-TC66C USB power meters. That closes a genuine gap: most DFL evaluations are simulated or containerized, and energy data from actual constrained devices is rare. The fact that the fully-connected physical run matched the virtualized baseline exactly (82.0/81.0 F1 on both datasets) is a good sanity check and suggests the harness is not broken.\n\nThe soft spots are in the experimental claims, not the testbed. The abstract says \"denser topologies lead to better outcomes,\" but Table II is one run per topology, ten rounds, one local epoch, IID partitions, and no error bars. The F1 spread on MNIST (74.7–82.0) could plausibly be seed noise with a short-horizon MLP. FashionMNIST is even less clean: ring (4 edges) scored 68.8 while star (3 edges) scored 75.0, so the ordering is not monotonic in edge count. \"Denser\" is doing work the data do not support. The virtual baseline only covers fully connected, so the \"replicated learning behavior\" conclusion is narrower than the text implies.\n\nOther limitations are real but not fatal: no code or configuration artifacts released, only two small datasets, IID only, and the \"random\" topology is not precisely specified. For a demo paper that is acceptable if claims stay local; it becomes a problem when the abstract generalizes. I would also want the power data analyzed more carefully, since energy per round across topologies is one of the genuinely novel measurements here and the paper only reports totals.\n\nBottom line: the testbed is worth having, and the energy-measurement angle is a legitimate contribution for DFL sustainability work. But the paper needs repeated runs with seeds, multi-topology virtual baselines, and a softened topology statement. I would send it to peer review with expectation of heavy revision, mainly because the artifact could help the community and the current flaws are fixable. I would not cite the topology result as it stands, but I would cite the testbed if they release code.","headline":"A useful physical testbed demo whose topology claim is not supported by un-replicated single runs; worth reviewing for the artifact, not for the conclusion.","tokens_in":4981,"tokens_out":2066,"would_cite":false,"duration_ms":21571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Denser communication topologies improve decentralized federated learning accuracy, and a four-device physical testbed can measure the real energy cost of training.","keywords":["decentralized federated learning","physical testbed","edge devices","energy consumption monitoring","communication topology","Raspberry Pi","Jetson Nano","resource-constrained training"],"falsifier":"Repeat the MNIST experiments on the same testbed with at least five independent runs per topology, using the same datasets and model, and compare the spread of F1 scores; if the within-topology variation overlaps the between-topology gaps reported in Table II (e.g., 82.0 vs 76.0 vs 78.5 vs 74.7), then the topology ordering is not a stable effect. A second check would re-run under non-IID partitions and with more rounds (e.g., 50) to see whether the ordering persists.","tokens_in":3978,"feed_emoji":"⚡","tokens_out":9715,"duration_ms":80340,"temperature":0.7,"pith_summary":"This paper argues that a physical testbed of four off-the-shelf edge devices—three Raspberry Pi 4 boards and an NVIDIA Jetson Nano—can serve as a realistic platform for decentralized federated learning (DFL), reproducing the model accuracy of Docker-based simulation while adding trustworthy measurements of CPU, memory, bandwidth, and energy use. Its central empirical claim is that communication topology affects model performance: in experiments on MNIST and FashionMNIST, the fully connected topology reached the highest F1 scores (around 81–82%), while ring, star, and random topologies scored lower. The authors also show that the physical testbed's F1 scores match a virtualized baseline on the fully connected topology, despite the edge devices completing training much more slowly. If correct, this gives DFL researchers a practical, energy-aware testbed for studying realistic deployments on resource-constrained hardware.","feed_headline":"A 4-device testbed: dense topology beats sparse in federated learning","feed_subtitle":"This small rig of Pis and a Jetson matches virtual accuracy while logging real watts per topology.","key_machinery":"The load-bearing object is the physical testbed itself: three Raspberry Pi 4 Model B units and one NVIDIA Jetson Nano connected over local Ethernet, running NEBULA, a decentralized federated learning platform, with three modifications—lightweight HTTP servers for configuration, periodic metric streaming, and JT-TC66C USB multimeters for real-time power and energy logging. This hardware arrangement turns the abstract variable of 'communication topology' into concrete overlay networks (fully connected, star, ring, random), and it produces the resource and energy numbers that a virtualized Docker setup cannot. The testbed's role in the argument is to provide the experimental conditions under which the F1 differences and resource measurements are observed, so the entire topology claim stands on the fidelity of this device configuration.","core_discovery":"On the paper's own terms, the central discovery is that a physical testbed can stand in for virtualized decentralized federated learning without losing model quality, and that the testbed exposes a topology effect that a simulation baseline alone would not quantify. Using three Raspberry Pi 4 devices and one Jetson Nano running the NEBULA platform with power monitoring, the authors ran 10 rounds of FedAvg with one local epoch on IID partitions of MNIST and FashionMNIST across four topologies. Fully connected training achieved average F1 scores of 82.0 (MNIST) and 81.0 (FashionMNIST), matching the virtualized Docker baseline exactly on those numbers, while star (76.0/75.0), ring (78.5/68.8), and random (74.7/64.9) topologies scored lower on average. Measured resource use stayed between 25.7% and 29.1% CPU, about 33% RAM, 3.0–3.5 W power, and 1249–1404 J total energy, leading the authors to conclude that the testbed replicates learning behavior under realistic conditions and is feasible for DFL research.","pith_inferences":["The paper uses only IID data partitions and a short 10-round horizon; a natural next test is whether the topology ordering persists under non-IID partitions, which are the norm in federated learning.","Because the power-monitoring module reports joules per run, one could directly compare energy efficiency per accuracy point across topologies, turning topology selection into a sustainability decision rather than an accuracy-only one.","The fact that the virtualized baseline matched physical F1 with a much faster run is itself informative: it suggests the simple MLP model is not bottlenecked by the edge hardware, so more complex models or larger datasets might expose differences the testbed is designed to capture.","A controlled experiment that varies node degree while holding edge count constant could isolate whether the star topology's central-node advantage comes from centrality or simply from having one node aggregate more updates."],"forward_implications":["Model performance in DFL is not fixed by the aggregation algorithm alone; the overlay topology is a first-order factor, with denser connections yielding better average F1 in these experiments.","Physical testbeds of four inexpensive edge devices can reproduce the accuracy of Docker-based DFL experiments, so future accuracy studies can trust virtualized baselines while using physical runs to obtain energy and resource measurements.","Energy monitoring on real hardware is now possible for DFL, enabling quantitative comparison of total joules consumed per topology or per round.","The measured headroom (roughly 25–29% CPU, about 3–3.5 W per device) suggests that current commodity edge devices can sustain DFL training without exhausting memory or power budgets, supporting on-device training feasibility.","The star topology's behavior—central nodes bearing higher communication loads and showing higher accuracy than edge nodes—indicates that the distribution of communication load within a topology matters, not just the number of edges."],"supporting_citations":[{"why":"supplies the DFL fundamentals and motivation (no central server, no single point of failure) that frame the testbed's purpose.","marker":"[2]"},{"why":"defines FedAvg, the training algorithm used across all physical and virtualized scenarios.","marker":"[3]"},{"why":"provides the NEBULA/Fedstellar DFL platform that the testbed adapts for physical deployment and extends with power monitoring.","marker":"[5]"},{"why":"provides the MNIST dataset used for the main accuracy and resource comparisons in Table II.","marker":"[6]"},{"why":"provides the FashionMNIST dataset used as the second benchmark for evaluating topology effects.","marker":"[7]"}],"fun_headline_variants":["Physical testbed matches virtual federated learning, topology matters","Dense topology wins on real edge devices: FedAvg testbed","Raspberry Pi and Jetson testbed proves dense network beats sparse in DFL","Edge testbed shows topology impact in decentralized FL, matches simulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result that denser topologies improve accuracy rests on one run per setting with only ten training rounds and evenly distributed data; if the few-point F1 gaps between topologies come from run-to-run noise, or disappear when data is unevenly spread across devices, that conclusion collapses.","fun_headline_variants_meta":{"raw":{"variants":["Physical testbed matches virtual federated learning, topology matters","Dense topology wins on real edge devices: FedAvg testbed","Raspberry Pi and Jetson testbed proves dense network beats sparse in DFL","Edge testbed shows topology impact in decentralized FL, matches simulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000874,"raw_usage":{"total_tokens":3766,"prompt_tokens":910,"completion_tokens":2856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2780}},"tokens_in":526,"tokens_out":2856,"duration_ms":20511,"temperature":1.0,"reasoning_tokens":2780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:04:43.037342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the MNIST experiments on the same testbed with at least five independent runs per topology, using the same datasets and model, and compare the spread of F1 scores; if the within-topology variation overlaps the between-topology gaps reported in Table II (e.g., 82.0 vs 76.0 vs 78.5 vs 74.7), then the topology ordering is not a stable effect. A second check would re-run under non-IID partitions and with more rounds (e.g., 50) to see whether the ordering persists.","supporting_citations":[{"cited_title":"Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges,","cited_arxiv_id":null,"evidence_quote":"supplies the DFL fundamentals and motivation (no central server, no single point of failure) that frame the testbed's purpose."},{"cited_title":"Fedstellar: A platform for decentralized federated learning,","cited_arxiv_id":null,"evidence_quote":"provides the NEBULA/Fedstellar DFL platform that the testbed adapts for physical deployment and extends with power monitoring."}],"review_version":1}