{"id":"ee08e9d1-6c69-4ddc-93b0-e9a5cfb1c215","arxiv_id":"2507.02295","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Flotilla is a modular, resilient federated learning framework that runs on heterogeneous edge devices, supports sync and async strategies, and scales to 1000+ clients with low overhead.","lead":"Flotilla is a new open-source framework for running federated learning on real edge devices, with support for synchronous and asynchronous training strategies and recovery from client and server failures. It aims to give systems researchers a common platform to compare FL strategies on real hardware rather than in simulation alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Server failover is validated only for FedAvg; partial-state consistency for stateful async strategies is unproven, so the general resilience claim is not yet established.","rationale":"The reader's weakest_assumption identified exactly the load-bearing point: Sec. 3.5's admission that persisted state may be partial and that modules must estimate consistency on restore. My stress-test elaborates why this is not merely a theoretical caveat but a concrete correctness risk for the very strategies that distinguish Flotilla from Flower and OpenFL. The paper provides strong, reproducible experimental evidence for modularity, client-failure tolerance, and low overhead on real edge devices, and it is transparent about the consistency limitation. However, the server-failover experiment only exercises FedAvg, whose state is nearly trivial to reconcile; the framework's ability to resume stateful async strategies such as FedAT under arbitrary mid-round failures is unsupported. This does not refute the central claim, but it narrows it: the advertised 'seamless server failover' is demonstrated for a simple synchronous strategy, not for the modular composition of sync and async strategies that the paper positions as its primary contribution. The proposed test would settle this by injecting failures at precise points where the partial-state inconsistency would manifest. Other concerns, such as the missing derivation of the 92.5% weak-scaling efficiency or the use of Flotilla's dataloader in the Flower comparison, are real but less decisive: the scaling curves and resource plots still provide evidence, and the comparisons are clearly documented. I therefore keep the reader's CONDITIONAL verdict rather than escalating to REJECT, and I agree that the stated weak assumption is the correct locus for the condition.","tokens_in":32833,"tokens_out":4040,"duration_ms":48576,"concrete_test":"Run FedAT (or FedAsync) with Redis-backed external state on the Docker-208 cluster, select clients per round, and at a random mid-round point kill the Leader process after one client's local model has been persisted but before the corresponding client-selection counter is updated. Fail over to a standby Leader and continue training for 50 rounds. Compare final accuracy and per-round progress to a no-failure control, and at each restore assert the FedAT invariants (e.g., CS tierAggNum <= Agg tierAggNum for every tier, and tier-model versions are consistent with the selected-client lists). Repeat with 10 random kill points. If any run deadlocks, diverges, or violates an invariant, the server-failover claim does not generalize beyond FedAvg.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline resilience claim is that Flotilla tolerates server failures and resumes a session on another server within hundreds of milliseconds of detection. Section 3.5 explicitly concedes that when a server fails mid-round, the externally persisted state 'may be a partial state,' and that modules 'need to estimate the consistency of the keys they wish to use,' with stronger consistency models deferred to future work. The empirical support in Sec. 4.4.1, however, only exercises this mechanism with CCNN/CIFAR10-IID/FedAvg on the Pi cluster. FedAvg's Aggregation State is a simple dictionary of client weights for the current round, so a partial state can be reconciled by simply waiting for the remaining clients or re-selecting; the invariants that must survive a crash are trivial. The same is not true for the stateful strategies the paper highlights. FedAT, for example, maintains tier models, update counts, and selected-client lists whose consistency depends on the relative progression of the Client Selection State and Aggregation State (e.g., CS's tierAggNum must match or lag the Agg's tierAggNum). If the server dies after a tier model is written to Redis but before the CS counter is advanced, a restored session could double-count an update, lose a tier model, or deadlock waiting for a client already aggregated. No fault-injection experiment covers FedAsync or FedAT, and no consistency check is applied to the restored state. Thus the demonstrated sub-second failover is a property of a single, nearly stateless aggregation strategy, not a validated property of the framework for the modular async strategies that are central to the paper's novelty. The missing derivation of the 92.5% weak-scaling efficiency is secondary; this gap directly affects the headline resilience contribution and the claim that users can 'rapidly compose' arbitrary sync and async strategies on a failure-tolerant platform.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Flotilla, a federated learning (FL) framework designed for modular composition of client-selection and aggregation strategies, stateless clients, and server-side session state that can be checkpointed to disk or externalized to Redis. The authors implement five FL strategies (FedAvg, TiFL, HACCS, FedAsync, FedAT) and evaluate them on a Raspberry Pi cluster, a Jetson cluster, and containerized clusters with up to 1080 clients. The paper further claims client-failure tolerance and server-failure failover, reports weak-scaling performance versus Flower, and compares resource usage (CPU, memory, round time) with Flower, OpenFL, and FedML on the Pi cluster. The central claim is that Flotilla provides a scalable, modular, and resilient platform for FL systems research and deployment on heterogeneous edge hardware.","tokens_in":33251,"tokens_out":7332,"duration_ms":73468,"significance":"The paper's main strengths are its breadth of empirical evaluation: real hardware (three Pi types, four Jetson types), 1080 containerized clients, five strategies spanning synchronous/asynchronous aggregation, fault-injection experiments for client and server failures, and a direct comparison with three established FL frameworks. The finding that simple baselines match or beat sophisticated strategies on real hardware is a useful reproducibility-oriented result, and the stated open-source release would benefit the community. If the resilience and scalability claims are supported by the intended fixes, Flotilla would be a competitive platform. However, several headline claims—general server failover for stateful strategies, the 92.5% weak-scaling efficiency, and resource comparison on Jetsons—are not backed by the current evidence.","major_comments":[{"comment":"The general server-failover claim is not supported for the stateful asynchronous strategies that the paper highlights. Sec. 3.5 concedes that the externally persisted state may be a partial state and that modules need to estimate the consistency of the keys they wish to use, with stronger consistency models deferred to future work. The only server-failure experiment (Sec. 4.4.1) uses CCNN/CIFAR10-IID/FedAvg, whose Aggregation State is a simple stash of local models for the current round and is reconcilable by waiting for the remaining clients. The FedAT pseudocode in Appendix A.1, by contrast, maintains tierAggNum counters in both the Client Selection State and the Aggregation State, and a tier model is written before the CS counter is advanced; a crash in that window can double-count an update or deadlock the recovered session. Since no fault-injection experiment covers FedAsync or FedAT, the paper's claim that the external state store makes Flotilla generally resilient to server failures (Sec. 5) is an extrapolation rather than a demonstrated result. In addition, Contribution 2's wording that training is resumed within 820ms of detection is misleading: the 820ms is only leader startup plus state restoration, while actual resumption of training takes an additional ~171s to complete the partial round. The claim should be qualified to state the partial-round completion time explicitly.","section":"Sec. 3.5 and Sec. 4.4.1"},{"comment":"The paper claims a weak-scaling efficiency of 92.5% in Contribution 3, citing Sec. 4.5, but Sec. 4.5 never defines or reports a weak-scaling efficiency metric. The section reports end-to-end training times and overhead percentages for 56, 112, 160, 208, and 1080 clients, but no efficiency number appears. Either the metric must be defined and computed from the reported data, or the claim should be removed.","section":"Sec. 1.4 Contribution 3 and Sec. 4.5"},{"comment":"The abstract states that Flotilla's resource usage on Raspberry Pis and Nvidia Jetson edge accelerators is comparable to or better than three state-of-the-art FL frameworks, but the only cross-framework resource comparison is performed on the Raspberry Pi cluster (Fig. 13). No CPU, memory, or round-time comparison on the Jetson cluster is presented. The claim should be restricted to the Pi cluster or supported by additional experiments.","section":"Abstract and Sec. 4.6.2"},{"comment":"The client-failure tolerance experiment uses only CCNN/CIFAR10-IID/FedAvg, and the paper itself attributes the negligible accuracy impact to the IID data distribution. The resilience claims in the abstract and conclusions are not conditioned on IID data, so the current experiment is insufficient to establish client-failure tolerance for non-IID workloads, where the loss of clients with particular label distributions can bias the global model. The evaluation should either include a non-IID fault-injection run or the claims should be qualified.","section":"Sec. 4.4.3"}],"minor_comments":[{"comment":"The table header reads 'Jenson-Shannon Score' but should be 'Jensen-Shannon Score'.","section":"Sec. 4.1.3, Table 5"},{"comment":"The legend labels and the text descriptions are inconsistent: the text describes 'Single Machine Fail' while the figure shows 'Single Machine Fails' and 'Server Resuming.' Please align the terminology and clarify which lines correspond to each setup.","section":"Sec. 4.4.1, Fig. 10a"},{"comment":"The sentence 'This avoiding users having to write any code if existing modules suffice' should read 'This avoids users having to write any code if existing modules suffice.'","section":"Sec. 2.2.1"},{"comment":"The phrase 'A discrete checkpointing strategy' should likely be 'A disk checkpointing strategy,' and the distinction between disk checkpointing and the Redis external state store should be made explicit.","section":"Sec. 3.5"},{"comment":"The text describes OpenFL memory usage in MiB while Fig. 13 plots memory in GB; please harmonize the units for consistency.","section":"Sec. 4.6.2"},{"comment":"The sentence 'Client missing 5 consecutive heartbeats are marked inactive' should be 'Clients missing five consecutive heartbeats are marked inactive.'","section":"Sec. 4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's first page states that the paper was published in the Journal of Parallel and Distributed Computing with DOI 10.1016/j.jpdc.2025.105103. If this is not a resubmission to that same journal, the editor should verify whether prior publication or dual submission is a concern. Otherwise, the main technical issue is the gap between the advertised generality of server-failover resilience and the narrow experimental support (FedAvg only); the other unsupported claims (92.5% weak-scaling efficiency, Jetson resource comparison) are more straightforward to fix by either adding the missing analysis or qualifying the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is worth reading if you care about FL systems rather than FL theory. Flotilla is a framework, and the authors did the heavy lifting: real Pi and Jetson clusters, 1080 containers, five strategies implemented in modest LOC, and direct comparison against Flower, OpenFL, and FedML. The modular design with client selection and aggregation as separate stateful modules is genuinely nice, and runtime model delivery to stateless clients is a real feature that other frameworks lack.\n\nThe weak-scaling result at 208 clients is solid, and the 1080-client comparison showing Flower's overhead growing to 54.9% while Flotilla stays at 1.7% is the kind of concrete number that makes the case. Checkpointing overheads are reported honestly, including the 0.07% overhead for Redis state persistence.\n\nThe main gap is server resilience. The paper claims failover within ~820ms, but the fault injection is only for CCNN/CIFAR10-IID/FedAvg on the Pi cluster. FedAvg's aggregation state is just a dict of weights for the current round; a partial state is trivially recoverable. The paper itself admits in Sec 3.5 that the persisted state 'may be a partial state' and that modules 'need to estimate the consistency of the keys they wish to use.' That is not a mechanism, it is a caveat. For FedAT, a strategy the paper highlights, the state includes tier models and update counts that must be coordinated; no experiment shows that a crash between a tier-model write and the counter update is handled correctly. So the general claim that Flotilla is a resilient platform for async strategies is not yet established by the evidence.\n\nSecondary issues: the abstract's 92.5% weak-scaling efficiency does not appear in the body. The Flower resource comparison uses Flotilla's dataloader for Flower, which biases the results—though the 1080-client scaling comparison is about orchestration, so it is less affected. Accuracy comparisons appear to be single runs with no error bars; given the hardware variability they report, that is a real limitation.\n\nWho should read it: systems researchers building or benchmarking FL frameworks, and anyone who wants an honest apples-to-apples evaluation of FL strategies on real hardware. The finding that FedAvg and FedAsync match or beat TiFL/HACCS/FedAT in most configurations is useful evidence against simulation-only claims.\n\nI would send it to peer review. It is a solid systems paper with substantial reproducible engineering, but I would ask the authors to either fault-inject a stateful async strategy (FedAsync or FedAT) and show recovery, or soften the resilience claims to 'supported for stateless aggregation strategies.' And add the missing 92.5% derivation or remove the number. This is fixable in revision.","headline":"Flotilla is a genuine systems contribution with real experiments, but the headline server-failover claim is only demonstrated for the simplest strategy and the 92.5% weak-scaling number is missing from the body.","tokens_in":33762,"tokens_out":2250,"would_cite":true,"duration_ms":25758,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a state-centered, event-driven design can make an FL framework simultaneously modular, resilient, and scalable, with server failover in under a second and 1.7% overhead at 1080 clients.","keywords":["federated learning","asynchronous aggregation","server failover","client fault tolerance","edge devices","scalability","modular FL framework","Redis state store"],"falsifier":"Set up a FedAT or TiFL session with Redis-backed states, kill the leader after the client-selection state records the chosen clients for a tier but before the aggregation state has a matching entry, then let a standby leader resume; if the resumed run's round count, tier bookkeeping, or accuracy trajectory differs from an uninterrupted run in a way that mirrors the partial write, the failover claim fails for strategies that do not add their own consistency checks.","tokens_in":32644,"feed_emoji":"🔁","tokens_out":8202,"duration_ms":88119,"temperature":0.7,"pith_summary":"The paper is trying to establish that a federated learning framework can be built around modular, event-driven client-selection and aggregation modules plus an externalized session state, and that this design is enough to make the framework competitive across the axes FL systems actually need: rapid composition of synchronous and asynchronous strategies, tolerance of client dropouts and server death, small resource footprint on low-end edge devices, and scaling beyond a thousand clients. If that claim holds, Flotilla gives systems researchers and practitioners a single platform on which novel FL strategies can be implemented in tens to hundreds of lines, deployed unchanged from simulation to real Raspberry Pi and Jetson clusters, and compared against each other on equal terms. The paper supports the claim by implementing five FL strategies, injecting failures on 200+ clients and between two servers, and benchmarking resource usage against three established FL frameworks.","feed_headline":"Flotilla resumes a failed FL server in under a second","feed_subtitle":"Modular selection and aggregation states keep synchronous and asynchronous strategies training across 1000+ edge clients.","key_machinery":"The load-bearing mechanism is the state-centric training lifecycle with pluggable strategy modules. The central objects are the five persistent session states and the two module interfaces, `clientSelect(...)` and `aggregate(...)`, each of which receives read-only access to other modules' states and read-write access to its own. The event-driven loop is what allows synchronous and asynchronous behavior to coexist: the client-selection module is invoked on every client response and can either return new clients or defer, while the aggregation module can stash local models or return a new global model. Failure recovery is declared to work by reconstructing these states from Redis or disk and resuming the same event loop.","core_discovery":"On the paper's own terms, the discovery is that federated learning orchestration can be reduced to an event-driven lifecycle over a small set of named session states, and that this reduction is what buys modularity, resilience, and scale simultaneously. Flotilla's leader maintains five state objects — Client Info, Training Session, Client Selection, Client Training, and Aggregation — and exposes each one to client-selection (CS) and aggregation (Agg) modules with either read-only or read-write wrappers. Every client response triggers the lifecycle in the same order (selection, training, aggregation, validation), so a synchronous strategy like FedAvg can defer aggregation until all selected models arrive while FedAsync aggregates immediately on receipt; the same interfaces implement TiFL, HACCS, and FedAT. Clients are stateless: they receive model code on demand, send heartbeats over MQTT, and return trained models over gRPC, so a client failure just removes one participant from the pool. The leader's session state can be mirrored to Redis or checkpointed to disk, letting a standby server restore and resume a session in under a second with only the interrupted round lost. The paper's evaluations then report that this design yields 59–246 lines of code per implemented strategy, near-identical accuracy when 89 of 208 clients fail, 1.7% framework overhead at 1080 clients versus Flower's 54.9%, and resource usage on Raspberry Pi and Jetson clusters comparable to or better than Flower, OpenFL, and FedML.","pith_inferences":["A testable extension is to use the Redis-backed state as a hook for checkpointing at finer granularity than one round; the paper's own overhead numbers (about 143 ms per disk checkpoint for LeNet and 24.8 MiB of cumulative state for CCNN) suggest that mid-round resumption could be made nearly continuous.","The partial-state caveat implies a design pressure on future FL strategies: authors who want reliable failover should specify which of their state keys must be written atomically, and the paper explicitly leaves stronger built-in consistency for future work.","The observation that sophisticated strategies did not beat FedAvg and FedAsync on real hardware, despite tuning to the settings in their papers, points to a community-level need to report systems-level convergence results rather than only simulated accuracy; the paper itself draws this conclusion for the five strategies it implemented.","Flotilla's state model could be reused for other orchestration tasks, such as multiple concurrent sessions or hierarchical FL, which the paper names as future work but does not implement."],"forward_implications":["A researcher can move a strategy from single-machine simulation to a real edge cluster by changing a YAML file, since the same leader and client deployment executes both modes.","Client failures no longer require restarting a session: heartbeat detection plus a per-round timeout lets training continue with the remaining pool, and the paper reports near-identical final accuracy when 89 of 208 clients were killed.","Server failover becomes a deployable option: with Redis mirroring, another server can resume the session in under a second, losing at most the partial round, plus only a few seconds of checkpointing cost every five rounds.","Large-scale containerized FL emulation is practical: Flotilla's overhead stays at 1.7% of wall time with 1080 clients and 100 training calls per round, where the paper measures Flower's overhead at 54.9%.","An apples-to-apples comparison on real hardware can change conclusions drawn from simulation; the paper's five-strategy runs find FedAvg and FedAsync often match or beat the sophisticated strategies on final accuracy within a fixed time budget."],"supporting_citations":[{"why":"Defines FedAvg, the synchronous aggregation baseline that anchors Flotilla's modularity demonstration and comparisons.","marker":"[12]"},{"why":"Supplies FedAsync, the staleness-weighted asynchronous aggregation strategy that Flotilla implements.","marker":"[19]"},{"why":"Supplies TiFL, the tier-based client selection strategy used to demonstrate Flotilla's interface flexibility.","marker":"[17]"},{"why":"Supplies HACCS, the heterogeneity-aware clustered client selection strategy implemented in Flotilla.","marker":"[18]"},{"why":"Supplies FedAT, the asynchronous tiered strategy whose cross-tier coordination exercises Flotilla's state model.","marker":"[20]"},{"why":"Provides Flower, the framework Flotilla compares against for resource usage and for the 1080-client scaling test.","marker":"[25]"},{"why":"Provides OpenFL, one of three frameworks in the Raspberry Pi resource-footprint comparison.","marker":"[26]"},{"why":"Provides FedML, another baseline for the resource-footprint comparison on Raspberry Pi devices.","marker":"[27]"}],"fun_headline_variants":["FL framework cuts server failover to under a second","Flotilla scales federated learning to 1000+ resilient clients","Modular FL framework survives client failures with fast resume","Flotilla enables rapid failover and modular FL strategies","Edge FL framework with sub-second recovery and 1000+ clients"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"When a server dies mid-round, the saved session state may be only partially written; Flotilla's fast failover works only if each FL strategy's modules can tell which parts of that state are still consistent and safely resume from it.","fun_headline_variants_meta":{"raw":{"variants":["FL framework cuts server failover to under a second","Flotilla scales federated learning to 1000+ resilient clients","Modular FL framework survives client failures with fast resume","Flotilla enables rapid failover and modular FL strategies","Edge FL framework with sub-second recovery and 1000+ clients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":2049,"prompt_tokens":1154,"completion_tokens":895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":770,"completion_tokens_details":{"reasoning_tokens":811}},"tokens_in":770,"tokens_out":895,"duration_ms":7384,"temperature":1.0,"reasoning_tokens":811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:32:16.485797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set up a FedAT or TiFL session with Redis-backed states, kill the leader after the client-selection state records the chosen clients for a tier but before the aggregation state has a matching entry, then let a standby leader resume; if the resumed run's round count, tier bookkeeping, or accuracy trajectory differs from an uninterrupted run in a way that mirrors the partial write, the failover claim fails for strategies that do not add their own consistency checks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FedAvg, the synchronous aggregation baseline that anchors Flotilla's modularity demonstration and comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TiFL, the tier-based client selection strategy used to demonstrate Flotilla's interface flexibility."},{"cited_title":"Wolfrath, N","cited_arxiv_id":null,"evidence_quote":"Supplies HACCS, the heterogeneity-aware clustered client selection strategy implemented in Flotilla."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FedAT, the asynchronous tiered strategy whose cross-tier coordination exercises Flotilla's state model."},{"cited_title":"Foley, M","cited_arxiv_id":null,"evidence_quote":"Provides OpenFL, one of three frameworks in the Raspberry Pi resource-footprint comparison."}],"review_version":1}