{"id":"d1f0bc8d-47ce-4f00-b8b6-92619647d62b","arxiv_id":"2412.19991","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FLUDE combines dependability-aware device selection, local model caching, and stale-aware model distribution to make federated learning faster, more accurate, and more efficient when many devices are unreliable.","lead":"A new federated learning framework, FLUDE, keeps unreliable phones and edge devices in training by estimating which devices will finish, caching half-finished local models, and sending fresh global models only to the most stale devices. Tests on 120 real devices show higher accuracy, faster time-to-accuracy, and 24 to 41 percent lower communication cost than existing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run stochastic evaluation: several FLUDE margins are under one percentage point, so the claimed accuracy and speedup advantages are not yet statistically established.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports keeping that verdict. The strongest claim is empirical, so the empirical evidence must be robust. The manuscript reports only single runs of a stochastic pipeline: random undependability, random online/offline dynamics, random bandwidth, and random exploration. The reported differences are sometimes very small, making run-to-run variance a direct threat to the central quantitative claim. Meanwhile, the reader's weakest_assumption focuses on the generative model matching the estimator; that is a real generalization concern, but the immediate threat to the numbers in this paper is the absence of repeated trials. I therefore see the concern as related but not identical, hence 'partial.' The proposed seed-ensemble rerun would settle whether the headline margins are noise. No fabrication or internal inconsistency is apparent; the ablations are informative and the system design is coherent. The verdict should remain CONDITIONAL: the framework may well work, but the current statistical support is insufficient to accept the performance claims as settled.","tokens_in":20547,"tokens_out":6360,"duration_ms":72083,"concrete_test":"Rerun the complete comparison (four datasets, five algorithms) with at least 10 independent seeds, seeding separately the per-device undependability events, online/offline transitions, bandwidth fluctuations, and the exploration randomness in Algorithm 1. Report the mean and standard deviation (or, better, paired differences) for final accuracy/AUC, time-to-accuracy, and communication cost. If the FLUDE-versus-FedSEA margin on Google Speech (0.30%) or any other sub-percentage-point margin falls within one standard deviation of the paired difference, the claimed improvements cannot be considered reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is quantitative: FLUDE beats four baselines by 2.28%–7.43% accuracy, 1.2x–3.2x time-to-accuracy, and 23.71%–40.71% communication cost. But every reported number is a single draw from a highly stochastic process. In Section 5.2, per-device undependability is randomly realized each round, online/offline states flip every 10 minutes, WiFi bandwidth fluctuates between 1 and 30 Mb/s, and Algorithm 1 uses randomized epsilon-greedy exploration. Algorithm 2 additionally terminates each round when |S| x R uploads arrive, so round length and aggregation size are random. No seeds, confidence intervals, or repeated runs are reported for any table or figure. Several decisive margins are small: FLUDE exceeds FedSEA by only 0.30% final accuracy on Google Speech (Table 1) and by 1.29% on CIFAR-100 versus Oort, and Figure 4 presents single time-to-accuracy traces. If run-to-run variance from device failures, offline transitions, bandwidth, and exploration is comparable to these margins, then the claimed ordering of methods is not established. This is load-bearing because the paper's headline conclusion is precisely a quantitative improvement under stochastic undependability; the design may still be sound, but the evidence as reported does not separate signal from noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLUDE, a federated learning framework designed for environments in which devices are undependable, i.e., they may drop out during local training or be offline at round start. The framework has three main mechanisms: (1) an adaptive device selector that estimates each device's dependability with a Beta-Bernoulli Bayesian model, balances participation frequency with a penalty term, and uses epsilon-greedy exploration; (2) a local model caching mechanism that preserves training progress on devices that fail mid-round; and (3) a staleness-aware model distribution strategy that decides whether a selected device receives the latest global model or resumes from its cached model. The authors implement FLUDE on 40 OPPO smartphones and 80 NVIDIA Jetson devices and evaluate it on CIFAR-10, CIFAR-100, Google Speech, and Avazu against Oort, AsyncFedED, SAFA, and FedSEA, reporting 2.28%–7.43% higher final accuracy, 1.2x–3.2x time-to-accuracy improvements, and 23.71%–40.71% communication-cost reductions. The paper also includes ablation studies of the selector and the model distributor, and robustness experiments varying offline and undependability rates.","tokens_in":20690,"tokens_out":4409,"duration_ms":45506,"significance":"If the reported results hold, FLUDE would be a practically valuable system for federated learning on unreliable mobile and edge fleets. The physical testbed with 120 heterogeneous devices, the four diverse tasks, and the component-wise ablations are genuine strengths: they test the framework under realistic bandwidth dynamics and device heterogeneity rather than in a pure simulator. The component ablations in Section 5.4 give some confidence that each mechanism contributes, and the robustness analysis in Section 5.5 directly targets the paper's core claim. However, the central quantitative claims are currently supported by single runs of a highly stochastic process, and the evaluation's failure model coincides with the estimator's generative assumption. These issues are load-bearing because the paper's headline contribution is precisely a quantitative improvement in accuracy, speed, and communication cost under stochastic undependability.","major_comments":[{"comment":"The headline quantitative claims rest on a single run of a stochastic process. In the setup of Section 5.2, each device's undependability is randomly realized each round, online/offline states flip every 10 minutes, WiFi bandwidth fluctuates between 1 and 30 Mb/s, Algorithm 1 uses randomized epsilon-greedy exploration, and Algorithm 2 terminates rounds at a random number of received uploads. Yet no seeds, confidence intervals, or repeated runs are reported for any table or figure. Several margins are small: FLUDE exceeds FedSEA by only 0.30 percentage points on Google Speech and by 1.29 percentage points over Oort on CIFAR-100 in Table 1. If run-to-run variance is comparable to these margins, the claimed ordering of methods is not statistically established. This is load-bearing because the abstract and introduction claim specific quantitative improvements. Please report multiple independent runs per configuration with means and variances (or a similar statistical summary), and where the claim is an ordering, include a significance test or state why the process variance is negligible.","section":"Section 5.2, Table 1, Figure 4"},{"comment":"The evaluation generates device failures from a model that matches the estimator's assumptions. In Section 5.2, each device is assigned a fixed undependability rate drawn from a normal distribution, and each round the device fails with that fixed probability. This is exactly the fixed-probability Beta-Bernoulli generative model assumed by the dependability estimator in Section 4.1, Eq. (1). The estimator is therefore tested on its own model class. The robustness claim in Section 5.5 would be considerably stronger if the failure process were made non-stationary (e.g., dependability rates drifting over time) or correlated across devices (e.g., regional network outages), and if the sensitivity of Algorithm 2's round termination rule, which uses predicted uploads based on the estimated dependability, were measured under such mismatch.","section":"Section 5.2 vs. Section 4.1, Eq. (1)"},{"comment":"Several key hyperparameters that directly control the core mechanisms are hand-set, and no sensitivity analysis is reported. These include the participation-frequency penalty sigma=0.5, the staleness-adjustment coefficients lambda=1 and mu=0.5 in Eq. (4), the exploration-factor schedule, and the Beta(2,2) prior. The device selector and the staleness-aware distributor depend on these values, so it is currently unknown whether the reported gains are robust to reasonable parameter changes. Please add a sensitivity study over these parameters, or provide a theoretical argument that the framework's performance is insensitive to their values.","section":"Section 5.2, Eq. (4), Algorithm 1"}],"minor_comments":[{"comment":"FLUDE's own CIFAR-10 time-to-accuracy is 2.68 hours in Table 1 but 2.39 hours in Table 2 for the same final accuracy of 87.52%; please reconcile the numbers or state whether the stopping criterion differs between the two tables.","section":"Section 5.3, Tables 1 and 2"},{"comment":"The text in Section 5.3 reports that FLUDE reduces communication cost by 23.71%–49.7%, while the abstract and the introduction state 23.71%–40.71%; these ranges need to be reconciled.","section":"Section 5.3 vs. Abstract/Introduction"},{"comment":"The robustness study in Section 5.5 uses a normal distribution with variance 0.05, whereas Section 5.2 sets the variance to 0.04; please clarify whether this difference is intentional.","section":"Section 5.5 vs. Section 5.2"},{"comment":"In Eq. (1), s and f are described as the number of successes and failures, but it is not stated whether they are counts since the previous update or cumulative counts; if cumulative, repeated updates would double-count observations, so please state explicitly that they are the counts since the last update.","section":"Section 4.1, Eq. (1)"},{"comment":"The staleness-threshold update in Eq. (4) divides by H_old and N_old, but no handling is specified for initial rounds in which H_old or N_old is zero; please add a boundary condition.","section":"Section 4.3, Eq. (4)"},{"comment":"The while loop for budget adjustment has no explicit termination guarantee if the predicted cost B_pred does not fall below B_max after an iteration; please add a bound on the number of iterations or a fallback when the budget cannot be met.","section":"Algorithm 2, Lines 6-11"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the single-run stochastic evaluation is the main gating issue and is addressable within the scope of the manuscript by adding repeated runs with statistical summaries in the same testbed. I do not see a load-bearing error that would force rejection. The paper's claim to be 'the first' framework addressing device undependability in practical systems is not systematically supported by a comparison with prior work on unreliable or unavailable devices, but I did not make that a major comment because the technical contribution stands independently of that claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: FLUDE is a sensible integration of known ideas — dependability-aware selection, participation-frequency penalty, local model caching, and staleness-aware distribution — and the paper gives each component an ablation. That's real work, and the framing around undependable devices at scale is a real bottleneck that the cited baselines don't directly attack.\n\nWhat's new: treating dependability as a Beta-Bernoulli posterior per device and using it to pick participants, cap participation, and decide when to redistribute the global model. That combination is not in Oort, SAFA, FedSEA, or AsyncFedED. The hardware evaluation on 120 devices across four tasks is a genuine strength, and the ablation tables show each module contributes. The math is straightforward Bayes plus a threshold-adjustment rule; I see no red flags there. The citation pattern looks appropriate, though the claim to be the \"first\" framework for undependable devices is broader than the related work supports.\n\nThe soft spots are real but concentrated in the evaluation statistics. Every number in Table 1 and Figures 4 and 5 is a single draw from a highly stochastic process: per-device failure probabilities realized each round, online/offline flips every ten minutes, WiFi bandwidth fluctuating between 1 and 30 Mb/s, epsilon-greedy exploration, and a round-termination rule based on random arrivals. No seeds, no repeated runs, no error bars. Some claimed margins are small — 0.30% over FedSEA on Google Speech, 1.29% over Oort on CIFAR-100. Run-to-run variance from this setup could easily be that large. So the ordering of methods is not statistically established yet, even though the design is plausible.\n\nSecond soft spot: the simulator generates failures from fixed per-device probabilities, which is exactly the model FLUDE's Beta-Bernoulli estimator assumes. That is mild circularity, not fatal — the mechanism is being tested on its own generative assumption. But the paper should acknowledge this explicitly and ideally add a time-varying or correlated failure model (e.g., regional outages) to show the design does not depend on that match. Also, no artifact is released, which limits independent verification.\n\nWho this is for: people building production FL systems with flaky phone fleets. The architecture decisions are worth borrowing. A serious referee should ask for repeated trials, error bars, and an artifact before the quantitative claims are cited as established.","headline":"Useful systems paper with a plausible design and real hardware, but the headline numbers are single draws from a stochastic process and need repeated trials before the ordering over baselines is settled.","tokens_in":21349,"tokens_out":1925,"would_cite":true,"duration_ms":20483,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLUDE keeps federated learning accurate and fast when many devices drop out mid-training.","keywords":["federated learning","device undependability","participant selection","model caching","staleness-aware distribution","resource efficiency","non-IID data","mobile edge training"],"falsifier":"Run FLUDE in a scenario where undependability is caused by a shared external event, such as a regional network outage that knocks out many devices at once, and compare its time-to-accuracy and final accuracy against a random-selection baseline; if the predicted number of completed uploads $|S| \\times R$ systematically overshoots actual arrivals and FLUDE's advantage over Oort disappears, the robustness claim fails.","tokens_in":20230,"feed_emoji":"📱","tokens_out":4692,"duration_ms":43617,"temperature":0.7,"pith_summary":"This paper tries to show that federated learning can stay accurate, fast, and cheap even when a large share of participant devices repeatedly drop out mid-training. It proposes FLUDE, a framework that predicts each device's probability of finishing a training round from its history, selects dependable devices while balancing how often each device is chosen, keeps interrupted training progress in a local cache, and sends the fresh global model only to devices whose cached model is too stale. On four tasks with smartphone and edge-device hardware, the paper reports 2.28%-7.43% higher final accuracy, 1.2x-3.2x faster time-to-accuracy, and 23.71%-40.71% lower communication cost than four existing FL systems. If the results hold, FLUDE would make FL practical in exactly the environments where it is now considered unreliable, and would let model training use devices with useful data that current systems exclude.","feed_headline":"FLUDE keeps federated learning accurate when devices drop out","feed_subtitle":"Reports up to 7.4% higher accuracy, 3.2x faster convergence, and 40% less communication cost.","key_machinery":"The load-bearing object is a per-device dependability score $R(i)$, estimated as the expected value of a Beta distribution $\\mathrm{Beta}(\\alpha, \\beta)$ updated by Bayesian counting of completed versus interrupted training attempts. This score feeds a selection priority $P(i) = R(i) \\times (Q/q_i)^{\\mathbb{1}(Q < q_i)} \\times \\sigma$, which applies a participation-frequency penalty, and a multi-armed-bandit exploration term keeps unobserved devices in the running. A local model cache preserves the latest training state so interrupted devices resume from a checkpoint, and a staleness-aware distributor uses an adaptive threshold $W_{\\text{new}}$ to decide which cached models are too stale and must be refreshed with the global model. Round termination uses the sum over selected devices of predicted dependability, $|S| \\times R$, as the expected number of uploads, so the server stops waiting when the likely useful work is done.","core_discovery":"The paper's central claim is that device undependability should be treated as a measurable, learnable property rather than a reason to exclude devices. FLUDE estimates each device's dependability as the expected value of a Beta distribution over success and failure counts, re-prioritizes devices so that high-dependability devices are preferred yet no device participates too often, explores unfamiliar devices with a bandit-style fraction of selections, caches local training state so an interrupted device resumes instead of restarting, and adaptively chooses which rejoining devices must download the latest global model based on a moving staleness threshold. The paper reports that this combination, implemented on 120 physical devices across image classification, speech recognition, and click-through-rate prediction, beats four synchronous and asynchronous FL baselines in final accuracy, wall-clock time to target accuracy, and communication cost.","pith_inferences":["The dependability score could double as a fairness or incentive signal: devices that prove reliable earn more selection, while operators could compensate less-dependable devices differently, something the paper does not discuss.","The staleness-threshold adjuster could transfer to asynchronous FL protocols that currently use fixed staleness cutoffs; this paper only tests it in its own synchronous-with-deadline scheme.","Because the evaluation generates failures from fixed per-device probabilities matching the estimator's generative model, the reported gains may shrink under correlated failures such as regional outages; a test with such correlations would be the natural next experiment.","The Beta prior's initial parameters are a free input, and the paper suggests setting them from manufacturer statistics, so a practitioner could tune them from fleet data, making part of the benefit depend on prior quality."],"forward_implications":["FL systems can keep training in environments where roughly 30-40% of devices are undependable without excluding them, recovering accuracy lost to dropped updates.","Wall-clock time to a target accuracy falls by 1.2x-3.2x, so fixed training budgets in hours buy more accuracy.","Communication cost to reach a target accuracy falls by roughly a quarter to two-fifths, mainly because cached checkpoints remove repeated global-model downloads.","Balancing participation frequency reduces bias toward frequently available devices, so the global model generalizes better across classes and devices with little participation.","The same caching mechanism also reduces wasted computation from restarting local training from scratch after interruptions."],"supporting_citations":[{"why":"Supplies the production-system evidence that many devices fail to meet training conditions, motivating the undependability problem.","marker":"[9]"},{"why":"Defines FedAvg, the base federated aggregation protocol that FLUDE extends with selection, caching, and distribution.","marker":"[10]"},{"why":"Oort, the guided-participant-selection baseline that FLUDE compares against and must beat.","marker":"[12]"},{"why":"SAFA, the semi-asynchronous baseline whose idle-waiting limitation FLUDE's round termination targets.","marker":"[11]"},{"why":"FedSEA, the semi-asynchronous baseline with training-time balancing that FLUDE compares against.","marker":"[15]"},{"why":"AsyncFedED, the asynchronous baseline with staleness-weighted aggregation that FLUDE compares against.","marker":"[16]"},{"why":"Supplies the Beta distribution used to model the prior over device dependability.","marker":"[25]"},{"why":"Provides the Bayesian theorem used to update dependability estimates from success and failure counts.","marker":"[26]"},{"why":"Supplies the multi-armed bandit formulation and exploration factor for the device selection strategy.","marker":"[27]"}],"fun_headline_variants":["FLUDE turns undependable devices into FL assets","FLUDE: Federated learning that survives flaky devices","FLUDE boosts accuracy and cuts costs with unreliable devices","Adaptive FL framework FLUDE handles undependable devices at scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes each device has a fixed probability of finishing a round and that this probability can be learned from past successes and failures; if failures are correlated across devices or shift unexpectedly over time, the dependability estimates, the predicted upload count, and the round-termination rule can all be wrong.","fun_headline_variants_meta":{"raw":{"variants":["FLUDE turns undependable devices into FL assets","FLUDE: Federated learning that survives flaky devices","FLUDE boosts accuracy and cuts costs with unreliable devices","Adaptive FL framework FLUDE handles undependable devices at scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2989,"prompt_tokens":933,"completion_tokens":2056,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1989}},"tokens_in":549,"tokens_out":2056,"duration_ms":15214,"temperature":1.0,"reasoning_tokens":1989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:41:09.997821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FLUDE in a scenario where undependability is caused by a shared external event, such as a regional network outage that knocks out many devices at once, and compare its time-to-accuracy and final accuracy against a random-selection baseline; if the predicted number of completed uploads $|S| \\times R$ systematically overshoots actual arrivals and FLUDE's advantage over Oort disappears, the robustness claim fails.","supporting_citations":[{"cited_title":"Towards federated learning at scale: System design","cited_arxiv_id":null,"evidence_quote":"Supplies the production-system evidence that many devices fail to meet training conditions, motivating the undependability problem."},{"cited_title":"Oort: Efficient federated learning via guided participant selection","cited_arxiv_id":null,"evidence_quote":"Oort, the guided-participant-selection baseline that FLUDE compares against and must beat."},{"cited_title":"Safa: A semi-asynchronous protocol for fast federated learning with low overhead.IEEE Transactions on Computers, 70(5):655– 668, 2020","cited_arxiv_id":null,"evidence_quote":"SAFA, the semi-asynchronous baseline whose idle-waiting limitation FLUDE's round termination targets."},{"cited_title":"Fedsea: A semi-asynchronous federated learning framework for extremely heterogeneous devices","cited_arxiv_id":null,"evidence_quote":"FedSEA, the semi-asynchronous baseline with training-time balancing that FLUDE compares against."},{"cited_title":"Multitude of beta distributions with applications","cited_arxiv_id":null,"evidence_quote":"Supplies the Beta distribution used to model the prior over device dependability."}],"review_version":1}