{"id":"bb805851-93ca-477e-8937-e70b9d294c3a","arxiv_id":"2505.21727","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FedCostAware reports cutting cloud costs for synchronous federated learning by up to 72 percent using smarter spot-instance lifecycle management.","lead":"FedCostAware is a scheduler that cuts cloud costs for synchronous federated learning by turning off spot instances while clients wait and restarting them before the next training round. If the measured savings hold, budget-constrained hospitals and labs could run federated training on cheaper spot hardware without switching to less accurate asynchronous methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cost-savings claim is only tested in a preemption-free window: no spot preemptions occurred, so the checkpoint-recovery and schedule-adjustment paths that justify fault tolerance were never exercised, and volatility could erase the measured savings.","rationale":"The reader's conditional verdict already identifies the same load-bearing weakness: the experiments observed no spot preemptions and the scheduler's cost advantage depends on predictable spin-up and termination/prewarming behavior. My stress-test concurs and sharpens the point: because the paper explicitly claims fault tolerance as a key contribution, the absence of any preemption means that entire code path is untested. A single six-hour window with no preemptions does not establish behavior under realistic spot volatility, and preemptions are the mechanism that could turn the measured savings into net losses due to round delays and forced idle time on surviving instances. I do not think this requires moving the verdict from CONDITIONAL to REJECT; the central idea is plausible and the measured savings in a stable environment are internally consistent. It does mean the acceptance condition should include a preemption-stress experiment and repeated trials, exactly the kind of condition the reader already set. No ad hominem is intended; this is a gap in experimental coverage, not a claim of misconduct.","tokens_in":11404,"tokens_out":5400,"duration_ms":61204,"concrete_test":"Run a preemption-injection ablation on Fed-ISIC2019: for each of, say, 10 seeds, randomly preempt one client's spot instance in a random subset of rounds (frequencies 0, 1, 3, and 6 preemptions over 20 rounds), and compare FedCostAware vs static spot and on-demand on total cost and wall-clock time. Also launch 20 identical g5.xlarge instances to measure the spin-up distribution and feed that distribution into a bootstrap simulation of Listing 1. If the mean additional saving over static spot drops below, say, 10 percentage points or becomes negative at realistic preemption rates, the headline 'significantly reduces costs' must be scoped to preemption-free or low-preemption conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical claim about reducing costs on spot instances, but the evaluation never exercises the defining failure mode of spot instances. Section IV-B states 'No spot preemptions occurred during our experiments, even in long-running sessions exceeding six hours,' and all reported numbers in Table I are single runs without error bars. The scheduler's extra saving over a static spot cluster comes from terminating fast clients during the synchronous wait and prewarming them before the next round (Listing 1). That saving is real only if the spin-up estimate is accurate and no preemption disrupts the round. A preemption of the slowest client, or of a prewarmed fast client after restart, triggers the checkpoint/recovery path, delays the round, and may force already-running clients to idle; if this happens often, the measured additional saving over spot (about 24%) and the 60-72% saving over on-demand can shrink or vanish. Because the fault-tolerant and dynamic-schedule-adjustment mechanisms claimed as key contributions were never exercised, the central claim is currently established only for a preemption-free environment, not for the volatile spot market the paper targets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedCostAware, a scheduler for synchronous federated learning that runs client training on cloud spot instances and reduces cost by terminating instances during idle periods caused by stragglers, then pre-warming them before the next round based on estimated spin-up times. The system also performs checkpointing for fault tolerance, dynamically adjusts schedules after preemptions, and enforces per-client budgets. Experiments on Fed-ISIC2019, AI-READI, CIFAR-10, and MNIST report cost savings of 60-72% over on-demand instances and roughly 25% over a static spot-instance baseline, while preserving the synchronous aggregation protocol.","tokens_in":11625,"tokens_out":3213,"duration_ms":35476,"significance":"If the central claim holds, FedCostAware addresses a real operational cost in cross-silo federated learning: idle GPU time during synchronous rounds. The natural-partition experiments on Fed-ISIC2019 and AI-READI are a genuine strength, since they do not rely on artificially injected delays. The cost measurements are taken directly from billed instance time, so the reported savings are not an artifact of a fitted parameter or a normalization convention. The paper is also honest about the absence of preemptions in its experiments. However, the evaluation is incomplete for a system whose main selling point is cost-effective use of volatile spot instances, because the fault-tolerance and dynamic-schedule-adjustment paths are never exercised, and the paper provides no accuracy results or repeated trials.","major_comments":[{"comment":"The central empirical claim is tested only in a preemption-free window. Section IV-B states that no spot preemptions occurred, even in sessions exceeding six hours, so the checkpoint-recovery and dynamic schedule adjustment mechanisms described in Section III-D were never exercised. Since spot preemption is the defining failure mode of the target resource class, the paper needs fault-injection experiments in which instances are terminated mid-round at controlled rates; the evaluation should report recovery overhead, round-delay impact, and total cost under varying preemption frequencies. Without such experiments, the claim that FedCostAware 'significantly reduces cloud computing costs' is established only for a benign spot market, not for the volatile conditions the system is designed to handle.","section":"Section IV-B and Section III-D"},{"comment":"All cost numbers in Table I come from single runs without error bars or repeated trials. Spot prices, boot times, and network conditions vary over time and across regions, so a single measurement cannot support the claimed savings percentages as stable quantities. The paper should report the mean and spread over multiple runs, and ideally over more than one AWS region or availability zone, to support the generality of the cost-savings claim.","section":"Section IV-A and Table I"},{"comment":"The paper reports only training time and cost, never model accuracy or convergence. Because FedCostAware keeps the synchronous aggregation protocol, one expects accuracy to match a standard synchronous baseline, but the paper does not demonstrate this. A practical cost-saving system must show that the savings do not degrade the trained model; the authors should add final accuracy or loss per dataset for FedCostAware, the static spot baseline, and the on-demand baseline.","section":"Section IV-B and Table I"},{"comment":"The headline savings of 72.22% come from CIFAR-10, where per-epoch times are artificially scaled by client-specific factors, while the most realistic datasets show lower savings (70.47% and 67.18%). The paper does not state how these scaling factors were chosen or whether they were selected to create a particular straggler spread. Since the magnitude of idle-time savings depends directly on the spread of client finishing times, the authors should report the scaling factors, justify them, and present a sensitivity analysis showing how savings vary with the degree of heterogeneity.","section":"Section IV-A and Table I"}],"minor_comments":[{"comment":"The notation is inconsistent: the text uses T_epoch cold and T_epoch warm, while Listing 1 uses T_epoch_cold and T_epoch_warm; also F_s is used both for the estimated slowest finish time inside the function and for the slowest client's finish time in the surrounding text. Please unify the notation.","section":"Listing 1 and Section III-B"},{"comment":"The 'Savings (%)' column does not specify the baseline for each row. For the two Sync FL rows the savings are relative to on-demand, and for the FedCostAware row they are also relative to on-demand, but the column header alone is ambiguous. Please add a footnote or clarify the comparison baseline.","section":"Table I"},{"comment":"The abstract states 'up to a 70% reduction' while the conclusion and Table I report 72.22%; this inconsistency should be corrected.","section":"Abstract and Section V"},{"comment":"The sentence about P4 ($30/hr) and P5 ($100/hr) GPUs is speculative; the paper does not test these instance types. It would be better phrased as an expectation rather than a demonstrated scaling result.","section":"Section IV-B"},{"comment":"The phrase 'all datasets remain within the secure cloud accounts of the respective clients' is slightly ambiguous given that the system provisions instances across providers and accounts; the subsequent explanation via S3 presigned URLs for model updates should be integrated more clearly.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems contribution with a plausible and directly measured cost-saving mechanism, and the natural-partition results are a solid starting point. The main risk is that the evaluation does not exercise the fault-tolerance path at all, which is a substantial gap for a spot-instance scheduler. The requested fault-injection experiments and accuracy comparisons are within the scope of a revision and would materially strengthen the paper. There is no evidence of citation manipulation; the use of APPFL and FedCompass from the same group is disclosed and the central cost claim does not depend on those citations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: FedCostAware is a system paper, not an ML method paper. It keeps synchronous FL intact and attacks the cost of idle GPU time by terminating spot instances of fast clients and prewarming them before the next round. That combination — idle-time termination, prewarming, and budget-aware exclusion — is genuinely new relative to Oort, HACCS, FedCompass, and spot-DNN-style work. The cost table is internally consistent and the savings over on-demand (60–72%) are plausible from billed instance time, especially on Fed-ISIC2019 and AI-READI, where no artificial scaling was used.\n\nSoft spots: single runs without error bars or repeated trials. No accuracy comparison against baselines, so we don't know whether the dynamic instance lifecycle affects convergence. No spot preemptions occurred, so the checkpoint recovery and dynamic schedule adjustment — the parts that justify spot instance use — are never exercised. The stress-test concern is real: a preemption of the slowest client, or of a prewarmed fast client, could delay the round and shrink the measured savings. But that is a limitation of the evaluation, not a flaw in the design; the system has mechanisms for it, they're just untested. MNIST and CIFAR rely on client-specific scaling factors to simulate stragglers, which is fine for exercising the scheduler but reduces the weight of those numbers. The spin-up time prediction is central and only tested via EMA; the reported at-most-3-min extra delay is decent but not a stress test. Multi-cloud and region selection are claimed but not demonstrated — the experiments are all AWS.\n\nOverall, the core scheduler idea is sound, the evaluation is honest (it explicitly notes no preemptions), and the cost arithmetic checks out. The paper would be substantially stronger with code artifacts, repeated trials, accuracy parity, and injected preemptions.\n\nWho this is for: people building cloud FL deployments with tight budgets, especially biomedical consortia. A serious referee should engage. It deserves peer review, not a desk reject. I would cite it in a systems-oriented FL paper. Not a conceptual breakthrough, but a useful and defensible engineering contribution. My recommendation: engage, with requests for artifacts and preemption injection.","headline":"A cost-aware spot-instance scheduler for synchronous FL that delivers real savings in a benign cloud window, though the fault-tolerance machinery that justifies spot use is never exercised.","tokens_in":12145,"tokens_out":1555,"would_cite":true,"duration_ms":15516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedCostAware turns the idle wait in synchronous federated learning into a cost-saving strategy on cloud spot instances, cutting GPU bills by up to 72% compared with on-demand instances.","keywords":["federated learning","cloud computing","spot instances","cost optimization","scheduling","straggler problem","synchronous FL","instance lifecycle management"],"falsifier":"Run the same federated workload on spot instances while deliberately doubling the observed spin-up times or forcing preemptions in the middle of a round; if the wall-clock time per round grows or the total cost exceeds the always-on spot baseline, the central claim that lifecycle management reliably reduces cost would be refuted.","tokens_in":11220,"feed_emoji":"💸","tokens_out":8488,"duration_ms":72653,"temperature":0.7,"pith_summary":"The paper claims that the idle time inherent to synchronous federated learning—fast clients waiting on stragglers—can be converted into direct cost savings on cloud spot instances. It introduces FedCostAware, a scheduler that terminates a client's spot instance as soon as its local training finishes and pre-warms the instance just before the next round is expected to start, using per-client estimates of epoch time and spin-up time. The authors report that this reduces federated training cost by roughly 60% compared to always-on spot instances and up to 72.22% compared to on-demand instances across four datasets. The significance is that synchronous FL, which is preferred for model accuracy and stability, can become affordable on volatile, discount-priced cloud capacity instead of requiring expensive on-demand GPUs.","feed_headline":"Terminating idle spot instances cuts federated learning costs 72%","feed_subtitle":"Powering down spot instances during straggler waits and pre-warming them before each round saves up to 72% on GPU costs.","key_machinery":"The termination-and-prewarm scheduler, driven by a calibration phase and exponential moving average updates, is the central mechanism. During the first two rounds the system measures each client's cold-start epoch time, warm-start epoch time, and instance spin-up time. Thereafter, whenever a client finishes early, the scheduler estimates the round's slowest finish time, computes the idle window, and if the idle window exceeds the spin-up time plus a threshold, it terminates the instance and schedules a pre-warm at the estimated slowest finish minus spin-up minus a buffer. The same estimates are updated online with an exponential moving average, and a separate re-estimation step adjusts the pre-warm schedule when a spot preemption forces a crashed client to recover from a checkpoint.","core_discovery":"The central claim is that a cost-aware scheduler can make synchronous federated learning on spot instances cheaper than both conventional spot usage and on-demand usage while leaving the aggregation protocol unchanged. The mechanism is to treat the wait for the slowest client as billable idle time and eliminate it: each client that finishes early has its instance terminated, and a pre-warming queue restarts it at a time computed from the estimated slowest finish time minus the estimated spin-up time, with a buffer to avoid delaying the next round. The paper demonstrates on MNIST, CIFAR-10, AI-READI, and Fed-ISIC2019 that this lifecycle management yields total-cost reductions of 67–72% relative to on-demand instances and about 6–11 percentage points of additional savings relative to naive spot usage, with negligible added wall-clock delay (at most about 3 minutes per experiment from spin-up estimation error).","pith_inferences":["The paper's experiments saw no spot preemptions, so the dynamic schedule adjustment and checkpoint recovery are untested under real interruptions; a natural next step is to force preemptions during training and measure whether cost savings survive the recovery overhead.","The same terminate-and-prewarm pattern applies beyond federated learning to any synchronous distributed job with stragglers, such as data-parallel deep learning with heterogeneous workers, so long as per-worker completion times can be predicted.","The fixed threshold and EMA estimates could be replaced by a learned predictor of boot time and spot price, potentially recovering savings in settings where spin-up time is volatile or spot prices fluctuate within a round.","If cloud providers move to finer-grained billing, the idle periods being eliminated may already be cheap, which would reduce the advantage of termination over simply idling; the scheduler's benefit is largest under per-hour billing, which is the current norm."],"forward_implications":["If the reported savings hold, synchronous federated learning can be deployed on spot instances at roughly 30–40% the cost of on-demand deployment, making GPU-based FL feasible for budget-constrained institutions such as hospitals.","The approach preserves the synchronous aggregation protocol, so existing convergence guarantees and model quality of synchronous FL are retained rather than traded away for cost.","Because the savings scale linearly with the hourly GPU price, the same scheduler would yield proportionally larger dollar savings on high-end accelerators ($30–$100 per hour).","Client-defined budgets become enforceable: a client whose remaining budget cannot cover the next round is automatically excluded, so no participant exceeds its financial limit.","Checkpoint-and-resume makes long FL runs resilient to spot preemptions, provided the pre-warming schedule is updated to the crashed client's new estimated finish time."],"supporting_citations":[{"why":"Supplies the cluster-management layer that the scheduler extends with custom node termination and pre-warming logic.","marker":"[27]"},{"why":"The federated learning framework into which FedCostAware is integrated, providing the training and aggregation loop.","marker":"[30]"},{"why":"The updated version of that framework that contributes checkpointing and extensibility hooks for the scheduler.","marker":"[31]"},{"why":"Provides the naturally partitioned Fed-ISIC2019 medical imaging benchmark used to measure real-world savings.","marker":"[35]"},{"why":"Defines the straggler and heterogeneity problem that motivates converting idle time into cost savings.","marker":"[8]"},{"why":"Provides the dual Dirichlet partitioning method and a compute-power-aware scheduling precedent that FedCostAware extends.","marker":"[11]"},{"why":"Supplies one of the synthetic image datasets (MNIST) used in the cost experiments.","marker":"[32]"},{"why":"Supplies another synthetic image dataset (CIFAR-10) used in the cost experiments.","marker":"[33]"}],"fun_headline_variants":["Cost-aware scheduler for federated learning cuts cloud costs 72%","Terminate idle spot instances to slash federated learning costs 72%","Smart lifecycle of spot instances slashes FL cloud costs by 72%","FedCostAware kills idle spot instances to cut federated learning costs 72%","Managing spot instance lifecycles cuts FL costs by 72%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost savings depend on the assumption that instance spin-up time can be predicted accurately enough that terminating and restarting an instance is cheaper than keeping it idle; if boot times are volatile or spot instances are unavailable when pre-warming starts, synchronous rounds will stall and the savings vanish.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware scheduler for federated learning cuts cloud costs 72%","Terminate idle spot instances to slash federated learning costs 72%","Smart lifecycle of spot instances slashes FL cloud costs by 72%","FedCostAware kills idle spot instances to cut federated learning costs 72%","Managing spot instance lifecycles cuts FL costs by 72%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001158,"raw_usage":{"total_tokens":4780,"prompt_tokens":910,"completion_tokens":3870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":3774}},"tokens_in":526,"tokens_out":3870,"duration_ms":29373,"temperature":1.0,"reasoning_tokens":3774,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:23:38.157367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same federated workload on spot instances while deliberately doubling the observed spin-up times or forcing preemptions in the middle of a round; if the wall-clock time per round grows or the total cost exceeds the always-on spot baseline, the central claim that lifecycle management reliably reduces cost would be refuted.","supporting_citations":[{"cited_title":"APPFL: open-source software framework for privacy-preserving federated learning,","cited_arxiv_id":null,"evidence_quote":"The federated learning framework into which FedCostAware is integrated, providing the training and aggregation loop."},{"cited_title":"Flamby: Datasets and benchmarks for cross-silo federated learning in realistic healthcare settings,","cited_arxiv_id":null,"evidence_quote":"Provides the naturally partitioned Fed-ISIC2019 medical imaging benchmark used to measure real-world savings."},{"cited_title":"FedCompass: Efficient cross-silo federated learning on heterogeneous client devices using a computing power-aware scheduler,","cited_arxiv_id":null,"evidence_quote":"Provides the dual Dirichlet partitioning method and a compute-power-aware scheduling precedent that FedCostAware extends."},{"cited_title":"The mnist database of handwritten digits,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the synthetic image datasets (MNIST) used in the cost experiments."}],"review_version":1}