{"id":"f3846db5-9722-4b08-a330-8cd95464c4a0","arxiv_id":"1908.08082","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dynamic resizing of Horovod-based ring architecture training jobs is feasible, and a simulator using measured run times cuts average job completion time by more than half under moderate contention.","lead":"This paper shows that deep learning jobs using ring-style GPU communication can be paused, checkpointed, and restarted with more GPUs at low cost, and uses that fact in a simulated scheduler. In moderate cluster contention, the simulated scheduler more than halves average job completion time compared with always using eight GPUs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The simulated scheduling gain is computed in-sample: every job is the same ResNet-110 run used to fit the speed and convergence models, so the claimed >2x improvement in Table 3 is not yet evidence about real workloads.","rationale":"The paper makes two separable claims: (1) checkpoint/restart is cheap and resizing speeds up the same job; (2) a scheduler using this can halve average job time. Claim (1) has direct measurements, though it would be stronger with final accuracy. Claim (2) is only a simulation with no out-of-sample validation. The reader's conditional verdict captures this. My stress-test agrees with that weakest assumption; no new objection beyond it. I would keep the verdict CONDITIONAL rather than reject, because the measured feasibility facts are genuine and the simulation is a plausible extension, but not yet evidence. The concrete test would settle whether the simulated gain survives a held-out workload.","tokens_in":6870,"tokens_out":7415,"duration_ms":77494,"concrete_test":"Release the simulator (or a faithful reimplementation) and run a leave-one-out / held-out experiment: fit Eq. 1 and Eq. 5 on the existing ResNet-110 runs plus at least one additional workload (e.g., a Transformer or ResNet-50 on a different dataset), then simulate the held-out workload's arrival process and recompute Table 3 with multiple arrival seeds. If the Precompute/Exploratory advantage over the best fixed strategy drops below the reported >2x in moderate contention, the headline claim is workload-specific rather than a general scheduler result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 7's transfer from measured runs to a simulated scheduler. The resource model f(w) (Eq. 5) and the convergence fit (Eq. 1) are fitted to the experimental ResNet-110/CIFAR-10 runs, and the simulator then predicts the convergence of arriving jobs from 'data from previously completed runs' with no held-out set, no second workload, and no published simulator. Since every simulated job is essentially a copy of the fitting workload, Table 3's comparison (e.g., Precompute 2.63 h vs Eight 6.20 h under moderate contention) may reflect the fitted curves rather than a general property of dynamic ring-architecture scheduling. The direct measurements (94.5% scaling efficiency, ~10 s checkpoint/restart) are legitimate, but they do not by themselves establish the more-than-halving claim. A second weakness is that Table 2 reports steps/epochs only, not final validation accuracy, so the Eq. 7 learning-rate rescaling is not verified to preserve model quality; if the resized runs finish at a worse optimum, 'faster completion' is not an apples-to-apples comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that dynamic (elastic) scheduling of distributed deep learning jobs is feasible for ring-allreduce architectures, not just parameter-server architectures. It derives a resource-to-speed model f(w) in Eq. (5) and a convergence-in-epochs model in Eq. (1), formulates resource allocation as an NP-hard integer program, and proposes a doubling heuristic. Experiments with Horovod/ResNet-110 on CIFAR-10 measure 94.5% scaling efficiency from 4 to 8 GPUs and roughly 10 seconds of checkpoint/restart overhead. Using those runs, a scheduler simulation in Section 7 reports more than a halving of average job completion time under moderate contention relative to fixed 8-GPU scheduling. The direct experimental measurements support feasibility; the headline scheduling gain rests on an in-sample simulation that has not been validated on held-out workloads.","tokens_in":7099,"tokens_out":6433,"duration_ms":62874,"significance":"If the claims held, the paper would provide a useful feasibility result for dynamic resizing of ring-architecture training jobs and a concrete scheduling heuristic. The direct measurements—94.5% scaling efficiency and about 10 seconds of restart overhead—are credible and valuable on their own. The proposed model and doubling heuristic are well motivated by the communication structure of all-reduce. However, the central quantitative claim (more than halving of average job time) currently comes from a simulator that is fitted and evaluated on the same ResNet-110/CIFAR-10 workload, with no held-out jobs, no prediction-error analysis, and no report of final model accuracy. The paper therefore establishes feasibility but does not yet establish the magnitude of the scheduling benefit for new workloads; it is a solid basis for a major revision rather than an accept.","major_comments":[{"comment":"The more-than-halving claim is computed in-sample. The simulation predicts each job's convergence 'based on data from previously completed runs,' and those runs are the same ResNet-110/CIFAR-10 jobs used to fit the convergence model in Eq. (1) and the resource model in Eq. (5). With no held-out workload, no second model/dataset, and no reported prediction error, the comparison between Precompute and Eight may simply reflect the fitted curves rather than a general property of ring-architecture scheduling. Please validate the simulator on held-out jobs or at least report cross-validated prediction errors and error bars on the Table 3 entries.","section":"§7, Table 3"},{"comment":"The resizing experiments do not report the quality of the converged model. Table 2 lists steps, epochs, and wall-clock time but no final validation accuracy or loss for the uninterrupted baselines or the checkpoint/resume runs. Since Eq. (7) changes the global batch size and learning rate at resume time, 'faster completion' is only an apples-to-apples comparison if the resumed runs reach the same final accuracy as the baselines. Please report the final validation accuracy for every row of Table 2, or explicitly state the convergence criterion used.","section":"§6, Table 2"},{"comment":"The doubling heuristic's motivating premise—that non-power-of-two allocations such as 9 GPUs have disproportionately worse per-GPU performance and can trap the Optimus greedy heuristic—is asserted but never measured in the paper. Table 1 profiles only 1, 2, 4, and 8 GPUs, and no experimental data are presented for w = 3, 5, 6, 7, etc. Consequently, the claimed advantage of the doubling heuristic over the Optimus greedy heuristic is not demonstrated by the evidence in this manuscript; at minimum, a sensitivity analysis or direct measurements for a few non-power-of-two configurations would be needed.","section":"§4.2, Eq. (6)"}],"minor_comments":[{"comment":"The text says the 5k-stop experiment saves 'close to 50 min (~32%)' and the 10k-stop experiment saves 'close to 36 min (~23%)', but Table 2 gives 126 min for the 4-GPU baseline and 104 and 113 min for the resized runs, i.e., differences of 22 and 13 minutes. Please reconcile these numbers or correct the computation.","section":"§6, Table 2"},{"comment":"The header of Table 2 contains spacing and column-label errors ('# GP U sinit stepsstop #GP U snew stepstot epochs Ttot'), which make the table harder to read; please reformat the column names.","section":"Table 2"},{"comment":"The sentence 'In all cases, we can fit a NNLS model for each value of w' is unclear: Eq. (5) is presented as a single functional form with coefficients θ0..θ3 learned from (w, f(w)) data points. Please clarify whether the θ coefficients are global or fitted separately for each w.","section":"§3.2, Eq. (5)"},{"comment":"The description of the exploratory strategy ('gives a new job 8 GPUs for the first ten minutes to run for 2.5 minutes at each of 1, 2, 4 and 8 GPUs') is ambiguous about whether these are sequential or parallel runs and about how the measured speeds are aggregated. Please clarify.","section":"§7"},{"comment":"The paper reports a single profiling run per configuration and no variance or confidence intervals for the scaling efficiency or for the ~10 second restart overhead; adding repeat measurements would strengthen the direct feasibility claim.","section":"§5, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a feasibility/system study rather than a fully validated scheduling result. The direct Horovod resizing measurements are the strongest contribution; the simulation study in Section 7 needs held-out validation and accuracy measurements before the headline improvement can be taken as established. Please consider whether the journal's standards require an end-to-end scheduler implementation or whether a carefully validated simulation is sufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:1908.08082. The paper's direct measurements are the real contribution: checkpointing and restarting a Horovod job from 4 to 8 GPUs costs about 10 seconds, scaling efficiency is 94.5%, and resizing saves 23–32% wall-clock time. Those numbers support the feasibility claim that dynamic scheduling of ring-architecture jobs is practical. The doubling heuristic is a sensible addition to Optimus, because it avoids getting stuck on bad intermediate allocations (8→9 vs 8→16). That part is worth reading.\n\nThe soft spot is the scheduler simulation. Table 3's 'more than halving' comes from a simulator that predicts convergence using the same fitted model (Eq. 5) and 'data from previously completed runs' — effectively every simulated job is a copy of the ResNet-110 runs used to fit the model. There's no held-out workload, no second model/dataset, no error bars, and no code released. So the >2x improvement is not yet evidence about real workloads; it's an in-sample demonstration that the fitted curves favor the precompute/exploratory strategies. The paper itself hints at this in Section 9, saying future work will use 'real workloads with a simulated schedule rather than a simulated schedule with simulated workloads.' That's an honest acknowledgment that the current simulation is not a measured result.\n\nSecond issue: Table 2 reports steps and total epochs but not final validation accuracy. The learning-rate rescaling of Eq. 7 follows Goyal et al., but without accuracy numbers we can't tell if a resized run converges to the same quality. If the 4→8 runs finish at a worse optimum, 'faster completion' isn't apples-to-apples.\n\nThese concerns are not fatal. The direct feasibility evidence stands on its own. The simulation is an engineering extrapolation, not a measured claim. For a cluster-scheduling audience, this is a useful extension of Optimus to ring architectures with some concrete numbers. I'd bring it to a reading group. A serious referee should engage with it, but the burden should be on the authors to validate the scheduler on held-out workloads or with a real implementation. My recommendation: accept with major revision, not desk reject.","headline":"Direct resizing measurements are credible, but the headline simulated scheduler gain is an in-sample artifact and should be read as a hypothesis.","tokens_in":7639,"tokens_out":2094,"would_cite":true,"duration_ms":19938,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ring-architecture deep learning jobs can be stopped, resized, and restarted cheaply enough that dynamic scheduling more than halves average job completion time in simulation.","keywords":["dynamic scheduling","ring all-reduce","GPU cluster","checkpoint and restart","doubling heuristic","resource allocation","distributed deep learning"],"falsifier":"Run the proposed scheduler on a real cluster with jobs whose speed and convergence were not used to fit the model, and compare predicted vs. actual completion times and final accuracies; the claim fails if, for instance, a ResNet-110 run resumed from a 4-GPU checkpoint onto 8 GPUs with the Eq. 7 learning-rate doubling reaches markedly lower test accuracy than an uninterrupted 8-GPU run, or if measured stop/restart overhead on a loaded cluster is far above the simulated 10 seconds.","tokens_in":6657,"feed_emoji":"⚙️","tokens_out":8401,"duration_ms":73734,"temperature":0.7,"pith_summary":"Deep learning jobs on GPU clusters are usually trained with a fixed number of GPUs, but clusters are shared and demand shifts. This paper tries to establish that ring-architecture training jobs—the all-reduce alternative to parameter servers—can be dynamically resized: stopped, checkpointed, and restarted with more or fewer GPUs without a meaningful penalty. To support this, it derives a resource-to-speed model for ring all-reduce algorithms, formulates the allocation problem as a non-convex, NP-hard integer program, and introduces a doubling heuristic that assigns workers in powers of two to escape local optima. Experiments on ResNet-110 with CIFAR-10 show stop-restart time near 10 seconds and 94.5% scaling efficiency from 4 to 8 GPUs; a scheduler simulation reports average job time more than halved under moderate contention. If correct, the same cluster can finish more training jobs in the same wall-clock time by shifting GPUs between jobs as contention changes.","feed_headline":"Resizing ring-jobs dynamically can halve training time","feed_subtitle":"Low stop-and-restart cost and 94.5% scaling efficiency let schedulers shift GPUs between jobs.","key_machinery":"The carrying mechanism is the resource-to-speed model of Eq. 5, $f(w) = (\\theta_0 (m/w) + \\theta_1 (w-1) + \\theta_2 (w-1)(n/w) + \\theta_3)^{-1}$, where $w$ is the number of workers, $m$ is the minibatch size per worker, $n$ is the model size, and the $\\theta$'s are non-negative coefficients learned for each job. This single functional form can represent ring, doubling-halving, and binary-block all-reduce algorithms with different coefficient behaviors. The scheduling problem is a non-convex, non-linear, NP-hard integer program minimizing the sum of job completion times subject to total GPU capacity; the doubling heuristic solves it by starting every job at one GPU and repeatedly adding $w_j$ workers to the job maximizing $(Q_j/f(w_j) - Q_j/f(2w_j))/w_j$, the average per-GPU reduction in remaining time. Doubling matters because increasing from 8 to 9 GPUs can look worse than increasing from 8 to 16 GPUs, so a unit-step greedy heuristic can get stuck in a local optimum. The learning-rate rescaling rule $lr_{new} = (w_{new}/w_{last}) \\cdot lr_{last}$ is what makes checkpointed resumes on more GPUs preserve convergence.","core_discovery":"The central discovery is twofold: ring-architecture jobs have low stop-and-restart cost and near-linear scaling, so dynamic resizing is practically viable; and a doubling-based heuristic can allocate GPUs to jobs in a way that respects the non-monotonic performance of ring all-reduce. The authors model time per minibatch as a combination of forward/backward propagation and all-reduce communication, fit a speed-per-worker function $f(w)$ to each job by non-negative least squares, and solve the scheduling problem by repeatedly giving a job enough extra workers to double its allocation, selecting the job with the largest average marginal time improvement per GPU. On a single 8-GPU node with ResNet-110 on CIFAR-10, resuming from a 4-GPU checkpoint onto 8 GPUs saved about 32% of training time when stopped at 51 epochs and about 23% when stopped at 102 epochs, with roughly 10 seconds of checkpoint/stop/restart overhead. In simulation on a 64-GPU cluster with Poisson job arrivals, the precompute scheduling strategy more than halved average job completion time relative to fixed 8-GPU allocations under moderate contention.","pith_inferences":["A next step the authors leave implicit: deploy the precompute scheduler on a live cluster and compare measured completion times to the simulated halving, which would also test whether the fitted speed model generalizes to new jobs.","The same doubling heuristic could in principle be applied to parameter-server jobs, since the speed model's functional form covers both architectures and the heuristic is what avoids local-optimum stalls.","The checkpoint-resume results suggest dynamic scheduling could double as preemption: suspend a low-priority job, admit a short high-priority job, then resume the suspended job on a different worker count with minimal penalty.","For models sensitive to learning-rate changes, the Eq. 7 rescaling rule may not preserve accuracy; if so, the scheduler would need a short warm-up or a different adaptation rule before reallocating GPUs."],"forward_implications":["On a 64-GPU cluster with Poisson arrivals and moderate contention, the precompute scheduler more than halves average job completion time compared with fixed 8-GPU allocations.","Rescaling a job from 4 to 8 GPUs at 51 epochs saves roughly 32% of wall-clock time; stopping at 102 epochs still saves about 23%, with stop/restart overhead around 10 seconds.","The doubling heuristic lets the scheduler reach 16-GPU allocations that a unit-increment greedy heuristic would miss, because the 8-to-9 step looks unprofitable while the 8-to-16 step is not.","Because placement is simpler in ring architectures (there are no parameter servers to place), the scheduling gains can be realized without adding placement complexity.","Exploratory scheduling—giving new jobs 8 GPUs briefly to sample speeds at 1, 2, 4, and 8 GPUs—performs well under moderate contention but poorly under extreme contention, where the 7.5 minutes of exploration is too costly."],"supporting_citations":[{"why":"Horovod is the ring-architecture framework on which the stop/restart and scaling experiments are run.","marker":"[10]"},{"why":"Doubling-halving and binary-block all-reduce algorithms define the communication cost structure the speed model encodes.","marker":"[9]"},{"why":"The parameter-server dynamic scheduler formulation and its NNLS resource fitting are the starting point this paper adapts to ring architectures.","marker":"[8]"},{"why":"Supplies the learning-rate scaling rule (Eq. 7) used when resuming a job on more GPUs.","marker":"[2]"},{"why":"ResNet-110 is the model used in the profiling and checkpoint/restart experiments.","marker":"[4]"},{"why":"Supplies the alpha/beta/gamma communication-cost coefficients used in Eqs. 2-4.","marker":"[11]"}],"fun_headline_variants":["Ring-job resizing can more than halve average completion time","Stop, restart, resize: ring jobs finish in half the time","Doubling GPU allocation halves training time for ring jobs","Dynamic ring scheduling cuts average job time by half","Horovod jobs adapt: stop, restart, and run twice as fast"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulation's promised halving of average job time assumes that the per-job speed and convergence models learned from previously completed runs predict new jobs of the same kind, and that resuming from a checkpoint with the doubled learning rate reaches the same model quality as uninterrupted training; neither assumption is tested on held-out jobs or measured accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Ring-job resizing can more than halve average completion time","Stop, restart, resize: ring jobs finish in half the time","Doubling GPU allocation halves training time for ring jobs","Dynamic ring scheduling cuts average job time by half","Horovod jobs adapt: stop, restart, and run twice as fast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1457,"prompt_tokens":957,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":573,"tokens_out":500,"duration_ms":5105,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:49:53.786249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed scheduler on a real cluster with jobs whose speed and convergence were not used to fit the model, and compare predicted vs. actual completion times and final accuracies; the claim fails if, for instance, a ResNet-110 run resumed from a 4-GPU checkpoint onto 8 GPUs with the Eq. 7 learning-rate doubling reaches markedly lower test accuracy than an uninterrupted 8-GPU run, or if measured stop/restart overhead on a loaded cluster is far above the simulated 10 seconds.","supporting_citations":[{"cited_title":"Rabenseifner","cited_arxiv_id":null,"evidence_quote":"Horovod is the ring-architecture framework on which the stop/restart and scaling experiments are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Doubling-halving and binary-block all-reduce algorithms define the communication cost structure the speed model encodes."},{"cited_title":"Open source high performance computing","cited_arxiv_id":null,"evidence_quote":"The parameter-server dynamic scheduler formulation and its NNLS resource fitting are the starting point this paper adapts to ring architectures."},{"cited_title":"Abadi, P","cited_arxiv_id":null,"evidence_quote":"Supplies the learning-rate scaling rule (Eq. 7) used when resuming a job on more GPUs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ResNet-110 is the model used in the profiling and checkpoint/restart experiments."}],"review_version":1}