{"id":"50d734ae-6920-41cd-8b20-74cc5478907d","arxiv_id":"2506.04531","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hierarchical asynchronous local SGD method with regional parameter servers and global model merging is claimed to train small LLMs up to 7.5x faster than DiLoCo in simulated geo-distributed settings.","lead":"HALoS is a training method for language models whose compute is spread across separate data centers. It adds a local coordinator per region that collects updates and only occasionally talks to a global coordinator, cutting slow cross-region communication and reducing idle time, with claimed speedups in simulation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.6 is proved for a combined merge-and-gradient LPS update, not for Algorithm 1's ordered update-then-merge, so the convergence guarantee does not yet cover the reported method.","rationale":"The reader identified the right weakest assumption. I could not find a stronger load-bearing objection. The empirical evaluation is simulator-based with a single worker-speed draw and hyperparameters tuned on Pythia-70M; that limits generality but does not by itself invalidate the time-to-loss claim, and the released code plus trace-driven replay on real GPUs is meaningful supporting evidence. The proof–algorithm gap is more serious because the paper's advertised contribution includes a rigorous convergence analysis for HALoS. Appendix E's LPS recurrence merges and applies gradients at every step, whereas Algorithm 1 treats gradient updates and global-model merges as separate queued events gated by K; the closed-form comparison shows these differ for K>1. The theorem therefore currently analyzes a stylized variant, and the correct fix is either an equivalence lemma (e.g., reindexing timestamps and choosing α and t_last so the recurrences coincide) or a proof adapted to the queued event semantics with K-gating reflected in the staleness bounds. This leaves the verdict CONDITIONAL: accept only after the proof is aligned with Algorithm 1. Since the reader's verdict is already CONDITIONAL, no change is needed.","tokens_in":23872,"tokens_out":4224,"duration_ms":51974,"concrete_test":"Derive the one-cycle update of both recurrences with β_l=0, constant gradient g, fixed global model Θ_old, and no intervening global messages, starting from θ_0. Algorithm 1: after K local steps θ_K=θ_0−Kηg, then the merge gives θ=(1−α)(θ_0−Kηg)+αΘ_old. Appendix E's recurrence gives θ_K=(1−α)^Kθ_0+[1−(1−α)^K]Θ_old−ηg(1−(1−α)^K)/α. These are unequal for general K>1 (e.g., α=0.25, K=32, Θ_old=1, θ_0=0, ηg=0.1 gives −2.15 vs ≈0.60). If the authors instead prove a mapping between the two recurrences, the mismatch is resolved; absent that, Theorem 4.6 should be relabeled as analyzing an idealized combined-update rule, not Algorithm 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E's LPS recurrence is θ_{l,t+1} = (1−α)θ_{l,t} + αΘ_{lg(l,t)} − η_t[(1−β_l)∇_l(θ_{l,t}) + β_l m_{l,t+1}], i.e., a merge with the global model and a momentum gradient step are applied at every LPS time step. Algorithm 1 (lines 9–20) instead applies only ModelUpdate(θ_{t−1}, δ, η_l, β_l) when a worker delta arrives, and only when a new global model Θ_i arrives does it set θ_t ← (1−α)θ_t + αΘ_i; these events share a queue, sending to the GPS is gated by t−t_last=K, and several local updates can occur between merges. A convex combination applied every step is not the same operation as one applied once after K local updates: with K>1 and a fixed stale global model, the proof's model is pulled toward the stale Θ on every local step, while Algorithm 1's model takes K pure gradient steps and is then blended once. Consequently Theorem 4.6's bound—including the stated O(1/(1−β_g)) and staleness terms—does not, as written, govern the update rule whose speedups are reported. The theoretical contribution is load-bearing for the claim of a rigorous convergence analysis, and this mismatch is the weakest point of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HALoS, a hierarchical asynchronous local-SGD framework for geo-distributed LLM training. HALoS places a local parameter server (LPS) in each region and a global parameter server (GPS) that receives accumulated updates from LPSs; workers perform H local SGD steps and send deltas to their LPS, which is allowed to keep updating while a stale global model is in flight and later merges the global model with the local model using a weight alpha. The paper contributes an algorithm, a non-convex convergence theorem (Theorem 4.6) with a proof in Appendix E, and a trace-driven simulation study reporting up to 7.5x faster time-to-loss than DiLoCo and up to 2.1x faster than Async-Local-SGD, with benchmark accuracy matching or exceeding synchronous SGD.","tokens_in":24124,"tokens_out":7604,"duration_ms":81045,"significance":"If Theorem 4.6 genuinely governed Algorithm 1, the paper would be a useful contribution to geo-distributed LLM training: it combines a concrete hierarchical design, a convergence analysis that explicitly separates local and global momentum effects, and an unusually careful simulation methodology using measured inter-region bandwidths, profiled H100 step times, and a published implementation. The empirical section is a strength: the trace-driven simulator is described in enough detail to be reproduced, the hyperparameter sweep is disclosed, and the baselines include DiLoCo with dynamic updates, which isolates the benefit of asynchrony. The main significance hinges on the theoretical claim, and that claim currently applies to an update rule that differs from the algorithm whose speedups are reported. With a corrected or properly scoped proof, the paper could be accepted; without it, the central claim of a rigorous convergence analysis is not established.","major_comments":[{"comment":"Theorem 4.6 is proved for a LPS update rule that is not the rule in Algorithm 1. The recurrence for theta_{l,t+1} displayed in Appendix E immediately after Assumption E.5 applies the merge with the global model at every LPS time step, whereas Algorithm 1 (lines 9-20) applies a momentum-based ModelUpdate when a worker delta arrives and applies the convex combination theta_t <- (1-alpha)theta_t + alpha Theta_i only when a new global model arrives; with K > 1 there are gradient-only updates between merges. These two operations do not commute: in the proof's combined update the gradient step is scaled by (1-alpha) and the stale Theta pulls the model at every step, while Algorithm 1 takes pure gradient steps and then blends once. The proof also uses a gradient evaluated at the current LPS model, while the worker sends a delta produced after H local SGD steps, and the experiments (Appendix D) state that HALoS uses delayed Nesterov momentum with delays d_g and d_l, which do not appear in Algorithm 1 or in the proof. The GPS update in the proof is a convex combination of local gradients and momenta, not the difference delta = theta_t - theta_{t_last} received in Algorithm 1. Consequently, Theorem 4.6 does not, as written, govern the algorithm whose speedups are reported. This is load-bearing for the paper's claim of a rigorous convergence analysis, so the proof must be repaired or the theorem must be explicitly scoped to an idealized variant with an equivalence argument.","section":"Appendix E vs. Algorithm 1"},{"comment":"The bounded-staleness constants D_g and D_l are assumed but never bounded in terms of the algorithm's parameters or the network model. The second constant term in Theorem 4.6 is O(L^2(D_g^2 + D_l^2)), yet the paper does not derive any dependence of D_g or D_l on H, K, alpha, worker speeds, or inter-region bandwidth/latency. As stated, the theorem cannot explain the central trade-off that larger K reduces communication but increases staleness, nor can it yield the claimed beta_g = 0.5 recommendation; those conclusions are supported only by the hyperparameter sweep in Figure 5. Please derive explicit bounds on D_g and D_l from the algorithm parameters (or state that the theorem is conditional on unknown staleness constants), and connect those bounds to the communication model used in the simulations.","section":"Section 4.2, Assumption 4.5 and Eq. (2)"}],"minor_comments":[{"comment":"The theorem statement says 'Under certain assumptions' without naming them; it should explicitly reference Assumptions 4.1-4.5 so that the main theorem is self-contained.","section":"Section 4.2, Theorem 4.6 statement"},{"comment":"The sentence 'Overall, our derived bound is tight considering both perspectives' is not supported by a matching lower bound or a constructed example; please temper this claim or provide a formal tightness argument.","section":"Section 4.2, tightness claim"},{"comment":"In the final displayed chain of the proof, the term D_l^2 appears without the L^2 factor that multiplies D_g^2 and appears in the preceding lines; this appears to be a typo but should be fixed for consistency.","section":"Appendix E, last displayed chain"},{"comment":"The text says the HALoS model is trained on 223B tokens to match Pythia checkpoints trained on 207B tokens; please clarify whether the extra tokens are required by the method or are an artifact of the training schedule, since token counts are reported as a key efficiency metric.","section":"Section 5.1, benchmark comparison"}],"recommendation":"major_revision","confidential_remarks":"The mismatch between Appendix E and Algorithm 1 is the decisive issue. It is not a presentation flaw: the proof's recurrence, the GPS aggregation rule, and the delayed Nesterov setting used in experiments all differ from the pseudocode in Algorithm 1. I do not see an internal inconsistency in the simulation methodology itself; the speedup numbers are plausible as trace-driven results. If the authors can supply a corrected proof for Algorithm 1 (or precisely characterize the idealized variant and prove equivalence), the paper could be acceptable. I would also ask them to state explicitly which momentum update (plain or delayed Nesterov) the theorem covers and to relate D_g and D_l to H, K, alpha, and the communication model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the systems work is real: HALoS is a hierarchical asynchronous local-SGD design with server-side update accumulation and global model merging, evaluated with a careful trace-driven simulator, and it shows large time-to-loss gains over DiLoCo and Async-Local-SGD. Second, the theoretical result as written doesn't govern the algorithm. Theorem 4.6's proof in Appendix E analyzes an LPS update that combines the merge with a momentum gradient step at every time step, while Algorithm 1 performs gradient updates whenever worker deltas arrive and only merges with the global model when a new global model arrives, every K accumulated updates. Those are different dynamics. The bound's staleness terms D_g and D_l are never tied to K, H, alpha, or the network delays, so the 'rigorous convergence analysis' promised in the abstract is currently unsupported for the reported method.\n\nCredit where it's due: the specific combination—fully asynchronous at both tiers, local update accumulation, global merging, two-level momentum—is new relative to DiLoCo, Async-Local-SGD, and hierarchical federated learning methods. The simulator is more careful than most systems papers: it uses measured inter-region bandwidths, profiled per-step times on H100s, and replays event traces on real GPUs, with code released. Ablations are thorough, and the Shakespeare result—global momentum hurting under heterogeneous data—is a nice confirmation of the paper's intuition. The gains also hold across Pythia scales and two other model families under the same hyperparameters, which strengthens the empirical case.\n\nBeyond the proof mismatch, the speedups come from simulation with a single worker-speed draw and no error bars or multiple seeds. That is common in this subfield, but it makes the headline numbers less robust than they look. Hyperparameters are tuned on Pythia-70M and carried over, and the beta_g=0.5 recommendation is the minimizer of the worst-case bound, then confirmed in the same sweep—so it's not a parameter-free prediction. Both are minor relative to the theory gap.\n\nBottom line: this is a systems paper that deserves referee time, not a desk reject. The empirical contribution stands on its own, but the theory section needs major revision—either prove the ordered update-then-merge rule or explicitly scope the claim. As written, I wouldn't rely on the convergence guarantee.\n\nRecommendation: send to peer review, with the algorithm-theorem correspondence as the main review question.","headline":"A genuinely useful hierarchical async-local-SGD system with careful simulation, but the advertised convergence proof analyzes a different update rule than Algorithm 1, so the theoretical claim does not yet cover the reported method.","tokens_in":24720,"tokens_out":3335,"would_cite":false,"duration_ms":38181,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68W15","90C26"],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-tier parameter-server scheme, HALoS trains geo-distributed LLMs up to 7.5x faster than DiLoCo and 2.1x faster than Async-Local-SGD while matching synchronous-SGD quality, backed by a non-convex convergence bound.","keywords":["hierarchical asynchronous training","local SGD","geo-distributed LLM training","parameter server","global model merging","non-convex convergence analysis","momentum","trace-driven simulation"],"falsifier":"A concrete check would be to replay Algorithm 1's exact event ordering (gradient updates whenever a worker's delta arrives, merge only when a global model returns) in the same simulator and compare the resulting model trajectories with the combined update rule $\\theta_{l,t+1} = (1-\\alpha)\\theta_{l,t} + \\alpha\\Theta_{lg(l,t)} - \\eta_t[(1-\\beta_l)\\nabla_l(\\theta_{l,t}) + \\beta_l m_{l,t+1}]$ that the proof analyzes; any material divergence means the theorem is not bounding the measured method. The decisive empirical falsifier is a live geo-distributed run in which HALoS fails to match the synchronous-SGD validation loss in the claimed time, or diverges when the worker-to-LPS ratio is uneven.","tokens_in":23617,"feed_emoji":"🌐","tokens_out":15175,"duration_ms":111623,"temperature":0.7,"pith_summary":"HALoS is a training method for large language models whose accelerators sit in different geographical regions connected by slow, low-bandwidth links. The paper's claim is that inserting a local parameter server inside each region — so workers talk only to a nearby server over fast links, that server accumulates their updates and periodically ships a single delta to a distant global server, and the returning global model is blended into the local model instead of replacing it — eliminates most of the idle waiting that slows both synchronous and flat asynchronous methods. If the claim holds, geo-distributed pretraining becomes practical at near-synchronous quality: experiments report up to 7.5x faster convergence to the same validation loss than DiLoCo, up to 2.1x faster than Async-Local-SGD, and a 68.6x time advantage over fully synchronous SGD, with matching or better benchmark accuracy. The paper also proves a non-convex convergence bound, Theorem 4.6, which it uses to explain why local momentum should stay high while global momentum must be moderate.","feed_headline":"Two-tier async training speeds geo-distributed LLMs up to 7.5x","feed_subtitle":"Local parameter servers keep workers busy and merge a global model back, matching synchronous-SGD quality.","key_machinery":"The argument is carried by the two-tier update loop with server-side accumulation and $\\alpha$-weighted global model merging. Workers take $H$ local SGD steps and send a gradient delta to their regional LPS; the LPS applies it with local momentum $\\beta_l$, and every $K$ updates ships the accumulated difference $\\Delta = \\theta_t - \\theta_{t_{\\mathrm{last}}}$ to the GPS; the GPS applies it with global momentum $\\beta_g$ and returns the new global model, which the LPS absorbs as $\\theta_t \\leftarrow (1-\\alpha)\\theta_t + \\alpha\\Theta_i$. The proof of Theorem 4.6 recasts the LPS step as a single combined update $\\theta_{l,t+1} = (1-\\alpha)\\theta_{l,t} + \\alpha\\Theta_{lg(l,t)} - \\eta_t[(1-\\beta_l)\\nabla_l(\\theta_{l,t}) + \\beta_l m_{l,t+1}]$, and the structure of the resulting bound — momentum terms of order $1/(1-\\beta_l)$ locally but $\\frac{1}{\\beta_g^3(1-\\beta_g)^3}$ globally — yields the paper's practical prescription: keep local momentum near 0.9, keep global momentum near 0.5, and drop global momentum entirely when regional data is strongly non-i.i.d.","core_discovery":"On the paper's own terms, HALoS establishes that a two-level parameter-server hierarchy lets asynchronous local SGD hide both slow inter-region communication and straggling heterogeneous workers: each regional local server keeps applying momentum-based gradient updates while awaiting the slow global round trip, and when a fresh global model arrives it is merged in with weight $\\alpha \\approx 0.25$ rather than overwriting the local model, so no local progress is discarded. Empirically, the claim is that Pythia-70M/160M/410M models trained this way reach the same validation loss as fully synchronous SGD training — matching or exceeding it on eight downstream benchmarks — while cutting time-to-loss by up to 7.5x versus DiLoCo, 3.9x versus DiLoCo with dynamic updates, 2.1x versus Async-Local-SGD, and 68.6x versus synchronous SGD in the simulated geo-distributed environment. The theoretical claim is Theorem 4.6, a bound of the form $O(1/T)$ plus constant terms governed by heterogeneity $\\sigma^2$ and asynchronous staleness $D_g^2 + D_l^2$, which the paper presents as the first convergence guarantee for hierarchical asynchronous distributed optimization with momentum.","pith_inferences":["All wall-clock numbers come from a trace-driven simulator that combines H100 profiling with published inter-region bandwidth measurements; a live multi-region deployment would be the decisive test of whether the 7.5x and 2.1x speedups survive real network variability.","The proof in Appendix E analyzes a combined merge-plus-gradient update, while Algorithm 1 merges only when a fresh global model arrives, possibly after several gradient updates; an equivalence argument or a proof for the event-driven rule is needed before Theorem 4.6 can be said to govern the measured algorithm.","The observed divergence under naive one-LPS-per-region grouping when worker counts differ (2 versus 6 per region) suggests HALoS's gains rest on balanced LPS workloads, making adaptive LPS formation a natural next step.","A testable extension falls out of the theory: the optimal global momentum should decrease as cross-region data heterogeneity increases, a prediction the Shakespeare experiment already points toward."],"forward_implications":["Geo-distributed LLM pretraining can be made communication-efficient without a model-quality penalty: HALoS matches or exceeds synchronous-SGD accuracy on all eight reported downstream benchmarks while shortening time-to-loss.","The convergence bound gives a concrete rule for setting momentum in hierarchical asynchronous training (high local momentum near 0.9, moderate global momentum near 0.5) and warns that global momentum should be reduced or disabled when regions hold conflicting data.","Server-side update accumulation (K around 32) cuts the global server's workload and stabilizes cross-region gradient variance, so the approach scales to larger numbers of regions and workers.","The method carries over to other model families (Llama, Qwen) with hyperparameters tuned once on a small Pythia model, and composes with model parallelism by treating each worker as an accelerator group."],"supporting_citations":[{"why":"DiLoCo, the synchronous momentum-based local-SGD baseline that HALoS reports up to 7.5x faster convergence than.","marker":"(Douillard et al., 2023)"},{"why":"Async-Local-SGD, the strongest asynchronous baseline (up to 2.1x), whose delayed Nesterov momentum HALoS adopts for a fair comparison.","marker":"(Liu et al., 2024)"},{"why":"Source of the measured inter-region bandwidth matrix used to drive the geo-distributed simulator.","marker":"(Jaghouar et al., 2024)"},{"why":"Supplies the bounded-heterogeneity and bounded-staleness assumptions used in the proof; removing momentum from Theorem 4.6 recovers its bound.","marker":"(Nguyen et al., 2022)"},{"why":"Momentum-SGD analysis whose bound Theorem 4.6 recovers when the staleness contributions are removed.","marker":"(Liu et al., 2020)"},{"why":"FedAH, the semi-synchronous hierarchical baseline that the technique ablation in Table 2 improves on by 3.3x.","marker":"(Wang & Wang, 2022)"},{"why":"Pythia model suite whose checkpoints and trained-token order define the token-to-loss and downstream-benchmark evaluation targets.","marker":"(Biderman et al., 2023)"},{"why":"The Pile dataset on which the Pythia training runs and the benchmark comparisons are performed.","marker":"(Gao et al., 2020)"},{"why":"Provides the point-to-point communication model used to compute every simulated wall-clock time in the evaluation.","marker":"(Valiant, 1990)"}],"fun_headline_variants":["Hierarchical async SGD cuts geo-distributed LLM training time by 7.5x","Two-tier async training speeds geo-distributed LLMs up to 7.5x","Local parameter servers enable 7.5x faster async LLM training","HALoS: 7.5x faster geo-distributed LLM training via local servers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof covers a simplified local-server update in which merging with the global model and applying the worker gradient happen together in one step, but the actual algorithm merges with a freshly received global model only occasionally, after several gradient updates have already been applied; if these two ways of updating are not the same, the mathematical guarantee does not cover the algorithm whose speedups the paper reports.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical async SGD cuts geo-distributed LLM training time by 7.5x","Two-tier async training speeds geo-distributed LLMs up to 7.5x","Local parameter servers enable 7.5x faster async LLM training","HALoS: 7.5x faster geo-distributed LLM training via local servers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1728,"prompt_tokens":1015,"completion_tokens":713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":631,"tokens_out":713,"duration_ms":6865,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:40:33.406982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to replay Algorithm 1's exact event ordering (gradient updates whenever a worker's delta arrives, merge only when a global model returns) in the same simulator and compare the resulting model trajectories with the combined update rule $\\theta_{l,t+1} = (1-\\alpha)\\theta_{l,t} + \\alpha\\Theta_{lg(l,t)} - \\eta_t[(1-\\beta_l)\\nabla_l(\\theta_{l,t}) + \\beta_l m_{l,t+1}]$ that the proof analyzes; any material divergence means the theorem is not bounding the measured method. The decisive empirical falsifier is a live geo-distributed run in which HALoS fails to match the synchronous-SGD validation loss in the claimed time, or diverges when the worker-to-LPS ratio is uneven.","supporting_citations":[{"cited_title":"Federated learning with buffered asynchronous aggregation","cited_arxiv_id":null,"evidence_quote":"Supplies the bounded-heterogeneity and bounded-staleness assumptions used in the proof; removing momentum from Theorem 4.6 recovers its bound."},{"cited_title":"An improved analysis of stochastic gradient descent with momentum","cited_arxiv_id":null,"evidence_quote":"Momentum-SGD analysis whose bound Theorem 4.6 recovers when the staleness contributions are removed."},{"cited_title":"Asynchronous Hierarchical Federated Learning","cited_arxiv_id":"2206.00054","evidence_quote":"FedAH, the semi-synchronous hierarchical baseline that the technique ablation in Table 2 improves on by 3.3x."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the point-to-point communication model used to compute every simulated wall-clock time in the evaluation."}],"review_version":1}