{"id":"8a97b888-6c73-4ccf-90d4-b7a4bf1a5c48","arxiv_id":"2506.10911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"NoLoCo trains large language models without any all-to-all synchronization by using pairwise weight averaging and random pipeline routing, matching or slightly beating DiLoCo in experiments.","lead":"NoLoCo is a new way to train large AI models that skips the expensive all-reduce step used by distributed training. Each machine only exchanges model weights with one randomly chosen partner, and the authors show it can match or slightly beat a strong low-communication baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Even granting the paper's independence assumption, Eq. 55 contains an apparent factor-γ error in the covariance term, so Theorem 3's stability condition for γ is not derived; because γ is never reported, the experiments do not verify the claimed regime.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, which matches my assessment. The central empirical claim is not obviously false: Table 2 shows NoLoCo close to DiLoCo across many configurations, the random-routing ablation is sensible, and the latency model, while simulated, is not absurd. The weakest point is the theory. Even granting all independence assumptions, the covariance identity in Eq. 55 appears to have the wrong power of γ, and that factor directly controls the variance recurrence and the stability bound in Eq. 74. Since γ is absent from the experimental section, the connection between theory and experiment is unverifiable from the text. A corrected derivation and disclosed γ values would settle this. No adversarial reading of the authors is needed; this is an internal algebraic inconsistency plus a missing hyperparameter. The empirical side also lacks error bars and uses shared suboptimal hyperparameters, but that is a strength-of-evidence issue rather than a proof-level obstacle, so I do not recommend changing the reader's conditional verdict.","tokens_in":18424,"tokens_out":7491,"duration_ms":84396,"concrete_test":"Independently re-derive Eq. 55 from Eq. 2/31: with δ_t,i = αδ_{t−1,i} − (β/n)Σ_j Δ_t,j − γ(φ_t,i − (1/n)Σ_j φ_t,j), compute Cov(φ_t,i, δ_t,i) after dropping the same terms the text drops. If the result is −γ((n−1)/n)V(φ_t,i) instead of −γ²((n−1)/n)V(φ_t,i), recompute the d_V coefficient in Eq. 68 and the γ bounds in Eq. 74. If the corrected condition differs from the printed one, Theorem 3 is not established as written, and the paper should report every γ used in Table 2 and Fig. 3, or extract it from the GitHub code and rerun Table 2 with γ inside and outside the corrected bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is that the convergence guarantee anchoring the central claim is not established even on the paper's own assumptions. In Appendix A, the slow weights are admitted not to be independent, and Conjecture 1 is assumed without proof. More concretely, Eq. 55 states Cov(φ_t,i, δ_t,i) ≈ −γ²((n−1)/n)V(φ_t,i), but the only direct dependence of δ_t,i on φ_t,i in Eq. 2/31 is the term −γ(φ_t,i − (1/n)Σ_j φ_t,j). If the omitted δ_{t−1} and Δ terms are negligible as the text states, this covariance must be linear in γ: Cov ≈ −γ((n−1)/n)V. The γ² version changes the coefficient in Eqs. 62-68 and hence the stability range for γ in Eq. 74 and the variance claim of Theorem 3. Section 4 reports α=0.5, β=0.7, and group size n=2, but never reports γ, so the reader cannot check whether the runs lie in the claimed stable region. The empirical results in Table 2 and Fig. 3 may still be valid, but they are not tied to the stated theory as written; the central convergence claim is therefore conditional on a corrected derivation and on disclosure of γ.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NoLoCo, a distributed training method that eliminates collective communication (all-reduce) by performing outer-optimizer synchronization between randomly chosen pairs of replicas and by using random pipeline routing, with a modified Nesterov momentum update that includes a local averaging term. The authors provide a convergence analysis for a quadratic loss (expected value tends to zero, variance proportional to inner learning rate squared) and report language-model experiments on 125M-6.8B models comparing NoLoCo to DiLoCo and FSDP. They claim up to 4% faster convergence than DiLoCo and estimate an order-of-magnitude reduction in synchronization cost. The manuscript includes source code, an explicit latency model, and ablations of random routing and batch size.","tokens_in":18782,"tokens_out":8681,"duration_ms":90520,"significance":"If the central claims were established, NoLoCo would be a meaningful advance for low-bandwidth, high-latency distributed training: removing global blocking all-reduce and replacing it with point-to-point pair averaging is an appealing design, and the theoretical variance-control prediction gives a concrete mechanism for eventual consistency. The paper is candid about several limitations (suboptimal hyperparameters, latency model vs measured deployment, and the independence assumption). However, the theoretical guarantee is not currently established: the expectation recurrence has a sign error, the covariance term in the variance recurrence has an incorrect power of gamma, and the proof relies on an admitted-false independence assumption and an unproven conjecture. The empirical support for the headline speed-up is weakened by single-run comparisons with untuned hyperparameters and no error bars. These issues are fixable in revision, so the underlying idea remains worth pursuing.","major_comments":[{"comment":"The expectation recurrence has a sign error. From Eq. (2)/Eq. (31), delta_t,i = alpha delta_{t-1,i} - (beta/n) sum_j Delta_{t,j} - gamma(phi_t,i - (1/n) sum_j phi_t,j). Taking expectations, the gamma term vanishes by Lemma 1, so E(delta_t,i) = alpha E(delta_{t-1,i}) - beta E(Delta_{t,i}), not the plus sign used in Eq. (32). This sign propagates through Eqs. (33)-(43), and the matrix D in Eq. (43) should be I + alpha I - beta(B^m - I), not I + alpha I + beta(B^m - I). Consequently the stability condition 'beta > alpha' below Eq. (53) is derived from the wrong recurrence. The proof of Theorem 2 must be re-derived with the correct sign; this is load-bearing for the paper's central convergence claim.","section":"Appendix A.2, Eq. (32)"},{"comment":"The covariance approximation contains an incorrect power of gamma. The only direct dependence of delta_t,i on phi_t,i in Eq. (31) is the term -gamma(phi_t,i - (1/n) sum_j phi_t,j). Under the text's stated neglect of covariances with Delta and delta_{t-1}, the leading term is Cov(phi_t,i, delta_t,i) approximately -gamma((n-1)/n)V(phi_t,i), linear in gamma, not -gamma^2((n-1)/n)V as written. This error changes the coefficient of the second term in Eq. (62), the definition of d_V in Eq. (69), and hence the stability window for gamma in Eq. (74). As written, Theorem 3's variance bound is not derived; the corrected recurrence should be solved and the stability condition updated.","section":"Appendix A.3, Eq. (55)"},{"comment":"The convergence proof relies on assumptions that are acknowledged in the text to be false or unproven. In Appendix A.2 the authors state that slow weights across replicas are 'generally not independent' and yet assume independence for the convergence analysis; Conjecture 1 for the variance of the deviation from the group mean is stated without proof. The text asserts that the error 'will become smaller when the data parallel world size becomes larger' but provides no quantitative bound. Since the variance recurrence (Eqs. (56)-(70)) and Theorem 3 depend on Conjecture 1, the guarantee is conditional on an unverified assumption. Please either prove the conjecture under a stated weaker condition, or provide a bound on the error and state the theorem as conditional.","section":"Appendix A.2, Conjecture 1"},{"comment":"The hyperparameter gamma is never reported. Eq. (2) includes gamma, and the theoretical stability condition (Eq. (74)) is a constraint on gamma. Section 4 reports alpha=0.5, beta=0.7, and group size n=2 for NoLoCo but does not state the gamma value used in any run. Without this value the reader cannot check whether the experiments operate in the claimed stable regime, and the experimental configuration is not reproducible. Please report gamma for every configuration and, ideally, provide an ablation over gamma around the predicted stability boundary.","section":"Section 4, Eq. (2)"},{"comment":"The empirical comparison supporting the headline 'up to 4% faster convergence' is not backed by uncertainty estimates. Table 2 and Fig. 2 report single training runs without error bars or repeated seeds, and Section 5.1 states that the batch size and learning rate are taken from an FSDP-optimized study and are 'likely sub-optimal' for both DiLoCo and NoLoCo. The differences between DiLoCo and NoLoCo in Table 2 are often small (e.g., 26.8 vs 26.4 for the small model on Reddit with DP=8) and could be within run-to-run noise. Please provide multiple seeds with error bars (or at least report the number of runs) and either tune hyperparameters for both methods or clearly label the results as a preliminary comparison rather than evidence of faster convergence.","section":"Section 5.1, Table 2"}],"minor_comments":[{"comment":"There is a typo: 'NiLoCo' should be 'NoLoCo' in the sentence 'we observe that NiLoCo is slightly better than DiLoCo'.","section":"Section 5.1"},{"comment":"The y-axis label sigma/sigma_max is not defined in the text; please state explicitly whether sigma is the standard deviation of a single parameter, averaged over all parameters, and how sigma_max is computed.","section":"Fig. 3B"},{"comment":"The characteristic equations involve matrix square roots and matrix inequalities (e.g., '0 < r_2 <= r_1 <= D') that are not well-defined for general matrices; these should be stated in terms of the eigenvalues of D or of A.","section":"Eqs. (45)-(46)"},{"comment":"There appears to be an index inconsistency in the summation: the term B^{k-1} appears where B^k is used elsewhere; please check and correct the indexing.","section":"Eq. (24)"},{"comment":"In Eq. (21), the variance of a sum of gradient terms is approximated by the sum of variances without a justification for neglecting covariances of non-consecutive fast weights; please make this approximation explicit and, if possible, bound the neglected terms.","section":"Appendix A.1, Eq. (21)"},{"comment":"The random-routing ablation is presented with ratios (sigma/sigma_base, L/L_base) but without error bars; given the small effects (10-15%), please clarify how many runs these ratios are based on.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The main theoretical proof has a sign error and a gamma-power error; both are fixable, but the corrected stability conditions might differ, so the authors should verify that the experiments lie in the stable region after correction. The paper would benefit from a careful proofreading pass and from reporting gamma. I do not see grounds for rejection, but the current version overstates the guarantees in the abstract and Section 6."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is Eq. 2: a Nesterov-style outer update that averages weights with a randomly chosen partner, plus random pipeline routing. If it works, it removes the all-reduce from DiLoCo's outer steps and replaces it with point-to-point exchanges. That is a meaningful step for low-communication training.\n\nWhat the paper does well: it ships code, runs 125M–6.8B parameter models, and reports a clean correlation between replica weight variance and the inner learning rate (0.91–0.97), consistent with the intuition that the LR schedule controls replica divergence.\n\nThe soft spots are real and load-bearing. The convergence proof in Appendix A rests on an independence assumption the authors admit is false, and on Conjecture 1, which is unproven. More concretely, Eq. 55 gives Cov(φ, δ) ≈ −γ²((n−1)/n)V(φ), but the direct dependence of δ on φ is the term −γ(φ_i − φbar), so that covariance must be linear in γ, not γ². That error propagates into the stability condition for γ in Eq. 74, so the stated bound is not derived. On top of that, γ is never reported in the experiments, so no one can check whether the runs lie in the claimed stable regime. The stress-test note is correct on this.\n\nThe empirical comparison is also weaker than the abstract suggests: hyperparameters are suboptimal for both methods (the authors admit this), runs are single-shot with no error bars, and the headline \"one order of magnitude faster\" communication benefit comes from a log-normal simulation, not a measured deployment.\n\nThat said, the paper is honest about its main limitations, and the idea is worth testing further. With a corrected derivation, a reported γ, and a real geo-distributed benchmark, it could be a solid contribution.\n\nThis is a paper for the distributed training community. A serious referee could fix the theory and demand better empirics, so I'd send it to review, but with major revision expected.","headline":"A plausible no-all-reduce training method with a load-bearing proof error and an unreported key hyperparameter; worth reviewing but not yet convincing.","tokens_in":19269,"tokens_out":4100,"would_cite":false,"duration_ms":39668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes NoLoCo, a modified Nesterov-momentum optimizer that synchronizes model weights only between randomly chosen pairs of replicas, and argues this removes the need for all-reduce and all collective communication while…","keywords":["NoLoCo","low-communication training","decentralized training","distributed optimization","Nesterov momentum","pipeline parallelism","all-reduce","large language model training"],"falsifier":"Measure the actual variance of $\\phi_{t,i} - (1/n)\\sum_j \\phi_{t,j}$ during training and compare it with Conjecture 1's prediction $2((n-1)/n)^2\\mathrm{Var}(\\phi_{t,i})$; if the mismatch does not shrink as the number of data-parallel workers grows, the proof's independence assumption is doing the work and the theorem would not transfer to the correlated setting. A second, direct test is a real geo-distributed run with hundreds of accelerators: if wall-clock time per outer step does not drop by roughly $\\log_2(n)$ relative to a tree all-reduce and validation perplexity does not track the reported gap to DiLoCo, the practical speed-up claim fails.","tokens_in":18249,"feed_emoji":"🌐","tokens_out":11119,"duration_ms":104764,"temperature":0.7,"pith_summary":"Training large language models normally assumes a tightly connected cluster because data-parallel training must keep all replicas synchronized. NoLoCo is an optimizer designed to break that assumption: it never performs an all-reduce or any global collective operation, instead implicitly synchronizing model weights by partially averaging each replica's slow weights with one randomly chosen other replica at each outer step. The paper argues that this modified Nesterov momentum update keeps replicas from diverging, and it reports that models from 125M to 6.8B parameters converge comparably to, and up to 4% faster than, the low-communication DiLoCo baseline on two language modeling datasets. If the claim holds, large-model training could work over low-bandwidth, high-latency networks instead of dedicated supercomputing fabrics, because the synchronization cost per step grows only logarithmically with worker count and no global barrier idles fast workers.","feed_headline":"NoLoCo drops all-reduce, matching DiLoCo's convergence","feed_subtitle":"Synchronizing only between random pairs could make large-model training work on slow, high-latency networks.","key_machinery":"The load-bearing object is the modified Nesterov momentum outer update of Eq. (2): $\\delta_{t,i} = \\alpha\\delta_{t-1,i} - (\\beta/n)\\sum_j \\Delta_{t,j} - \\gamma(\\phi_{t,i} - (1/n)\\sum_j \\phi_{t,j})$. The third term is what turns pairwise averaging into implicit synchronization: each worker's slow weights are dragged toward the random subgroup mean, and because the subgroup is re-sampled every outer step, information spreads across all workers without any collective call. The convergence argument studies the standard quadratic stochastic loss $L(\\theta)=\\frac12(\\theta-c)^T A(\\theta-c)$ with $c\\sim\\mathcal{N}(0,\\Sigma)$, which reduces the expected-value recursion to a linear recurrence whose roots stay below 1 when $\\beta>\\alpha$ and $0<\\omega\\Lambda_i\\le 1$; the variance recursion then gives $\\mathrm{Var}(\\phi_{t,i})\\propto\\omega^2$, so the inner learning rate controls replica divergence. Random pipeline routing is the second mechanism: by letting a stage receive inputs from any replica of the previous stage, it mixes weights across data-parallel pipelines and is reported to lower replica variance by up to 15% at a small cost in validation loss.","core_discovery":"The central claim is that explicit all-to-all synchronization is not necessary for low-communication large-model training. In NoLoCo, the outer update is $\\delta_{t,i} = \\alpha\\delta_{t-1,i} - (\\beta/n)\\sum_j \\Delta_{t,j} - \\gamma(\\phi_{t,i} - (1/n)\\sum_j \\phi_{t,j})$, where $\\Delta_{t,i} = \\theta_{t+1,i} - \\phi_{t,i}$ is the local outer gradient, $\\alpha$ is momentum, $\\beta$ the outer learning rate, and $\\gamma$ the local averaging strength, with the group of size $n=2$ re-sampled randomly each outer step. The third term pulls each worker's slow weights toward the group average, so weights are implicitly synchronized even though no accelerator ever talks to all others. On a quadratic stochastic loss the paper proves the expected slow weights converge to the optimum and the variance across replicas is proportional to the square of the inner learning rate, which lets a cosine learning-rate schedule drive eventual consistency. Empirically, NoLoCo matches or beats DiLoCo in validation perplexity across model sizes and worker counts while requiring no global blocking communication, replacing a tree all-reduce that costs about $2t_c\\log_2(n)$ with pairwise averaging that costs about $2t_c$.","pith_inferences":["Beyond the paper: if pairwise random averaging is enough for convergence, the same principle could be combined with gradient compression or asynchronous updates, since correctness no longer depends on a global barrier.","Beyond the paper: the claimed wall-clock advantage rests on a log-normal latency model rather than a measured geo-distributed deployment, so the natural next experiment is a real run across distant accelerators, which would also reveal how often a random pair lands on a slow link.","Beyond the paper: the independence approximation in the proof gets better as the data-parallel world size grows, so the theory predicts the method becomes more reliable exactly in the regime where all-reduce is most expensive; testing convergence at hundreds or thousands of workers would probe this.","Beyond the paper: if replica variance is truly controlled by the inner learning rate, a learning-rate schedule could be tuned to make the final ensemble agree to arbitrary precision without any explicit synchronization step, giving a parameter-server-free path to a single merged model."],"forward_implications":["The outer synchronization cost per step drops from about $2t_c\\log_2(n)$ for a tree all-reduce to about $2t_c$ for pairwise averaging, so the advantage grows with worker count and with message-latency variance.","Removing the global barrier eliminates the idling where fast workers wait for slow ones; the paper's simulations put the total-time saving near 20% at 1024 accelerators with 100 inner steps per outer step.","The replicas remain an ensemble rather than identical copies, and the standard deviation across replicas tracks the inner learning rate (empirical Pearson correlation between 0.91 and 0.97), so a learning-rate schedule can be used to drive eventual consistency to arbitrary numerical accuracy.","NoLoCo's final validation perplexity is a few percent worse than fully sharded data parallel training but up to 4% better than DiLoCo, with the gap to fully synchronized training shrinking for larger models and smaller data-parallel world sizes.","Random pipeline routing alone, without any outer synchronization, reduces replica weight variance by up to 15% while raising validation perplexity by up to 4%, and this tradeoff becomes less pronounced for larger models."],"supporting_citations":[{"why":"Defines DiLoCo, the inner-outer low-communication baseline with periodic all-reduce that NoLoCo must match and exceed.","marker":"Douillard et al. (2023)"},{"why":"Supplies the look-ahead optimizer structure of slow and fast weights, as well as the quadratic stochastic loss used in the convergence proof.","marker":"Zhang et al. (2019)"},{"why":"Introduces epidemic learning's local subset averaging, the direct precursor to NoLoCo's pairwise outer synchronization.","marker":"De Vos et al. (2023)"},{"why":"Provides the random pipeline routing scheme that NoLoCo uses to implicitly mix weights across replicas.","marker":"Ryabinin et al. (2023)"},{"why":"Establishes DiLoCo scaling behavior with model size and data-parallel world size, the trend NoLoCo reports matching.","marker":"Charles et al. (2025)"},{"why":"Supplies the model sizes, global batch sizes, and learning-rate schedule used in the experiments.","marker":"Zhang et al. (2022)"},{"why":"Documents that the outer all-reduce step can take minutes in highly distributed networks, motivating the removal of global communication.","marker":"Jaghouar et al. (2024)"}],"fun_headline_variants":["NoLoCo drops all-reduce, still matches DiLoCo","NoLoCo: no global sync, same convergence as DiLoCo","Pairwise sync only: NoLoCo rivals DiLoCo","NoLoCo eliminates all-reduce without hurting convergence","No collective communication: NoLoCo matches DiLoCo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof assumes that the model copies behave as statistically independent replicas, even though the authors acknowledge that the routing and previous averaging steps make them correlated, so the theoretical guarantee rests on that approximation being good enough.","fun_headline_variants_meta":{"raw":{"variants":["NoLoCo drops all-reduce, still matches DiLoCo","NoLoCo: no global sync, same convergence as DiLoCo","Pairwise sync only: NoLoCo rivals DiLoCo","NoLoCo eliminates all-reduce without hurting convergence","No collective communication: NoLoCo matches DiLoCo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1732,"prompt_tokens":1135,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":751,"completion_tokens_details":{"reasoning_tokens":511}},"tokens_in":751,"tokens_out":597,"duration_ms":6671,"temperature":1.0,"reasoning_tokens":511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:13:36.378657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual variance of $\\phi_{t,i} - (1/n)\\sum_j \\phi_{t,j}$ during training and compare it with Conjecture 1's prediction $2((n-1)/n)^2\\mathrm{Var}(\\phi_{t,i})$; if the mismatch does not shrink as the number of data-parallel workers grows, the proof's independence assumption is doing the work and the theorem would not transfer to the correlated setting. A second, direct test is a real geo-distributed run with hundreds of accelerators: if wall-clock time per outer step does not drop by roughly $\\log_2(n)$ relative to a tree all-reduce and validation perplexity does not track the reported gap to DiLoCo, the practical speed-up claim fails.","supporting_citations":[],"review_version":1}