{"id":"145e7f33-af96-4b22-b00e-1e8096616c33","arxiv_id":"2504.17672","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoCoDC uses Taylor-expansion delay compensation and adaptive fragment scheduling to reach a target perplexity up to 21% fewer steps than Streaming DiLoCo in simulated cross-region LLM training.","lead":"CoCoDC is a new method for training large language models across distant data centers, combining overlapping communication and computation with a delay compensation step. In a simulated four-worker test with a 150M-parameter model, it reached the target perplexity up to 21% faster than Streaming DiLoCo.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoCoDC's 21% step reduction is confounded: it performs 8 fragment syncs per H vs 4 for Streaming DiLoCo (Eq. 9, Sec. IV.A); matched-frequency ablation is needed to attribute the gain to delay compensation and adaptive selection.","rationale":"The strongest claim is Table I's 21% step reduction. For this claim to justify the paper's mechanism-based attribution, the comparison must isolate delay compensation and adaptive selection from synchronization volume. Section IV.A sets K=4 for both methods but sets gamma=0.4, and Eq. 9 yields N=8 fragment syncs per H for CoCoDC, while Streaming DiLoCo performs only K=4. This is an uncontrolled variable: streaming methods with more frequent pseudo-gradient exchanges can converge faster per step, so Table I cannot distinguish the proposed mechanisms from the effect of twice the communication rounds. The paper also self-admits that final metrics may not show substantial improvements and provides no wall-clock measurement or code release. These issues all trace to the same missing control. I agree with the reader's weakest_assumption. A matched-frequency ablation would settle the concern; until then the central claim is conditional rather than established. Since the reader's verdict is already CONDITIONAL, no adjustment is needed.","tokens_in":10630,"tokens_out":4349,"duration_ms":45187,"concrete_test":"Run the Section IV.A setup with identical H=100, K=4, tau=5, model, and dataset, then add: (1) Streaming DiLoCo with the fixed round-robin schedule but N=8 fragment syncs per H (each shard synchronized twice); (2) CoCoDC with gamma adjusted so N=4, matching Streaming DiLoCo's sync count; (3) CoCoDC with delay compensation disabled (reverting to Eq. 3 blending) but adaptive selection kept at N=8. Compare steps-to-PPL-20 and final PPL over at least 3 seeds. If condition (1) matches CoCoDC's 10,292 steps, or condition (2) loses to Streaming DiLoCo, the 21% improvement is attributable to communication volume rather than the proposed mechanisms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Table I, Fig. 2) is a step-count comparison: CoCoDC reaches PPL 20 in 10,292 steps vs 13,021 for Streaming DiLoCo, a 21.0% reduction. The most load-bearing concern is that this comparison does not control the number of fragment synchronizations. Streaming DiLoCo is set up with K=4 shards and syncs each shard exactly once per H=100 steps (4 total fragment all-reduce rounds per H). CoCoDC uses Eq. 9 with gamma=0.4, which Section IV.A states results in 8 synchronizations per H steps—twice the communication rounds and twice the pseudo-gradient volume. More frequent global updates can independently accelerate convergence, so the 21% step reduction may be an artifact of synchronization frequency rather than the Taylor delay compensation (Eqs. 7-8) or the adaptive selection metric (Eq. 11). The paper provides no ablation matching synchronization frequency, no wall-clock comparison to account for the added communication cost, and no error bars. Thus the mechanism-based claim that delay compensation and adaptive selection explain the improvement is unsupported by the current experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoCoDC, a distributed training framework for cross-region LLM training that extends Streaming DiLoCo with two mechanisms: a Taylor-expansion-based delay compensation step (Section III.A, Eqs. (4)-(8)) and an adaptive fragment transmission policy (Section III.B, Eqs. (9)-(12)). The experimental section simulates four datacenter workers on a single 4-GPU server using a 150M-parameter LLaMA-style model trained on the C4 dataset, comparing against DiLoCo and Streaming DiLoCo. The paper reports that CoCoDC reaches validation perplexity 20 in 10,292 steps versus 13,021 for Streaming DiLoCo (a 21.0% reduction) and 10,821 for DiLoCo, and achieves the lowest final perplexity (18.0357).","tokens_in":10882,"tokens_out":6599,"duration_ms":61290,"significance":"If the mechanisms worked as described, the paper would address a real problem: Streaming DiLoCo's communication-computation overlap introduces staleness and partial-synchronization inconsistency, and a low-overhead correction plus smarter fragment scheduling could improve convergence per step. The empirical validation uses an external benchmark (C4) rather than a self-referential or constructed target, which is a strength, and the paper identifies a genuine limitation of the Streaming DiLoCo schedule. However, the central contribution is currently compromised by a sign inconsistency in the delay-compensation derivation and by an experimental comparison that does not control synchronization frequency or measure wall-clock time. The free constants λ and γ are hand-set without sensitivity analysis. With corrected equations and matched ablations, the framework could be a useful contribution; in its current form, the paper's main claims are not fully supported.","major_comments":[{"comment":"The sign of the local change rate is internally inconsistent. Eq. (4) defines g_m^p,tp = (θ^m_p,tp − θ^m_p,tl)/τ. Since tl = tp + τ, this is the negative of the average local parameter change over the overlap interval, not the change rate in the direction of training. Eq. (8) then computes θ^m_p,tl ← θ^g_p,tp + g_corr_p,tp × τ, which moves the compensated state opposite to the local update direction. A Taylor extrapolation from tp to tl requires the positive rate (θ^m_p,tl − θ^m_p,tp)/τ, or equivalently subtracting the quantity defined in Eq. (4). Please correct the sign and verify that Algorithm 1 and the actual implementation follow the corrected formula; as written, the derivation does not support the compensation mechanism.","section":"Section III.A, Eqs. (4)-(8)"},{"comment":"The headline step-count comparison is confounded by synchronization frequency. Streaming DiLoCo with K=4 shards synchronizes each fragment once per H=100 steps, i.e., 4 fragment all-reduce rounds per H. CoCoDC is run with γ=0.4, which the paper states results in 8 synchronizations per H steps—twice the number of communication rounds and twice the pseudo-gradient volume. Since more frequent global updates can independently accelerate convergence, the 21.0% step reduction in Table I cannot be attributed specifically to delay compensation or adaptive selection. Please add a matched-frequency ablation, e.g., CoCoDC restricted to K synchronizations per H or Streaming DiLoCo allowed N=8 synchronizations per H, and report the results.","section":"Section IV.A, Eq. (9), and Table I"},{"comment":"The reported results depend on hand-set hyperparameters λ=0.5 and γ=0.4, with no sensitivity analysis or ablations. λ scales the Hessian approximation in Eq. (7), so the compensation term is not parameter-free, and γ directly controls the number of synchronizations in Eq. (9). Without varying these values, or at least reporting a small grid, the observed improvements may reflect tuning to the chosen setting rather than the proposed mechanisms. Please provide sensitivity curves for λ and γ, and ideally multiple random seeds, since Figs. 1 and 2 show single-run curves without error bars.","section":"Section IV.A, Eqs. (7) and (9)"},{"comment":"The paper's third contribution states that CoCoDC significantly reduces wall-clock training time, but the evaluation reports only training steps and validation metrics. No wall-clock time, communication time, or bandwidth utilization measurements are given. This is especially important because CoCoDC performs more synchronization rounds than Streaming DiLoCo, so a step-count advantage does not automatically translate into a wall-clock advantage. Please add wall-clock comparisons and a communication-overhead breakdown to support the training-speed claim.","section":"Section IV.B and Contributions"}],"minor_comments":[{"comment":"Line 3 of Algorithm 1 says the correction is applied to \"global state θ^m_p,tp\"; to match Eq. (8) and the algorithm input, this should be θ^g_p,tp.","section":"Algorithm 1, line 3"},{"comment":"The notation ⊙ is used both for the outer product of g with itself and then for the three-term product g ⊙ g ⊙ (Δθ/H). The dimensions are ambiguous; please define whether the Hessian action is a matrix-vector product and write it consistently, for example as λ g gᵀ (Δθ/H) or a diagonal approximation.","section":"Section III.A, Eq. (7)"},{"comment":"There is a typo in the text: \"negelects\" should be \"neglects.\"","section":"Section III.B"},{"comment":"The setup states overlap depth τ=5, but it does not explain how the simulated network latency/bandwidth maps to the measured values Tc and Ts used in Eq. (9). Reporting these values would make the stated N=8 synchronizations per H reproducible.","section":"Section IV.A"},{"comment":"The paper mentions possible non-IID data distributions across datacenters, but the experimental section does not describe how the C4 data is partitioned among workers. Please clarify whether the training data is shuffled globally or partitioned per worker.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Eq. (4) and the synchronization-frequency confound are serious enough that the paper should not be accepted in its current form. Both issues appear fixable within the manuscript's scope: the sign can be corrected and verified against the implementation, and the experimental claims can be supported with matched-frequency ablations, wall-clock measurements, and hyperparameter sensitivity studies. Given the ambiguity introduced by the sign error, access to the released code or implementation would substantially help verify that the reported experiments used the corrected compensation direction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoCoDC attacks a real problem: Streaming DiLoCo's overlap introduces stale updates and partial model inconsistency, and the paper proposes explicit delay compensation plus adaptive fragment transmission. The delay compensation is a Taylor-expansion correction applied to pseudo-gradients accumulated over H>1 local steps, which generalizes the H=1 asynchronous SGD delay compensation from [20]-[22]. The adaptive transmission picks fragments by pseudo-gradient norm and syncs some twice per H. That combination is new in this setting. The experiments are on C4 with a 150M LLaMA-style model, four emulated workers, and compare against DiLoCo and Streaming DiLoCo. The validation data is external, so the gains are not self-referential.\n\nThe main problem is the comparison. Streaming DiLoCo syncs each of 4 fragments once per H=100 steps, so 4 all-reduce rounds. CoCoDC with gamma=0.4 does 8 syncs per H, according to Eq. 9 and the text. Doubling the synchronization frequency alone can accelerate convergence, so the claimed 21% step reduction cannot be attributed to delay compensation or adaptive selection without an ablation that matches the number of syncs (e.g., a Streaming DiLoCo variant that also does 8 syncs, or a CoCoDC variant with N=4). Without that, the headline result is confounded.\n\nThere are smaller issues. The derivation has a sign problem: Eq. 4 defines the change rate as (theta_tp - theta_tl)/tau, which is the negative of the usual average parameter change over the overlap, yet Eq. 8 adds gcorr * tau to the global state. If the equations are taken literally, the correction moves the parameters in the opposite direction of the local updates. This could be a typo, but it makes the method hard to reproduce. lambda=0.5 and gamma=0.4 are hand-set with no sensitivity analysis. No wall-clock time is reported even though the abstract and introduction claim wall-clock gains. No error bars or multiple seeds. Code is not released. These are all addressable.\n\nSo my bottom line: the paper has a plausible framework and a potentially useful combination of ideas, but the central claim is not yet supported. It deserves peer review, not desk rejection, because the confound is fixable and the problem is relevant. A serious referee should ask for matched-frequency ablations, wall-clock measurements, sensitivity on lambda and gamma, and a corrected derivation.\n\nWho is this for? People working on cross-region or low-communication LLM training. I'd bring it to a reading group to discuss the experimental design, but I wouldn't cite it in its current form.","headline":"Plausible framework and a new combination of ideas, but the headline 21% step reduction is confounded by twice the synchronization frequency and the derivation has a sign error.","tokens_in":11414,"tokens_out":3927,"would_cite":false,"duration_ms":36092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoCoDC cuts cross-region LLM training to target perplexity in 21.0% fewer steps than Streaming DiLoCo, with a lower final perplexity.","keywords":["distributed training","cross-region LLM training","communication-computation overlap","delay compensation","adaptive transmission","staleness","Taylor expansion","DiLoCo"],"falsifier":"Run an experiment that matches synchronization counts: give Streaming DiLoCo the same number of fragment synchronizations per $H$ that CoCoDC uses (8 per 100 in the reported configuration) and give CoCoDC a round-robin schedule with that same count, then compare steps to reach perplexity 20. If the 21.0% gap disappears or shrinks drastically, the gain comes from extra communication, not from delay compensation and adaptive selection.","tokens_in":10375,"feed_emoji":"🌐","tokens_out":12234,"duration_ms":95883,"temperature":0.7,"pith_summary":"The paper argues that the two problems introduced by overlapping communication with computation in cross-region language-model training—stale global updates and partial model synchronization—can be explicitly corrected, and that doing so makes distributed training faster and more stable. It proposes CoCoDC, which adds a Taylor-expansion delay-compensation step that estimates what the global update would have become during the overlap delay, and an adaptive transmission scheduler that syncs fast-changing model fragments more often. In a simulated four-worker cross-region setup on a 150M-parameter LLaMA-style transformer trained on C4, CoCoDC reaches perplexity 20.0 in 10,292 steps, 21.0% fewer than Streaming DiLoCo and 4.9% fewer than DiLoCo, with a slightly lower final perplexity. If correct, this shows that the wall-clock cost of cross-region training can be lowered without sacrificing convergence quality.","feed_headline":"Cross-region LLM training gets 21% fewer steps to target perplexity","feed_subtitle":"Taylor-expansion delay correction plus adaptive fragment sync beats both DiLoCo baselines on speed and final perplexity.","key_machinery":"The carrying object is the delay-compensated update in Eq. (8): after receiving the stale global state $\\theta^g_{p,t_p}$, the worker computes its observed change rate $g = (\\theta^m_{p,t_p}-\\theta^m_{p,t_l})/\\tau$, corrects it to $g_{\\mathrm{corr}} = g + \\lambda (g \\odot g) \\odot (\\Delta\\theta/H)$ using the per-step divergence $\\Delta\\theta/H$ from the global state, and applies $\\theta^m_{p,t_l} \\leftarrow \\theta^g_{p,t_p} + g_{\\mathrm{corr}}\\,\\tau$. The $\\odot$ term is a Fisher-information-motivated stand-in for the Hessian that would otherwise be intractable. The second mechanism is the selection rule $R_p = \\|\\Delta\\theta^g_p\\|_2 / I_p$: the next fragment to synchronize is the one with the largest average change rate since its last sync, with a hard rule that any fragment unsynced for $H$ steps is chosen first. Together they convert a stale, partially synchronized update into an approximation of the current ideal global state while using idle bandwidth for the fragments that need it most.","core_discovery":"The paper's central claim is that the staleness and inconsistency introduced by overlapping communication with computation can be corrected well enough that low-communication cross-region training converges at least as well as blocking synchronization while using less wall-clock time. CoCoDC's delay compensation estimates the ideal global update at the current local step by a first-order Taylor expansion around the received global state, with the intractable Hessian replaced by a scaled outer product of local parameter-change rates. Its adaptive transmission then synchronizes the fragment with the largest averaged global pseudo-gradient norm first, subject to a maximum staleness of $H$ steps. In the reported experiments this yields faster convergence to perplexity 20 and a better final validation perplexity than both DiLoCo and Streaming DiLoCo, which the paper reads as evidence that explicit staleness handling restores the quality lost by overlap.","pith_inferences":["The head-to-head comparison is not mechanism-matched: with $\\gamma=0.4$, Eq. (9) gives 8 fragment synchronizations per $H=100$ steps for CoCoDC while Streaming DiLoCo performs $K=4$, so the 21.0% step reduction could partly reflect a higher update frequency rather than the compensation and adaptive selection themselves.","The $\\lambda$-scaled term $\\lambda (g \\odot g) \\odot (\\Delta\\theta/H)$ is structurally a diagonal natural-gradient correction, so a natural extension is to connect $\\lambda$ to the outer optimizer's momentum and learning rate and to test whether a time-varying $\\lambda$ helps as the overlap depth grows.","Because fragment selection uses globally averaged pseudo-gradients, the no-coordination property assumes workers observe the same $\\Delta\\theta^g_p$ history; under heterogeneous data or stragglers, local $R_p$ estimates can diverge, weakening the deterministic selection argument."],"forward_implications":["In the simulated 4-worker, 150M-parameter setting, CoCoDC reaches perplexity 20.0 in 10,292 steps, 21.0% fewer than Streaming DiLoCo (13,021) and 4.9% fewer than DiLoCo (10,821).","CoCoDC reaches a lower final validation perplexity (18.0357) than both DiLoCo (18.1608) and Streaming DiLoCo (18.6457), indicating the correction does not trade final quality for speed.","Because communication and computation overlap, CoCoDC avoids the blocking all-reduce wait of DiLoCo, so wall-clock time per step is lower even when the number of steps is similar.","The paper expects the benefits to grow under more aggressive cross-region conditions, where higher latency forces larger overlap depth and longer local training intervals and therefore stronger staleness and inconsistency."],"supporting_citations":[{"why":"Defines DiLoCo's H-step pseudo-gradient synchronization protocol that CoCoDC inherits and compares against.","marker":"[12]"},{"why":"Introduces Streaming DiLoCo's fragment-wise streaming synchronization and overlap, the main baseline CoCoDC claims to outperform.","marker":"[14]"},{"why":"Supplies the Taylor-expansion delay-compensation idea for stale gradients that CoCoDC adapts to pseudo-gradients with H greater than 1.","marker":"[20]"},{"why":"Provides another delay-compensation scheme for asynchronous training that motivates correcting stale updates before applying them.","marker":"[22]"},{"why":"Supports approximating the Hessian action by the gradient outer product through the Fisher information connection.","marker":"[23]"},{"why":"Grounds the gradient-outer-product approximation in natural-gradient theory, supporting the lambda-scaled correction term.","marker":"[24]"},{"why":"Provides the distributed communication primitives used to emulate ring all-reduce across the simulated workers.","marker":"[25]"},{"why":"Defines the LLaMA-style transformer architecture used for the 150M-parameter evaluation model.","marker":"[26]"},{"why":"Supplies the C4 training and validation corpus on which loss and perplexity are measured.","marker":"[27]"}],"fun_headline_variants":["CoCoDC: 21% fewer training steps for cross-region LLMs","Cross-region LLM training: 21% speedup via delay correction","Taylor-expansion fix cuts cross-region LLM training steps by 21%","Adaptive sync + delay comp: cross-region LLMs converge 21% faster","Overlap without staleness: cross-region LLM training saves 21% steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison credits the speedup to CoCoDC's two mechanisms even though the configuration synchronizes fragments roughly twice as often as Streaming DiLoCo, and the paper provides no ablation that isolates the mechanisms at a matched synchronization frequency.","fun_headline_variants_meta":{"raw":{"variants":["CoCoDC: 21% fewer training steps for cross-region LLMs","Cross-region LLM training: 21% speedup via delay correction","Taylor-expansion fix cuts cross-region LLM training steps by 21%","Adaptive sync + delay comp: cross-region LLMs converge 21% faster","Overlap without staleness: cross-region LLM training saves 21% steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":2171,"prompt_tokens":947,"completion_tokens":1224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1117}},"tokens_in":563,"tokens_out":1224,"duration_ms":8712,"temperature":1.0,"reasoning_tokens":1117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:34:07.727318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an experiment that matches synchronization counts: give Streaming DiLoCo the same number of fragment synchronizations per $H$ that CoCoDC uses (8 per 100 in the reported configuration) and give CoCoDC a round-robin schedule with that same count, then compare steps to reach perplexity 20. If the 21.0% gap disappears or shrinks drastically, the gain comes from extra communication, not from delay compensation and adaptive selection.","supporting_citations":[{"cited_title":"Asynchronous Stochastic Gradient Descent with Delay Compensation,","cited_arxiv_id":null,"evidence_quote":"Supplies the Taylor-expansion delay-compensation idea for stale gradients that CoCoDC adapts to pseudo-gradients with H greater than 1."},{"cited_title":"Asynchronous Training Schemes in Distributed Learning with Time Delay,","cited_arxiv_id":null,"evidence_quote":"Provides another delay-compensation scheme for asynchronous training that motivates correcting stale updates before applying them."},{"cited_title":"The elements of statistical learning. vol. 1 Springer series in statistics,","cited_arxiv_id":null,"evidence_quote":"Supports approximating the Hessian action by the gradient outer product through the Fisher information connection."},{"cited_title":"Natural Gradient Works Efficiently in Learning,","cited_arxiv_id":null,"evidence_quote":"Grounds the gradient-outer-product approximation in natural-gradient theory, supporting the lambda-scaled correction term."},{"cited_title":"Pytorch distributed: Experiences on accelerating data parallel training,","cited_arxiv_id":null,"evidence_quote":"Provides the distributed communication primitives used to emulate ring all-reduce across the simulated workers."},{"cited_title":"Llama: Open and efficient foundation language models,","cited_arxiv_id":null,"evidence_quote":"Defines the LLaMA-style transformer architecture used for the 150M-parameter evaluation model."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer,","cited_arxiv_id":null,"evidence_quote":"Supplies the C4 training and validation corpus on which loss and perplexity are measured."}],"review_version":1}