{"id":"a086b5b8-9a3e-4ea7-a413-cd06d22cc126","arxiv_id":"2412.07210","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EDiT trains LLMs with local SGD plus model sharding, adding a pseudo-gradient penalty to stabilize loss; it reports modest quality gains and higher throughput, though a proof of convergence contains an error.","lead":"EDiT is a new method for training large language models across many GPUs, combining local updates with model sharding so workers communicate less often. It reports faster training and comparable or better model quality than existing methods, with an asynchronous variant for uneven clusters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof uses β_t ≥ 1, but Eq. 4 defines β_t ≤ 1, so the claimed O(log T/√T) convergence rate is not established.","rationale":"The most load-bearing concern is the invalid convergence proof: the central theoretical claim of an O(log T/√T) rate rests on a false bound on β_t. This is a concrete, checkable error and is more decisive than the empirical fragility noted by the reader, because it does not depend on experimental variance. The reader's weakest assumption focused on the pseudo-gradient EMA z-test and separately noted that β_t is treated as deterministic; my concern is a distinct but closely related mathematical gap (the β_t ≥ 1 lower bound), so agreement is partial rather than full. I did not choose the lack of error bars as the primary concern, although it is real: the FineWeb-Edu loss gap between EDiT and Baseline is 0.004, and without multiple seeds or confidence intervals the claim that EDiT 'surpasses' Baseline is not statistically supported. That issue would also need to be addressed for a fully convincing empirical claim. The verdict remains CONDITIONAL because the method is plausible, code is available, and the empirical trends are mostly consistent; however, Theorem 1 should be corrected or removed, and the sensitivity of the pseudo-gradient hyperparameters (δ, α, ϕ) should be tested before the stated convergence guarantee is accepted.","tokens_in":21502,"tokens_out":6560,"duration_ms":71834,"concrete_test":"Independently re-derive the step from Eq. 17 to Eq. 18 in Appendix A.4 without invoking β_t ≥ 1, replacing it with the correct 0 < β_t ≤ 1 from Eq. 4, and check whether any O(log T/√T) bound can still be recovered. As a complementary check, log β_t values during a representative EDiT run (e.g., Llama 1B on FineWeb-Edu) and report the fraction of synchronization steps with β_t < 1; if that fraction is positive, the proof's premise is violated in the very regime the theorem is meant to cover.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 1 in Appendix A.4 contains a reversed inequality that breaks the argument. Equation 4 defines the clip coefficient as β_t = min(ϕ/(Ḡ_t + ε), 1), so by construction 0 < β_t ≤ 1. However, between Eq. 17 and Eq. 18 the proof states 'Since from Equation 4, we have 1 ≤ β_t ≤ ϕ/ε'. The lower bound is false. Eq. 17 upper-bounds a sum of ν η_{t,p} β_t E[‖∇L(θ_{t,p})‖²]; passing to a bound on Σ η E[‖∇L‖²] requires β_t ≥ 1, not β_t ≤ 1. Because β_t can be arbitrarily close to zero when the synchronized pseudo-gradient norm is large, the displayed O(log T/√T) bound does not follow from the preceding inequalities. A related issue is Eq. 12, which treats E[β_t Σ_j w_{t,j} g_{t,p}^{(j)}] as β_t E[g_{t,p}]; β_t and the data-dependent weights w_{t,j} depend on pseudo gradients accumulated over the last τ local steps, which include g_{t,p}, so the expectation does not factor as written. The convergence theorem is therefore unsupported as a proof, even though the empirical method may still be sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EDiT, a Local-SGD-based distributed training method for large language models that combines model sharding with layer-wise parameter synchronization during the forward pass, a pseudo-gradient penalty strategy (anomaly elimination, weighted averaging, and gradient clipping), and a fully asynchronous variant A-EDiT. The authors report experiments on Llama models from 350M to 7B parameters on FineWeb-Edu and an in-house dataset, comparing against Baseline, Post Local SGD, DiLoCo, and CO2/CO2*. They claim lower training loss and validation perplexity than the baselines, higher throughput, better robustness to stragglers and bandwidth constraints, and favorable elastic-training behavior. Section 5 states a convergence theorem with rate O(log T / sqrt(T)) for EDiT, proved in Appendix A.4.","tokens_in":21767,"tokens_out":3903,"duration_ms":42812,"significance":"If the empirical results are reproducible and the convergence claim is valid, EDiT is a practically relevant contribution: it addresses a real gap in making Local SGD compatible with model sharding for LLM training, releases code, evaluates at scales up to 7B, and includes a useful ablation of its stability mechanisms. The main theoretical claim, however, is currently unsupported because of a clear error in the proof, and the empirical superiority over the Baseline rests on very small differences without variance estimates. The significance is therefore conditional: the engineering contribution is credible, but the paper's stronger claims need correction and additional statistical support.","major_comments":[{"comment":"The proof of Theorem 1 contains a reversed inequality that invalidates the argument. Equation (4) defines beta_t = min(phi / (Gbar_t + epsilon), 1), so by construction 0 < beta_t <= 1, yet the proof states 'Since from Equation 4, we have 1 <= beta_t <= phi/epsilon'. The lower bound is false, and beta_t may be arbitrarily close to zero when the synchronized pseudo-gradient norm is large. The step from Eq. (17) to Eq. (18) drops beta_t from the left-hand side under the sum, which requires beta_t >= 1, not beta_t <= 1. Consequently the displayed O(log T / sqrt(T)) bound does not follow from the preceding inequalities. The theorem must either be reproved with a valid lower bound on beta_t or replaced by a weaker statement that accounts for the clipping factor.","section":"Appendix A.4, Eqs. (4) and (17)-(18)"},{"comment":"The equality E[beta_t sum_j w_{t,j} g_{t,p}^{(j)}] = beta_t E[g_{t,p}] is not justified. The coefficient beta_t and the weights w_{t,j} depend on pseudo-gradients accumulated over the last tau local steps, which include g_{t,p} itself; they are not constants or independent of the current gradient. The expectation therefore does not factor as written. This is a second load-bearing gap in the proof of Theorem 1, and it must be fixed before the convergence claim can be accepted.","section":"Appendix A.4, Eq. (12)"},{"comment":"The claim that EDiT 'even surpasses the Baseline' on FineWeb-Edu rests on differences of 0.004 in training loss (2.848 vs. 2.852) and 0.04 in validation PPL (17.38 vs. 17.42). These values are reported as averages of the last 10 points, with no standard deviations, no multiple seeds, and no significance test. At LLM pretraining scale, such differences are within typical run-to-run noise. The authors should either provide repeated-run statistics or temper the claim to 'comparable to Baseline' for these metrics.","section":"Section 4.2, Figure 4"}],"minor_comments":[{"comment":"The anomaly elimination mechanism relies on fixed hyperparameters delta=3 and alpha=0.02, and the gradient clip threshold is set to phi=10, but no sensitivity analysis is provided. The paper should report how performance varies with these values, since the stability gains claimed for the pseudo-gradient penalty depend on them.","section":"Section 3.2"},{"comment":"The notation in Theorem 1 and Appendix A.4 suppresses the dependence of beta_t, w_{t,j}, and the pseudo-gradient norms on the module index l and on the worker index i. This obscures the fact that the clipping coefficient is computed per module per synchronization event. The theorem should make these dependencies explicit.","section":"Section 5, Theorem 1"},{"comment":"Throughput and TFLOPS are reported as averages over 100 steps without variance or repeated measurements. A brief statement about measurement noise would strengthen the acceleration claims, especially for the small differences between EDiT and CO2 on the 350M model.","section":"Section 4.3, Table 2"},{"comment":"The elastic-training conclusion is based on single-curve comparisons of validation PPL for Baseline and EDiT. Because the final gaps are modest and no repeated experiments are reported, the claim that EDiT 'validates' the learning-rate invariance hypothesis should be phrased more cautiously.","section":"Section 4.4, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's main theoretical contribution is not yet reliable because of the two proof gaps in Appendix A.4. The empirical contribution is plausible but would benefit from variance reporting. These are fixable within the scope of a revision, so I do not recommend rejection, but the convergence theorem should not be cited as established in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the systems contribution is worth engaging with; the convergence theorem is not. EDiT gives a practical recipe for running local SGD on sharded LLMs, and the 7B experiment is the first serious large-scale test of that class of methods. But the proof of Theorem 1 has a load-bearing inequality error, and the reported quality gains over the baseline are within noise.\n\nWhat's new: combining FSDP-style sharding with periodic synchronization, layer-wise sync in the forward pass with prefetch, and a pseudo-gradient penalty that does anomaly elimination, weighted averaging, and clipping. That combination is genuinely new for LLM-scale local SGD. The code is available, the ablation shows each penalty component matters, and the elasticity observation — optimal LR independent of worker count — is interesting and useful.\n\nWhere it falls down: the theory. Equation 4 defines β_t = min(ϕ/(Ḡ_t+ε),1), so β_t ≤ 1. The proof between Eq. 17 and Eq. 18 asserts 1 ≤ β_t ≤ ϕ/ε and uses that to pass from a bound on Σ ν η β E‖∇L‖² to a bound on Σ η E‖∇L‖². The lower bound is false; the direction of the inequality is exactly backwards. Equation 12 also factors E[β_t Σ w_j g_j] as β_t E[g], but β_t and w_j depend on the same pseudo-gradients being summed. So the O(log T/√T) claim is not established. That is a real gap, not a nitpick.\n\nThe empirical evidence is also thinner than the abstract suggests. On FineWeb-Edu, EDiT's final loss is 2.848 vs baseline 2.852, and PPL 17.38 vs 17.42. Those differences could be noise; there are no seeds or error bars, and the in-house dataset is private. The throughput and straggler results are more solid — 5% throughput gain at 7B and graceful degradation under lag are plausible from the design.\n\nRisk of overreaction: the broken proof doesn't make the method invalid. The empirical story stands on its own, and the techniques are reasonable. But the authors should either fix the proof or clearly state the convergence result as conjectural/empirically observed.\n\nWho should read this: anyone building communication-efficient training for 1B-7B LLMs. It deserves a proper referee because the engineering contribution is real and the experiments, while thin on statistics, are at the right scale. I would send to review but require correcting or removing the theorem and adding variance estimates.","headline":"Worth reading for the systems design; don't rely on the convergence proof — the theorem's bound is unsupported by its own equations.","tokens_in":22355,"tokens_out":3532,"would_cite":true,"duration_ms":35811,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68W15"],"pacs":[],"model":"deepseek-v4-flash","headline":"EDiT claims a sharded Local SGD recipe that beats full-sync LLM training on loss with a 19ms sync delay.","keywords":["EDiT","Local SGD","distributed training","model sharding","large language models","asynchronous training","pseudo gradient penalty","elastic training"],"falsifier":"Log the per-worker pseudo-gradient norms over a real pre-training run and compare how often the EMA z-test flags workers against the outlier rate a Gaussian tail at $\\delta=3$ implies: if healthy workers are flagged more than a fraction of a percent of the time, or genuine anomalies are missed, the anomaly elimination is miscalibrated and the stability gains do not follow. A separate check targets the proof: record the sequence of clip coefficients $\\beta_t$ across steps; the appendix's Equation 12 treats $\\beta_t$ as deterministic, so if measured $\\beta_t$ shows non-negligible variance under identical conditioning, the stated $O(\\log T/\\sqrt{T})$ rate is not established by the argument given.","tokens_in":21271,"feed_emoji":"⚡","tokens_out":18902,"duration_ms":144543,"temperature":0.7,"pith_summary":"EDiT's aim is to make Local SGD practical for pre-training large language models, where earlier local-update methods fail because they keep full model copies in memory, add non-overlapped communication, and destabilize on noisy web-scale data. The paper combines Local SGD with parameter sharding on a two-dimensional device mesh: workers synchronize sharded parameters layer by layer during the forward pass, overlapping sync with computation, and keep all extra optimizer state sharded and CPU-offloadable. At each synchronization point a pseudo-gradient penalty rejects outlier workers by an EMA z-test, reweights the rest by their pseudo-gradient norms, and clips the aggregated update, which the authors argue suppresses loss spikes and even turns worker diversity into better updates. In Llama experiments up to 7B parameters, EDiT posts the lowest training loss on two datasets and the lowest validation perplexity on FineWeb-Edu, beating the fully synchronous baseline, with only 19ms of added sync delay, and the asynchronous A-EDiT variant keeps throughput high under a consistent straggler. If these results hold, EDiT is a drop-in replacement for synchronous LLM pre-training that needs far less communication, tolerates stragglers, and supports elastic resizing without learning-rate retuning.","feed_headline":"EDiT out-trains full-sync LLM training with a 19ms sync delay","feed_subtitle":"Sharded Local SGD with pseudo-gradient penalties posts the lowest training loss at a fraction of the sync cost.","key_machinery":"The load-bearing mechanism is the pseudo-gradient penalty applied inside the Sync() routine at each synchronization step. Every worker computes the pseudo-gradient $\\Delta^{(i,l)}_t = \\theta^{(i,l)}_{t,\\tau} - \\theta^{(i,l)}_t$; its norm $G^{(i,l)}_t$ is screened by an EMA z-score (Eq. 1) with fixed threshold $\\delta = 3$ and decay $\\alpha = 0.02$; flagged workers are eliminated; the survivors are combined with weights $w_{t,i} = \\exp(-G^{(i,l)}_t) / \\sum_j \\exp(-G^{(j,l)}_t)$ (Eq. 2), clipped by $\\beta_t = \\min(\\phi/(\\bar{G}_t + \\epsilon), 1)$ (Eq. 4), and fed to the outer Nesterov optimizer, with a rollback to the last synchronized parameters if every worker is flagged. Around this sits the engineering core: the $M \\times N$ device mesh with model-shard groups handling all-gather/reduce-scatter for each layer and model-sync groups exchanging only their shard of the parameters during the forward pass, which is what converts synchronization from a batch-wide stall into an overlapped, memory-cheap operation. The convergence theorem then analyzes this Sync operation as a clipped, weighted average of inner SGD updates.","core_discovery":"The paper's central claim is that the two things that kept Local SGD out of LLM training—memory blowup from full-model copies and synchronization overhead—can both be removed by sharding parameters along one mesh dimension and synchronizing periodically along the other, so that each worker holds exactly its shard at all times. Synchronization then happens per layer at the start of the forward pass with a prefetch of the next layer, making the periodic sync nearly free (19ms versus 160ms for Post Local SGD) and letting the method scale to 7B parameters where Post Local SGD, DiLoCo, and CO2 run out of memory. The second claim is that training stability on noisy corpora can be bought explicitly: the pseudo-gradient penalty filters each worker's accumulated parameter change at sync time, eliminating statistical outliers, down-weighting large-norm workers, and clipping the combined update, so that loss spikes are suppressed and EDiT recovers faster than DiLoCo, whose workers spike together. The paper reports EDiT reaching the lowest training loss on both the FineWeb-Edu and in-house datasets and the lowest validation perplexity on FineWeb-Edu (17.38, below the Baseline's 17.42), with Theorem 1 giving a convergence rate of $O(\\log T/\\sqrt{T})$ to a stationary point.","pith_inferences":["If the EMA z-test is well calibrated, the per-layer anomaly flags and weights it produces could serve as a free training-time diagnostic for data quality: workers repeatedly flagged as anomalous are likely consuming low-quality shards, which suggests using EDiT's penalty statistics to drive data curation or resharding.","The fixed thresholds ($\\delta=3$, $\\alpha=0.02$) are the fragile constants of the method; on corpora with heavy-tailed or non-stationary gradient norms—say, during a learning-rate change or a domain shift—the z-test will misclassify healthy workers, so an adaptive or distribution-free anomaly rule is the natural next experiment.","Because A-EDiT switches the sync trigger from steps to wall-clock time, lagging workers accumulate systematically different numbers of local updates; an untested consequence is that their effective learning rates diverge, and the anomaly filter may end up masking exactly the stale updates that time-based sync creates.","The elasticity result hints at a cheaper hyperparameter search protocol for sharded Local SGD generally: tune on a single worker, then scale out without changing the learning rate, a recipe the paper demonstrates only for EDiT and that would be worth testing with other outer optimizers."],"forward_implications":["EDiT extends the reach of Local SGD to model scales that defeat its predecessors: Post Local SGD, DiLoCo, and CO2 all run out of memory at 3B or below, while EDiT trains 3B and 7B Llama models at higher throughput than the synchronous baseline.","Because the optimal learning rate stays fixed as workers are added (Fig. 6), EDiT supports elastic training: scaling workers from 1 to 16 or back does not require re-tuning, and in simulated 1-2-4-8 and 8-4-2-1 schedules EDiT beats Baseline PPL by 4.5% and 2.6%.","The pseudo-gradient penalty makes Local SGD robust to low-quality corpus shards: on the noisy in-house dataset, where DiLoCo's performance visibly declines, EDiT stays within noise of the Baseline on loss and beats it on average benchmark score.","A-EDiT, which synchronizes on wall-clock time rather than step count, keeps TFLOPS near the straggler-free level when one node is consistently slow, while the step-synchronous EDiT and Baseline degrade sharply.","Theorem 1 places EDiT's convergence at $O(\\log T/\\sqrt{T})$ under $L$-smoothness, bounded gradients, and unbiased noise, matching the standard Local SGD rate despite the additional penalty, weighting, and clipping."],"supporting_citations":[{"why":"Foundational Local SGD paradigm of parallel local updates with periodic averaging that EDiT extends.","marker":"Zhang et al., 2016"},{"why":"Post Local SGD, the warmup-then-local-sync baseline whose 160ms sync overhead EDiT improves to 19ms.","marker":"Lin et al., 2019"},{"why":"DiLoCo, the main Local-SGD-for-LLM baseline; supplies the AdamW-inner/Nesterov-outer setup EDiT reuses.","marker":"Douillard et al., 2023"},{"why":"CO2/CO2*, the overlap-and-memory baseline that EDiT matches in throughput while avoiding OOM.","marker":"Sun et al., 2023"},{"why":"ZeRO, the model-sharding framework EDiT is built on to make Local SGD memory-feasible for LLMs.","marker":"Rajbhandari et al., 2020"},{"why":"SlowMo, whose theoretical framework underpins the proof of Theorem 1's convergence rate.","marker":"Wang et al., 2019"},{"why":"Provides the FineWeb-Edu dataset used in the main convergence, PPL, and benchmark experiments.","marker":"Lozhkov et al., 2024"},{"why":"Source of the norm-based reweighting idea behind EDiT's weighted averaging of pseudo-gradients.","marker":"Thakkar et al., 2023"},{"why":"muP initialization, which lets hyperparameters transfer from 350M to 7B and enables the scale-up experiments.","marker":"Yang et al., 2021"},{"why":"Asynchronous Local SGD for language modeling, framing the staled-model problem A-EDiT must handle.","marker":"Liu et al., 2024"}],"fun_headline_variants":["EDiT: Local SGD that scales to 7B with 19ms sync","Sharded Local SGD: 19ms sync, lowest training loss","Pseudo-gradient penalty: EDiT tames loss spikes","A-EDiT: Fully async LLM training for heterogeneous clusters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the EMA z-test with fixed threshold $\\delta=3$ and decay $\\alpha=0.02$ reliably separates anomalous workers from healthy ones using only pseudo-gradient norms; the paper assumes this separation works on real, noisy, non-stationary gradient distributions and offers no sensitivity analysis of the thresholds.","fun_headline_variants_meta":{"raw":{"variants":["EDiT: Local SGD that scales to 7B with 19ms sync","Sharded Local SGD: 19ms sync, lowest training loss","Pseudo-gradient penalty: EDiT tames loss spikes","A-EDiT: Fully async LLM training for heterogeneous clusters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1716,"prompt_tokens":1069,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":568}},"tokens_in":685,"tokens_out":647,"duration_ms":7704,"temperature":1.0,"reasoning_tokens":568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:56:07.700903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Log the per-worker pseudo-gradient norms over a real pre-training run and compare how often the EMA z-test flags workers against the outlier rate a Gaussian tail at $\\delta=3$ implies: if healthy workers are flagged more than a fraction of a percent of the time, or genuine anomalies are missed, the anomaly elimination is miscalibrated and the stability gains do not follow. A separate check targets the proof: record the sequence of clip coefficients $\\beta_t$ across steps; the appendix's Equation 12 treats $\\beta_t$ as deterministic, so if measured $\\beta_t$ shows non-negligible variance under identical conditioning, the stated $O(\\log T/\\sqrt{T})$ rate is not established by the argument given.","supporting_citations":[{"cited_title":"Don't use large mini-batches, use local sgd","cited_arxiv_id":null,"evidence_quote":"Post Local SGD, the warmup-then-local-sync baseline whose 160ms sync overhead EDiT improves to 19ms."},{"cited_title":"Tuning large neural networks via zero-shot hyperparameter transfer","cited_arxiv_id":null,"evidence_quote":"muP initialization, which lets hyperparameters transfer from 350M to 7B and enables the scale-up experiments."}],"review_version":1}