{"id":"e974aed9-cb77-432b-a8e0-49ee275dfed0","arxiv_id":"2511.11505","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FarSkip-Collective makes Mixture-of-Experts communication non-blocking by feeding later layers already-available activations, and self-distillation keeps converted 16B–109B models within ~1–2.5% of the original on average.","lead":"FarSkip-Collective rewires how activations flow through Mixture-of-Experts layers so GPU-to-GPU communication runs while computation is still happening, instead of waiting for it. The authors convert 16B–109B open-source MoE models into this form and report near-original accuracy after self-distillation, plus faster training and inference.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Capability-retention claim rests on distillation runs whose early-stopping signal (MBPP+) is also in the final Tab. 1 average; held-out evaluation is needed to rule out selection bias.","rationale":"The reader's verdict is already CONDITIONAL and its weakest assumption centers on whether the FCSD distillation recipe, with MBPP+-based early stopping and no held-out evaluation, genuinely repairs the distribution shift. My stress-test identifies the same load-bearing concern, narrowed to the selection-bias mechanism: MBPP+ appears both as the early-stopping signal and as one of the eleven tasks averaged in Tab. 1. If held-out evaluation confirms the margins, the central claim is supported and I would move toward ACCEPT; until then, CONDITIONAL is the correct verdict. I do not see an internal inconsistency that would require REJECT. The paper's systems results have number inconsistencies (e.g., abstract '32.6%' vs §5.2 '1.34x'), but those are less central to the capability-retention thesis and are addressable without changing the main claim.","tokens_in":18349,"tokens_out":7287,"duration_ms":65557,"concrete_test":"Independently evaluate the released FCSD checkpoints (or, if unreleased, re-run FCSD) on a pre-registered held-out suite that was never used for early stopping or hyperparameter selection — e.g., MMLU-Pro, GPQA, LiveCodeBench, IFEval, plus the existing tasks excluding MBPP+. If the FCSD-vs-original gap on this held-out suite is within the same ~1-2.5 pt margin as Tab. 1, the selection-bias concern does not land. Additionally, rerun early stopping on a disjoint validation split (e.g., a 500-example subset of MMLU-Pro) and recompute Tab. 1 averages without MBPP+ to confirm the margins are unchanged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that all layers of 16B-109B open-source MoEs can be converted with negligible capability loss (Tab. 1) — depends on the FCSD checkpoints being representative of the recipe, not artifacts of model selection. In §5.1 the authors state that early stopping is performed on MBPP+ ('patience of 20 evaluations and performance delta of 2%'), and Tab. 1 includes MBPP+ in the 'Avg' column for every model. Thus the very benchmark used to trigger early stopping is also part of the reported average. Because the paper reports a single checkpoint per model and no error bars or seeds, a reader cannot tell how much of the 'within 1%' (Llama 4 Scout) or 'within 2.5%' margins is due to selection on that benchmark. This is not an accusation of cherry-picking; it is a standard selection-bias concern. The same issue affects the hyperparameter sweeps: LR/BS are chosen by short sweeps (§5.1), and while selection uses training loss, the early-stopping criterion uses an evaluation set. If the final checkpoint is chosen when MBPP+ stops improving, then MBPP+ performance is optimistically biased, and the average that contains it is too. The recovery premise may be true, but the current Tab. 1 evidence does not isolate it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FarSkip-Collective, a connectivity modification for MoE layers that replaces each sub-block's input with an outdated or partial activation, so that dispatch/combine/all-reduce communication can run concurrently with computation. It introduces FCSD, a KL self-distillation recipe against the frozen original checkpoint, and reports that fully converted DeepSeek-V2-Lite (16B), Qwen-3-30B, and Llama-4 Scout (109B) retain within 2.5, 2.2, and 0.9 points of the original instruction-tuned models on an 11-benchmark average. It also reports overlapped Megatron-LM training (88–89% all-to-all overlap) and vLLM/SGLang inference speedups (up to 1.34x prefill). The central claim is that the modified connectivity is viable at up to 109B scale without material capability loss.","tokens_in":18708,"tokens_out":11376,"duration_ms":90751,"significance":"If the capability-retention result holds, this is a practically useful contribution: it demonstrates at three scales, including a 109B model, that a simple dependency drop followed by self-distillation against the original checkpoint can recover most of an instruction-tuned MoE's accuracy. The SFT baseline and the no-training degradation curve in Fig. 3 support the need for the distillation, and the 50B-token pretraining control in Appendix A is a valuable check that the architecture itself is trainable without a teacher. The reported overlap numbers and speedups, if reproducible, would also be relevant for large-scale MoE training and serving. The paper does not currently ship code or checkpoints, but the evaluation is systematic across eleven benchmarks and three model families.","major_comments":[{"comment":"The early-stopping signal for FCSD is MBPP+ ('patience of 20 evaluations and performance delta of 2%'), and MBPP+ is also one of the eleven entries in the average used to claim capability retention. This makes the reported average partially circular: the checkpoint is selected to be good on a benchmark that is then included in the reported evaluation. I verified that deleting the MBPP+ column from Tab. 1 does not change the qualitative conclusion (the average gaps remain roughly 2.6, 1.5, and 0.9 points), so the result is not an artifact of that single column; however, selection on MBPP+ can in principle also affect the other ten entries, and with a single checkpoint per model and no seeds or error bars, the reader cannot quantify this. Please report performance on a held-out validation set not used for early stopping, or exclude MBPP+ from the reported average, or provide multiple seeds","section":"§5.1, Tab. 1"},{"comment":"The headline inference numbers do not match the body. The abstract states '32.6% speedup in Time To First Token when serving a converted DeepSeek-V3 architecture ... in SGLang and achieve 97.3% communication-computation overlap during the prefill stage'; the body reports 97.6% all-reduce overlap for DeepSeek-V2 in vLLM (not SGLang/DeepSeek-V3), and Fig. 6 reports SGLang prefill speedups of 1.18x–1.34x depending on context/batch, not a single 32.6% number. Please align the abstract with the measured configuration and, if overlap percentage for the SGLang/DeepSeek-V3 setting is claimed, report it explicitly.","section":"Abstract vs §5.2 (Tab. 3, Fig. 6)"},{"comment":"The FCSD recipe as stated is not fully specified. The final number of training tokens per model is only given as 'up to 10B'; the batch-size and learning-rate sweeps are described but the selected values for each model are not reported; and no seed information is given. Because the main claim depends on the final checkpoints being representative of a reproducible recipe, please provide per-model token counts, the selected hyperparameters, and the number of seeds. Also, Tab. 2 contains duplicate rows for 'KL (FAR100%)' and 'KL' with identical values; clarify whether this is intentional or a typo.","section":"§5.1, FCSD recipe"}],"minor_comments":[{"comment":"Equation (6) is missing the initial residual term o0; as written it defines o_k as a sum of sub-block outputs without the embedding/residual base. This is presumably a typo, but it makes the formula incorrect as stated.","section":"Eq. (6)"},{"comment":"The conclusion states that the models are converted 'while retaining 99% of the model accuracy'. The reported averages imply retention of about 96.1% for DeepSeek-V2-Lite, 97.1% for Qwen-3-30B, and 98.8% for Llama-4 Scout. Please either soften the claim ('within 2.5% on average') or restrict it to the Llama-4 Scout result.","section":"Abstract/§7"},{"comment":"References [23] and [24] are the same paper (Liu et al., 'Is your code generated by ChatGPT really correct?'). Please merge.","section":"References"},{"comment":"The 'KL + INTER. L2' row reports a single number after a sweep over L2 coefficients, but the coefficient value or range is not given. This makes the comparison hard to reproduce.","section":"Tab. 2"},{"comment":"The legend labels 'Reg. FarSkip' are ambiguous; it should read 'Regular' and 'FarSkip-Collective' for clarity.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's main capability claim is likely defensible, but the early-stopping-on-MBPP+ issue touches the central evidence and should be fixed before acceptance. I would ask for a held-out evaluation (or removal of MBPP+ from the reported average), reconciliation of the abstract's speedup/overlap numbers with the body, and the missing recipe details. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real extension, not just a rehash. Ladder-Residual and Kraken already had the outdated/partial activation idea for dense models with tensor parallelism, but Dukler et al. take it to all layers of MoE expert parallelism, develop a self-distillation recipe that plausibly repairs the distribution shift, and report evals at 16B-109B. The systems section is also competently done: 88%+ overlap on all-to-all communication in training, meaningful TTFT gains in vLLM/SGLang, execution traces, and implementation details that look reproducible in principle.\n\nWhat I trust: Table 1 is internally consistent. FCSD tracks the original within 1-2.5 points on an 11-benchmark average across three models, the SFT baseline is much worse, and the 50B-token pretraining run shows the modified architecture does not collapse from scratch. That is genuine supporting evidence, not padding.\n\nWhere it gets soft: MBPP+ is used for early stopping in Section 5.1 AND included in the Tab. 1 averages. That is a textbook selection-bias leak. With one checkpoint per model and no seeds or error bars, the reported margins are optimistic by an unknown amount. This does not kill the paper, but the capability-retention claim needs held-out evaluation—drop MBPP+ from the final average or use a separate validation set for early stopping. My guess is the result survives, but as written you cannot tell.\n\nAlso: the speedup numbers are not reconciled. Abstract says 32.6% TTFT speedup and 97.3% overlap; the body says up to 18.5% for vLLM, 1.34x for SGLang, and 95.3/97.6 overlap; training overlap is 88.9 in the abstract and 88.4 in the body. These may all be consistent under different settings, but the paper does not say so. And despite the promise to open-source, no code or checkpoints are currently available; for a systems paper that lowers confidence.\n\nThe literature coverage is fine; prior work is cited fairly. This is not a theory paper and no one should pretend otherwise.\n\nVerdict: send it to peer review. A serious reviewer should focus on the validation protocol and number consistency, not on whether the idea is sound—it mostly is. I would cite it once artifacts exist.","headline":"Worth refereeing, with one real methodology caveat: early stopping on MBPP+ contaminates the headline averages.","tokens_in":19191,"tokens_out":2056,"would_cite":true,"duration_ms":20593,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FarSkip-Collective rewires MoE layers to overlap communication with computation, retaining accuracy within 1% on a 109B model.","keywords":["Mixture of Experts","communication overlap","expert parallelism","knowledge distillation","self-distillation","MoE inference","all-to-all communication","LLM serving"],"falsifier":"A replication attempt that converts a 100B+ MoE (e.g., DeepSeek-V3) to FarSkip-Collective, then evaluates the model on the same eleven benchmarks but additionally on held-out tasks not used in early stopping (e.g., GPQA, MATH-500, or a code benchmark other than MBPP+); if the average drop exceeds the claimed ~1% on those held-out tasks, the capability-retention claim would be falsified.","tokens_in":18274,"feed_emoji":"⚡","tokens_out":1304,"duration_ms":13908,"temperature":0.7,"pith_summary":"This paper claims that the blocking communication costs in Mixture of Experts models—the all-to-all dispatches and combines, and all-reduce calls—can be eliminated by changing the model's residual connectivity, without sacrificing capabilities at scale. The authors modify every layer of three open-source MoE models (16B to 109B) so each sub-block starts from a slightly outdated or partial activation, then use self-distillation against the original checkpoint to recover performance. They report fully converted Llama 4 Scout retains on average 75.1 vs 76.0 across eleven benchmarks, with similar results for DeepSeek-V2-Lite and Qwen-3-30B. They also implement the overlap explicitly in Megatron-LM, vLLM, and SGLang, achieving up to 97% communication-computation overlap in prefill and double-digit TTFT speed-ups.","feed_headline":"Rewired MoE layers run all collectives in parallel and keep accuracy","feed_subtitle":"Llama 4 Scout stays within 1% of its original release after every layer is converted and self-distilled.","key_machinery":"FarSkip-Collective connectivity (the modified residual graph) and FCSD (FarSkip-Collective Self-Distillation). The connectivity chooses for each layer an available activation that is either 'outdated' (previous layer output) or 'partial' (accumulating the attention and shared-expert outputs but not the routed experts), making the next sub-block's input independent of in-flight communication. FCSD is a training recipe that uses the original model as a fixed teacher, KL divergence as the loss, and MBPP+ early stopping to stabilize training and recover distributional alignment.","core_discovery":"FarSkip-Collective replaces the standard residual connection into each sub-block with a 'far-skipped' input: the attention sub-block feeds on the partial activation that includes the shared-expert output but not the routed experts, and the MoE sub-block feeds on the previous layer's full output without the latest attention contribution. This removes the dependency chains that make Dispatch and Combine collectives blocking, so computation can proceed during communication. The paper's central claim is that this rewiring, despite being an architectural change, does not inherently reduce model capability: after a self-distillation recipe (FCSD) using KL divergence to the frozen original model on","pith_inferences":["The dependency-dropping trick suggests a broader principle: residual networks may tolerate bounded 'staleness' in the residual stream, raising the question of how far the lag can be pushed—the paper only tests one-block lag and leaves multi-block variants as future work.","One could test the method on models without an available instruction-tuned teacher, since the recipe relies on the original checkpoint as the distillation target; a test on a base (pretrained-only) MoE would probe whether chat-tuning is a prerequisite for recovery.","The MBPP+ early-stopping proxy is a clever but narrow instability detector; extending FCSD to other tools (perplexity, validation-set KL) could make the recipe more robust in practice.","If the accuracy retention generalizes to larger MoEs (e.g., >500B parameters) and to multimodal or reasoning-tuned models, it would substantially change the cost structure of serving large sparse models."],"forward_implications":["If the claims hold, MoE training and inference can avoid the classic exposed-communication bubbles without changing the parameter layout, enabling more efficient expert parallelism at larger scales.","Converting a model to FarSkip connectivity requires no retraining from scratch; a relatively cheap self-distillation pass (<10B tokens, about 100-1000x less compute than pretraining) suffices for capability recovery.","The same non-blocking pattern generalizes to both all-to-all (expert parallelism) and all-reduce (tensor parallelism) collectives, as demonstrated in Megatron-LM and in vLLM/SGLang serving.","Larger parallelism settings (e.g., EP=32) show increasing end-to-end speed-ups (up to 1.22x) as communication becomes more dominant, suggesting the method becomes more valuable as models grow.","The architecture itself trains from scratch on par with standard connectivity at 16B scale (50B tokens), pointing to viability beyond conversion of existing checkpoints."],"fun_headline_variants":["FarSkip rewires MoEs to overlap comms, keeps accuracy","Rewire MoE layers to skip blocking collectives, retain accuracy","FarSkip rewiring overlaps comm in MoEs, saves 32.6% TTFT","Converted Llama 4 Scout within 1% of original after far-skip rewiring","MoE blocks use far-skip inputs so all-to-all runs in parallel"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The recovery premise: KL self-distillation against the frozen original checkpoint, on fewer than 10B instruction tokens, with per-model learning rate and batch size tuned by short sweeps and early stopping on MBPP+, is enough to repair the distribution shift introduced by feeding outdated or partial activations into every layer.","fun_headline_variants_meta":{"raw":{"variants":["FarSkip rewires MoEs to overlap comms, keeps accuracy","Rewire MoE layers to skip blocking collectives, retain accuracy","FarSkip rewiring overlaps comm in MoEs, saves 32.6% TTFT","Converted Llama 4 Scout within 1% of original after far-skip rewiring","MoE blocks use far-skip inputs so all-to-all runs in parallel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3027,"prompt_tokens":801,"completion_tokens":2226,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2122}},"tokens_in":545,"tokens_out":2226,"duration_ms":15112,"temperature":1.0,"reasoning_tokens":2122,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T22:09:44.144299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication attempt that converts a 100B+ MoE (e.g., DeepSeek-V3) to FarSkip-Collective, then evaluates the model on the same eleven benchmarks but additionally on held-out tasks not used in early stopping (e.g., GPQA, MATH-500, or a code benchmark other than MBPP+); if the average drop exceeds the claimed ~1% on those held-out tasks, the capability-retention claim would be falsified.","supporting_citations":[],"review_version":1}