Pith. sign in

REVIEW 3 major objections 5 minor 46 references

FarSkip-Collective rewires MoE layers to overlap communication with computation, retaining accuracy within 1% on a 109B model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

FarSkip-Collective makes Mixture-of-Experts communication non-blocking by feeding later layers already-available activations, and self-distillation keeps converted 16B–109B models within ~1–2.5% of the original on average.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Worth refereeing, with one real methodology caveat: early stopping on MBPP+ contaminates the headline averages. the 3 major comments →

arxiv 2511.11505 v3 pith:I3VXMYSC submitted 2025-11-14 cs.LG

FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models

Yonatan Dukler , Guihong Li , Deval Shah , Jiang Liu , Vikram Appia , Emad Barsoum This is my paper
classification cs.LG
keywords Mixture of Expertscommunication overlapexpert parallelismknowledge distillationself-distillationMoE inferenceall-to-all communicationLLM serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the blocking communication costs in Mixture of Experts models—the all-to-all dispatches and combines, and all-reduce calls—can be eliminated by changing the model's residual connectivity, without sacrificing capabilities at scale. The authors modify every layer of three open-source MoE models (16B to 109B) so each sub-block starts from a slightly outdated or partial activation, then use self-distillation against the original checkpoint to recover performance. They report fully converted Llama 4 Scout retains on average 75.1 vs 76.0 across eleven benchmarks, with similar results for DeepSeek-V2-Lite and Qwen-3-30B. They also implement the overlap explicitly in Megatron-LM, vLLM, and SGLang, achieving up to 97% communication-computation overlap in prefill and double-digit TTFT speed-ups.

Core claim

FarSkip-Collective replaces the standard residual connection into each sub-block with a 'far-skipped' input: the attention sub-block feeds on the partial activation that includes the shared-expert output but not the routed experts, and the MoE sub-block feeds on the previous layer's full output without the latest attention contribution. This removes the dependency chains that make Dispatch and Combine collectives blocking, so computation can proceed during communication. The paper's central claim is that this rewiring, despite being an architectural change, does not inherently reduce model capability: after a self-distillation recipe (FCSD) using KL divergence to the frozen original model on

What carries the argument

FarSkip-Collective connectivity (the modified residual graph) and FCSD (FarSkip-Collective Self-Distillation). The connectivity chooses for each layer an available activation that is either 'outdated' (previous layer output) or 'partial' (accumulating the attention and shared-expert outputs but not the routed experts), making the next sub-block's input independent of in-flight communication. FCSD is a training recipe that uses the original model as a fixed teacher, KL divergence as the loss, and MBPP+ early stopping to stabilize training and recover distributional alignment.

Load-bearing premise

The recovery premise: KL self-distillation against the frozen original checkpoint, on fewer than 10B instruction tokens, with per-model learning rate and batch size tuned by short sweeps and early stopping on MBPP+, is enough to repair the distribution shift introduced by feeding outdated or partial activations into every layer.

What would settle it

A replication attempt that converts a 100B+ MoE (e.g., DeepSeek-V3) to FarSkip-Collective, then evaluates the model on the same eleven benchmarks but additionally on held-out tasks not used in early stopping (e.g., GPQA, MATH-500, or a code benchmark other than MBPP+); if the average drop exceeds the claimed ~1% on those held-out tasks, the capability-retention claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, MoE training and inference can avoid the classic exposed-communication bubbles without changing the parameter layout, enabling more efficient expert parallelism at larger scales.
  • Converting a model to FarSkip connectivity requires no retraining from scratch; a relatively cheap self-distillation pass (<10B tokens, about 100-1000x less compute than pretraining) suffices for capability recovery.
  • The same non-blocking pattern generalizes to both all-to-all (expert parallelism) and all-reduce (tensor parallelism) collectives, as demonstrated in Megatron-LM and in vLLM/SGLang serving.
  • Larger parallelism settings (e.g., EP=32) show increasing end-to-end speed-ups (up to 1.22x) as communication becomes more dominant, suggesting the method becomes more valuable as models grow.
  • The architecture itself trains from scratch on par with standard connectivity at 16B scale (50B tokens), pointing to viability beyond conversion of existing checkpoints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dependency-dropping trick suggests a broader principle: residual networks may tolerate bounded 'staleness' in the residual stream, raising the question of how far the lag can be pushed—the paper only tests one-block lag and leaves multi-block variants as future work.
  • One could test the method on models without an available instruction-tuned teacher, since the recipe relies on the original checkpoint as the distillation target; a test on a base (pretrained-only) MoE would probe whether chat-tuning is a prerequisite for recovery.
  • The MBPP+ early-stopping proxy is a clever but narrow instability detector; extending FCSD to other tools (perplexity, validation-set KL) could make the recipe more robust in practice.
  • If the accuracy retention generalizes to larger MoEs (e.g., >500B parameters) and to multimodal or reasoning-tuned models, it would substantially change the cost structure of serving large sparse models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FarSkip-Collective, a connectivity modification for MoE layers that replaces each sub-block's input with an outdated or partial activation, so that dispatch/combine/all-reduce communication can run concurrently with computation. It introduces FCSD, a KL self-distillation recipe against the frozen original checkpoint, and reports that fully converted DeepSeek-V2-Lite (16B), Qwen-3-30B, and Llama-4 Scout (109B) retain within 2.5, 2.2, and 0.9 points of the original instruction-tuned models on an 11-benchmark average. It also reports overlapped Megatron-LM training (88–89% all-to-all overlap) and vLLM/SGLang inference speedups (up to 1.34x prefill). The central claim is that the modified connectivity is viable at up to 109B scale without material capability loss.

Significance. If the capability-retention result holds, this is a practically useful contribution: it demonstrates at three scales, including a 109B model, that a simple dependency drop followed by self-distillation against the original checkpoint can recover most of an instruction-tuned MoE's accuracy. The SFT baseline and the no-training degradation curve in Fig. 3 support the need for the distillation, and the 50B-token pretraining control in Appendix A is a valuable check that the architecture itself is trainable without a teacher. The reported overlap numbers and speedups, if reproducible, would also be relevant for large-scale MoE training and serving. The paper does not currently ship code or checkpoints, but the evaluation is systematic across eleven benchmarks and three model families.

major comments (3)
  1. [§5.1, Tab. 1] The early-stopping signal for FCSD is MBPP+ ('patience of 20 evaluations and performance delta of 2%'), and MBPP+ is also one of the eleven entries in the average used to claim capability retention. This makes the reported average partially circular: the checkpoint is selected to be good on a benchmark that is then included in the reported evaluation. I verified that deleting the MBPP+ column from Tab. 1 does not change the qualitative conclusion (the average gaps remain roughly 2.6, 1.5, and 0.9 points), so the result is not an artifact of that single column; however, selection on MBPP+ can in principle also affect the other ten entries, and with a single checkpoint per model and no seeds or error bars, the reader cannot quantify this. Please report performance on a held-out validation set not used for early stopping, or exclude MBPP+ from the reported average, or provide multiple seeds
  2. [Abstract vs §5.2 (Tab. 3, Fig. 6)] The headline inference numbers do not match the body. The abstract states '32.6% speedup in Time To First Token when serving a converted DeepSeek-V3 architecture ... in SGLang and achieve 97.3% communication-computation overlap during the prefill stage'; the body reports 97.6% all-reduce overlap for DeepSeek-V2 in vLLM (not SGLang/DeepSeek-V3), and Fig. 6 reports SGLang prefill speedups of 1.18x–1.34x depending on context/batch, not a single 32.6% number. Please align the abstract with the measured configuration and, if overlap percentage for the SGLang/DeepSeek-V3 setting is claimed, report it explicitly.
  3. [§5.1, FCSD recipe] The FCSD recipe as stated is not fully specified. The final number of training tokens per model is only given as 'up to 10B'; the batch-size and learning-rate sweeps are described but the selected values for each model are not reported; and no seed information is given. Because the main claim depends on the final checkpoints being representative of a reproducible recipe, please provide per-model token counts, the selected hyperparameters, and the number of seeds. Also, Tab. 2 contains duplicate rows for 'KL (FAR100%)' and 'KL' with identical values; clarify whether this is intentional or a typo.
minor comments (5)
  1. [Eq. (6)] Equation (6) is missing the initial residual term o0; as written it defines o_k as a sum of sub-block outputs without the embedding/residual base. This is presumably a typo, but it makes the formula incorrect as stated.
  2. [Abstract/§7] The conclusion states that the models are converted 'while retaining 99% of the model accuracy'. The reported averages imply retention of about 96.1% for DeepSeek-V2-Lite, 97.1% for Qwen-3-30B, and 98.8% for Llama-4 Scout. Please either soften the claim ('within 2.5% on average') or restrict it to the Llama-4 Scout result.
  3. [References] References [23] and [24] are the same paper (Liu et al., 'Is your code generated by ChatGPT really correct?'). Please merge.
  4. [Tab. 2] The 'KL + INTER. L2' row reports a single number after a sweep over L2 coefficients, but the coefficient value or range is not given. This makes the comparison hard to reproduce.
  5. [Fig. 4] The legend labels 'Reg. FarSkip' are ambiguous; it should read 'Regular' and 'FarSkip-Collective' for clarity.

Circularity Check

1 steps flagged

Capability-retention numbers are partially selected: the same MBPP+ benchmark used for early stopping is included in the reported average; the other benchmarks and speedup measurements are independent.

specific steps
  1. other [Section 5.1 (early-stopping description); Table 1 (MBPP+ column and Avg)]
    "For the early stopping validation we use the MBPP+[23] dataset as a fast proxy for detecting instabilities and evaluate every 1000 training steps with a patience of 20 evaluations and performance delta of 2%."

    The checkpoints reported in Table 1 are selected by this exact early-stopping criterion on MBPP+. Table 1 then lists MBPP+ as one of the 11 evaluation tasks and includes it in the Avg column used for the 'within 1%'/'within 2.5%' capability-retention claims. Thus the reported MBPP+ axis is not an independent evaluation of the chosen checkpoint; it is the selection signal itself, so that row and the part of Avg it contributes are optimistically biased. This does not make the entire claim circular, because the other ten benchmarks, the pre-training proxy, and the overlap/speedup measurements are not used in model selection, but it means one reported axis is statistically connected to the selection procedure.

full rationale

The paper's central claim is empirical rather than a derived prediction: FarSkip-Collective is an engineering recipe, and matching the original checkpoint via KL self-distillation is the intended mechanism, not a hidden tautology. There is no fitted parameter renamed as a result, and the downstream benchmarks are external to the training objective. The main circularity concern is the early-stopping loop: final checkpoints are chosen by monitoring MBPP+, and the same MBPP+ score is included in Table 1's average that supports the headline capability-retention numbers. This makes that benchmark and part of the average non-independent. The remaining ten benchmarks, the from-scratch pre-training comparison, and the communication-overlap speedups are independent evidence, so the central claim is only partially affected. The one self-citation [39] is not load-bearing: Eq. 5 cites it for intermediate activation distillation, but Table 2 shows KL + INTER. L2 underperforms the final KL-only FCSD recipe, so the citation does not carry the result.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central method is a connectivity rewrite plus a distillation recipe, so there are no fitted physical constants; however, several recipe choices are tuned on the same models/benchmarks: LR/BS sweeps, token budget, and early stopping on MBPP+. The hand-picked partial/outdated input modes and the overlap inequality are the main assumptions. No new physical or conceptual entities are introduced.

free parameters (5)
  • per-model learning rate = one of {2e-5, 4e-5, 8e-5} selected by short sweep; exact values per model not reported
    Section 5.1: short sweeps with training loss are used to pick LR; FCSD results depend on it.
  • per-model batch size = one of {2^16, 2^17, 2^18} tokens; not reported per model
    Section 5.1: batch-size sweep is run for each model before the main distillation.
  • early-stopping criterion = MBPP+ with patience=20 evaluations and delta=2%
    Section 3.1: training is stopped based on MBPP+, which is also one of the 11 benchmarks in the final evaluation table.
  • distillation token budget = up to 10B tokens; exact per-model counts not given
    Section 5.1 states 'up to 10B tokens' but does not report how many tokens each model actually used.
  • sub-block input mode = attention uses partial (8b), MoE uses outdated (8a)
    Section 3: the choice of partial vs outdated per sub-block is hand-selected to balance accuracy and overlap, not automatically optimized.
axioms (6)
  • standard math MoE transformer residual updates can be written as Eq. 6, and skipping a sub-block output changes inputs to Eq. 7 with at most one sub-block staleness.
    Section 3, Eq. 6-8: this is the algebraic basis of FarSkip-Collective.
  • domain assumption GPUs can run communication kernels concurrently with compute kernels on independent streams/queues.
    Section 4: the implementation relies on torch.dist async mode and cuda streams.
  • domain assumption For the tested settings, T_Dispatch + T_Combine <= T_layer - T_routed - T_gate, so the overlap window is long enough to hide communication.
    Section 4.1, Eq. 9: the speedup claims depend on this inequality; the paper notes it may not hold for extremely sparse MoEs, large-scale topologies, or small-message decode.
  • domain assumption KL self-distillation to the original model's logits on instruction data recovers the original distribution under modified connectivity within a small token budget.
    Section 3.1/5.1: this is the load-bearing empirical premise, validated only for three instruction-tuned/chat checkpoints.
  • domain assumption PyTorch autograd Sequence Number reprioritization can safely reorder the backward graph without changing gradient values.
    Section 4.1: training overlap in the backward pass relies on this framework-internal behavior.
  • ad hoc to paper MBPP+ is a fast, sensitive proxy for FCSD training instability.
    Section 3.1: introduced specifically to early-stop FCSD; also appears in the final evaluation average (Tab. 1).

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models." pith.science (2026). https://pith.science/paper/I3VXMYSC

@misc{pith2026251111505,
  author       = {Pith},
  title        = {Pith review of: FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3VXMYSC}},
  note         = {Machine review of arXiv:2511.11505}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Blocking communication presents a major hurdle in running MoEs efficiently in distributed settings. To address this, we present FarSkip-Collective which modifies the architecture of modern models to enable overlapping of their computation with communication. Our approach modifies the architecture to skip connections in the model and it is unclear a priori whether the modified model architecture can remain as capable, especially for large state-of-the-art models and while modifying all of the model layers. We answer this question in the affirmative and fully convert a series of state-of-the-art models varying from 16B to 109B parameters to enable overlapping of their communication while achieving accuracy that is comparable with their original open-source releases. For example, we convert Llama 4 Scout (109B) via self-distillation and achieve average accuracy within 1% of its instruction tuned release averaged over a wide range of downstream evaluations. In addition to demonstrating retained accuracy of the large modified models, we realize the benefits of FarSkip-Collective through optimized implementations that explicitly overlap communication with computation, accelerating both training and inference in existing frameworks. For inference, we demonstrate 32.6% speedup in Time To First Token when serving a converted DeepSeek-V3 architecture with expert parallelism in SGLang and achieve 97.3% communication-computation overlap during the prefill stage. During training, our approach enables 88.9% communication overlap of the all-to-all communication collectives when pre-training DeepSeek-V3 MoE layers with expert parallelism.

Figures

Figures reproduced from arXiv: 2511.11505 by Deval Shah, Emad Barsoum, Guihong Li, Jiang Liu, Vikram Appia, Yonatan Dukler.

Figure 1
Figure 1. Figure 1: FarSkip-Collective modifies the connectivity between sub-blocks to avoid blocking com [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FarSkip-Collective MoE layer main operator execution. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy of Qwen-3-30B MoE modified with N FarSkip-Collective layers without training. Modified layers applied to the last N layers (blue) and first N layers (red). Modern GPUs, equipped with hundreds of in￾dependent Compute Units (Streaming Multipro￾cessors), can process multiple Queues (Streams) of kernels independently by scheduling work on different sets of processing units at the same time [44, 30, 8]… view at source ↗
Figure 4
Figure 4. Figure 4: Time To First Token (prefill stage) with vLLM inference engine under varying prompt [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: DeepSeek-V3 (L = 6) FarSkip￾Collective training speed-up under different Expert-Parallelism sizes and batch size con￾figurations [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: DeepSeek-V3 Time To First Token (prefill stage) with SGLang under varying batch-size [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Time Between Tokens (decode stage) with SGLang DeepSeek-V3 for 2-node inference under varying prompt length. Each model is served with EP=16 for the MLP sub￾block and TP=16 for attention. Unlike the prefill phase, LLM decoding is memory￾bandwidth-bound; especially in large MoEs such as DeepSeek-V3. In single-node settings, the large parameter count that needs to be loaded per-GPU, translates to slower deco… view at source ↗
Figure 8
Figure 8. Figure 8: MoE pretraining loss with regular and FarSkip-Collective architectures. We pretrain [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Model generations of a fully converted FarSkip-Collective DeepSeek-V2-Lite model [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: DeepSeek-V2 vLLM prefill inference layer execution (Top) regular connectivity (Bottom) FarSkip-Collective. In the bottom figure the all-reduce collectives are overlapped during the attention and MoE sub-blocks by running asynchronously on a second hardware queue. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: DeepSeek-V2-Lite pre-training forward-pass layer execution (Top) regular connectivity (Bottom) FarSkip-Collective. In the bottom, all-to-all communication is overlapped with computation, the first call corresponds to Dispatch which gets overlapped with the core-attention computation. In the second call, the all-to-all corresponds to Combine and is overlapped with the shared-expert and the next layer’s q, … view at source ↗
Figure 12
Figure 12. Figure 12: DeepSeek-V2-Lite pre-training backward-pass layer execution (Top) regular connectiv￾ity (Bottom) FarSkip-Collective. The backward-pass operator execution order is “hijacked” from the default torch.autograd Sequence Number ordering to re-order operations for overlap. In particular, routed-expert backward computation launches immediately after the finished synchronization point of the Combine all-to-all bac… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 14 linked inside Pith

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. InThirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  3. [3]

    Gpt-neox- 20b: An open-source autoregressive language model

    Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox- 20b: An open-source autoregressive language model. InProceedings of BigScience Episode ...

    2022

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1, 2018. 13

    Pith/arXiv arXiv 2018

  6. [6]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021

    2021

  7. [7]

    Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models.arXiv preprint arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    DeepSeek-AI. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  10. [10]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    DeepSeek-AI, Zhihong Shao, Damai Dai, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  12. [12]

    Apple intelligence foundation language models

    Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Kewei Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wiseman, Syd Evans, Tao Lei, Viv...

    arXiv 2024

  13. [13]

    Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  14. [14]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  15. [15]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V . Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. InAdvances in Neural Information Processing Systems, volume 32, pages 103–112, 2019

    2019

  17. [17]

    Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras, editors,Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics

    2016

  18. [18]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Team Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  20. [20]

    Tensor-parallelism with partially synchronized activations.arXiv preprint arXiv:2506.19645, 2025

    Itay Lamprecht, Asaf Karnieli, Yair Hanani, Niv Giladi, and Daniel Soudry. Tensor-parallelism with partially synchronized activations.arXiv preprint arXiv:2506.19645, 2025

    arXiv 2025

  21. [21]

    Fanxin Li, Shixiong Zhao, Yuhao Qing, Xusheng Chen, Xiuxian Guan, Sen Wang, Gong Zhang, and Heming Cui. Fold3d: Rethinking and parallelizing computational and communicational tasks in the training of large dnn models.IEEE Transactions on Parallel and Distributed Systems, 34(5):1432–1449, 2023. 14

    2023

  22. [22]

    Infinity instruct: Scaling instruction selection and synthesis to enhance language models, 2025

    Jijie Li, Li Du, Hanyu Zhao, Bo wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. Infinity instruct: Scaling instruction selection and synthesis to enhance language models, 2025

    2025

  23. [23]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  24. [24]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  25. [25]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. Meta AI Blog, April 2025

    2025

  26. [26]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InEMNLP, 2018

    2018

  27. [27]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. InProceedings of the International Conference for High Perform...

    2021

  28. [28]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sun- daraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, V olodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao...

    Pith/arXiv arXiv 1906

  29. [29]

    Kraken: Inherently parallel transformers for efficient multi-device inference

    Rohan Baskar Prabhakar, Hengrui Zhang, and David Wentzlaff. Kraken: Inherently parallel transformers for efficient multi-device inference. InAdvances in Neural Information Processing Systems, volume 37, 2024

    2024

  30. [30]

    Distributed w/ torchtitan: Introducing async tensor parallelism in pytorch

    PyTorch Team. Distributed w/ torchtitan: Introducing async tensor parallelism in pytorch. PyTorch Discussion Forum, 2024

    2024

  31. [31]

    Qwen3 technical report, 2025

    Team Qwen. Qwen3 technical report, 2025

    2025

  32. [32]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pages 3505–3506, New York, NY , USA, 2020. Association for Computing Machinery

    2020

  33. [33]

    Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 2019

    Pith/arXiv arXiv 1907

  34. [34]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  35. [35]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis...

    2019

  36. [36]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 15

    2023

  37. [37]

    Gpt-j-6b: A 6 billion parameter autoregressive language model.https://github.com/kingoflolz/mesh-transformer-jax, May 2021

    Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model.https://github.com/kingoflolz/mesh-transformer-jax, May 2021

    2021

  38. [38]

    Overlap communication with dependent computa- tion via decomposition in large deep learning models

    Shibo Wang, Jinliang Wei, Amit Sabne, et al. Overlap communication with dependent computa- tion via decomposition in large deep learning models. InASPLOS, pages 93–106, 2023

    2023

  39. [39]

    Zebra- llama: Towards extremely efficient hybrid models

    Mingyu Yang, Mehdi Rezagholizadeh, Guihong Li, Vikram Appia, and Emad Barsoum. Zebra- llama: Towards extremely efficient hybrid models. InAdvances in Neural Information Process- ing Systems, volume 39, 2025. NeurIPS 2025

    2025

  40. [40]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  41. [41]

    Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping.arXiv preprint arXiv:2501.06589, 2025

    Muru Zhang, Mayank Mishra, Zhongzhu Zhou, William Brandon, Jue Wang, Yoon Kim, Jonathan Ragan-Kelley, Shuaiwen Leon Song, Ben Athiwaratkun, and Tri Dao. Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping.arXiv preprint arXiv:2501.06589, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, Wenfeng Liang, Ying He, Yuqing Wang, Yuxuan Liu, and Y . X. Wei. Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures.arXiv preprint arXiv:2505.09343, 2025

    arXiv 2025

  43. [43]

    Genqa: Generating millions of instructions from a handful of prompts.arXiv preprint arXiv:2406.10323, 2024

    Henry Hengyuan Zhao, Pan Zhou, Difei Gao, and Mike Zheng Shou. Genqa: Generating millions of instructions from a handful of prompts.arXiv preprint arXiv:2406.10323, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel.Proceedings of the VLDB Endowment, 16(12):3848–3860, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Nguyen, Bernard Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel.Proceedings of the VLDB Endowment, 16(12):3848–3860, 2023

    2023

  45. [45]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural I...

    2024

  46. [46]

    hijacked

    Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. Nanoflow: Towards optimal large language model serving throughput. In Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation,...

    2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.