Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoE all-to-all communication can be cut to its theoretical minimum by grouping co-activated experts on the same device, yielding 1.5x speedups with no quality loss after fine-tuning.

desk verdict Clever BRIM kernel and a clean cost metric, but the '>1.5x' claim is not fully backed and the profiling-to-deployment transfer is untested. read the letter →

arxiv 2505.13345 v1 pith:JIXP4TK6 submitted 2025-05-19 cs.LG cs.DC

Shuqing Luo , Pingzhi Li , Jie Peng , Hanrui Wang , Yang (Katie) Zhao , Yu (Kevin) Cao , Yu Cheng , Tianlong Chen This is my paper · ORCID
classification cs.LGcs.DC
keywords mixture-of-expertsexpertparallelismall-to-allcommunicationoptimizationcollaborationpruningMoEtraininginferencesparsematrixmultiplication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the dominant communication cost in mixture-of-experts (MoE) models—the all-to-all exchange that can consume over 40% of training runtime—can be drastically reduced without sacrificing model quality. The authors introduce Occult, an algorithm-system co-design that first reschedules expert placement so that experts frequently co-activated by the same token live on the same GPU, then optionally prunes the routing choices to enforce device locality. They report more than 1.5x wall-clock speedups across three MoE-LLMs and three tasks, with comparable or better fine-tuned accuracy than standard top-k routing. The practical significance is that communication-efficient expert parallelism could make large MoE models cheaper to train and serve.

What carries the argument

The central object is the collaboration graph P, a normalized matrix of expert co-activation counts, along with the communication budget metric CT, the average number of token replicas per token. Algorithm 1, inspired by farthest point sampling, partitions experts into device groups by encouraging intra-collaboration and discouraging inter-collaboration. The Bidirectional Re-Index Matrix (BRIM) is a sparse-indexing data structure that enables a custom Triton-based sparse matrix multiplication kernel to perform token scattering and merging across two aggregation stages, avoiding extra memory allocation and enabling the streamlined all-to-all communication.

What would settle it

Run a communication-intensive decoding task on a domain different from the profiling data (for example, code generation when profiling used Wikipedia text) and measure whether CT stays near the profiled value and whether Occult's speedup over standard top-k training persists; a large drop in speedup or a sudden quality collapse would indicate the placement does not transfer.

Watch

Extended reading notes

Core claim

Occult establishes that the all-to-all communication in expert-parallel MoE layers can be reformulated as collaborative communication, where co-activated experts are either intra-collaborated (same device) or inter-collaborated (different devices). By measuring the average number of token replicas per token, a quantity CT that correlates linearly with wall-clock runtime, the paper shows that maximizing intra-collaboration reduces CT and that a fixed expert placement derived from profiling data can achieve this without changing the computation's results. When further coupling this placement with collaboration pruning, which restricts each token's routing to experts on a limited set of devices and is realized via modified fine-tuning, CT reaches its theoretical lower bound ceil(k*Nd/Ne) while maintaining comparable or superior task accuracy compared to standard fine-tuning.

Load-bearing premise

The fixed expert placement and pruning policy, both computed once from a profiling dataset, must remain near-optimal on the actual training and inference workloads, meaning the router's output distribution has to stay stable between profiling and deployment.

Editorial extensions

If this is right

  • Occult without pruning delivers exact computational results while cutting CT by about 20%, which directly translates into latency savings for prefilling, decoding, and training.
  • Occult with two-device collaboration pruning plus fine-tuning matches or exceeds standard top-k routing quality on multiple benchmarks, while being faster than popular frameworks such as vLLM, DeepSpeed, FSDP, and Tutel.
  • The method scales to 8 and 16 GPUs, with stable latency trends as batch size grows, and supports larger batch sizes under a fixed memory budget.
  • Occult is orthogonal to general serving and training frameworks, so it can be integrated with them for additional efficiency gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The collaboration-graph formulation frames expert placement as a graph partitioning problem; one could extend it to dynamically update placement during training to adapt to shifting router distributions, which the paper does not explore.
  • The similarity-table-based pruning suggests a testable extension: recomputing the expert similarity table periodically during fine-tuning could improve robustness to distribution shift, a scenario the paper does not evaluate.
  • CT could serve as a standard, task-agnostic reporting metric for communication efficiency in MoE systems, making speedup claims easier to compare across hardware and workloads.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Occult, an algorithm-system co-design for reducing all-to-all communication in expert-parallel Mixture-of-Experts (MoE) training and inference. It introduces a collaborative-communication perspective, measuring communication cost via CT (average token replicas per device), and two optimizations: (1) an offline expert-placement rescheduling algorithm (Algorithm 1) that places frequently co-activated experts on the same device, and (2) a collaboration-pruning technique that constrains each token's routed experts to a small set of devices and is materialized by modified fine-tuning. The system also contributes a custom sparse matrix multiplication kernel and the BRIM data structure for efficient tensor state transitions. Experiments on OLMoE, Qwen-MoE, and DeepSeek-MoE report latency gains over Tutel, MegaBlocks, vLLM, and FSDP, and benchmark quality after pruning on 23 tasks. The no-prune variant preserves exact routing and is claimed to reduce communication cost by roughly 20% in CT.

Significance. If the central claims hold, Occult provides a valuable perspective on MoE communication overhead: the CT metric is a simple, task-agnostic measure of all-to-all cost, and the no-prune placement rescheduling is an exact optimization with a clear lower bound. The pruning variant is a practical way to trade a small, controlled amount of routing flexibility for large communication savings, and the authors provide detailed kernel-level pseudocode and release code. However, the paper's headline claim of 'more than 1.5x speed up across multiple tasks and models' is contradicted by its own experimental numbers in one baseline comparison, and the transferability of the offline profiling step to new routers or data distributions is not demonstrated. The significance is therefore conditional: the core mechanism appears sound, but the stated speed and quality guarantees are not yet supported by the evidence as written.

major comments (4)
  1. [Abstract and Section 4.3 (Figure 9)] The abstract claims 'more than 1.5× speed up across multiple tasks and models' without qualification, but Section 4.3 reports that for Qwen-MoE Occult is '0.71× faster than MegaBlocks' and for DeepSeek-MoE '0.66× faster than MegaBlocks' in prefilling, i.e., Occult is slower than MegaBlocks in those settings. This is a direct contradiction between the headline speedup claim and the presented measurements. Please reconcile the abstract with the full results, or restrict the speedup claim to the specific baselines and configurations where it holds.
  2. [Section 3.2, Algorithm 1, and Appendix A.3 (Eq. 6-8)] The expert placement L and the similarity table T are both derived offline from a profiling dataset using the original router. However, the pruning method in Section 3.3 fine-tunes the model with a constrained routing policy, so the deployed router can have a different co-activation distribution from the one used to build L and T. The only supporting analyses are Figure 3, which shows that the maximal connected subgraph converges to the complete graph as batch size grows, and Figure 8, which shows one expert's collaboration pattern under pruning; neither measures router drift between profiling and deployment, nor between pre- and post-fine-tune routers. Without such a robustness test, the claimed speedups and quality guarantees may not transfer beyond the specific profiling distribution. Please add an experiment that measures CT or placement quality on held-out distributions or after fine-tuning, and compare the pre- and post-fine-tune router co-activation statistics.
  3. [Section 4.2 and Tables 3-5] The paper's abstract and introduction state that collaboration pruning achieves 'comparable or superior quality compared to the standard fine-tuning,' but the full results show that 1-GPU pruning consistently degrades quality on many benchmarks. For example, in Table 3 (OLMoE), Router-based Prune 1 GPU gives WSC 75.09 vs. 86.08 for standard SFT, MMLU 38.30 vs. 49.46, and PIQA 73.45 vs. 80.85. The paper later qualifies that 2-device pruning is the sweet spot, but the unqualified claim in the abstract is not supported. Please either qualify the quality claim to the 2-device setting or present the headline claim as 'comparable or superior quality with 2-device pruning.'
  4. [Section 2.2, Eq. (5)] The lower-bound formula 1≤⌈k·Nd/Ne⌉≤CT≤CT≤min{k,Nd} is presented as a general bound, but the derivation assumes that each device holds exactly Ne/Nd experts and that the router can place k selected experts into the fewest possible devices. In a fixed placement obtained from Algorithm 1, not every k-subset of experts can achieve this packing, so the lower bound is an ideal bound over all placements, not a per-placement guarantee. Please clarify the status of this inequality: it is a theoretical minimum over placements, not a claim about the CT achieved by the specific placement returned by Algorithm 1.
minor comments (6)
  1. [Section 2.2, Eq. (5)] The notation for the lower and upper bounds of CT is visually confusing: the text reads 'CT and CT' and '1≤⌈k·Nd/Ne⌉≤C_T≤CT≤min{k,Nd}', mixing underlined and overlined symbols. Please use distinct symbols (e.g., CT_lower and CT_upper) consistently throughout the paper.
  2. [Table 1] The claim of a 'strong linear correlation' between CT and runtime is based on a single configuration with 2^14 prompt tokens on one model. Please provide more configurations (different models, token counts, or device counts) and report the correlation coefficient, so the reader can judge the strength of the linear relationship.
  3. [Appendix A.3] The similarity table is declared as T∈Z^{Ne×Ne} (integer-valued), but Eqs. (6)-(8) compute cosine or squared-cosine similarities, which are real numbers in [0,1]. Please clarify whether the table stores real values or is quantized, and if quantized, describe the quantization procedure.
  4. [Section 3.3] The pruning rule 'The Nd devices are determined by traversing the tokens’ k routed experts in descending order of routing score until Nd devices are reached' is underspecified when multiple selected experts reside on the same device or when k < Nd. Please provide a precise algorithm or pseudocode for this device-range selection and for the similarity-based replacement criterion.
  5. [Section 4.1] The paper repeatedly references 'a profiling dataset' but never specifies which dataset is used for profiling, how many tokens it contains, or whether it is the same for all models and tasks. This information is needed to assess the transferability concern raised above and to reproduce the placement and similarity table.
  6. [Figure 1] The caption and the figure show speedups for 'multiple models and tasks,' but the figure does not report the specific baselines or error bars. Adding the raw latencies or the baseline names to the figure would make it easier to verify the headline claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the speedup and quality claims are validated against external baselines, and the CT metric and lower bound are defined independently of the measured latencies.

full rationale

The paper's central claims do not reduce to their own inputs. CT is defined in Section 2.2 as the average number of token replicas, and the empirical latencies in Table 1 are measured end-to-end, not fitted from CT. The lower bound in Eq. (5), C_T = ceil(k*Nd/Ne), is a straightforward combinatorial statement about how many devices k routed experts must span under even expert placement; it is not derived from the runtime data. The expert placement in Algorithm 1 is a heuristic partition of a profiled collaboration graph P, and the reported reduction in CT and runtime is an empirical result of running that placement, not an identity forced by the algorithm's objective. The collaboration-pruning quality claims are evaluated against standard fine-tuning on external benchmarks (Tables 3-5 and Figure 7), so 'comparable or superior quality' is not an artifact of the pruning construction. The only overlapping-author citation is the use of MC-MoE's empirical insight to build the similarity table in Appendix A.3; that heuristic is subsequently validated within the paper by comparing router-based and similarity-based pruning, and it is not the basis of the speedup numbers. The concern that the offline placement and similarity table may not transfer under router drift is a robustness/generalization risk, not a circularity. Therefore the derivation chain is self-contained, and no prediction is equivalent by construction to a fitted parameter or prior result.

Assumptions & free parameters 0 free parameters · 6 assumptions · 4 invented entities

No explicit fitted scalar parameters are introduced; the placement and pruning policies depend on profiled data, and the number of target devices Nd is a configuration choice, not a fitted constant. The axioms above are the load-bearing background assumptions, several of which are empirical and may not generalize.

assumptions (6)
  • domain assumption All-to-all communication is a dominant fraction of MoE runtime in the tested settings and reducing it translates into wall-clock speedup.
    Stated in the abstract and Section 2; supported only by the authors' own latency measurements (Table 1), not by an external benchmark.
  • domain assumption The collaboration graph P built from a profiling dataset is representative of routing patterns during deployment.
    Invoked in Section 3.2 to justify a fixed rescheduled placement; no distribution-shift analysis is provided.
  • domain assumption Router-logit cosine similarity is a valid proxy for expert interchangeability when pruning.
    Adopted from MC-MoE (Li et al., 2023b) in Section 3.3 and Appendix A.3; the paper relies on this borrowed empirical insight.
  • ad hoc to paper The maximal connected subgraph of the collaboration graph converges to the complete graph as batch size grows, making per-batch optimal placement unattainable.
    Section 2.4 and Figure 3 present this as an empirical finding for the tested models; it is used to justify the fixed-placement co-design and is not proven.
  • standard math Standard CUDA/Triton semantics and floating-point arithmetic behave as expected in the custom kernels.
    Assumed throughout Section 3.1 and the appendix algorithms; not formally verified.
  • domain assumption The single-node PCIe A6000 testbed is representative of the 'at scale' setting claimed in the abstract.
    All experiments are on one node with 4-16 GPUs (Section 4.1); no multi-node or high-bandwidth interconnect results are reported.
invented entities (4)
  • BRIM (Bidirectional Re-Index Matrix) independent evidence
    purpose: Guides dispatch/combine and sparse matrix multiplication to handle reduced token replicas without extra HBM allocations.
    Central new data structure; correctness is empirically testable via the released code and the described kernels.
  • CT (collaborative communication metric) independent evidence
    purpose: Measures average token replicas per token as a communication budget proxy and optimization objective.
    New metric; its claimed linear correlation with runtime is shown on the authors' own configurations in Table 1.
  • ORI/SFD/EPD tensor states independent evidence
    purpose: Define token multiplicity at each pipeline stage so that intra-collaboration reduces replicas while keeping outputs consistent.
    Implementation abstractions used by BRIM and the sparse kernels; testable via the pseudocode and code.
  • Expert similarity table T independent evidence
    purpose: Enables similarity-based replacement of pruned experts during collaboration pruning.
    Built from router-logit cosine similarity over a profiling dataset; follows MC-MoE's empirical insight and is reproducible with the released code.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference." pith.science (2026). https://pith.science/paper/JIXP4TK6

@misc{pith2026250513345,
  author       = {Pith},
  title        = {Pith review of: Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIXP4TK6}},
  note         = {Machine review of arXiv:2505.13345}
}
abstract

Mixture-of-experts (MoE) architectures could achieve impressive computational efficiency with expert parallelism, which relies heavily on all-to-all communication across devices. Unfortunately, such communication overhead typically constitutes a significant portion of the total runtime, hampering the scalability of distributed training and inference for modern MoE models (consuming over $40\%$ runtime in large-scale training). In this paper, we first define collaborative communication to illustrate this intrinsic limitation, and then propose system- and algorithm-level innovations to reduce communication costs. Specifically, given a pair of experts co-activated by one token, we call them "collaborated", which comprises $2$ cases as intra- and inter-collaboration, depending on whether they are kept on the same device. Our pilot investigations reveal that augmenting the proportion of intra-collaboration can accelerate expert parallelism at scale. It motivates us to strategically optimize collaborative communication for accelerated MoE training and inference, dubbed Occult. Our designs are capable of either delivering exact results with reduced communication cost or controllably minimizing the cost with collaboration pruning, materialized by modified fine-tuning. Comprehensive experiments on various MoE-LLMs demonstrate that Occult can be faster than popular state-of-the-art inference or training frameworks (more than $1.5\times$ speed up across multiple tasks and models) with comparable or superior quality compared to the standard fine-tuning. Code is available at $\href{https://github.com/UNITES-Lab/Occult}{https://github.com/UNITES-Lab/Occult}$.

Figures

Figures reproduced from arXiv: 2505.13345 by the authors.

Figure 1
Figure 1. Latency Comparison with Multiple Models & Tasks. Occult can accelerate training & inference for modern MoE￾LLMs on communication-intensive tasks. 1. Introduction Transformer-based (Vaswani, 2017) large language mod￾els (LLMs) (Achiam et al., 2023; Touvron et al., 2023) have demonstrated tremendous success in a broad spectrum of downstream tasks, and scaling up model parameters is the de facto approach to train more … view at source ↗
Figure 2
Figure 2. MoE workflows with 3 all-to-all communication strategies. We take 2 devices (D0 and D1) for expert parallel, and D j i denotes tokens from device i dispatched to device j. 2. Preliminary 2.1. Mixture-of-Experts and Expert Collaboration Given an input token embedding x, the output of an MoE layer is the weighted sum of outputs from the Ne experts {E0, . . . , ENe−1}: MoE(x) = N Xe−1 i=0 R(x)i · Ei(x), (1) where R(x)i… view at source ↗
Figure 3
Figure 3. Empirical correlation between the expert amount in the maximal connected sub-graph and token amount in a mini-batch for top-2, 4, 6, and 8 routing. We examine the last MoE layer for each model with the same inputs. four MoE-LLMs with top-2, 4, 6, and 8 routing. The results show that the maximal connected subgraph converges to the complete graph as the mini-batch token count increases, implying that optimal expert pa… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: MoE workflow with proposed bidirectional re-index-guided all-to-all communication and expert computing. We take the forward process for illustration, where token tensors are stated as ORI, SFD and EPD, compactly stored on HBM and guided by BRIMs. 1 0 2 0 1 x 2 x 3 0 1 …
Figure 5
Figure 5. Figure 5: Functions of 2 BRIMs. Both BRIM0 and BRIM1 are 2D matrices. BRIM0’s dimensions are defined by the number of devices for expert parallelism (first dimension) and the token count in the ORI tensor (second dimension). BRIM1’s dimensions are defined by the number of local …
Figure 6
Figure 6. Figure 6: Correlation between the functions of BRIM and the basic operations in MoE workflow. Each BRIM provides 2 functions. BRIM0 is employed for Dispatch & Combine, while BRIM1 is adopted for the sparse matrix multiplication. ing inter-collaboration. Each sub-graph represents…
Figure 7
Figure 7. Figure 7: Performance Comparison for Collaboration Pruning. Comprehensive evaluation across three MoE architectures shows performance trends under different pruning strategies. Note that 4-device collaboration pruning is equivalent to standard training with original top-k routin…
Figure 8
Figure 8. Figure 8: Collaboration Analysis for OLMoE using 4-way ex￾pert parallelism. Examining expert 0 in the first MoE layer shows stable collaboration between no-pruning and 2-GPU scenarios, while 1-GPU pruning leads to some patterns’ vanishing. A6000 GPUs, each with 48GB HBM memory. …
Figure 9
Figure 9. Figure 9: Prefilling Latency Comparison with 4 GPUs. Occult achieves superior efficiency under scaled workload [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Decoding Latency Comparison with 4 GPUs. Analysis with fixed prompt tokens (12800) and batch size (512) demonstrates Occult’s consistent latency advantages on communication-intensive decoding tasks [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Training Latency Comparison with 4 GPUs. With fixed sequence length of 512 tokens, Occult demonstrates supe￾rior performance across varying batch sizes. Training We evaluate the average latency per training step over 1k iterations with sequence length 512. We evalu￾at…
Figure 12
Figure 12. Figure 12: More training latency comparison for expert parallelism frameworks. Owning to the communication- and memory-efficient design, Occult achieves superior training efficiency under both 8- and 16-way expert parallelism configurations [PITH_FULL_IMAGE:figures/full_fig_p00…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 30 canonical work pages

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone.arXiv preprint arXiv:2404.14219,

    arXiv

  2. [6]

    R., Cross, J., C ¸elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al

    Costa-juss`a, M. R., Cross, J., C ¸elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. No language left behind: Scal- ing human-centered machine translation.arXiv preprint arXiv:2207.04672,

    arXiv

  3. [8]

    A., and Gardner, M

    Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., and Gardner, M. A dataset of information-seeking questions and answers anchored in research papers.arXiv preprint arXiv:2105.03011,

    arXiv

  4. [9]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    arXiv

  5. [10]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. InPro- ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252–262,

    2018

  6. [11]

    Race: Large-scale reading comprehension dataset from exami- nations.arXiv preprint arXiv:1704.04683,

    Lai, G., Xie, Q., Liu, H., Yang, Y ., and Hovy, E. Race: Large-scale reading comprehension dataset from exami- nations.arXiv preprint arXiv:1704.04683,

    arXiv

  7. [15]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large lan- guage models.arXiv preprint arXiv:2402.14800,

    Lu, X., Liu, Q., Xu, Y ., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large lan- guage models.arXiv preprint arXiv:2402.14800,

    arXiv

  8. [16]

    Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts

    Luo, S., Peng, J., Li, P., Wang, H., and Chen, T. Hexa- moe: Efficient and heterogeneous-aware moe accelera- tion with zero computation redundancy.arXiv preprint arXiv:2411.01288,

    work page Pith review arXiv

Show all 50 references
  1. [17]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789,

    arXiv

  2. [19]

    Jetmoe: Reaching llama2 performance with 0.1 m dollars.arXiv preprint arXiv:2404.07413, 2024a

    Shen, Y ., Guo, Z., Cai, T., and Qin, Z. Jetmoe: Reaching llama2 performance with 0.1 m dollars.arXiv preprint arXiv:2404.07413, 2024a. Shen, Y ., Stallone, M., Mishra, M., Zhang, G., Tan, S., Prasad, A., Soria, A. M., Cox, D. D., and Panda, R. Power scheduler: A batch size an...

    2013 arXiv

  3. [21]

    Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  4. [23]

    doi: 10.18653/v1/W18-5446

    Association for Compu- tational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446/. Wei, Y ., Du, J., Jiang, J., Shi, X., Zhang, X., Huang, D., Xiao, N., and Lu, Y . Aptmoe: Affinity-aware pipeline tun- ing for moe models on bandwidth-constrained gp...

    doi

  5. [24]

    A broad- coverage challenge corpus for sentence understanding through inference

    Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. InProceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volum...

    2018

  6. [25]

    Moe-I2: Compressing mixture of experts models through inter- expert pruning and intra-expert low-rank decomposition

    Yang, C., Sui, Y ., Xiao, J., Huang, L., Gong, Y ., Duan, Y ., Jia, W., Yin, M., Cheng, Y ., and Yuan, B. Moe-I2: Compressing mixture of experts models through inter- expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01016,

    arXiv

  7. [26]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  8. [27]

    accuracy 83.52 75.09 86.45 73.63 83.52 86.08 ✓ Winogrande (Sakaguchi et al., 2021)accuracy 68.35 61.40 68.11 61.56 68.82 69.53 ✓ ASDiv (Miao et al.,

    2021

  9. [28]

    accuracy 4.56 2.86 9.59 2.86 8.33 9.37 ✓ OpenBookQA (Mihaylov et al., 2018)accuracy 33.20 26.00 34.60 23.60 32.40 34.20 ✓ PIQA (Bisk et al.,

    2018

  10. [29]

    accuracy 80.09 73.45 80.41 71.98 80.47 80.85 ✓ HellaSwag (Zellers et al., 2019)accuracy 58.07 48.14 57.53 47.14 56.99 59.91 ✓ SST-2 (Socher et al.,

    2019

  11. [30]

    accuracy 62.61 75.92 86.35 74.66 85.44 83.94 ✓ MultiNLI (Williams et al., 2018)accuracy 40.96 35.82 41.93 33.64 43.23 45.04 ✓ QASPER (Dasigi et al., 2021)F1 score 90.53 93.88 97.03 97.03 97.03 97.79 ✓ MRPC (Dolan & Brockett, 2005)accuracy 51.96 66.18 67.65 68.63 68.14 65.93 ✓ ...

    2018

  12. [31]

    accuracy 52.65 51.35 50.94 50.28 49.99 50.05 ✓ MMLU (Hendrycks et al., 2021)accuracy 50.54 38.30 47.86 37.59 46.88 49.46 ✓ RACE (Lai et al.,

    2021

  13. [32]

    accuracy 94.60 91.20 94.40 91.00 94.20 94.70 ✓ PROST (Aroca-Ouellette et al., 2021)accuracy 28.24 25.80 28.99 26.80 27.84 29.16 ✓ BoolQ (Clark et al.,

    2021

  14. [33]

    accuracy 74.50 71.96 76.64 68.59 75.96 77.77 ✓ COPA (Roemmele et al., 2011)accuracy 89.00 80.00 85.00 75.00 86.00 89.00 ✓ LogiQA (Liu et al.,

    2011

  15. [34]

    accuracy 23.35 23.04 23.04 22.73 20.89 22.73 ✓ COQA (Reddy et al., 2019)exact match score56.42 50.90 56.37 47.63 53.23 55.35 ✓ COQA (Reddy et al.,

    2019

  16. [35]

    accuracy 82.05 79.49 81.68 78.39 82.78 80.59 ✓ Winogrande (Sakaguchi et al., 2021)accuracy 68.43 68.11 69.06 66.06 70.24 70.01 ✓ ASDiv (Miao et al.,

    2021

  17. [36]

    accuracy 4.38 16.36 12.28 12.49 10.63 11.28 ✓ OpenBookQA (Mihaylov et al., 2018)accuracy 30.40 27.80 29.40 28.60 32.20 29.60 ✓ PIQA (Bisk et al.,

    2018

  18. [37]

    accuracy 79.71 78.62 80.25 78.51 79.87 80.36 ✓ HellaSwag (Zellers et al., 2019)accuracy 57.95 55.29 57.19 54.73 57.74 57.42 ✓ SST-2 (Socher et al.,

    2019

  19. [38]

    accuracy 68.46 78.21 77.41 73.17 87.04 82.22 ✓ MultiNLI (Williams et al., 2018)accuracy 49.77 49.83 51.88 46.41 54.40 52.50 ✓ QASPER (Dasigi et al., 2021)F1 score 90.81 98.78 97.03 98.04 86.65 97.54 ✓ MRPC (Dolan & Brockett, 2005)accuracy 76.47 71.57 78.19 76.72 75.49 78.92 ✓ ...

    2018

  20. [39]

    accuracy 57.44 53.25 56.05 52.00 51.18 54.99 ✓ MMLU (Hendrycks et al., 2021)accuracy 60.86 56.85 60.13 56.23 57.25 59.99 ✓ RACE (Lai et al.,

    2021

  21. [40]

    accuracy 94.40 95.40 95.60 95.70 95.50 95.20 ✓ PROST (Aroca-Ouellette et al., 2021)accuracy 30.50 30.01 31.22 30.08 32.47 31.41 ✓ BoolQ (Clark et al.,

    2021

  22. [41]

    accuracy 79.57 78.38 79.88 77.22 81.41 80.40 ✓ COPA (Roemmele et al., 2011)accuracy 84.00 81.00 86.00 79.00 83.00 83.00 ✓ LogiQA (Liu et al.,

    2011

  23. [42]

    Apart from these tasks, router-based pruning outperforms similarity-based pruning on11 benchmarks, while similarity-based pruning outperforms router-based pruning on7benchmarks

    accuracy 30.41 30.26 31.03 31.80 31.95 29.03 ✓ COQA (Roemmele et al., 2011)exact match score64.40 65.93 66.77 66.28 64.48 66.23 ✓ COQA (Roemmele et al., 2011)F1 score 78.60 78.15 80.04 79.30 77.53 79.48 ✓ 18 Optimizing Collaborative Communication across Experts for Accelerated...

    2011

  24. [43]

    accuracy 84.98 78.75 83.88 82.05 84.62 84.62 ✓ Winogrande (Sakaguchi et al., 2021)accuracy 70.48 66.30 70.40 66.22 70.72 71.19 ✓ ASDiv (Miao et al.,

    2021

  25. [44]

    accuracy 0.91 4.77 2.82 1.65 3.08 4.95 ✓ OpenBookQA (Mihaylov et al., 2018)accuracy 32.20 29.60 33.40 29.20 34.20 34.20 ✓ PIQA (Bisk et al.,

    2018

  26. [45]

    accuracy 78.73 77.15 79.92 77.26 79.76 79.11 ✓ HellaSwag (Zellers et al., 2019)accuracy 58.06 54.05 58.36 53.71 58.31 58.49 ✓ SST-2 (Socher et al.,

    2019

  27. [46]

    accuracy 64.68 65.71 76.72 59.40 73.05 78.33 ✓ MultiNLI (Williams et al., 2018)accuracy 42.30 36.77 45.65 38.01 44.42 41.55 ✓ QASPER (Dasigi et al., 2021)F1 score 93.06 90.24 95.74 98.04 97.03 91.67 ✓ MRPC (Dolan & Brockett, 2005)accuracy 68.63 68.14 68.38 67.65 68.38 68.38 ✓ ...

    2018

  28. [47]

    accuracy 49.50 53.12 50.14 50.17 50.80 49.83 ✓ MMLU (Hendrycks et al., 2021)accuracy 37.95 36.16 42.43 34.69 41.91 38.66 ✓ RACE (Lai et al.,

    2021

  29. [48]

    accuracy 92.80 93.30 93.50 94.70 93.50 93.40 ✓ PROST (Aroca-Ouellette et al., 2021)accuracy 28.72 28.26 29.60 28.91 28.79 28.64 ✓ BoolQ (Clark et al.,

    2021

  30. [49]

    accuracy 72.39 68.69 68.87 73.39 70.15 71.93 ✓ COPA (Roemmele et al., 2011)accuracy 90.00 84.00 89.00 86.00 87.00 88.00 ✓ LogiQA (Liu et al.,

    2011

  31. [50]

    accuracy 25.35 24.42 25.96 25.65 25.65 25.96 ✓ COQA (Reddy et al., 2019)exact match score64.15 64.02 63.83 62.25 62.80 63.17 ✓ COQA (Reddy et al.,

    2019

  32. [2012]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313,

    Li, A., Gong, B., Yang, B., Shan, B., Liu, C., Zhu, C., Zhang, C., Guo, C., Chen, D., Li, D., et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313,

    arXiv

  33. [2013]

    11 Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference Taori, R., Gulrajani, I., Zhang, T., Dubois, Y ., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca: A strong, replicable instruction-following model.Stanfor...

    2023

  34. [2017]

    GLUE: A multi-task benchmark and analysis plat- form for natural language understanding

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bow- man, S. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Linzen, T., Chrupała, G., and Alishahi, A. (eds.),Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing an...

    2018

  35. [2018]

    Olmoe: Open mixture-of-experts language models.arXiv preprint arXiv:2409.02060,

    Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., et al. Olmoe: Open mixture-of-experts language models.arXiv preprint arXiv:2409.02060,

    arXiv

  36. [2019]

    Prost: Physical reasoning about objects through space and time

    Aroca-Ouellette, S., Paik, C., Roncone, A., and Kann, K. Prost: Physical reasoning about objects through space and time. InFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4597–4608,

    2021

  37. [2020]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- ma...

    2019

  38. [2021]

    H., and Jiang, Y

    Liu, J., Wang, J. H., and Jiang, Y . Janus: A unified dis- tributed training framework for sparse mixture-of-experts models. InProceedings of the ACM SIGCOMM 2023 Conference, pp. 486–498,

    2023

  39. [2022]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models.arXiv preprint arXiv:2401.06066,

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y ., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models.arXiv preprint arXiv:2401.06066,

    arXiv

  40. [2023]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y ., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319,

    1905 arXiv

  41. [2024]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  42. [2025]

    Accelerating distributed{MoE} training and inference with lina

    Li, J., Jiang, Y ., Zhu, Y ., Wang, C., and Xu, H. Accelerating distributed{MoE} training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 945–959, 2023a. Li, P., Zhang, Z., Yadav, P., Sung, Y .-L., Cheng, Y ., Bansal, M., and Chen, T. M...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.