Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DIVE claims that pruning a dense LLM on different calibration datasets surfaces domain-specialized experts, and that reassembling these pruned FFNs into an MoE beats prior conversion methods at the same active-parameter and retraining…

desk verdict A genuinely new dense-to-MoE recipe with a real benchmark-selection soft spot; worth reviewing seriously, but the headline gains need an OOD check. read the letter →

arxiv 2506.09351 v1 pith:34ORHNJD submitted 2025-06-11 cs.CL

Yuchen Feng , Bowen Shen , Naibin Gu , Jiaxuan Zhao , Peng Fu , Zheng Lin , Weiping Wang This is my paper · ORCID
classification cs.CL
keywords mixture-of-expertsdense-to-sparseconversionstructuredpruningexpertdiversitydomainaffinitymininglargelanguagemodelsparameter-efficientretrainingLLMreconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most ways of turning a dense language model into a mixture-of-experts either duplicate the same feed-forward network or split it randomly, so the experts start out nearly identical and need heavy retraining to differentiate. DIVE starts from a different observation: when the same dense model is structurally pruned on different calibration datasets, the resulting pruned models perform very differently across domains, and that variation tracks the content of the calibration data. The paper uses this pruning sensitivity as a source of expert diversity, clustering 24 calibration tasks into domains, pruning the FFN once per domain, and reassembling the pruned FFNs as experts. With only routers, experts, and normalization layers retrained (under 1% of parameters on TinyLlama-1.1B, 0.5B tokens for routers plus 5B for experts), DIVE reports lower perplexity on WikiText2 and LAMBADA and higher average accuracy across 11 tasks than LLM-Pruner, FLAP, and LLaMA-MoE at the same activated-parameter budget. If this holds, converting existing dense checkpoints into efficient MoE models becomes cheaper and less wasteful of the dense model's latent specialization.

What carries the argument

The load-bearing mechanism is domain affinity mining followed by pruning-based expert reconstruction. The paper prunes the same dense model on each of 24 calibration datasets, scores each pruned model by normalized perplexity on all 24 evaluation tasks, computes Pearson correlations between calibration datasets, and hierarchically clusters the datasets into 8 domains. Each expert is then produced by pruning the original FFN with a calibration set mixed from one cluster, so expert indices are aligned with domains across all layers. A two-stage retraining recovers performance: first routers are trained densely with a temperature-scaled softmax that approximates top-k selection and matches the sparse inference behavior, then experts and normalization layers are trained sparsely with low-rank adaptation while attention and embeddings stay frozen.

What would settle it

Re-run the domain affinity mining with a held-out set of calibration and evaluation tasks that shares no benchmark with the final evaluation, cluster the same model into 8 domains, and compare DIVE against random clustering at the same retraining budget; if the advantage shrinks to noise on the original 11 tasks, the diversity benefit was an artifact of benchmark overlap.

Watch

Extended reading notes

Core claim

The central claim is that a pretrained dense LLM already contains the materials for diverse experts: pruning its FFN on different calibration datasets produces pruned models with different domain strengths, and these pruned models can be merged into an MoE whose experts inherit those strengths. On TinyLlama-1.1B with eight experts, one active at 50% of the original FFN per expert, DIVE 1/8 reaches WikiText2 perplexity 13.52 (sequence length 1024), LAMBADA perplexity 24.84, and 42.17% average accuracy over 11 tasks, compared with 19.57, 87.27, and 39.34% for LLaMA-MoE 1/8, 14.51, 33.22, and 41.42% for FLAP, and 17.59, 56.66, and 39.57% for LLM-Pruner under the same retraining budget. With two of eight experts active at 25% each, DIVE 2/8 also beats LLaMA-MoE 2/8 on the same metrics, and both DIVE variants keep their advantage when retraining is extended to a 15B-token budget. The paper interprets these results as evidence that domain-aware pruning-based initialization, not additional capacity, is what makes the reconstructed MoE better.

Load-bearing premise

The expert domains are chosen using the same 24-task benchmark family that is later used to judge the method, so the pipeline stands on the assumption that the domain-affinity pattern seen there is a stable property of the dense model's weights rather than a quirk of that particular task suite.

Editorial extensions

If this is right

  • Existing dense checkpoints can be converted into MoE models without pretraining new experts from scratch, since the experts are pruned fragments of the original FFN.
  • The retraining bill is small: only routers, experts, and normalization layers are updated, which the paper reports as under 1% of parameters, with dense training on 0.5B tokens and sparse training on 5B tokens for TinyLlama-1.1B.
  • Expert identities are interpretable: routing distributions follow the domain clusters built during calibration, so tokens from math, reading comprehension, and other domains land on distinct experts.
  • The advantage persists with more retraining data: at a 15B-token budget, DIVE 1/8 and 2/8 still beat LLaMA-MoE counterparts on language modeling and downstream tasks.
  • The same recipe is intended to transfer to other Llama-style models and other backbones, since domain clustering is reported for LLaMA2-7B, OPT-6.7B, and Qwen2.5-7B as well as TinyLlama-1.1B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: repeat the whole pipeline with fully disjoint calibration and evaluation suites, so that the tasks used to choose expert domains never appear in the final benchmark, and check whether the domain clusters transfer to unseen tasks.
  • The clustering tables show domain groupings for several backbones, but end-to-end reconstruction results are reported only for TinyLlama-1.1B; running the full pipeline at 7B scale and beyond is a direct test of whether pruning-induced diversity survives in larger models.
  • Because expert indices are consistent across layers and tied to domains, DIVE could be combined with expert merging or layer-wise routing analysis to reduce active parameter counts further, or with neuron-sharing residual experts to retain shared computation.
  • The temperature-scaled router pretraining is a transferable recipe for any top-k MoE conversion: train the router densely with softened logits, then switch to sparse expert tuning, independent of how the experts were initialized.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes DIVE, a method that converts a dense LLM into a Mixture-of-Experts model by structurally pruning the FFN modules with calibration sets chosen through a domain affinity mining procedure. The method has three stages: (1) computing a normalized-perplexity matrix over 24 pruning/evaluation tasks and clustering them via Pearson correlation to define expert domains; (2) pruning the dense FFN on each domain's calibration data to initialize specialized experts; and (3) retraining only the routers, LoRA adapters on experts, and normalization layers using a 0.5B-token dense stage followed by a 5B-token sparse stage. On TinyLlama-1.1B, DIVE 1/8 and 2/8 report lower WikiText2/LAMBADA perplexity and higher average downstream-task accuracy than LLM-Pruner, FLAP, and LLaMA-MoE at matched active FFN parameters, with gains persisting after 15B tokens. The paper also presents routing distribution analyses and ablations of the domain affinity mining and of the retraining targets.

Significance. If the reported results are robust, DIVE is a useful contribution to dense-to-MoE conversion: it offers a concrete, pruning-based way to initialize specialists that are more diverse than random splits or simple up-cycling, and it is accompanied by a structured analysis of domain affinity, per-backbone clustering tables, routing visualizations, and a public code release. The central claim that DIVE outperforms existing methods at equal active parameters is, however, not yet established at the level of generality stated in the abstract: the expert partition is selected using the test sets of the same benchmarks on which the model is then evaluated, and the full reconstruction pipeline is demonstrated on a single 1.1B model. The paper is clear and well organized, and the fix for the main concern is within reach.

major comments (2)
  1. [Section 3.1 (Eqs. 5-6), Appendix D.1, Appendix F] The domain affinity matrix that determines the expert partition is computed on the test sets of the 24-task suite, and the headline evaluation then reports perplexity/accuracy on seven of those same tasks (SciQ, PIQA, WinoGrande, MathQA, HellaSwag, LogiQA, BoolQ) plus WikiText2. This creates a selection loop: the expert clusters are chosen using the test sets of the very benchmarks on which DIVE is claimed to outperform the baselines, whereas LLM-Pruner, FLAP, and LLaMA-MoE are not given an equivalent test-set-driven architecture choice. The OOD subset (ARC-e, ARC-c, OBQA, MMLU) and LAMBADA are independent but small and not fully disjoint in domain from the calibration families (e.g., OBQA is commonsense QA, and ARC contains science questions similar to SciQ). To support the general claim of a diversity benefit, the authors should either redo the affinity mining with only training/validation splits, or restructure the main tables to report ID and OOD results separately and demonstrate that the OOD gains are consistent and beyond noise.
  2. [Section 4.1, Tables 1-2, and Limitations] The empirical validation of the reconstruction pipeline is performed on a single model, TinyLlama-1.1B. Table 10 reports domain clusterings for OPT-6.7B and Qwen2.5-7B, but no DIVE reconstruction, retraining, or evaluation is reported for those backbones. The abstract and introduction state that DIVE outperforms existing pruning and MoE reconstruction methods without restricting the claim to TinyLlama, and the Limitations section only mentions that the models are no larger than 7B. The generality of the method across architectures and sizes is therefore not yet demonstrated; the authors should either add at least one additional reconstruction experiment (e.g., on OPT-6.7B or LLaMA2-7B) or explicitly scope the central claim to the evaluated settings.
minor comments (6)
  1. [Equation (4) and surrounding text] There is a typographical comma in the subscript of S^ell_{:,j,} and the sentence 'we calculate the pruning mask M^ell_{t_i} base on the fluctuation variance' should read 'based on'.
  2. [Section 4.1] The text says all methods use 'identical retraining procedures,' but MoE methods receive an additional 0.5B-token dense router training stage that the pruning baselines do not receive; please clarify the exact token budget for each method and, if possible, compare at equal total budgets.
  3. [Table 2] The header 'ARC-c (25)' is ambiguous; it should be 'ARC-c (25-shot)' to match the main text.
  4. [Table 4 caption] The caption says 'DIVE 1/8 reconstructed without sparse training' but the comparison includes FLAP models that are pruned and not retrained; the caption should state the retraining status of both sides explicitly.
  5. [Table 10] The token 'SST2' appears in the TinyLlama 75% clustering while other entries use 'SST-2'; please unify the naming.
  6. [Figure 1] The caption would benefit from a one-sentence explanation of what 'Expertised' means, since the term is introduced only in the table and figure without definition.

Circularity Check

1 steps flagged · score 6.0 of 10

Expert domains are clustered using test-set PPL of the same ID benchmarks that headline the comparison, so part of DIVE's reported gain is selection on the evaluation target rather than a general diversity advantage.

  1. fitted input called prediction [Section 3.1-3.2, Eq. (5)-(6), Algorithm 1 (lines 1-4), Appendix D.1, Appendix F, Tables 2, 3, 9]
    "using their training sets as calibration datasets and test sets as the evaluation datasets. ... for (t_i,t_j)∈T×T do P_i,j←PPL(PRUNE(M,t_i),t_j); G←CLUSTER(NORM(P),N,dist=corr). ... ID tasks include SciQ, PIQA, WinoGrande, MathQA, HellaSwag, LogiQA, and BoolQ."

    Algorithm 1 builds the expert-domain partition G directly from PPL(PRUNE(M,t_i),t_j) over the 24-task suite; Eqs. (5)-(6) normalize and cluster this matrix over the same evaluation tasks. Appendix D.1 confirms test sets are the evaluation sets, so the partition is selected to make pruned experts 'good' on those test sets. Appendix F counts seven headline benchmarks (SciQ, PIQA, WinoGrande, MathQA, HellaSwag, LogiQA, BoolQ) as ID tasks from the same 24, and WikiText2 is also in the list. The reported ID-task and WikiText2 gains are thus evaluations on the objective used to construct the experts, not independent predictions; baselines get no equivalent test-set-driven architecture choice.

full rationale

The core reconstruction steps are not circular: pruning masks are computed from fluctuation importance scores, experts are pruned FFNs, and retraining uses routers, LoRA, and normalization modules on SlimPajama. There is no load-bearing self-citation chain; the self-citations in the related work are not used to justify the method. The circular content is confined to the benchmark loop: the affinity matrix that defines expert domains is measured on the test sets of the same 24 datasets that later serve as ID evaluation, so part of the measured superiority is a selection effect rather than an out-of-sample prediction. Because the OOD tasks (ARC-e, ARC-c, OBQA, MMLU) and LAMBADA were not used in the clustering and still show improvements, the method has independent support and the circularity is partial, not total.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on roughly six tuned or hand-chosen numbers (cluster count, router temperature, LoRA rank, calibration budget, sparsity ratio, retraining tokens) and five assumptions inherited from FLAP, the clustering geometry, the retraining recipe, frozen attention, and the corpus sample. No new theoretical entities are introduced: 'experts' are pruned FFN modules, routers and LayerNorms are standard components, and the domain-affinity heatmap is an analysis artifact rather than a fabricated entity. The free parameters are standard training choices rather than physics-style constants, but the '<1% tuned parameters' claim is inconsistent with the stated LoRA rank-8 setup.

free parameters (6)
  • Number of expert clusters N = 8
    Hierarchical clustering is truncated to 8 domains for every backbone and pruning ratio (Table 10); no sensitivity analysis for N is reported, and N is both a method hyperparameter and the MoE architecture choice.
  • Router temperature t = 0.05 for DIVE 1/8, 0.5 for DIVE 2/8
    Chosen per model by validation loss (Appendix B.3, Table 6), so the sharpness of routing during dense router training is tuned rather than derived; baselines receive their own tuned values, keeping the comparison fair.
  • LoRA rank and alpha = rank 8, alpha 16, dropout 0.1
    Fixed PEFT settings (Table 5); the '<1% tuned parameters' claim depends on rank, and at rank 8 on all experts the trainable fraction is about 1.8% of TinyLlama-1.1B, so the claim is inconsistent with the stated setup.
  • Pruning calibration size and sample length = 1024 samples, length 256
    Set in Appendix A.2 for all pruning runs; the domain-affinity signal (Figure 2) may depend on this small calibration budget, and no sensitivity check is reported.
  • Per-expert FFN sparsity = 50% for 1/8, 25% for 2/8
    Sparsity levels are fixed to equalize activated-parameter counts across methods (Section 4.1); the paper does not sweep sparsity to test whether DIVE's advantage depends on the ratio.
  • Retraining token budget = 0.5B tokens dense, 5B tokens sparse, 15B total in the extension
    The efficiency claim is relative to this budget; margins over LLaMA-MoE grow from 5B to 15B tokens (Table 7), so the reported advantage is budget-dependent, though the same budget is applied to baselines.
assumptions (5)
  • domain assumption FLAP fluctuation variance (Eq. 4) is a valid channel-importance measure for carving domain specialists.
    DIVE inherits FLAP's importance score without testing alternatives (e.g., Wanda or SparseGPT); if FLAP scores were domain-blind or noisy, the affinity heatmap in Figure 2 would not reflect true weight diversity.
  • ad hoc to paper Normalized-PPL Pearson correlations (Eqs. 5-6) define a meaningful domain geometry for experts.
    The 24-task evaluation suite defines both the affinity measure and the OOD/ID categories; the resulting 8 clusters (Table 10) are partly an artifact of which tasks were chosen, and the hierarchical linkage is not specified.
  • domain assumption Expert specializations survive reassembly and PEFT retraining.
    Algorithm 1 assumes a 0.5B-token router warm-up plus 5B-token LoRA restores the pruned-model domain structure; routing analysis (Figure 5) supports this qualitatively, but no quantitative measure of specialization preservation is given.
  • domain assumption Attention, embeddings, and the LM head can stay frozen after FFN replacement.
    Supported by the WikiText2 ablation (Figure 7), which shows under 0.4 PPL difference, but the ablation covers one benchmark, and the assumption is carried into downstream-task claims without additional evidence.
  • domain assumption A random 5B-token SlimPajama sample covers the domains needed for recovery.
    SlimPajama is standard, but the paper does not describe domain balancing of the sampled 5B tokens, so niche-domain routing quality could depend on the sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts." pith.science (2026). https://pith.science/paper/34ORHNJD

@misc{pith2026250609351,
  author       = {Pith},
  title        = {Pith review of: DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/34ORHNJD}},
  note         = {Machine review of arXiv:2506.09351}
}
read the original abstract

Large language models (LLMs) with the Mixture-of-Experts (MoE) architecture achieve high cost-efficiency by selectively activating a subset of the parameters. Despite the inference efficiency of MoE LLMs, the training of extensive experts from scratch incurs substantial overhead, whereas reconstructing a dense LLM into an MoE LLM significantly reduces the training budget. However, existing reconstruction methods often overlook the diversity among experts, leading to potential redundancy. In this paper, we come up with the observation that a specific LLM exhibits notable diversity after being pruned on different calibration datasets, based on which we present a Diversity-Enhanced reconstruction method named DIVE. The recipe of DIVE includes domain affinity mining, pruning-based expert reconstruction, and efficient retraining. Specifically, the reconstruction includes pruning and reassembly of the feed-forward network (FFN) module. After reconstruction, we efficiently retrain the model on routers, experts and normalization modules. We implement DIVE on Llama-style LLMs with open-source training corpora. Experiments show that DIVE achieves training efficiency with minimal accuracy trade-offs, outperforming existing pruning and MoE reconstruction methods with the same number of activated parameters.

Figures

Figures reproduced from arXiv: 2506.09351 by the authors.

Figure 1
Figure 1. Example inputs and outputs of a dense LLM [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The heatmap shows normalized perplexity (PPL) with a 75% FFN pruning ratio on LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Demonstration of diversity-enhanced reconstruction of LLMs from dense to MoE. In each lane, the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Perplexity curves of DIVE and the baselines, [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Routing distribution of DIVE 1/8 (left) and DIVE 2/8 (right), shown as activation ratios of the correspond [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance curves of DIVE and baselines [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of DIVE 1/8 (left) and DIVE 2/8 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Training and validation loss for (a) DIVE 1/8 and DIVE 2/8 with and without temperature coefficient ( [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Full routing distribution of DIVE 1/8 (left) and DIVE 2/8 (right), shown as activation ratios of the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Blink dynamically expands high-saliency visual tokens and drops them when attention shifts, improving LLaVA-1.5 and LLaVA-NeXT across seven multimodal benchmarks.

Reference graph

Works this paper leans on

59 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    AF Agarap. 2018. https://arxiv.org/abs/1803.08375 Deep learning using rectified linear units (relu) . ArXiv preprint, abs/1803.08375

    arXiv 2018

  2. [2]

    Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar

    Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. https://doi.org/10.18653/v1/2024.acl-long.678 LLM in a flash: Efficient large language model inference with limited memory . In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    doi:10.18653/v1/2024.acl-long.678 2024

  3. [3]

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/N19-1245 M ath QA : Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

    doi:10.18653/v1/n19-1245 2019

  4. [4]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. https://doi.org/10.1609/AAAI.V38I10.28960 Fluctuation-based adaptive structured pruning for large language models . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Sympo...

    doi:10.1609/aaai.v38i10.28960 2024

  5. [5]

    Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. https://aaai.org/ojs/index.php/AAAI/article/view/6239 PIQA: reasoning about physical commonsense in natural language . In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IA...

    work page 2020

  6. [6]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language...

    doi:10.18653/v1/n19-1300 2019

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    arXiv 2018

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . ArXiv preprint, abs/2110.14168

    arXiv 2021

Show all 59 references
  1. [9]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  2. [10]

    Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. 2023. https://arxiv.org/abs/2307.02628 Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference . ArXiv preprint, abs/2307.02628

    2023 arXiv

  3. [11]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. 2005. https://aclanthology.org/I05-5002 Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005)

    2005

  4. [12]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/N19-1246 DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  5. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  6. [14]

    Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, and Weiping Wang. 2024. https://aclanthology.org/2024.findings-acl.447 Light- PEFT : Lightening parameter-efficient fine-tuning via early pruning . In Findings of the Association for Computational Linguistics ACL 2024, pages...

    2024

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In 9th International Conference on Learning Representations, ICLR 2021,...

    2021

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  9. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. https://arxiv.org/abs/2401.04088 Mixtral of experts . ArXiv preprint, abs/2401.04088

    2024 arXiv

  10. [18]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019 doi

  11. [19]

    Aran Komatsuzaki, Joan Puigcerver, James Lee - Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. https://openreview.net/pdf?id=T5nUQDrM4u Sparse upcycling: Training mixture-of-experts from dense checkpoints . In The E...

    2023

  12. [20]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages...

    2017 doi

  13. [21]

    Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2

    1989

  14. [22]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. https://openreview.net/forum?id=qrwe7XHTmYb Gshard: Scaling giant models with conditional computation and automatic sharding . In 9th Inte...

    2021

  15. [23]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. https://doi.org/10.24963/ijcai.2020/501 Logiqa: A challenge dataset for machine reading comprehension with logical reasoning . In Proceedings of the Twenty-Ninth International Joint Conference on ...

    2020 doi

  16. [24]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/44956951349095f74492a5471128a7e0-Abstract-Conference.html Llm-pruner: On the structural pruning of large language models . In Advances in Neural Information Processing Systems 3...

    2023

  17. [25]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. https://openreview.net/forum?id=Byj72udxe Pointer sentinel mixture models . In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proc...

    2017

  18. [26]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  19. [27]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  20. [28]

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. 2016. https://doi.org/10.18653/v1/P16-1144 The LAMBADA dataset: Word prediction requiring a broad discourse cont...

    2016 doi

  21. [29]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res., ...

    2020

  22. [30]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  23. [31]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. https://arxiv.org/abs/2404.02258 Mixture-of-depths: Dynamically allocating compute in transformer-based language models . ArXiv preprint, abs/2404.02258

    2024 arXiv

  24. [32]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. https://dl.acm.org/doi/10.1145/3394486.3406703 Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters . In KDD '20: The 26th ACM SIGKDD Conference on Knowledg...

    2020

  25. [33]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. https://doi.org/10.1162/tacl_a_00266 C o QA : A conversational question answering challenge . Transactions of the Association for Computational Linguistics, 7:249--266

    2019 doi

  26. [34]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. https://aaai.org/ojs/index.php/AAAI/article/view/6399 Winogrande: An adversarial winograd schema challenge at scale . In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Th...

    2020

  27. [35]

    Noam Shazeer. 2020. https://arxiv.org/abs/2002.05202 Glu variants improve transformer . ArXiv preprint, abs/2002.05202

    2020 arXiv

  28. [36]

    Bowen Shen, Zheng Lin, Yuanxin Liu, Zhengxiao Liu, Lei Wang, and Weiping Wang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.112 COST - EFF : Collaborative optimization of spatial and temporal efficiency with slenderized multi-exit language models . In Proceedings of the 2...

    2022 doi

  29. [37]

    Bowen Shen, Zheng Lin, Daren Zha, Wei Liu, Jian Luan, Bin Wang, and Weiping Wang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.582 Pruning large language models to intra-module low-rank architecture with transitional activations . In Findings of the Association for Comp...

    2024 doi

  30. [38]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  31. [39]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Emp...

    2013

  32. [40]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2023. https://arxiv.org/abs/2312.12456 Powerinfer: Fast large language model serving with a consumer-grade gpu . ArXiv preprint, abs/2312.12456

    2023 arXiv

  33. [41]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  34. [42]

    Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou ' , and Daniel Cer. 2022. https://doi.org/10.18653/v1/2022.acl-long.346 SP o T : Better frozen model adaptation through soft prompt transfer . In Proceedings of the 60th Annual Meeting of the Association for Computational Lingui...

    2022 doi

  35. [43]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://openreview.net/forum?id=rJ4km2R5t7 GLUE: A multi-task benchmark and analysis platform for natural language understanding . In 7th International Conference on Learning Represe...

    2019

  36. [44]

    Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei L \"u , Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. 2024. https://arxiv.org/abs/2406.06563 Skywork-moe: A deep dive into training techniques for mixture-of-experts language models . ArXiv preprint, a...

    2024 arXiv

  37. [45]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. https://doi.org/10.18653/v1/W17-4413 Crowdsourcing multiple choice science questions . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  38. [46]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  39. [47]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  40. [48]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. https://openreview.net/forum?id=09iOdaeOzp Sheared llama: Accelerating language model pre-training via structured pruning . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus...

    2024

  41. [49]

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. https://doi.org/10.18653/v1/2022.acl-long.107 Structured pruning learns compact and accurate models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 151...

    2022 doi

  42. [50]

    Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. 2024. https://arxiv.org/abs/2406.06282 Powerinfer-2: Fast large language model inference on a smartphone . ArXiv preprint, abs/2406.06282

    2024 arXiv

  43. [51]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  44. [52]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . ArXiv preprint, abs/2401.02385

    2024 arXiv

  45. [53]

    Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. https://arxiv.org/abs/1810.12885 Record: Bridging the gap between human and machine commonsense reading comprehension . ArXiv preprint, abs/1810.12885

    2018 arXiv

  46. [54]

    Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html Character-level convolutional networks for text classification . In Advances in Neural Information Processing Systems 28: Annual Co...

    2015

  47. [55]

    Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. https://doi.org/10.18653/v1/2022.findings-acl.71 M o E fication: Transformer feed-forward layers are mixtures of experts . In Findings of the Association for Computational Linguistics: ACL 2022,...

    2022 doi

  48. [56]

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.890 LL a MA - M o E : Building mixture-of-experts from LL a MA with continual pre-training . In Proceedings of the 2024 Conference on Empiri...

    2024 doi

  49. [57]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. https://arxiv.org/abs/2308.07633 A survey on model compression for large language models . ArXiv preprint, abs/2308.07633

    2023 arXiv

  50. [58]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.