Pith. sign in

REVIEW 3 major objections 8 minor 50 references

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One shared delta plus eight sparse masks turns a dense LLM into a mixture of experts in a single instruction-tuning pass.

desk verdict A clean, well-engineered method for sparse upcycling with genuinely useful memory savings, but the central 'expert discovery' claim is not actually tested by the experiments. read the letter →

arxiv 2506.12597 v1 pith:QBGHUMTF submitted 2025-06-14 cs.LG

Shengzhuang Chen , Ying Wei , Jonathan Richard Schwarz This is my paper
classification cs.LG
keywords sparsemixture-of-expertsLLMupcyclinginstructiontuningstructuredsparsityexpertroutingcross-taskgeneralizationparameter-efficientfine-tuningmask-basedexperts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SIMoE instruction-tuning aims to convert a dense pre-trained LLM into a mixture-of-experts model in one end-to-end fine-tuning stage, without manually choosing which layers become experts or dividing the data into domains. Its core assertion is that a single shared trainable parameter delta, gated by eight neuron-level sparse masks and blended by an input-dependent router, can represent the specialized knowledge the instruction data calls for. If correct, upcycling stops being a hand-designed architectural surgery and becomes an automatic optimization problem that also saves roughly two-thirds of the parameters compared with standard upcycling. The paper reports that this recipe beats full fine-tuning and existing upcycling baselines on cross-task generalization benchmarks while keeping the trained model roughly two-thirds smaller.

What carries the argument

The load-bearing object is the SIMoE linear-layer replacement $\theta = \theta_{\mathrm{pre}} + \sum_{i=1}^M \alpha_i (z_i \odot \theta_\delta)$, where $\theta_{\mathrm{pre}}$ is frozen, $\theta_\delta$ is one shared trainable delta, $z_i$ are binary masks drawn from a hard-concrete distribution, and $\alpha_i$ come from a softmax router applied to the final token embedding of the prompt. A constrained-Lagrangian objective enforces an L0 sparsity level $\tau$ on the masks, an orthogonality penalty $\|ZZ^\top - I\|^2$ pushes the masks apart, and the shared delta lets gradients flow across experts. This triple mechanism is what lets the model decide which linear layers, and which neurons within them, become specialized, rather than upcycling every feed-forward network (FFN) block by hand.

What would settle it

Replace the trained router with a fixed uniform average of the eight expert masks and re-evaluate on the same 12 unseen SNI categories; if the fixed variant stays within about 0.3 ROUGE-L of the reported 63.26/65.71, the input-dependent routing is not the source of the gains. Alternatively, measure per-domain routing entropy across the 12 unseen categories: near-maximum entropy or near-deterministic one-expert routing would both indicate the router is not doing the learned specialization the paper claims.

Watch

Extended reading notes

Core claim

The paper claims that an SMoE can be created from a frozen dense LLM by adding a small shared expert delta $\theta_\delta$ to every linear layer, multiplying it by expert-specific hard-concrete masks $z_i$, and summing $M=8$ such masked deltas with router weights $\alpha_i$ that depend on the prompt: $\theta = \theta_{\mathrm{pre}} + \sum_i \alpha_i (z_i \odot \theta_\delta)$. Training with a sparsity constraint of $\tau=75\%$ and an orthogonality penalty yields experts that are sparse, partially overlapping, and semantically aligned with task families. On the 12 unseen categories of Super-NaturalInstructions the paper reports SIMoE at 63.26 average ROUGE-L for Llama3.2-3B and 65.71 for Llama3-8B, against 60.76/64.07 for full fine-tuning and 61.84/65.05 for sparse upcycling; on the Tülu-v3 suite it reports 61.1 average with 10.04B total parameters, versus 60.9 for BTX with 30.58B. The intended conclusion is that where to upcycle and how to specialize experts can be learned rather than prescribed.

Load-bearing premise

The load-bearing premise is that one shared trainable delta gated by eight neuron-level masks, chosen by a router that reads only the last token of the prompt, can contain all the specialized knowledge in 1,616 tasks; if the router saturates or the masks are too coarse, the model degenerates into a single global update and the claimed gains disappear.

Editorial extensions

If this is right

  • Upcycling no longer requires domain labels: the sparsity-constrained optimization discovers which neurons encode which knowledge, so there is no need to partition the instruction data into math, code, safety, and other domains beforehand.
  • Model capacity scales with the number of masks rather than the number of full expert copies; with $\tau=75\%$ the resulting model uses roughly a third of the parameters of a standard upcycled SMoE at inference.
  • The router's instance-level soft blend means a single forward pass can combine knowledge from several experts, which the paper links to gains on 7 of 12 unseen SNI categories and to a 10-point safety margin on the DoAnythingNow jailbreak set.
  • Because upcycling happens at every linear layer but only a sparse subset survives, the learned pattern can be inspected post hoc, and the paper shows that layer-norm and attention key/value/output matrices carry more expert capacity than FFN gates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-delta-plus-masks representation is as general as the paper suggests, the same mechanism should transfer to vision-language models and other non-NLP backbones; the paper itself lists that setting as untested.
  • A natural next experiment would be to measure router entropy on held-out tasks: if prompts within one domain route to the same expert nearly always, instance-level routing may be functioning as task-level routing with a learned task embedding.
  • The sparsity constraint $\tau$ acts as a direct compute dial, so one could train a small sweep of $\tau$ to map out an accuracy-versus-parameter frontier, which the paper does not present.
  • The measured safety improvements hint that learned sparse residual specialization avoids some of the catastrophic forgetting that full fine-tuning induces, but the paper does not separate this effect from the smaller number of updated parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces SIMoE, an end-to-end instruction-tuning method that upcycles a dense pretrained LLM into a sparse mixture-of-experts-style model. SIMoE learns M=8 neuron-level binary masks over a single shared delta parameter tensor θδ, so that the model parameters become θ = θ_pre + Σ_i α_i (z_i ⊙ θδ), with α produced by an instance-level router. The training objective combines a negative log-likelihood loss, an orthogonality penalty on the masks, and a sparsity constraint enforced via a Lagrangian. The authors evaluate on Super-NaturalInstructions (SNI) and the Tülu-v3 suite using Llama3.2-3B, Llama3-8B, and Llama3.1-8B seeds, reporting higher average ROUGE-L and benchmark scores than full fine-tuning and upcycling/BTX baselines, with lower total parameter counts.

Significance. If the results are taken at face value, SIMoE offers a practical way to adapt a dense LLM into a parameter-efficient, partially specialized model, and the optimization formulation (sparsity constraint, orthogonality penalty, hard-concrete masks) is clearly specified and internally consistent. The method is also attractive for its memory savings relative to multi-expert upcycling. However, the significance is conditional on two unresolved points: the architecture's equivalence to a single gated sparse adapter (since θδ is shared across experts) is not tested, and the headline differences over baselines are small and are reported without variance. These points directly affect the central claim of automatic expert discovery.

major comments (3)
  1. [Section 3.1.2 and Eq. (9)] Because all experts share the same θδ, the effective residual update is θδ ⊙ (Σ_{i=1}^M α_i z_i). Thus the model is mathematically equivalent to a single sparse residual adapter whose mask is the router-weighted sum of the learned masks; no expert contains any parameter value that differs from another expert on any shared coordinate. The paper never ablates M=1 (a single mask with no router) or a fixed/uniform router, and Table 4 tests only L.U., O.P., S.C., and I.R. Without such a control, the reported gains over Full FT and Upcycling could be produced by a regularized sparse adapter, and the central claim that SIMoE 'automatically identifies multiple specialized experts' is unsupported. Please add these two controls (M=1, and M=8 with α_i fixed to 1/M) and report the resulting performance, along with routing statistics such as average α entropy and per-domain mask activation, to substantiate that the multi-mask, input-dependent routing provides a genuine benefit.
  2. [Section 4.2, Tables 1 and 2; Section 4.3, Tables 4 and 5] All results are from single runs with no seeds or error bars. In the strongest comparison, SIMoE exceeds the Upcycling baseline by 0.66 ROUGE-L on SNI-8B (Table 1) and by 0.2 points over BTX on Tülu-v3 (Table 2); these margins are within the typical run-to-run variance of instruction tuning. The abstract's claim that SIMoE 'consistently achieves state-of-the-art performance' is not supported by a single seed. Please report mean and standard deviation over at least three seeds for all main tables, or otherwise demonstrate that the evaluation setup is deterministic and that conclusions are stable across seeds.
  3. [Tables 3 and 8; Fig. 1] The claimed 'performance-compute trade-off' is based on total parameter count, but the router α is a dense softmax, so every expert mask contributes to every forward pass. The effective combined mask Σ_i α_i z_i can be dense even when individual masks are 75% sparse, so SIMoE's inference FLOPs may be comparable to a dense model of size θ_pre + θδ, not the reduced 'activated' parameter count implied by Table 8. Please report inference FLOPs or measured latency, and compare with the active parameter count of the Upcycling/BTX baseline (13.64B for the 8B seed) on the same basis. This is necessary to support the central efficiency claim.
minor comments (8)
  1. [Eq. (3) and Appendix A.2] The notation L0(Z) is used both as a count of nonzeros and as a fraction (the constraint 1 - L0(Z) ≥ τ only makes sense for a normalized sparsity). Please define L0(Z) explicitly, e.g., L0(Z) = ||Z||_0 / (M·D), and keep the convention throughout.
  2. [Table 2] The Tülu v3 8B SFT row is presumably taken from Lambert et al. (2025). Please clarify whether this baseline was retrained under the same settings and compute budget as SIMoE, or whether the comparison is against the published official numbers; if the latter, note the possible mismatch in training stages.
  3. [Section 4.2 and Introduction] The text refers to a '0.6%' improvement over Tülu-v3-8B-SFT, but Table 2 shows an absolute difference of 0.7 points (61.1 vs 60.4). Please reconcile the percentages and specify whether they are relative or absolute.
  4. [Figure 1] The percentages in the left panel (27.8%, 31.8%) are not defined in the caption; please state the reference value and whether these are memory reductions relative to the Upcycling baseline. In the right panel, clarify the distinction between 'Total' and 'Active' for the Upcycling bars.
  5. [Table 5] The β values are written as '5e−6' etc.; use consistent scientific notation and add a column header description (e.g., β coefficient for orthogonality penalty). Also state that these sensitivity results are from single runs, consistent with the main tables.
  6. [Section 3.1.3] The choice of instance-level routing is justified only by Table 4(d) on a single run. Please either add variance information for this ablation or soften the claim that it is 'empirically identified as the optimal choice.'
  7. [Related Work and Section 3] The relationship to the authors' prior SIE paper (Chen et al., 2024) is mentioned only in passing. Please add a paragraph that spells out the concrete novelties of SIMoE relative to SIE (e.g., scaling to LLM upcycling, instance-level router, sparsity-constrained Lagrangian, orthogonality penalty) to help readers assess the contribution.
  8. [Appendix C.1/Table 7] The full-granular vs structured mask comparison is performed on Llama3.2-1B, while the main results use 3B and 8B models. Please state whether the conclusion is assumed to transfer or provide a similar comparison at the main scales.

Circularity Check

1 steps flagged · score 6.0 of 10

The SIMoE derivation is internally self-contained, but the SNI headline number is a selected maximum over hyperparameters evaluated on the same unseen-category benchmark, making that part of the empirical claim partially circular.

  1. fitted input called prediction [Section 4.3 'Hyperparameter sensitivity', Table 5; hyperparameter values fed to Tables 1 and 6]
    "We evaluate the trained models on the SNI benchmark using the Llama3 8B model. ... Optimal performance is achieved with β∈[5e−6,5e−5] and a τ of 0.75."

    The SNI benchmark used for hyperparameter selection is the same 12 unseen-category evaluation later reported as SIMoE's headline result in Table 1. The paper chooses β and τ as the values that maximize average ROUGE-L on this target benchmark (Table 5, best average 65.71), and the same configuration is then reported as SIMoE's Table 1 score (65.71 for Llama3 8B; Table 6 fixes β=5e-6 and τ=0.75). Thus the claimed 1.6% gain over the 65.05 baseline is not an out-of-sample prediction: it is the maximum of a hyperparameter grid evaluated directly on the test set.

full rationale

The core SIMoE construction is not circular by definition: Eq. (9) defines the upcycled weights from a frozen pretrained tensor, per-expert masks, a shared delta, and router coefficients, and Section 3.2 derives a sparsity-constrained Lagrangian objective from that definition. No equation in the derivation assumes its own conclusion, and the Tülu-v3 benchmark results are external and not affected by the SNI hyperparameter selection. The authors' self-citation to their prior SIE work (Chen et al., 2024) is not load-bearing: the paper states the full reparameterization, sparsity penalty, and training objective rather than deferring to that citation for correctness. The missing M=1 or random-router control is a support gap for the 'expert discovery' claim, but it is not circularity. The one concrete circular step is the SNI hyperparameter search: Section 4.3 tunes β and τ on the SNI benchmark and then Table 1 reports the best configuration's score as SIMoE's SNI performance, so the headline SNI gain is partially a fitted maximum rather than an independent evaluation. This warrants a partial-circularity score of 6; if the hyperparameters had been selected on a validation split or pre-registered before touching the unseen categories, the score would drop to 0-2.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method adds no new physical or ontological entities. The expert masks, shared expert delta, and router are optimization parameters within an existing model. The learnable parts are the hard-concrete mask logits, the shared expert delta, the router MLP, and the Lagrangian multiplier, together with three hand-chosen hyperparameters (τ, β, M) that are tuned on the evaluation benchmark. The method rests on standard reparameterization and optimization tools plus domain assumptions about how and where knowledge is stored in transformer weights.

free parameters (5)
  • Sparsity target τ = 0.75
    Chosen from {0, 0.5, 0.75, 0.9} using the SNI evaluation categories (Table 5); all main experiments use τ=0.75.
  • Orthogonality coefficient β = 5e-6
    Chosen from {0, 5e-6, 5e-5, 5e-4} on the SNI evaluation set (Table 5); main experiments use 5e-6.
  • Maximum number of experts M = 8
    Set by hand (Section 4.1); no sensitivity study over M is reported.
  • Initial expected mask sparsity = 0.05
    Hand-set in Appendix B.1 so each mask starts with P(z=0)=0.05; affects the optimization trajectory.
  • Router MLP configuration = not specified
    Appendix B.1 describes an MLP on the final token embedding but gives no hidden sizes, depth, or activation details.
assumptions (6)
  • standard math Hard-concrete reparameterization with the deterministic median substitution yields differentiable masks with exact zeros.
    Adopted from Louizos et al. (2018) and Schwarz and Teh (2022); used in Section A.1 to optimize L0 sparsity.
  • domain assumption Structured sparsity at the granularity of input neurons is expressive enough to capture expert specialization.
    Section 3.1.1 motivates neuron-level masks for scalability and hardware friendliness; no expressiveness bound is given.
  • domain assumption A softmax router over the final prompt-token embedding provides sufficient instance-level routing signal.
    Section 3.1.3 selects instance-level routing; validated only by one ablation (Table 4(d)).
  • domain assumption The orthogonality penalty on masks induces complementary yet cooperative experts.
    Section 3.1.2; the paper observes lower overlap ratios but provides no theoretical guarantee.
  • domain assumption Anchoring to frozen θ_pre as θ_pre plus a sum of masked updates preserves pretrained knowledge and prevents catastrophic forgetting.
    Section 3.1.1 relies on the additive residual being small enough to preserve pretrained behavior.
  • standard math Solving the Lagrangian with projected gradient ascent and resetting λ to zero reaches the constrained sparsity solution.
    Appendix A.2, based on Gallego-Posada et al. (2022); no convergence analysis for the LLM setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts." pith.science (2026). https://pith.science/paper/QBGHUMTF

@misc{pith2026250612597,
  author       = {Pith},
  title        = {Pith review of: Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QBGHUMTF}},
  note         = {Machine review of arXiv:2506.12597}
}
read the original abstract

We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.

Figures

Figures reproduced from arXiv: 2506.12597 by the authors.

Figure 1
Figure 1. Compute cost in terms of (left) peak memory per GPU during upcycling instruction-tuning of 3B and 8B pre-trained LLMs, and (right) number of model parameters during inference. suboptimal performance and diminish the general￾ization capabilities of upcycled LLMs. The bene￾fits of dynamically identifying rather than manu￾ally specifying components are well documented (Von Oswald et al., 2021; Schwarz et al., 2021; Che… view at source ↗
Figure 3
Figure 3. Overview of the proposed Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning approach. SIMoE conceptually resembles the MoE principle in routing and combining specialized parameter components through soft merging, while it differs in implementation from conventional MoE architectures by defining each expert as a specific subset of sparse parameters within a shared network. Specifically, SIMoE upcycles … view at source ↗
Figure 5
Figure 5. Expert overlap ratios of upcycled LLMs post [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages

  1. [1]

    Shengzhuang Chen, Jihoon Tack, Yunqiao Yang, Yee Whye Teh, Jonathan Richard Schwarz, and Ying Wei. 2024. Unleashing the power of meta-tuning for few-shot generalization through sparse interpolated experts. arXiv preprint arXiv:2403.08477

    work page Pith review arXiv 2024

  2. [2]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. https://arxiv.org/abs/2401.06066 Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models . Preprint, arXiv:2...

    arXiv 2024

  3. [3]

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. https://doi.org/10.18653/v1/2022.acl-long.489 S table M o E : Stable routing strategy for mixture of experts . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7085--7095, Dublin, Ireland. Associ...

    doi:10.18653/v1/2022.acl-long.489 2022

  4. [4]

    William Fedus, Barret Zoph, and Noam M. Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1--120:39

    work page 2021

  5. [5]

    Jose Gallego-Posada, Juan Ramirez, Akram Erraqabi, Yoshua Bengio, and Simon Lacoste-Julien. 2022. https://openreview.net/forum?id=XUvSYc6TqDF Controlled sparsity via constrained optimization or: How i learned to stop tuning penalties and love constraints . In Thirty-Sixth Conference on Neural Information Processing Systems

    work page 2022

  6. [6]

    Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. 2024. https://arxiv.org/abs/2308.14761 Unified concept editing in diffusion models . Preprint, arXiv:2308.14761

    arXiv 2024

  7. [7]

    Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://arxiv.org/abs/2410.07524 Upcycling large language models into mixture of experts . Preprint, arXiv:2410.07524

    arXiv 2024

  8. [8]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. https://doi.org/10.1145/3695988 Large language models for software engineering: A systematic literature review . ACM Trans. Softw. Eng. Methodol., 33(8)

    doi:10.1145/3695988 2024

Show all 50 references
  1. [9]

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. https://doi.org/10.18653/v1/2024.acl-long.696 Harder task needs more experts: Dynamic routing in M o E models . In Proceedings of the 62nd...

    2024 doi

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  3. [11]

    Wangyi Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. 2025. https://aclanthology.org/2025.coling-main.636/ Improved sparse upcycling for instruction tuning . In Proceedings of the 31st International Conference on Computational Linguistics, pages 9485--9498, Abu Dhabi, ...

    2025

  4. [12]

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. https://openreview.net/forum?id=T5nUQDrM4u Sparse upcycling: Training mixture-of-experts from dense checkpoints . In The E...

    2023

  5. [13]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2025 arXiv

  6. [14]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. \ GS \ hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations

    2021

  7. [15]

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning

    2021

  8. [16]

    Smith, and Luke Zettlemoyer

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022. https://arxiv.org/abs/2208.03306 Branch-train-merge: Embarrassingly parallel training of expert language models . Preprint, arXiv:2208.03306

    2022 arXiv

  9. [17]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  10. [18]

    Zeming Lin, Halil Akin, Roshan Rao, Brian L. Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. 2022. Evolutionary-scale prediction of atomi...

    2022

  11. [19]

    Tianlin Liu, Joan Puigcerver, and Mathieu Blondel. 2023. Sparsity-constrained optimal transport. In The Eleventh International Conference on Learning Representations

    2023

  12. [20]

    Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V

    S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning

    2023

  13. [21]

    Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. https://openreview.net/forum?id=H1Y8hhg0b Learning sparse neural networks through l_0 regularization . In International Conference on Learning Representations

    2018

  14. [22]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf Locating and editing factual associations in gpt . In Advances in Neural Information Processing Sy...

    2022

  15. [23]

    Llama Team Meta. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  16. [24]

    Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, ...

    2025

  17. [25]

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. 2023. Soft merging of experts with adaptive routing. ArXiv, abs/2306.03745

    2023 arXiv

  18. [26]

    Basil Mustafa, Carlos Riquelme Ruiz, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022. Multimodal contrastive learning with LIM oe: the language-image mixture of experts. In Advances in Neural Information Processing Systems

    2022

  19. [27]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  20. [28]

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. https://openreview.net/forum?id=L8ifDX5XNq LISA : Layerwise importance sampling for memory-efficient large language model fine-tuning . In The Thirty-eighth Annual Conference on Neural Inf...

    2024

  21. [29]

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. 2023. From sparse to soft mixtures of experts. ArXiv, abs/2308.00951

    2023 arXiv

  22. [30]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. https://openreview.net/forum?id=hTEGyKf0dZ Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on ...

    2024

  23. [31]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1)

    2020

  24. [32]

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr \'e Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. In Neural Information Processing Systems

    2021

  25. [33]

    Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. In Neural Information Processing Systems

    2021

  26. [34]

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, Alhussein Fawzi, Josh Grochow, Andrea Lodi, Jean-Baptiste Mouret, Talia Ring...

    2023

  27. [35]

    Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. 2021. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 34:28889--28903

    2021

  28. [36]

    Jonathan Schwarz and Yee Whye Teh. 2022. https://openreview.net/forum?id=Cct7kqbHK6 Meta-learning sparse compression networks . Transactions on Machine Learning Research

    2022

  29. [37]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations

    2017

  30. [38]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. https://doi.org/10.1145/3658644.3670388 "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . In Proceedings of the 2024 on ACM SIGSAC Conference on Co...

    2024

  31. [39]

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wen Li, Wen tau Yih, Jason E Weston, and Xian Li. 2024. https://openreview.net/forum?id=nqLAuMOF6n Branch-train-mix: Mixing expert LLM s into a mixture-of-experts LLM...

    2024

  32. [40]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  33. [41]

    Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and Jo \ a o Sacramento. 2021. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 34:5250--5263

    2021

  34. [42]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  35. [43]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In International Conference on Learning Representations

    2022

  36. [44]

    Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Nicolaus Foerster, Phil Blunsom, Sebastian Ruder, Ahmet \"U st \"u n, and Acyr Locatelli. 2024. https://openreview.net/forum?id=BDrWQTrfyI BAM ! just like that: Simple and ef...

    2024

  37. [45]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023. Instruction tuning for large language models: A survey. ArXiv, abs/2308.10792

    2023

  38. [46]

    Dai, Zhifeng Chen, Quoc V Le, and James Laudon

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V Le, and James Laudon. 2022. Mixture-of-experts with expert choice routing

    2022

  39. [47]

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. https://doi.org/10.1162/coli_a_00502 Can large language models transform computational social science? Computational Linguistics, 50(1):237--291

    2024 doi

  40. [48]

    Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations

    2022

  41. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.