Pith. sign in

REVIEW 2 major objections 5 minor 46 references

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single pretrained parameter bank can instantly instantiate a specialist model for any domain mixture.

desk verdict The architecture is genuinely novel and worth citing, but the headline empirical claim is confounded by an eightfold training-budget gap between Soup-of-Experts and its baselines, so 'best specialists' is not established. read the letter →

arxiv 2502.01804 v1 pith:N33MRVA5 submitted 2025-02-03 cs.LG cs.CL

Pierre Ablin , Angelos Katharopoulos , Skyler Seto , David Grangier This is my paper · ORCID
classification cs.LGcs.CL
keywords modelmergingparameteraveragingdomainspecializationlanguagemodelingimportancesamplingmixtureofexpertslow-rankadaptationtask-adaptivepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that one pretraining run can encode a family of small specialist models, so that when a target dataset arrives, a specialist can be obtained by merging parameters rather than retraining. The core claim is that with a shared parameter bank $S$, expert vectors $E_1,\dots,E_n$, and coefficients $\alpha=\phi_\omega(h)$ produced by a small MLP from domain weights $h$, the instantiated model $\theta = S + \sum_j \alpha_j E_j$ can be trained once to have low loss on every mixture $\text{mix}(h)$. In experiments on 110M-parameter GPT-2 style models over 4096 RedPajama2 domains and 16 Pile domains, the soup achieves the best average specialized loss among generic pretraining, per-domain experts, and CRISP, while staying close to generic pretraining on the generic loss. If correct, pretraining cost no longer scales with the number of specialization tasks; specialization becomes a quick forward pass through the router and a parameter sum.

What carries the argument

The load-bearing object is the linear combination operator $\text{Combine}(S,E,\alpha) = S + \sum_j \alpha_j E_j$, together with the learned router $\phi_\omega: \mathbb{R}^k \to \mathbb{R}^n$ that maps domain weights $h$ to expert coefficients $\alpha$. The training objective is $L(S,E,\omega) = \mathbb{E}_{h\sim\pi}[\mathbb{E}_{x\sim\text{mix}(h)}[\ell(\text{Combine}(S,E,\phi_\omega(h)), x)]]$, with $\pi$ a meta-distribution over domain weights that in the main experiments samples $s=4$ sparse domains uniformly. This mechanism carries the argument because it turns model merging from a post-hoc fine-tuning operation into a pretraining objective: the experts are jointly optimized so that their linear interpolation, not any individual expert, yields good models across the entire simplex of mixtures.

What would settle it

Construct a specialization set whose documents are systematically outside the pretraining domain clusters, for example a language or file format absent from RedPajama2; compute $h_{\text{spe}}$ with Algorithm 3, instantiate the soup without training, and compare against (a) generic pretraining, (b) a CRISP model trained on $\text{mix}(h_{\text{spe}})$, and (c) the same soup with $\alpha$ optimized by gradient descent on the target data. If the flash-instantiated model fails to beat generic pretraining while the gradient-optimized $\alpha$ succeeds, the representation assumption collapses.

Watch

Extended reading notes

Core claim

The central claim is that pretraining can amortize specialization: instead of training one model per domain mixture, learn a parameterized family of models via parameter averaging. For any non-negative domain weights $h$ summing to one, the model $\text{Combine}(S,E,\phi_\omega(h)) = S + \sum_{j=1}^n \alpha_j(h) E_j$ should have low next-token loss on data sampled from $\text{mix}(h)$, where $\phi_\omega$ is a two-layer MLP mapping histograms over pretraining domains to expert coefficients. Training samples random domain weights $h \sim \pi$, instantiates the corresponding model, samples a mini-batch from $\text{mix}(h)$, and backpropagates through the loss. At specialization time, a target dataset $D_{\text{spe}}$ is mapped to domain weights $h_{\text{spe}}$ using the nearest-centroid rule of Algorithm 3, fed through the MLP, and the resulting parameters are merged directly, with no optimization on the target data. The paper reports that this procedure yields the best specialized loss among the compared methods on 16 Pile domains, nearly matches generic pretraining on the generic loss, and preserves its advantage through fine-tuning.

Load-bearing premise

A target specialization set can be faithfully summarized by a mixture-weight vector $h_{\text{spe}}$ computed with the BERT nearest-centroid rule, and the router trained on pretraining mixtures will produce good coefficients for that vector without ever seeing target samples.

Editorial extensions

If this is right

  • A single pretrained soup can instantiate a different specialist for each incoming dataset with no gradient updates, so specialization cost becomes a forward pass through a small MLP plus a parameter sum.
  • Pretraining cost is decoupled from the number of specialization tasks: CRISP and domain experts scale one pretraining run per task or domain, while the soup does not.
  • The soup's advantage survives fine-tuning: fine-tuning the instantiated specialist keeps the gain over fine-tuning a generic model, saving up to 10 million fine-tuning tokens on the uspto domain in the reported experiments.
  • Specialization data can be scarce: with roughly three target samples the instantiated soup already matches generic pretraining, and 100 to 1000 samples are enough to reach the full benefit of the method.
  • Low-rank experts can reduce the total parameter count, but at fixed parameter budget dense experts are more parameter-efficient; low-rank experts remain a fallback for heavily resource-constrained settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The flash specialization procedure inherits its weakness from the domain-weight representation: if a target distribution lies far from the pretraining simplex, the nearest-centroid estimate $h_{\text{spe}}$ may be meaningless, and the method has no built-in correction because $\alpha$ is never adapted from target samples.
  • The router is trained only on pretraining mixtures, so its predictions for out-of-simplex or novel target mixtures are extrapolations; a testable extension is to train the router on a richer meta-distribution or to add a second stage that optimizes $\alpha$ directly on the target when compute allows.
  • The soup trades storage flexibility for instant adaptability: a single 14B-parameter soup stores the same total parameter count as 128 individual 110M experts, but it can serve any mixture without knowing the target in advance, which matters when many specialists must be shipped quickly.
  • The linear-combination subspace may transfer to other modalities or tasks where data domains cluster, since the method only requires a domain-sampling procedure and a loss, not language-specific machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Soup-of-Experts (SoE), an asymmetrical architecture in which a small model is instantiated as a linear combination of a shared parameter vector S and n expert vectors E_j, with combination coefficients produced by a small MLP applied to domain-mixture weights h. Pretraining samples h from a meta-distribution π, instantiates Θ = S + Σ_j α_j E_j, and optimizes the next-token prediction loss on a batch drawn from mix(h). At specialization time, a target dataset is summarized into weights h_spe via the nearest-centroid method (Algorithm 3), and Θ = Combine(S,E,φ(h_spe)) is used without further training. The paper evaluates on 16 Pile domains with 110M-parameter GPT-2 models, comparing against generic pretraining, per-domain experts, and CRISP, and reports ablations on model scale, support size s, low-rank experts, and number of samples for weight estimation. The central claim is that SoE yields the best specialists while staying close to generic pretraining on the generic loss.

Significance. Training a parameter bank whose linear combinations can be selected by a learned function of domain weights is a novel and practically relevant idea, and the evaluation design is mostly sound: the 16 Pile specialization domains and their estimated weights are held out during SoE pretraining, and the reported losses are measured on those held-out domains. The algorithms are described precisely enough to be reimplemented, and the ablations on support size, low-rank experts, and model scale give useful guidance. The main weakness is the unequal training budget in the headline comparison, which currently prevents the paper from supporting its strongest empirical claim.

major comments (2)
  1. [Appendix A, Figure 4] The headline comparison is confounded by unequal training budgets. Appendix A states that the Soup-of-Experts and generic pretraining run for 1024K iterations (134B tokens), while domain experts and CRISP run for only 128K iterations (17B tokens) per model. Since Figure 4 plots loss against training time, the baselines are evaluated after one eighth of the tokens used by SoE. Figure 9 shows the per-domain CRISP and domain-expert curves still decreasing at 17B tokens, so there is no evidence of saturation. Consequently, the claim in Section 3.2 that SoE 'achieves the best performance among all methods on the specialized domains' is not established by the reported experiment. I request a token-matched comparison: either train CRISP and the domain experts to 134B tokens, or report SoE losses at 17B tokens alongside the baselines at 17B tokens.
  2. [Section 3.2] The description of the x-axis in Figure 4 is ambiguous and possibly inconsistent. The text says that for the multi-model baselines the reported time is the total training time taken by all models, but the CRISP and domain-expert curves appear to lie in the same 0-20h range as SoE, which would not be the case if 16 CRISP models and 64 domain experts were aggregated (each baseline model runs only 128K iterations). Please state clearly whether the x-axis for each baseline is per-model time or total time, and make the comparison readable under both conventions.
minor comments (5)
  1. [Appendix A] The phrase '128K iterations (134B tokens)' is arithmetically inconsistent with the stated batch size (128 sequences of length 1024), which gives approximately 17B tokens; this appears to be a typo that should be corrected.
  2. [Section 3.4, Figure 7] The main experiments use a meta-distribution support size of s=4, while the ablation reports the lowest average specialized loss at s=2; please justify the choice of s=4 or state explicitly that the difference from the optimal value is small.
  3. [Section 3] The paper does not report variance or multiple seeds; given the small differences between some curves (e.g., Figure 6), a statement about run-to-run variability would strengthen the empirical claims.
  4. [Algorithm 3] Algorithm 3 refers to 'Bert(x)' without specifying the exact BERT model or preprocessing used to compute document embeddings; please add this detail or a pointer to the implementation.
  5. [Throughout] There are several typos: 'proportionnal' in the Figure 4 caption, 'an domain weights' in the Figure 2 caption, and 'litterature' in Section 4; these should be fixed in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Soup-of-Experts objective, router, and held-out specialization evaluation are self-contained, and the only self-cited input (Algorithm 3) is fully specified in the paper and not fitted to the model.

full rationale

The central derivation is the training objective L(S,E,omega) = E_{h~pi}[L(Combine(S,E,phi_omega(h)),h)] (Eq. 3), and the paper's claim is that a model minimizing this objective, evaluated on held-out Pile domains via estimated mixture weights h_spe, yields strong specialists. The specialization domains and h_spe are explicitly stated to be unseen during pretraining ('We highlight that these specialist domains and specialist domain weights are never used or seen during the pre-training phase for all methods except for CRISP'), so the evaluation is an external generalization test rather than a refit of the training objective. Algorithm 3, attributed to Grangier et al. (2024b) and written out in full, computes h_spe from BERT nearest-centroid counts; it is an input to the router, not a parameter fitted to the Soup-of-Experts loss, and its output is not a quantity that the SoE objective minimizes by construction. The linear-combination ansatz (Eq. 2) is supported by external model-merging references rather than by a self-cited uniqueness claim. The unequal iteration budgets in Appendix A (1024K for SoE/generic versus 128K for domain experts/CRISP) create a training-cost confound in the headline comparison, but that is a correctness risk, not a circular reduction of the prediction to its inputs. No step in the derivation chain equates a fitted parameter with a predicted quantity or imports an unverified load-bearing conclusion from the authors' prior work.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on three domain assumptions rather than on a formal derivation: linear combinability of trained parameters, generalization of the router to unseen mixtures, and the sufficiency of centroid-estimated domain weights. These are tested empirically at a limited scale, but they are not proven.

free parameters (4)
  • Meta-distribution support size s = 4 for main experiments; ablation shows s=2 gives best specialist loss
    Hand-chosen to favor sparse domain weights; not derived from theory and not the value that optimizes the reported specialist loss.
  • Number of experts n = 128
    Chosen for main experiments; no dense-expert ablation shown for n, despite the low-rank experiments varying n.
  • Learning rate for Soup-of-Experts = 1e-4
    Selected by grid search over {1e-4, 3e-4, 1e-3}; the other methods use 3e-4.
  • Number of pretraining domains k = 4096 via 64x64 hierarchical k-means
    Clustering granularity follows Grangier et al. 2024b; it affects centroid quality in Algorithm 3 and the input dimension of the router.
assumptions (4)
  • domain assumption Linear combinations of model parameters can yield good models
    Inherited from the model-merging literature and invoked in Section 2.3 to justify Equation 2.
  • domain assumption A small MLP trained on random domain mixtures generalizes to unseen domain-weight vectors, including sparse held-out combinations
    Needed for specialization without retraining; tested empirically in Section 3.2 and Figure 7, but not proven.
  • domain assumption Target specialization datasets are well represented by mixture weights h_spe estimated via the BERT nearest-centroid rule
    Algorithm 3 is taken from Grangier et al. 2024b; specialization uses only this h_spe and never optimizes alpha on target data, as described in Section 2.7.
  • domain assumption Pretraining corpus clustering with sentence-BERT k-means yields meaningful domains whose mixtures span the Pile target sets
    The clustering and centroid construction follow Grangier et al. 2024b, Section 3; if centroids are poor proxies, the estimated weights mislead the router.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging." pith.science (2026). https://pith.science/paper/N33MRVA5

@misc{pith2026250201804,
  author       = {Pith},
  title        = {Pith review of: Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N33MRVA5}},
  note         = {Machine review of arXiv:2502.01804}
}
read the original abstract

Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint.

Figures

Figures reproduced from arXiv: 2502.01804 by the authors.

Figure 2
Figure 2. Data mixture sampling Given several pretraining do￾mains D1, . . . , Dk, an domain weights h1, . . . , hk, we can train a model on the mixture mix(h) = Pk i=1 hiDi, using the sampling procedure described in Algorithm 1. Domain weights have a criti￾cal impact on the downstream performance. find domain weights hspe such that mix(hspe) ≃ Dspe and then ii) train the model by minimizing L(Θ, hspe). This importance-sampli… view at source ↗
Figure 3
Figure 3. Quickly instantiating a small model from a pre￾trained Soup-of-Experts Given a specialist dataset with a few samples, we compute the domain weights using Algorithm 3. The domain weights are then passed through the Soup-of-Experts’ MLP to get the coefficients α that are then used to merge the experts. This process is quick since the MLP is small, and it requires no training. The fastest way is to obtain domain weight… view at source ↗
Figure 4
Figure 4. Training curves of the different methods. The average specialized loss is the average of the loss of the models over 16 domains from the Pile. The generic loss is the loss of the models on the standard pre-training distribution of RedPajamav2. The x-axis is the training time. This number is roughly proportion￾nal to number of tokens processed, since in this setting, the cost of instantiating the Soup-of-Experts is s… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: The gains of Soup-of-Experts during pretraining are maintained during fine-tuning and sometimes lead to large savings. On each of the 16 domains from the PILE, we fine-tune the corresponding instantiated Soup-of-Experts and generic model, with a limited number of fine-…
Figure 6
Figure 6. Figure 6: Left: Impact of model scale We train Soup-of-Experts and generic models with different instantiated model sizes. We observe that Soup-of-Experts maintain their advantage across the three scales considered here. Right: Impact of the number of samples in the specializati…
Figure 7
Figure 7. Figure 7: Role of the support size for the meta-distribution π. The training meta-distribution π draws random domain weights by first sampling s random domains, and then takes domain weights uniformly at random on those s domains. We investigate the impact of s, the support size…
Figure 8
Figure 8. Figure 8: Low-rank experts We investigate the possibility to use low-rank experts as detailed in Section 2.6. Low rank experts allow increasing the number of experts at a fixed total number of parameters, by diminishing the rank of experts. We train multiple Soup-of-Experts with…
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Impact of the number of samples in the specific set on the instantiated Soup-of-Experts. Detailed results from [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Impact of the support size of the meta-distribution. Detailed results from [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 14 canonical work pages

  1. [1]

    Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501.12370, 2025

    arXiv 2025

  2. [2]

    Ensemble of averages: Improving model selection and boosting performance in domain generalization

    Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 35: 0 8265--8277, 2022

    2022

  3. [3]

    A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M

    Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  4. [4]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    1901

  5. [5]

    Fusing finetuned models for better pretraining

    Choshen, L., Venezian, E., Slonim, N., and Katz, Y. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044, 2022

    arXiv 2022

  6. [6]

    X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y

    Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066, 2024. URL https://arxiv.org/abs/2401.06066

    arXiv 2024

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [8]

    Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models

    Dimitriadis, N., Frossard, P., and Fleuret, F. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. In International Conference on Machine Learning, pp.\ 8015--8052. PMLR, 2023

    work page 2023

Show all 46 references
  1. [9]

    Understanding emergent abilities of language models from the loss perspective

    Du, Z., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024

    2024 arXiv

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Doge: Domain reweighting with generalization estimation

    Fan, S., Pagliardini, M., and Jaggi, M. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023

    2023 arXiv

  4. [12]

    Dynamic gradient alignment for online data mixing

    Fan, S., Grangier, D., and Ablin, P. Dynamic gradient alignment for online data mixing. arXiv preprint arXiv:2410.02498, 2024

    2024 arXiv

  5. [13]

    A review of sparse expert models in deep learning

    Fedus, W., Dean, J., and Zoph, B. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667, 2022 a

    2022 arXiv

  6. [14]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022 b

    2022

  7. [15]

    Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al

    Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024

    2024 arXiv

  8. [16]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  9. [17]

    A., and Zettlemoyer, L

    Gonen, H., Iyer, S., Blevins, T., Smith, N. A., and Zettlemoyer, L. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037, 2022

    2022 arXiv

  10. [18]

    Adaptive training distributions with scalable online bilevel optimization

    Grangier, D., Ablin, P., and Hannun, A. Adaptive training distributions with scalable online bilevel optimization. Transactions on Machine Learning Research (TMLR), 2024 a . URL https://openreview.net/forum?id=JP1GVyF5i5

    2024

  11. [19]

    Task-adaptive pretrained language models via clustered-importance sampling

    Grangier, D., Fan, S., Seto, S., and Ablin, P. Task-adaptive pretrained language models via clustered-importance sampling. arXiv preprint arXiv:2410.03735, 2024 b

    2024 arXiv

  12. [20]

    Hard mixtures of experts for large scale weakly supervised vision

    Gross, S., Ranzato, M., and Szlam, A. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 6865--6873, 2017

    2017

  13. [21]

    MiniLLM : Knowledge distillation of large language models

    Gu, Y., Dong, L., Wei, F., and Huang, M. MiniLLM : Knowledge distillation of large language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5h0qf7IBZZ

    2024

  14. [22]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  15. [23]

    Y., Pang, T., Du, C., and Lin, M

    Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  16. [24]

    T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  17. [25]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  18. [26]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  19. [27]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  20. [28]

    Scaling laws for fine-grained mixture of experts

    Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi \'o ro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr \'o l, K., Odrzyg \'o \'z d \'z , T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024

    2024 arXiv

  21. [29]

    Evaluating quantized large language models

    Li, S., Ning, X., Wang, L., Liu, T., Shi, X., Yan, S., Dai, G., Yang, H., and Wang, Y. Evaluating quantized large language models. In International Conference on Machine Learning (ICML), 2024

    2024

  22. [30]

    LLM-Pruner : On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. LLM-Pruner : On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023

    2023

  23. [31]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Ortiz-Jimenez, G., Favero, A., and Frossard, P. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\...

    2023

  24. [32]

    Dense training, sparse inference: Rethinking training of mixture-of-experts language models

    Pan, B., Shen, Y., Liu, H., Mishra, M., Zhang, G., Oliva, A., Raffel, C., and Panda, R. Dense training, sparse inference: Rethinking training of mixture-of-experts language models. arXiv preprint arXiv:2404.05567, 2024

    2024 arXiv

  25. [33]

    Diverse weight averaging for out-of-distribution generalization

    Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35: 0 10821--10836, 2022

    2022

  26. [34]

    Model ratatouille: Recycling diverse models for out-of-distribution generalization

    Rame, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., and Lopez-Paz, D. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Inter...

    2023

  27. [35]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

    Rame, A., Couairon, G., Dancette, C., Gaya, J.-B., Shukor, M., Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [36]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  29. [37]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538

    2017 arXiv

  30. [38]

    Realistic evaluation of model merging for compositional generalization

    Tam, D., Kant, Y., Lester, B., Gilitschenski, I., and Raffel, C. Realistic evaluation of model merging for compositional generalization. arXiv preprint arXiv:2409.18314, 2024

    2024 arXiv

  31. [39]

    Efficient large language models: A survey

    Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., Chowdhury, M., and Zhang, M. Efficient large language models: A survey. Transactions on Machine Learning Research (TMLR), 2024

    2024

  32. [40]

    T., Wu, T., Song, D., Mittal, P., and Jia, R

    Wang, J. T., Wu, T., Song, D., Mittal, P., and Jia, R. GREATS : Online selection of high-quality data for llm training in every iteration. In Advances in Neural Information Processing Systems, 2024

    2024

  33. [41]

    Redpajama: an open dataset for training large language models

    Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., et al. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12372, 2024

    2024 arXiv

  34. [42]

    Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A

    Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Ch...

    2022

  35. [43]

    Structured pruning learns compact and accurate models

    Xia, M., Zhong, Z., and Chen, D. Structured pruning learns compact and accurate models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1513--1528, Du...

    2022 doi

  36. [44]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  37. [45]

    M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q

    Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429, 2023

    2023 arXiv

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.