Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoSLD proposes sharing the upper LoRA matrix across all experts and applying dropout to it, and reports the highest average multi-task accuracy among the compared LoRA-mixture methods while cutting trainable parameters.

desk verdict Sharing the LoRA A-matrix across experts is a neat, plausible trick, but the headline accuracy gain is not yet trustworthy because the model's hyperparameters appear to be tuned on the test sets while baselines were not, and the claimed t-test significance lacks supporting variance. read the letter →

arxiv 2412.08946 v1 pith:4OZBHKNW submitted 2024-12-12 cs.LG cs.AIcs.CL

Lulu Zhao , Weihao Zeng , Xiaofeng Shi , Hua Zhou This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords parameter-efficientfine-tuningLoRAmixtureofexpertsmulti-tasklearningparametersharingdropoutlargelanguagemodelsLLaMA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoSLD is a parameter-efficient variant of mixture-of-experts fine-tuning for large language models. The paper's central claim is that in multi-task training, the upper projection matrix $A$ of a LoRA pair should be shared by all experts in a layer, while each expert keeps its own lower projection matrix $B$, and that applying dropout to $A$ improves optimization. On six commonsense reasoning datasets with LLaMA2-7B, MoSLD reports an average mixture accuracy of 71.56%, against 69.15% for plain LoRA and 70.00% for MoLA, while training 1.389 billion parameters compared with MoLA's 2.228 billion. The mechanism matters because it offers a simple way to reduce the parameter cost of LoRA-based MoE while mitigating cross-task interference and knowledge forgetting.

What carries the argument

The central object is the per-layer shared matrix $A_l$, called the general-feature matrix of the LoRA decomposition, combined with the router-weighted sum in Eq. (3): $h_l = W_0x + \frac{\alpha}{r}\sum_{k=1}^K S_k^l(x)A_{k,l}B_{k,l}x$, where all $A_{k,l}$ are equal across experts. This sharing collapses $N_l$ low-rank matrices into one per layer, and the dropout update in Eq. (2), $A'_l = \mathrm{Mask}\odot A_l$ with rescaling by $1/(1-p)$, is what the paper credits with balancing gradient updates and preventing overfitting in the shared matrix. A load-balancing loss is added on top of the router to keep token assignments spread among experts.

What would settle it

Train MoSLD and an unshared mixture-of-experts LoRA baseline on two tasks that require opposite input-to-output mappings, such as the same prompt templates with reversed answer labels; if per-task accuracy on the reversed task falls below the unshared baseline, the shared $A$ cannot represent both task-specific transformations and the general-feature premise fails.

Watch

Extended reading notes

Core claim

The paper argues that the two low-rank matrices in LoRA are naturally decoupled: matrix $A$ acts as a general-feature extractor shared across tasks, and matrix $B$ carries task-specific features. MoSLD operationalizes this by setting $A_{1,l} = A_{2,l} = \cdots = A_{N_l,l}$ for every layer $l$, keeping one shared matrix per layer, and training a router to select the top-$K$ experts for each token, with the expert output computed as $h_l = W_0x + \frac{\alpha}{r}\sum_{k=1}^{K} S_k^l(x)A_{k,l}B_{k,l}x$. A Bernoulli dropout mask is applied to the shared $A_l$ during updates and rescaled by $1/(1-p)$, which the paper credits with balancing updates between the shared and expert-specific matrices and preventing overfitting. The reported consequence is that this design outperforms plain LoRA, MoLoRA, SiRA, MoLA, and MixLoRA in the mixture setting on the six datasets used, and improves out-of-domain generalization on MMLU.

Load-bearing premise

The load-bearing premise is that in a LoRA pair the upper projection matrix $A$ genuinely encodes general cross-task features while the lower projection matrix $B$ encodes task-specific features, so a single shared $A$ per layer can serve all experts without becoming a bottleneck; the paper itself lists visualizing this distinction as future work.

Editorial extensions

If this is right

  • If MoSLD is correct, LoRA-based mixture-of-experts becomes substantially cheaper: sharing one matrix per layer cuts trainable parameters from MoLA's 2.228 billion to 1.389 billion on LLaMA2-7B while raising average mixture accuracy from 70.00% to 71.56%.
  • The sharing mechanism is orthogonal to layer-wise expert allocation; the paper's Figure 7 indicates that putting more experts in higher layers still helps, so the two ideas can be combined without conflict.
  • Dropout on the shared matrix provides a practical regularizer: a dropout probability of 0.1 improves both single and mixture results over the no-dropout variant MoSL, and larger dropout rates degrade performance.
  • The method scales with backbone size: average mixture accuracy rises from 71.56% on LLaMA2-7B to 79.96% on LLaMA-33B, with the single-to-mixture gain growing from 1.36 points to 1.91 points.
  • For task mixtures with little commonality, such as adding GSM8K math reasoning to commonsense QA, MoSLD turns the usual mixture penalty into a small gain, suggesting the shared matrix does not force harmful coupling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that sharing $A$ acts as an implicit regularizer that forces experts to differentiate only through $B$; a direct test would compare the cosine similarity of expert $B$ matrices under MoSLD versus an unshared mixture-of-experts LoRA baseline.
  • The factor-splitting premise is asserted rather than measured, since the paper lists visualizing the general/specific feature split as future work; if a task mixture requires $A$ to carry task-specific information, the shared matrix becomes a capacity bottleneck, and we would expect MoSLD to underperform unshared mixture-of-experts LoRA on such mixtures.
  • We would expect the dropout-on-$A$ trick to transfer to other shared-parameter fine-tuning designs, such as shared adapters or shared prefix parameters, because the update imbalance it addresses is generic to any frequently updated shared module.
  • The reported out-of-domain gains on MMLU suggest the shared matrix stores transferable features; a testable extension is to freeze $A$ after multi-task training and fine-tune only new $B$ matrices for unseen tasks, which should be cheaper and possibly more stable than training full LoRA from scratch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes MoSLD, a parameter-sharing variant of mixture-of-LoRA for multi-task fine-tuning of large language models. In each Transformer layer, a single low-rank matrix A is shared by all experts while each expert retains its own B matrix, and dropout is applied to A during training. The authors evaluate on six commonsense reasoning datasets under single-task and mixed-task training, reporting a higher average accuracy than LoRA and several MoE-LoRA baselines, with 1.389B trainable parameters versus 2.228B for MoLA. The paper also includes scaling experiments across model sizes, an out-of-domain evaluation, and an ablation of the dropout component.

Significance. If the comparative results are valid, the proposal is a simple and parameter-efficient modification of LoRA-MoE that is largely orthogonal to existing methods. The inclusion of scaling results to 33B, an out-of-domain test, and an ablation isolating dropout are strengths. However, the empirical support for the headline claim is weakened by test-set hyperparameter selection and an unsupported significance statement; the conceptual claim that A carries general features is not directly evidenced. These issues are addressable and do not invalidate the underlying idea.

major comments (3)
  1. [§5.2 and Table 1] The dropout ratio (0.1) and expert-allocation scheme (2,4,6,8) used in Table 1 are selected by evaluating MoSLD on the six test sets under the mixture setting (Figures 5 and 7), and rank r=8 is selected in Appendix B (Table 6). This is test-set selection, not validation-based selection, and no comparable tuning is reported for the baselines. The reported 1.56-point gain over MoLA may therefore reflect the hyperparameter search rather than the proposed mechanism. Please report model selection on a held-out validation split, or tune all baselines under the same protocol, and clearly state the selection procedure in the main text.
  2. [Table 1] The caption states '(p < 0.01 under t-test)' but provides no variance information, per-run results, or test details. With only six paired dataset means, a two-sided paired t-test of the reported mean differences does not reach p < 0.01 (a rough calculation gives p ≈ 0.02), and the effective sample size is small. Please report the per-run accuracies, the exact test performed, the test statistic, and the degrees of freedom, or remove the significance claim.
  3. [Section 3.1 and Limitations (3)] The design rationale is that A is a 'general-feature matrix' and B is a 'specific-feature matrix,' but the paper states in Limitation (3) that visualizing this phenomenon is future work. The ablation MoSL vs. MoSLD isolates the dropout contribution only; the contribution of sharing A is not directly ablated because no variant with separate A per expert under identical hyperparameters is compared. The comparison to MoLA is a baseline comparison, not a controlled ablation, since expert allocation and other settings differ. Please add a controlled ablation or rephrase the causal attribution.
minor comments (6)
  1. [Eq. (3)] In Eq. (3), the expert matrices are written as A_{k,l} and B_{k,l}, but A is shared so it should be A_l; this makes the sharing mechanism explicit and avoids confusion with Eq. (2).
  2. [Eq. (4)] The load-balancing loss in Eq. (4) is not a well-formed equation; the expression after 'pk =' needs to be written out cleanly, and the notation c_k and s_k should be defined consistently with the text.
  3. [Table 5] The column 'LoRA number' is ambiguous: please clarify whether it counts matrices per layer for both Q and V projections and how the average expert number 5 is derived from the (2,4,6,8) allocation.
  4. [Table 2] The row 'MoSLD (matrix B)' should specify whether dropout is applied to B instead of A or whether the sharing mechanism is applied to B; the text should define the variant precisely.
  5. [Section 4.4] The statement that baselines 'trail behind LoRA by 1.20%, 1.06%, 1.98%, and 1.00%' should refer to percentage points, not percentages, to avoid quantitative ambiguity.
  6. [Figure 1] Figure 1's axes and the meaning of 'Score (mixture)-Score (single)' are not explained in the text; please add a sentence describing the figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoSLD's empirical claims are benchmark comparisons, not derivations from their own inputs.

full rationale

MoSLD's contribution is an architectural change (sharing matrix A across experts in Eq. 3 and applying dropout to A in Eq. 2) evaluated against external benchmarks, not a quantity derived from its own inputs. The reported accuracies in Table 1 are measured on held-out test sets of OBQA, CSQA, Race, MCTest, Arc-e, and Arc-c, and the out-of-domain claim uses MMLU. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation is invoked. Self-citations (Zhao et al. 2022a,b; Zeng et al. 2023) appear only in related-work and baseline descriptions. The central assumption that A captures general features while B captures task-specific features is explicitly unverified (Limitation 3: the authors "intend to visualize this phenomenon in the future"), but that is an unsupported premise, not a circular derivation. Likewise, the hyperparameter searches in Section 5.2 (dropout ratio in Figure 5 and expert allocation in Figure 7) are performed on the mixture test sets before being used in Table 1, which is a soundness and comparability concern rather than circularity, because the reported numbers are empirical measurements and the method's claims do not reduce by construction to those choices. The unsupported t-test claim (p<0.01) is an evidentiary weakness, not a circular-reasoning step. The paper is self-contained as an empirical evaluation of an architectural variant.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests mostly on benchmark measurements. The main supported assumptions are the LoRA low-rank update, the MoE routing paradigm, the general/specific factor split of A and B, and the dropout benefit. Free parameters such as dropout ratio, expert allocation, rank, and top-K are chosen by hand, and for the first three the paper reports test-set sweeps.

free parameters (4)
  • dropout ratio p = 0.1
    Chosen from a sweep reported on test-set accuracy in Figure 5, then used in all main experiments.
  • per-layer expert allocation = (8,6,4,2) for layer groups 1-8, 9-16, 17-24, 25-32
    Selected using the test-set comparison in Figure 7; the final allocation is used in the headline Table 1 results.
  • rank r = 8
    Set to 8 in Section 4.3 and supported by a test-set rank sweep in Table 6, where 8 or 16 is best.
  • top-K selected experts = 2
    Set in Section 4.3 as a standard sparse MoE choice; not swept.
assumptions (4)
  • domain assumption Fine-tuning updates of a pretrained model lie approximately in a low-rank subspace, so LoRA's BA decomposition is sufficient.
    Invoked by using LoRA decomposition in Eq. (3); this is the standard LoRA premise from Hu et al. 2022.
  • ad hoc to paper Matrix A in LoRA captures general or shared features while matrix B captures task-specific features.
    Assumed in Section 3.1 to justify sharing A across experts. Limitation (3) says visualizing this split is future work, so the premise is asserted without direct evidence.
  • domain assumption Sparse top-K routing with a load-balancing auxiliary loss is effective for mixing LoRA experts.
    Borrowed from the MoE literature and used in Eq. (1) and Eq. (4); the formula for the balancing loss is garbled in the text.
  • domain assumption Dropout on the shared parameter matrix alleviates optimization imbalance and overfitting.
    Applied through Eq. (2) and motivated by LoRA dropout literature, mainly Wang et al. 2024.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning." pith.science (2026). https://pith.science/paper/4OZBHKNW

@misc{pith2026241208946,
  author       = {Pith},
  title        = {Pith review of: MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OZBHKNW}},
  note         = {Machine review of arXiv:2412.08946}
}
read the original abstract

Recently, LoRA has emerged as a crucial technique for fine-tuning large pre-trained models, yet its performance in multi-task learning scenarios often falls short. In contrast, the MoE architecture presents a natural solution to this issue. However, it introduces challenges such as mutual interference of data across multiple domains and knowledge forgetting of various tasks. Additionally, MoE significantly increases the number of parameters, posing a computational cost challenge. Therefore, in this paper, we propose MoSLD, a mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these challenges by sharing the upper projection matrix in LoRA among different experts, encouraging the model to learn general knowledge across tasks, while still allowing the lower projection matrix to focus on the unique features of each task. The application of dropout alleviates the imbalanced update of parameter matrix and mitigates parameter overfitting in LoRA. Extensive experiments demonstrate that our model exhibits excellent performance in both single-task and multi-task scenarios, with robust out-of-domain generalization capabilities.

Figures

Figures reproduced from arXiv: 2412.08946 by the authors.

Figure 1
Figure 1. The increase between mixture setting and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the share mechansim and dropout strategy in our MoSLD. Noted that the matrix A is shared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview of our proposed Mixture-of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: A comparision of performance for LoRA, MoLA, MoSL, and MoSLD on single and mixture settings for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results of six datasets under different dropout [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Different data amount of OpenOrca between MoSLD and MoLA on six datasets. Here, we use the mixture [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Different allocation strategies for the number [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

  2. Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Sci-LoRA dynamically mixes domain-specific LoRA adapters and achieves state-of-the-art lay paraphrasing across twelve domains without needing domain labels at inference.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    Shaoxiang Chen, Zequn Jie, and Lin Ma. 2024. https://arxiv.org/abs/2401.16160 Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms . Preprint, arXiv:2401.16160

    arXiv 2024

  2. [2]

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022. https://openreview.net/forum?id=mWaYC6CZf5 On the representation collapse of sparse mixture of experts . In Advances in Neural Information Processing Systems

    work page 2022

  3. [3]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

  4. [4]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. https://arxiv.org/abs/2312.09979 Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin . Preprint, arXiv:2312.09979

    arXiv 2024

  5. [5]

    William Fedus, Jeff Dean, and Barret Zoph. 2022 a . https://arxiv.org/abs/2209.01667 A review of sparse expert models in deep learning . Preprint, arXiv:2209.01667

    arXiv 2022

  6. [6]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022 b . Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1)

    work page 2022

  7. [7]

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. 2024. https://arxiv.org/abs/2403.03432 Mixture-of-loras: An efficient multitask tuning for large language models . Preprint, arXiv:2403.03432

    arXiv 2024

  8. [8]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. 2024. https://arxiv.org/abs/2402.08562 Higher layers need more lora experts . Preprint, arXiv:2402.08562

    arXiv 2024

Show all 36 references
  1. [9]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  2. [10]

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Chao Du, Tianyu Pang, and Min Lin. 2024. https://openreview.net/forum?id=w8eCnnq57m Lorahub: Efficient cross-task generalization via dynamic lo RA composition

    2024

  3. [11]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. https://doi.org/10.1162/neco.1991.3.1.79 Adaptive mixtures of local experts . Neural Computation, 3(1):79--87

    1991 doi

  4. [12]

    Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. 2021. https://arxiv.org/abs/2109.10465 Scalable and efficient moe training for multitask multilingual models . Preprint...

    2021 arXiv

  5. [13]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages...

    2017 doi

  6. [14]

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024. https://arxiv.org/abs/2404.15159 Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts . Preprint,...

    2024 arXiv

  7. [15]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  8. [16]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2023. https://arxiv.org/abs/2310.18339 Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications . Preprint, arXiv:2310.18339

    2023 arXiv

  9. [17]

    Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. 2024. https://arxiv.org/abs/2402.12851 Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models . Preprint, arXiv:2402.12851

    2024 arXiv

  10. [18]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  11. [19]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  12. [20]

    Burges, and Erin Renshaw

    Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. https://aclanthology.org/D13-1020 MCT est: A challenge dataset for the open-domain machine comprehension of text . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pa...

    2013

  13. [21]

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr\' e Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/48237d9f2dea8c74c2a72126cf63d933-Paper.pdf Scaling vision with sparse mi...

    2021

  14. [22]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. https://openreview.net/forum?id=B1ckMDqlg Outrageously large neural networks: The sparsely-gated mixture-of-experts layer . In International Conference on Learning ...

    2017

  15. [23]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  16. [24]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  17. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  18. [26]

    Sheng Wang, Liheng Chen, Jiyue Jiang, Boyang Xue, Lingpeng Kong, and Chuan Wu. 2024. https://arxiv.org/abs/2403.00812 Lora meets dropout under a unified framework . Preprint, arXiv:2403.00812

    2024 arXiv

  19. [27]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024. https://openreview.net/forum?id=uWvKBCYh4S Mixture of lo RA experts . In The Twelfth International Conference on Learning Representations

    2024

  20. [28]

    Ted Zadouri, Ahmet \"U st \"u n, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. 2024. https://openreview.net/forum?id=EvDeiLv7qc Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning . In The Twelfth International Conf...

    2024

  21. [29]

    Weihao Zeng, Lulu Zhao, Keqing He, Ruotong Geng, Jingang Wang, Wei Wu, and Weiran Xu. 2023. https://doi.org/10.18653/v1/2023.acl-long.793 Seen to unseen: Exploring compositional generalization of multi-attribute controllable dialogue generation . In Proceedings of the 61st Ann...

    2023 doi

  22. [30]

    Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. 2023. https://openreview.net/forum?id=5r3e27I9Gy Composing parameter-efficient modules with arithmetic operation . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  23. [31]

    Lulu Zhao, Fujia Zheng, Weihao Zeng, Keqing He, Ruotong Geng, Huixing Jiang, Wei Wu, and Weiran Xu. 2022 a . https://doi.org/10.1145/3477495.3531933 Adpl: Adversarial prompt-based domain adaptation for dialogue summarization with knowledge disentanglement . In Proceedings of t...

    2022

  24. [32]

    Lulu Zhao, Fujia Zheng, Weihao Zeng, Keqing He, Weiran Xu, Huixing Jiang, Wei Wu, and Yanan Wu. 2022 b . https://doi.org/10.18653/v1/2022.naacl-main.357 Domain-oriented prefix-tuning: Towards efficient and generalizable fine-tuning for zero-shot dialogue summarization . In Pro...

    2022 doi

  25. [33]

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf Mixture-of-experts with expert...

    2022

  26. [34]

    Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, and Lei Meng. 2023. https://arxiv.org/abs/2311.09179 Sira: Sparse mixture of low rank adaptation . Preprint, arXiv:2311.09179

    2023 arXiv

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.