Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By giving shallow layers lower-rank adapters and deep layers higher-rank adapters, HILO claims to beat existing adapter-MoE fine-tuning in accuracy while using fewer trainable and active parameters.

desk verdict A plausible incremental idea — per-layer rank in adapter-MoE — but the main comparisons are compromised by selecting the rank schedule on the same benchmarks and by confounding rank with expert count and parameter budget. read the letter →

arxiv 2502.03884 v1 pith:YFHFDQTW submitted 2025-02-06 cs.LG cs.AI

Peizhuang Cong , Wenpu Liu , Wenhan Yu , Haochen Zhao , Tong Yang This is my paper · ORCID
classification cs.LGcs.AI
keywords mixtureofexpertsLoRAparameter-efficientfine-tuninglow-rankadaptationhierarchicalrankallocationlayer-wiseexpertlargelanguagemodelsactiveparameters
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in mixture-of-adapter-experts fine-tuning of large language models, the rank of each adapter expert is as important as the number of experts per layer. It introduces HILO, a hierarchical configuration that sets lower ranks for shallow layers and higher ranks for deeper layers, while also varying the number of experts. The central claim is that this joint tuning improves accuracy over existing methods (MoLA, AlphaLoRA, AdaMoE) while introducing the same or fewer trainable parameters and reducing active parameters by about 37.5 percent. If correct, this would make per-layer rank scheduling a cheap and effective efficiency lever for LoRA-based MoE fine-tuning. It also implies that layer representational complexity can be matched at the granularity of adapter capacity, not just expert count.

What carries the argument

The central object is the hierarchical rank-setting formula of HILO. It computes the rank of layer $i$ as a piecewise-linear interpolation between a minimum rank $r_{\min}$ and a maximum rank $r_{\max}$ over the layer index, optionally holding the rank constant for every $l$ layers. This formula is the load-bearing mechanism because it defines the capacity of each adapter expert and, together with the expert count, directly determines the trainable and active parameter counts. The supporting mechanism is the empirical observation (presented as output distribution plots) that shallow-layer adapter outputs are closer to zero, which motivates lower ranks for shallow layers.

What would settle it

Run an ablation that reverses HILO's rank schedule, assigning the highest ranks to the shallowest layers while keeping the same total number of trainable parameters and the same expert counts. If the reversed schedule matches or outperforms HILO's accuracy on the six reported datasets, the depth-increasing rank hierarchy is not the source of the gains. Alternatively, compare HILO against a uniform-rank baseline with the same total trainable parameters and the same number of experts per layer; if the uniform baseline achieves equal or higher accuracy, then the rank schedule adds no value beyond the expert-count allocation.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that existing mixture-of-adapter-experts architectures optimize only the number of adapter experts per layer and neglect the rank of each adapter, which also determines the parameter size and fitting capacity. By analyzing the output distributions of FFN layers and their adapters during fine-tuning of Llama 2-7B, the paper observes that shallow layers produce adapter outputs closer to zero relative to the original network, suggesting that shallow layers need lower adapter capacity. HILO operationalizes this by assigning ranks that increase with layer depth (for example, 2, 4, 6, 8 across quarter-layer groups) and adjusting expert counts to meet a fixed trainable-parameter budget. The paper reports that HILO outperforms MoLA, AlphaLoRA, and AdaMoE on six benchmarks while using equal or smaller trainable parameter sizes and fewer active parameters.

Load-bearing premise

The load-bearing premise is that shallow layers genuinely need less adapter capacity than deep layers, so assigning them low rank and fewer experts does not sacrifice accuracy; this is inferred from the observation that shallow-layer adapter outputs are close to zero, not from a causal or accuracy-based test.

Editorial extensions

If this is right

  • If HILO's central claim is correct, adapter-MoE fine-tuning can be made simultaneously more accurate and cheaper by replacing a uniform rank with a depth-increasing rank schedule.
  • For any fixed trainable-parameter budget, the method suggests that total capacity should be redistributed toward deeper layers, both in the number of experts and in the rank of each expert.
  • Active parameters during inference drop by roughly 37.5% relative to the vanilla mixture-of-adapter-experts baseline, which could lower per-token inference cost in deployed LoRA-MoE models.
  • The method implies that layer-wise representational complexity can be matched at adapter granularity, so fine-tuning efficiency may improve by treating rank as a per-layer hyperparameter rather than a global constant.
  • The reported gains over AlphaLoRA (about 1.01 percentage points average) and over AdaMoE (about 1.04 percentage points average) suggest that rank heterogeneity is complementary to expert-count allocation and token-adaptive routing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's motivation, that shallow-layer adapter outputs are close to zero, could be tested more directly: one could ablate by assigning the highest ranks to the shallowest layers and measure whether accuracy degrades, which would make the depth-increasing direction load-bearing.
  • The hierarchical rank principle may generalize beyond layer index: the same output-distribution analysis could identify other structural factors (e.g., attention vs. FFN, or residual-stream magnitude) that predict where extra rank pays off, potentially leading to learned or adaptive rank allocation.
  • Because HILO operates purely at configuration level (rank and expert count), it could be combined with quantization or pruning methods: low-rank shallow adapters might tolerate more aggressive quantization than high-rank deep adapters, further shrinking the fine-tuned model.
  • The reported improvements are demonstrated on Llama 2-7B and six datasets; a natural testable extension is whether the depth-increasing rank schedule also holds for larger models, different architectural families, or non-commonsense tasks such as instruction tuning or code generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HILO, a hierarchical configuration scheme for mixtures of adapter experts in LLM fine-tuning. Unlike prior methods that only vary the number of LoRA experts per layer, HILO also varies the rank of each adapter expert across layers, using fewer and lower-rank experts in shallow layers and more and higher-rank experts in deep layers. The authors motivate this design with an analysis of output-value distributions across layers, then evaluate HILO on six datasets (ScienceQA, CommonsenseQA, OpenBookQA, MRPC, COLA, RTE) using Llama 2-7B. They report that HILO outperforms MoLA, AlphaLoRA, and AdaMoE in accuracy while reducing trainable and active parameters, with the main comparisons carried out at matched total trainable-parameter sizes.

Significance. The paper addresses a relevant design question in parameter-efficient fine-tuning: whether adapter-expert rank, in addition to expert count, should vary across layers. The proposed HILO scheme is simple, easy to implement, and the evaluation covers three reasoning tasks and three GLUE tasks with a widely used base model. The paper also explicitly tracks active-parameter reductions during inference, which is a useful efficiency metric. If the reported results are robust, the contribution would be a modest but useful architectural guideline for adapter-MoE fine-tuning. However, the current evidence is weakened by the fact that the adopted rank configuration was selected on the same evaluation datasets, by ablations that change several variables at once, and by the absence of any variance or seed information. The claimed gains are therefore plausible but not yet firmly established.

major comments (5)
  1. [Section 5.2 (Exploratory Studies) and Tables 1–2] The adopted rank configuration '2468' is selected in Figure 6 as the best among five configurations evaluated on the same six datasets that are later reported as the main results in Tables 1 and 2. Because the configuration selection is not made on a held-out split, the reported average improvements (1.01 percentage points over AlphaLoRA and 1.04 percentage points over AdaMoE) are partly a test-set selection artifact. Please provide an independent validation split for configuration selection, or otherwise report selection-controlled results.
  2. [Section 5.2 (1) and Table 1] In the HILO versus MoLA and HILO versus AlphaLoRA comparisons, the number of experts per layer, the rank schedule, and the active-parameter count all change simultaneously while only the total trainable-parameter size is matched. Consequently, the reported gains cannot be attributed specifically to the hierarchical rank setting; they could also result from the different expert-count distribution or from having more, smaller experts. An ablation that holds expert allocation and total trainable parameters constant while varying only the rank schedule is needed to isolate the contribution of rank.
  3. [Table 3 (Ablation Studies)] The Mix versus HILO comparison in Table 3 varies both the rank schedule and the trainable-parameter budget (0.63 versus 0.39), so the 1.30% accuracy difference does not isolate the effect of hierarchical rank. Please add an equal-budget ablation with identical expert counts and activation policy, varying only the rank sequence, to support the claim that hierarchical rank, rather than the reduced parameter count, drives the improvement.
  4. [Sections 3.1–3.2] The motivating evidence that shallow layers need lower rank is distributional only, and Section 3.2 explicitly states that minimizing small-value proportions is not the final objective. The monotone-rank hypothesis is therefore not supported by an accuracy-based test. For example, permuting the rank schedule under a fixed parameter budget would test whether the depth-increasing order rather than the specific rank values is responsible for the observed gains.
  5. [All results tables (Tables 1–3, Figures 5–6)] All empirical results are reported as single numbers without error bars, seeds, or variance information. Given that several reported differences are small (e.g., a 0.08-point decrease on CQA and a 0.01-point increase on RTE in Table 1), the robustness of the central claim cannot be assessed. Please report multiple seeds with means and standard deviations, or otherwise justify why variance is negligible.
minor comments (5)
  1. [Section 5.2, first paragraph] The text 'reduces the rank values progressively from deep to shallow layers, grouped every eight layers, to 8, 4, 6, and 2' is inconsistent with the monotone shallow-to-deep schedule 2, 4, 6, 8 used in Figure 6 and Section 4.3. Please clarify whether the listed values are in deep-to-shallow or shallow-to-deep order and correct the list accordingly.
  2. [Section 5.1 and Table 1] The symbol '▽' is used both for MoLA's expert-allocation strategy and for HILO's hierarchical rank configuration, which makes Table 1 difficult to interpret. Please introduce distinct notations for these two settings and use them consistently.
  3. [Section 4.3] The sentence introducing the formula says 'the number of adapter experts in each layer can be expressed as' but the formula defines r_i, which the surrounding text treats as a rank value. Please align the prose with the formula and clearly distinguish r_min/r_max (ranks) from the expert-count variables used in Section 4.1.
  4. [Throughout] There are several typos and inconsistencies, including 'Large Langue Model' in the Introduction, 'for for every l layers' in Section 4.3, and mixed capitalization of 'Vanilla'/'vanilla' and 'HILO'/'H ILO'. A careful proofreading pass is recommended.
  5. [Figure 6] The caption does not identify which marker corresponds to which rank configuration, and the subplot arrangement is not described. Please add a legend (e.g., labeling the green diamond as the 2468 configuration) so the reader can verify the selection of the 2468 setting.

Circularity Check

1 steps flagged · score 6.0 of 10

The rank schedule used in the headline comparisons was selected from five candidates on the same six evaluation datasets, so the reported gains are partly a selection artifact rather than an independent prediction.

  1. fitted input called prediction [Section 5.2, Exploratory Studies (Figure 6; configuration used in Tables 1-3)]
    "HILO is evaluated by various rank configurations, specifically grouping every eight layers with ranks set as 2448, 2288, 2468, 2488, and 2888 from shallow to deep layers, with the corresponding trainable parameters gradually increasing. The parameter size of 2468 configuration is assigned unit 1. The accuracy results for these five configurations across six datasets are shown in Figure 6. It can be observed that the configuration of 2468 (green ♦) achieves the best balance between parameter size and accuracy. This configuration is also adopted in the above experiments."

    All main HILO results in Tables 1-3 use the 2468 rank schedule. That schedule is not derived a priori from the formula in Section 4.3; instead it is chosen in the Exploratory Studies as the best of five candidates according to accuracy on the same six datasets (SQA, CQA, OQA, MRPC, COLA, RTE) that are later reported as the method's performance. The paper explicitly says 'This configuration is also adopted in the above experiments.' With no held-out split or pre-registered schedule described, the headline improvements over MoLA, AlphaLoRA, and AdaMoE are partly a selection artifact: the configuration was picked because it scored best on the evaluation set, so the reported average gains are not independent predictions of HILO as a method.

full rationale

This paper's central claim is that the hierarchical rank schedule HILO improves fine-tuning accuracy while reducing trainable and active parameters. The main experimental support is not a derivation from first principles but a set of benchmark comparisons. The only part that reduces to its own input is the choice of the specific rank configuration: Section 5.2's Exploratory Studies evaluates five schedules (2448, 2288, 2468, 2488, 2888) on the exact six datasets later used as headline results, declares 2468 the best balance, and then states 'This configuration is also adopted in the above experiments.' Since the central comparisons in Tables 1-3 all use 2468, the reported average improvements (1.01% over AlphaLoRA, 1.04% over AdaMoE) are not out-of-sample predictions of a pre-specified method; they are the result of selecting a hyperparameter on the evaluation set. This is a genuine, if partial, circularity: the 'prediction' is fitted to the benchmark. I do not count the expert-count/rank confound as circular, though it is a separate weakness: HILO versus MoLA/AlphaLoRA varies the number of experts, the rank schedule, and active parameters simultaneously, so the improvement cannot be attributed cleanly to rank alone. Similarly, the paper's own distributional motivation is explicitly disclaimed as the objective in Section 3.2 ('the final goal of model fine-tuning is not to minimize such proportions'), so it provides only plausibility, not proof. The rank list in Section 5.2 ('8, 4, 6, and 2' from deep to shallow) is also non-monotonic and inconsistent with the monotone formula in Section 4.3, but this is a correctness or consistency issue rather than circularity. Overall, the configuration selection is load-bearing for the headline numbers, so the score is 6.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a small number of hand-chosen rank parameters and a domain assumption about layer-wise capacity needs. No new physical or architectural entities are introduced beyond the adapter experts already present in the cited literature.

free parameters (5)
  • Minimum rank rmin = 2
    Set by hand in Section 4.3 as the base rank for shallow layers; no sensitivity analysis or derivation.
  • Maximum rank rmax = 8 in main experiments; 16 and 32 listed as options
    Chosen by hand; values other than 8 are not evaluated in the main comparisons.
  • Layer grouping interval l = 8
    Groups of eight layers for Llama 2-7B; no ablation of the grouping interval is reported.
  • Adopted rank configuration = 2,4,6,8 across four eight-layer groups
    Selected in Section 5.2 exploratory studies as the best balance on the same six evaluation datasets, then used in all main comparisons.
  • Top-K activation = 2
    Most experiments use Top-2 activation; the paper mentions Top-P and placeholder variants but does not fully specify them.
assumptions (6)
  • domain assumption Shallow layers perform general feature extraction and deep layers learn specialized features, so required adapter capacity increases with depth.
    Stated in Section 3.1 and used to justify the rank hierarchy; supported only by output magnitude plots, not by accuracy measurements.
  • domain assumption The distribution of adapter output magnitudes relative to original network outputs is an informative proxy for how much fine-tuning capacity a layer needs.
    Used in Section 3.1 and Figure 2; the paper notes in Section 3.2 that minimizing small outputs is not the final goal, so the link to accuracy is asserted.
  • domain assumption MoE layer capability scales with expert count and rank but with diminishing returns once individual experts exceed layer requirements.
    Used in Section 3.1 to justify jointly tuning expert count and rank; no formal capacity model is provided.
  • domain assumption Baseline configurations (MoLA and AlphaLoRA expert allocations) and their reported parameter budgets are correctly transcribed and comparable.
    The paper relies on the original papers for layer-wise expert counts and does not independently validate those numbers beyond its tables.
  • domain assumption Results on the six chosen datasets transfer to other fine-tuning tasks.
    No held-out validation of the rank schedule or cross-task transfer test is reported.
  • standard math Standard linear algebra and softmax gating for MoE-LoRA behave as in prior work.
    The output equation in Section 4.1 uses weighted sums of expert outputs with normalized gate probabilities, consistent with prior MoE-LoRA methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning." pith.science (2026). https://pith.science/paper/YFHFDQTW

@misc{pith2026250203884,
  author       = {Pith},
  title        = {Pith review of: Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFHFDQTW}},
  note         = {Machine review of arXiv:2502.03884}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable success across various tasks, accompanied by a continuous increase in their parameter size. Parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address the challenges of fine-tuning LLMs by significantly reducing the number of trainable parameters. Recent studies have integrated LoRA with Mixture of Experts (MoE) architectures, leveraging multiple adapter experts and gating mechanisms to further improve fine-tuning performance. However, existing approaches primarily focus on adjusting the allocations of adapter experts per layer to optimize the introduced trainable parameter size, while neglecting a critical factor of adapters' rank. To this end, we propose a hierarchical scheme for expert allocation and rank configuration, HILO, which dynamically adjusts the number and rank of adapter experts across layers, matching the varying representational complexity of model layers in adapter-granularity. Extensive experiments on multiple benchmark tasks demonstrate that HILO outperforms existing methods in accuracy while introducing fewer trainable parameters, providing an efficient and practical solution for fine-tuning LLMs.

Figures

Figures reproduced from arXiv: 2502.03884 by the authors.

Figure 1
Figure 1. Optimization focus of adapter experts. Existing works primarily optimize the number of adapters, whereas this paper ex￾plores both the number and rank of adapters to improve fine-tuning performance. adapters) into the model’s weight matrices, enabling efficient fine-tuning for downstream tasks without introducing train￾able parameters size substantially. Based on LoRA, several studies have further optimized it by in… view at source ↗
Figure 3
Figure 3. Proportions of values (<10−3 ) of vanilla method, MoLA, and HILO. (1) the proportions of three methods in the 1st layer are very close; (2) except 1st layer, MoLA has lower pro￾portions than Vanilla and HILO in 4 and 2 layers, respectively; (3) HILO exhibits lower proportions than Vanilla and MoLA in 5 layers. deep layer, rather than equally, is a straightforward and effec￾tive allocation strategy to maximize the ef… view at source ↗
Figure 2
Figure 2. Analysis of output values between original model weight [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: HILO Design 1 to 8) of HILO are presented in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Accuracy comparison between HILO and MoLA. HILO assigns experts and rank as ▽ across layers. Adapter Experts Parameter ↓ Rank Top-K Active Trainable Accuracy (%) ↑ 8 2 1.03 0.63 88.44 ▽ 2 0.77 0.39 89.74 8 ▽ 1.65 0.63 90.56 ▽ ▽ 1.23 0.39 90.24 [PITH_FULL_IMAGE:figures…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 24 canonical work pages

  1. [1]

    A survey on evaluation of large language models

    [Chang et al., 2024] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xi- aoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transac- tions on Intelligent Systems and Technology , 15(3):1–45,

    work page 2024

  2. [3]

    Qlora: Efficient fine- tuning of quantized llms

    [Dettmers et al., 2024] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient fine- tuning of quantized llms. Advances in Neural Information Processing Systems, 36,

    work page 2024

  3. [4]

    Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin

    [Dou et al., 2024] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pa...

    work page 2024

  4. [5]

    Glam: Efficient scaling of language models with mixture- of-experts

    [Du et al., 2022] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture- of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR,

    work page 2022

  5. [7]

    Higher layers need more lora experts

    [Gao et al., 2024] Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562,

    arXiv 2024

  6. [8]

    Parameter-efficient fine- tuning for large models: A comprehensive survey

    [Han et al., 2024] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608,

    arXiv 2024

  7. [9]

    Lora+: Efficient low rank adaptation of large models

    [Hayou et al., 2024] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learn- ing,

    work page 2024

  8. [10]

    Parameter-efficient transfer learning for nlp

    [Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, Stanis- law Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Interna- tional conference on machine learning, pages 2790–2799. PMLR,

    work page 2019

Show all 33 references
  1. [12]

    Harder tasks need more experts: Dynamic routing in moe mod- els

    [Huang et al., 2024] Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Li- wei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe mod- els. In The 62nd Annual Meeting of the Association for Computational ...

    2024

  2. [13]

    Adaptive mix- tures of local experts

    [Jacobs et al., 1991] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mix- tures of local experts. Neural computation, 3(1):79–87,

    1991

  3. [15]

    Gshard: Scaling giant models with conditional computa- tion and automatic sharding

    [Lepikhin et al., 2021] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computa- tion and automatic sharding. In International Conference on Learn...

    2021

  4. [17]

    Prefix- tuning: Optimizing continuous prompts for generation

    [Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processi...

    2021

  5. [18]

    Mixlora: Enhancing large language models fine- tuning with lora based mixture of experts

    [Li et al., 2024] Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine- tuning with lora based mixture of experts. arXiv preprint arXiv:2404.15159,

    2024 arXiv

  6. [19]

    Learn to ex- plain: Multimodal reasoning via thought chains for sci- ence question answering

    [Lu et al., 2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to ex- plain: Multimodal reasoning via thought chains for sci- ence question answering. Advances in Neural Information Processi...

    2022

  7. [20]

    Can a suit of armor con- duct electricity? a new dataset for open book question an- swering

    [Mihaylov et al., 2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor con- duct electricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing , pa...

    2018

  8. [22]

    Alphalora: Assigning lora experts based on layer training quality

    [Qing et al., 2024] Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush V osoughi. Alphalora: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pro- cessing, pag...

    2024

  9. [23]

    Scaling vision with sparse mixture of experts

    [Riquelme et al., 2021] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr´e Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595,

    2021

  10. [24]

    Hash layers for large sparse mod- els

    [Roller et al., 2021] Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse mod- els. Advances in Neural Information Processing Systems, 34:17555–17566,

    2021

  11. [25]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    [Talmor et al., 2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational L...

    2019

  12. [26]

    Llama 2: Open foundation and fine-tuned chat models

    [Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    2023 arXiv

  13. [29]

    Parameter-efficient sparsity crafting from dense to mixture-of-experts for instruction tuning on gen- eral tasks

    [Wu et al., 2024a] Haoyuan Wu, Haisheng Zheng, Zhuolun He, and Bei Yu. Parameter-efficient sparsity crafting from dense to mixture-of-experts for instruction tuning on gen- eral tasks. arXiv preprint arXiv:2401.02731,

    arXiv

  14. [30]

    Moral: Moe aug- mented lora for llms’ lifelong learning

    [Yang et al., 2024a] Shu Yang, Muhammad Asif Ali, Cheng- Long Wang, Lijie Hu, and Di Wang. Moral: Moe aug- mented lora for llms’ lifelong learning. arXiv preprint arXiv:2402.11260,

    arXiv

  15. [31]

    Xmoe: Sparse models with fine-grained and adaptive expert se- lection

    [Yang et al., 2024b] Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. Xmoe: Sparse models with fine-grained and adaptive expert se- lection. In Findings of the Association for Computational Linguistics ACL 2024, pages 11664–11674,

    2024

  16. [32]

    Adamoe: Token- adaptive routing with null experts for mixture-of-experts language models

    [Zeng et al., 2024] Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token- adaptive routing with null experts for mixture-of-experts language models. In Findings of the Association for Com- putational Linguistics ACL 2024,

    2024

  17. [33]

    Mixture-of-experts with expert choice routing

    [Zhou et al., 2022] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022

    2022

  18. [1991]

    Vera: Vector-based random matrix adaptation

    [Kopiczko et al., 2024] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations,

    2024

  19. [2017]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    [Wang et al., 2019] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Confer- ence on Learning Representations,

    2019

  20. [2018]

    Mul- timodal contrastive learning with limoe: the language- image mixture of experts

    [Mustafa et al., 2022] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language- image mixture of experts. Advances in Neural Information Processing Systems, 35:9564–9576,

    2022

  21. [2019]

    Lora: Low-rank adaptation of large language mod- els

    [Hu et al., 2022] Edward J Hu, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language mod- els. In The Tenth International Conference on Learning Representations,

    2022

  22. [2021]

    The power of scale for parameter-efficient prompt tuning

    [Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045–3059,

    2021

  23. [2022]

    Switch transformers: Scaling to trillion param- eter models with simple and efficient sparsity

    [Fedus et al., 2022] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion param- eter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39,

    2022

  24. [2023]

    Attention is all you need

    [Vaswani, 2017] A Vaswani. Attention is all you need. Ad- vances in Neural Information Processing Systems,

    2017

  25. [2024]

    Deepseek- moe: Towards ultimate expert specialization in mixture- of-experts language models

    [Dai et al., 2024] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseek- moe: Towards ultimate expert specialization in mixture- of-experts language models. In The 62nd Annual Meeting of the Associat...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.