Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Two Is Better Than One: Rotations Scale LoRAs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding an input-dependent rotation to each LoRA output before the usual weighted-sum gating, RadarGate claims, enlarges what LoRA-MoE can express and lets it keep improving as the number of experts grows.

desk verdict Plausible incremental LoRA-MoE gating with a broken theory claim and mixed but interesting scaling evidence. read the letter →

arxiv 2505.23184 v1 pith:DFRKANLC submitted 2025-05-29 cs.LG cs.SE

Hongcan Guo , Guoshun Nan , Yuan Yang , Diyang Zhang , Haotian Li , Zhican Chen , Qinchuan Zhou , Yuhan Ran
show 4 more authors
Xinye Cao Sicong Leng Xiaofeng Tao Xudong Jiang
This is my paper
classification cs.LGcs.SE
keywords LoRAMixtureofExpertsgatingmechanismrotationmatrixparameter-efficientfine-tuningscalabilityconvexconegeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the performance collapse of LoRA-based mixture-of-experts (LoRA-MoE) as the number of experts grows is caused by the weighted-sum form of the gating mechanism, which confines outputs to a convex cone—the set of all non-negative weighted combinations—of the LoRA vectors. It proposes RadarGate, which rotates each LoRA output by an input-dependent learned rotation matrix before the usual non-negative weighting, giving the gate an extra degree of freedom beyond magnitude scaling. The paper argues this expands the hypothesis space and the effective output space, and reports that across 21 tasks and LoRA counts up to 40, the rotation sustains performance where plain weighted-sum gates decline. A sympathetic reader would care because it identifies a concrete mechanism—an extra geometric degree of freedom—for scaling parameter-efficient adaptation of large language models, not merely a larger network.

What carries the argument

The load-bearing object is the block-diagonal rotation matrix $R_i \in \mathbb{R}^{d_{\text{out}}\times d_{\text{out}}}$ whose $2\times2$ blocks $R_i^{(m)} = \begin{pmatrix}\cos\alpha_{ri}^{(m)} & -\sin\alpha_{ri}^{(m)} \\ \sin\alpha_{ri}^{(m)} & \cos\alpha_{ri}^{(m)}\end{pmatrix}$ rotate pairs of coordinates of the $i$-th LoRA output. The angles $\alpha_{ri}$ are not free constants: they are generated per input as $\alpha_{ri} = \big((x A_i B_i) \odot (x \sum_{A_jB_j \in S_i} A_jB_j)\big)\theta_r$, a learnable, input-dependent function of the Hadamard product between the LoRA output and the sum of the other LoRA outputs, so the rotation encodes pairwise relative relations between LoRA representations. This rotation is what changes the basis vectors themselves before the StretchGate applies the conventional weighted sum; the paper's Lemma 2 relies on the union of these rotated cones being strictly larger than the fixed cone to argue that targets outside the fixed cone become reachable.

What would settle it

For a trained RadarGate, measure on held-out inputs whether each target output $\Delta y_{\text{target}}(x)$ falls inside that input's cone $H'(x)$ spanned by the rotated LoRA outputs; if the typical distance to $H'(x)$ is no smaller than the distance to the unrotated cone $H$, the claimed rotation-driven expansion is not what produces the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that the well-documented performance collapse of LoRA-based mixture-of-experts as expert count grows is caused by the gating mechanism's weighted-sum form, not by the LoRA modules themselves. Because existing gates compute $\Delta y = \sum_i g_i v_i$ with $g_i \ge 0$ and $\sum_i g_i = 1$, the combined output is confined to the convex cone $H = \operatorname{conv}\{v_1,\dots,v_n\}$ of the LoRA outputs, which only allows magnitude scaling along fixed directions. RadarGate replaces this with a two-step transform: a RotationGate multiplies each LoRA output $v_i$ by an input-dependent block-diagonal rotation matrix $R_i(x;\theta_r)$, producing rotated vectors $\tilde v_i$, and a StretchGate then applies the usual non-negative, sum-to-one weights. The paper argues (Lemma 1) that this enlarges the hypothesis space from $K_{\text{gate}}$ to $K_{\text{ours}}$, and (Lemma 2) that the union over inputs of the rotated cones $\bigcup_x H'(x)$ strictly contains the original fixed cone $H$, so outputs outside $H$ become reachable. On this basis it claims better fitting, better generalization, and sustained performance when LoRA count grows from 5 to 40.

Load-bearing premise

RadarGate's theoretical case for better generalization depends on the assumption that, for each individual input, the target output lies inside that input's own rotated cone—the paper only proves that the union of such cones over all inputs is larger, not that each input's own cone reaches its own target.

Editorial extensions

If this is right

  • Scaling LoRA-MoE to 40 experts no longer follows the inverted-U pattern of baseline gates: RadarGate reports near-monotonic improvement with up to an 8% accuracy gain over the best baselines.
  • The rotation gate adds asymptotically negligible overhead—computational and memory complexity stay at the same order as existing gating methods—so the scalability gain does not come at a cost in efficiency.
  • The rotation behavior is contrastive: semantically similar LoRA representations are pulled together while distant ones are pushed apart, which the paper links to faster and more stable convergence of the gating network.
  • RadarGate also sustains a 5–10% advantage over baselines as the base model grows from 110M to 8B parameters, and in low-sample regimes (50 training samples).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same geometric degree-of-freedom argument could apply to any weighted-sum ensemble of vectors—task-vector merging, adapter stacking, or multi-modal adapter fusion—so a natural extension is testing whether input-dependent rotations before combination give comparable scaling benefits there.
  • A control experiment that swaps the rotation matrices for equal-parameter input-dependent linear maps (without orthogonality) would separate the 'rotation as geometry' explanation from mere added capacity; the paper does not run this control.
  • The contrastive alignment the authors observe suggests a regularizer on the rotation angles could explicitly encourage semantic separation between LoRA experts, making the gate's routing decisions more interpretable.
  • The per-input reachability gap (union vs. own cone) is testable: recording achieved vs. required outputs per held-out input would show whether the generalization gain actually comes from rotation-expanded reachability or from the extra parameters acting as a more flexible interaction layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RadarGate, a gating mechanism for LoRA-MoE that rotates LoRA output vectors through input-dependent relative angles (RotationGate) before weighting them (StretchGate). The motivation is that standard convex-combination gating restricts outputs to a fixed convex cone, causing underfitting and poor generalization when LoRA experts scale. The authors prove two lemmas: nested hypothesis spaces reduce fitting error, and the union of rotated dynamic cones strictly contains the fixed cone. They evaluate on GLUE, MMLU, WMT14, MATH, GSM8K, and GPQA across HydraLoRA, MoLE, and OMoE architectures, and report improvements in fitting, generalization, scaling, convergence, and low-sample settings.

Significance. The idea of adding rotational degrees of freedom to LoRA-MoE gating is original and easy to implement, and the complexity analysis suggests low overhead. If the theoretical claim were established and the empirical results robust, this would be a modest but useful contribution to parameter-efficient fine-tuning. The paper ships no code yet, and the main theoretical argument currently has a quantifier gap; the empirical tables are not as uniformly favorable as the text claims. With repair of the theory and more careful evaluation, the work could become a solid contribution.

major comments (3)
  1. [§4.2 and Appendix C.3] Lemma 2 proves only that the union over all inputs of the dynamic cones H′(x) strictly contains the fixed cone H. For a fixed test input x, however, the model's output is confined to the single dynamic cone H′(x), not to the union over other inputs, so the sentence “Thus Δytarget ∈ ∪x H′(x)\H can still be fitted” does not follow. Appendix C.3 repeats the same union quantifier and relies on a “core idea” transfer instead of a proof. Since rotations preserve norms, every H′(x) lies in the ball of radius max_i ‖v_i‖; targets outside that ball are unreachable at every input, so “outside H” is not sufficient for fitting. Please either prove per-input reachability or substantially temper the theoretical generalization claim.
  2. [Table 1 and §5.3 Obs.❷] The claim of 5–10% improvements over learnable baselines is contradicted by the reported numbers. On SST-2 under the first learnable-architecture block, Nexus achieves 70.67% while RadarGate reports 48.69%, and under the second block RadarGate's 56.30% trails Nexus's 57.33%; similar non-improvements appear on QNLI, RTE, and several MMLU rows. The text says RadarGate is best on 'over 90% of tasks,' but Table 1 does not support this. Please provide per-architecture aggregate win rates and clarify which columns belong to which architecture.
  3. [§5.1 and §5.3] All results are single point estimates with no standard deviations, number of seeds, or significance tests. Many reported differences are only 1–2 percentage points, so the load-bearing empirical claim that RadarGate sustains performance at larger scales needs uncertainty quantification before it can be accepted.
minor comments (5)
  1. [§5.3] Figure 3 references in Section 5.3 appear as unresolved '(??)' placeholders; all figure citations need to be corrected.
  2. [§4.3 and Appendix C.4] Equation (10) uses 'O r' and 'O s' while Equation (40) defines Or and Os with subscript notation; please make the notation consistent and state explicitly which complexity is for RadarGate.
  3. [§4.1] The statement that 'the absolute value of the angle is meaningless, and only the relative value of the rotation angle matters' is an unproved axiom; please either formalize the relative-angle invariance or report an experiment that tests it.
  4. [Table 1] Table 1's merged-column layout makes it difficult to determine which gating method is evaluated under which LoRA architecture; please reformat with clear column spans.
  5. [§5.4] The case study reports angle and weight values from what appears to be a single run; it should be labeled as illustrative rather than as statistical evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical lemmas are tautological or geometric but not used as fitted predictions, and no self-citation chain is load-bearing; the generalization argument's quantifier gap is a correctness issue, not circularity.

full rationale

The paper's derivation chain contains no step that reduces to its own inputs. Lemma 1 (Section 4.2, proof in Appendix C.1) restates that a larger hypothesis space cannot have a larger optimal fitting error; this is a definitional property, but the paper independently establishes K_gate ⊆ K_ours by the explicit θ_r=0 construction in Appendix C.2, so the containment is not assumed from the lemma. Lemma 2 is a geometric statement about unions of rotated convex hulls; its proof is self-contained and does not presuppose the fitting claim. The load-bearing use of Lemma 2 in Section 4.2 ('Thus Δytarget ∈ ∪_x H′(x)\H can still be fitted') is logically too strong: for a fixed test input x the model output lies in H′(x), not in the union over all inputs, so membership in the union does not by itself establish reachability at x. This is a non-circular logical gap in the generalization argument, and Appendix C.3 partially acknowledges it ('Although the original statement of the lemma is for fixed vi, the core idea...'). The empirical evaluation is independent: the gating methods are trained on FLAN subsets and evaluated on held-out GLUE, MMLU, WMT14, MATH, GSM8K, and GPQA benchmarks, with no test-set fitting or relabeled prediction. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in by citation. Accordingly, the circularity score is 0; any concern about the generalization proof belongs to correctness risk rather than circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the expressiveness of the learned rotations; the freedom introduced by θr and θs is what expands the hypothesis space. The domain assumptions are standard for LoRA-MoE. No new physical entities are postulated.

free parameters (6)
  • θr (rotation angle projection) = learned from data
    Maps input and cross-LoRA interaction features to rotation angles α_ri (Eq. 8). Central learnable parameters of RotationGate.
  • θs (gate weight projection) = learned from data
    Projects normalized concatenated LoRA outputs to gating logits (Eq. 1/21). Standard gating parameters.
  • top-k = 2 in main results (Table 1)
    Number of experts activated; chosen by hand, affects the convex cone size and scalability results.
  • temperature τ = not specified
    Softmax temperature in the gating network (Eq. 1); chosen by hand, affects sharpness of routing.
  • LoRA rank r and α = r=8, α=32
    Standard LoRA hyperparameters; determine the dimensionality of LoRA outputs and are chosen by hand.
  • low-rank factorization dim r_a for θr = not specified
    Introduced to reduce complexity (Eq. 10); value is not reported, making exact implementation ambiguous.
assumptions (5)
  • standard math Rotation matrices are orthogonal and norm-preserving (Eq. 7 and Eq. 19).
    Used to argue that rotations change direction without scaling LoRA outputs, an essential property for the claimed space expansion.
  • standard math Convex hull properties: non-negative weights summing to 1 confine outputs to the convex cone (Eq. 3).
    Basis for the observed limitation of existing gating mechanisms.
  • domain assumption The ideal target modification Δy_target can be expressed as a convex combination of LoRA outputs (Eq. 2).
    Assumes the fitting target is representable by the LoRA module outputs; not proven for real tasks.
  • domain assumption Pretrained weights and LoRA matrices are frozen; only gating parameters are trained.
    Standard for LoRA-MoE; allows isolation of gating effectiveness but limits joint adaptation.
  • ad hoc to paper The absolute value of the angle is meaningless, and only the relative value of the rotation angle matters.
    Design choice in Section 4.1 that justifies computing angles from pairwise LoRA relation Map(L_i).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Two Is Better Than One: Rotations Scale LoRAs." pith.science (2026). https://pith.science/paper/DFRKANLC

@misc{pith2026250523184,
  author       = {Pith},
  title        = {Pith review of: Two Is Better Than One: Rotations Scale LoRAs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DFRKANLC}},
  note         = {Machine review of arXiv:2505.23184}
}
read the original abstract

Scaling Low-Rank Adaptation (LoRA)-based Mixture-of-Experts (MoE) facilitates large language models (LLMs) to efficiently adapt to diverse tasks. However, traditional gating mechanisms that route inputs to the best experts may fundamentally hinder LLMs' scalability, leading to poor generalization and underfitting issues. We identify that the root cause lies in the restricted expressiveness of existing weighted-sum mechanisms, both within and outside the convex cone of LoRA representations. This motivates us to propose RadarGate, a novel geometrically inspired gating method that introduces rotational operations of LoRAs representations to boost the expressiveness and facilitate richer feature interactions among multiple LoRAs for scalable LLMs. Specifically, we first fuse each LoRA representation to other LoRAs using a learnable component and then feed the output to a rotation matrix. This matrix involves learnable parameters that define the relative angular relationship between LoRA representations. Such a simple yet effective mechanism provides an extra degree of freedom, facilitating the learning of cross-LoRA synergies and properly tracking the challenging poor generalization and underfitting issues as the number of LoRA grows. Extensive experiments on 6 public benchmarks across 21 tasks show the effectiveness of our RadarGate for scaling LoRAs. We also provide valuable insights, revealing that the rotations to each pair of representations are contrastive, encouraging closer alignment of semantically similar representations during geometrical transformation while pushing distance ones further apart. We will release our code to the community.

Figures

Figures reproduced from arXiv: 2505.23184 by the authors.

Figure 1
Figure 1. (a) Composable LoRA-MoE performs even worse than vanilla LoRA. (b) Poor general￾ization of different LoRA-MoE architectures as the number of LoRA grows. (c) Underfitting of various gating methods as the LoRA scales up. However, as the number of LoRA grows, ex￾isting LoRA-MoE gating methods may limit the LLMs’ scalability and face two critical chal￾lenges regarding generalization and underfitting [PITH_FULL_IMAGE:fi… view at source ↗
Figure 2
Figure 2. Workflow of the proposed RadarGate. Two key ingredients are RotationGate and StretchGate. RotationGate takes LoRA representations as inputs and then proceeds to three steps, including 1) LoRA representation categorization, 2) rotation angles generation, and 3) angles injection. The rotated LoRA representation will be stretched in magnitude by StretchGate to get the output. The coefficient gi only scales the magnitud… view at source ↗
Figure 3
Figure 3. Performance on Fitting and Ablation. Figure (a) shows performance of fitting capability on same-source training/test sets, while Figures (b) and (c) show ablation results for RadarGate’s StretchGate and RotationGate components [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Scaling performance comparison on three benchmarks. Figures (a), (b), and (c) compare the scaling performance of RadarGate with four baselines as the number of modules increases. Figure (d), (e), and (f) show performance across different model sizes. Obs.❸ When the num…
Figure 5
Figure 5. Figure 5: A GLUE case study visualizes RadarGate. The proposed RadarGate can correctly integrate representations along global norm weight and local angular weight by mid-training, yielding the correct answers,while the Gate in the previous MoLE method fails to generate the corre…
Figure 6
Figure 6. Figure 6: Discussion. Figure (a) shows the convergence speed of different gating methods in the same MoLE architecture with five modules. Figure (b) presents the visualization of our proposed RadarGate. Figure (c) compares method performance in MoLE under different sample sizes.…
Figure 7
Figure 7. Figure 7: The performance variations of different gates within the HydraLoRA architecture across [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: The performance variations of different gates within the MoLE architecture across six [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: The performance variations of different gates within the OMoE architecture across six [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: The accuracy of different gates within the HydraLoRA architecture across six different [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: The accuracy of different gates within the MoLE architecture across six different bench [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: The accuracy of different gates within the OMoE architecture across six different bench [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Figures (a), (b), and (c) show the loss convergence plots for module counts of 5, 15, and [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: The accuracy variations of different gates within the HydraLoRA architecture across six [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: The accuracy variations of different gates within the MoLE architecture across six different [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: The accuracy variations of different gates within the OMoE architecture across six different [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

  1. [1]

    Mtlora: Low-rank adaptation approach for efficient multi-task learning

    Agiza, A., Neseem, M., and Reda, S. Mtlora: Low-rank adaptation approach for efficient multi-task learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  2. [2]

    Y ., Rajbhandari, S., Awan, A

    Aminabadi, R. Y ., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InSC22: International Conference for High Performance Computing, Networking, Storage and Analysis, 2022

    work page 2022

  3. [3]

    and Purwar, A

    B, G. and Purwar, A. Evaluating the efficacy of open-source llms in enterprise-specific RAG systems: A comparative study of performance and scalability.CoRR, 2024

    work page 2024

  4. [4]

    Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. s. Findings of the 2014 workshop on statistical machine translation. InProceedings of the Ninth Workshop on Statistical Machine Translation, 2014

    work page 2014

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  6. [6]

    Omoe: Diversifying mixture of low-rank adaptation by orthogonal finetuning

    Feng, J., Pu, Z., Hu, T., Li, D., Ai, X., and Wang, H. Omoe: Diversifying mixture of low-rank adaptation by orthogonal finetuning. 2025

    work page 2025

  7. [7]

    Nexus: Specialization meets adaptability for efficiently training mixture of experts.CoRR, 2024

    Gritsch, N., Zhang, Q., Locatelli, A., Hooker, S., and ¨Ust¨un, A. Nexus: Specialization meets adaptability for efficiently training mixture of experts.CoRR, 2024

    work page 2024

  8. [8]

    Fusemoe: Mixture-of- experts transformers for fleximodal fusion

    Han, X., Nguyen, H., Harris, C., Ho, N., and Saria, S. Fusemoe: Mixture-of- experts transformers for fleximodal fusion. In Globersons, A., Mackey, L., Bel- grave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.),Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 202...

    work page 2024

Show all 35 references
  1. [9]

    Fastmoe: A fast mixture-of-expert training system.arXiv preprint arXiv:2103.13262, 2021

    He, J., Qiu, J., Zeng, A., Yang, Z., Zhai, J., and Tang, J. Fastmoe: A fast mixture-of-expert training system.arXiv preprint arXiv:2103.13262, 2021

    2021 arXiv

  2. [10]

    Mea- suring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mea- suring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  3. [11]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations, 2022

    2022

  4. [12]

    Y ., Pang, T., Du, C., and Lin, M

    Huang, C., Liu, Q., Lin, B. Y ., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic loRA composition. InFirst Conference on Language Modeling, 2024. 10

    2024

  5. [13]

    Tutel: Adaptive mixture-of-experts at scale.Proceedings of Machine Learning and Systems, 2023

    Hwang, C., Cui, W., Xiong, Y ., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., et al. Tutel: Adaptive mixture-of-experts at scale.Proceedings of Machine Learning and Systems, 2023

    2023

  6. [14]

    Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

    Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., Yang, M., and Rhu, M. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference. 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2023

    2024

  7. [15]

    T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. InThe Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023

    2023

  8. [16]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 1991

    1991

  9. [17]

    Matrix-transformation based low-rank adapta- tion (mtlora): A brain-inspired method for parameter-efficient fine-tuning.arXiv preprint arXiv:2403.07440, 2024

    Liang, Y ., Wang, Y ., Li, Y ., and Zeng, Y . Matrix-transformation based low-rank adapta- tion (mtlora): A brain-inspired method for parameter-efficient fine-tuning.arXiv preprint arXiv:2403.07440, 2024

    2024 arXiv

  10. [18]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  11. [19]

    Scaling laws for fine-grained mixture of experts

    Ludziejewski, J., Krajewski, J., Adamczewski, K., Pi´oro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr ´ol, K., Odrzyg ´o´zd´z, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. InICLR 2024 Workshop on Mathematical and Empirical...

    2024

  12. [20]

    Lsh-moe: Communication-efficient moe training via locality-sensitive hashing

    Nie, X., Liu, Q., Fu, F., Zhu, S., Miao, X., Li, X., Zhang, Y ., Liu, S., and Cui, B. Lsh-moe: Communication-efficient moe training via locality-sensitive hashing. InAd- vances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Sy...

    2024

  13. [21]

    Mode: Effective multi-task parameter efficient fine-tuning with a mixture of dyadic experts.arXiv preprint arXiv:2408.01505, 2024

    Ning, L., Lara, H., Guo, M., and Rastogi, A. Mode: Effective multi-task parameter efficient fine-tuning with a mixture of dyadic experts.arXiv preprint arXiv:2408.01505, 2024

    2024 arXiv

  14. [22]

    M., Charlin, L., Roux, N

    Ostapenko, O., Su, Z., Ponti, E. M., Charlin, L., Roux, N. L., Caccia, L., and Sordoni, A. Towards modular llms by building and reusing a library of loras. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024

    2024

  15. [23]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  16. [24]

    V ., Hinton, G

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V ., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 201...

    2017

  17. [25]

    D., Ng, A., and Potts, C

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of EMNLP, 2013

    2013

  18. [26]

    Hydralora: An asymmetric lora architecture for efficient fine-tuning.arXiv preprint arXiv:2404.19245, 2024

    Tian, C., Shi, Z., Guo, Z., Li, L., and Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning.arXiv preprint arXiv:2404.19245, 2024

    2024 arXiv

  19. [27]

    W., Lester, B., Du, N., Dai, A

    Wei, J., Bosma, M., Zhao, V ., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V . Finetuned language models are zero-shot learners. InInternational Conference on Learning Representations, 2022

    2022

  20. [28]

    Mixture of lora experts

    Wu, X., Huang, S., and Wei, F. Mixture of lora experts. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. 11

    2024

  21. [29]

    Flex-moe: Modeling arbitrary modality combination via the flexible mixture- of-experts

    Yun, S., Choi, I., Peng, J., Wu, Y ., Bao, J., Zhang, Q., Xin, J., Long, Q., and Chen, T. Flex-moe: Modeling arbitrary modality combination via the flexible mixture- of-experts. 2024. URL http://papers.nips.cc/paper_files/paper/2024/hash/ b2f2af5403042b1344f4e93b35fb67d9-Abstr...

    2024

  22. [30]

    Composing parameter-efficient modules with arithmetic operations.ArXiv, 2023

    Zhang, J., Chen, S., Liu, J., and He, J. Composing parameter-efficient modules with arithmetic operations.ArXiv, 2023

    2023

  23. [31]

    Galore: Memory- efficient LLM training by gradient low-rank projection

    Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y . Galore: Memory- efficient LLM training by gradient low-rank projection. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024

    2024

  24. [32]

    Multi-lora composition for image generation.CoRR, 2024

    Zhong, M., Shen, Y ., Wang, S., Lu, Y ., Jiao, Y ., Ouyang, S., Yu, D., Han, J., and Chen, W. Multi-lora composition for image generation.CoRR, 2024

    2024

  25. [33]

    M., Le, Q

    Zhou, Y ., Lei, T., Liu, H., Du, N., Huang, Y ., Zhao, V ., Dai, A. M., Le, Q. V ., Laudon, J., et al. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems, 2022

    2022

  26. [34]

    Y ., Dai, A

    Zhou, Y ., Lei, T., Liu, H., Du, N., Huang, Y ., Zhao, V . Y ., Dai, A. M., Chen, Z., Le, Q. V ., and Laudon, J. Mixture-of-experts with expert choice routing. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.),Advances in Neural Information Proc...

    2022

  27. [35]

    Uni-perceiver-moe: Learning sparse generalist models with conditional moes

    Zhu, J., Zhu, X., Wang, W., Wang, X., Li, H., Wang, X., and Dai, J. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.),Advances in Neural Information Processing Systems 3...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.