Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

CoMoL claims that a mixture of LoRA experts can specialize per token while keeping parameter and compute counts at the level of a single LoRA, because each expert can be stored as a small core matrix inside shared low-rank projection bases.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:50 UTC pith:6O56DUAM

load-bearing objection A clean architectural trick that genuinely gets MoE-LoRA down to LoRA-level parameter/FLOP budgets, but the empirical claim of consistent SOTA is mostly noise, and the shared-subspace assumption is untested. the 3 major comments →

arxiv 2603.00573 v2 pith:6O56DUAM submitted 2026-02-28 cs.CL

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Jie Cao , Zhenxuan Fan , Zhuonan Wang , Tianwei Lin , Ziyuan Zhao , Rolan Yan , Wenqiao Zhang , Feifei Shao
show 3 more authors
Hongwei Wang Jun Xiao Siliang Tang
This is my paper
classification cs.CL
keywords core spaceMoE-LoRAparameter-efficient fine-tuningtoken-level routinglow-rank adaptationexpert mergingLLM fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper targets a known inefficiency in MoE-LoRA fine-tuning: expert diversity costs N independent projection pairs, so trainable parameters and per-token compute both grow with the number of experts. CoMoL's proposal is to reparameterize each LoRA expert as a shared pair of projection matrices plus an r×r core matrix, route tokens among experts inside that low-rank core space, and soft-merge the selected cores before applying the shared projections. The claimed outcome is per-token expert adaptation at roughly the parameter count, FLOPs, and wall-clock latency of standard LoRA, with experiments on mathematical reasoning and code generation at the 8B and 14B scale. A sympathetic reading is that the algebraic reduction of N LoRA computations into one is the main insight, and the shared-subspace premise is what makes it work. The paper's own Limitations section (between the conclusion and acknowledgments) flags that learning capacity across fine-tuning scenarios remains underexplored, so the empirical scope is two task families.

Core claim

The central move is the identity ΔW = U_B M V_A^T, where M is the r×r core matrix of a LoRA pair. Because every expert update can be written this way, the token-weighted combination Σ_i G(x)_i U_B M_i V_A^T x rearranges to U_B (Σ_i G(x)_i M_i) V_A^T x. CoMoL accordingly shares U_B and V_A across all experts, treats each M_i as the only expert-specific parameter, routes using the already-computed r-dimensional vector V_A^T x, and forms one merged core per token. The paper argues that this keeps token-level routing while cutting expert parameter complexity from O(N(m+n)r) to O((m+n)r + N r^2) and expert compute to that of a single LoRA, and reports that it matches or beats MoE-LoRA baselines o

What carries the argument

The core matrix of a LoRA expert: after reducing the paired matrices, M = Σ_B V_B^T U_A Σ_A, so any LoRA update is ΔW = U_B M V_A^T. CoMoL makes U_B and V_A shared, stores each expert as M_i, routes in the low-rank space via x̂ = V_A^T x, and soft-merges the selected M_i per token. This object carries the load: it converts a sum over N LoRA experts into a single low-rank update whose r×r coefficient matrix varies with the input token.

Load-bearing premise

The load-bearing premise, stated in Section 3.2, is that all experts share essentially the same low-rank projection subspace (the bases U_B and V_A), with only the r×r core differing; if real expert specializations live in different subspaces, CoMoL collapses toward a single constrained LoRA and the token-level expert benefit disappears.

What would settle it

Train independent full LoRA adapters on two deliberately different domains, compute the principal angles between their left and right singular subspaces, and check whether one pair of bases can span both. If the subspaces are far apart, CoMoL's fixed U_B and V_A cannot represent both specializations; a complementary ablation that frees U_B and V_A per expert and yields a large accuracy jump would also show how much of the result rests on the shared-subspace assumption.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Router parameters no longer scale with the hidden dimension: the gating matrix is N×r instead of N×n, so the expert count can rise without a matching memory bill.
  • Per-token soft-merging keeps the FLOPs of one LoRA layer while preserving input-adaptive specialization, unlike instance-level merging.
  • Empirically, CoMoL reaches 64 experts on both 8B- and 14B-scale models, where a leading MoE-LoRA baseline runs out of GPU memory at 16 experts.
  • Training time is roughly half that of soft-weighted MoE-LoRA baselines and about a quarter that of sparse MoE baselines, with inference latency close to LoRA.
  • If the shared-subspace premise holds, the method becomes a drop-in replacement for the LoRA branch in transformer layers and inherits LoRA's broad applicability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not stress-test its shared-subspace premise; one test is to train independent LoRAs on two well-separated domains and measure the principal angles between their singular subspaces. Large angles would predict that CoMoL's fixed U_B and V_A cannot span both domains.
  • Token-level soft-merging makes the routed update a convex combination of core matrices, which suggests CoMoL could also serve as a learnable interpolator for task composition: new tasks could be added by inserting an M_i and adjusting router weights only.
  • The Limitations section explicitly leaves cross-scenario learning capacity open; an editor's inference is that the reported gains may weaken when a task needs many distinct high-rank directions, since the shared bases cap the total subspace available.
  • Because the router lives in the same reduced space as the merged core, the routing weights occupy interpretable coordinates of the LoRA subspace; inspecting M_i overlap or routing entropy over a dataset could reveal whether experts specialize by token type or merely re-weight a common direction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CoMoL, a Mixture-of-LoRA-Experts variant that reparameterizes every LoRA expert as U_B M_i V_A^T with a single shared pair of singular bases (U_B, V_A) and an expert-specific r×r core matrix M_i. A token-level router is projected into the same low-rank space, and the activated cores are soft-merged into one effective core before the low-rank projection is applied. The authors argue this reduces the per-expert parameter cost from O(N(m+n)r) to O((m+n)r + N r^2), keeps FLOPs comparable to a single LoRA layer, and preserves token-level expert diversity. Experiments on math reasoning (Qwen3-8B/14B) and code generation (Qwen3-8B, Llama3.1-8B) compare CoMoL against Standard LoRA and several MoE-LoRA baselines, with additional scaling and wall-clock experiments.

Significance. If the central assumption is valid, CoMoL would be a practically useful bridge between parameter-efficient LoRA and token-adaptive MoE: it gives near-LoRA parameter counts while retaining input-dependent expert weighting. The algebraic derivation in Section 3 is clean and the parameter/FLOP arithmetic follows from the construction; the paper also provides code, a fairly broad set of baselines, two backbone families, and wall-clock measurements. These are genuine strengths. The significance, however, hinges on an empirically untested hypothesis: that independently useful LoRA experts concentrate their updates in a shared low-rank subspace. Without direct evidence for that hypothesis, the claim that CoMoL preserves MoE-LoRA adaptability is not yet established. The statistical support for 'consistently outperforming' is also weak, with most math-table differences within one standard deviation and no error bars on code results.

major comments (3)
  1. [§3.2, Eqs. (11)–(13)] The shared-subspace assumption is load-bearing. Eq. (11) writes every expert as U_B M_i V_A^T with one shared (U_B, V_A); Eq. (13) then gives the per-token merged update as U_B (Σ_i G_i M_i) V_A^T, whose rank is at most r regardless of N. The paper labels this a 'hypothesis' but provides no analysis or experiment testing when it holds, how training maintains suitable shared bases, or how much expert diversity is lost. The empirical comparison against methods with independent B_i,A_i is suggestive but does not isolate the assumption: a degenerate baseline (a single LoRA whose r×r core is token-scaled by the router) could potentially match CoMoL's expressiveness. Please add an explicit test of expert diversity (e.g., measuring the effective rank/alignment of learned expert cores, or comparing against a single-LoRA-plus-token-scalar-core baseline) and a discussion of failure modes when expe
  2. [Tables 2 and 3; Table 4] The abstract's claim of 'consistently outperforming existing methods' is not statistically supported. In Table 2 the average accuracy for CoMoL is 84.48±1.40, while FlyLoRA is 83.64±0.40 and SparseMoA is 83.69±0.76; the differences are within one standard deviation. On AQuA, CoMoL has 44.62±5.32, a huge interval. Table 3 has the same issue (e.g., CoMoL 86.34±1.01 vs. SparseMoA 86.16±0.77 on average). Table 4 reports pass@k without any error bars or seed counts, and Figure 2 likewise appears to show single runs. Please report per-seed numbers or confidence intervals, run more seeds where feasible, and avoid 'consistent' claims unless a paired significance test or a clear effect-size criterion supports them.
  3. [§4.5, Table 6] The efficiency claim 'comparable to Standard LoRA' needs qualification. Table 6 shows training wall-clock time of 2:19:55 for CoMoL versus 0:56:47 for LoRA — a 2.5× gap. The text correctly attributes the lower training cost to being roughly half of MoLoRA/HydraLoRA, but the conclusion says 'computational costs comparable to Standard LoRA.' If the comparison is limited to forward expert FLOPs (as in Table 1, which excludes router overhead), please state that explicitly and reconcile the two statements. This matters because practitioners will read the abstract and conclusion as a broader efficiency claim.
minor comments (4)
  1. [Tables 2–4] Typos and formatting: 'SV AMP' appears in Tables 2–3; 'HydraLora' is spelled inconsistently in Table 4. Table 1 should clarify that the '×' ratios are relative to Standard LoRA and that the routing overhead is excluded from the FLOPs column.
  2. [Eq. (14)–(15)] The Core Space Routing reuses V_A^T x as the router input and also as the right projection in the main adapter. It would be helpful to discuss whether this coupling constrains the router or creates interference between routing and adaptation, and to add an ablation that uses a separately learned projection for the router.
  3. [§4.3] Figure 2 would benefit from error bars and explicit seed counts, especially since the text claims 'consistently outperforms' at all ranks. Currently the reader cannot assess whether the differences are meaningful.
  4. [Limitations] The Limitations section appropriately states that no systematic benchmark exists for learning capacity across PEFT methods. Given that admission, the paper should also acknowledge that the shared-subspace assumption is a distinct limitation and point to future work that directly tests it.

Circularity Check

0 steps flagged

No significant circularity: core-space reparameterization is algebraic and the efficiency claims are tested against independent baselines.

full rationale

I traced the derivation chain across Sections 3.1–3.3 and the experiments. Equation 9 is the exact SVD of the LoRA matrices B and A; Equation 10 defines the core matrix M as Σ_B V_B^T U_A Σ_A, so Equation 11 follows algebraically rather than assuming the conclusion. Equations 12–13 and 14–15 are obtained by substitution and associativity of matrix multiplication, so the parameter-count and FLOP reductions are by-construction consequences of the shared-subspace design, not a hidden reuse of experimental outputs. The shared-subspace claim in Section 3.2 is explicitly labeled a hypothesis, and the experiments compare CoMoL against independent baselines (LoRA, MoLoRA, HydraLoRA, MoLA, AdaMoLE, SparseMoA, DenseLoRA, FlyLoRA) on held-out evaluation benchmarks; no parameter is fitted to a subset and then reported as a prediction of a closely related quantity. The only self-referential citations, notably Cao et al. 2025 for SparseMoA latency and setup, are contextual/empirical support and are not load-bearing for CoMoL's central construction or main comparisons. No derivation step reduces to its own inputs, so I find no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No fitted constants; the only hand-chosen values are architecture hyperparameters (rank r, expert count N) reported in the experiments. The central claim rests on the shared-subspace hypothesis, which is stated but not independently validated.

axioms (5)
  • domain assumption Individual LoRA experts share a substantially similar latent subspace, so a single pair (U_B, V_A) can serve all experts.
    Section 3.2: 'we hypothesize that individual experts share a substantially similar latent subspace defined by the singular bases U_B and V_A.' This is the core load-bearing assumption; if false, per-expert specialization is lost.
  • standard math The reduced SVD reparameterization ΔW = U_B M V_A^T with shared U_B, V_A preserves the expressive power of per-expert low-rank updates.
    Section 3.1: follows from SVD if U_B, V_A are the expert's own singular bases, but becomes an approximation when they are shared across experts and trained jointly.
  • standard math Token-level soft-merging via the distributive property (Eq. 13) is equivalent to applying experts separately (Eq. 12).
    The algebra is correct: G_i scalars commute with matrix products; used to justify FLOP reduction.
  • domain assumption The low-rank router input xhat = V_A^T x retains enough information for routing decisions.
    Section 3.3: routing in projected core space reduces router parameters from O(Nn) to O(Nr); no analysis of information loss for routing.
  • domain assumption Math14k and CodeAlpaca-20k fine-tuning transfers to the evaluation benchmarks used.
    Standard empirical transfer assumption; not proven beyond reported results.

pith-pipeline@v1.3.0-alltime-deepseek · 13534 in / 12564 out tokens · 122533 ms · 2026-08-02T19:50:33.346329+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameter-efficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instance-level routing. To address these issues, we propose Core Space Mixture of LoRA (\textbf{CoMoL}), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: core space experts and core space routing. Core space experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core space routing dynamically selects and activates the appropriate core experts for each token, enabling fine-grained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL retains the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to standard LoRA, consistently outperforming existing methods across multiple tasks.

Figures

Figures reproduced from arXiv: 2603.00573 by Feifei Shao, Hongwei Wang, Jie Cao, Jun Xiao, Rolan Yan, Siliang Tang, Tianwei Lin, Wenqiao Zhang, Zhenxuan Fan, Zhuonan Wang, Ziyuan Zhao.

Figure 1
Figure 1. Figure 1: Comparison of LoRA-based methods. (a) LoRA injects trainable low-rank matrices A and B into transformer layers to approximate weight updates. (b) MoE-LoRA employs multiple LoRA experts with a routing network to select and activate experts based on input tokens. (c) CoMoL fuses expert parameters in the low-rank core space, preserving token-level routing while maintaining expert-specific expressiveness with … view at source ↗
Figure 2
Figure 2. Figure 2: Performance and trainable parameter comparison between CoMoL and LoRA across different ranks on the code generation benchmark. CoMoL w/o CR CoMoL # Experts Math # Param Math # Param Qwen3-8B 8 84.47 33.39M 84.48 25.16M 16 82.50 42.02M 84.00 25.56M 32 82.49 59.28M 82.24 26.34M 64 84.50 93.78M 82.72 27.91M Qwen3-14B 8 86.08 47.90M 86.34 35.82M 16 85.92 60.46M 86.24 36.26M 32 86.13 85.52M 86.06 37.13M 64 86.3… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

    cs.LG 2026-07 conditional novelty 6.0

    PMDRouter selects LoRAs zero-shot by decoding scale-normalized linear response energy from one adapter-free backbone prefill, and leads most internal-signal baselines on a new multi-granularity EPM bench.

Reference graph

Works this paper leans on

38 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    Jie Cao, Tianwei Lin, Hongyang He, Rolan Yan, Wenqiao Zhang, Juncheng Li, Dongping Zhang, Siliang Tang, and Yueting Zhuang. 2025. Moa: Heterogeneous mixture of adapters for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2506.05928

    arXiv 2025

  3. [3]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation

    2023

  4. [4]

    Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter-efficient tuning: Are we really there yet? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2612--2626

    2022

  5. [5]

    Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    Pith/arXiv arXiv 2021

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    Pith/arXiv arXiv 2021

  7. [7]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. http://arxiv.org/abs/2312.09979 LoRAMoE : Alleviate World Knowledge Forgetting in Large Language Models via MoE - Style Plugin . arXiv preprint. ArXiv:2312.09979 [cs]

    Pith/arXiv arXiv 2024

  8. [8]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and V. S. Subrahmanian. 2024. https://doi.org/10.48550/arXiv.2402.08562 Higher Layers Need More LoRA Experts . arXiv preprint. ArXiv:2402.08562 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.08562 2024

  9. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. https://doi.org/10.48550/arXiv.2110.04366 Towards a Unified View of Parameter - Efficient Transfer Learning . arXiv preprint. ArXiv:2110.04366 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2110.04366 2022

  11. [11]

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 523--533

    2014

  12. [12]

    Houlsby, A

    N. Houlsby, A. Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and S. Gelly. 2019. https://www.semanticscholar.org/paper/29ddc1f43f28af7c846515e32cc167bc66886d0c Parameter- Efficient Transfer Learning for NLP . ArXiv

    2019

  13. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. http://arxiv.org/abs/2106.09685 LoRA : Low - Rank Adaptation of Large Language Models . arXiv preprint. ArXiv:2106.09685 [cs]

    Pith/arXiv arXiv 2021

  14. [14]

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. http://arxiv.org/abs/2304.01933 LLM - Adapters : An Adapter Family for Parameter - Efficient Fine - Tuning of Large Language Models . arXiv preprint. ArXiv:2304.01933 [cs]

    Pith/arXiv arXiv 2023

  15. [15]

    Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585--597

    2015

  16. [16]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.243 The Power of Scale for Parameter - Efficient Prompt Tuning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045--3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.243 2021

  17. [17]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix- Tuning : Optimizing Continuous Prompts for Generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( Volume 1: Long Papers ) , pages 4582--4597...

    doi:10.18653/v1/2021.acl-long.353 2021

  18. [18]

    Tianwei Lin, Jiang Liu, Wenqiao Zhang, Zhaocheng Li, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Hao Jiang, and 1 others. 2024. Teamlora: Boosting low-rank adaptation with expert collaboration and competition. arXiv preprint arXiv:2408.09856

    Pith/arXiv arXiv 2024

  19. [19]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146

    Pith/arXiv arXiv 2017

  20. [20]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950--1965

    2022

  21. [21]

    Zefang Liu and Jiahua Luo. 2024. https://doi.org/10.48550/arXiv.2405.00361 AdaMoLE : Fine - Tuning Large Language Models with Adaptive Mixture of Low - Rank Adaptation Experts . arXiv preprint. ArXiv:2405.00361 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.00361 2024

  22. [22]

    Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. 2024. http://arxiv.org/abs/2402.12851 MoELoRA : Contrastive Learning Guided Mixture of Experts on Parameter - Efficient Fine - Tuning for Large Language Models . arXiv preprint. ArXiv:2402.12851 [cs]

    Pith/arXiv arXiv 2024

  23. [23]

    Lin Mu, Xiaoyu Wang, Li Ni, Yang Li, Zhize Wu, Peiquan Jin, and Yiwen Zhang. 2025. Denselora: Dense low-rank adaptation of large language models. arXiv preprint arXiv:2505.23808

    Pith/arXiv arXiv 2025

  24. [24]

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. 2024. https://doi.org/10.48550/arXiv.2306.03745 Soft Merging of Experts with Adaptive Routing . arXiv preprint. ArXiv:2306.03745 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.03745 2024

  25. [25]

    Accurate and efficient low-rank model merging in core space

    Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Bart omiej Twardowski, Andrew D Bagdanov, Simone Calderara, and Joost van de Weijer. Accurate and efficient low-rank model merging in core space. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

  26. [26]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191

    Pith/arXiv arXiv 2021

  27. [27]

    Subhro Roy and Dan Roth. 2016. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413

    Pith/arXiv arXiv 2016

  28. [28]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    Pith/arXiv arXiv 2017

  29. [29]

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. 2024. http://arxiv.org/abs/2404.19245 HydraLoRA : An Asymmetric LoRA Architecture for Efficient Fine - Tuning . arXiv preprint. ArXiv:2404.19245 [cs]

    Pith/arXiv arXiv 2024

  30. [30]

    Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.388 AdaMix : Mixture -of- Adaptations for Parameter -efficient Model Tuning . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 5744--5760, Abu...

    work page doi:10.18653/v1/2022.emnlp-main.388 2022

  31. [31]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. https://www.aclweb.org/anthology/2020.emnlp-demos.6 Transformers...

    2020

  32. [32]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024. https://doi.org/10.48550/arXiv.2404.13628 Mixture of LoRA Experts . arXiv preprint. ArXiv:2404.13628 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.13628 2024

  33. [33]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  34. [34]

    Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. 2023. http://arxiv.org/abs/2309.05444 Pushing Mixture of Experts to the Limit : Extremely Parameter Efficient MoE for Instruction Tuning . arXiv preprint. ArXiv:2309.05444 [cs]

    Pith/arXiv arXiv 2023

  35. [35]

    Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, and Lei Meng. 2023. https://doi.org/10.48550/arXiv.2311.09179 SiRA : Sparse Mixture of Low Rank Adaptation . arXiv preprint. ArXiv:2311.09179 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.09179 2023

  36. [36]

    Flylora: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts

    Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. Flylora: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

  37. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  38. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...