A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

Boyu Diao; Junzhou Xu

REVIEW 4 major objections 4 minor 24 references

A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Allocating LoRA experts to the most gradient-sensitive parameter blocks raises accuracy on eight commonsense benchmarks to 83.2% with only 1.60% of parameters trainable.

desk verdict Sensitivity-based expert allocation for LoRA-MoE is a genuinely new idea, but the reported accuracy gains are selection outcomes, not verified predictions, and the parameter-count comparison with MoLA is misstated. read the letter →

arxiv 2505.06272 v1 pith:WB3WFHFZ submitted 2025-05-06 cs.LG cs.AI

Junzhou Xu , Boyu Diao This is my paper · ORCID

classification cs.LGcs.AI

keywords LoRAMixtureofExpertsparameter-efficientfine-tuningparametersensitivitygradient-basedexpertallocationmulti-tasklearningcommonsensereasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the per-block expert counts in a LoRA-MoE fine-tune can be chosen automatically from gradient information rather than from hand-set heuristics such as putting more experts in higher layers. The proposed LoRA-SMoE method samples a small batch of task data, computes each parameter block's sensitivity as the cumulative sum of squared gradients, and spends a fixed expert budget on the most sensitive blocks. With the budget split separately between self-attention and MLP modules and set at the top 60% of sensitive blocks, it reports higher average accuracy than fixed-allocation baselines while training fewer parameters. If correct, this means expert budgets can be tuned cheaply per task, which matters for resource-constrained multi-task fine-tuning.

What carries the argument

The load-bearing object is a per-block sensitivity score: for each matrix block (Q, K, V, O in attention; up, down, gate in MLP), the score is the sum of squared gradients $s_n = \sum g_n^2$ computed over a small sample of task data. Expert counts are then assigned to the top 60% of blocks under a fixed total budget, either across all blocks (uniform), separately within attention and MLP (separate), or per matrix (independent). The fine-tuning backbone is an asymmetric LoRA-MoE with a shared low-rank matrix $A$ and task-specific $B_i$ matrices combined by soft routing, $y = W_0 x + \sum_{i=1}^{E}\omega_i B_i A x$, so 'experts' are collections of $B$ matrices. Sensitivity is computed one parameter group at a time, keeping peak memory comparable to ordinary LoRA fine-tuning.

What would settle it

Compute sensitivity on a held-out set of tasks, then allocate experts to the least sensitive 60% of blocks under the same separate attention/MLP budgets; if that allocation matches or beats the most-sensitive allocation, the sensitivity signal is not what drives the gain. Alternatively, if a unified allocation beats the separate-budget method on a different base model or task family, the attention/MLP split is the operative ingredient rather than the gradient signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that parameter sensitivity, measured as the cumulative sum of squared gradients over backpropagation on a small sample, predicts where LoRA experts should be placed: within a fixed budget, assigning more experts to the most sensitive parameter blocks improves downstream accuracy, provided the budget is allocated separately to self-attention and MLP modules. On eight commonsense reasoning benchmarks with a 36-layer 3B instruction-tuned model, the authors report 83.2% average accuracy for LoRA-SMoE-S at the top 60% sensitivity threshold, with 1.60% trainable parameters, compared with 82.9% for HydraLoRA (2.51%) and 82.8% for MoLA (1.38%). They also find that attention layers are most sensitive in higher layers and that MLP middle layers are least sensitive, which refines rather than overturns the existing rule that higher layers need more LoRA experts. A unified allocation that simply takes the most sensitive blocks across all modules underperforms, so the attention/MLP separation is part of the claim rather than a detail.

Load-bearing premise

The load-bearing premise is that a block's summed squared gradient on a small data sample tells where LoRA experts will help most, a premise the paper's own unified-allocation result shows is incomplete without the extra attention/MLP budget split.

Editorial extensions

If this is right

Expert allocation no longer needs hand-tuned layer rules: a sensitivity scan on 36 to 288 samples, taking roughly 10 to 66 seconds in the paper's measurements, fixes per-block expert counts before training begins.
Dropping the least sensitive 40% of parameter blocks from the expert budget cuts trainable parameters from 2.51% (HydraLoRA) to 1.60% while the average accuracy rises from 82.9% to 83.2%, and the 60% threshold outperforms both 20% and 80% thresholds.
Attention and MLP modules need separate expert budgets, because a unified sensitivity ranking overallocates experts to MLP blocks and performs worse than the separate-budget variant.
The sensitivity scan adds only forward and backward passes on a small sample, so the allocation procedure is inexpensive enough to run per task before fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If gradient sensitivity is as stable across sample sizes as the paper's consistency table suggests, the same per-block allocation could be computed once and reused across related tasks, though the paper only verifies one 3-billion-parameter model.
The reported insensitivity of MLP middle layers points to a more aggressive recipe the authors did not test: freeze or share a single expert across those blocks and move the saved parameter budget into attention blocks.
The sensitivity metric is model-agnostic, so the allocation recipe could plausibly extend to other parameter-efficient mechanisms such as adapters or prompt vectors, but the paper does not test that transfer.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know about this paper is that the idea is better than the evidence. LoRA-SMoE allocates LoRA-MoE experts by gradient sensitivity, with separate budgets for attention and MLP blocks. On eight commonsense benchmarks on Qwen2.5-3B, the S variant averages 83.2%, slightly above HydraLoRA (82.9%) and MoLA (82.8%), while using fewer parameters than HydraLoRA. The cheap sensitivity measure looks stable: Table III shows that expert selections made with 36 samples are over 85% consistent with the 288-sample selections for most datasets. The finding that unified sensitivity selection (LoRA-SMoE-U) underperforms while the module-separated version (S) wins is interesting, and suggests that sensitivity alone is not the whole story.

The soft spots are real and load-bearing. There are no seeds or error bars, and the differences between methods are around 0.3 points. More importantly, the 60% threshold and the choice of the S variant come from comparing test-set averages in the same Table II used for the final claims, so the reported numbers are selected, not predicted. The 'fewer trainable parameters' claim also misreads the table: LoRA-SMoE-S uses 1.60% trainable parameters versus MoLA's 1.38%, so it is not fewer than MoLA. On top of that, the U variant falls below HydraLoRA, which means the success of S depends on the manually imposed separation of attention/MLP budgets rather than on the sensitivity principle alone. Code is promised but not available, so independent verification is not yet possible.

What is genuinely useful here is the sensitivity measurement itself and the consistency analysis across sample sizes. That part is cheap, reproducible, and could be adopted by other LoRA-MoE work. The paper is honest in its limitations: no theoretical derivation, single model scale. But the main empirical claim needs stronger evidence before it can be trusted.

I would send this to peer review, because the idea deserves a serious referee and the experiments can be meaningfully improved: multi-seed runs, fixed hyperparameters chosen on a validation split or a subset of tasks, and a clearer parameter-budget comparison. Right now I would not cite the accuracy gains as a reliable result, but I would cite the sensitivity consistency observation once the code is out and verified.

Referee Report

4 major / 4 minor

Summary. The paper proposes LoRA-SMoE, a method for allocating per-module expert budgets in LoRA-based MoE fine-tuning. The allocation is driven by a sensitivity score defined as the sum of squared gradients computed on small sampled subsets of each task's training data. Three variants are introduced: unified selection (LoRA-SMoE-U), separate selection for self-attention and MLP modules (LoRA-SMoE-S), and per-matrix selection (LoRA-SMoE-I). On Qwen2.5-3B-Instruct across eight commonsense reasoning benchmarks, the authors report that LoRA-SMoE-S with a 60% retention budget achieves an average accuracy of 83.2%, compared with 82.9% for HydraLoRA and 82.8% for MoLA, while also studying the stability of the sensitivity estimates under different sample sizes and the consistency of expert selection across tasks.

Significance. If the central empirical claim were verified, this would be a useful contribution: a cheap, gradient-based sensitivity measure could replace manual expert-allocation heuristics in LoRA-MoE fine-tuning, and the paper's sensitivity-stability analysis (Table III) addresses an important practical question about sample sizes. The code link and the experiments on a 3B model also provide a concrete basis for reproduction. However, the reported gains over HydraLoRA and MoLA are small, are computed on the same benchmarks used to choose the 60% threshold and the S variant, and are presented without any variance or significance estimates. In addition, the claim of reducing trainable parameters relative to MoLA is not supported by the paper's own Table II. The core idea is plausible and the limitations are honestly stated, but the evidence as presented does not yet establish the advertised advantage.

major comments (4)

[Section IV-C and Table II] The 60% redundancy level is selected by comparing LoRA-SMoE-S(20%), LoRA-SMoE-S(40%), LoRA-SMoE-S(60%), and LoRA-SMoE-S(80%) on the same eight test benchmarks that are later used for the final comparison in Table II. Likewise, the S variant is chosen over U and I in Section IV-D using the same test-table averages. This means the reported 83.2% average is a selection outcome, not a predictive evaluation on an independent benchmark suite. Please provide an evaluation on held-out tasks, or use a separate validation split for model selection and report the test-set results only once, so the reported numbers are not outcomes of choosing hyperparameters on the test set.
[Table II] No random seeds, error bars, or significance tests are reported anywhere in the paper. The headline differences are 83.2% versus 82.9% for HydraLoRA and 83.2% versus 82.8% for MoLA, and per-task differences are often 0.1 to 0.3 points. On eight- or ten-thousand-example commonsense benchmarks, these differences are well within the range of seed-to-seed variation. The paper should report multiple seeds with mean and standard deviation for at least the main comparison, and ideally a paired significance test, before the claimed improvement can be evaluated.
[Section IV-D, Table II] The abstract and conclusion state that the proposed method reduces the number of trainable parameters, but Table II lists Tuned/Total as 1.38% for MoLA-▽(8642) and 1.60% for LoRA-SMoE-S(60%). LoRA-SMoE-S therefore trains more parameters than MoLA, not fewer. The sentence in Section IV-D claiming 'superior performance compared to MoLA-▽ under the same parameters' is also inconsistent with this table. Please correct the parameter-efficiency claim and provide a parameter-matched comparison, or state explicitly that the comparison against MoLA uses a larger parameter budget.
[Section IV-D, LoRA-SMoE-U] LoRA-SMoE-U(60%), the pure sensitivity-ranked allocation, achieves 82.55%, below HydraLoRA's 82.9%, and the paper itself states that 'allocating experts based only on parameter sensitivity is insufficient.' The success of LoRA-SMoE-S therefore depends on the manually imposed separate selection of attention and MLP budgets, not on the sensitivity criterion alone. The paper should justify this module-level constraint as part of the method, show that it is not itself a hand-crafted heuristic, and assess how sensitive the result is to the split between the two modules.

minor comments (4)

[Section IV-C] The sentence 'we conducted experiments on a dataset of size 15k' is ambiguous: it is not clear whether the eight datasets were each subsampled to 15,000 examples, whether a single 15,000-example mixture was used, or whether '15k' refers to something else. Please specify the exact data used for the redundancy exploration.
[Introduction and Reference [12]] The method name is written inconsistently as 'MOLA' in the introduction and as 'MoLA' in the experimental sections and reference [12]. Please use a single spelling throughout.
[Equation (3)] In Equation (3), 'sof tmax' should be 'softmax'. In addition, the routing layer R and the input x are not defined precisely; please state the dimensions of R and whether x is the layer input, the hidden state, or a concatenation of both.
[Section II-C and Section V] The paper states in Section II-C that 'MLP layers exhibit higher parameter sensitivity values compared to attention layers,' while Section I says that 'parameters in the middle layers of MLPs appear relatively insensitive.' These statements are not contradictory if mean sensitivity is meant in the first and per-layer distribution is meant in the second, but the distinction should be stated explicitly to avoid confusing readers about the actual sensitivity pattern.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported accuracy gains are test-set selection outcomes: the 60% budget and the S variant were chosen after inspecting Table II on the same eight benchmarks, so 83.2% is not an independent prediction.

fitted input called prediction [Section IV-C (Preliminary Exploration of Parameter Redundancy), Section IV-D (Method Comparison), Table II]
"Our method achieved better results by retaining only the top 60 % most sensitive experts, indicating that more experts do not necessarily lead to better performance and confirming the existence of expert redundancy. Therefore, in subsequent experiments, we consistently selected the top 60 % most sensitive parameters for optimization. ... The experimental results (see table II) show that our LoRA-SMoE-S achieved the best performance."

The 60% budget is chosen because it 'achieved better results' on the same eight test benchmarks that later support the conclusion, and the S variant is chosen after inspecting Table II on those same benchmarks. The headline average of 83.2% is thus the best cell of a small configuration grid evaluated on the test sets, not an independent prediction of a pre-specified method. The 'therefore' sentence makes the test-set selection explicit, and the same table is then cited as evidence of superiority. Additionally, LoRA-SMoE-U's underperformance shows that the sensitivity ranking alone does not force the reported gains; the success of LoRA-SMoE-S depends on the additional manually imposed separation of attention and MLP budgets, which is also fitted to the same evaluation results.

full rationale

The paper contains no self-citation chain or definitional circularity: the sensitivity score is computed from gradients on training subsets independently of test accuracy, and the core allocation rule does not reference the target metric. The circularity burden is in the experimental protocol. In Section IV-C the authors choose the 60% threshold because, in Table II, it 'achieved better results' on the very eight benchmarks used for evaluation; Section IV-D then selects S over U and I after comparing the same table. The headline 83.2% is therefore a test-set selection outcome, not a predictive evaluation of a pre-specified method. The differences over HydraLoRA (0.3 points) and MoLA (0.4 points) are within the range of configuration-grid variation, and no seeds, confidence intervals, or significance tests are reported, so the numbers cannot distinguish selection noise from method benefit. The paper's own Limitations passage concedes the sensitivity analysis is empirical rather than rigorously derived, which further reduces the force of any mathematical-circularity critique but does not fix the selection problem. The separate claim that LoRA-SMoE-S matches MoLA 'under the same parameters' is also contradicted by Table II (1.60% vs 1.38%), but that is a factual inconsistency, not circularity. Overall, the central empirical claim currently reduces, at least in part, to fitted hyperparameter selection on the evaluation set: score 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on two fitted choices (the 60% threshold and the S variant) selected using the evaluation benchmarks, and on the assumption that full-weight gradient sensitivity predicts LoRA-effectiveness. The sensitivity measure itself is not fitted to the target metric.

free parameters (3)

Redundancy level (top-x% sensitive blocks retained) = 60%
Section IV-C tries 20%, 40%, 60%, 80% and selects 60% because it gives the best average on the eight test benchmarks. This selection is based on the same test sets used for the final comparison, so the reported result is partly a product of this choice.
Allocation variant (U/S/I) = S (separate self-attention and MLP selection)
Section IV-D reports that S outperforms U and I; the variant is chosen from the test results. This is a discrete hyperparameter selected on the evaluation set.
Sensitivity sample size C = 108
Section IV-B sets C=108. Table III shows expert-selection consistency is stable across sample sizes, so this choice is less critical, but it remains a hand-set hyperparameter.

assumptions (4)

domain assumption Gradient norms with respect to pretrained parameters are a valid proxy for where LoRA adapters should be placed.
Section III-A defines sensitivity as the sum of squared gradients and uses it to allocate experts. This assumption is not theoretically derived and is partially contradicted by the U variant's underperformance (Section IV-D).
domain assumption Multi-task fine-tuning on the eight benchmarks can be treated with a single shared model using soft routing.
Section IV-A and IV-B fine-tune Qwen2.5-3B-Instruct jointly on BoolQ, PIQA, SIQA, Winogrande, ARC-E, ARC-C, OBQA, and HellaSwag, following the HydraLoRA setup.
standard math The pretrained weights are frozen and only LoRA/B matrices are trained.
LoRA definition in Eq. (1) and the fine-tuning module in Eq. (2); the base model W0 is frozen.
domain assumption The sensitivity computed on a small sample (C=108) generalizes to the full training distribution.
Section IV-E provides consistency evidence across sample sizes, but generalization to the training distribution is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning." pith.science (2026). https://pith.science/paper/WB3WFHFZ

@misc{pith2026250506272,
  author       = {Pith},
  title        = {Pith review of: A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WB3WFHFZ}},
  note         = {Machine review of arXiv:2505.06272}
}

read the original abstract

As deep learning models expand, the pre-training-fine-tuning paradigm has become the standard approach for handling various downstream tasks. However, shared parameters can lead to diminished performance when dealing with complex datasets involving multiple tasks. While introducing Mixture-of-Experts (MoE) methods has alleviated this issue to some extent, it also significantly increases the number of parameters required for fine-tuning and training time, introducing greater parameter redundancy. To address these challenges, we propose a method for allocating expert numbers based on parameter sensitivity LoRA-SMoE (A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning). This method rapidly assesses the sensitivity of different tasks to parameters by sampling a small amount of data and using gradient information. It then adaptively allocates expert numbers within a given budget. The process maintains comparable memory consumption to LoRA (Low-Rank Adaptation) while ensuring an efficient and resource-friendly fine-tuning procedure. Experimental results demonstrate that compared to SOTA fine-tuning methods, our LoRA-SMoE approach can enhance model performance while reducing the number of trainable parameters. This significantly improves model performance in resource-constrained environments. Additionally, due to its efficient parameter sensitivity evaluation mechanism, LoRA-SMoE requires minimal computational overhead to optimize expert allocation, making it particularly suitable for scenarios with limited computational resources. All the code in this study will be made publicly available following the acceptance of the paper for publication. Source code is at https://github.com/EMLS-ICTCAS/LoRA-SMoE

Figures

Figures reproduced from arXiv: 2505.06272 by the authors.

**Figure 1.** Parameter blocks sensitivity of Qwen2.5-3B-Instruct on eight datasets, the darker the color, the higher the sensitivity. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** LoRA-SMoE Framework Overview(steps A and B show our method for one layer’s parameter blocks) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Parameter blocks sensitivity obtained with different sampling sizes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 5 canonical work pages

[1]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , 2019, pp. 4171–4186

2019
[2]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023
[3]

P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,

X. Liu, K. Ji, Y . Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” arXiv preprint arXiv:2110.07602 , 2021

arXiv 2021
[4]

Clip-kd: An empirical study of clip model distillation,

C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, B. Diao, and Y . Xu, “Clip-kd: An empirical study of clip model distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 952–15 962

2024
[5]

Sketch-fusion: a gradient compression method with multi-layer fusion for communication-efficient distributed training,

L. Dai, L. Gong, Z. An, Y . Xu, and B. Diao, “Sketch-fusion: a gradient compression method with multi-layer fusion for communication-efficient distributed training,” Journal of Parallel and Distributed Computing, vol. 185, p. 104811, 2024

work page 2024
[6]

etag: Class- incremental learning via embedding distillation and task-oriented gener- ation,

L. Huang, Y . Zeng, C. Yang, Z. An, B. Diao, and Y . Xu, “etag: Class- incremental learning via embedding distillation and task-oriented gener- ation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 11, 2024, pp. 12 591–12 599

work page 2024
[7]

Continual learning in the frequency domain,

R. Liu, B. Diao, L. Huang, Z. An, Z. An, and Y . Xu, “Continual learning in the frequency domain,” Advances in Neural Information Processing Systems, vol. 37, pp. 85 389–85 411, 2024

work page 2024
[8]

Brain-inspired fast-and slow-update prompt tuning for few-shot class- incremental learning,

H. Ran, X. Gao, L. Li, W. Li, S. Tian, G. Wang, H. Shi, and X. Ning, “Brain-inspired fast-and slow-update prompt tuning for few-shot class- incremental learning,” IEEE Transactions on Neural Networks and Learning Systems, 2024

work page 2024

Show all 24 references

[9]

Lora: Low-rank adaptation of large language models

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

2022
[10]

Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning,

T. Zadouri, A. ¨Ust¨un, A. Ahmadian, B. Ermis ¸, A. Locatelli, and S. Hooker, “Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning,” arXiv preprint arXiv:2309.05444 , 2023

2023 arXiv
[11]

Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment,

S. Dou, E. Zhou, Y . Liu, S. Gao, J. Zhao, W. Shen, Y . Zhou, Z. Xi, X. Wang, X. Fan et al., “Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment,” arXiv preprint arXiv:2312.09979, vol. 4, no. 7, 2023

2023 arXiv
[12]

Higher layers need more lora experts,

C. Gao, K. Chen, J. Rao, B. Sun, R. Liu, D. Peng, Y . Zhang, X. Guo, J. Yang, and V . Subrahmanian, “Higher layers need more lora experts,” arXiv preprint arXiv:2402.08562 , 2024

2024 arXiv
[13]

Sgdr: Stochastic gradient descent with warm restarts,

I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016

2016 arXiv
[14]

Sensitivity-aware visual parameter-efficient fine-tuning,

H. He, J. Cai, J. Zhang, D. Tao, and B. Zhuang, “Sensitivity-aware visual parameter-efficient fine-tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 825–11 835

2023
[15]

Hydralora: An asym- metric lora architecture for efficient fine-tuning,

C. Tian, Z. Shi, Z. Guo, L. Li, and C.-Z. Xu, “Hydralora: An asym- metric lora architecture for efficient fine-tuning,” Advances in Neural Information Processing Systems , vol. 37, pp. 9565–9584, 2024

2024
[16]

Boolq: Exploring the surprising difficulty of natural yes/no questions,

C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044 , 2019

1905 arXiv
[17]

Piqa: Reasoning about physical commonsense in natural language,

Y . Bisk, R. Zellers, J. Gao, Y . Choi et al. , “Piqa: Reasoning about physical commonsense in natural language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432– 7439

2020
[18]

Socialiqa: Commonsense reasoning about social interactions,

M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y . Choi, “Socialiqa: Commonsense reasoning about social interactions,” arXiv preprint arXiv:1904.09728, 2019

1904 arXiv
[19]

Winogrande: An adversarial winograd schema challenge at scale,

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM , vol. 64, no. 9, pp. 99–106, 2021

2021
[20]

Think you have solved question answering? try arc, the ai2 reasoning challenge,

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , 2018

2018 arXiv
[21]

Can a suit of armor conduct electricity? a new dataset for open book question answering,

T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” arXiv preprint arXiv:1809.02789 , 2018

2018 arXiv
[22]

Hel- laswag: Can a machine really finish your sentence?

R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019

1905 arXiv
[23]

Qwen2. 5 technical report,

A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

2024 arXiv
[24]

Decoupled weight decay regularization,

I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

2017 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , 2019, pp. 4171–4186

2019

[2] [2]

Gpt-4 technical report,

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

arXiv 2023

[3] [3]

P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,

X. Liu, K. Ji, Y . Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” arXiv preprint arXiv:2110.07602 , 2021

arXiv 2021

[4] [4]

Clip-kd: An empirical study of clip model distillation,

C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, B. Diao, and Y . Xu, “Clip-kd: An empirical study of clip model distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 952–15 962

2024

[5] [5]

Sketch-fusion: a gradient compression method with multi-layer fusion for communication-efficient distributed training,

L. Dai, L. Gong, Z. An, Y . Xu, and B. Diao, “Sketch-fusion: a gradient compression method with multi-layer fusion for communication-efficient distributed training,” Journal of Parallel and Distributed Computing, vol. 185, p. 104811, 2024

work page 2024

[6] [6]

etag: Class- incremental learning via embedding distillation and task-oriented gener- ation,

L. Huang, Y . Zeng, C. Yang, Z. An, B. Diao, and Y . Xu, “etag: Class- incremental learning via embedding distillation and task-oriented gener- ation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 11, 2024, pp. 12 591–12 599

work page 2024

[7] [7]

Continual learning in the frequency domain,

R. Liu, B. Diao, L. Huang, Z. An, Z. An, and Y . Xu, “Continual learning in the frequency domain,” Advances in Neural Information Processing Systems, vol. 37, pp. 85 389–85 411, 2024

work page 2024

[8] [8]

Brain-inspired fast-and slow-update prompt tuning for few-shot class- incremental learning,

H. Ran, X. Gao, L. Li, W. Li, S. Tian, G. Wang, H. Shi, and X. Ning, “Brain-inspired fast-and slow-update prompt tuning for few-shot class- incremental learning,” IEEE Transactions on Neural Networks and Learning Systems, 2024

work page 2024