{"id":"84ebb020-1d4d-491f-953d-f8e830e5b2ba","arxiv_id":"2505.06272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Gradient-based sensitivity scoring automatically allocates LoRA-MoE expert budget across parameter blocks; the module-separated variant matches or slightly beats prior methods with fewer trainable parameters.","lead":"This paper proposes LoRA-SMoE, a way to decide where to place LoRA experts in a mixture-of-experts fine-tuning model by measuring how sensitive each parameter block is to the task. It aims to reduce the number of trainable parameters while preserving or slightly improving accuracy on common reasoning benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed 0.3–0.4 pt accuracy gains are not statistically supported (no seeds/error bars) and key hyperparameters were chosen on the same eight test benchmarks; the central empirical claim is currently unverified.","rationale":"The reader's weakest assumption focuses on the validity of gradient sensitivity as a proxy for LoRA adapter utility; this is a real concern and is supported by the paper's own LoRA-SMoE-U result. However, the more load-bearing issue is that the empirical comparison itself lacks the statistical grounding needed to support any superiority claim: single runs, no error bars, and selection of the 60% threshold and the S variant on the exact eight benchmarks used for the headline numbers. Even if the sensitivity proxy were perfectly valid, the 0.3–0.4 point differences could easily be within run-to-run noise, and the reported numbers cannot be distinguished from selection artifacts. The extra parameters versus MoLA further confound the comparison. A conditional acceptance would require the authors to provide multi-seed results and a separated evaluation; without those, the central claim is not yet verified, so the verdict should shift from CONDITIONAL to UNVERDICTED.","tokens_in":9109,"tokens_out":6012,"duration_ms":59824,"concrete_test":"Run LoRA-SMoE-S(60%), LoRA-SMoE-U(60%), HydraLoRA, and MoLA-▽ with at least 5 random seeds on the eight benchmarks; report mean and standard deviation per task and a paired bootstrap over the eight-task average. Then pre-register the 60% budget and the S variant before evaluating the same pipeline on a held-out set (e.g., a subset of MMLU or eight other QA tasks) to eliminate test-set selection. If the mean advantage of S over HydraLoRA and MoLA is not consistently replicated and outside overlap, or if the held-out advantage disappears, the claimed superiority is not established.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central empirical claim (Table II, Section IV-D) rests on average accuracy differences of only 0.3 points over HydraLoRA (83.2 vs 82.9) and 0.4 points over MoLA-▽ (83.2 vs 82.8). No seed variance, confidence intervals, or significance tests are reported. Because these eight benchmarks are the same ones used in Section IV-C to choose the 60% budget and in Section IV-D to select the S variant over U and I, the reported numbers are selection outcomes, not predictive evaluations. The claim of 'fewer trainable parameters' is also not correct against MoLA (1.60% vs 1.38% in Table II), so the comparison with MoLA is confounded by parameter budget. Furthermore, the paper's own LoRA-SMoE-U result (82.55%) underperforms HydraLoRA, showing that the sensitivity ranking alone does not improve performance and that the success of LoRA-SMoE-S depends on the additional, manually imposed separation of attention/MLP budgets. Without a separated evaluation on unseen tasks or multiple random seeds, the central claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoRA-SMoE, a method for allocating per-module expert budgets in LoRA-based MoE fine-tuning. The allocation is driven by a sensitivity score defined as the sum of squared gradients computed on small sampled subsets of each task's training data. Three variants are introduced: unified selection (LoRA-SMoE-U), separate selection for self-attention and MLP modules (LoRA-SMoE-S), and per-matrix selection (LoRA-SMoE-I). On Qwen2.5-3B-Instruct across eight commonsense reasoning benchmarks, the authors report that LoRA-SMoE-S with a 60% retention budget achieves an average accuracy of 83.2%, compared with 82.9% for HydraLoRA and 82.8% for MoLA, while also studying the stability of the sensitivity estimates under different sample sizes and the consistency of expert selection across tasks.","tokens_in":9303,"tokens_out":4564,"duration_ms":44568,"significance":"If the central empirical claim were verified, this would be a useful contribution: a cheap, gradient-based sensitivity measure could replace manual expert-allocation heuristics in LoRA-MoE fine-tuning, and the paper's sensitivity-stability analysis (Table III) addresses an important practical question about sample sizes. The code link and the experiments on a 3B model also provide a concrete basis for reproduction. However, the reported gains over HydraLoRA and MoLA are small, are computed on the same benchmarks used to choose the 60% threshold and the S variant, and are presented without any variance or significance estimates. In addition, the claim of reducing trainable parameters relative to MoLA is not supported by the paper's own Table II. The core idea is plausible and the limitations are honestly stated, but the evidence as presented does not yet establish the advertised advantage.","major_comments":[{"comment":"The 60% redundancy level is selected by comparing LoRA-SMoE-S(20%), LoRA-SMoE-S(40%), LoRA-SMoE-S(60%), and LoRA-SMoE-S(80%) on the same eight test benchmarks that are later used for the final comparison in Table II. Likewise, the S variant is chosen over U and I in Section IV-D using the same test-table averages. This means the reported 83.2% average is a selection outcome, not a predictive evaluation on an independent benchmark suite. Please provide an evaluation on held-out tasks, or use a separate validation split for model selection and report the test-set results only once, so the reported numbers are not outcomes of choosing hyperparameters on the test set.","section":"Section IV-C and Table II"},{"comment":"No random seeds, error bars, or significance tests are reported anywhere in the paper. The headline differences are 83.2% versus 82.9% for HydraLoRA and 83.2% versus 82.8% for MoLA, and per-task differences are often 0.1 to 0.3 points. On eight- or ten-thousand-example commonsense benchmarks, these differences are well within the range of seed-to-seed variation. The paper should report multiple seeds with mean and standard deviation for at least the main comparison, and ideally a paired significance test, before the claimed improvement can be evaluated.","section":"Table II"},{"comment":"The abstract and conclusion state that the proposed method reduces the number of trainable parameters, but Table II lists Tuned/Total as 1.38% for MoLA-▽(8642) and 1.60% for LoRA-SMoE-S(60%). LoRA-SMoE-S therefore trains more parameters than MoLA, not fewer. The sentence in Section IV-D claiming 'superior performance compared to MoLA-▽ under the same parameters' is also inconsistent with this table. Please correct the parameter-efficiency claim and provide a parameter-matched comparison, or state explicitly that the comparison against MoLA uses a larger parameter budget.","section":"Section IV-D, Table II"},{"comment":"LoRA-SMoE-U(60%), the pure sensitivity-ranked allocation, achieves 82.55%, below HydraLoRA's 82.9%, and the paper itself states that 'allocating experts based only on parameter sensitivity is insufficient.' The success of LoRA-SMoE-S therefore depends on the manually imposed separate selection of attention and MLP budgets, not on the sensitivity criterion alone. The paper should justify this module-level constraint as part of the method, show that it is not itself a hand-crafted heuristic, and assess how sensitive the result is to the split between the two modules.","section":"Section IV-D, LoRA-SMoE-U"}],"minor_comments":[{"comment":"The sentence 'we conducted experiments on a dataset of size 15k' is ambiguous: it is not clear whether the eight datasets were each subsampled to 15,000 examples, whether a single 15,000-example mixture was used, or whether '15k' refers to something else. Please specify the exact data used for the redundancy exploration.","section":"Section IV-C"},{"comment":"The method name is written inconsistently as 'MOLA' in the introduction and as 'MoLA' in the experimental sections and reference [12]. Please use a single spelling throughout.","section":"Introduction and Reference [12]"},{"comment":"In Equation (3), 'sof tmax' should be 'softmax'. In addition, the routing layer R and the input x are not defined precisely; please state the dimensions of R and whether x is the layer input, the hidden state, or a concatenation of both.","section":"Equation (3)"},{"comment":"The paper states in Section II-C that 'MLP layers exhibit higher parameter sensitivity values compared to attention layers,' while Section I says that 'parameters in the middle layers of MLPs appear relatively insensitive.' These statements are not contradictory if mean sensitivity is meant in the first and per-layer distribution is meant in the second, but the distinction should be stated explicitly to avoid confusing readers about the actual sensitivity pattern.","section":"Section II-C and Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible idea and an honest limitations section, but the central empirical claim is currently unverified because the key hyperparameters were selected on the test benchmarks and no variance information is provided. The parameter-efficiency claim relative to MoLA is factually wrong in the current tables. These issues are fixable with additional experiments and a rewriting of the claims, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know about this paper is that the idea is better than the evidence. LoRA-SMoE allocates LoRA-MoE experts by gradient sensitivity, with separate budgets for attention and MLP blocks. On eight commonsense benchmarks on Qwen2.5-3B, the S variant averages 83.2%, slightly above HydraLoRA (82.9%) and MoLA (82.8%), while using fewer parameters than HydraLoRA. The cheap sensitivity measure looks stable: Table III shows that expert selections made with 36 samples are over 85% consistent with the 288-sample selections for most datasets. The finding that unified sensitivity selection (LoRA-SMoE-U) underperforms while the module-separated version (S) wins is interesting, and suggests that sensitivity alone is not the whole story.\n\nThe soft spots are real and load-bearing. There are no seeds or error bars, and the differences between methods are around 0.3 points. More importantly, the 60% threshold and the choice of the S variant come from comparing test-set averages in the same Table II used for the final claims, so the reported numbers are selected, not predicted. The 'fewer trainable parameters' claim also misreads the table: LoRA-SMoE-S uses 1.60% trainable parameters versus MoLA's 1.38%, so it is not fewer than MoLA. On top of that, the U variant falls below HydraLoRA, which means the success of S depends on the manually imposed separation of attention/MLP budgets rather than on the sensitivity principle alone. Code is promised but not available, so independent verification is not yet possible.\n\nWhat is genuinely useful here is the sensitivity measurement itself and the consistency analysis across sample sizes. That part is cheap, reproducible, and could be adopted by other LoRA-MoE work. The paper is honest in its limitations: no theoretical derivation, single model scale. But the main empirical claim needs stronger evidence before it can be trusted.\n\nI would send this to peer review, because the idea deserves a serious referee and the experiments can be meaningfully improved: multi-seed runs, fixed hyperparameters chosen on a validation split or a subset of tasks, and a clearer parameter-budget comparison. Right now I would not cite the accuracy gains as a reliable result, but I would cite the sensitivity consistency observation once the code is out and verified.","headline":"Sensitivity-based expert allocation for LoRA-MoE is a genuinely new idea, but the reported accuracy gains are selection outcomes, not verified predictions, and the parameter-count comparison with MoLA is misstated.","tokens_in":9873,"tokens_out":3422,"would_cite":false,"duration_ms":32870,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Allocating LoRA experts to the most gradient-sensitive parameter blocks raises accuracy on eight commonsense benchmarks to 83.2% with only 1.60% of parameters trainable.","keywords":["LoRA","Mixture of Experts","parameter-efficient fine-tuning","parameter sensitivity","gradient-based expert allocation","multi-task learning","commonsense reasoning"],"falsifier":"Compute sensitivity on a held-out set of tasks, then allocate experts to the least sensitive 60% of blocks under the same separate attention/MLP budgets; if that allocation matches or beats the most-sensitive allocation, the sensitivity signal is not what drives the gain. Alternatively, if a unified allocation beats the separate-budget method on a different base model or task family, the attention/MLP split is the operative ingredient rather than the gradient signal.","tokens_in":8861,"feed_emoji":"🎯","tokens_out":9260,"duration_ms":78564,"temperature":0.7,"pith_summary":"This paper tries to show that the per-block expert counts in a LoRA-MoE fine-tune can be chosen automatically from gradient information rather than from hand-set heuristics such as putting more experts in higher layers. The proposed LoRA-SMoE method samples a small batch of task data, computes each parameter block's sensitivity as the cumulative sum of squared gradients, and spends a fixed expert budget on the most sensitive blocks. With the budget split separately between self-attention and MLP modules and set at the top 60% of sensitive blocks, it reports higher average accuracy than fixed-allocation baselines while training fewer parameters. If correct, this means expert budgets can be tuned cheaply per task, which matters for resource-constrained multi-task fine-tuning.","feed_headline":"Gradient-based expert allocation out-scores fixed LoRA-MoE","feed_subtitle":"Top 60% most gradient-sensitive blocks hit 83.2% accuracy with 1.60% trainable parameters.","key_machinery":"The load-bearing object is a per-block sensitivity score: for each matrix block (Q, K, V, O in attention; up, down, gate in MLP), the score is the sum of squared gradients $s_n = \\sum g_n^2$ computed over a small sample of task data. Expert counts are then assigned to the top 60% of blocks under a fixed total budget, either across all blocks (uniform), separately within attention and MLP (separate), or per matrix (independent). The fine-tuning backbone is an asymmetric LoRA-MoE with a shared low-rank matrix $A$ and task-specific $B_i$ matrices combined by soft routing, $y = W_0 x + \\sum_{i=1}^{E}\\omega_i B_i A x$, so 'experts' are collections of $B$ matrices. Sensitivity is computed one parameter group at a time, keeping peak memory comparable to ordinary LoRA fine-tuning.","core_discovery":"The paper's central claim is that parameter sensitivity, measured as the cumulative sum of squared gradients over backpropagation on a small sample, predicts where LoRA experts should be placed: within a fixed budget, assigning more experts to the most sensitive parameter blocks improves downstream accuracy, provided the budget is allocated separately to self-attention and MLP modules. On eight commonsense reasoning benchmarks with a 36-layer 3B instruction-tuned model, the authors report 83.2% average accuracy for LoRA-SMoE-S at the top 60% sensitivity threshold, with 1.60% trainable parameters, compared with 82.9% for HydraLoRA (2.51%) and 82.8% for MoLA (1.38%). They also find that attention layers are most sensitive in higher layers and that MLP middle layers are least sensitive, which refines rather than overturns the existing rule that higher layers need more LoRA experts. A unified allocation that simply takes the most sensitive blocks across all modules underperforms, so the attention/MLP separation is part of the claim rather than a detail.","pith_inferences":["If gradient sensitivity is as stable across sample sizes as the paper's consistency table suggests, the same per-block allocation could be computed once and reused across related tasks, though the paper only verifies one 3-billion-parameter model.","The reported insensitivity of MLP middle layers points to a more aggressive recipe the authors did not test: freeze or share a single expert across those blocks and move the saved parameter budget into attention blocks.","The sensitivity metric is model-agnostic, so the allocation recipe could plausibly extend to other parameter-efficient mechanisms such as adapters or prompt vectors, but the paper does not test that transfer."],"forward_implications":["Expert allocation no longer needs hand-tuned layer rules: a sensitivity scan on 36 to 288 samples, taking roughly 10 to 66 seconds in the paper's measurements, fixes per-block expert counts before training begins.","Dropping the least sensitive 40% of parameter blocks from the expert budget cuts trainable parameters from 2.51% (HydraLoRA) to 1.60% while the average accuracy rises from 82.9% to 83.2%, and the 60% threshold outperforms both 20% and 80% thresholds.","Attention and MLP modules need separate expert budgets, because a unified sensitivity ranking overallocates experts to MLP blocks and performs worse than the separate-budget variant.","The sensitivity scan adds only forward and backward passes on a small sample, so the allocation procedure is inexpensive enough to run per task before fine-tuning."],"supporting_citations":[{"why":"Defines low-rank adaptation (LoRA), the adapter mechanism that each expert is built from.","marker":"[9]"},{"why":"Supplies the sensitivity definition (cumulative squared gradients) and the idea of tuning only sensitive parameters.","marker":"[14]"},{"why":"Provides the asymmetric shared-A/multiple-B HydraLoRA architecture used as the fine-tuning backbone and as the main baseline.","marker":"[15]"},{"why":"Provides the MoLA baseline and the higher-layers-need-more-experts allocation rule that this paper refines.","marker":"[12]"},{"why":"Identifies the 36-layer 3-billion-parameter instruction-tuned model used in all experiments.","marker":"[23]"}],"fun_headline_variants":["Sensitivity-driven expert placement outperforms fixed LoRA-MoE","Gradient sensitivity selects LoRA experts, boosting accuracy","Sensitivity-aware expert allocation cuts trainable parameters","LoRA-MoE: allocate experts by parameter sensitivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a block's summed squared gradient on a small data sample tells where LoRA experts will help most, a premise the paper's own unified-allocation result shows is incomplete without the extra attention/MLP budget split.","fun_headline_variants_meta":{"raw":{"variants":["Sensitivity-driven expert placement outperforms fixed LoRA-MoE","Gradient sensitivity selects LoRA experts, boosting accuracy","Sensitivity-aware expert allocation cuts trainable parameters","LoRA-MoE: allocate experts by parameter sensitivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2783,"prompt_tokens":1058,"completion_tokens":1725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":1659}},"tokens_in":674,"tokens_out":1725,"duration_ms":11497,"temperature":1.0,"reasoning_tokens":1659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:49:35.492999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute sensitivity on a held-out set of tasks, then allocate experts to the least sensitive 60% of blocks under the same separate attention/MLP budgets; if that allocation matches or beats the most-sensitive allocation, the sensitivity signal is not what drives the gain. Alternatively, if a unified allocation beats the separate-budget method on a different base model or task family, the attention/MLP split is the operative ingredient rather than the gradient signal.","supporting_citations":[{"cited_title":"Hydralora: An asym- metric lora architecture for efficient fine-tuning,","cited_arxiv_id":null,"evidence_quote":"Provides the asymmetric shared-A/multiple-B HydraLoRA architecture used as the fine-tuning backbone and as the main baseline."}],"review_version":1}