{"id":"fc24064d-205e-4240-b162-1beb24f55f9b","arxiv_id":"2412.08946","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new LoRA-MoE variant that shares the up-projection matrix across experts and adds dropout on it improves multi-task fine-tuning accuracy on six commonsense reasoning datasets.","lead":"This paper proposes a way to fine-tune large language models on many tasks at once using a mixture of small adapter modules that share one of their two low-rank matrices. The method reports better multi-task accuracy than existing adapter-mixture baselines on six reasoning benchmarks, while using fewer trainable parameters than one leading baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy advantage may be an artifact of hyperparameter selection on the test sets, not evidence for the sharing/dropout mechanism.","rationale":"The reader's weakest_assumption concerns whether matrix A truly captures general features. While relevant to the paper's mechanistic interpretation, the empirical headline—highest average accuracy—does not strictly depend on A being 'general' in a semantic sense; even a shared low-rank input projection could yield strong performance through the mixture of B matrices. The more decisive threat to the central claim is the evaluation protocol: §5.2 shows hyperparameters selected on test sets, creating a bias that could fully explain the 1.56-point advantage over MoLA. This is a concrete, correctable flaw and should be a condition for acceptance. I therefore keep the reader's CONDITIONAL verdict. I partially agree with the reader because the test-set tuning was listed as a secondary issue there, but the primary weakest_assumption (general vs. specific factorization) is not, in my view, the most load-bearing element.","tokens_in":14600,"tokens_out":5040,"duration_ms":48739,"concrete_test":"Re-run the mixture-setting experiments with all methods (MoSLD and every baseline) using only validation-based hyperparameter selection: hold out 10% of each training set as validation, tune dropout ratio and expert allocation on validation, then report test-set accuracy once. If MoSLD's average advantage over MoLA under this fair protocol falls below the reported 1.56 points or fails significance testing, the headline claim is not supported. Alternatively, provide a table with per-run results, standard deviations, and the exact t-test procedure for the current numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §5.2, the dropout ratio and per-layer expert allocation for MoSLD are selected by evaluating on the six test sets under the mixture setting (Figure 5 and Figure 7). The chosen values (dropout 0.1, allocation (2,4,6,8)) are then used in Table 1 and Table 5 to produce MoSLD's reported 71.56% average. Baselines such as MoLA, LoRA, MoLoRA, and SiRA are reported with default configurations and were not given the same test-set-based hyperparameter search. This asymmetrical protocol inflates MoSLD's performance because the model is effectively optimized on the evaluation data while competitors are not. The claimed 1.56-point gain over MoLA could therefore be an artifact of test-set tuning rather than evidence for the shared-A mechanism. The concern is compounded by the unsupported t-test claim (p<0.01): no variance or test details are reported, and a rough paired t-test over the six dataset means yields p≈0.02. This methodological issue must be resolved before the central comparative claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MoSLD, a parameter-sharing variant of mixture-of-LoRA for multi-task fine-tuning of large language models. In each Transformer layer, a single low-rank matrix A is shared by all experts while each expert retains its own B matrix, and dropout is applied to A during training. The authors evaluate on six commonsense reasoning datasets under single-task and mixed-task training, reporting a higher average accuracy than LoRA and several MoE-LoRA baselines, with 1.389B trainable parameters versus 2.228B for MoLA. The paper also includes scaling experiments across model sizes, an out-of-domain evaluation, and an ablation of the dropout component.","tokens_in":14775,"tokens_out":4888,"duration_ms":46223,"significance":"If the comparative results are valid, the proposal is a simple and parameter-efficient modification of LoRA-MoE that is largely orthogonal to existing methods. The inclusion of scaling results to 33B, an out-of-domain test, and an ablation isolating dropout are strengths. However, the empirical support for the headline claim is weakened by test-set hyperparameter selection and an unsupported significance statement; the conceptual claim that A carries general features is not directly evidenced. These issues are addressable and do not invalidate the underlying idea.","major_comments":[{"comment":"The dropout ratio (0.1) and expert-allocation scheme (2,4,6,8) used in Table 1 are selected by evaluating MoSLD on the six test sets under the mixture setting (Figures 5 and 7), and rank r=8 is selected in Appendix B (Table 6). This is test-set selection, not validation-based selection, and no comparable tuning is reported for the baselines. The reported 1.56-point gain over MoLA may therefore reflect the hyperparameter search rather than the proposed mechanism. Please report model selection on a held-out validation split, or tune all baselines under the same protocol, and clearly state the selection procedure in the main text.","section":"§5.2 and Table 1"},{"comment":"The caption states '(p < 0.01 under t-test)' but provides no variance information, per-run results, or test details. With only six paired dataset means, a two-sided paired t-test of the reported mean differences does not reach p < 0.01 (a rough calculation gives p ≈ 0.02), and the effective sample size is small. Please report the per-run accuracies, the exact test performed, the test statistic, and the degrees of freedom, or remove the significance claim.","section":"Table 1"},{"comment":"The design rationale is that A is a 'general-feature matrix' and B is a 'specific-feature matrix,' but the paper states in Limitation (3) that visualizing this phenomenon is future work. The ablation MoSL vs. MoSLD isolates the dropout contribution only; the contribution of sharing A is not directly ablated because no variant with separate A per expert under identical hyperparameters is compared. The comparison to MoLA is a baseline comparison, not a controlled ablation, since expert allocation and other settings differ. Please add a controlled ablation or rephrase the causal attribution.","section":"Section 3.1 and Limitations (3)"}],"minor_comments":[{"comment":"In Eq. (3), the expert matrices are written as A_{k,l} and B_{k,l}, but A is shared so it should be A_l; this makes the sharing mechanism explicit and avoids confusion with Eq. (2).","section":"Eq. (3)"},{"comment":"The load-balancing loss in Eq. (4) is not a well-formed equation; the expression after 'pk =' needs to be written out cleanly, and the notation c_k and s_k should be defined consistently with the text.","section":"Eq. (4)"},{"comment":"The column 'LoRA number' is ambiguous: please clarify whether it counts matrices per layer for both Q and V projections and how the average expert number 5 is derived from the (2,4,6,8) allocation.","section":"Table 5"},{"comment":"The row 'MoSLD (matrix B)' should specify whether dropout is applied to B instead of A or whether the sharing mechanism is applied to B; the text should define the variant precisely.","section":"Table 2"},{"comment":"The statement that baselines 'trail behind LoRA by 1.20%, 1.06%, 1.98%, and 1.00%' should refer to percentage points, not percentages, to avoid quantitative ambiguity.","section":"Section 4.4"},{"comment":"Figure 1's axes and the meaning of 'Score (mixture)-Score (single)' are not explained in the text; please add a sentence describing the figure.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is worth considering. The main risk is the test-set hyperparameter selection, which is common but needs to be disclosed and corrected. The missing code/reproducibility details and the rough notation also need attention. I would not reject on the current evidence, but the revision must include validation-based selection and full statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea—share the A matrix across LoRA experts, keep per-expert B matrices, and add dropout on A—is simple, plausible, and not present in the cited MoLA/MixLoRA line. The efficiency numbers are real: 1.389B trainable parameters versus MoLA's 2.228B, with a higher reported average. That is worth taking seriously.\n\nWhat the paper does well: the architecture is clearly described, the scaling study across 7B/13B/33B gives useful breadth, and the additional mixtures with GSM8K and OpenOrca are the right kind of robustness checks. The dropout-on-A motivation is reasonable and the paper honestly flags that the general/specific decomposition is not yet directly observed.\n\nThe soft spots are real and load-bearing for the central claim. First, Section 5.2 shows dropout ratio and per-layer expert allocation being selected by evaluating on the six test sets under the mixture setting. The baselines, as far as I can tell, are not given the same test-set search. That asymmetric protocol means the 71.56% average may reflect selection bias rather than the sharing/dropout mechanism. Second, the p<0.01 t-test is unsupported: no standard deviations or per-run numbers are given, and a rough paired t-test on the six dataset means gives p≈0.02, not <0.01. Third, the out-of-domain evidence is a figure with no numeric table, so it is hard to quantify.\n\nThese are fixable. Release code and per-run results, move hyperparameter selection to a validation split or at least show the main conclusion holds under defended hyperparameters, and report error bars. Until then, the 71.56% should be treated as an upper bound, not a verified effect.\n\nWorth a serious referee: yes, but as a major-revision candidate, not an accept. I would not cite it in my own work until the protocol is cleaned up, but I would be curious to see the revision.","headline":"Sharing the LoRA A-matrix across experts is a neat, plausible trick, but the headline accuracy gain is not yet trustworthy because the model's hyperparameters appear to be tuned on the test sets while baselines were not, and the claimed t-test significance lacks supporting variance.","tokens_in":15349,"tokens_out":2181,"would_cite":false,"duration_ms":23550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoSLD proposes sharing the upper LoRA matrix across all experts and applying dropout to it, and reports the highest average multi-task accuracy among the compared LoRA-mixture methods while cutting trainable parameters.","keywords":["parameter-efficient fine-tuning","LoRA","mixture of experts","multi-task learning","parameter sharing","dropout","large language models","LLaMA"],"falsifier":"Train MoSLD and an unshared mixture-of-experts LoRA baseline on two tasks that require opposite input-to-output mappings, such as the same prompt templates with reversed answer labels; if per-task accuracy on the reversed task falls below the unshared baseline, the shared $A$ cannot represent both task-specific transformations and the general-feature premise fails.","tokens_in":14357,"feed_emoji":"🧠","tokens_out":14258,"duration_ms":120515,"temperature":0.7,"pith_summary":"MoSLD is a parameter-efficient variant of mixture-of-experts fine-tuning for large language models. The paper's central claim is that in multi-task training, the upper projection matrix $A$ of a LoRA pair should be shared by all experts in a layer, while each expert keeps its own lower projection matrix $B$, and that applying dropout to $A$ improves optimization. On six commonsense reasoning datasets with LLaMA2-7B, MoSLD reports an average mixture accuracy of 71.56%, against 69.15% for plain LoRA and 70.00% for MoLA, while training 1.389 billion parameters compared with MoLA's 2.228 billion. The mechanism matters because it offers a simple way to reduce the parameter cost of LoRA-based MoE while mitigating cross-task interference and knowledge forgetting.","feed_headline":"Sharing one LoRA matrix lifts accuracy, cuts parameters","feed_subtitle":"MoSLD reports 71.56% average accuracy on six reasoning datasets while using 1.389B trainable parameters.","key_machinery":"The central object is the per-layer shared matrix $A_l$, called the general-feature matrix of the LoRA decomposition, combined with the router-weighted sum in Eq. (3): $h_l = W_0x + \\frac{\\alpha}{r}\\sum_{k=1}^K S_k^l(x)A_{k,l}B_{k,l}x$, where all $A_{k,l}$ are equal across experts. This sharing collapses $N_l$ low-rank matrices into one per layer, and the dropout update in Eq. (2), $A'_l = \\mathrm{Mask}\\odot A_l$ with rescaling by $1/(1-p)$, is what the paper credits with balancing gradient updates and preventing overfitting in the shared matrix. A load-balancing loss is added on top of the router to keep token assignments spread among experts.","core_discovery":"The paper argues that the two low-rank matrices in LoRA are naturally decoupled: matrix $A$ acts as a general-feature extractor shared across tasks, and matrix $B$ carries task-specific features. MoSLD operationalizes this by setting $A_{1,l} = A_{2,l} = \\cdots = A_{N_l,l}$ for every layer $l$, keeping one shared matrix per layer, and training a router to select the top-$K$ experts for each token, with the expert output computed as $h_l = W_0x + \\frac{\\alpha}{r}\\sum_{k=1}^{K} S_k^l(x)A_{k,l}B_{k,l}x$. A Bernoulli dropout mask is applied to the shared $A_l$ during updates and rescaled by $1/(1-p)$, which the paper credits with balancing updates between the shared and expert-specific matrices and preventing overfitting. The reported consequence is that this design outperforms plain LoRA, MoLoRA, SiRA, MoLA, and MixLoRA in the mixture setting on the six datasets used, and improves out-of-domain generalization on MMLU.","pith_inferences":["We infer that sharing $A$ acts as an implicit regularizer that forces experts to differentiate only through $B$; a direct test would compare the cosine similarity of expert $B$ matrices under MoSLD versus an unshared mixture-of-experts LoRA baseline.","The factor-splitting premise is asserted rather than measured, since the paper lists visualizing the general/specific feature split as future work; if a task mixture requires $A$ to carry task-specific information, the shared matrix becomes a capacity bottleneck, and we would expect MoSLD to underperform unshared mixture-of-experts LoRA on such mixtures.","We would expect the dropout-on-$A$ trick to transfer to other shared-parameter fine-tuning designs, such as shared adapters or shared prefix parameters, because the update imbalance it addresses is generic to any frequently updated shared module.","The reported out-of-domain gains on MMLU suggest the shared matrix stores transferable features; a testable extension is to freeze $A$ after multi-task training and fine-tune only new $B$ matrices for unseen tasks, which should be cheaper and possibly more stable than training full LoRA from scratch."],"forward_implications":["If MoSLD is correct, LoRA-based mixture-of-experts becomes substantially cheaper: sharing one matrix per layer cuts trainable parameters from MoLA's 2.228 billion to 1.389 billion on LLaMA2-7B while raising average mixture accuracy from 70.00% to 71.56%.","The sharing mechanism is orthogonal to layer-wise expert allocation; the paper's Figure 7 indicates that putting more experts in higher layers still helps, so the two ideas can be combined without conflict.","Dropout on the shared matrix provides a practical regularizer: a dropout probability of 0.1 improves both single and mixture results over the no-dropout variant MoSL, and larger dropout rates degrade performance.","The method scales with backbone size: average mixture accuracy rises from 71.56% on LLaMA2-7B to 79.96% on LLaMA-33B, with the single-to-mixture gain growing from 1.36 points to 1.91 points.","For task mixtures with little commonality, such as adding GSM8K math reasoning to commonsense QA, MoSLD turns the usual mixture penalty into a small gain, suggesting the shared matrix does not force harmful coupling."],"supporting_citations":[{"why":"Defines LoRA's low-rank decomposition $W_0+\\Delta W=W_0+BA$, the substrate that MoSLD's sharing mechanism modifies.","marker":"Hu et al., 2022"},{"why":"Supplies the sparsely-gated mixture-of-experts routing mechanism, including top-K expert selection and gating, that MoSLD adapts to LoRA experts.","marker":"Shazeer et al., 2017"},{"why":"MoLA is the strongest mixture-of-LoRA baseline in the comparison and the source of the layer-wise expert allocation idea MoSLD follows.","marker":"Gao et al., 2024"},{"why":"MixLoRA is the concurrent multi-task LoRA-MoE baseline that MoSLD compares against to argue that shared matrices are more parameter-efficient.","marker":"Li et al., 2024"},{"why":"Provides the unified view of LoRA and dropout that motivates applying dropout to the general-feature matrix $A$.","marker":"Wang et al., 2024"},{"why":"Supplies evidence of parameter redundancy among experts, which the paper cites to justify sharing one matrix across experts.","marker":"Fedus et al., 2022b"},{"why":"MoLoRA is a parameter-efficient MoE baseline whose single-task results are used to show plain MoE-LoRA does not help in the single setting.","marker":"Zadouri et al., 2024"},{"why":"Supplies the LLaMA2-7B backbone on which the main results and scaling experiments are run.","marker":"Touvron et al., 2023b"}],"fun_headline_variants":["Shared LoRA matrix boosts multi-task accuracy","MoSLD: one shared matrix, huge parameter savings","Dropout on shared LoRA balances updates, cuts params","Mix-of-shared LoRAs: 71.56% accuracy, few trainable params","Sharing LoRA's A matrix lifts multi-task generalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that in a LoRA pair the upper projection matrix $A$ genuinely encodes general cross-task features while the lower projection matrix $B$ encodes task-specific features, so a single shared $A$ per layer can serve all experts without becoming a bottleneck; the paper itself lists visualizing this distinction as future work.","fun_headline_variants_meta":{"raw":{"variants":["Shared LoRA matrix boosts multi-task accuracy","MoSLD: one shared matrix, huge parameter savings","Dropout on shared LoRA balances updates, cuts params","Mix-of-shared LoRAs: 71.56% accuracy, few trainable params","Sharing LoRA's A matrix lifts multi-task generalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1286,"prompt_tokens":969,"completion_tokens":317,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":585,"tokens_out":317,"duration_ms":3939,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:21:44.017270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MoSLD and an unshared mixture-of-experts LoRA baseline on two tasks that require opposite input-to-output mappings, such as the same prompt templates with reversed answer labels; if per-task accuracy on the reversed task falls below the unshared baseline, the shared $A$ cannot represent both task-specific transformations and the general-feature premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MoLoRA is a parameter-efficient MoE baseline whose single-task results are used to show plain MoE-LoRA does not help in the single setting."}],"review_version":1}