REVIEW 4 major objections 5 minor 10 references
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Composing frozen LoRA adapters works better with hard per-token selection than with soft weighted blending, and needs far fewer trainable parameters.
desk verdict A clean construction and honest evaluation, but the unit-scale mechanism is confounded with expert freezing and the token-level advantage only shows on a contrived mixed-domain test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is a token-level top-1 router trained with a straight-through estimator: the forward pass selects exactly one expert, applies its LoRA update at weight 1, and the backward pass uses the softmax probabilities as a surrogate gradient. The paper also formalizes the scale-mismatch it is correcting: a soft mixture computes W = W_base + sum_i ω_i A_i B_i, so each frozen adapter's update is multiplied by ω_i < 1, which the authors argue breaks the unit-scale assumption the LoRA training used.
What would settle it
Run the same frozen expert on the same task with its update scaled by 1.0 versus, say, 0.5, keeping everything else identical; if accuracy does not drop, the scale mismatch is not the reason soft routing loses. Alternatively, train a soft-routing mixture with frozen experts and no preservation loss and compare directly with hard routing.
Extended reading notes
Core claim
The central claim is that the integration stage for frozen LoRA experts should be a pure routing problem: keeping every expert fixed and learning only when to apply which one, with the chosen expert applied at unit scale. The paper identifies a mechanism for why soft mixtures lose: routing weights below one rescale each LoRA update and change the effective function the adapter performs. Hard top-1 routing through a straight-through estimator restores the original unit-scale application and still allows end-to-end training of the router. Empirically, on instruction-tuned models from 1B to 8B parameters across five distinct tasks, the hard-routed mixture achieves average accuracy at least as h
Load-bearing premise
The load-bearing premise is that applying a frozen LoRA update with a weight less than one degrades the adapter's trained behavior; the paper never isolates this effect from the simultaneous choice of not retraining the experts.
Editorial extensions
If this is right
- Frozen LoRA experts can be composed into one model without access to the original training data and without retraining the experts, as long as a small labeled set for routing is available.
- The number of trainable parameters in the integration stage does not grow with the number of experts, since only a shared router and small attention LoRA are trained.
- Normalized soft top-2 mixtures often behave like near top-1 selection, suggesting hard routing captures the same behavior more directly and with sparser computation.
- Token-level routing handles inputs that mix multiple domains better than prompt-level routing, which must commit to a single expert for the whole prompt.
Reading between the lines
- If the scale-mismatch mechanism is the real cause of soft routing's losses, then even a properly selected expert applied at fractional scale should underperform the same expert at unit scale; this is a directly testable prediction the paper does not run.
- The unit-scale principle suggests a natural extension to an abstention route: when no expert is a good match, the model could fall back to the base computation graph, which the authors list as future work but do not evaluate.
- The parameter-efficiency advantage should widen with larger expert pools, since soft-routing baselines that retrain experts incur per-expert costs that hard routing avoids.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hard-Routed MoR-LoRA, a two-stage framework for composing independently trained LoRA adapters. Stage I trains domain-specific reasoning experts with RLVF/GRPO; Stage II freezes all experts and trains only a shared token-level top-1 router with a straight-through estimator plus a small attention LoRA on distilled expert traces. The central claim is that soft MoE-style routing scales frozen LoRA updates by routing probabilities, violating the unit-scale additive update assumed during LoRA training, whereas hard top-1 routing preserves each expert's trained function and yields better or comparable performance with far fewer trainable parameters. The headline results are on LLaMA-3B and 8B with five benchmarks: 72.07 vs 70.81 (3B) and 79.80 vs 78.33 (8B) against LoRAMixer normalized top-2, with roughly 73M vs 606M and 109M vs 1.133B trainable parameters, respectively.
Significance. If the central mechanism were cleanly established, the work would be a useful contribution to modular and parameter-efficient LLM adaptation: it combines RLVF-trained experts, a frozen-expert composition principle, and an STE-based token-level router, and it ships code plus detailed hyperparameters. The manuscript also contains honest and useful control analyses: the prompt-level routing baseline, the mixed-domain and unseen-dataset evaluations, the Stage II data-budget ablation, and Appendix C's proof that normalized top-1 soft routing has zero router gradient are all valuable. However, the significance is currently tempered by the fact that the main mechanistic claim is not isolated from expert retraining, and by the absence of any variance estimates in a reinforcement-learning pipeline.
major comments (4)
- [§4.1, Eq. (8) and Tables 6/7] The central scale-mismatch mechanism is confounded with expert retraining. The LoRAMixer baselines in Tables 6/7 continue training the adapters with an L2 preservation loss, while Hard-Routed MoR-LoRA freezes them. Thus the reported gains (72.07 vs 70.81 at 3B; 79.80 vs 78.33 at 8B) are equally consistent with the alternative explanation that retraining degrades independently trained experts, independent of soft weighting. A soft-routing baseline over the same frozen experts is missing — e.g., a learned soft top-1/top-2 over the frozen FFN LoRAs, or LoRAMixer without the preservation loss. The Appendix E soft-train/hard-inference variant does not disentangle this, since it stays within the hard-routing family. Note also that the prompt-level frozen-expert baseline in Table 1 achieves 72.79/79.87, comparable to or above the proposed method on clean single-domain tasks, suggesting that fre
- [§5.1, Table 5 vs Table 16] The RLVF-versus-SFT expert comparison is not controlled: RLVF uses learning rate 5e-6 and 2 epochs, while SFT uses 1e-6 and 10 epochs. The claim that 'RLVF produces stronger reasoning experts' is therefore confounded with optimization budget and learning-rate schedule. Since Stage I expert quality is load-bearing for the entire two-stage result, please either match hyperparameters or provide a sweep showing the effect is due to the RLVF objective rather than the training configuration.
- [All experimental tables] No seeds, error bars, or confidence intervals are reported anywhere. This is especially problematic for an off-policy GRPO pipeline with temperature 1.1, four sampled generations per prompt, and small Stage II budgets (1000 samples per dataset). The headline margins over LoRAMixer normalized top-2 are ≈1.2–1.5 points, which may be within run-to-run noise. Please report means and standard deviations (or bootstrap intervals) over at least 3–5 seeds for the main tables, including Tables 5–13.
- [§5.4, Fig. 4] The histogram of dominant routing weights under normalized soft top-2 is presented as evidence that soft routing 'collapses' toward single-expert selection, but it is computed for LoRAMixer, where the experts have been retrained under the soft objective. It therefore does not characterize what a soft router over the original frozen experts would do, and it is not causally connected to the performance gap. Moreover, a mean dominant weight of 0.71 leaves 0.29 on a second expert; whether that residual weight is harmful is exactly what the missing frozen-expert soft baseline would test.
minor comments (5)
- [§2, Related Work] There is a garbled sentence: 'There is a rich body of research have Several works, therefore, studiedy how to combine multiple pretrained adapters.' Please rewrite.
- [Appendix F.1 and Table 14] 'SV AMP' should be 'SVAMP' (the dataset name) throughout the appendix and in Table 14.
- [Table 5] The 1B baseline GSM8K value is shown as '03.49' and the RLVF value as '00.05'; the leading zeros are inconsistent with the rest of the table.
- [Figure 1 caption] Typo: 'the routerselects a single expert' should be 'the router selects a single expert'.
- [References] The BoolQ reference title is misspelled as 'BooQ: Exploring the Surprising Difficulty...'.
Circularity Check
No circular derivation: the hard-routing mechanism is definitional but tested empirically; the main limitation is an experimental confound, not a circular step.
full rationale
The paper's central claim is that hard top-1 routing preserves the unit-scale update under which independently trained LoRA experts were trained, and that this preserves expert behavior. This is not derived from its conclusion. The scaling-mismatch argument in Sec. 4.1 (Eq. 8) is a mathematical observation about applying a routing weight omega_i to a LoRA update; it does not assume the empirical result that hard routing outperforms soft routing. The empirical comparison in Tables 6/7 and Appendix A.2 tests that claim against LoRAMixer baselines. A genuine validity concern exists: the LoRAMixer baselines both use soft routing and retrain adapters with a preservation loss, so routing type and expert-retraining status vary together; a frozen-expert soft-routing baseline would isolate the mechanism. But this is an experimental confound (correctness risk), not circularity. The Appendix C derivation that normalized top-1 soft routing has zero router gradient is a standalone mathematical proof, and Appendix E's soft-train/hard-inference comparison is an independent manipulation, not a restatement of the conclusion. Figure 4's observation that normalized top-2 routing assigns a dominant weight of about 0.71 is a post-hoc descriptive statistic, not a fitted quantity presented as a prediction. The paper is benchmarked against external datasets and baselines, and the self-citations in Related Work (e.g., Ostapenko et al. 2023/2024, Su et al. 2024, Wen et al. 2025, which include Zhan Su) are contextual and not load-bearing: no uniqueness theorem or central premise is imported from those papers. No fitted parameter is renamed as a prediction, and no ansatz is smuggled in via citation. Therefore no circular step meeting the required quote-and-reduction standard is present.
Assumptions & free parameters
free parameters (3)
- RLVF reward constants (correct +1.0, incorrect −1.1, think-format +0.2, JSON-on-wrong +0.1) =
+1.0 / −1.1 / +0.2 / +0.1 (Table 15)
- Stage II attention LoRA rank =
128 (default; sweep in Table 9)
- Stage I training budgets for SFT vs RLVF =
SFT: lr 1e−6, 10 epochs; RLVF: lr 5e−6, 2 epochs (Table 16)
assumptions (4)
- ad hoc to paper Soft routing weights p_i change the effective function of a frozen LoRA update (Eq. 8), and this scale mismatch is the operative cause of soft-routing losses.
- standard math The straight-through estimator (Eq. 12) provides a useful training signal for the router.
- domain assumption Four RLVF generations per prompt at temperature 1.1 with an offline importance-ratio objective suffice to train stable domain experts.
- domain assumption A shared linear token-level router can learn expert selection from ~1000 distilled traces per domain and transfers to mixed prompts.
Cite this review
Pith. "Pith review of Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs." pith.science (2026). https://pith.science/paper/RD3KWTOI
@misc{pith2026260631413,
author = {Pith},
title = {Pith review of: Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs},
year = {2026},
howpublished = {\url{https://pith.science/paper/RD3KWTOI}},
note = {Machine review of arXiv:2606.31413}
}
read the original abstract
Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose \textbf{Hard-Routed MoR-LoRA}, a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Hard STEis our default method: the forward pass selects the top-1 expert deterministically, while gradients are propagated through the soft router probabilities
-
[2]
We an- neal the temperature from τ= 1.5 to τ= 0.1 exponentially and keep the final temperature fixed for the last10%of training
Gumbel-Softmax(Jang et al., 2016) uses hard Gumbel-Softmax sampling with a contin- uous relaxation in the backward pass. We an- neal the temperature from τ= 1.5 to τ= 0.1 exponentially and keep the final temperature fixed for the last10%of training
2016
-
[3]
DeepSeek-R1: Incentivizing Reasoning Capa- bility in Large Language Models via Reinforcement Learning.arXiv preprint arXiv:2501.12948. Neil Houlsby, Sebastian Jastrzebski, Andrzej Brooks, Rosanne de Vries, Andrea Guedj, and Grégory Ne- matzadeh. 2019. Parameter-Efficient Transfer Learn- ing for NLP. InInternational Conference on Ma- chine Learning (ICML)....
arXiv 2019
-
[6]
Xun Wu, Shaohan Huang, and Furu Wei
MARVEL: Modular Abstention for Reliable and Versatile Expert LLMs. Xun Wu, Shaohan Huang, and Furu Wei. 2024. Mixture of LoRA Experts.arXiv preprint arXiv:2404.13628. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, and 1 oth- ers. 2025. Qwen2.5 Technical Report.Preprint, arXiv:241...
arXiv 2024
-
[7]
SiRA: Sparse Mixture of Low-Rank Adapta- tion.arXiv preprint arXiv:2311.09179. 12 A Detailed Experimental Results This section provides thefull per-taskresults corre- sponding to the averaged metrics reported in the pa- per. All experiments follow the two-stage pipeline: training standalone experts (Stage I) and integrat- ing them through routing (Stage I...
arXiv 2025
-
[10]
Soft-train/hard-inferencetrains the model with a probability-scaled top-1 expert output, but removes the softmax scaling at inference time and applies the selected expert with unit weight. This variant tests whether unit-scale routing is sufficient only at inference time, or whether the same behavior is also necessary during training. All methods use the ...
arXiv 2021
-
[2013]
arXiv preprint arXiv:1308.3432
Estimating or Propagating Gradients through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language Models are Few-shot Learners.Advances in neural in...
arXiv 2020
-
[2021]
Are NLP Models really able to Solve Sim- ple Math Word Problems? InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 2080–2094. Zhen Shao and 1 others. 2024. DeepSeekMath: Push- ing the Limits of Mathematical Reasoning in Open Language Models.arXiv preprint...
arXiv 2021
Show all 10 references
-
[2023]
InNeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing
A Case Study of Instruction Tuning With Mix- ture of Parameter-efficient Experts. InNeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing. Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Lau- rent Charlin, Nicolas Le Roux, Matheus Pereira, Lu- cas Caccia, an...
2023 arXiv
-
[2025]
Daya Guo, Dejian Yang, Haowei Zhang, and 1 others
Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786. Daya Guo, Dejian Yang, Haowei Zhang, and 1 others
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.