{"id":"399e3a49-a319-4ff9-89f3-7c82f7d5e703","arxiv_id":"2608.07890","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Router-norm changes induced by lightweight fine-tuning, for example router-only LoRA, provide a one-shot expert-pruning score that preserves MMLU-Pro accuracy far better than magnitude or random pruning on Mixtral-8x7B.","lead":"The paper shows that briefly fine-tuning a Mixture-of-Experts language model with a tiny adapter, such as LoRA on the router only, reveals which experts can be safely removed. This makes a theoretically motivated expert-pruning criterion practical for large models, cutting memory by about half with a modest accuracy loss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Qwen transfer experiment is circular: the 20-expert base was pre-selected by the same router-sensitivity criterion, so Section 8.3 cannot independently validate cross-model transfer.","rationale":"The reader's formal weakest_assumption is the mechanistic premise that router movement is a faithful proxy for expert utility. That is a real assumption, but the most concrete and falsifiable weakness is the Qwen preprocessing circularity, which the reader flags in the rationale but does not make the primary concern. The Qwen experiment is load-bearing because it is the sole evidence that the criterion transfers across model family, training regime, and domain. Section 4 discloses that the 20-expert base was itself selected using the same router-only LoRA sensitivity, so the subsequent 20-to-10 pruning comparison cannot distinguish the criterion's own transfer from the effect of the initial preselection. The random and magnitude baselines in Table 7 are applied only after that preselection, so they do not control for it. Configuration selection keyed to MMLU-Pro is also a limitation, but the paper reports complete sweeps, independent runs, and replication statistics, so it is less damaging than the circular transfer test. The proposed control experiment would settle the issue directly. Since the reader's verdict is already CONDITIONAL and this concern reinforces that condition rather than overturning the Mixtral core, no verdict change is needed.","tokens_in":21574,"tokens_out":8216,"duration_ms":103533,"concrete_test":"Run the Qwen control: from the original 60-expert Qwen1.5-MoE, build three 20-expert bases—one by the paper's router-only LoRA selection, one by random selection, and one by router weight magnitude—then apply the identical mathematical fine-tuning and the identical router-sensitivity scoring and 20-to-10 pruning from Section 8.3. Compare mean accuracy over the same eleven benchmarks. If the router-selected base yields roughly the same 49.7% and the random or magnitude bases yield clearly lower retention under the same 20-to-10 pruning, the transfer claim survives. If pruning from a random base closes the gap, the Qwen result is an artifact of initial preselection. Also report the overlap between experts removed in the initial 60-to-20 reduction and those removed in the 20-to-10 pruning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim includes transfer to Qwen1.5-MoE (Abstract, Section 8.3, Table 7), where router sensitivity retains 49.7% mean accuracy after pruning half of 20 experts while random pruning collapses to 4.4%. This is the only cross-family, full-fine-tuning validation. Section 4 states the original 60-expert Qwen model was reduced to 20 experts per layer 'using a router-only LoRA adaptation to select the experts to remove.' The resulting 20-expert checkpoint is then fully fine-tuned and pruned again by the same criterion. The 49.7% result therefore measures re-pruning of a checkpoint whose expert set was already selected by the criterion itself. The random and magnitude baselines are computed only at the second stage, on that pre-selected base, so they do not control for the initial selection. An alternative explanation is that the initial 60-to-20 router-sensitivity reduction, rather than the full-fine-tuning sensitivity, is what makes the surviving experts resilient to subsequent pruning. This circularity directly undercuts the abstract's transfer claim and the 'provably motivated ... practical at scale' conclusion, even though the paper openly discloses the preprocessing. The Mixtral results are not affected, but the cross-model generalization argument rests on a tainted experiment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using the magnitude of the per-expert change in router weights induced by a lightweight PEFT run (or, for frozen routers, the shift in router logits) as a one-shot expert-pruning score for MoE language models. On Mixtral-8x7B-Instruct, the authors report that router-only LoRA (0.002% of parameters) yields better MMLU-Pro accuracy after pruning than all-module LoRA at matched rank, that the signal improves monotonically with LoRA rank, that multiplicative IA3 roughly matches router LoRA, and that accuracy under pruning decays quasi-linearly. They also report a transfer test on Qwen1.5-MoE fine-tuned for mathematics, keeping 49.7% mean accuracy over eleven benchmarks with half the experts removed. The paper includes controlled sweeps, replication runs, deployment measurements, and released code and checkpoints.","tokens_in":21761,"tokens_out":5930,"duration_ms":61909,"significance":"The Mixtral results are internally consistent and supported by replication analysis; the central comparison between router-only and all-module LoRA exceeds the stated measurement resolution, and the controlled sweeps are a useful empirical mapping of when the criterion works. If the Qwen transfer claim could be independently validated, this would be a practically significant recipe for expert pruning in deployment settings that already include PEFT. However, the Qwen experiment is circular as a transfer test, and the paper's abstract and conclusion overstate the cross-family evidence. The paper is unusually transparent about its limitations, which is a strength, but the current framing does not fully reflect them.","major_comments":[{"comment":"The Qwen transfer result is circular. Section 4 states that the original 60-expert Qwen model was first reduced to 20 experts per layer 'using a router-only LoRA adaptation to select the experts to remove.' Section 8.3 then reports the 49.7% retention after re-pruning this pre-selected 20-expert base with the same criterion, with random and magnitude baselines computed only at the second stage. This does not independently validate cross-model transfer, because the initial 60-to-20 selection may itself be responsible for the resilience of the surviving experts; the baselines do not control for that initial selection. Since this is the only cross-family, full-fine-tuning validation in the paper, the abstract's claim that 'the criterion also transfers to Qwen1.5-MoE' is not supported. Please either re-run the full experiment on the original 60-expert Qwen model (e.g., using the ZeRO-offloading setup already described), or explicitly recast Section 8.3 as a study of re-pruning a pre-selected base and remove or carefully qualify the transfer claims in the abstract and conclusion.","section":"Section 4, Section 8.3, Table 7, Abstract"},{"comment":"The pruning score used in the main results is the magnitude of the router-norm change s_e in Eq. (4), but the theoretical guarantee from Chowdhury et al. is stated for the signed norm change. Section 11 acknowledges that the sign question is open, yet the abstract and conclusion repeatedly describe the approach as 'provably motivated.' As written, this overstates the theoretical grounding for the exact score being evaluated. Please either report the signed-score variant in the main results (or at minimum show that the choice of sign does not affect the qualitative findings), or soften 'provably motivated' to 'empirically motivated' in the abstract and conclusion.","section":"Section 3.2, Eq. (4), Section 11, Abstract"}],"minor_comments":[{"comment":"In the sensitivity scoring code, the line `router.__wrapped__(lb) if False else router(lb)` appears to be a leftover debugging construct; simplify it to `router(lb)`.","section":"Appendix D"},{"comment":"The section title 'How Should Adaptation Perturb the Model?' suggests a causal conclusion, but the IA3 and Houlsby comparisons are run under unmatched budgets and the section itself notes this. Consider a more cautious title or an explicit caveat in the first paragraph.","section":"Section 7"},{"comment":"The note distinguishing the 4-bit memory measurement (24.2GB) from the 24.2B parameter count of the four-experts-pruned model is helpful, but the row labels '1 pruned' through '4 pruned' can be misread as the number of experts retained; consider labeling the column 'Experts retained' for clarity.","section":"Table 9"},{"comment":"The text states that MMLU-Pro loses 21.0 points end to end for selection D, but Table 6 does not include MMLU-Pro; adding a row or footnote with that number would make the cross-benchmark comparison easier to verify.","section":"Section 8.2"}],"recommendation":"major_revision","confidential_remarks":"The Qwen circularity is disclosed by the authors but is not reflected in the abstract or conclusion, which is a problem for the paper's central transfer claim. The Mixtral results are strong and could stand alone as a paper about router-sensitivity-based pruning in a single model family; the revision should either add an independent Qwen experiment or carefully scope the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something useful: it takes Chowdhury et al.'s expert-pruning criterion, previously defined for full fine-tuning, and shows that a cheap PEFT run can elicit the same signal. The central Mixtral finding is robust: router-only LoRA (0.002% of parameters) produces a ranking that, at half experts removed, beats magnitude and random pruning by a wide margin (27.54% vs 15.88%/16.31%), far above their stated 2.2-point resolution. The rank sweep is clean and monotone, the module-ladder result (confining adaptation to the router beats spreading it) is genuinely informative, and the IA3-vs-Houlsby comparison suggests perturbation geometry matters more than parameter count. They report replication runs, release code and checkpoints, and state their measurement resolution honestly. That is real empirical work.\n\nThe soft spots are real but localized. The Qwen transfer experiment (Section 8.3) is not an independent validation: the 20-expert base was itself reduced from 60 experts using router-sensitivity selection. The 49.7% retention after pruning to 10 experts therefore measures re-pruning of a checkpoint already selected by the same criterion. Random and magnitude baselines are only computed at the second stage, so they do not control for the initial selection. The authors disclose the preprocessing in Section 4, but the abstract still claims the criterion 'transfers' to Qwen, and that claim rests on a circular experiment. The Mixtral results are unaffected, but the generalization argument needs to be reframed as a within-criterion consistency check, not a transfer validation.\n\nThe second concern is minor: configuration selection was keyed to MMLU-Pro, and the sweeps use single runs with a 2.2-point resolution. They acknowledge this in the Discussion. The precise ordering of module-ladder rungs under two points is unreliable, but the endpoint comparisons (router-only vs all-module, IA3 vs Houlsby) are large enough to hold. The mechanistic claim that router changes capture expert utility is plausible and supported by the module-ladder evidence, though not proven.\n\nThis paper is for researchers and practitioners working on MoE compression. The Mixtral core deserves a serious referee. The Qwen transfer section needs major revision: either recompute baselines on the original 60-expert model or explicitly scope the claim. I would send it to peer review.","headline":"The Mixtral core is a solid, honest empirical study of a cheap expert-pruning signal; the Qwen transfer claim is weakened by a circular pre-selection and should be re-scoped.","tokens_in":22334,"tokens_out":2517,"would_cite":true,"duration_ms":27047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A brief router-only fine-tune can identify which experts to prune in Mixture-of-Experts models.","keywords":["mixture-of-experts","expert pruning","router sensitivity","parameter-efficient fine-tuning","LoRA","one-shot pruning","MMLU-Pro","model compression"],"falsifier":"Apply router-only LoRA at rank 8 to a second coarse-grained Mixture-of-Experts model, prune half the experts one shot, and evaluate on MMLU-Pro; if accuracy falls below magnitude-based or random pruning, or if the decay curve shows a cliff beyond one expert, the router-sensitivity signal is not general.","tokens_in":21347,"feed_emoji":"✂️","tokens_out":6130,"duration_ms":57427,"temperature":0.7,"pith_summary":"This paper claims that the signal needed to prune experts from a Mixture-of-Experts language model can be recovered from a brief, parameter-efficient fine-tuning run rather than from full fine-tuning. The authors' recipe is to attach a small adapter, read off how much each expert's router gate vector (or, when the router is frozen, its logits) moved, and delete the least-changed experts in one shot. On Mixtral-8×7B-Instruct, restricting LoRA to the router weights, training 0.002% of parameters, outperforms adapting all modules at the same rank once half the experts are removed (27.54% vs. 24.42% MMLU-Pro), and the ranking transfers to Qwen1.5-MoE fine-tuned on mathematics, where half the experts can be removed while retaining 49.7% mean accuracy over eleven math benchmarks. The practical stake is that expert pruning is the direct route to shrinking MoE memory and latency, and the paper shows the accuracy cost of compression is quasi-linear and predictable rather than a cliff.","feed_headline":"Router-only LoRA marks the MoE experts you can safely delete","feed_subtitle":"A 0.002%-parameter fine-tune ranks experts; half removed, MMLU-Pro stays near 28% while memory drops 49%.","key_machinery":"The load-bearing object is the router gate matrix $W_g \\in \\mathbb{R}^{E \\times d}$, whose rows are the gate vectors $w_e$; the router is the only place where the decision of which experts serve a token is expressed as parameters. The pruning score is the per-expert $\\ell_2$ norm change, $s_e = \\left| \\|w_e^{(T)}\\|_2 - \\|w_e^{(0)}\\|_2 \\right|$, computed between the pretrained and adapted checkpoints, with a logit-delta variant $s^{\\text{logit}}_e = \\frac{1}{\\sqrt{N}}\\|\\Delta L_{:,e}\\|_2$ for adaptations such as IA3 and Houlsby adapters that leave $W_g$ frozen. The argument is that this score measures how much the task revises the router's reliance on each expert, and that pruning the least-revised experts one shot, with no retraining or calibration, preserves accuracy in proportion to how faithfully the adapter was forced and disciplined in moving the router.","core_discovery":"At the paper's center is a measurement: adapt briefly, then rank experts by the magnitude of the change adaptation induces in their router vectors, $s_e = \\left|\\|w_e^{(T)}\\|_2 - \\|w_e^{(0)}\\|_2\\right|$, and remove the experts with the smallest scores from every layer. The authors establish, on Mixtral-8×7B-Instruct, that this router-sensitivity ranking is strongest when adaptation is confined to the router, improves monotonically with adapter rank up to the largest rank tested, and survives when the router is left frozen and the score is instead read from $\\ell_2$ shifts in router logits. They further show that unconstrained additive adapters degrade the signal, while multiplicative scaling matches direct router adaptation. With half the experts removed, MMLU-Pro accuracy under router-guided pruning is 27.54% at rank 8 and 28.76% at rank 16, roughly 1.8 times the accuracy of magnitude-based or random pruning at maximal compression, while memory falls 49% and per-token latency 37%. The criterion transfers to Qwen1.5-MoE fully fine-tuned on mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed where random pruning collapses to 4.4%.","pith_inferences":["If the mechanism is right, the same ranking should improve further when the signed norm change is used instead of the magnitude, since contracting gate norms are observable in the sensitivity maps; the paper leaves that re-ranking as an open experiment.","The Qwen transfer is partly confounded because the 20-expert starting checkpoint was itself produced by router-sensitivity selection from the 60-expert model, so a fully independent cross-family test would begin from the original checkpoint.","The paper's view of adaptation as a measuring instrument inverts standard training guidance: broad LoRA attachment is best for task loss but worst for pruning signal, suggesting future adaptation recipes should be chosen by measurement legibility rather than by downstream accuracy.","If the adaptation corpus is truly a steering wheel, then mixing corpora with known capability labels before adaptation should yield an expert ranking that trades off protected capabilities controllably; this is a direct testable extension."],"forward_implications":["Practitioners who already run a LoRA fine-tune before deployment can obtain an expert-pruning ranking as a byproduct, at no extra training cost.","Compression targets can be priced in advance: on the measured hardware each removed expert saves roughly 3GB of memory and 3.5 to 4.5ms per token, while MMLU-Pro accuracy decays quasi-linearly.","Adaptation configuration is a control surface: adapting on a corpus that exercises a capability protects the experts that capability relies on, so a practitioner who needs arithmetic preserved should adapt on arithmetic.","The signal does not require writing to the router; frozen-router methods such as multiplicative scaling can be scored through logit shifts, broadening the set of usable adapters.","Router-guided pruning is competitive with calibration-based criteria at moderate compression while using only a tiny fraction of their signal cost."],"supporting_citations":[{"why":"Supplies the provable criterion that experts with the smallest router-norm change during fine-tuning can be pruned, which this paper extends to lightweight adaptation.","marker":"[3]"},{"why":"Defines LoRA, the low-rank adapter used to induce and measure router changes.","marker":"[12]"},{"why":"Defines IA3, the multiplicative adaptation whose frozen-router result ties with direct router LoRA.","marker":"[23]"},{"why":"Defines Houlsby bottleneck adapters, the unconstrained additive geometry that degrades the pruning signal.","marker":"[11]"},{"why":"Provides the REAP one-shot pruning baseline on shared benchmarks and the gate-plus-activation scoring that router sensitivity is compared with.","marker":"[18]"},{"why":"Surveys sixteen expert-dropping criteria on Mixtral, supplying the calibration-based band and the magnitude baseline.","marker":"[15]"},{"why":"Reports DiEP differentiable expert pruning and the calibration-corpus effect on GSM8K used to interpret task conditioning.","marker":"[1]"}],"fun_headline_variants":["Tiny router LoRA finds half of MoE experts are prunable","Prune half of MoE experts with 0.002% router-only LoRA","Router sensitivity from tiny LoRA pinpoints deletable experts","Half of MoE experts can go after 0.002% router LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Expert utility for a task is expressed primarily in how adaptation moves the router's gate vectors or logits, and that movement is not obscured when adaptation also modifies attention and expert weights.","fun_headline_variants_meta":{"raw":{"variants":["Tiny router LoRA finds half of MoE experts are prunable","Prune half of MoE experts with 0.002% router-only LoRA","Router sensitivity from tiny LoRA pinpoints deletable experts","Half of MoE experts can go after 0.002% router LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001273,"raw_usage":{"total_tokens":5306,"prompt_tokens":1142,"completion_tokens":4164,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":758,"completion_tokens_details":{"reasoning_tokens":4082}},"tokens_in":758,"tokens_out":4164,"duration_ms":33153,"temperature":1.0,"reasoning_tokens":4082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:42:36.066773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply router-only LoRA at rank 8 to a second coarse-grained Mixture-of-Experts model, prune half the experts one shot, and evaluate on MMLU-Pro; if accuracy falls below magnitude-based or random pruning, or if the decay curve shows a cliff beyond one expert, the router-sensitivity signal is not general.","supporting_citations":[{"cited_title":"A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts","cited_arxiv_id":null,"evidence_quote":"Supplies the provable criterion that experts with the smallest router-norm change during fine-tuning can be pruned, which this paper extends to lightweight adaptation."},{"cited_title":"Parameter-efficient transfer learning for NLP","cited_arxiv_id":null,"evidence_quote":"Defines Houlsby bottleneck adapters, the unconstrained additive geometry that degrades the pruning signal."}],"review_version":1}