Pith. sign in

REVIEW 1 major objections 6 minor 25 references

One-shot expert pruning can delete half of a coding model's experts without statistically detectable loss on code generation, but the selection rule that wins changes completely between model families.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:07 UTC pith:MTWFGY6H

load-bearing objection Solid empirical MoE-pruning paper with an overclaimed abstract: the 50% no-loss coding result holds, but the off-domain damage claim rests on the very perplexity metric the paper disqualifies. the 1 major comments →

arxiv 2607.16721 v1 pith:MTWFGY6H submitted 2026-07-18 cs.LG

Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding

Anik Jha This is my paper
classification cs.LG
keywords mixture-of-expertsexpert pruningmodel compressioncoding modelsquantizationperplexityroutingone-shot selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that domain-focused, one-shot expert pruning is a viable way to shrink mixture-of-experts coding models: for both families tested, removing half the experts leaves code-generation accuracy statistically unchanged, with the lost capability concentrated almost entirely outside coding. It also argues that no published selection criterion transfers across model families — the criterion that wins on one model trails badly on the other — so any real deployment needs a per-model shootout judged by functional code evaluation, not by perplexity. The paper further shows that perplexity can rank a broken model above an intact one, that a lightweight fine-tune recovers about half of what aggressive pruning loses, and that pruning beats full-model quantization only below roughly 3 bits per weight. A sympathetic reader would care because this indicates expert pruning can roughly halve the memory footprint of open coding models on a single machine, provided the choice of experts is validated on the actual task.

Core claim

On two open-weight MoE coding models from different families, deleting 50% of the experts per layer leaves greedy pass@1 on the primary Python code benchmark statistically indistinguishable from the base model, while general-chat perplexity degrades up to 2.3x, confirming that pruned capacity is mostly off-domain. The decisive finding is that the ranking criterion that achieves this no-loss result is not portable: on one family the gated-activation-norm criterion wins and routing mass collapses; on the other, routing mass matches the base model while the norm criterion loses 16.5 points. The paper therefore claims that one-shot pruning works, but only under functional, task-matched validatio

What carries the argument

The central mechanism is per-expert flow statistics collected in one profiling pass over a coding-heavy calibration corpus: routed-token counts, gate mass, full-softmax probability, and first/second moments of gated activation norms, plus a domain-contrastive score defined as the ratio of target-domain to general-domain flow. Surgery then deletes experts and the corresponding router rows directly in the checkpoint, with byte-identical verification. The machinery does the work of locating which experts to keep, and the paper's controlled comparison shows the contrastive ratio destroys quality while magnitude-based gated norms carry the result; the same statistics also feed the perplexity demo

Load-bearing premise

The one-shot expert-importance statistics are collected while the intact model's router is still active, and the paper assumes these flows remain a valid ranking after the router is renormalized over survivors — if post-pruning routing shifts which experts the model actually uses, the no-loss result could be an artifact of the proxy.

What would settle it

Take the same pruned 50%-keep specialist and re-run the profiling pass after surgery: measure the actual post-pruning gate mass and flow of surviving experts on the calibration corpus. If the post-pruning ranking of experts (by flow) correlates poorly with the one-shot ranking, or if the model's own router would have chosen a different keep set, the one-shot proxy fails. Alternatively, evaluate the 50%-keep point on a larger code benchmark (1000+ problems) with a pre-registered paired test: a significant drop in pass@1 would overturn the headline no-loss claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners with a fixed memory budget can remove half the experts of these coding MoEs, roughly halving model size on disk, without measurable coding loss on the primary benchmark, when selection is tailored to the model.
  • Published one-shot pruning criteria should not be assumed to transfer across model families; a deployed specialist requires a per-model criterion shootout with functional evaluation.
  • Perplexity should not be used as the gate for pruning studies, since it can rate a broken random-pruned model above the intact base model (a 58-point pass@1 deficit with better code perplexity).
  • Quantizing the full model down to 3 bits per weight dominates pruning at equal memory; pruning is the better choice only below that boundary.
  • A lightweight fine-tune that keeps the routed experts frozen recovers about half of the aggressive-pruning gap on the code benchmark, in contrast to findings in translation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the criterion reversal holds generally, then the one-shot-pruning literature's habit of validating on a single family may have produced results that are partly family-specific artifacts; a useful next test is a three-family or four-family sweep with the same protocol.
  • The perplexity result implies that any pruning paper that reports only perplexity — a large share of the field — could be hiding functional collapse; re-evaluating those models on a functional benchmark is a cheap, decisive check.
  • The agentic repair finding suggests that single-shot benchmarks overstate compression penalties broadly; deployment loops with execution feedback may make deep compression far less costly than static evals imply, which would shift the pruning-versus-quantization crossover in real systems.
  • The failure of causal importance estimation at expert granularity hints that the no-loss region is a property of diffuse, distributed capacity rather than discrete deletable units; if so, expert pruning is bounded by capacity redundancy, and finer-grained structural pruning may extend the flat region.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper studies one-shot expert pruning of two MoE coding models (Qwen3.6-35B-A3B and Gemma-4-26B-A4B) at keep ratios 0.75, 0.50, 0.375, and 0.25 under five selection criteria, with functional pass@1 evaluation on HumanEval+ and MBPP+ (greedy, Q8_0 quantized) and matched perplexity measurements. It reports that 50% of experts can be removed without statistically detectable loss on HumanEval+ when the best per-family criterion is used (REAP on Qwen, routing mass on Gemma), while MBPP+ is unchanged on Gemma but loses 4.5 points on Qwen; that the best criterion flips between families; that perplexity can rank a broken model above an intact one (random-pruned Qwen scores 4.82 code ppl vs 5.38 base while losing 57.9 pass@1 points); that pruning vs quantization at equal memory shows a crossover at roughly 3 bits; and that a LoRA self-distillation recovery with frozen routed experts recovers about half of the 25%-keep HumanEval+ gap. Five pre-specified stress tests (better calibration, guarded selection, causal importance, failure attribution, agentic repair) leave the quantization crossover standing.

Significance. If correct, the paper is a valuable empirical contribution: it demonstrates a no-loss half-pruning result on two families, documents criterion non-transferability with paired CIs, and gives a practical crossover against quantization. The methodological strengths are real: matched imatrix quantization on both sides, byte-level verification of surgery and frozen parameters, pre-registered stress tests with kill criteria, and a released pipeline. The principal weakness is that the advertised 'intended trade' — damage landing almost entirely on abilities outside coding — rests on a metric (perplexity) that the paper itself disqualifies in §5.4, and no functional general-domain evaluation is provided. This is fixable, but it means the manuscript's abstract and §5.1 currently overstate the off-domain evidence.

major comments (1)
  1. [Abstract; §5.1; §5.4] The abstract and §5.1 claim that pruning damage 'lands almost entirely on abilities outside coding' and call the general-chat perplexity rise (2.3x at 50% keep, 6x below) 'the intended trade, not a defect.' But §5.4 argues that held-out perplexity 'cannot carry a pruning result on its own,' citing a random-pruned Qwen that beats base on code perplexity (4.82 vs 5.38) while losing 57.9 pass@1 points, and concludes that 'functional evaluation is the only measurement that consistently ranks checkpoints the way a user would.' No functional evaluation of general-domain abilities (MMLU, GSM8K, instruction following, or agentic) is reported. The off-domain damage claim rests only on a metric the paper itself disqualifies. This internal inconsistency is load-bearing for the advertised trade; either add a functional general-domain benchmark or limit the claim to 'perplexity on general text degrad
minor comments (6)
  1. [Title; Abstract] The title 'Half the Experts, All the Code' overclaims: on Qwen, MBPP+ loses a significant 4.5 points at 50% keep (p=0.006), and on Gemma only the routing-mass criterion achieves the no-loss claim. The abstract's 'primary code benchmark' hedge helps, but the title is still stronger than the results. Consider qualifying it.
  2. [§1; §3] Typos and spacing errors: 'parallelexpertsubnetworks' and 'adomain-contrastivescore' appear in the introduction; please fix.
  3. [§3.2] The text promises 'our equivalence tests (below)' but no formal equivalence-testing procedure (e.g., TOST) is described. The paired bootstrap interval comparisons are useful, but either describe them as bounding intervals or add a formal equivalence test with a prespecified margin.
  4. [§5.4] The 'up to 30-fold per sample' and '14x over the full holdout' template-inflation numbers are for Gemma only. The text later clarifies this, but the first mention could be ambiguous to readers comparing with Qwen numbers in the same paragraph.
  5. [§5.5] The sentence 'quantizing the full model is never worse' is stronger than the evidence: only two comparisons in Table 5 reach significance, and the rest are within noise. Please qualify as 'on the measured configurations' or similar.
  6. [§5.6] The phrase 'better selection cannot repair it' is too strong given the same section reports on-policy calibration (0.787→0.817) and union-guard selection (0.829) improving the 25%-keep specialist. The evidence supports 'better selection helps but cannot close the gap to the 2-bit base,' not 'cannot repair.'

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical comparisons against external benchmarks, not reductions to fitted inputs or self-citations.

full rationale

The paper's derivation chain is empirical and self-contained. The headline result—removing half the experts leaves HumanEval+ statistically unchanged—is a direct measurement on external benchmark suites (HumanEval+, MBPP+) under greedy pass@1, not a quantity computed from fitted parameters that are then 'predicted.' The selection criteria (REAP, routing mass, MAN, random, contrastive) are either taken from prior published work or introduced as candidates and explicitly tested; the paper's own contrastive score is introduced, evaluated, and rejected by the data ('Used alone it destroys the model at both ratios'), so no claimed result reduces to its definition. The one-shot proxy assumption in Section 3.2 ('measured flows are estimates of post-surgery behavior') is a stated modeling caveat, not a circular step, and the paper reports equivalence tests to bound the mechanical part of the routing shift. There is no load-bearing self-citation: the references are external prior work, and no uniqueness theorem or prior result from the same author is invoked to force a conclusion. Recovery tuning in Section 5.7 uses base-model outputs for distillation, but that is disclosed, does not feed back into the pruning comparisons, and is not presented as an independent prediction. The skeptic concern about off-domain 'damage' resting only on perplexity, which Section 5.4 disqualifies as a pruning metric, is a legitimate internal-consistency issue, but it is not circularity: it does not make any derivation equivalent to its inputs. Overall, no step reduces by construction to a fitted value or to a self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claims depend on benchmark representativeness, calibration mix choice, and the one-shot routing proxy. There are no invented physical entities. The free parameters are design choices that are disclosed but not swept.

free parameters (3)
  • Calibration mix ratio (70/30 coding-to-general) = 70% code / 30% general, 2,047 samples
    Chosen by hand; the pruning frontier and criterion rankings could shift with this composition. Reported in Table 1 and Section 4.
  • Calibration sampling seed (single) = not stated
    The paper acknowledges a single seed and single mix composition; sensitivity to calibration randomness is only partially quantified via criterion overlap.
  • LoRA rank and healing hyperparameters = rank 64, 95.2M trainable params, <3 hours
    Used in recovery tuning; no ablation on rank, learning rate, or epochs, so the recovery fraction (52% of HumanEval+ gap) may depend on these choices.
axioms (4)
  • domain assumption HumanEval+ and MBPP+ pass@1 under greedy decoding are valid measures of 'whether the model writes correct code'.
    Used throughout as the functional yardstick; the paper reports 164/378 items, but real coding workloads may not match these distributions (Section 4).
  • domain assumption The 70/30 code-to-general calibration corpus (2,047 samples) defines the target coding domain for expert selection.
    The paper notes single seed/mix and possible overlap of Evol-CodeAlpaca with HumanEval; a different calibration distribution could change 'which experts to keep' (Sections 4, 6).
  • domain assumption One-shot selection statistics measured under base routing are adequate proxies for post-surgery routing behavior.
    Invariants are tested only against a synthetic model and the paper explicitly warns that measured flows are estimates of post-surgery behavior (Sections 3.2-3.3).
  • domain assumption General-chat perplexity is a meaningful proxy for non-coding capability loss.
    The paper itself calls general-text perplexity a coarse proxy and does not claim the specialists are safe general assistants (Section 6).

pith-pipeline@v1.3.0-alltime-deepseek · 13345 in / 14441 out tokens · 144749 ms · 2026-08-01T20:07:49.345786+00:00 · methodology

0 comments
read the original abstract

The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token. That pool is why these models do not fit on the machines most developers own, yet for a user who only wants coding help, most experts encode abilities that will never be invoked. We ask how many experts can be removed, and which, by pruning two recent open-weight MoE models from different families (Qwen3.6-35B-A3B and Gemma-4-26B-A4B) under five selection strategies, judged the way a user would: by whether the model still writes correct code. Half the experts can be removed from either model with no statistically detectable loss on the primary code benchmark, and the damage lands almost entirely on abilities outside coding, the intended trade. But the winning strategy flips between the two models, so a recipe validated on one family cannot be assumed to work on another. We further show that perplexity, the metric much of the pruning literature leans on, can rate a broken model above an intact one; that a lightweight fine-tune recovers about half of what aggressive pruning loses; and that against quantizing the full model to the same memory, pruning wins only where quantization would have to drop below 3 bits per weight. Five attempts to overturn that crossover, with failure criteria fixed in advance (better calibration, guarded selection, causal expert importance, failure attribution, and an agentic evaluation letting each model repair its failures from execution feedback), all leave it standing; the last shows single-shot benchmarks overstate compression penalties broadly, as one repair turn erases the 2-bit quantization penalty entirely. Expert pruning works, but it demands per-model validation on the task the model will actually serve.

Figures

Figures reproduced from arXiv: 2607.16721 by Anik Jha.

Figure 1
Figure 1. Figure 1: The pipeline. A calibration corpus is routed through the base model once to collect [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) HumanEval+ against experts kept under REAP (greedy pass@1 at Q8 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy against model size for every deployable artifact in Table 5: the 50%-pruned [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 9 linked inside Pith

  1. [2]

    Gemma-4-26b-a4b

    Gemma Team, Google DeepMind . Gemma-4-26b-a4b. https://huggingface.co/google/gemma-4-26B-A4B-it, 2026. Model card and released checkpoint; architecture facts verified from config.json, safetensors index, and installed modeling source

    2026

  2. [3]

    llama.cpp

    Georgi Gerganov and contributors. llama.cpp. https://github.com/ggml-org/llama.cpp, 2026

    2026

  3. [4]

    It takes a MAESTRO to prune bad experts

    Palaash Goel, Ayush Maheshwari, and Tanmoy Chakraborty. It takes a MAESTRO to prune bad experts. arXiv preprint arXiv:2607.08601, 2026. https://arxiv.org/abs/2607.08601

    Pith/arXiv arXiv 2026

  4. [6]

    Less is MoE : Trimming experts in domain-specialist language models

    Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, and Heather Miller. Less is MoE : Trimming experts in domain-specialist language models. arXiv preprint arXiv:2606.05538, 2026. https://arxiv.org/abs/2606.05538

    Pith/arXiv arXiv 2026

  5. [7]

    REAP the experts: Why pruning prevails for one-shot MoE compression

    Mike Lasby et al. REAP the experts: Why pruning prevails for one-shot MoE compression. arXiv preprint arXiv:2510.13999, 2025. ICLR 2026. https://arxiv.org/abs/2510.13999

    Pith/arXiv arXiv 2025

  6. [8]

    Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023. https://arxiv.org/abs/2305.01210

    Pith/arXiv arXiv 2023

  7. [9]

    AIMER : Calibration-free task-agnostic MoE pruning

    Zongfang Liu, Guangyi Chen, Shengkun Tang, Yifan Shen, Huan Wang, and Xin Yuan. AIMER : Calibration-free task-agnostic MoE pruning. arXiv preprint arXiv:2603.18492, 2026 a . https://arxiv.org/abs/2603.18492

    Pith/arXiv arXiv 2026

  8. [10]

    How to score experts for one-shot MoE expert pruning: A unified formulation and selection principle

    Zongfang Liu, Jinghui Zhang, Zijian Ma, Guangyi Chen, and Xin Yuan. How to score experts for one-shot MoE expert pruning: A unified formulation and selection principle. arXiv preprint arXiv:2606.15716, 2026 b . https://arxiv.org/abs/2606.15716

    arXiv 2026

  9. [11]

    Martin, Lucas Bandarkar, and Nanyun Peng

    Liu O. Martin, Lucas Bandarkar, and Nanyun Peng. Extracting small translation specialists from LLMs by aggressively pruning experts. arXiv preprint arXiv:2605.28042, 2026. https://arxiv.org/abs/2605.28042

    Pith/arXiv arXiv 2026

  10. [12]

    Qwen3.6-35b-a3b

    Qwen Team . Qwen3.6-35b-a3b. https://huggingface.co/Qwen/Qwen3.6-35B-A3B, 2026. Model card and released checkpoint; architecture facts verified from config.json and safetensors index

    2026

  11. [14]

    MoE pathfinder: Trajectory-driven expert pruning

    Xican Yang, Yuanhe Tian, and Yan Song. MoE pathfinder: Trajectory-driven expert pruning. arXiv preprint arXiv:2512.18425, 2025. https://arxiv.org/abs/2512.18425

    arXiv 2025

  12. [15]

    2025 , note =

    Lasby, Mike and others , journal =. 2025 , note =

    2025

  13. [16]

    How to Score Experts for One-Shot

    Liu, Zongfang and Zhang, Jinghui and Ma, Zijian and Chen, Guangyi and Yuan, Xin , journal =. How to Score Experts for One-Shot. 2026 , note =

    2026

  14. [17]

    It Takes a

    Goel, Palaash and Maheshwari, Ayush and Chakraborty, Tanmoy , journal =. It Takes a. 2026 , note =

    2026

  15. [18]

    and Bandarkar, Lucas and Peng, Nanyun , journal =

    Martin, Liu O. and Bandarkar, Lucas and Peng, Nanyun , journal =. Extracting Small Translation Specialists from. 2026 , note =

    2026

  16. [19]

    arXiv preprint arXiv:2504.06792 , year =

    Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations , author =. arXiv preprint arXiv:2504.06792 , year =

    Pith/arXiv arXiv

  17. [20]

    arXiv preprint arXiv:2607.01444 , year =

    On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain , author =. arXiv preprint arXiv:2607.01444 , year =

    Pith/arXiv arXiv

  18. [21]

    arXiv preprint arXiv:2504.07807 , year =

    Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models , author =. arXiv preprint arXiv:2504.07807 , year =

    Pith/arXiv arXiv

  19. [22]

    2025 , note =

    Yang, Xican and Tian, Yuanhe and Song, Yan , journal =. 2025 , note =

    2025

  20. [23]

    2026 , note =

    Liu, Zongfang and Chen, Guangyi and Tang, Shengkun and Shen, Yifan and Wang, Huan and Yuan, Xin , journal =. 2026 , note =

    2026

  21. [24]

    Is Your Code Generated by

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by. 2023 , note =

    2023

  22. [25]

    2026 , howpublished =

    Qwen3.6-35B-A3B , author =. 2026 , howpublished =

    2026

  23. [26]

    2026 , howpublished =

    llama.cpp , author =. 2026 , howpublished =

    2026

  24. [27]

    He, Haoze and Zou, Xinkai and Jiang, Xuan and Ding, Xingyuan and Qu, Ao and Li, Juncheng Billy and Miller, Heather , journal =. Less is. 2026 , note =

    2026

  25. [28]

    2026 , howpublished =

    Gemma-4-26B-A4B , author =. 2026 , howpublished =

    2026