{"id":"68abe8c4-3a63-4061-9f97-0f79a007fa40","arxiv_id":"2505.03804","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MoEQuant combines expert-balanced self-sampled calibration data with affinity-weighted quantization error to improve low-bit quantization of MoE LLMs.","lead":"This paper presents MoEQuant, a quantization method for Mixture-of-Experts language models that improves 4-bit and 3-bit weight compression. It selects calibration data via model self-sampling with expert balance, and reweights quantization errors by gating affinities.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main gains over AWQ/GPTQ are not causally isolated: MoEQuant changes calibration data and quantization objective simultaneously, and EBSS's balance guidance is never ablated against plain self-sampling.","rationale":"The paper's core idea is plausible, and the full-pipeline results are consistent across three MoE families and 3/4-bit settings. The GPTQ ablation in Table 5 gives some evidence that EBSS and AGQ each contribute small gains relative to a Wikitext2-calibrated GPTQ baseline, which is real support and should not be dismissed. However, the central claim as stated—MoEQuant outperforms AWQ and GPTQ—rests on comparisons that differ in calibration data, and the EBSS ablation never removes the balance term, so the proposed expert-balance mechanism is not isolated from the generic effect of model-generated calibration data. The AGQ quasi-linearity issue in Eq. 17, flagged by the reader, is real but secondary: Eq. 18 can be read as a direct heuristic weighting of tokens by gate affinity, and Table 5 gives modest empirical support for AGQ. The cross-calibration comparison in the main tables is the more load-bearing threat because it bears directly on the headline outperformance claim. These issues are addressable with a small set of controlled experiments, so conditional acceptance is appropriate; since the reader already recommended CONDITIONAL, this stress-test does not change the verdict.","tokens_in":15928,"tokens_out":10701,"duration_ms":110766,"concrete_test":"On Qwen-MoE-14B and DeepSeek-MoE-16B, run (a) AWQ and GPTQ with the exact EBSS-generated calibration set, (b) GPTQ with self-sampled calibration but tau = infinity (balance term disabled), and (c) GPTQ with randomly self-sampled sequences of equal length and count. Compare 4-bit averages on the same nine tasks. If (a) closes the Table 1 gap to MoEQuant, the headline outperformance is attributable to calibration data rather than AGQ/EBSS; if (b) matches EBSS, the expert-balance guidance is not the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 1, every MoEQuant row uses the EBSS self-generated calibration set while every AWQ/GPTQ baseline uses Wikitext2 (Sec. 5.1; table note). Calibration distribution is a first-order factor in PTQ, so the reported outperformance over AWQ/GPTQ is a comparison of a full pipeline against baselines with a different calibration set. Table 5 partially controls this for GPTQ (the EBSS-only row improves DeepSeek/Mixtral averages by +0.86/+1.73), but no AWQ decomposition is given. More importantly, EBSS itself is never decomposed: there is no condition using the same self-sampling without the expert-balance term (tau -> infinity in Eq. 13), no random-self-sampling control, and no 'self-sampling + balance but no AGQ' row for AWQ. The balance term in Eq. 13 is evaluated on the prefix S, not on the candidate S||v, so within a beam branch the balance score is identical for every next-token candidate; it cannot directly steer token-level expert usage. Thus the mechanism claimed to produce expert balance may contribute little beyond the choice of calibration distribution. The abstract's '>10 point HumanEval gain for DeepSeekMoE-16B under 4-bit' is also not visible in any 4-bit table row; the largest 4-bit HumanEval delta over GPTQ in Table 1 is +4.55 on Mixtral.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MoEQuant is a post-training quantization framework for mixture-of-experts (MoE) large language models. The paper identifies two challenges: inter-expert imbalance (uneven calibration-sample distribution across experts) and intra-expert imbalance (varying token-expert affinities). To address them, it proposes Expert-Balanced Self-Sampling (EBSS), which generates calibration data via self-sampling with a beam search guided by cumulative token probabilities and an expert-balance penalty, and Affinity-Guided Quantization (AGQ), which incorporates gating affinities into the quantization loss and Hessian computation. Experiments on Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B at 4-bit and 3-bit weight quantization show consistent average-score improvements over RTN, AWQ, and GPTQ, with ablations reported for the GPTQ-based variant. The abstract additionally claims a 'more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization', which is not visible in any table.","tokens_in":16275,"tokens_out":8532,"duration_ms":83412,"significance":"If the claims hold, MoEQuant would make low-bit deployment of MoE LLMs substantially more practical: the paper reports 3.2x memory savings and 1.2-2.1x decoder speedups at 4-bit, with accuracy close to floating point on several benchmarks. The problem is well-motivated, the experimental suite is broad (three model families, base and instruction-tuned variants, two bitwidths, nine tasks), and the EBSS/AGQ decomposition for the GPTQ variant is a useful ablation. The main empirical pattern, that MoEQuant variants consistently beat AWQ and GPTQ in most evaluated settings, is plausible. However, the causal attribution of these gains is currently incomplete: the baselines use a different calibration set, the expert-balance component of EBSS is never isolated, the AGQ derivation relies on an untested homogeneity approximation for SiLU/GELU activations, and hyperparameters are selected on the same benchmark used for final results. These issues are fixable but require additional experiments and a corrected abstract.","major_comments":[{"comment":"The abstract's headline claim of 'more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization' is not supported by any table. In Table 1, the 4-bit DeepSeek-MoE-16B HumanEval score is 22.56 for GPTQ and 25.00 for MoEQuant++ (a gain of 2.44 points); in Table 2, the largest visible gain over GPTQ is 8.54 points on DeepSeek-MoE-16B-Chat (13.41 to 21.95). Please either correct the abstract or identify the specific omitted result that supports the claim.","section":"Abstract; Tables 1 and 2"},{"comment":"The AGQ derivation moves the gating affinity c_i inside the activation function f by writing f(c_i x W_gate), relying on the 'quasi-linear property' of f. This step is exact only for positively homogeneous activations with c_i > 0, such as ReLU; for SiLU or GELU, which are used in the SwiGLU FFNs of Mixtral, DeepSeek, and Qwen MoE models, f(c_i z) is not equal to c_i f(z). The paper neither tests the accuracy of this approximation nor discusses its failure mode, yet the affinity-weighted Hessian H = X diag(c) X^T in Eq. (19) depends on it. Please quantify the approximation error for SiLU/GELU activations or revise the derivation to make its assumptions explicit.","section":"Sec. 4.3, Eq. (17)"},{"comment":"The expert-balance term in Eq. (13) is evaluated on the prefix S, not on the extended candidate S||v, so for a fixed prefix it is constant across all next-token candidates and cannot directly steer token-level expert usage; it only differentiates branches in the beam search. The paper never ablates this term (for example by taking tau to infinity) and never compares EBSS with unguided self-sampling. Table 5 ablates EBSS as a whole, which simultaneously changes the calibration distribution, the sampling procedure, and the balance penalty, so the specific contribution of the expert-balance mechanism is not demonstrated. Please add a decomposition that isolates the balance component.","section":"Sec. 4.2, Eqs. (13)-(14); Table 5"},{"comment":"The baseline comparison is confounded by the calibration set: in Table 1, every MoEQuant row uses EBSS-generated calibration data, while every AWQ/GPTQ baseline uses Wikitext2, as stated in the table note. Table 5 provides a GPTQ decomposition, but no AWQ decomposition is given. Without an 'AWQ + EBSS' condition, the reported gains of MoEQuant+ over AWQ cannot be attributed to either EBSS or AGQ; they could be due to the calibration distribution alone. Please add AWQ ablations with matched calibration data.","section":"Sec. 5.1, Table 1 note"},{"comment":"The hyperparameters tau and w are selected by maximizing the average score over the same seven-task benchmark used for the final reported results (Tables 6 and 7). This is selection on the evaluation set and can inflate the reported improvements, especially because the hyperparameter sweep is performed on DeepSeek-MoE-16B and then applied to all models. Please tune on a validation split or clearly separate hyperparameter search from final evaluation.","section":"Sec. 5.2 and Appendix A.1, Tables 6-7"}],"minor_comments":[{"comment":"Equation (12) writes PPL(M, S||v) using P(v|S) as if it were already a log-probability; the expression should use log P(v|S), consistent with Eq. (13).","section":"Sec. 4.2, Eq. (12)"},{"comment":"The notation 'arg topk_x (f(w,x)' in Eq. (14) is malformed; please define the top-k selection operator precisely, and make the relationship between S, S_i, and the branch set S unambiguous.","section":"Sec. 4.2, Eq. (14)"},{"comment":"The text refers to 'GAQ' where AGQ is meant; please correct this typo.","section":"Sec. 5.2"},{"comment":"The framework name is written inconsistently as 'MoE-Quant' in the abstract and 'MoEQuant' elsewhere; please standardize.","section":"Abstract and throughout"},{"comment":"The description of shared experts is confusing: the text says 'topk shared experts with the highest affinities are utilized' and defines K as topk over i in {1,...,m}, but shared experts are normally always activated. Please clarify the routing notation.","section":"Sec. 3, Eq. (1)"},{"comment":"The 3-bit results for MoEQuant+ on DeepSeek-MoE-16B report C4 perplexity of 4924 while GPTQ reports 11.66, yet MoEQuant+ has better task scores; this large discrepancy between perplexity and downstream accuracy is unexplained and should be addressed or qualified.","section":"Appendix A.2, Table 9"},{"comment":"The speedup and memory-savings comparison only reports MoEQuant against FP16; it would be informative to compare against the other quantized baselines, since the relative efficiency claim depends on the implementation.","section":"Sec. 5.2, Table 4"},{"comment":"The hyperparameter sweeps report average scores without variance or multiple seeds; given the small differences between adjacent settings (e.g., 39.98 vs 40.01), statistical noise could change the selected values.","section":"Appendix A.1, Tables 6-7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a competent PTQ-for-MoE submission with two mechanisms worth knowing about: EBSS, a self-sampling calibration generator that balances expert usage, and AGQ, which reweights the quantization objective by gating affinities. On three MoE models, 4-bit and 3-bit, the combined pipeline beats GPTQ and AWQ on most tasks. That result holds up in the tables. The ablation in Table 5 shows both pieces earn their keep: EBSS alone adds ~0.2-1.7 average points over Wikitext2, AGQ adds a bit, and the combination is better. This is a genuine, if incremental, contribution.\n\nThe soft spots are real but not fatal. First, the abstract's \"more than 10 points HumanEval gain for DeepSeek-MoE-16B under 4-bit\" is unsupported. The largest 4-bit HumanEval delta over GPTQ is about 8.5 points (DeepSeek-Chat, Table 2); in Table 1 it tops out near 4.5. That's a definite overclaim. Second, the main comparison is partly confounded: MoEQuant uses EBSS calibration, baselines use Wikitext2. Table 5 partially controls this for GPTQ and shows EBSS alone helps, but there's no AWQ decomposition, so part of the reported AWQ gains may be calibration-set effects. Third, EBSS's balance term is not isolated. The balance score in Eq. 13 is computed on the prefix, not the candidate, so within a branch it doesn't steer token selection. A control with plain self-sampling (tau → infinity) would show whether the expert-balance guidance adds anything beyond generating from the model. That's a reasonable experimental request, not a fatal flaw.\n\nOn AGQ: the derivation in Eq. 17 moves gating coefficients inside the activation, which is exact for ReLU but heuristic for GELU/SiLU. However, the actual loss (Eq. 18) and the Hessian (Eq. 19) are exactly the ones for a per-token gating-weighted quadratic, so the approximation may not be load-bearing. Still, the paper should say so.\n\nNet: the central claim—that MoE-specific calibration and affinity weighting help PTQ for MoE LLMs—is plausible and supported by the data, but the paper oversells the headline number and would be strengthened by controls. I'd send it to a competent referee.","headline":"Solid incremental PTQ-for-MoE work with a defensible central claim, but the abstract overstates the HumanEval gain and the main comparison is partly confounded by calibration set choice.","tokens_in":16763,"tokens_out":4132,"would_cite":true,"duration_ms":38027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoEQuant claims that two expert-aware fixes, balanced calibration sampling and gating-affinity weighting, let Mixture-of-Experts LLMs be quantized to 3–4 bits with near-floating-point accuracy, beating GPTQ and AWQ on most evaluated tasks.","keywords":["Mixture-of-Experts","post-training quantization","LLM quantization","expert-balanced calibration","affinity-guided quantization","Hessian weighting","self-sampling calibration","low-bit language models"],"falsifier":"Quantize a GELU- or SiLU-based MoE model (Mixtral-8x7B uses SiLU) with AGQ's approximated Hessian $H=(X\\cdot\\sqrt{c})(X\\cdot\\sqrt{c})^\\top$, and in parallel quantize the same layers with an exact affinity weighting obtained by accumulating each token's squared error weighted by its gating score without commuting $c_i$ through the activation. Comparing C4 perplexity and HumanEval accuracy at 3 bits between the two would settle whether Eq. (17)'s commutation assumption carries the reported gains.","tokens_in":15749,"feed_emoji":"⚖️","tokens_out":8522,"duration_ms":76673,"temperature":0.7,"pith_summary":"The paper argues that the reason Mixture-of-Experts LLMs lose so much accuracy under post-training quantization is not weight statistics alone but two imbalances: calibration samples land unevenly across experts, and the gating weights that define each token's affinity to its assigned experts are ignored. To fix the first, it proposes Expert-Balanced Self-Sampling (EBSS), a data-free way to generate a calibration set with low perplexity and balanced expert usage. To fix the second, it proposes Affinity-Guided Quantization (AGQ), which folds gating coefficients into the per-layer quantization loss and Hessian. The paper reports that on Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B, MoEQuant improves accuracy over AWQ and GPTQ baselines at 4-bit and 3-bit weight quantization, including more than 10 accuracy points gained on HumanEval for DeepSeek-MoE-16B under 4-bit, while cutting memory use by more than 3.2×.","feed_headline":"MoE models keep near-full accuracy at 3-4 bits with new quantization","feed_subtitle":"Balanced calibration and gating-aware weighting restore reasoning accuracy lost by 4-bit quantization.","key_machinery":"The load-bearing object is the affinity-weighted Hessian $H=(X\\cdot\\sqrt{c})(X\\cdot\\sqrt{c})^\\top$, with per-token gating scores $c$ folded into the second-order sensitivity statistic that GPTQ-style methods use for weight updates. It carries Eq. (17)'s assumption that gating affinity is, approximately, a scale factor on inputs at every linear layer of the expert. The other machinery is EBSS's beam-like self-sampling search: it keeps $w$ branches, scores next-token continuations by cumulative log-probability plus an expert-balance penalty $\\sigma/\\tau$, and caches historical probabilities to cut calibration-set search complexity from $O(m^n)$ to $O(wn)$.","core_discovery":"The central claim is that token-expert affinity—the gating probability $c_i$ assigned to a routed token—can and should act as a per-token importance weight inside expert quantization. The paper derives an approximation (Eq. 17) in which the gating coefficient commutes through the expert feedforward network, so a token with affinity $c_i$ affects each linear layer as if its input were scaled by $c_i$. On that basis it redefines the quantization loss as a weighted error sum and the GPTQ-style Hessian as $H=(X\\cdot\\sqrt{c})(X\\cdot\\sqrt{c})^\\top=(X\\cdot c)X^\\top$. Combined with EBSS, which uses model self-sampling plus probability-guided path pruning and a deferred expert-balance penalty to build calibration data, the framework reports near-floating-point accuracy on many tasks at 4 bits and improved average scores at 3 bits.","pith_inferences":["If the reported gains transfer to newer routers, AGQ's key idea—weighting second-order sensitivity by gating affinity—could also be applied to quantizing attention or the routing probabilities themselves, where no analogous calibration set exists.","EBSS's joint objective of low perplexity and balanced expert use is a candidate general recipe for calibration-data selection in any sparse or conditional model, not only MoE LLMs.","Because Eq. (17) commutes the gating weight through the expert network, the approximation may matter most for models using GELU or SiLU at 3 bits; a reader should test AGQ with exact affinity weighting before applying it to such models at very low bitwidths.","A testable extension is to combine EBSS with larger calibration budgets or evaluation-domain data to determine how much of the reported gain comes from training-distribution alignment versus expert balancing."],"forward_implications":["On Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B, 4-bit MoEQuant improves average accuracy over both AWQ and GPTQ, with the largest jumps on reasoning and generation tasks such as HumanEval and GSM8K.","DeepSeek-MoE-16B gains more than 10 accuracy points on HumanEval under 4-bit quantization compared with baseline methods.","At 3-bit weight quantization, MoEQuant layered on GPTQ raises average task scores from 35.85 to 36.47 for DeepSeek-MoE-16B and from 45.03 to 49.75 for Mixtral-8x7B.","Wrapping either AWQ or GPTQ saves more than 3.2× memory and gives at least 1.17× decoder speedup on the tested MoE models at 4 bits.","The ablation study shows the two components are additive: EBSS alone contributes about 1.3 accuracy points and AGQ alone about 2 points on DeepSeek-MoE-16B, and their combination exceeds both."],"supporting_citations":[{"why":"Supplies the Hessian-based layer-wise quantization procedure and weight-update rule that AGQ modifies.","marker":"(Frantar et al., 2022)"},{"why":"Baseline activation-aware weight quantization method that MoEQuant wraps, and source of the per-layer quantization-error objective.","marker":"(Lin et al., 2023)"},{"why":"Supplies the Hadamard rotation used in the GPTQ implementation to remove weight outliers before quantization.","marker":"(Ashkboos et al., 2024)"},{"why":"Defines the DeepSeekMoE architecture with shared and routed experts used in the evaluations.","marker":"(Dai et al., 2024)"},{"why":"Defines the Mixtral-8x7B architecture and gating scheme used in the evaluations.","marker":"(Jiang et al., 2024)"},{"why":"Defines the Qwen-MoE architecture with shared and routed experts used in the evaluations.","marker":"(Qwen, 2024)"},{"why":"Provides the optimal-brain-damage Hessian sensitivity principle that GPTQ's Hessian and AGQ's affinity-weighted Hessian build on.","marker":"(LeCun et al., 1989)"},{"why":"Supplies the self-sampling idea that EBSS uses to generate data-free calibration text aligned with the model's distribution.","marker":"(Liu et al., 2023)"}],"fun_headline_variants":["MoE 4-bit quantization gains 10+ points with affinity guidance","Expert-balanced sampling and affinity weights restore MoE accuracy","MoEQuant: Balanced calibration lifts 4-bit MoE reasoning","Affinity-aware quantization salvages MoE models at low bits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The crux is a single algebraic step: the paper treats each token's gating weight as a scalar that can be moved inside every layer of the expert's neural network; this is exact only for positively homogeneous activations such as ReLU (with positive weights) and approximate for GELU or SiLU, and the approximation is never tested.","fun_headline_variants_meta":{"raw":{"variants":["MoE 4-bit quantization gains 10+ points with affinity guidance","Expert-balanced sampling and affinity weights restore MoE accuracy","MoEQuant: Balanced calibration lifts 4-bit MoE reasoning","Affinity-aware quantization salvages MoE models at low bits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2446,"prompt_tokens":1044,"completion_tokens":1402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":1330}},"tokens_in":660,"tokens_out":1402,"duration_ms":11571,"temperature":1.0,"reasoning_tokens":1330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:25:40.269386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Quantize a GELU- or SiLU-based MoE model (Mixtral-8x7B uses SiLU) with AGQ's approximated Hessian $H=(X\\cdot\\sqrt{c})(X\\cdot\\sqrt{c})^\\top$, and in parallel quantize the same layers with an exact affinity weighting obtained by accumulating each token's squared error weighted by its gating score without commuting $c_i$ through the activation. Comparing C4 perplexity and HumanEval accuracy at 3 bits between the two would settle whether Eq. (17)'s commutation assumption carries the reported gains.","supporting_citations":[],"review_version":1}