Pith. sign in

REVIEW 5 major objections 8 minor 2 cited by

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

T0 review · 5 major / 8 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoEQuant claims that two expert-aware fixes, balanced calibration sampling and gating-affinity weighting, let Mixture-of-Experts LLMs be quantized to 3–4 bits with near-floating-point accuracy, beating GPTQ and AWQ on most evaluated tasks.

desk verdict Solid incremental PTQ-for-MoE work with a defensible central claim, but the abstract overstates the HumanEval gain and the main comparison is partly confounded by calibration set choice. read the letter →

arxiv 2505.03804 v1 pith:LDULQC7K submitted 2025-05-02 cs.LG cs.AI

Xing Hu , Zhixuan Chen , Dawei Yang , Zukang Xu , Chen Xu , Zhihang Yuan , Sifan Zhou , Jiangyong Yu This is my paper · ORCID
classification cs.LGcs.AI
keywords Mixture-of-Expertspost-trainingquantizationLLMexpert-balancedcalibrationaffinity-guidedHessianweightingself-samplinglow-bitlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the reason Mixture-of-Experts LLMs lose so much accuracy under post-training quantization is not weight statistics alone but two imbalances: calibration samples land unevenly across experts, and the gating weights that define each token's affinity to its assigned experts are ignored. To fix the first, it proposes Expert-Balanced Self-Sampling (EBSS), a data-free way to generate a calibration set with low perplexity and balanced expert usage. To fix the second, it proposes Affinity-Guided Quantization (AGQ), which folds gating coefficients into the per-layer quantization loss and Hessian. The paper reports that on Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B, MoEQuant improves accuracy over AWQ and GPTQ baselines at 4-bit and 3-bit weight quantization, including more than 10 accuracy points gained on HumanEval for DeepSeek-MoE-16B under 4-bit, while cutting memory use by more than 3.2×.

What carries the argument

The load-bearing object is the affinity-weighted Hessian $H=(X\cdot\sqrt{c})(X\cdot\sqrt{c})^\top$, with per-token gating scores $c$ folded into the second-order sensitivity statistic that GPTQ-style methods use for weight updates. It carries Eq. (17)'s assumption that gating affinity is, approximately, a scale factor on inputs at every linear layer of the expert. The other machinery is EBSS's beam-like self-sampling search: it keeps $w$ branches, scores next-token continuations by cumulative log-probability plus an expert-balance penalty $\sigma/\tau$, and caches historical probabilities to cut calibration-set search complexity from $O(m^n)$ to $O(wn)$.

What would settle it

Quantize a GELU- or SiLU-based MoE model (Mixtral-8x7B uses SiLU) with AGQ's approximated Hessian $H=(X\cdot\sqrt{c})(X\cdot\sqrt{c})^\top$, and in parallel quantize the same layers with an exact affinity weighting obtained by accumulating each token's squared error weighted by its gating score without commuting $c_i$ through the activation. Comparing C4 perplexity and HumanEval accuracy at 3 bits between the two would settle whether Eq. (17)'s commutation assumption carries the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that token-expert affinity—the gating probability $c_i$ assigned to a routed token—can and should act as a per-token importance weight inside expert quantization. The paper derives an approximation (Eq. 17) in which the gating coefficient commutes through the expert feedforward network, so a token with affinity $c_i$ affects each linear layer as if its input were scaled by $c_i$. On that basis it redefines the quantization loss as a weighted error sum and the GPTQ-style Hessian as $H=(X\cdot\sqrt{c})(X\cdot\sqrt{c})^\top=(X\cdot c)X^\top$. Combined with EBSS, which uses model self-sampling plus probability-guided path pruning and a deferred expert-balance penalty to build calibration data, the framework reports near-floating-point accuracy on many tasks at 4 bits and improved average scores at 3 bits.

Load-bearing premise

The crux is a single algebraic step: the paper treats each token's gating weight as a scalar that can be moved inside every layer of the expert's neural network; this is exact only for positively homogeneous activations such as ReLU (with positive weights) and approximate for GELU or SiLU, and the approximation is never tested.

Editorial extensions

If this is right

  • On Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B, 4-bit MoEQuant improves average accuracy over both AWQ and GPTQ, with the largest jumps on reasoning and generation tasks such as HumanEval and GSM8K.
  • DeepSeek-MoE-16B gains more than 10 accuracy points on HumanEval under 4-bit quantization compared with baseline methods.
  • At 3-bit weight quantization, MoEQuant layered on GPTQ raises average task scores from 35.85 to 36.47 for DeepSeek-MoE-16B and from 45.03 to 49.75 for Mixtral-8x7B.
  • Wrapping either AWQ or GPTQ saves more than 3.2× memory and gives at least 1.17× decoder speedup on the tested MoE models at 4 bits.
  • The ablation study shows the two components are additive: EBSS alone contributes about 1.3 accuracy points and AGQ alone about 2 points on DeepSeek-MoE-16B, and their combination exceeds both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported gains transfer to newer routers, AGQ's key idea—weighting second-order sensitivity by gating affinity—could also be applied to quantizing attention or the routing probabilities themselves, where no analogous calibration set exists.
  • EBSS's joint objective of low perplexity and balanced expert use is a candidate general recipe for calibration-data selection in any sparse or conditional model, not only MoE LLMs.
  • Because Eq. (17) commutes the gating weight through the expert network, the approximation may matter most for models using GELU or SiLU at 3 bits; a reader should test AGQ with exact affinity weighting before applying it to such models at very low bitwidths.
  • A testable extension is to combine EBSS with larger calibration budgets or evaluation-domain data to determine how much of the reported gain comes from training-distribution alignment versus expert balancing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. MoEQuant is a post-training quantization framework for mixture-of-experts (MoE) large language models. The paper identifies two challenges: inter-expert imbalance (uneven calibration-sample distribution across experts) and intra-expert imbalance (varying token-expert affinities). To address them, it proposes Expert-Balanced Self-Sampling (EBSS), which generates calibration data via self-sampling with a beam search guided by cumulative token probabilities and an expert-balance penalty, and Affinity-Guided Quantization (AGQ), which incorporates gating affinities into the quantization loss and Hessian computation. Experiments on Qwen-MoE-14B, DeepSeek-MoE-16B, and Mixtral-8x7B at 4-bit and 3-bit weight quantization show consistent average-score improvements over RTN, AWQ, and GPTQ, with ablations reported for the GPTQ-based variant. The abstract additionally claims a 'more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization', which is not visible in any table.

Significance. If the claims hold, MoEQuant would make low-bit deployment of MoE LLMs substantially more practical: the paper reports 3.2x memory savings and 1.2-2.1x decoder speedups at 4-bit, with accuracy close to floating point on several benchmarks. The problem is well-motivated, the experimental suite is broad (three model families, base and instruction-tuned variants, two bitwidths, nine tasks), and the EBSS/AGQ decomposition for the GPTQ variant is a useful ablation. The main empirical pattern, that MoEQuant variants consistently beat AWQ and GPTQ in most evaluated settings, is plausible. However, the causal attribution of these gains is currently incomplete: the baselines use a different calibration set, the expert-balance component of EBSS is never isolated, the AGQ derivation relies on an untested homogeneity approximation for SiLU/GELU activations, and hyperparameters are selected on the same benchmark used for final results. These issues are fixable but require additional experiments and a corrected abstract.

major comments (5)
  1. [Abstract; Tables 1 and 2] The abstract's headline claim of 'more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization' is not supported by any table. In Table 1, the 4-bit DeepSeek-MoE-16B HumanEval score is 22.56 for GPTQ and 25.00 for MoEQuant++ (a gain of 2.44 points); in Table 2, the largest visible gain over GPTQ is 8.54 points on DeepSeek-MoE-16B-Chat (13.41 to 21.95). Please either correct the abstract or identify the specific omitted result that supports the claim.
  2. [Sec. 4.3, Eq. (17)] The AGQ derivation moves the gating affinity c_i inside the activation function f by writing f(c_i x W_gate), relying on the 'quasi-linear property' of f. This step is exact only for positively homogeneous activations with c_i > 0, such as ReLU; for SiLU or GELU, which are used in the SwiGLU FFNs of Mixtral, DeepSeek, and Qwen MoE models, f(c_i z) is not equal to c_i f(z). The paper neither tests the accuracy of this approximation nor discusses its failure mode, yet the affinity-weighted Hessian H = X diag(c) X^T in Eq. (19) depends on it. Please quantify the approximation error for SiLU/GELU activations or revise the derivation to make its assumptions explicit.
  3. [Sec. 4.2, Eqs. (13)-(14); Table 5] The expert-balance term in Eq. (13) is evaluated on the prefix S, not on the extended candidate S||v, so for a fixed prefix it is constant across all next-token candidates and cannot directly steer token-level expert usage; it only differentiates branches in the beam search. The paper never ablates this term (for example by taking tau to infinity) and never compares EBSS with unguided self-sampling. Table 5 ablates EBSS as a whole, which simultaneously changes the calibration distribution, the sampling procedure, and the balance penalty, so the specific contribution of the expert-balance mechanism is not demonstrated. Please add a decomposition that isolates the balance component.
  4. [Sec. 5.1, Table 1 note] The baseline comparison is confounded by the calibration set: in Table 1, every MoEQuant row uses EBSS-generated calibration data, while every AWQ/GPTQ baseline uses Wikitext2, as stated in the table note. Table 5 provides a GPTQ decomposition, but no AWQ decomposition is given. Without an 'AWQ + EBSS' condition, the reported gains of MoEQuant+ over AWQ cannot be attributed to either EBSS or AGQ; they could be due to the calibration distribution alone. Please add AWQ ablations with matched calibration data.
  5. [Sec. 5.2 and Appendix A.1, Tables 6-7] The hyperparameters tau and w are selected by maximizing the average score over the same seven-task benchmark used for the final reported results (Tables 6 and 7). This is selection on the evaluation set and can inflate the reported improvements, especially because the hyperparameter sweep is performed on DeepSeek-MoE-16B and then applied to all models. Please tune on a validation split or clearly separate hyperparameter search from final evaluation.
minor comments (8)
  1. [Sec. 4.2, Eq. (12)] Equation (12) writes PPL(M, S||v) using P(v|S) as if it were already a log-probability; the expression should use log P(v|S), consistent with Eq. (13).
  2. [Sec. 4.2, Eq. (14)] The notation 'arg topk_x (f(w,x)' in Eq. (14) is malformed; please define the top-k selection operator precisely, and make the relationship between S, S_i, and the branch set S unambiguous.
  3. [Sec. 5.2] The text refers to 'GAQ' where AGQ is meant; please correct this typo.
  4. [Abstract and throughout] The framework name is written inconsistently as 'MoE-Quant' in the abstract and 'MoEQuant' elsewhere; please standardize.
  5. [Sec. 3, Eq. (1)] The description of shared experts is confusing: the text says 'topk shared experts with the highest affinities are utilized' and defines K as topk over i in {1,...,m}, but shared experts are normally always activated. Please clarify the routing notation.
  6. [Appendix A.2, Table 9] The 3-bit results for MoEQuant+ on DeepSeek-MoE-16B report C4 perplexity of 4924 while GPTQ reports 11.66, yet MoEQuant+ has better task scores; this large discrepancy between perplexity and downstream accuracy is unexplained and should be addressed or qualified.
  7. [Sec. 5.2, Table 4] The speedup and memory-savings comparison only reports MoEQuant against FP16; it would be informative to compare against the other quantized baselines, since the relative efficiency claim depends on the implementation.
  8. [Appendix A.1, Tables 6-7] The hyperparameter sweeps report average scores without variance or multiple seeds; given the small differences between adjacent settings (e.g., 39.98 vs 40.01), statistical noise could change the selected values.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor test-set selection in EBSS hyperparameter tuning; the core quantization derivation is self-contained and not circular.

  1. fitted input called prediction [Appendix A.1, Tables 6-7; final results in Table 1]
    "The experiments are performed on DeepSeek-MoE-16B across seven tasks, as shown in Table 6 and Table 7. When τ is set to 1.2, the average score across datasets is maximized. Similarly, setting w to 4 yields optimal results, with further increases in w offering only marginal score improvements while significantly increasing generation time."

    The same seven-task average used to select τ and w in Appendix Tables 6-7 on DeepSeek-MoE-16B is the AVG.TEXT column reported for MoEQuant++ in Table 1. Choosing hyperparameters to maximize this benchmark and then reporting the same benchmark as evidence of superiority is selection on the evaluation set: the final headline averages are not independent predictions but partly optimized values. This is a mild 'fitted input called prediction' issue; it does not infect the quantization derivation itself, since only two scalar hyperparameters are involved and the core method is still evaluated on external tasks.

full rationale

The core derivation chain of MoEQuant is not circular. EBSS constructs a calibration set by optimizing Eq. (9) over model perplexity and expert balance, using self-sampling rather than task labels; AGQ reweights the quantization loss and Hessian via gating affinities (Eqs. 18-19). Neither quantity is fitted to the downstream task accuracies, and final accuracy is measured on held-out external benchmarks, so the central claim does not reduce to its inputs by construction. The main comparison changes calibration distribution and quantization objective simultaneously, and the abstract's 'more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit' is not visible in any 4-bit table row; these are experimental-validity and correctness concerns, not circularity. The only circularity-adjacent step is the EBSS hyperparameter selection: τ and w are tuned on the same seven-task benchmark whose average is then reported as the final MoEQuant++ score, mildly compromising the independence of the headline numbers. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces two tunable hyperparameters and relies on two unproven approximations: the quasi-linearity of FFN activations in AGQ and the transferability of self-sampled calibration in EBSS. No new physical entities are postulated.

free parameters (2)
  • tau (τ) = 1.2 for DeepSeek-MoE-16B
    In Eqs. (9)-(10), τ controls the importance of expert balance in the EBSS objective. Table 6 shows τ=1.2 chosen by sweeping on DeepSeek-MoE-16B across seven tasks, so it is tuned to the evaluation benchmark.
  • branch number w = 4
    Beam width in the EBSS search (Eq. 14). Table 7 shows w=4 chosen as a trade-off, with larger w giving marginal improvements; tuned on the same benchmark.
assumptions (3)
  • ad hoc to paper The gating affinity c_i can be treated as a per-token scaling of the expert's input features (Eq. 17) so that it propagates through all linear layers of the FFN.
    Used to justify the affinity-weighted loss (Eq. 18) and Hessian (Eq. 19). Exact for ReLU with positive c_i, approximate for GELU and SiLU; the approximation is not tested.
  • domain assumption Self-sampled calibration data from the model itself matches the model's pre-training distribution well enough to improve quantization generalization.
    EBSS relies on this to argue the generated samples are representative. The paper shows low perplexity on generated data but does not directly verify transfer to downstream tasks except through final benchmark results.
  • domain assumption Expert balance, defined as the standard deviation of usage frequency (Eqs. 3-4), is a valid proxy for calibration quality.
    The objective (Eq. 9) couples perplexity and expert std dev. No theoretical guarantee that balancing usage reduces quantization error on downstream tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance." pith.science (2026). https://pith.science/paper/LDULQC7K

@misc{pith2026250503804,
  author       = {Pith},
  title        = {Pith review of: MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDULQC7K}},
  note         = {Machine review of arXiv:2505.03804}
}
read the original abstract

Mixture-of-Experts (MoE) large language models (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scalability, have achieved higher performance while reducing computational costs. However, these models face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), a widely used method for compressing LLMs, encounters severe accuracy degradation and diminished generalization performance when applied to MoE models. This paper investigates the impact of MoE's sparse and dynamic characteristics on quantization and identifies two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts. To address these challenges, we propose MoEQuant, a novel quantization framework tailored for MoE LLMs. MoE-Quant includes two novel techniques: 1) Expert-Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a calibration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metrics as guiding factors. 2) Affinity-Guided Quantization (AGQ), which incorporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on different experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial performance gains (more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization) and boosts efficiency.

Figures

Figures reproduced from arXiv: 2505.03804 by the authors.

Figure 2
Figure 2. Sample distribution on the first MoE layer of [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The MoE structure in LLMs. The router selects [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Perplexity performance on DeepSeek-MoE-16B [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Illustrative diagram of EBSS. The expert distri [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  2. MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

    cs.LG 2025-09 reject novelty 5.0 of 10

    In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.

Reference graph

Works this paper leans on

22 extracted references · 2 canonical work pages · cited by 2 Pith papers

  1. [5]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  2. [6]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models

    Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y ., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models. arXiv preprint arXiv:2401.06066,

    arXiv

  3. [7]

    Similarly, settingw to 4 yields optimal results, with further increases inw offering only marginal score improvements while significantly increasing generation time

    Whenτ is set to 1.2, the average score across datasets is maximized. Similarly, settingw to 4 yields optimal results, with further increases inw offering only marginal score improvements while significantly increasing generation time. Table 6: Differentτ on avg scores across 7 tasks for DeepSeek-MoE-16B withMoEQuant ++. τ 1.0 1.1 1.2 1.3 1.4 1.5 AVG. 39.8...

    work page 2025

  4. [9]

    Gptq: Accurate post-training quantization for generative pre- trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

    arXiv

  5. [10]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J

    URL https://zenodo.org/records/12608602. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    arXiv 2009

  6. [15]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 , 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical rep...

    arXiv 2025

  7. [16]

    The race to efficiency: A new perspective on ai scaling laws

    Lu, C.-P. The race to efficiency: A new perspective on ai scaling laws. arXiv preprint arXiv:2501.02156,

    arXiv

  8. [18]

    Gptvq: The blessing of dimensionality for llm quanti- zation

    van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bas- toul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. Gptvq: The blessing of dimensionality for llm quanti- zation. arXiv preprint arXiv:2402.15319,

    arXiv

Show all 22 references
  1. [19]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438,

    arXiv

  2. [20]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  3. [21]

    Appendix A.1

    11 Submission and Formatting Instructions for ICML 2025 A. Appendix A.1. Ablation study In this section, we provide the complete comparison of results for our method EBSS and AGQ. As shown in Table 5, taking DeepSeek-MoE-16B as an example, when applied alone, EBSS brings a nea...

    2025

  4. [1989]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978,

    arXiv

  5. [1991]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,

    arXiv

  6. [1994]

    W., and Keutzer, K

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629,

    arXiv

  7. [2015]

    Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339,

    arXiv

  8. [2016]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

    arXiv

  9. [2017]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y ., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319,

    1905 arXiv

  10. [2019]

    L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456,

    arXiv

  11. [2021]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    1905 arXiv

  12. [2022]

    Learning fac- tored representations in a deep mixture of experts

    Eigen, D., Ranzato, M., and Sutskever, I. Learning fac- tored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314,

    arXiv

  13. [2023]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  14. [2024]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.