Pith. sign in

REVIEW 5 major objections 5 minor 13 references

MoBiQuant claims that a single LLM checkpoint can serve every precision from 2 to 6 bits by routing each token to a token-specific number of recursively quantized weight slices, matching fixed-precision PTQ without recalibration.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:47 UTC pith:7Z2SRVGQ

load-bearing objection Solid, novel any-precision quantization with a clean slice-composability result, but the core token-routing claim lacks the random-routing control that would prove it. the 5 major comments →

arxiv 2602.20191 v2 pith:7Z2SRVGQ submitted 2026-02-21 cs.LG cs.AIcs.CL

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Dongwei Wang , Jinhee Kim , Seokho Han , Denis Gudovskiy , Yohei Nakata , Tomoyuki Okuno , KhayTze Peong , Kang Eun Jeon
show 3 more authors
Jong Hwan Ko Yiran Chen Huanrui Yang
This is my paper
classification cs.LG cs.AIcs.CL
keywords any-precision quantizationpost-training quantizationtoken-adaptive routingrecursive residual quantizationoutlier migrationelastic LLM inferencemixture-of-bitsweight-only quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that static post-training quantization fails to generalize across bit-widths because the tokens that dominate quantization error change with precision—a phenomenon it calls outlier migration. From this it concludes that precision should be assigned per token rather than per model or layer. Its MoBiQuant framework decomposes each weight into a shared most-significant bit slice plus recursive residual slices, so that summing a prefix of slices yields a valid weight at the corresponding total bit-width. A lightweight router learns a binary mask deciding which residual slices each token needs, with an adjustable threshold that trades accuracy for average bit budget at runtime. Experiments on several open large language models report perplexity matching or beating static PTQ methods across 2–6 bits while supporting fine-grained average-bit control and practical kernel speedups.

Core claim

The central claim is that quantization error is token- and precision-dependent: the rank ordering of token sensitivities reshuffles as bit-width changes, so calibration parameters fit to one precision mis-target another. The paper's discovery is that this outlier migration can be tamed by decoupling quantization into composable bit slices—a base slice and residual slices—and letting a learned router activate a per-token subset. Because slices are constructed with floor-aligned codes, halved scales, and centered zero points, activating a prefix of slices reconstructs the same coarse integer code plus a bounded zero-mean remainder, so elastic precision switching does not corrupt the representa

What carries the argument

MoBiSlice is a recursive residual quantizer that decomposes a weight matrix into E=4 slices of 2 bits each: the first slice quantizes the original weight, and each later slice quantizes the residual left by all previous slices. Composability rests on floor-aligned integer codes, a scale that halves with each slice (s_{e+1}=s_e/2^{b_e}), and a centered zero point (z_e=2^{b_e-1}) for residual slices; under an assumption that the discarded remainder is approximately uniform, truncation adds only a bounded zero-mean term and cannot flip a bit of the coarser code. MoBiRoute is a two-layer MLP router with a sigmoid gating function annealed to a binary mask, plus an adjustable threshold that raises

Load-bearing premise

The slice-summing construction is only equivalent to direct quantization at the target bit-width if the discarded residual bits are approximately uniformly distributed, so truncation never flips a bit of the coarser code; real LLM weights with heavy-tailed residuals could break that equivalence.

What would settle it

Quantize a weight matrix directly at a given bit-width and also reconstruct it by summing the base slice plus truncated residual slices from MoBiSlice. If the two reconstructions differ by more than one coarse quantization step on a substantial fraction of entries—or if the two models show measurably different perplexity on the same calibration set—then the composability claim fails. A concrete test uses a weight distribution with highly skewed residuals (for example, a linear layer with a few very large channels) and compares per-entry error between slice-summed and directly quantized weights

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • One checkpoint can be served at any average bit-width in the 2–6 bit range by adjusting a single global threshold, with no per-bit recalibration and no checkpoint reload.
  • Token-adaptive precision reduces outlier migration, letting elastic inference match static PTQ performance even in the 3–4 bit regime.
  • Fine-grained average-bit control gives a serving system a continuous perplexity–throughput trade-off curve, rather than a few discrete precision points.
  • Bit-major packing and binary matrix multiplication make lower average bit-widths translate into real memory and latency savings, up to 2.7x over the FP16 baseline in the reported kernel.
  • The same slicing and routing mechanism extends to activation quantization, not only weight-only quantization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If outlier migration is a general property of quantized LLMs, token-adaptive precision may also inform other per-token compute decisions, such as expert selection in mixture-of-experts models or token dropping in long-context serving.
  • The composable slice construction is not tied to one PTQ backbone: any calibration scheme that preserves the floor-aligned, scale-halved, centered-residual structure could likely reuse the same elastic mechanism.
  • The paper's counterintuitive result—retaining some tokens at a lower precision than the calibration bit-width improves overall perplexity—suggests that token-level bit allocation can sometimes invert the usual 'more bits is better' ordering, which may open a separate line of adaptive compression.
  • A natural testable extension is a router trained with a curriculum over target budgets rather than a single 3-bit target; the paper's own ablations show the training target biases generalization, so a multi-target schedule might widen the elastic operating range.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. MoBiQuant proposes a post-training quantization framework for elastic LLM inference. It decomposes weights into 2-bit residual slices via recursive quantization (MoBiSlice) and uses a lightweight token-level router (MoBiRoute) to activate a variable number of slices per token, thereby supporting average bit-widths between 2 and 6 bits from a single checkpoint. The paper motivates the design through an 'outlier migration' phenomenon, claiming that tokens that are problematic at one precision differ from those problematic at another, and that static PTQ parameters overfit to a single precision. Experiments on LLaMA-2/3 models compare MoBiQuant against static PTQ baselines and any-precision methods on perplexity, zero-shot accuracy, and kernel throughput, reporting that MoBiQuant matches or surpasses static PTQ while preserving elasticity.

Significance. If the claims hold, MoBiQuant would be a practically valuable contribution: it offers fine-grained, token-adaptive precision control from a single calibration pass, avoids per-bit retraining, and includes a custom kernel to realize memory and latency gains. The Appendix B derivation of slice composability is a genuine attempt to put the 'many-in-one' construction on formal footing, and the benchmark coverage across multiple LLaMA-family models is broad. However, the central causal claim—that token-sensitive routing, rather than the mere availability of mixed precisions, drives the gains—is not tested with the required control. Several internal inconsistencies and reporting gaps (notably the C4 calibration row in Table 2 and the absence of error bars) prevent acceptance in the current form.

major comments (5)
  1. [Sec. 5.2 / Fig. 4 / App. C.2] The paper's central claim is that MoBiRoute's sensitivity-based routing is what enables MoBiQuant to match or surpass static PTQ at a given average bit budget. However, all baselines in Fig. 4 and Table 1 use globally uniform precision; there is no control that assigns tokens to bit slices at random or by a sensitivity-agnostic rule while keeping the same average bit budget per layer. Without such a control, the reported improvements could be due to the availability of a mixture of precisions (some tokens at 4-bit, others at 2-bit) rather than the router's predictions. Figure 5 is a qualitative visualization, not a quantitative comparison against random routing. Additionally, App. C.2 states that every target bit budget requires calibrating layer-wise thresholds on the calibration set; therefore the Abstract's 'without repeated calibration' claim is overstated and should be qualified.
  2. [Sec. 5.4 / Table 2] The prose in Sec. 5.4 states that 'MoBiQuant consistently achieves superior perplexity across most calibration–evaluation set pairs' and that it is robust to calibration set choice. Table 2 directly contradicts this for the C4 calibration row: MoBiQuant gives PPL 26.4 on Wiki vs. OmniQuant's 22.6, and 43.9 vs. 38.1 on PTB, i.e., substantially worse in both cases. Appendix E.3 reports the same numbers. This is not a minor wording issue; it undermines the robustness claim. The authors should either correct the prose, explain the conditions under which C4 calibration fails, or rerun the experiment.
  3. [Eq. (3) / Appendix B] The 'many-in-one' reconstruction in Eq. (3) asserts that summing recursively quantized residual slices produces a weight equivalent to direct quantization at the target precision. Appendix B proves a bound for truncating a merged integer code under the assumption that the discarded remainder is approximately uniform, and with specific choices of scales and zero points. However, the proof does not establish that, for real weight residuals, the sum of dequantized slices equals direct quantization at b bits. If the residual distribution violates the uniform-remainder assumption, the elastic PPL results in Fig. 4 would measure approximation error relative to true any-precision quantization, not any-precision performance. The authors should empirically validate composability by comparing slice-summed weights against direct quantization at each supported bit-width on the actual models used.
  4. [All experimental tables] No error bars, multiple seeds, or variance information are reported anywhere in the paper. This is particularly important in Table 1, where several differences between MoBiQuant and OmniQuant are on the order of 0.01–0.03 PPL (e.g., LLaMA-2-7B 3-bit: 6.07 vs. 6.03; LLaMA-2-13B 4-bit: 5.08 vs. 5.02). Without an estimate of calibration noise, the claim that MoBiQuant 'matches or surpasses' static PTQ is not statistically supported. At minimum, the authors should report the standard deviation over at least three calibration runs for the main comparisons.
  5. [Sec. 4.2 / App. D.2] The training target b is a free hyperparameter and is set to 3-bit by default. Appendix D.2 shows that this choice materially affects performance outside the trained region: training at 2.5-bit improves sub-3.0-bit PPL at the cost of higher-bit performance, while training at higher targets degrades sub-3.0-bit PPL substantially. The paper's broad claims of 'seamless switching across 2–6-bit' are therefore only demonstrated for a router trained at a 3.0-bit target. The authors should either show that the main conclusions hold for other training targets or clearly scope the elasticity claims to the trained operating point.
minor comments (5)
  1. [Sec. 4.2 / Eq. (6)] Notation is inconsistent: Eq. (6) uses X_i and Y_i, while Eq. (4) defines scores for a sequence X with tokens indexed by i. The relationship between X_i (token embeddings) and X (full sequence) should be clarified. Also, the bit size b_j in Eq. (9) is not defined there; it appears later as b_e in Sec. 4.1.
  2. [Appendix E titles] Appendix E.2 is titled 'Full results for Fig. 1' but the table it refers to is Table 1; similarly, Appendix E.3 is titled 'Full results for Fig. 2' but refers to Table 2. These cross-reference errors should be fixed.
  3. [Fig. 1] The bar chart in Fig. 1 reports 'PPL↓ on Wikitext2' with numbers 9.11, 7.31, 9.01, 7.41, 10.06, but the text does not explain all bars (e.g., the 10.06 value). Please add a figure legend that maps each bar to the condition in the text.
  4. [Abstract / Sec. 5.5] The Abstract mentions 'throughput gains of up to 1.34×' over any-precision methods, while Sec. 5.5 reports 'up to 2.7× speedup over FP16.' These are different metrics, but the paper should state both clearly and specify the comparison baselines to avoid confusion.
  5. [Sec. 2 / References] Some references are incomplete or inconsistent: e.g., 'Kim et al., 2025' is cited as both 'Truncquant' (Appendix B) and 'SqueezeLLM' (Sec. 2) with different author lists; please disambiguate and unify the citation format.

Circularity Check

0 steps flagged

No by-construction circularity: MoBiQuant's elastic PPL results are externally benchmarked rather than forced by its fitting equations.

full rationale

MoBiQuant's derivation is not circular. Eq. (3) defines a reconstruction W(b)=Σ W_e from recursively quantized residuals; this is a construction, and its validity is tested empirically against static PTQ baselines on held-out perplexity and zero-shot tasks, not derived from the same fitted numbers. Appendix B's floor-aligned composability proof rests on stated assumptions (uniform remainder, scale halving, centered zero point); if those fail the method degrades, which is a correctness risk, not an equivalence to its inputs. App. C.2's layer-wise threshold calibration controls the realized average bit ratio, so the reported bit budgets are enforced rather than predicted; however, the resulting PPL/accuracy numbers are measured on test sets against external baselines, so no fitted quantity is renamed as a prediction. The default target b=3 and logarithmic schedule are selected by ablations (App. D), i.e., hyperparameter tuning, not circular reasoning. The sole overlapping-author citation (Kim et al. 2025 for truncation-ready floor mapping) supplies a design convention that the paper re-derives arithmetically and validates externally; it is not an unverified uniqueness theorem used to force the conclusion. Concerns about missing random-routing controls are experimental-validity issues, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The method rests on learnable router/quantizer parameters calibrated on 128 WikiText2 sequences and per-layer thresholds calibrated for each target bit budget; these are fitted quantities, not a parameter-free derivation. The conceptual entity "outlier migration" is presented with visual evidence only. No formal verification is provided.

free parameters (5)
  • Target training bit-width b = 3.0 bits (default)
    Chosen after ablation D.2 because it gives the best PPL trade-off across WikiText2/C4/PTB; changing it changes elastic PPL curves.
  • Per-layer inference thresholds δ = Quantile thresholds over calibration router scores (values not reported)
    Appendix C.2 fits a threshold per layer for each requested target bit ratio; this is a per-target fitted quantity affecting the reported elastic PPL.
  • Router and quantization parameters Θr, Θq (incl. LWC/LET from OmniQuant) = Learned on 128 WikiText2 sequences, 2048 tokens each
    Eq. 10 jointly fits the router MLP and quantizer clipping/transform parameters; these are the main calibrated quantities.
  • Router regularization schedule and weight λ = Logarithmic; λ=1e-5 in App. E.1
    Schedule type and strength are selected by PPL comparison on evaluation sets (App. D.1) and affect target-precision exploration.
  • Slice configuration E=4, b_e=2 = slice_bits_list = 2222
    Default design for 2–8-bit support; no systematic justification for this exact decomposition.
axioms (5)
  • domain assumption Uniform residual assumption: discarded remainder r in Eq. 16 is approximately uniform over {0,...,2^p-1}.
    Used in Appendix B to show zero-mean truncation error (Eq. 19); if real residuals are non-uniform, bias can appear in slice reconstruction.
  • ad hoc to paper Floor-aligned integer codes with scale refinement guarantee composability of slices (Eq. 3).
    The many-in-one representation assumes summing independently quantized residual slices recovers the target-precision quantizer; only a bounded-error argument is given for the two-slice case.
  • domain assumption A 2-layer MLP router can predict per-token quantization difficulty from hidden states.
    Eqs. 4 and 6 rely on router scores generalizing to new tokens and precisions; no proof or guarantee is provided.
  • domain assumption Outlier migration is a stable, precision-dependent property of LLM activations and the dominant cause of PTQ generalization failure.
    Motivation in Sec. 3 is based on visualizations (Fig. 2) without a quantitative metric; if wrong, token-adaptive precision may not be the right remedy.
  • domain assumption Calibration distribution (WikiText2) is representative enough for thresholds to transfer to evaluation distributions.
    Inference thresholds are fit on the calibration set (App. C.2); C4/PTB ablations show partial robustness but no deployment-shift guarantee.
invented entities (1)
  • Outlier migration no independent evidence
    purpose: Explains why static PTQ calibration parameters fail across bit-widths and motivates token-adaptive bit selection.
    Introduced as a named phenomenon; supported only by in-paper error-distribution plots (Fig. 2 and Fig. 5) with no quantitative definition or external verification.

pith-pipeline@v1.3.0-alltime-deepseek · 17732 in / 18392 out tokens · 165870 ms · 2026-08-02T21:47:39.616728+00:00 · methodology

0 comments
read the original abstract

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent \textit{outlier migration} phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose \texttt{MoBiQuant}, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates \textit{outlier migration} with a token-aware router to dynamically select the optimal inference precision of each token.Extensive experiments show that \texttt{MoBiQuant} matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods.

Figures

Figures reproduced from arXiv: 2602.20191 by Denis Gudovskiy, Dongwei Wang, Huanrui Yang, Jinhee Kim, Jong Hwan Ko, Kang Eun Jeon, KhayTze Peong, Seokho Han, Tomoyuki Okuno, Yiran Chen, Yohei Nakata.

Figure 1
Figure 1. Figure 1: Precision generalization performance on LLaMA3- 8B. Existing PTQ methods poorly generalize under calibra￾tion–inference precision mismatch, e.g., calibrating for 3-bit but in￾ferring at 4-bit (green). Incorporating token-aware bit adjustment partially mitigates this degradation (pink). Our elastic MoBiQuant improves generalization achieving performance parity with the 4-bit calibrated model (orange). Well-… view at source ↗
Figure 2
Figure 2. Figure 2: Per-token quantization error distribution at layer 5 in LLaMA3-8B: quantization outliers are highly non-uniform at each bit-width. Tokens well-fitted at 4-bit can become outliers at 3-bit, while 4-bit outliers may not be the primary error source at 3-bit. LLMs without changing the original weights. Typically, PTQ involves a calibration phase, during which an addi￾tional set of quantization parameters Θq is… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of MoBiQuant. (a) MoBiQuant enables token-adaptive elastic inference for linear blocks in LLMs, consisting of two components: (b) MoBiSlice, which decomposes FP16 weights into bit slices via recursive quantization, providing multiple precisions; and (c) MoBiRoute, which is trained to route each token to the optimal bit slice under a budget, achieving token-adaptive precision adjustment. 4. The MoB… view at source ↗
Figure 4
Figure 4. Figure 4: Elastic-inference performance comparison. MoBiQuant outperforms PTQ baselines across multiple unseen target bit-widths on various LLaMA-family models, maintaining smooth and fine-grained precision scaling even at extremely low 2–3-bit precision range [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average precision of different MoBiQuant linear blocks, showing block-level variation and higher precisions in initial lay￾ers. tion shifts consistently with the target inference bit-width. Specifically, with the 3-bit inference, most tokens are as￾signed to 2- or 4-bit precisions, whereas for the 5-bit infer￾ence, the majority of tokens shift toward the 4–6-bit range. Overall, the average precision remain… view at source ↗
Figure 5
Figure 5. Figure 5: Per-token quantization error distribution for MoBiQuant at layer 5 in LLaMA3-8B: MoBiQuant shows largely reduced outlier migration phenomenon and, hence, improves generalization for elastic precision switching. Token-wise precision distributions. Our method enables each token to adaptively select its optimal bit-width. We collect statistics on token bit assignments for elastic infer￾ence across different t… view at source ↗
Figure 8
Figure 8. Figure 8: Overview of MoBiQuant kernel implementation: binary matrix multiplication (BMMA) using parallel stream-based execu￾tion for bit-major packed slices with permutations [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Inference speedups for MoBiQuant over the FP16 base￾line on LLaMA-2-7B. Evaluated on an NVIDIA A100 GPU. icantly improve memory coalescing and cache locality. Stream-based bit slice overlap. The activation frequency of bit slices in MoBiQuant is inherently skewed, as the most significant bits (MSBs) are invoked more frequently than other slices. Sequential execution of these slices often leads to hardware … view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of PPL (↓) sweep between different scheduling methods across WikiText2, C4, PTB datasets. We adopt Logarithmic scheduling for our regularization scheme because it converges to its intended target precision during inference, and also aligns naturally with the router temperature annealing used in the router gating mechanism. The router temperature is annealed following a logarithmic trajectory, t… view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of PPL (↓) sweep between different target bit-widths for training across WikiText2, C4, PTB datasets. E. Additional Experiments E.1. Elasticity with Activation Quantization [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: PPL (↓) tradeoff under 4 bit activation quantization on LLaMA2-13B. We compare with SmoothQuant (Xiao et al., 2023) and OmniQuant (Shao et al., 2024) W4A4 baselines. The elasticity of MoBiQuant also extends to activation quantization, as shown in [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 9 linked inside Pith

  1. [7]

    C., Jeon, K

    Kim, J., Yoon, S., Lee, T., Lee, J. C., Jeon, K. E., and Ko, J. H. Truncquant: Truncation-ready quantization for dnns with flexible weight bit precision.arXiv preprint arXiv:2506.11431,

    Pith/arXiv arXiv

  2. [8]

    and Patterson, D

    Ma, X. and Patterson, D. Challenges and research di- rections for large language model inference hardware. arXiv:2601.05047,

    arXiv

  3. [10]

    J., and Lee, D

    Park, G., Bae, J., Kwon, B., Kim, B., Kwon, S. J., and Lee, D. AnyBCQ: Hardware efficient flexible binary-coded quantization for multi-precision LLMs. arXiv:2510.10467,

    arXiv

  4. [11]

    Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models.arXiv:2307.09288,

    Pith/arXiv arXiv

  5. [12]

    Overall Algorithm of MoBiQuant

    10 MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs Appendix Contents A. Overall Algorithm of MoBiQuant .......................................................................................................... 11 B. MoBiSlice Construction: Formulation and Further Analysis................................................................

    2025

  6. [13]

    LET preserves the main linear path by transforming the input activation and compensating the linear weights so that the layer output remains equivalent

    and router training. LET preserves the main linear path by transforming the input activation and compensating the linear weights so that the layer output remains equivalent. For a linear layer, Y=XW+B, ˜X= (X−δ)⊘s, ˜W=s⊙W, ˜B=B+δW,(22) where X∈R T×d denote the input activations with T tokens and d input channels; W∈R d×m the weight matrix; and B∈R m the b...

    arXiv 2024

  7. [2019]

    Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv:1803.05457,

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC, the AI2 reasoning chal- lenge.arXiv:1803.05457,

    Pith/arXiv arXiv

  8. [2021]

    FGMP: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated LLM inference.arXiv:2504.14152,

    Hooper, C., Sakr, C., Keller, B., Venkatesan, R., Keutzer, K., Shao, S., and Khailany, B. FGMP: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated LLM inference.arXiv:2504.14152,

    Pith/arXiv arXiv

  9. [2022]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv:2210.17323,

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv:2210.17323,

    Pith/arXiv arXiv

  10. [2023]

    PrefixQuant: Static quantization beats dynamic through prefixed outliers in LLMs.arXiv:2410.05265,

    Chen, M., Liu, Y ., Wang, J., Bin, Y ., Shao, W., and Luo, P. PrefixQuant: Static quantization beats dynamic through prefixed outliers in LLMs.arXiv:2410.05265,

    Pith/arXiv arXiv

  11. [2024]

    W., and Keutzer, K

    Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference.arXiv:2103.13630,

    Pith/arXiv arXiv

  12. [2025]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv:2401.04088,

    Pith/arXiv arXiv

  13. [2026]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    9 MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Pith/arXiv arXiv