Pith. sign in

REVIEW 3 major objections 6 minor 24 references

The paper claims that fractional bit-widths assigned by a Fisher-salience quantile eliminate combinatorial search in mixed-precision LLM quantization, giving higher accuracy per byte and faster inference than optimization-based allocation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:47 UTC pith:DWHDEG65

load-bearing objection Search-free fractional block allocation is a genuine and useful idea, but a group-size confound weakens the extreme-low-bit comparisons; the core results at 2.5–3.5 BPW look credible. the 3 major comments →

arxiv 2602.01027 v2 pith:DWHDEG65 submitted 2026-02-01 cs.LG

SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models

Xin Nie , Haicheng Zhang , Liang Dong , Beining Feng , Jinhong Weng , Guiling Sun This is my paper
classification cs.LG
keywords mixed-precision quantizationlarge language modelsfractional bit-widthFisher informationblock-wise bit allocationrow-column reorderingLUT-based GEMMhardware-friendly inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mixed-precision quantization is normally an expensive combinatorial search: pick an integer bit-width per layer to fit a memory budget. SFMP dissolves the search by changing both the unit and the rule: the unit becomes a two-dimensional block of weights, and the rule is a single threshold on Fisher-information salience, with the top alpha-quantile of blocks getting the ceiling bit-width and the rest the floor bit-width. The paper reports that this one-pass allocation matches or exceeds optimizer-based allocations under the same memory, and cuts the configuration time for a 70-billion-parameter model from about 44 hours to 9 minutes. A reader should care because it removes the two barriers to practical low-bit LLM deployment: the hours of search, and the irregular memory formats that slow inference.

Core claim

SFMP claims that the NP-hard discrete optimization of layer-wise bit allocation collapses to a one-line rule once bit-width is allowed to be fractional and the allocation unit is a block. First, estimate per-weight global salience from a small calibration set using diagonal Fisher information. Reorder each weight matrix's rows and columns so salient weights cluster, partition into fixed-size blocks, sum salience per block, then assign ceil(b) bits to every block whose salience sum is at or above the alpha-quantile and floor(b) bits to all others, where alpha is the fractional part of the target average bits-per-weight. No search, no solver, no hyperparameters. The paper reports that on an 8-

What carries the argument

The load-bearing object is a salience threshold: bit(Block) = ceil(b) if Sal(Block) >= tau_alpha, else floor(b), where tau_alpha is the alpha-quantile of block salience sums and b is the target average bits per weight (Eq. 10). Three mechanisms carry it. Fractional bit-width restricts candidates to {floor(b), ceil(b)}, fixing the fraction of high-bit blocks to b - floor(b) and turning a discrete allocation into a continuous problem. Row-column reordering permutes each weight matrix by descending row- and column-salience sums, aggregating scattered salient weights into block-shaped clusters with only a small activation-reordering cost at runtime. A unified GEMM kernel then decomposes each blo

Load-bearing premise

The method's bit allocation is only as good as its salience measure: it assumes that the sum of diagonal Fisher information over a block, estimated from about a thousand calibration samples, correctly ranks which blocks hurt the loss most when quantized, so that simply taking the top alpha-quantile of blocks is a near-optimal allocation.

What would settle it

Take a small transformer (around 1-billion parameters) and compute the true loss increase for every possible block-level bit assignment on a validation set - or at least compare the top-alpha blocks selected by diagonal Fisher salience with the top-alpha blocks selected by an exact Hessian-vector-product loss estimate. If the two sets differ substantially, or if an exhaustive search finds a bit allocation that beats SFMP's threshold allocation by more than a point or two at the same memory budget, the claim that a single salience quantile replaces optimization would be refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At a fixed memory budget, bit allocation becomes a one-pass computation, cutting configuration time for a 70-billion-parameter model from tens of hours to minutes.
  • Inference latency decreases as average bit-width decreases, because the one-bit LUT-based kernel's cost scales linearly with bits per weight and removes runtime dequantization, unlike conventional low-bit kernels.
  • Extreme low-precision budgets (around 2.25-2.5 bits per weight) remain usable, with the paper reporting roughly six percentage points higher average zero-shot accuracy than an optimization-based baseline at the same memory on an 8-billion-parameter model.
  • Because the allocation is parameter-free and requires no solver, the procedure applies to any model scale without exponential search, making mixed precision practical for very large models.
  • The block-major layout and unified kernel mean a single implementation serves every bit-width, avoiding precision-specific branching in deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the threshold rule is robust across models, bit allocation becomes a closed-form function of the memory budget, so the full accuracy-versus-memory curve of a model could be swept in minutes, turning quantization from an offline preparation stage into an interactive deployment knob.
  • The row-column reordering step could be chained with input or activation rotations that homogenize weight magnitudes, concentrating salience further and potentially pushing the viable bit-width below 2.25 - a combination the paper does not explore.
  • The ablation showing that group size affects accuracy more than block height suggests an adaptive per-block group-size rule, chosen by the same salience criterion, could recover additional accuracy under a fixed memory budget; the paper explicitly leaves this as future work.
  • The two-bit-candidate restriction is a simplification; the same quantile allocation logic extends naturally to non-adjacent candidate sets where hardware can express them, trading granularity for a wider dynamic range.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SFMP, a search-free mixed-precision quantization framework for LLMs. The method defines a fractional bit-width per weight matrix, restricts candidate precisions to {floor(b), ceil(b)}, and allocates ceil(b) to the most salient blocks, where salience is the sum of diagonal Fisher information over a block (Eq. 10). Row/column reordering aggregates salient weights into blocks, and a one-bit LUT-based GEMM kernel enables unified, hardware-friendly execution at arbitrary average bit-width. Experiments on LLaMA3.1 8B/70B and Qwen3 8B/14B/32B report gains over AMQ and BitStack at BPW 2.5–4.0, over GPTQ/AWQ/SliM-LLM at 2.25–4.0, large reductions in configuration search cost (e.g., 44h to 0.15h for 70B), and improved inference throughput on several GPUs.

Significance. If the results hold, SFMP is a practically significant contribution: it replaces expensive ILP/heuristic bit-allocation search with a deterministic, salience-based rule, and it packages the result in a block-wise format with a unified kernel that is substantially faster than dequantization-based kernels. The paper is unusually transparent about ablations, hyperparameters, and limitations, and it ships code. The central allocation rule is simple, reproducible, and grounded in a standard Fisher-information approximation. The strongest evidence is the clean comparison against AMQ at matched group size 128 in Table 1, where SFMP improves average zero-shot accuracy by 5–6 points at BPW 2.5 and also improves perplexity on several settings. The main weakness is that some fixed-precision comparisons are confounded by group size, and the reported memory numbers are not transparently reconciled with BPW.

major comments (3)
  1. [§5.1, Tables 3 and 12; Eq. (10); Appendix H.1.2, Table 6] At BPW 2.25 and 3.25, SFMP uses group size 256 while all baselines (GPTQ, AWQ, SliM-LLM) are evaluated at group size 128. Group size directly changes the split between weight bits and scale/zero-point overhead. The paper's own ablation, Table 6, shows this choice is decisive: for LLaMA3.1-8B at BPW 2.25, changing nb from 128 to 256 moves average zero-shot accuracy from 37.42% to 57.69%; at BPW 3.25 it moves from 70.77% to 72.35%. Therefore the large gains in Table 3 (e.g., 57.69 vs. 40.67 for SliM-LLM at 2.25) are substantially attributable to group-size selection, not to the Fisher-salience allocation rule. The sentence in §5.1 that 'by setting the group size to 256, SFMP enables 12.5% of the weights to be quantized at 3-bit' confirms this mechanism. Because Appendix H.1.2 explicitly identifies group size as critical and leaves adaptive selection to future work, the current tables do no
  2. [§5.1, Tables 1/3; Fig. 1] The reported memory values in Tables 1 and 3 do not reconcile with BPW, group size, and FP16 model size. For LLaMA3.1-8B at BPW 2.5, g128, Table 1 reports 4,085 MB; a straightforward calculation from the FP16 memory (15,317 MB) and 2.5 bits/weight gives about 2,400 MB plus scale/zero-point overhead, leaving a gap of roughly 1.5 GB. The gap is much smaller for the 70B model. This matters because the central claim is 'under the same memory constraints'—if the memory numbers are not calculated consistently across methods, the tables cannot be independently verified. Please specify exactly what is included in 'Mem.' (all parameters including embeddings and norms? scale/zero-point? KV cache?) and provide the formula used to compute BPW and memory footprint.
  3. [§4.6; §5.2; Appendix H] The sentence 'The entire pipeline is parameter-free and requires no search' overstates the case. The method has at least four user-set hyperparameters: block size mb, group size nb, Fisher sample size, and reorder on/off. Section 5.2 even recommends disabling reordering at higher BPW, and Appendix H.1.2 shows the optimal nb varies with BPW. The method is search-free in the sense that it does not run a per-model numerical optimizer, but the configuration is not parameter-free and the authors did perform offline tuning (ablations) to select these values. Please rephrase to 'requires no per-model search' and report the sensitivity of the main claims to these choices in the main text, not only in the appendix.
minor comments (6)
  1. [Eq. (14)] The notation (P^T_{col})^{-1} is confusing; since permutation matrices are orthogonal, it equals P_col. Please simplify and clarify that the activation reordering is applied once per layer.
  2. [Table 3 caption] The note 'Memory overhead from extra quantization parameters in GPTQ and AWQ at w3, w4 is omitted as it is negligible' is inaccurate at low group sizes such as g128, where the overhead is 0.25 bits/weight and is exactly the reason the BPW numbers are what they are. Please state how overhead is handled for all rows.
  3. [§4.2] The text cites 'the empirical study from CMPQ (Zihan et al., 2025)' but the reference entry in the bibliography is missing the title/venue; it appears only as 'Channel-wise mixed-precision quantization for large language models, 2025.' Please complete the reference.
  4. [Fig. 10 caption] The caption says 'The numbers on the left indicate the BPW per configuration,' but the left axis appears to be a layer/block index. Please clarify.
  5. [Table 4] The 'Cost (h)' for SFMP (0.05h for 8B, 0.15h for 70B) should state what is included: Fisher estimation, quantization, and perhaps reordering. It would be helpful to separate Fisher estimation time from the rest.
  6. [§5.2, reordering discussion] The statement that reordering gains diminish as BPW increases is supported by Fig. 7, but the recommendation 'at higher BPW, we recommend disabling reordering' appears to contradict the main results in Table 1, where reordering is enabled at BPW 4.0. Please clarify whether the reported Table 1 numbers use reordering at all BPWs and whether disabling it changes the headline numbers.

Circularity Check

0 steps flagged

No circularity: bit allocation is a deterministic closed-form function of Fisher salience; the group-size mismatch is a benchmark-confounding issue, not a derivation-level circular step.

full rationale

I inspected the paper's derivation chain: Fisher-diagonal salience estimation (Eq. 2, Appendix C), block salience summation (Eq. 9), quantile-threshold allocation (Eq. 10), row-column reordering (Eqs. 11-13), and the unified one-bit LUT GEMM (Eq. 15). The allocation rule is fully determined by the average bit budget b and the salience scores; no measured accuracy or perplexity values are fed back into the method, and no fitted parameter is later relabeled as a prediction. The restriction to two candidate bit-widths {floor(b), ceil(b)} is justified by an external empirical observation from CMPQ, not by the paper's own results. The only self-referential elements are hyperparameter choices (block/group size nb, Fisher sample size, mb) chosen after ablations; this is standard tuning, not derivation-level circularity. The group-size mismatch in Tables 3/12 at BPW 2.25/3.25 (SFMP g256 vs baselines g128) is a legitimate controlled-comparison concern, but it is not circularity: the paper explicitly attributes the improvement to the larger group size and quantifies the effect in its own ablation (Table 6), so the cause is disclosed rather than disguised as a prediction. The authors also do not rely on their own prior work; citations to SqueezeLLM, AMQ, and CMPQ are external and non-load-bearing in a circularity sense. No specific reduction of a claimed output to its own inputs can be exhibited, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

SFMP introduces no new physical or mathematical entities. Its free parameters are block size, group size, Fisher sample size, and reorder policy, all chosen by ablation rather than derived. The main unproven assumptions are the diagonal-Fisher salience proxy and the adequacy of two adjacent bit-widths.

free parameters (4)
  • Block size (mb, nb) = (512, 128) default; (512, 256) at BPW 2.25/3.25
    Set by hand via ablation (Appendix H.1); nb strongly and non-monotonically affects accuracy, so it is effectively tuned on the evaluation benchmarks.
  • Fisher estimation sample size = 1024
    Chosen after ablation (Table 7); 128–2048 samples all give competitive results, but 1024 is the selected operating point.
  • LUT group size = 8
    Hardware choice for 8-element lookup tables; not derived from accuracy considerations.
  • Reorder on/off = on at low BPW, off recommended at BPW=4
    Ablation shows reordering has near-zero benefit at 4-bit; the paper recommends disabling it then, a post-hoc setting.
axioms (6)
  • domain assumption The diagonal Fisher Information matrix approximates the Hessian; cross-weight interactions vanish.
    Invoked in Eq. (2) and Appendix C, Eqs. (19)–(22). Standard in pruning but an approximation that can fail when weights interact.
  • domain assumption Block salience is the sum of per-weight Fisher diagonals.
    Used in Eq. (9); assumes no interaction between weights within a block.
  • ad hoc to paper Restricting candidate precisions to {floor(b), ceil(b)} is near-optimal.
    Section 4.2 justifies this by an empirical insight from CMPQ, not by derivation; it shapes the whole method.
  • domain assumption Row/column reordering aggregates salient weights into compact blocks.
    Section 4.4 and Appendix E present empirical scatter plots; if salience does not become block-clustered after sorting, the allocation loses its accuracy benefit.
  • standard math The one-bit decomposition X W = sum_i 2^i X W_i is exact.
    Appendix B, Eq. (15); this is exact for integer weights once scales and zero-points are handled.
  • domain assumption Gradient statistics on 1k C4 samples produce reliable global salience.
    Section 5 and Appendix H.2; sample-size ablation shows robustness, but this is still an empirical assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 30483 in / 13180 out tokens · 144510 ms · 2026-08-03T05:47:29.526333+00:00 · methodology

0 comments
read the original abstract

Mixed-precision quantization is a promising approach for compressing large language models under tight memory budgets. However, existing mixed-precision methods typically suffer from one of two limitations: they either rely on expensive discrete optimization to determine precision allocation, or introduce hardware inefficiencies due to irregular memory layouts. We propose SFMP, a search-free and hardware-friendly mixed-precision quantization framework for large language models. The framework is built upon four novel ideas: Fractional bit-width, which extends integer bit-width for weight matrix to fractional value and transforms discrete precision allocation as a continuous problem; 2)Block-wise mixed-precision, enabling fine-grained precision within weight matrices while remaining hardware-friendly; 3)Row-column weight reordering, which aggregates salient weights via row and column reordering, incurring only a small activation reordering overhead during inference; 4)Unified GEMM kernel, which supports mixed-precision GEMM at arbitrary average bit-width. Extensive experiments demonstrate that SFMP outperforms state-of-the-art layer-wise mixed-precision methods under the same memory constraints, while significantly reducing quantization cost and improving inference efficiency. Code is available at https://github.com/Nkniexin/SFMP

Figures

Figures reproduced from arXiv: 2602.01027 by Beining Feng, Guiling Sun, Haicheng Zhang, Jinhong Weng, Liang Dong, Xin Nie.

Figure 1
Figure 1. Figure 1: Trade-off between memory usage and average zero￾shot accuracy on ARC-Easy, ARC-Challenge, PIQA, HellaSwag, WinoGrande, and BoolQ. LLMs in memory-constrained edge scenarios (Zhang et al., 2024; Hosseinzadeh & Khamfroush, 2025; Husom et al., 2025). Existing methods (Frantar et al., 2023; Xiao et al., 2023; Lin et al., 2024; Kim et al., 2024; Liu et al., 2025b) achieve near-lossless compression at 8-bit preci… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of two GEMM computation paradigms: Left) dequant-based GEMM; Right) one-bit LUT-based GEMM. (Huang et al., 2024) observes that salience often concen￾trates along specific rows or columns and reduces quanti￾zation error through column-wise partitioning. Similarly, Slim-LLM (Huang et al., 2025) exploits row-wise salience by introducing group-wise mixed-precision quantization, achieving improved ac… view at source ↗
Figure 3
Figure 3. Figure 3: Motivation of SFMP. Hardware Inefficiency of Element-, Group-, and Channel-Wise Quantization. Despite their accuracy ben￾efits, fine-grained mixed-precision schemes are often mis￾aligned with efficient hardware execution. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Pipeline of SFMP. by 2-bit weights”. Inspired by this observation, we limit the candidate bit-width set to two values, with the difference between them not exceeding 1. Therefore, for a given aver￾age bit b, the candidate quantization bit-widths are uniquely determined as: B = {⌊b⌋, ⌈b⌉}, (5) where ⌊·⌋ denotes the floor function and ⌈·⌉ denotes the ceiling function. The bit allocation problem now simplifie… view at source ↗
Figure 5
Figure 5. Figure 5: An example of unified mixed-precision GEMM. rather than forming spatially contiguous block-level clusters. Consequently, block-wise bit allocation is spatially mis￾aligned with the salience patterns. Therefore, prior to block￾wise bit allocation, we reorder the weight matrix based on row and column salience to achieve block-level aggregation of salient weights. Given the salience matrix Sl ∈ R m×n, we comp… view at source ↗
Figure 6
Figure 6. Figure 6: End-to-end throughput (tokens/s) of generating a sequence length of 128 with batchsize of 1. FP16 inference of LLaMA3.1 70B is not feasible on single A100 and H100 due to memory constraints [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of bit allocation over linear layers with different BPWs at Llama3.1 8B. The numbers on the left indicate the BPW per configuration. LLaMA3.1-70B. SliM-LLM can be executed with fewer GPUs, but configuring LLaMA3.1-70B still takes 8 hours. In contrast, SFMP directly assigns bit-widths by global salience. Its only cost is estimating the diagonal values of the Fisher Information Matrix using a … view at source ↗
Figure 8
Figure 8. Figure 8: Latency comparison of our unified mixed-precision ker￾nel , uniform quantization kernel from GPTQModel and cuBLAS FP16 kernel on A100. 2.25 2.5 3 3.5 4 0 50 100 150 200 Tokens / s 168 162 150 139 130 177 170 157 146 136 RTX3090 / LLaMA3.1-8B 2.25 2.5 3 3.5 4 0 10 20 30 40 50 60 52 49 43 38 35 55 52 45 40 36 A100 / LLaMA3.1-70B BPW Reorder No Reorder [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Throughput (tokens / s) comparison for end-to-end generation of 128 tokens with and without reordering. more pronounced for smaller models, whose inference is less dominated by GEMM operations compared to larger models. Moreover, the overhead of reordering consistently diminishes as BPW increases, since the growing cost of one-bit LUT-based GEMM amortizes the fixed reordering overhead. Search cost [PITH_F… view at source ↗
Figure 11
Figure 11. Figure 11: Detailed computation procedure of one-bit Lut-based GEMM. B. Details about One-Bit Lut-Based GEMM [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Inference throughput (tokens / s) comparison between SliM-LLM and GPTQ when generating 128 tokens with batch size 1. FP16 inference of LLaMA3.1 70B is not feasible on single A100 and H100 due to memory constraints. BPW denotes “bits per weight” 10th 30th 20th q gate down high low k v o up [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Weight salience distribution in the 10th , 20th , 30th layers of LLaMA3.1 8B D. Empirical Study on Inference Speed of Group-wise Mixed-Precision Methods We present an empirical study that compares the inference throughput of the group-wise mixed-precision method SliM-LLM (Huang et al., 2025) and the uniform quantization method GPTQ (Frantar et al., 2023). For SliM-LLM, we use the official released code, w… view at source ↗
Figure 14
Figure 14. Figure 14: CUDA implementation. H. Additional Ablation Analysis H.1. Impact of Block Size We study the impact of block size (mb, nb) on model accuracy. H.1.1. EFFECT OF mb. As shown in [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Visualization of bit allocation over linear layers with different BPWs at Llama3.1 70B. The numbers on the left indicate the BPW per configuration. L. More Results of Bit Allocation Visualizations [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Visualization of bit allocation over linear layers with different BPWs at Qwen3 8B. The numbers on the left indicate the BPW per configuration. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Visualization of bit allocation over linear layers with different BPWs at Qwen3 32B. The numbers on the left indicate the BPW per configuration [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 11 linked inside Pith

  1. [3]

    This decomposition is performed offline, incurring no runtime overhead

    with binary representations (1001, 0111, 0110, 0011), the vector for the lowest bit is (1, 1, 0, 1), and the vector for the highest bit is (1, 0, 0, 0). This decomposition is performed offline, incurring no runtime overhead. During inference, for an activation vector of the group size g, the operator precomputes the dot products between this activation ve...

    2025

  2. [10]

    FlexQuant: A flexible and effi- cient dynamic precision switching framework for LLM quantization

    Liu, F., Wang, Z., Xia, J., Zhao, J., Zhao, S., Li, J., Liu, J., Jiang, L., and Guan, H. FlexQuant: A flexible and effi- cient dynamic precision switching framework for LLM quantization. InFindings of the Association for Com- putational Linguistics: EMNLP 2025, pp. 4152–4161, November 2025a. Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- is...

    2025

  3. [11]

    J., and Lee, D

    Park, G., Bae, J., Kwon, B., Kim, B., Kwon, S. J., and Lee, D. Anybcq: Hardware efficient flexible binary-coded quantization for multi-precision llms.arXiv preprint arXiv:2510.10467, 2025a. Park, G., Kwon, H., Kim, J., Bae, J., Park, B., Lee, D., and Lee, Y . Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables. In2025 IEEE ...

    arXiv

  4. [12]

    and Savakis, A

    Ranjan, N. and Savakis, A. Lrp-qvit: Mixed-precision vision transformer quantization via layer-wise relevance propagation.arXiv preprint arXiv:2401.11243,

    Pith/arXiv arXiv

  5. [13]

    and Savakis, A

    Ranjan, N. and Savakis, A. Mix-qvit: Mixed-precision vision transformer quantization driven by layer im- portance and quantization sensitivity.arXiv preprint arXiv:2501.06357,

    Pith/arXiv arXiv

  6. [14]

    Outliertune: Efficient channel-wise quantization for large language models.arXiv preprint arXiv:2406.18832,

    Wang, J., Yin, Y ., Sun, H., Qi, Q., Wang, J., Zhuang, Z., Yang, T., and Liao, J. Outliertune: Efficient channel-wise quantization for large language models.arXiv preprint arXiv:2406.18832,

    Pith/arXiv arXiv

  7. [16]

    Additional Related Works A.1

    11 SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models Appendix Appendix Overview Appendix A: Additional Related Works Appendix B: Details about One-Bit Lut-Based GEMM Appendix C: Fisher-Information-Based Global Salience of Weight Appendix D: Empirical Study on Inference Speed of Group-wise Mixed-Pr...

    2023

  8. [19]

    13 SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models C

    leveraged CPU vectorized lookup instructions (A VX2/NEON) to enable efficient LUT operations on CPUs. 13 SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Models C. Fisher-Information-Based Global Salience of Weight The objective of quantization is to approximate the original full-precision weight W with...

    2024

  9. [20]

    For SliM-LLM, we use the official released code, while GPTQ is evaluated using GPTQModel

    and the uniform quantization method GPTQ (Frantar et al., 2023). For SliM-LLM, we use the official released code, while GPTQ is evaluated using GPTQModel

    2023

  10. [24]

    Memory overhead from extra quantization parameters in GPTQ and AWQ at w3, w4 is omitted as it is negligible. ModelMem.(MB)BPW Method Wiki2(↓) C4(↓)HellaS.(↑) WinoG.(↑) ARC-e(↑) ARC-c(↑) PIQA(↑) BoolQ(↑) Avg.(↑) 8B 15,623 16 FP16 9.73 13.30 74.93 68.66 80.85 56.65 77.47 86.64 74.20 4,238 2.25 GPTQw2g128 39.79 35.90 38.60 49.88 30.85 24.65 54.62 44.86 40.58...

    arXiv

  11. [128]

    The numbers on the left indicate the BPW per configuration

    BPW = 2.5 BPW = 3.0 Layer q k v o gate up down q k v o gate up down 0 2.01 2.04 2.66 2.96 2.12 2.29 2.70 2.03 2.08 2.72 3.00 2.74 2.95 3.00 1 2.04 2.07 3.00 2.90 2.17 2.56 2.96 2.08 2.13 3.00 3.00 2.99 3.00 3.00 2 2.11 2.13 3.00 2.67 2.26 2.72 2.83 2.22 2.25 3.00 2.91 3.00 3.00 3.00 3 2.13 2.17 3.00 2.97 2.15 2.68 2.76 2.32 2.45 3.00 3.00 2.98 3.00 3.00 4...

    arXiv

  12. [256]

    Memory overhead from extra quantization parameters in GPTQ and AWQ at w3, w4 is omitted as it is negligible. ModelMem.(MB)BPW Method Wiki2(↓) C4(↓)HellaS.(↑) WinoG.(↑) ARC-e(↑) ARC-c(↑) PIQA(↑) BoolQ(↑) Avg.(↑) 8B 15,317 16 FP16 6.15 8.89 78.99 72.93 81.19 53.41 81.39 82.15 75.01 3,877 2.25 GPTQw2g128 232 165 29.27 50.74 28.41 23.21 53.75 45.96 38.56AWQw2...

    arXiv

  13. [512]

    Impact of Sample Size for Fisher Estimation Table 7 reports the impact of the sample size used for Fisher information estimation on model performance

    Model BPW nb=64n b=128n b=256n b=512 LLaMA3.1 8B 2.25 (4894, 36.79) (2520, 37.42)(28.61, 57.69)(43.56, 57.39) 2.50 (737, 40.90) (14.49, 64.34)(14.05, 64.96)(17.93, 62.94) 3.00 (10.12, 68.40)(9.51, 69.74)(9.55, 69.51) (9.53, 68.58) 3.25 (8.83, 69.99) (8.41, 70.77)(7.60, 72.35)(7.60, 72.62) 3.50 (7.98, 71.76)(7.19, 72.97)(7.21, 73.26) (7.27, 73.25) LLaMA3.1...

    arXiv 2048

  14. [1983]

    Amq: Enabling automl for mixed-precision weight-only quan- tization of large language models

    Lee, S., Woo, S.-t., Jin, J.-g., Lee, C., and Park, E. Amq: Enabling automl for mixed-precision weight-only quan- tization of large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 35520–35538,

    2025

  15. [1997]

    Fgmp: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated llm inference.arXiv preprint arXiv:2504.14152,

    Hooper, C., Sakr, C., Keller, B., Venkatesan, R., Keutzer, K., Shao, S., and Khailany, B. Fgmp: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated llm inference.arXiv preprint arXiv:2504.14152,

    Pith/arXiv arXiv

  16. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  17. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  18. [2020]

    Efficientqat: Efficient quantization- aware training for large language models.arXiv preprint arXiv:2407.11062,

    Chen, M., Shao, W., Xu, P., Wang, J., Gao, P., Zhang, K., and Luo, P. Efficientqat: Efficient quantization- aware training for large language models.arXiv preprint arXiv:2407.11062,

    Pith/arXiv arXiv

  19. [2021]

    Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  20. [2022]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pp. 2924–2936, June

    2019

  21. [2023]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  22. [2024]

    Sign- roundv2: Closing the performance gap in extremely low- bit post-training quantization for llms.arXiv preprint arXiv:2512.04746,

    Cheng, W., Zhang, W., Guo, H., and Shen, H. Sign- roundv2: Closing the performance gap in extremely low- bit post-training quantization for llms.arXiv preprint arXiv:2512.04746,

    Pith/arXiv arXiv

  23. [2025]

    and Khamfroush, H

    Hosseinzadeh, M. and Khamfroush, H. Dilemma: Joint llm quantization and distributed llm inference over edge computing systems.arXiv preprint arXiv:2503.01704,

    Pith/arXiv arXiv

  24. [2560]

    Most approaches (Cheng et al., 2025; You et al.,

    To obtain acceptable solutions within a reasonable time, existing mixed-precision methods rely on heuristic strategies to reduce the search space. Most approaches (Cheng et al., 2025; You et al.,

    2025