Pith. sign in

REVIEW 4 major objections 6 minor 23 references

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GANQ makes 3- and 4-bit weights rival full precision by giving each channel its own codebook, selected to minimize layer output error rather than rounding error.

desk verdict GANQ is a solid, incremental quantization paper with a genuinely clean MIQP formulation and a fast GPU-parallel alternating solver, but the greedy S-update is an unquantified heuristic and the initial codebook is unspecified. read the letter →

arxiv 2501.12956 v3 pith:MGHHH5LO submitted 2025-01-22 cs.LG cs.AImath.OC

Pengxiang Zhao , Xiaoming Yuan This is my paper · ORCID
classification cs.LGcs.AImath.OC
keywords post-trainingquantizationnon-uniformlookuptablemixed-integerquadraticprogrammingGPUaccelerationlargelanguagemodelsmixed-precisionGEMMweight-only
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GANQ argues that the right way to do post-training weight quantization is to let each output channel choose a small codebook of representable values and assign every weight to the nearest codebook entry in a way that minimizes the layer's output error, not just the weight error. The paper frames this as a mixed-integer quadratic program per layer and solves it with a training-free, GPU-parallel alternating algorithm, so the optimization itself is cheap enough to run on a single GPU. In experiments on OPT and LLaMA models, GANQ lowers 3- and 4-bit perplexity relative to RTN, GPTQ, and OmniQuant, and it can be combined with outlier-splitting methods for further gains. Because the quantized form is a lookup-table codebook, inference avoids dequantization and runs up to 2.57 times faster than the FP16 baseline on a single RTX 4090.

What carries the argument

The machinery is per-channel codebook quantization algebraically coupled to the input statistics. Define $H = XX^\top$, take its Cholesky factor $L$, and rewrite the row objective as $\|W_i L - T_i S_i L\|^2$; because $L$ is lower triangular, the squared error decomposes into $n$ independent terms ordered by column, and the assignment for column $j$ can be fixed by a nearest-codebook lookup after subtracting the accumulated residual from earlier columns. This back-substitution turns a combinatorial search into $O(n)$ nearest-neighbor steps per row, and the same $L$ is shared across all rows, so all rows update in parallel as batched matrix operations. The codebook $T$ then updates in closed form via a Moore-Penrose inverse of the small $2^N \times 2^N$ matrix $S H S^\top$.

What would settle it

A reader could settle the accuracy claim by running GANQ on a single small layer ($n \le 16$, 4-bit) and comparing the final objective $\|WX - T S X\|^2$ to the global optimum found by enumerating all $16^n$ assignments; a large gap on realistic weight rows would show the method rests on an unquantified heuristic. A simpler test is to permute the columns before quantization: the optimal objective is permutation-invariant, but the greedy right-to-left solver is not, so any measurable degradation would expose where the heuristic loses accuracy.

Watch

Extended reading notes

Core claim

The paper's central claim is that a principled optimization model for LUT-based non-uniform quantization—minimize $\|WX - T S X\|_F^2$ over a per-channel codebook $T$ and a one-hot assignment $S$—can be solved efficiently at LLM scale by exploiting row-wise separability. Each row's problem is a mixed-integer quadratic program; GANQ alternates between a closed-form least-squares update of the codebook $T$ and a greedy back-substitution update of the assignment $S$ along the Cholesky factor of $XX^\top$. The paper reports that this yields lower perplexity than uniform-quantization baselines at 3 and 4 bits, near-FP16 zero-shot accuracy on LLaMA-2-7B, and, when paired with existing lookup-table inference kernels, a 2.57$\times$ speedup over FP16 with peak memory under 4.1 GB on OPT-6.7B.

Load-bearing premise

The accuracy claim depends on the unproven assumption that the greedy right-to-left back-substitution in Eq. (22), which chooses each column's codebook entry while ignoring its effect on earlier columns, produces a discrete assignment close enough to the true optimum that the layer-wise objective stays low.

Editorial extensions

If this is right

  • At 3 and 4 bits, GANQ reports lower WikiText-2 perplexity than RTN, GPTQ, and OmniQuant on every OPT and LLaMA model tested, including cases where a quantized model matches or beats the FP16 baseline.
  • Because the per-channel codebook adds only $2^N m$ extra parameters, the storage overhead stays within a few tenths of a percent of uniform per-channel quantization, so the memory savings of low-bit weights are preserved.
  • GANQ is orthogonal to outlier splitting: retaining 0.5% outliers or 10 full rows lets GANQ$^\star$ beat the same baselines on OPT and LLaMA models while keeping a sparse component for extreme weights.
  • Using existing LUT-based kernels, GANQ-quantized OPT-6.7B achieves up to a 2.57$\times$ speedup over FP16 and peak memory of 4.10 GB at 3 bits, compared with 12.91 GB for FP16.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the row-separable formulation depends only on $XX^\top$, not on the weight values themselves, so the same solver could be applied to other matrix-multiplication-bound tensors such as KV caches or MoE expert weights without re-deriving the algorithm.
  • Beyond the paper: the greedy back-substitution is analogous to successive interference cancellation, so a cheap local-refinement pass (re-assigning columns whose residual remains large) could close part of the suboptimality gap the paper concedes; this is a testable extension, not a claim in the paper.
  • Beyond the paper: the reported speedup uses existing LUT kernels, but the algorithm emits exactly the codebook-plus-index format those kernels consume, so GANQ's inference advantage should grow as kernel-level LUT engineering improves without changing the quantization procedure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GANQ, a post-training weight-only quantization method for LLMs in which each layer's weights are represented by a per-channel codebook and a low-bit index matrix, enabling lookup-table-based mixed-precision matrix multiplication. The authors formulate the layer-wise quantization problem as a mixed-integer quadratic program and solve it by alternating between an exact closed-form update of the codebooks and a Cholesky-based greedy back-substitution update of the index assignments. Experiments on OPT, LLaMA, LLaMA-2, LLaMA-3, and LLaMA-3.2 models at 3 and 4 bits report perplexity improvements over RTN, GPTQ, OmniQuant, and SqueezeLLM in most configurations, plus zero-shot and long-context evaluations, and a reported up-to-2.57x inference speedup over FP16 on an RTX 4090.

Significance. If the central claims hold, GANQ is a useful contribution: it offers a principled optimization-based alternative to heuristic clustering for non-uniform LUT quantization, is training-free, parallelizes naturally across GPU rows, and is compatible with existing outlier-splitting techniques such as those used by SqueezeLLM. The authors also make their code publicly available and report a broad set of models, bit-widths, and tasks. The significance is conditional, however, because the discrete index-update step is a heuristic with no formal guarantee and no ablation, and because the empirical claim of consistent superiority is slightly stronger than the reported numbers support.

major comments (4)
  1. [Section 3.2, Eq. (22)] The greedy back-substitution that solves the S_i subproblem is explicitly described as producing a sub-optimal solution, yet the paper provides neither an approximation bound nor an ablation. This is a load-bearing step because the T-update in Eq. (7) is computed from the current S, so an uncontrolled error in the greedy S-update can propagate through the alternating iterations. I request either a formal error bound or an empirical study: on small layers where exhaustive search is feasible, compare the greedy solution with the optimal one; report the objective value and downstream perplexity for the greedy update versus k-means initialization, random restarts, or a few iterations of local search; and report sensitivity to the number of alternating iterations K.
  2. [Algorithm 1] The initial codebook T0 appears only as an input to the algorithm; the text never specifies how T0 is chosen. In a nonconvex alternating scheme, the final codebook and index matrix can depend on this initialization, so the current description is incomplete for reproducibility. Please specify the initialization (e.g., uniform grid over the weight range, k-means centroids, or percentile-based values) and report a sensitivity experiment over several plausible initializations.
  3. [Tables 2 and 5] The abstract and Section 4.2 state that GANQ consistently outperforms state-of-the-art methods, but Table 5 shows that GANQ* ties SqueezeLLM on LLaMA-2-7B at 4-bit (5.57 vs. 5.57) and leads by only 0.01 on LLaMA-7B at 4-bit (5.76 vs. 5.77). No standard deviations, repeated runs, or significance tests are reported. Please soften the claim to 'generally improves' or add repeated-run statistics and significance tests, especially for the configurations where the margins are within 0.01-0.02 in perplexity.
  4. [Section 4.4 and Appendix B] The method has two free hyperparameters that directly affect the reported results: the number of alternating iterations K (given as K=10 in Section 4.4) and the outlier extraction ratio r (default r=0.5% in Appendix B). The paper reports no ablation for either parameter. Since the performance of the greedy S-update and the outlier-splitting step both depend on these choices, a sensitivity analysis is needed to establish that the reported gains are not tied to a narrow parameter setting.
minor comments (6)
  1. [Section 2] The phrase 'wight-only quantization' appears in the Related Work; this should be 'weight-only quantization'.
  2. [Table 3] In the GPTQ 4-bit row, the WinoGrande and Arc-e entries appear concatenated as '57.7275.25'; the table should be reformatted to show separate values.
  3. [Figure 2] The back-substitution diagram is difficult to read in the provided version; please redraw it with larger fonts and clearer annotations.
  4. [Appendix C] The C4 perplexity results are reported after calibrating on C4 data, and PTB results are reported only for OPT models. While the text justifies the OPT-only PTB choice, a non-overlapping calibration/evaluation split or PTB numbers for LLaMA models would make the evaluation more convincing.
  5. [Appendix B, Algorithm 2] The outlier extraction uses floor and ceiling operations on percentile cutoffs, so the actual number of outliers per row may not equal r times the row length; please clarify how the reported ratio is enforced or state that it is approximate.
  6. [Availability] The GitHub URL in the paper contains a space ('https://github.com/Evans -Z/GANQ'); please correct it so the link resolves.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimization objective, evaluation metrics, and algorithmic steps are distinct and the central claims are externally checkable.

full rationale

GANQ's derivation chain is self-contained and does not reduce to its inputs. The optimization model in Eq. (1)-(2) minimizes layer-wise output reconstruction error on calibration inputs; the reported evaluation metrics (perplexity on WikiText-2, C4, PTB; zero-shot accuracy; LongBench; GSM8K) are not part of that objective, so no fitted-to-target loop exists. Even the C4 perplexity result, which uses the same dataset family as calibration, is not circular because the objective is reconstruction error, not perplexity, and WikiText-2 and PTB provide independent external checks. The T-update in Eq. (7) is a closed-form least-squares solution, and the S-update in Eq. (22) is an explicitly acknowledged 'sub-optimal' greedy back-substitution heuristic; this is an algorithmic approximation, not a hidden reuse of the claimed result. The paper makes no load-bearing self-citations, and the LUT inference kernels are attributed to prior work by Kim et al. (2024), not to the present authors. No uniqueness theorem, ansatz, or known result is smuggled in via self-citation. Accordingly, no circular step meets the evidence bar.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities. The method has four hand-chosen or unspecified hyperparameters (K, r, full rows, initial codebook) and relies on five background assumptions; the central benchmark results are empirical, not derived from first principles.

free parameters (4)
  • Number of alternating iterations K = 10
    Number of S/T updates; larger K costs more compute. No sensitivity study is reported, so the choice may affect perplexity claims.
  • Outlier extraction ratio r for GANQ* = 0.5%
    Hand-set extraction ratio used in Appendix B and Table 5; only one value and one '10 full rows' variant are tested, so sensitivity is unknown.
  • Full row count for GANQ* on LLaMA-7B/2-7B = 10 rows
    Retained to match SqueezeLLM's default, a comparison-driven choice that changes the quantized matrix.
  • Initial codebook T^0 = unspecified
    Algorithm 1 requires T^0 but the paper does not state how it is initialized; the final codebook may depend on it.
assumptions (5)
  • standard math Cholesky decomposition requires positive definiteness; when missing, diagonal dominance preconditioning preserves the intended factorization.
    Used in Eq. (10) and Algorithm 1; the paper supplies a standard fix in Appendix A, so this is a routine background assumption.
  • domain assumption Layer-wise output reconstruction error on calibration inputs is a faithful proxy for end-to-end model quality.
    Eq. (1) defines the objective; the paper never proves a bound linking this objective to perplexity or zero-shot accuracy.
  • domain assumption Calibration sequences from the first C4 shard are representative of the test distributions, including C4 itself.
    Evaluation on C4 after calibrating on C4 makes those numbers partially in-distribution; WikiText-2 and PTB are the external checks.
  • ad hoc to paper Greedy back-substitution in Eq. (22) supplies a sufficiently good solution to the discrete S_i subproblem.
    The S update minimizes each residual term in isolation and is explicitly labeled sub-optimal; no approximation bound or ablation is provided, yet the method's accuracy rests on it.
  • domain assumption The LUT-based mpGEMM kernels of SqueezeLLM used for profiling support GANQ's per-channel codebooks and are the reason speedups transfer.
    Section 4.3 profiles with kernels from Kim et al.; kernel details are external and not re-implemented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models." pith.science (2026). https://pith.science/paper/MGHHH5LO

@misc{pith2026250112956,
  author       = {Pith},
  title        = {Pith review of: GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGHHH5LO}},
  note         = {Machine review of arXiv:2501.12956}
}
abstract

Large Language Models (LLMs) face significant deployment challenges due to their substantial resource requirements. While low-bit quantized weights can reduce memory usage and improve inference efficiency, current hardware lacks native support for mixed-precision General Matrix Multiplication (mpGEMM), resulting in inefficient dequantization-based implementations. Moreover, uniform quantization methods often fail to capture weight distributions adequately, leading to performance degradation. We propose GANQ (GPU-Adaptive Non-Uniform Quantization), a layer-wise post-training non-uniform quantization framework optimized for hardware-efficient lookup table-based mpGEMM. GANQ achieves superior quantization performance by utilizing a training-free, GPU-adaptive optimization algorithm to efficiently reduce layer-wise quantization errors. Extensive experiments demonstrate GANQ's ability to reduce the perplexity gap from the FP16 baseline compared to state-of-the-art methods for both 3-bit and 4-bit quantization. Furthermore, when deployed on a single NVIDIA RTX 4090 GPU, GANQ's quantized models achieve up to 2.57$\times$ speedup over the baseline, advancing memory and inference efficiency in LLM deployment.

Figures

Figures reproduced from arXiv: 2501.12956 by the authors.

Figure 1
Figure 1. (a) A comparison of two mpGEMM implementations: a dequantization-based approach (left) versus a LUT-based method (right). (b) Violin plots showing the first decoder layer’s weight distribution in the LLaMA-2-7B model, clearly illustrating their deviation from a uniform distribution. form quantization Q : R → [0, 2 N − 1] ∩ Z defined as Q(x) = clamp(⌊ x s ⌉) + z, 0, 2 N − 1), where ⌊·⌉ denotes rounding N is the targe… view at source ↗
Figure 2
Figure 2. An illustration of the back-substitution framework for determining Si, leveraging the lower triangular structure of L. we solve for (Si):,n−2 by minimizing the square of (19) – (20): idx = argmin s [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 5 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  3. [9]

    M., Hauth, A., et al

    Gemini Team, Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  4. [10]

    Fast matrix multiplications for lookup table-quantized llms

    Guo, H., Brandon, W., Cholakov, R., Ragan-Kelley, J., Xing, E., and Kim, Y . Fast matrix multiplications for lookup table-quantized llms. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 12419– 12433,

    work page 2024

  5. [12]

    Version 22.1

    URL https://www.ibm.com/docs/e n/icos. Version 22.1. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    arXiv 2001

  6. [13]

    X., Nie, J.-Y ., and Wen, J.-R

    Li, J., Tang, T., Zhao, W. X., Nie, J.-Y ., and Wen, J.-R. Pre- trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1–39, 2024a. Li, M., Lin, Y ., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y ., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models.arXiv preprint a...

    arXiv

  7. [14]

    Llm-qat: Data-free quantization aware training for large language models

    Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y ., Shi, Y ., Krishnamoorthi, R., and Chandra, V . Llm-qat: Data-free quantization aware training for large language models. InFindings of the Association for Computational Linguistics ACL 2024, pp. 467–484,

    work page 2024

  8. [15]

    A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B

    Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. InHuman Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994,

    work page 1994

Show all 23 references
  1. [17]

    Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023a

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models.arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P....

    arXiv

  2. [18]

    Augmenting black-box llms with medical textbooks for clinical question answering

    Wang, Y ., Ma, X., and Chen, W. Augmenting black-box llms with medical textbooks for clinical question answering. arXiv preprint arXiv:2309.02233,

    arXiv

  3. [19]

    Huggingface’s transformers: State-of-the- art natural language processing.arXiv preprint arXiv:1910.03771,

    Wolf, T. Huggingface’s transformers: State-of-the- art natural language processing.arXiv preprint arXiv:1910.03771,

    1910 arXiv

  4. [21]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  5. [22]

    V ., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V ., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    arXiv

  6. [23]

    The sparse component Wsparse is obtained by multiplying the weight matrix W element-wise with the outlier mask, while the dense component Wdense is obtained by subtracting the sparse component from the original matrix. 13 GANQ: GPU-Adaptive Non-Uniform Quantization for Large L...

  7. [1925]

    Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers.arXiv preprint arXiv:2210.17323,

    arXiv

  8. [2017]

    Llama-3: Meta ai’s latest language model

    Meta AI. Llama-3: Meta ai’s latest language model. ht tps://ai.meta.com/blog/meta-llama-3/ , 2024a. Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta. com/blog/llama-3-2-connect-2024-visio n-edge-mobile-devices/, September 2...

    2024 arXiv

  9. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  10. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  11. [2021]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    Dettmers, T., Svirschevski, R., Egiazarian, V ., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    arXiv

  12. [2022]

    Nupes: Non- uniform post-training quantization via power exponent search.arXiv preprint arXiv:2308.05600,

    Yvinec, E., Dapogny, A., and Bailly, K. Nupes: Non- uniform post-training quantization via power exponent search.arXiv preprint arXiv:2308.05600,

    arXiv

  13. [2023]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- ma...

    2019

  14. [2024]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877–1901,

    1901

  15. [2025]

    Version 12.0

    URL https://www.gurobi.com/doc umentation/. Version 12.0. Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding.arXiv preprint arXiv:1510.00149,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.