Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A convolutional-code quantizer takes LLM weights to about 2 bits with accuracy within roughly 2 points of 8-bit baselines.

desk verdict Clever bitshift decoding trick for trellis-coded quantization, but the headline accuracy claim is under-supported and the code cluster rests on a single-channel distributional assumption. read the letter →

arxiv 2507.07145 v1 pith:ZZVTF6PO submitted 2025-07-09 cs.LG

Zhaojing Zhou , Xunchao Li , Minghao Li , Handi Zhang , Haoshuang Wang , Wenbin Chang , Yiqun Liu , Qingqing Dang
show 3 more authors
Dianhai Yu Yanjun Ma Haifeng Wang
This is my paper
classification cs.LG
keywords convolutionalcodequantizationlow-bitweight-onlypost-trainingLLMcompressionbit-shiftdecodingclusterMixture-of-Expertsinferencesingle-GPUdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CCQ aims to show that weight-only post-training quantization of LLMs at 2.0–2.75 bits can keep both accuracy and inference speed. Instead of storing a codebook and doing index lookups, it stores weights as convolutional-code symbols built from overlapping $L$-bit states; dequantization becomes bit shifts, masks, and scalar multiplies, the same cost class as scalar WINT2 kernels. To reach true 2-bit rates, a code-cluster step uniform-quantizes the per-output-channel distribution of code values to UINT8, storing four weights in one byte while the underlying state precision stays 6 bits. The paper reports that DeepSeek-V3-0324 drops to 184GB and ERNIE-4.5-300B-A47B to 89GB, with average scores about two points below 8-bit baselines and GEMV latency close to scalar WINT2, which is what the paper relies on for single-GPU deployment.

What carries the argument

The load-bearing mechanism is the convolutional-code codebook built from overlapping states: in configuration $(L,N,S)$, every row is an $N$-vector of states related by $S$-bit transitions, so the whole row is identified by a $T=L+(N-1)S$ bit symbol, and the $i$-th state is recovered from the symbol by a bit shift and mask—arithmetic that replaces vector quantization's memory lookup. Hybrid encoding interleaves two configurations to hit storage data types while keeping the linear mapping. Code Cluster is the second engine: it uniform-quantizes the code-value distribution along output channels into $[0,255]$ with per-channel scale $\alpha$ and zero-point $\beta$ (Eq. 1), collapsing the $2^T$ encoding space while retaining 6-bit true precision of the underlying weights; this is what brings the rate to 2.06 bpw. Group-scale optimization closes the loop by minimizing per-group MSE in closed form, and the optimized scale is packed into redundant bits of the code groups or into channel super-scales. Together these pieces convert the accuracy-versus-speed trade-off of vector quantization into a lookup-free arithmetic pipeline.

What would settle it

Run CCQ's code search with configuration $(L=6,N=4,S=3)$ on a dense LLM, not just the MoE models in the paper, and compute per-channel histograms of the resulting code values before code clustering; if a substantial fraction of channels show heavy tails or multiple modes instead of an approximately normal shape, the uniform 8-bit mapping of Eq. (1) will add reconstruction error beyond the 6-bit state quantization, and the 2.06 bpw accuracy should degrade more than the paper's 'around 2%' statement. A direct check is to compare the channel-wise reconstruction MSE with and without the code-cluster step on those histograms.

Watch

Extended reading notes

Core claim

CCQ is a weight-only post-training quantization (PTQ) method whose central object is a convolutional-code codebook generated by the encoding rule rather than stored: for configuration $(L,N,S)$, each code word collects $N$ consecutive $L$-bit states with $S$ new bits per transition, so a word costs $T=L+(N-1)S$ bits and its state values are recovered by shifting and masking the word. Hybrid encoding alternates configurations such as $(L=3,N=3,S=2)$ and $(L=3,N=4,S=2)$ to pack seven numeric values into 16 bits, reaching 2.28 bpw. Code Cluster then quantizes the per-channel distribution of code values into an 8-bit range using asymmetric uniform quantization, so four codes fit in one UINT8 byte; because the codes were chosen at 6-bit state precision, the paper argues that clustering to a byte costs little accuracy. On DeepSeek-V3-0324, CCQ 2.06 bpw scores 87.76 average versus 90.01 for the 8-bit baseline; on ERNIE-4.5-300B-A47B it scores 87.36 versus 89.39 for WINT8, and the same 2.06 bpw kernel runs GEMV at 0.034 ms versus 0.027 ms for scalar WINT2 and 0.058 ms for VPTQ on one tested shape. The paper's claim is that this combination—about 2 points of degradation at a 68–71% memory cut with WINT2-class speed—is what makes single-GPU inference practical for these models.

Load-bearing premise

The method's 2.06-bit mode assumes that, for each output channel, the hidden code values occur with roughly the same bell-curve shape across the range, so one uniform 8-bit rescaling can represent them all; the paper shows this shape for one example channel and relies on it for the extra compression.

Editorial extensions

If this is right

  • At 2.06 bpw, DeepSeek-V3-0324's memory drops from 642GB (8-bit) to 184GB, and ERNIE-4.5-300B-A47B drops from 281GB (WINT8) to 89GB, which the paper reports as enabling single-GPU ERNIE serving without inter-card communication.
  • The bit-shift dequantization pipeline is compatible with standard high-throughput serving kernels: in the grouped-GEMM tests, CCQ's W2A16 operator is faster than W8A16 and W4A16 across the reported batch sizes and shapes.
  • Because no codebook is stored, deployment memory is essentially indices plus compressed scales, which gives a 2.75/2.5/2.28/2.06 bpw option set for different accuracy-size budgets.
  • The reported accuracy loss is concentrated in C-Eval and MMLU, so the method's 'minimal loss' claim is tied to those benchmarks, not to all tasks equally.
  • Open-sourcing the 2-bit ERNIE-4.5 model and engine makes the single-GPU deployment claim reproducible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 2.06 bpw mode leans on the paper's per-channel bell-curve observation; if that fails on heavier-tailed layers or dense models, the 2.28–2.75 bpw modes that skip code clustering remain the safer configurations.
  • Editorial inference: since adjacent states already overlap in the bit representation, the same bitshift/mask decoding could pair with other structured codes (for example polar-like constructions) to trade codebook size against reconstruction fidelity, a direction the paper only flags conceptually.
  • Editorial inference: the accuracy drop on instruction-following benchmarks suggests that layer-sensitive bit allocation may be needed for chat or agent workloads even when average scores stay close to 8-bit baselines.
  • Editorial inference: because dequantization is arithmetic with no table lookups, CCQ should compose with activation quantization or KV-cache compression without changing the kernel's memory access pattern, though the paper does not explore either combination.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CCQ, a weight-only post-training quantization (PTQ) method that replaces the codebook lookup of vector quantization with bit-shift operations on convolutional-code indices. It introduces hybrid encoding to reach 2.5 bpw and a 'code cluster' scheme that quantizes the code-index distribution per output channel to reach 2.06 bpw. The authors report 2.06–2.75 bpw results for DeepSeek-V3-0324 and ERNIE-4.5-300B-A47B, claiming memory reductions of roughly 70% with about a 2-point average accuracy drop, and operator-level benchmarks showing latency near scalar WINT2. They also open-source a 2-bit ERNIE 4.5 variant.

Significance. If the accuracy claims hold, CCQ is a practically valuable contribution: it is a codebook-free VQ-style scheme with bit-shift dequantization, and the open-sourced 2-bit ERNIE 4.5 model would enable single-GPU deployment. Strengths include the closed-form scale optimization (Eq. 4), explicit bpw accounting (Eq. 5), and the kernel-level comparison with VPTQ. However, the central 'minimal accuracy loss' claim is not yet supported by the evidence: the experiments compare only against 8-bit and 4-bit baselines, no comparison with state-of-the-art 2-bit PTQ methods is provided, and the code-cluster scheme's distributional assumption rests on a single histogram. The method is novel and defensible, but the evaluation is incomplete for the abstract's broad claims.

major comments (5)
  1. [Section 4.2, Tables 2 and 3, Section 7] The central accuracy claim of 'minimal accuracy loss' at 2.0–2.75 bpw is evaluated only against 8-bit and 4-bit baselines (WINT8, WINT4, GPTQ 4-bit, AWQ 4-bit); no comparison is made with existing 2-bit PTQ methods such as QTIP, VPTQ, AQLM, or QUIP#. Section 7 explicitly admits that 'further investigation is warranted for dense models under state-of-the-art 2-bit quantization methods such as QTIP and VPTQ.' Because the abstract and conclusion assert outstanding performance, the authors must either add such comparisons (including perplexity on standard corpora) or temper the claims to the actual comparison set.
  2. [Section 3.3, Figure 3] The 2.06-bpw Code Cluster mechanism assumes the convolutional code values on each output channel are approximately normally distributed, so that an affine uniform quantization to 8 bits (Eq. 1) is faithful. The paper supports this with a single histogram from one channel. If other channels, layers, or models yield heavy-tailed or multimodal distributions, the 8-bit uniform quantization will allocate bins poorly and add error that is unevenly distributed across the four weights decoded from one byte via bit-shifts [9,6,3,0]. This distributional premise is load-bearing for the headline 2-bit result; the paper should provide systematic evidence across channels, layers, and models, and ideally a robustness analysis of the cluster step.
  3. [Section 3.3] The statement 'the true quantized value of the weights is 6 bits' is misleading. After Code Cluster, each code value is quantized to an 8-bit integer (Eq. 1), and the four weights are extracted from 6-bit fields of that byte; the effective per-weight precision is set by the cluster bin width, not by L=6. The text should describe the actual precision of the decoded weights and explicitly note that the L=6 state precision applies only before the cluster quantization.
  4. [Table 3] The ERNIE 4.5 results are ambiguously labeled. The table lists rows WINT8, WINT4, and WINT2, while the text says 'our 2 bpw quantized model reduces GPU memory consumption by 68.33% compared to the WINT8 baseline.' It is unclear whether the WINT2 row is the CCQ result, a scalar 2-bit baseline, or both. If the WINT2 row is CCQ, the labeling is inconsistent with Tables 2 and 4; if it is a baseline, the CCQ results are missing. Please report the CCQ rows explicitly and clarify the baseline definitions.
  5. [Section 3.4, Eq. (4)] The scale optimization assumes the decoded values Wq are fixed. With Code Cluster, the decoded values also depend on the code scale alpha and zero-point beta (Eq. 1) and on the rounding in Algorithm 3. The paper does not specify whether the group scale S is optimized jointly with alpha and beta, or sequentially, and the MSE derivation in Eqs. (2)–(4) does not account for the effect of alpha/beta on the decoded values. This matters because the final dequantization error is a function of both quantizations; please state the optimization procedure and justify the chosen order.
minor comments (6)
  1. [Equation (4)] Equation (4) writes 'W·W q_i' but should read 'W_i · Wq_i' for clarity.
  2. [Section 5, Table 4] Table 4 compares VPTQ at its recommended 3 bpw with CCQ at 2.06 bpw; the latency comparison is not at matched bit-widths and should be noted in the text.
  3. [Section 4.1] The experimental setup does not specify the calibration data used for scale optimization and code-cluster fitting; please add these details (e.g., dataset, number of samples).
  4. [Figure 3] The caption of Figure 3 says '(L=6, N=4, N=3)'; the third parameter should be S=3.
  5. [Throughout] There are numerous typos and spacing issues, such as 'W AQ', 'Yiqun Liu,Wenbin Chang', and 'configurations(bpw=3)'; these should be corrected.
  6. [Section 4.1] The term 'WINT2' is used for the standard 2-bit baseline in Table 4 but is not defined; please clarify whether it is RTN 2-bit or another method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CCQ's accuracy claims are empirical results against external benchmarks, with standard fitted scales and no prediction that reduces to its own inputs.

full rationale

The derivation chain is self-contained in the circularity-relevant sense. Convolutional code quantization is defined constructively in Section 3.1: a codebook C is generated from an (L,N,S) configuration, group scales are computed from weights, and each weight group is matched to the codebook entry with minimum MSE. The scale update in Eq. 4 is a closed-form least-squares solution minimizing the reconstruction error in Eq. 2; this is standard parameter fitting, not a renamed prediction. Hybrid Encoding in Section 3.2 and Code Cluster in Section 3.3 are additional compression transforms applied to the already-chosen code indices; Eq. 1 uniformly quantizes the code-value distribution, and the paper evaluates the resulting accuracy empirically on external benchmarks (GSM8K, C-Eval, MMLU, BBH, DROP, CMath, MUSR) in Tables 2 and 3. The claim of "minimal accuracy loss" is therefore tested against independent data and baselines, not guaranteed by construction. The only self-referential elements are citations to the Baidu ERNIE technical report and the open-sourced ERNIE 4.5 model, and these are used as experimental subjects rather than as load-bearing authority for the method. Section 3.3's assumption that convolutional code values are approximately normal per output channel, supported by one histogram in Figure 3, is an empirical modeling premise; if it is violated the method may lose robustness, but that is a correctness risk, not circularity, because the uniform quantizer does not by definition produce the distribution it assumes. No fitted parameter is relabeled as a prediction, and no load-bearing step reduces to a self-citation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method depends on hand-chosen encoding configurations and group sizes, and on fitted scales (group scale, code scale, code zero-point, super scale). No new entities are introduced. The most fragile domain assumption is the near-normal distribution of code values that justifies the code cluster.

free parameters (5)
  • Group size g = 64 (example; not stated as tuned)
    Hand-chosen granularity for group-wise quantization and scale storage; affects bpw and encoding alignment.
  • Encoding configuration (L,N,S) = (4,3,2), (3,3,2)/(3,4,2), or (6,4,3) depending on bpw target
    Hand-selected to match storage data types, bit-shift masks, and compression targets.
  • Code scale alpha and code zero-point beta = per-output-channel fitted values
    Fitted to the distribution of convolutional code values in Eq. (1) for the code cluster; maps 15-bit codes into uint8.
  • Group scale S = per-group fitted
    Optimal scale from MSE minimization; Eq. (4) is the least-squares solution given the chosen codes.
  • Super scale s_super = per-channel fitted
    Used to quantize group scales for storage, adding another layer of fitted parameters.
assumptions (5)
  • standard math Convolutional/trellis code structure yields valid codebooks via state transitions (Marcellin and Fischer 1990; QTIP).
    Section 3.1 builds the codebook from (L,N,S) state transitions; this follows established coding theory.
  • standard math MSE-based codebook search (argmin over 2^T codes) is a valid nearest-neighbor quantizer.
    Algorithm 1 selects the code minimizing squared error; standard vector quantization premise.
  • domain assumption Convolutional code values per output channel are approximately normally distributed (code cluster assumption).
    Section 3.3 states the distribution is suitable for uniform quantization; only one channel shown in Figure 3.
  • domain assumption Padding groups to multiples of N with zeros does not compromise quantization accuracy.
    Section 3.1 states this without evidence; affects groups whose size is not a multiple of N.
  • domain assumption Benchmarks (GSM8K, C-Eval, MMLU, etc.) and the 8-bit/4-bit baselines are sufficient to demonstrate near-lossless quality.
    Section 4.1 defines the evaluation; missing perplexity and SOTA 2-bit comparisons make this assumption load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs." pith.science (2026). https://pith.science/paper/ZZVTF6PO

@misc{pith2026250707145,
  author       = {Pith},
  title        = {Pith review of: CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZVTF6PO}},
  note         = {Machine review of arXiv:2507.07145}
}
read the original abstract

The rapid scaling of Large Language Models (LLMs) elevates inference costs and compounds substantial deployment barriers. While quantization to 8 or 4 bits mitigates this, sub-3-bit methods face severe accuracy, scalability, and efficiency degradation. We propose Convolutional Code Quantization (CCQ), an inference-optimized quantization approach compressing LLMs to 2.0-2.75 bits with minimal accuracy loss. Departing from error-prone scalar quantization or slow vector quantization, CCQ integrates a hardware-aware bit-shift encoding and decoding solution with Convolutional Code, Hybrid Encoding, and Code Cluster, jointly overcoming accuracy-speed bottlenecks. We construct a lookup-free encoding space, enabling a linear mapping between the codebook and weight vectors, thereby optimizing inference performance. Meanwhile, by drawing on the concept of data mapping from vector quantization, we minimize the performance degradation of the model under extremely low-bit conditions. Experiments demonstrate that CCQ achieves outstanding performance on LLMs across various benchmarks. We compress DeepSeek-V3 (671B total parameters) to 184GB and ERNIE-4.5-300B-A47B to 89GB, enabling single-GPU deployment of ERNIE 4.5 and eliminating inter-card communication. The 2-bit ERNIE-4.5-300B-A47B model and inference engine have been open-sourced.

Figures

Figures reproduced from arXiv: 2507.07145 by the authors.

Figure 1
Figure 1. Convolutional code codebook construction. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A schematic diagram of the CCQ. The dequantization part only involves shifting and dequantization [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The code value distribution diagram of convo [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    LC-QAT achieves data-efficient 2-bit weight-only QAT for LLMs by representing quantized weights as a learned affine transform over discrete vectors, supporting end-to-end optimization from a high-quality PTQ start.

  2. LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    LC-QAT is a 2-bit weight-only vector quantization aware training framework for LLMs that uses linear-constrained affine mappings to achieve data-efficient optimization and outperform prior QAT methods.

Reference graph

Works this paper leans on

39 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hicham Badri and Appu Shaji. 2023. https://mobiusml.github.io/hqq_blog/ Half-quadratic quantization of large machine learning models

    work page 2023

  2. [2]

    Hicham Badri and Appu Shaji. 2024. https://mobiusml.github.io/1bit_blog/ Towards 1-bit machine learning models

    work page 2024

  3. [3]

    Baidu-ERNIE-Team. 2025. Ernie 4.5 technical report

    work page 2025

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  5. [6]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161

    arXiv 2019

  6. [7]

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. https://arxiv.org/abs/2401.06118 Extreme compression of large language models via additive quantization . Preprint, arXiv:2401.06118

    arXiv 2024

  7. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    arXiv 2022

  8. [9]

    Aaron Grattafiori, Abhimanyu Dubey, and et.al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

Show all 39 references
  1. [10]

    Robert Gray. 1984. Vector quantization. IEEE Assp Magazine, 1(2):4--29

    1984

  2. [11]

    Gray and David L

    Robert M. Gray and David L. Neuhoff. 1998. Quantization. IEEE transactions on information theory, 44(6):2325--2383

    1998

  3. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  4. [13]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270--1303

    2024

  5. [14]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in N...

    2023

  6. [15]

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629

    2023 arXiv

  7. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  8. [17]

    Banseok Lee, Dongkyu Kim, Youngcheon You, and Youngmin Kim. 2025. https://arxiv.org/abs/2506.13771 Littlebit: Ultra low-bit quantization via latent factorization . Preprint, arXiv:2506.13771

    2025

  9. [18]

    Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. 2025. Gptaq: Efficient finetuning-free quantization for asymmetric calibration. arXiv preprint arXiv:2504.02692

    2025 arXiv

  10. [19]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

    2024

  11. [20]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  12. [21]

    Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024 b . Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprint arXiv:2409.17066

    2024 arXiv

  13. [22]

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888

    2023 arXiv

  14. [23]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 c . Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750

    2024 arXiv

  15. [24]

    Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. 2024. https://arxiv.org/abs/2405.14852 Pv-tuning: Beyond straight-through estimation for extreme llm compression . Preprint, arXiv:2405.14852

    2024 arXiv

  16. [25]

    Michael W Marcellin and Thomas R Fischer. 1990. Trellis coded quantization of memoryless and gauss-markov sources. IEEE transactions on communications, 38(1):82--93

    1990

  17. [26]

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2023. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049

    2023 arXiv

  18. [27]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  19. [28]

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10--19

    2019

  20. [29]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024 a . Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396

    2024 arXiv

  21. [30]

    Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. 2024 b . Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597--59620

    2024

  22. [31]

    Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. 2024. Gptvq: The blessing of dimensionality in llm quantization. arXiv preprint arXiv:2402.15319

    2024 arXiv

  23. [32]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453

    2023 arXiv

  24. [33]

    Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023. Cmath: Can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306.16636

    2023 arXiv

  25. [34]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR

    2023

  26. [35]

    Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. https://arxiv.org/abs/2402.11295 Onebit: Towards extremely low-bit large language models . Preprint, arXiv:2402.11295

    2024 arXiv

  27. [36]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  28. [37]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  29. [38]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  30. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.