Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read QRazor claims a bit-level 'razoring' step can quantize LLM weights, activations, and KV cache to 4 bits with no fine-tuning, matching or beating rotation-based methods.

desk verdict A genuinely new 4-bit PTQ idea with a sign-bit ambiguity that must be fixed before the '4-bit' claim holds. read the letter →

arxiv 2501.13331 v2 pith:EXCCOOUC submitted 2025-01-23 cs.LG

Dongyoung Lee , Seungkyu Choi , Ik Joon Chang This is my paper · ORCID
classification cs.LG
keywords 4-bitquantizationpost-trainingLLMinferenceKVcachebit-levelcompressionhardwareaccelerationsignificance-basedencodingzero-shotreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QRazor claims that transformer-based LLMs can be quantized to 4 bits for weights, activations, and KV cache without fine-tuning, retraining, data reordering, or rotation. The scheme first converts tensors to 8- or 16-bit integers using absolute-max scaling, then compresses each group of integers down to its four most significant bits via significant data razoring (SDR). The paper argues this preserves the values that matter because leading-one positions in a group cluster into a narrow bit range, so a single razoring point plus four retained bits captures both outliers and ordinary values. On zero-shot reasoning benchmarks across LLaMA-2, LLaMA-3, and Mistral-7B, QRazor reports accuracy matching or beating prior 4-bit methods that require rotation or channel reassembly, and it shows that the compressed format can be multiplied directly with a small integer unit, cutting hardware power and area roughly in half.

What carries the argument

The mechanism is significant data razoring (SDR): a per-group bit-level compressor. For each group, it computes the bitwise OR of the sign-and-magnitude integers, locates the leading one (the razoring point), captures a fixed number of adjacent salient bits (four for 4-bit output), truncates the higher zero bits and lower bits, applies round-to-nearest with a special floor for elements whose salient bits are all ones, and stores a small flag giving the number of truncated least-significant bits. The flag lets the hardware shift the 4-bit values back into place during a multiply-accumulate, so a 4-bit multiplier plus a single barrel shifter performs the operation without dequantization. SDR's efficiency comes from replacing per-group floating-point scale factors and absolute-max computations with simple bitwise OR and shifts.

What would settle it

Run QRazor's W4A4KV4 configuration on a model known to have heavier activation outliers than the three tested families (for instance, an instruction-tuned or mixture-of-experts model) and measure zero-shot accuracy; if the leading-one positions spread beyond the observed 8th-to-12th-bit band or the fraction of groups with razoring points above the 12th bit substantially exceeds the reported 9%, the method's accuracy should collapse toward the group-size-128 results in the paper, where accuracy drops by 10+ points.

Watch

Extended reading notes

Core claim

The central claim is that reliable 4-bit quantization of LLMs does not require reshaping the data distribution; it only requires choosing the right base precision and then keeping the most informative bits. Starting from 8-bit weights and KV cache with 16-bit activations under static absolute-max scaling, QRazor's SDR stage finds, for each group of 16 to 128 elements, the highest '1' position in a bitwise OR, calls that the razoring point, and keeps the sign bit plus the four adjacent bits, rounding with an overflow guard. Because the leading-one positions of real transformer tensors concentrate between roughly the 8th and 12th bit, the technique discards mostly zeros in high positions and small values in low positions. The authors report that the resulting W4A4 and W4A4KV4 models beat SmoothQuant and QLLM by more than 12 points and QuaRot with round-to-nearest by more than 2.9 points on LLaMA-2-7B, and come close to QuaRot with GPTQ, while requiring only static scaling.

Load-bearing premise

The entire method rests on the empirical regularity that the leading-one positions of quantized values in a group concentrate in a narrow bit range, so one razoring point per group plus four retained bits can represent the group's important information; this was observed only on LLaMA-2, LLaMA-3, and Mistral-7B on five zero-shot tasks.

Editorial extensions

If this is right

  • Weights, activations, and KV cache all reach 4 bits in the reported W4A4KV4 runs, so long-context inference can cut KV-cache memory while keeping accuracy within a few points of the FP16 baseline.
  • Because activation and KV scaling are static per-tensor, there is no per-token or per-group scale computation at runtime, and no calibration beyond collecting 128 wikitext samples.
  • The proposed decompression-free MAC uses a 4x4 multiplier and one barrel shifter, with reported 57.8% power and 61.2% area savings versus decompress-then-multiply, and still beats INT8 MAC units by about a third on both metrics.
  • Group sizes of 16 or 32 give effective bit-widths of 4.25/4.125 data bits, comparable to standard per-tensor-group quantization, while avoiding the FP32/FP16 scale-factor overhead of those schemes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SDR's leading-one clustering holds broadly, then outlier handling in PTQ can be reformulated as a bit-plane selection problem rather than a distribution-reshaping problem; a natural check is measuring the spread of leading-one positions on instruction-tuned or mixture-of-experts models, where activations are known to shift.
  • The paper's own comparison with QuaRot(GPTQ) suggests the method is orthogonal to weight-reordering techniques: applying GPTQ to QRazor's 4-bit weights could close the remaining gap to the rotation-based baseline without adding rotation overhead.
  • The hardware savings rest on static scaling; extending the scheme to dynamic per-token scaling would reintroduce per-group arithmetic and erode most of the 57.8%/61.2% gains, so the hardware claim should be read as limited to the static-scaling configuration.
  • The SDR procedure is parameterized by the number of kept salient bits, so the same machinery should cover 3-bit or 5-bit compression; testing whether accuracy degrades smoothly with that parameter would map the reliability boundary of the method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. QRazor proposes a two-stage post-training quantization scheme for LLMs. In the first stage, weights, activations, and KV cache are quantized to 8-bit or 16-bit integers with absolute-max scaling (W8A16 or W8A16KV8). In the second stage, the integers are compressed to 4-bit using Significant Data Razoring (SDR): per group, the leading-one position (the 'razoring point') is found via a bitwise OR, and the four adjacent salient bits are retained along with a sign bit and flag bits. The paper reports zero-shot accuracies for LLaMA-2-7B/13B, LLaMA-3-8B, and Mistral-7B on five tasks, claiming performance similar to or better than SOTA 4-bit methods such as QuaRot(GPTQ) and QServe, without fine-tuning or rotation. It also presents an integer-based decompression-free MAC unit and reports 57.8% power and 61.2% area savings relative to a 16x8 INT MAC.

Significance. If the representation and comparison issues are resolved, the SDR idea is a genuinely simple and hardware-friendly alternative to rotation-based outlier suppression: it avoids Hadamard transforms, uses bitwise operations, and enables direct low-precision arithmetic. The inclusion of a synthesized Verilog MAC unit and detailed accuracy tables across three model families are strengths, and the method appears to produce a working PTQ recipe at near-4-bit precision. However, the significance hinges on three load-bearing points: the storage format must truly be 4-bit (or be honestly reported as ~5-bit), the SOTA comparison must include the strongest baselines in every row, and the 'reliable and effortless' generalization must be supported beyond the tested model families. The paper's claims currently outrun the evidence on all three points.

major comments (4)
  1. [Section 4.2, Algorithm 1, Tables 2 and 4] The representation is internally inconsistent. The text says 'we retain only the sign bit and the selected salient bits,' which for a 4-bit target means 5 bits per stored value, yet the effective-bit tables report 4.125-4.25 bits and count only flag-bit overhead. If the sign is stored separately, the true memory cost is about 5.125-5.25 bits/value and the '4-bit quantization' claim is not accurate; if the sign is not stored, the described encoding cannot represent negative values. Algorithm 1's conversion for negative inputs (concat(si,j, 2's complement(esi,j))) is not a standard two's complement encoding, and the subsequent MSB truncation treats positive and negative operands asymmetrically. This issue is load-bearing because the effective-bit, memory-savings, and hardware-multiplier (4x4) claims all depend on the actual storage format.
  2. [Table 2] The 'similar or better than SOTA' claim is selective. For LLaMA-2-7B, QuaRot(GPTQ) achieves an average of 65.64, while QRazor scores 63.88 (W4A4 g32), 63.65 (W4A4KV4 g16), and 61.17 (W4A4KV4 g32); only W4A4 g16 at 65.91 slightly exceeds QuaRot(GPTQ). For LLaMA-2-13B, QuaRot(GPTQ) averages 69.79 while the best QRazor result is 68.59. The abstract's comparison against SmoothQuant, QLLM, and QuaRot(RTN) omits the stronger GPTQ baselines. Please report the strongest available QuaRot configuration in every model row or restrict the headline claim to the configurations where the comparison actually holds.
  3. [Section 4.2, Figure 2, Table 4] The reliability premise is established only for three model families on five zero-shot tasks with 128-sample calibration. The statement that only 9% of groups exceed the 12th bit position is load-bearing for the 'reliable' claim, and Table 4 shows how sensitive the method is when this premise weakens: LLaMA-2-7B W4A4KV4 average accuracy drops from 63.65 at group size 16 to 47.36 at group size 128. Models with more frequent activation outliers (instruction-tuned, mixture-of-experts, long-context workloads, or larger scales) are not covered, so the blanket 'reliable and effortless' wording overgeneralizes. Please add stress tests on such models or explicitly scope the claim.
  4. [Section 5.4, Table 5] The hardware savings claim is not an apples-to-apples comparison. The abstract states '57.8% power and 61.2% area savings compared to arithmetic operations performed after decompression,' but the INT16x8 baseline in Table 5 does not include the decompression shifters needed to reconstruct the two operands to base precision; only the proposed design includes a shifter. A fair baseline must add the decompression logic (e.g., barrel shifters for both operands) before the 16x8 MAC. If the representation in Major Comment 1 is actually 5 bits, a 4x4 multiplier is also insufficient, and the power/area numbers need to be re-derived with the correct operand width.
minor comments (6)
  1. [Abstract and Section 5.3] There are typos in 'requirment' (abstract) and 'evalutate' (Section 5.3); Figure 2(c) labels 'Weght' instead of 'Weight'.
  2. [Table 1] The column abbreviations PQ, AE, AC, HS, and WG are not defined in the table or its caption; please spell out the task names or refer explicitly to Section 5.1.
  3. [Throughout] The name QuaRot is spelled inconsistently as 'Quarot' in multiple places (e.g., Introduction, Section 4, Table 2 caption); please use the original spelling consistently.
  4. [Section 5.1] The sentence 'Calibration and validation of zero-shot accuracy and perplexity were performed by randomly selecting 128 samples from Wikitext2 for all tasks' is ambiguous; it could be read as calibrating all tasks on Wikitext2. Please clarify that calibration uses Wikitext2 while evaluation uses the listed zero-shot benchmarks.
  5. [Table 4] The LLaMA-3-8B baseline average accuracy listed as 72.57 differs from the FP16 average computed from Table 2 (72.77); please reconcile these numbers.
  6. [Table 2] The LLaMA-3-8B HellaSwag FP16 entry is listed as '79.18s', which appears to be a typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: QRazor's SDR compression and accuracy results are empirically self-contained.

full rationale

The paper's derivation chain is not circular. QRazor is defined as a two-stage pipeline: absolute-max per-tensor/per-channel integer quantization (W8A16 or W8A16KV8), followed by significant data razoring (SDR), which finds a per-group razoring point with bitwise OR, keeps a fixed number of salient bits plus flag bits, and rounds with overflow protection. Accuracy results in Tables 2-4 and Appendix A.5-A.6 are empirical comparisons to SmoothQuant, QLLM, QuaRot, QServe, etc.; no headline number is obtained by substituting a fitted quantity back into its own defining equation. The choice of base precision is motivated by Table 1, which compares W8A8, W8A16, and FP16 on the same models; this is an experiment, not a definitional identification. The group-size sweep (Table 4) is standard hyperparameter selection on the evaluation suite, and the abstract's zero-shot claim rests on PIQA/ARC/HellaSwag/Winogrande, which are external to the 128-sample Wikitext2 calibration set. The only self-citation is O2A (Ho et al., 2020), a prior flag-bit design by co-author Chang; it is cited as related work and is not load-bearing for QRazor's central claims, which are supported by new experiments and Verilog synthesis. A separate internal inconsistency exists between the text saying the sign bit is retained alongside four salient bits and the stated 4-bit effective bit-widths; that is a correctness and consistency concern, not circularity, and does not change this verdict.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on no fitted constants, but on several domain assumptions: that per-tensor static scaling to W8A16 preserves accuracy (validated only on five tasks), that a single shared exponent per group of 16 to 32 elements captures the salient distribution, and that the leading-one distribution observed in LLaMA/Mistral models is representative. The only hand-selected values are the group size, base precision, and the number of salient bits, all chosen by ablations on the same benchmark.

free parameters (3)
  • Group size g = 16 and 32
    Chosen after ablation (Table 4) as the largest group sizes that keep average accuracy close to the FP16 baseline. This is a hand-selected hyperparameter, not derived from first principles.
  • Base precision scenario = W8A16 / W8A16KV8
    Selected from Table 1 because W8A16 is the coarsest precision that keeps zero-shot accuracy close to FP16 on the five test tasks. This is an empirical design choice.
  • Number of salient bits = 4
    The target precision; fixed by the 4-bit claim. Not fitted to data, but a design decision that defines the method.
assumptions (4)
  • domain assumption Absolute max scaling with 8-bit weights and 16-bit activations preserves accuracy close to FP16.
    Invoked in Section 4.1 and validated only in Table 1 on five commonsense tasks; if this base precision is lossy for other models, SDR compression adds further error.
  • standard math Bitwise OR of the magnitudes in a group identifies the maximum leading-one position, and this leading-one position is a sufficient shared exponent for the group.
    This is a standard property of binary numbers; the 4-bit truncation's error behavior is not analyzed, only empirically demonstrated in Figure 2.
  • domain assumption The leading-one distribution measured on LLaMA-2/3 and Mistral activations and weights is representative of other LLMs.
    Section 4.2 and Figure 2 state that only 9% of groups have a leading '1' beyond the 12th bit; the method's reliability depends on this proportion staying small for other models.
  • domain assumption Per-tensor static scaling for activations is sufficient when combined with SDR, avoiding per-token scaling.
    Stated in Section 4 as a design choice; validated only on the reported tasks and models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring." pith.science (2026). https://pith.science/paper/EXCCOOUC

@misc{pith2026250113331,
  author       = {Pith},
  title        = {Pith review of: Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXCCOOUC}},
  note         = {Machine review of arXiv:2501.13331}
}
read the original abstract

Large-scale language models (LLMs) excel in language processing tasks but face deployment challenges due to high memory and computational demands. While low-bit quantization, such as 4-bit techniques, offers a potential solution, these methods often suffer from significant accuracy loss or require considerable effort for implementation such as reordering, rotation, etc. To address these challenges, we propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based LLMs. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique, which retains only the four most salient bits. Without any additional requirment of fine-tuning or additional training, QRazor achieves performance similar or better compared to state-of-the-art in 4-bit quantization method, surpassing Smoothquant and QLLM by over 12 points and Quarot(RTN) by more than 2.9 points in zero-shot reasoning task accuracy on the LLaMA2-7B model. Additionally, we introduce an integer-based arithmetic unit optimized for QRazor, allowing direct low-precision operations on SDR data without decompression.

Figures

Figures reproduced from arXiv: 2501.13331 by the authors.

Figure 1
Figure 1. The overall concept of our QRazor scheme. 3. Preliminaries Absolute Max Scaling. The absolute max scaling (Latotzke et al., 2022) ensures that input data fits within a target bit￾width by utilizing the absolute maximum value of the data. The quantization process begins by identifying the absolute maximum value |Xmax| in the tensor, which is then used to calculate the scale factor. For example, in the case of 8-bit i… view at source ↗
Figure 2
Figure 2. Leading ’1’ position before the compression for (a) ac￾tivation and query, and (b) key. (c) portion of zeroed elements between “Before 4-bit compression” and “After 4-bit compression” of outliers, as mentioned above, this explains why QRazor consistently delivers reliable performance across various LLMs, as discussed in Section 5.2. It is important to note that in our compression, the trunca￾tion level dynamically v… view at source ↗
Figure 3
Figure 3. The concept of a commonly perceived decompression method(a) compared with the proposed decompression-free arithmetic(b). 4.3. Decompression-Free Arithmetic Operation We introduce an arithmetic unit designed to execute multiply-and-accumulate (MAC) operations required for matrix multiplication while preserving the compression of our QRazor, a method we refer to as decompression-free arithmetic. Typically, data compre… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: SDR Encoding Scheme Consisting of Bitwise OR Operations for detecting the razoring point. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The QRazor quantization flow in attention and feed-forward network layers 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 4 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Quik: Towards end-to-end 4-bit inference on generative large language models, 2023

    Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. Quik: Towards end-to-end 4-bit inference on generative large language models, 2023. URL https://arxiv.org/abs/2310.09259

    arXiv 2023

  3. [3]

    L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms, 2024. URL https://arxiv.org/abs/2404.00456

    arXiv 2024

  4. [4]

    Post-training 4-bit quantization of convolution networks for rapid-deployment, 2019

    Banner, R., Nahshan, Y., Hoffer, E., and Soudry, D. Post-training 4-bit quantization of convolution networks for rapid-deployment, 2019. URL https://arxiv.org/abs/1810.05723

    arXiv 2019

  5. [5]

    Quantease: Optimization-based quantization for language models, 2023

    Behdin, K., Acharya, A., Gupta, A., Song, Q., Zhu, S., Keerthi, S., and Mazumder, R. Quantease: Optimization-based quantization for language models, 2023. URL https://arxiv.org/abs/2309.01885

    arXiv 2023

  6. [6]

    Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2024. URL https://arxiv.org/abs/2307.13304

    arXiv 2024

  7. [7]

    Optimize weight rounding via signed gradient descent for the quantization of llms, 2024

    Cheng, W., Zhang, W., Shen, H., Cai, Y., He, X., Lv, K., and Liu, Y. Optimize weight rounding via signed gradient descent for the quantization of llms, 2024. URL https://arxiv.org/abs/2309.05516

    arXiv 2024

  8. [8]

    J., and Khailany, B

    Dai, S., Venkatesan, R., Ren, H., Zimmer, B., Dally, W. J., and Khailany, B. Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference, 2021. URL https://arxiv.org/abs/2102.04503

    arXiv 2021

Show all 40 references
  1. [9]

    B., Cavalcanti, G

    de Amorim, L. B., Cavalcanti, G. D., and Cruz, R. M. The choice of scaling technique matters for classification performance. Applied Soft Computing, 133: 0 109924, January 2023. ISSN 1568-4946. doi:10.1016/j.asoc.2022.109924. URL http://dx.doi.org/10.1016/j.asoc.2022.109924

    2023

  2. [10]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 3...

    2022

  3. [11]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023

    Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023. URL https://arxiv.org/abs/2306.03078

    2023 arXiv

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/2210.17323

    2023 arXiv

  5. [13]

    A framework for few-shot language model evaluation, 12 2023

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2023

  6. [14]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  7. [15]

    Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization

    Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture, volume ...

    2023

  8. [16]

    O-2a: Low overhead dnn compression with outlier-aware approximation

    Ho, N.-D., Le, M.-S., and Chang, I.-J. O-2a: Low overhead dnn compression with outlier-aware approximation. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp.\ 1--6, 2020. doi:10.1109/DAC18072.2020.9218594

    2020

  9. [17]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024. URL https://arxiv.org/abs/2401.18079

    2024 arXiv

  10. [18]

    W., and Keutzer, K

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization, 2024. URL https://arxiv.org/abs/2306.07629

    2024 arXiv

  11. [19]

    Post-training quantization for energy efficient realization of deep neural networks, 2022

    Latotzke, C., Balim, B., and Gemmeke, T. Post-training quantization for energy efficient realization of deep neural networks, 2022. URL https://arxiv.org/abs/2210.07906

    2022 arXiv

  12. [20]

    Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024

    Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024. URL https://arxiv.org/abs/2306.02272

    2024 arXiv

  13. [21]

    Norm tweaking: High-performance low-bit quantization of large language models, 2023 a

    Li, L., Li, Q., Zhang, B., and Chu, X. Norm tweaking: High-performance low-bit quantization of large language models, 2023 a . URL https://arxiv.org/abs/2309.02784

    2023 arXiv

  14. [22]

    Fptq: Fine-grained post-training quantization for large language models, 2023 b

    Li, Q., Zhang, Y., Li, L., Yao, P., Zhang, B., Chu, X., Sun, Y., Du, L., and Xie, Y. Fptq: Fine-grained post-training quantization for large language models, 2023 b . URL https://arxiv.org/abs/2308.15987

    2023 arXiv

  15. [23]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024 a

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024 a . URL https://arxiv.org/abs/2306.00978

    2024 arXiv

  16. [24]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving, 2024 b

    Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving, 2024 b . URL https://arxiv.org/abs/2405.04532

    2024 arXiv

  17. [25]

    Qllm: Accurate and efficient low-bitwidth quantization for large language models, 2024 a

    Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. Qllm: Accurate and efficient low-bitwidth quantization for large language models, 2024 a . URL https://arxiv.org/abs/2310.08041

    2024 arXiv

  18. [26]

    Llm-qat: Data-free quantization aware training for large language models, 2023

    Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. Llm-qat: Data-free quantization aware training for large language models, 2023. URL https://arxiv.org/abs/2305.17888

    2023 arXiv

  19. [27]

    Spinquant: Llm quantization with learned rotations, 2024 b

    Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Llm quantization with learned rotations, 2024 b . URL https://arxiv.org/abs/2405.16406

    2024 arXiv

  20. [28]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL https://arxiv.org/abs/1606.06031

    2016 arXiv

  21. [29]

    Patro, S. G. K. and Sahu, K. K. Normalization: A preprocessing stage, 2015. URL https://arxiv.org/abs/1503.06462

    2015 arXiv

  22. [30]

    Omniquant: Omnidirectionally calibrated quantization for large language models, 2024

    Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large language models, 2024. URL https://arxiv.org/abs/2308.13137

    2024 arXiv

  23. [31]

    Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C., Gonzalez, J. E., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu, 2023. URL https://arxiv.org/...

    2023 arXiv

  24. [32]

    A note on approximate hadamard matrices, 2024

    Steinerberger, S. A note on approximate hadamard matrices, 2024. URL https://arxiv.org/abs/2402.13202

    2024 arXiv

  25. [33]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023 arXiv

  26. [34]

    Outliertune: Efficient channel-wise quantization for large language models, 2024

    Wang, J., Yin, Y., Sun, H., Qi, Q., Wang, J., Zhuang, Z., Yang, T., and Liao, J. Outliertune: Efficient channel-wise quantization for large language models, 2024. URL https://arxiv.org/abs/2406.18832

    2024 arXiv

  27. [35]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023

    Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023. URL https://arxiv.org/abs/2304.09145

    2023 arXiv

  28. [36]

    Smoothquant: Accurate and efficient post-training quantization for large language models, 2024

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https://arxiv.org/abs/2211.10438

    2024 arXiv

  29. [37]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information ...

    2022

  30. [38]

    Rptq: Reorder-based post-training quantization for large language models, 2023

    Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. Rptq: Reorder-based post-training quantization for large language models, 2023. URL https://arxiv.org/abs/2304.01089

    2023 arXiv

  31. [39]

    Integer or floating point? new outlooks for low-bit quantization on large language models, 2023

    Zhang, Y., Zhao, L., Cao, S., Wang, W., Cao, T., Yang, F., Yang, M., Zhang, S., and Xu, N. Integer or floating point? new outlooks for low-bit quantization on large language models, 2023. URL https://arxiv.org/abs/2305.12356

    2023 arXiv

  32. [40]

    Atom: Low-bit quantization for efficient and accurate llm serving, 2024

    Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate llm serving, 2024. URL https://arxiv.org/abs/2310.19102

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.