Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Is (Selective) Round-To-Nearest Quantization All You Need?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that round-to-nearest quantization, with selective 8-bit modules and Marlin kernels, matches advanced calibration-based quantizers while running faster.

desk verdict Useful engineering work, but the headline accuracy claims are overreached and the selection procedure is not properly held out. read the letter →

arxiv 2505.15909 v1 pith:STECTTYN submitted 2025-05-21 cs.LG

Alex Kogan This is my paper
classification cs.LG
keywords round-to-nearestquantizationselectivepost-traininglargelanguagemodelservingMarlinkernelszero-shotaccuracytokengenerationlatencydata-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Round-to-nearest (RTN) is the simplest weight-quantization scheme: divide each weight by a scale, round to the nearest integer, and clamp to the target range. The paper argues that RTN, long dismissed in favor of calibration-based methods, becomes a viable and practical choice for serving large language models when two ingredients are added. First, selective quantization, which keeps a small subset of layers and linear modules at 8-bit precision while the rest stay at 4 bits, recovers almost all accuracy lost by plain RTN-4; at roughly 5 bits per weight on average, RTN-~5 matches or exceeds the average zero-shot accuracy of GPTQ, AWQ, and BitsAndBytes on all tested models. Second, Marlin-based kernels let RTN generate tokens up to 37% faster than the 16-bit baseline. If correct, this means a cheap, data-free, on-the-fly quantization path can serve as a practical alternative to more complex pipelines.

What carries the argument

The machinery is group-wise RTN with a symmetric scale factor $S = 2\max(|r|)/(2^b - 1)$, applied per group of weights to contain outliers, combined with a two-dimensional selection over the Transformer stack: horizontal selection chooses which layers stay at 8 bits, vertical selection chooses which of the four linear modules per layer stay at 8 bits, and hybrid selection combines the two into the RTN-~4, RTN-~5, and RTN-~6 variants. The other half of the machinery is a Marlin-style mixed-precision GEMM kernel that reshuffles quantized weights into an interleaved layout and hides dequantization behind memory access, plus a dual-path switch to plain 16-bit GEMM when the activation tensor is large, avoiding the kernel's slowdown on long prompts.

What would settle it

Evaluate the exact selective recipe on a held-out model and benchmark suite that were not used to choose which layers and modules stay at 8 bits; if the chosen modules no longer reduce wikitext perplexity or the average zero-shot accuracy drops below the GPTQ, AWQ, and BitsAndBytes numbers, the paper's recovery-rate claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that plain RTN-4's weakness is not intrinsic to rounding but concentrated in a small part of the model: the Transformer layer position that matters is model-dependent, and within the influential layers, leaving the QKV projection and the two feed-forward linear modules at 8 bits while quantizing the rest to 4 bits is enough to recover most accuracy. For Llama-3.1/3.3 70B, keeping modules 1+3+4 of the first layer at 8 bits and the rest at 4 bits yields a near-perfect recovery rate while adding less than 0.05 bits per weight on average; the coarser RTN-~5 variant (a quarter of layers at 8 bits) matches or exceeds the average zero-shot accuracy of GPTQ, AWQ, and BitsAndBytes on every tested model. At the same time, an RTN implementation built on Marlin kernels, a mixed-precision GEMM kernel originally designed for GPTQ, improves token generation throughput by up to 37% at 4 bits and up to 25% at 8 bits relative to the 16-bit baseline, with additional gains over the advanced methods at small batch sizes. The paper reads this as evidence that data-free RTN plus selective precision is a practical, cheaper alternative to calibration-dependent quantization.

Load-bearing premise

The selection of which layers and modules to keep at 8 bits is made by inspecting perplexity curves on the same wikitext data used for evaluation, so the paper's recovery-rate claims rest on the assumption that this manual, evaluation-informed choice generalizes to unseen tasks and models.

Editorial extensions

If this is right

  • Models can be quantized on the fly while loading, with no calibration data, so quantization can be applied on machines that cannot hold the full-precision model.
  • For very large models, a few extra 8-bit modules in selected layers close most of the 4-bit accuracy gap at a memory cost under 0.05 bits per weight.
  • RTN-~5 provides a practical tradeoff among accuracy, size, and speed, with roughly five bits per weight on average and faster token generation than the 16-bit baseline.
  • Token-generation speedups from the Marlin kernels are largest at small batch sizes, which is the typical interactive serving regime.
  • Varying how many layers and modules stay at 8 bits tunes the model's effective precision continuously between about 4 and 8 bits per weight.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's manual selection of layers and modules is made after looking at wikitext perplexity and then evaluated on the same tasks, so the strongest test of the approach is to hold out validation data and automate the selection, which the paper itself names as open work.
  • Selective quantization likely transfers to mixture-of-experts models, where individual experts rather than layers could be kept at higher precision, but the paper only offers this as a future direction.
  • The dual-path crossover point at an activation dimension of 1024 and the reported speedup magnitudes are hardware- and shape-dependent, so the 37% figure should be read as a demonstration on the tested H100 systems rather than a universal bound.
  • Because RTN is data-free and group-wise, the same selective scheme could be applied with 2- or 3-bit groups plus a few high-precision modules, though the paper does not test such configurations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that round-to-nearest (RTN) quantization, when combined with selective assignment of 8-bit precision to a few layers/modules and with a Marlin-kernel-based implementation, is a practical alternative to more sophisticated calibration-based PTQ methods such as GPTQ, AWQ, and BitsAndBytes. The authors first compare uniform RTN-8 and RTN-4 against these baselines on Llama-3.1/3.2 and several non-Llama models, then study horizontal (layer) and vertical (module) selection using wikitext perplexity curves, and finally report accuracy and token-generation latency for the resulting RTN-~4, RTN-~5, and RTN-~6 variants. The main claims are that RTN-~5 matches or exceeds GPTQ/AWQ/BnB in average zero-shot accuracy, that keeping modules 1+3+4 of one layer in 8-bit gives 'nearly perfect recovery' for Llama-3.1-70B at RTN-~4, and that the Marlin-based RTN implementation reduces per-token latency by up to 37% at 4-bit precision relative to the baseline. The paper explicitly lists the manual nature of layer/module selection as a limitation and says the code changes to vLLM are intended to be open-sourced but are not yet available.

Significance. If the accuracy and latency claims hold under an unbiased evaluation protocol, the paper would make a useful practical point: a data-free, calibration-free method can be competitive with calibration-based PTQ for LLM serving. The positive aspects are the breadth of the empirical study (five Llama/Phi/Dolly models up to 405B parameters), the integration with vLLM, the use of Marlin kernels with a dual-path activation-size heuristic, and the careful attention to prompt-chunking effects on GEMM behavior. The throughput comparison in Table 3 is genuinely informative, and the observation that Marlin kernels degrade for large activation dimensions is worth reporting. However, the central accuracy claim is currently difficult to separate from evaluation-informed configuration selection, because the same data that guide the choice of which layers and modules to keep at 8-bit precision are used to report the headline comparisons. A held-out or nested evaluation is needed before the 'matches or exceeds' statement can be taken at face value.

major comments (3)
  1. [§4.3 and Table 2] The headline comparison in Table 2 is a selected maximum rather than an unbiased accuracy estimate. Section 4.1 fixes the horizontal choice from the wikitext perplexity curves in Figure 1, and Section 4.2 fixes the vertical choice from the 16-way module search in Figure 2; Section 4.3 then evaluates the chosen RTN-~5/RTN-~6 configurations on the same wikitext set and the same six zero-shot tasks. Because no validation split and no separate selection criterion are used, the configuration is chosen by inspecting the very data on which the final accuracy is reported. The Limitations section acknowledges that selection is manual, but this is load-bearing for the main claim that RTN-~5 'matches or exceeds' GPTQ/AWQ/BnB. Please add a held-out evaluation protocol (for example, a perplexity-only selection set with zero-shot tasks reserved for the final comparison, or a nested procedure), or explicitly re-frame the quantitative comparisons as exploratory and model-specific.
  2. [Table 2, §4.3] The sentence that RTN-~5 'matches or exceeds performance of GPTQ, AWQ and BnB for all considered models' is not supported by the reported numbers. For Llama-3.1-8B, RTN-~5 averages 64.93 against 65.59 for BnB; for Llama-3.1-70B, 71.32 against 71.54 for AWQ; and for Llama-3.2-3B, 57.54 against 58.16 for BnB. These gaps may be within run-to-run noise, but no error bars, repeated runs, or statistical tests are provided, so the claim should be softened or substantiated with an error analysis.
  3. [Abstract, §6, and Table 2 (Llama-3.1-70B row)] The claim of 'nearly perfect recovery rate' for RTN-~4 on Llama-3.1-70B is not uniformly supported. The average zero-shot accuracy is close (71.55 vs. baseline 71.67), but the wikitext perplexity for RTN-~4 is 4.91 versus 3.94 for the baseline (and 8.19 for uniform RTN-4), a substantial relative degradation. Since the paper reports both accuracy and perplexity as measures of recovery, the claim needs to be qualified by defining the recovery rate, reporting per-task and per-metric results, and stating which metric the 'nearly perfect' language refers to.
minor comments (5)
  1. [Title page] The running title contains a typo: 'Round-To-Nearest Quantization All Y ou Need?' should read 'All You Need?'.
  2. [§2.1] There is a duplicated article in 'minimize the the mean-squared error of quantization'; please correct this typo.
  3. [§4.3] The text uses 'LLama-3.1 70B' with an unusual capitalization; please standardize to 'Llama-3.1-70B' for consistency with the tables.
  4. [§5 and Table 3] The speedup percentages in the abstract and §1 should be stated as latency reduction percentages, since Table 3 reports msec/token and the 37% figure corresponds to a 37% reduction in per-token latency rather than a 1.37x throughput multiplier.
  5. [Introduction, footnote 1] The paper says 'We intend to open-source all our changes to vLLM' but no code or repository link is provided in the submission. For a systems-oriented claim, please state the artifact availability status explicitly (e.g., a URL or a statement that code will be released upon publication).

Circularity Check

1 steps flagged · score 4.0 of 10

The selective-quantization recipe is read off the same wikitext perplexity curves that appear again in the final accuracy tables, so the reported recovery is partly a selected maximum rather than an independent out-of-sample prediction.

  1. fitted input called prediction [Section 4.3, Table 2; choices made from Figures 1 and 2 in Sections 4.1 and 4.2]
    "Following the observations above, we experiment with a hybrid selections strategy, in which we select X layers (first for all models but Llama3.1-405B and middle for Llama3.1-405B, based on the results in Figure 1) and modules ’1+3+4’ in those selected layers are quantized to 8 bit, while the rest of the model is quantized into 4 bit. ... The recovery rate of RTN-˜5 improves over RTN-4, and although it falls slightly below that of RTN-˜6, the accuracy of RTN-˜5 matches or exceeds performance of GPTQ, AWQ and BnB for all considered models."

    The horizontal choice (first, middle, or last layers) and the vertical choice (modules 1+3+4) are selected from wikitext perplexity sweeps in Figures 1 and 2. The same wikitext perplexity is then reported in Table 2, and the recovery-rate conclusions are drawn from that table. With 3x16 configurations scanned per model and no validation split, the RTN-~4, RTN-~5, and RTN-~6 results are best-case selections with respect to the very signal used to choose them, not independent confirmations of a fixed recipe. The six zero-shot tasks are not directly optimized by the selection, so the claim retains some independent content, but the headline 'matches or exceeds' summary is optimistically biased.

full rationale

This is an empirical study with no formal derivation, so the classic equation-level circularity patterns do not apply. The RTN formula, the Marlin-kernel integration, and the throughput measurements are self-contained implementation results with external benchmarks and do not reduce to the paper's inputs. The main circular element is the selective-quantization configuration: it is chosen per model from wikitext perplexity curves and then evaluated on the same perplexity metric and on nearby zero-shot tasks, without a held-out validation split. This makes the reported accuracy recovery of RTN-~5 and RTN-~4 a selected maximum over a nontrivial configuration search, not a prediction from an a-priori rule. The paper's own limitation statement acknowledges the manual selection. The score is 4 rather than higher because the zero-shot accuracy tasks and the latency results are not directly fitted by the selection procedure, and the paper is transparent about the manual nature of the recipe; it is a partial circularity in the central accuracy claim, not a fully tautological one.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The ledger is small: no new physical or mathematical entities, but the method's practical success depends on several hand-picked choices and on the representativeness of the evaluation suite. The most consequential item is the first free parameter, because the layer/module selection is tuned to the test data, and the paper's recovery claims would be on firmer ground if the recipe were fixed a priori or validated on held-out tasks.

free parameters (3)
  • layer/module selection for 8-bit precision = X=1/4 or 1/2 of first layers (middle for 405B); modules 1+3+4
    Chosen after inspecting wikitext perplexity curves in Figures 1 and 2; no held-out validation, so the selection is effectively fitted to the evaluation set.
  • dual-path activation-dimension threshold = 1024
    Hand-picked in Section 5 as the point where Marlin loses its speed advantage; affects the reported throughput numbers.
  • per-group scaling group size g = 128 (2^7)
    Standard choice inherited from Marlin and GPTQ; not fitted in this paper but a free hyperparameter of the method.
assumptions (3)
  • domain assumption Wikitext perplexity and six zero-shot tasks are sufficient proxies for LLM accuracy
    Section 3 uses only these metrics to conclude RTN recovery and to compare against GPTQ/AWQ/BnB; the paper does not evaluate on broader benchmarks such as MMLU or GSM8K.
  • domain assumption Marlin kernel speedups measured on individual GEMM shapes transfer to end-to-end vLLM serving
    Section 5 latency results assume that the kernel-level speedup survives integration, the dual-path switch, and the non-default max_num_batched_tokens=16K setting.
  • ad hoc to paper The manual, evaluation-informed selection does not invalidate the 'data-free' characterization
    RTN itself needs no calibration data, but the selective recipe is chosen from the same test data used to measure success; the paper calls the method data-free while relying on evaluation-set feedback for the recipe.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is (Selective) Round-To-Nearest Quantization All You Need?." pith.science (2026). https://pith.science/paper/STECTTYN

@misc{pith2026250515909,
  author       = {Pith},
  title        = {Pith review of: Is (Selective) Round-To-Nearest Quantization All You Need?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/STECTTYN}},
  note         = {Machine review of arXiv:2505.15909}
}
read the original abstract

Quantization became a necessary tool for serving ever-increasing Large Language Models (LLMs). RTN (Round-to-Nearest) is perhaps the simplest quantization technique that has been around well before LLMs surged to the forefront of machine learning (ML) research. Yet, it has been largely dismissed by recent and more advanced quantization methods that claim superiority over RTN in nearly every aspect of performance. This work aims to dispel this established point of view, showing that RTN is not only much cheaper to apply, but also its token generation throughput can be better than and accuracy can be similar to more advanced alternatives. In particular, we discuss our implementation of RTN based on the recent Marlin kernels and demonstrate how the accuracy of RTN can be gradually improved by selectively increasing the data precision format of certain model layers and modules. Based on our results, we argue that RTN presents a viable and practical choice for quantizing LLMs.

Figures

Figures reproduced from arXiv: 2505.15909 by the authors.

Figure 1
Figure 1. Perplexity of various models on the wikitext dataset (lower is better). Note that the range of the x-axis (number of layers) depends on the model. The y-axis does not start at zero for better readability. systematic study evaluating LLM accuracy across variety of tasks. 4 Selective quantization There are two dimensions in the Transformer archi￾tecture along which the quantization can be con￾sidered. First, there is … view at source ↗
Figure 2
Figure 2. Perplexity of various models on the wikitext dataset (lower is better). The y-axis does not start at zero for better readability. includes modules ’1’, ’2’, and ’4’. The perplexity results for various models and all possible selection modes can be found in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Speedup over the default GEMM implemen￾tation in PyTorch for tensor shapes corresponding to linear modules of the LLama3.1-8B model the LLama3.1-8B model are depicted in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [5]

    https://unsloth.ai/blog/ deepseekr1-dynamic

    Run DeepSeek R1 Dynamic 1.58-bit. https://unsloth.ai/blog/ deepseekr1-dynamic. Accessed on 03/01/2025. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi

    work page 2025

  2. [6]

    CoRR, abs/1806.08342

    Quantizing deep convolutional networks for efficient inference: A whitepaper. CoRR, abs/1806.08342. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gon- zalez, Hao Zhang, and Ion Stoica

    arXiv

  3. [8]

    https://ai.meta.com/blog/meta- llama-3-1

    Introducing Llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta- llama-3-1. Accessed on 02/20/2025. Meta

    work page 2025

  4. [9]

    https://ai.meta.com/blog/llama- 4-multimodal-intelligence

    The Llama 4 herd: The begin- ning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama- 4-multimodal-intelligence. Accessed on 04/09/2025. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi

    work page 2025

  5. [11]

    https://neuralmagic.com/blog/ introducing-machete-a-mixed-input-gemm- kernel-optimized-for-nvidia-hopper-gpus

    Introducing Machete, a Mixed-Input GEMM Kernel Optimized for NVIDIA Hopper GPUs. https://neuralmagic.com/blog/ introducing-machete-a-mixed-input-gemm- kernel-optimized-for-nvidia-hopper-gpus . Accessed on 02/07/2025. Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Reza Yazdani Aminabadi, Yuxiong He, Olatunji...

    work page 2025

  6. [13]

    CoRR, 2406.09904

    QQQ: Quality quattuor-bit quantization for large language models. CoRR, 2406.09904. Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai- Man Cheung, and Pascal Frossard

    arXiv

  7. [2016]

    Preprint, arXiv:1609.07843

    Pointer sentinel mixture mod- els. Preprint, arXiv:1609.07843. Meta

    arXiv

  8. [2018]

    CoRR, abs/1803.05457

    Think you have solved question an- swering? try ARC, the AI2 reasoning challenge. CoRR, abs/1803.05457. DeepSeek-AI

    arXiv

Show all 13 references
  1. [2019]

    CoRR, abs/1907.10641

    WINOGRANDE: an ad- versarial winograd schema challenge at scale. CoRR, abs/1907.10641. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant

    1907 arXiv

  2. [2021]

    CoRR, abs/2103.13630

    A survey of quantization methods for efficient neural network inference. CoRR, abs/2103.13630. Daniel Han and Michael Han

    arXiv

  3. [2023]

    CoRR, 2312.08583

    Ze- roquant(4+2): Redefining llms quantization with a new fp6-centric strategy for diverse generative tasks. CoRR, 2312.08583. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han

    arXiv

  4. [2024]

    CoRR, 2408.11743

    Marlin: Mixed- precision auto-regressive parallel inference on large language models. CoRR, 2408.11743. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, ...

    arXiv

  5. [2025]

    CoRR, abs/2501.12948

    Deepseek-r1: Incentivizing reason- ing capability in LLMs via reinforcement learning. CoRR, abs/2501.12948. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.