Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

QuEST: Stable Training of LLMs with 1-Bit Weights and Activations

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read QuEST claims that LLMs can be trained stably at 1-bit weights and activations, and that 4-bit precision is Pareto-optimal for accuracy at a given model size.

desk verdict QuEST delivers a credible QAT method with stable 1-bit training and strong 4-bit results, but the headline 4-bit-optimality claim rests on a scaling-law extrapolation that needs more support before I'd fully trust it. read the letter →

arxiv 2502.05003 v2 pith:PCJVVNDK submitted 2025-02-07 cs.LG

Andrei Panferov , Jiale Chen , Soroush Tabesh , Roberto L. Castro , Mahdi Nikdan , Dan Alistarh This is my paper · ORCID
classification cs.LG
keywords quantization-awaretraining1-bitweightsandactivationsHadamardtransformtrustgradientestimatorscalinglawslow-precisionLLMINT4inferencePareto-optimalprecision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QuEST is a quantization-aware training method that claims to keep large language models training stably when both weights and activations are reduced to as few as 1 bit, and to make 4-bit weights and activations the best accuracy-per-model-size choice in the overtrained regime. This matters because low-precision training directly cuts memory and inference cost, and prior work had put the practical floor at around 8 bits. The method combines a Hadamard transform with MSE-optimal Gaussian fitting on the forward pass, and a trust gradient estimator that masks gradients for entries whose quantization error is large. If the claims hold, models trained with QuEST can match or beat much larger BF16 baselines at the same memory footprint.

What carries the argument

Two components carry the method: a forward quantization step that applies a fast Hadamard transform to weights and activations, then projects the transformed values onto a uniform grid whose scale minimizes the L2 error against a standard Gaussian distribution, and a trust gradient estimator that, on the backward pass, multiplies the gradient in the Hadamard domain by a binary mask zeroing entries whose quantization error exceeds half a quantization interval. The Hadamard transform makes the trust mask less persistent in the original weight domain, allowing gradient flow to all weights while bounding the gradient estimation error from outliers.

What would settle it

Train 3-bit and 4-bit QuEST models of the same architecture at, say, 430M and 800M parameters with D/N set to 500 or 1000, repeated over several seeds; if 3-bit achieves equal or lower validation loss at the same memory and compute budget, the claimed ordering eff(4)/4 > eff(3)/3 is contradicted in the high-data regime.

Watch

Extended reading notes

Core claim

The paper's central claim is that a quantization-aware training method can train Llama-style models stably down to 1-bit weights and activations, and that the resulting scaling laws make 4-bit weights and activations Pareto-optimal: at a fixed model memory budget, a QuEST W4A4 model achieves lower validation loss than both higher-precision and lower-precision alternatives. This advances the prior state of the art, where 8-bit was identified as the optimal QAT precision. The paper reports stable convergence across model sizes from 30M to 1.6B parameters, with W4A4 models outperforming BF16 baselines almost four times larger in size, and it extends the same framework to FP4 and 2:4-sparse INT4 formats.

Load-bearing premise

The 4-bit-optimality conclusion rests on extrapolating a scaling law, fitted on models up to 800M parameters and mostly at 100 tokens per parameter, to larger models and much higher data ratios.

Editorial extensions

If this is right

  • QuEST models trained at W1A1 converge stably across Llama-style architectures from 30M to 1.6B parameters, following a consistent scaling law.
  • At fixed inference memory, QuEST W4A4 produces lower C4 validation loss than BF16 models roughly four times larger.
  • In the overtraining regime where the tokens-per-parameter ratio is high, the fitted precision efficiency eff(P)/P peaks at INT4, making 4-bit the recommended pre-training precision for runtime-constrained deployment.
  • Weight-only QuEST also yields stable scaling laws, with 2-bit weights appearing Pareto-dominant.
  • The framework extends to FP4 and 2:4-sparse INT4, where dense INT4 still shows the highest parameter efficiency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 4-bit optimality result extrapolates to much larger models, hardware makers could prioritize fast INT4 matrix units for training rather than treating low precision as an inference-only optimization.
  • Since the Hadamard transform changes the fitted coefficients but not the scaling-law ranking, the trust estimator may be the load-bearing component; isolating the estimator alone at larger scale would test this directly.
  • The same forward-backward recipe could transfer to other architectures such as encoder-decoder models or vision transformers, a direction the authors identify as future work.
  • The trust mask zeros a fraction of weights each step, and its interaction with adaptive per-parameter optimizers like AdamW may introduce dynamics the paper does not analyze in depth.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces QuEST, a quantization-aware training (QAT) method for LLMs that combines Hadamard normalization with MSE-optimal grid fitting and a new 'trust' gradient estimator that masks gradients for entries with large quantization error. The authors train Llama-family models from 30M to 1.6B parameters on C4 at precisions from 1 to 8 bits, reporting stable convergence down to W1A1, improved validation loss relative to STE, LSQ, QuaRot, and AdaBin, and scaling laws from which they conclude that INT4 is the Pareto-optimal precision for accuracy per unit of model memory and inference cost in the overtraining regime. They also provide GPU kernels with measured speedups.

Significance. If the central claims hold, the paper advances the state of the art: it shows, for the first time, that QAT can train LLMs stably at 1-bit weights and activations and that the optimal precision for overtrained models is lower than the 8-bit optimum reported by Kumar et al. (2024). The empirical comparisons in Table 1, the stability analysis in Figures 2 and 10, and the scaling-law fits across precisions are concrete and reproducible, and the release of code and kernels is a strong contribution. The main caveat is that the headline '4-bit optimal' claim is derived from a scaling-law extrapolation with limited data and no uncertainty quantification; the direct fixed-compute validation in Figure 13 is a single point. The method itself is clearly valuable even if the optimal-precision conclusion is later refined.

major comments (3)
  1. [4.3-4.4, Appendix C.1] Equation (5) is fitted with eff(P) estimated from runs at D/N=100 for N up to 800M, plus runs at D/N in {25,50} for N up to 100M and P in {2,3,4}; each configuration appears to be a single run with no confidence intervals. The conclusion in Section 4.4 that P=4 is optimal in the D/N->infinity regime relies on the fitted eff(P)/P values, e.g., Table 2 and Figure 3, whose margins (0.175 for P=4 vs 0.143 for P=3) may be within fitting noise. Moreover, at D/N=100 the two power-law terms in Equation (5) have the same N-dependence, so eff(P) and B are partly confounded. Please add uncertainty quantification (e.g., bootstrap over the fits or multiple seeds per configuration) and additional fixed-compute runs across a wider range of model sizes to support the extrapolation.
  2. [4.4, Section 5] The optimality analysis uses F=N P as the runtime cost, assuming a linear speedup in precision, but the measured end-to-end speedups in Figures 5-6 are substantially sublinear (1.3-1.5x for the 1.6B W4A4 model). Since the abstract describes INT4 as Pareto-optimal 'in terms of ... inference cost', this conclusion is not established under the actual measured inference cost; the paper should either use the measured speedups in the Pareto analysis or explicitly restrict the claim to the idealized linear-FLOPs proxy.
  3. [3.3, Appendix A.2] The outer trust scaling factor s* is tuned on 30M models (Figure 9) and then applied to all 1-bit runs, including the largest models in Figure 1. No evidence is given that the optimal s* is independent of model size; if it varies, the W1A1 scaling law and the eff(1) entry in Table 2 are affected. Please report the sensitivity of s* with model size or justify its transferability.
minor comments (5)
  1. [4.1, Table 4, Introduction] The introduction states that models up to 1.6B parameters are trained, but Section 4.1 lists model sizes only up to 800M and Table 4 omits the 1.6B configuration; please reconcile these statements.
  2. [Figure 3] Figure 3 plots several series with markers that are difficult to distinguish in black-and-white print; please use clearer labels or a table.
  3. [3.1] The sentence 'we believe we are the first to harness it for QAT' is a novelty claim without a citation; either provide a reference or soften the wording.
  4. [3.2] The trust estimator is motivated by the squared gradient difference in Equation (2), but no formal convergence guarantee is given; a one-sentence statement in the Discussion that this is an empirical heuristic would be helpful.
  5. [Table 2] The eff(P) values are reported to two decimal places; given the use of these values to select P=4, reporting more digits or confidence intervals would aid reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

The 4-bit optimality claim in the overtraining regime is read off fitted eff(P)/P values, but direct fixed-compute validation and raw Pareto data keep the central method self-contained.

  1. fitted input called prediction [Section 4.4 'Finding the Optimal Precision'; Equation (6) and Figure 3, using Table 2]
    "Recall that we can estimate eff(P) from the empirical scaling law (obtained in Section 4.3 and shown in Table 2). Thus, we can calculate eff(P)/P for any precision. Figure 3 suggests that 4-bit appears to be the optimal pre-training precision in this regime."

    The claimed optimum P=4 in the overtraining regime is selected as the argmax of eff(P)/P, where eff(P) are free parameters fitted to the same QuEST loss curves used to build Equation (5). The ordering of precisions is therefore a direct function of the fitted values rather than an independent prediction; the conclusion would change only if the fit changed. This is partially mitigated by the fixed-compute validation in Figure 13 and by the raw Pareto comparison in Figure 1, which do not depend on the scaling-law extrapolation.

full rationale

The QuEST method itself, including Hadamard preprocessing, MSE-optimal Gaussian fitting, trust masking, and stable W1A1 training curves, is directly empirical and self-contained, with kernel support and comparisons to STE, LSQ, QuaRot, and AdaBin. The one near-circular element is the overtraining-regime optimality claim in Section 4.4: P=4 is chosen by comparing eff(P)/P, with eff(P) fitted to the same loss data (Equation 5, Table 2). That specific prediction reduces to a comparison of fitted parameters, and the fitted eff(P) values carry no uncertainty intervals; potential confounding between B and eff(P) at fixed D/N=100 is a correctness risk, not circularity. The paper does provide an independent fixed-compute check (Figure 13) and raw Pareto evidence (Figure 1), so the central claim is not forced solely by the fit. The scaling-law ansatz is attributed to Frantar et al. (2025), a prior work with overlapping authorship, but the present paper re-fits the parameters to its own runs and validates the conclusion directly; this is a minor self-citation, not a load-bearing circularity.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The method introduces no invented physical or ontological entities. The main fitted quantities are the precision efficiencies eff(P) from the scaling-law fit and the 1-bit trust-scale s*. The Gaussian fitting scale alpha* is derived from a fixed N(0,1) target, not from training data, so it is counted under axioms rather than as a free parameter.

free parameters (8)
  • eff(1) = 0.02
    Fitted scaling-law efficiency for 1-bit QuEST on C4, used in the precision-optimality calculation.
  • eff(2) = 0.16
    Fitted scaling-law efficiency for 2-bit QuEST on C4, used in the precision-optimality calculation.
  • eff(3) = 0.43
    Fitted scaling-law efficiency for 3-bit QuEST on C4, used in the precision-optimality calculation.
  • eff(4) = 0.70
    Fitted scaling-law efficiency for 4-bit QuEST on C4, used in the precision-optimality calculation.
  • eff(8) = 1.02
    Fitted scaling-law efficiency for 8-bit QuEST on C4, used in the precision-optimality calculation.
  • eff(16) = 1.00
    Reference efficiency for BF16, normalized to 1.0, used in the precision-optimality calculation.
  • Outer trust scaling factor s* = 1.30
    Tuned by sweep on a 30M W1A1 model in Appendix A.2, then applied to all 1-bit runs; it changes the outer trust region in the 1-bit case.
  • Scaling-law coefficients A, B, E, alpha, beta = Not reported in full
    Fitted by Huber loss to C4 validation loss according to the Hoffmann et al. procedure; values not listed in the paper, but they define the fitted curve from which eff(P) and optimality are extracted.
assumptions (4)
  • domain assumption The transformer loss is gamma-smooth, so the small quantization-error term is bounded by gamma^2 T^2 |S_small|.
    Invoked in Section 3.2 to motivate the trust estimator; large-scale transformer losses are not known to satisfy global smoothness.
  • domain assumption After RMS normalization, weight and activation tensors can be treated as N(0,1) plus long tails; after Hadamard transform they are close enough to Gaussian that an MSE-optimal Gaussian grid is near-optimal.
    Section 3.1 relies on this to fix alpha* analytically instead of learning it; it is empirically motivated but not proven.
  • standard math Hadamard matrices are orthogonal and fast transforms exist, so applying HT to both matmul operands preserves the output.
    Section 3.3 leverages orthogonality; this part is mathematically exact.
  • domain assumption The scaling law L(N,D,P)=A/(N*eff(P))^alpha + B/D^beta + E is accurate across precisions and extrapolates to the overtrained regime.
    Equation 5 is the basis for the 4-bit optimality conclusion; it is fitted to a limited grid and is not derived from first principles.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QuEST: Stable Training of LLMs with 1-Bit Weights and Activations." pith.science (2026). https://pith.science/paper/PCJVVNDK

@misc{pith2026250205003,
  author       = {Pith},
  title        = {Pith review of: QuEST: Stable Training of LLMs with 1-Bit Weights and Activations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PCJVVNDK}},
  note         = {Machine review of arXiv:2502.05003}
}
read the original abstract

One approach to reducing the massive costs of large language models (LLMs) is the use of quantized or sparse representations for training or deployment. While post-training compression methods are very popular, the question of obtaining even more accurate compressed models by directly training over such representations, i.e., Quantization-Aware Training (QAT), is still open: for example, a recent study (arXiv:2411.04330) put the "optimal" bit-width at which models can be trained using QAT, while staying accuracy-competitive with standard FP16/BF16 precision, at 8-bits weights and activations. We advance this state-of-the-art via a new method called QuEST, for which we demonstrate optimality at 4-bits and stable convergence as low as 1-bit weights and activations. QuEST achieves this by improving two key aspects of QAT methods: (1) accurate and fast quantization of the (continuous) distributions of weights and activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust gradient estimator based on the idea of explicitly minimizing the error between the noisy gradient computed over quantized states and the "true" (but unknown) full-precision gradient. Experiments on Llama-type architectures show that QuEST induces stable scaling laws across the entire range of hardware-supported precisions, and can be extended to sparse representations. We provide GPU kernel support showing that models produced by QuEST can be executed efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.

Figures

Figures reproduced from arXiv: 2502.05003 by the authors.

Figure 1
Figure 1. The scaling law induced by QuEST when training Llama￾family models from 30 to 1.6B parameters on C4, with quan￾tized weights and activations from 1 to 4 bits, in the 100 to￾kens/parameter regime (harder compression uses proportionally more data at fixed memory). QuEST allows for stable training at 1- bit weights and activations (W1A1), and the QuEST W4A4 model is Pareto-dominant relative to BF16, with lower loss at … view at source ↗
Figure 2
Figure 2. Gradient alignment comparison for a 30M Llama model after training on 2.7B tokens in 8-bit precision. two Hadamard Transforms on the forward pass and two Inverse Hadamard transforms on the backward pass. For a Transformer model (Vaswani, 2017) with d blocks and hidden dimension h, and a batch containing b tokens, the MatMul complexity of the forward pass can be estimated as: b × d × h 2 . Then, the asymptotic cost o… view at source ↗
Figure 3
Figure 3. Illustration of the efficiency factors eff(P)/P, arising from our analysis, for different numerical precisions P, formats (INT, FP, INT+sparse) and methods. Higher is better. QuEST INT4 appears to have the highest efficiency. illustrating loss vs. model size. First, we observe that, remarkably, QuEST provides stable training down to 1-bit weights and activations, across model sizes, following a stable scaling law. S… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Additional scaling laws induced by QuEST: (a, left) compares INT, FP, and INT+sparse formats at 4-bit precision, (b, middle) shows the scaling laws for weight-only quantization, where 2-bit appears to be Pareto-dominant, while (c, right) shows that trust estimation ben…
Figure 5
Figure 5. Figure 5: Per-layer speedups for QuEST INT4 vs BF16, on a single RTX 4090 GPU. The results take into account quantiza￾tion/dequantization costs for QuEST, and include the cost of the Hadamard transform (orange bar). We present results for the 1.6B 4-bit QuEST model we trained, a…
Figure 7
Figure 7. Figure 7: Fraction of weights for which Mα∗ = 0 as a function of number of training iterations for a 30M model trained with QuEST. Moreover, we looked at the percentage of masked elements at a fixed iteration in the past, that remain masked at a fixed later iteration. We plot th…
Figure 8
Figure 8. Figure 8: Fraction of masked values retained from an old iteration to a new iteration for a 30M model trained with QuEST W8A8. A.2. The 1-bit Case To determine the optimal outer trust scaling factor s ∗ , discussed in Section 3.3, we conduct a sweep over s, varying the outer siz…
Figure 9
Figure 9. Figure 9: Performance of QuEST as a function of the outer trust scaling factor s for a 30M model pretraining. A.3. Zero-shot Evaluation of QuEST Models To assess the effectiveness of QuEST beyond perplexity, we conducted a comprehensive zero-shot evaluation on five established c…
Figure 10
Figure 10. Figure 10: Training loss curves for a 30M model trained on 3B tokens with W4A4 bitwidth, comparing QuEST (ours), LSQ, PACT, and BF16. (a) Full training loss curves, showing that QuEST closely follows BF16 and consistently outperforms LSQ, while PACT struggles with high loss. (b)…
Figure 11
Figure 11. Figure 11: Hyperparameter search for PACT on a 30M parameter model with 4-bit weights and activations, trained on 10% of the dataset. The search explores different values for learning rate scaling (LR Scale) and alpha weight decay, with validation loss indicated by the color gra…
Figure 12
Figure 12. Figure 12: Scaling law (5) fit for 3 and 4 bit QuEST with tokens/parameters ratios in {25, 50, 100}. As described in Section 4.3, we closely follow the fitting procedure of Hoffmann et al. (2022) for the scaling law (5) fitting. Specifically, we copied their grid of initializati…
Figure 13
Figure 13. Figure 13: Comparison of different QuEST precisions P at a fixed model size and training compute. The results in Section 4.4 suggest that 4-bit training is optimal in the D/N → ∞ regime. Here, we use the fitted scaling law (5) to verify that 4 bit is also close to optimal for D/…
Figure 14
Figure 14. Figure 14: Different QuEST precision performance as a function of tokens-to-parameters ratio at a fixed model memory footprint. The gray line indicates a 4-bit optimality threshold. training is optimal can, in fact, be easily achieved in practice. We validate this in practice by…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FP4 All the Way: Fully Quantized Training of LLMs

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A 7-billion parameter LLM trained almost entirely in 4-bit float precision matches a standard 16-bit baseline after a brief quantization-aware fine-tuning phase.

  2. Unified Scaling Laws for Compressed Representations

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A representation capacity derived from Gaussian fitting error predicts the training efficiency of sparse, quantized, and hybrid compressed models, and this capacity approximately multiplies across combined compression types.

  3. Scaling Law for Quantization-Aware Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.

  4. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

Reference graph

Works this paper leans on

37 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [7]

    I.-J., Srini- vasan, V ., and Gopalakrishnan, K

    Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srini- vasan, V ., and Gopalakrishnan, K. Pact: Parameterized clipping activation for quantized neural networks.arXiv preprint arXiv:1805.06085,

    arXiv

  2. [8]

    2:4 INT4

    As we can see, for the run without the HT, around 69% of masked elements at iteration 6000 (roughly halfway through training) remain masked at iteration 10000 (towards the end of the training). This percentage is more than twice as small for the run with the HT at 30%. This implies that the HT makes masks less persistent, as expected. In addition, we note...

    work page 2000

  3. [9]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    Dettmers, T., Svirschevski, R., Egiazarian, V ., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078,

    arXiv

  4. [11]

    org/abs/1702.03118

    URL https://arxiv. org/abs/1702.03118. Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization.arXiv preprint arXiv:1902.08153,

    arXiv 1902

  5. [12]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    arXiv

  6. [13]

    Scaling Laws for Sparsely-Connected Foundation Models

    URL https://arxiv.org/abs/ 2309.08520. Frantar, E., Evci, U., Park, W., Houlsby, N., and Alistarh, D. Compression scaling laws: Unifying sparsity and quantization,

    work page Pith review arXiv

  7. [14]

    Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y

    URL https://arxiv.org/ abs/2203.15556. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y . Binarized neural networks.Advances in neural information processing systems, 29,

    arXiv

  8. [17]

    F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A. Scaling laws for precision.arXiv preprint arXiv:2411.04330,

    arXiv

Show all 37 references
  1. [18]

    Spinquant–llm quantization with learned rotations

    Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V ., Tian, Y ., and Blankevoort, T. Spinquant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406,

    arXiv

  2. [19]

    Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F

    URL https://arxiv.org/abs/ 1711.05101. Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit llms: All large language models are in 1.58 bits,

    arXiv

  3. [20]

    Malinovskii, V ., Panferov, A., Ilin, I., Guo, H., Richt´arik, P., and Alistarh, D

    URLhttps://arxiv.org/abs/2402.17764. Malinovskii, V ., Panferov, A., Ilin, I., Guo, H., Richt´arik, P., and Alistarh, D. Pushing the limits of large lan- guage model quantization via the linearity theorem.arXiv preprint arXiv:2411.17525,

    arXiv

  4. [21]

    https://resources.nvidia.com/ en-us-blackwell-architecture

    URL https://arxiv.org/abs/ 2404.03605. 10 Training Accurate LLMs with Low-Bit Weights and Activations NVIDIA. Nvidia blackwell architecture technical brief. "https://resources.nvidia.com/ en-us-blackwell-architecture",

    arXiv

  5. [22]

    Snell, C., Lee, J., Xu, K., and Kumar, A

    URL https://arxiv.org/abs/ 1907.10641. Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters.arXiv preprint arXiv:2408.03314,

    1907 arXiv

  6. [23]

    Suresh, A

    URL https://arxiv.org/abs/ 2104.09864. Suresh, A. T., Yu, F. X., Kumar, S., and McMahan, H. B. Distributed mean estimation with limited communica- tion,

    arXiv

  7. [24]

    URL https://arxiv.org/abs/1611. 00429. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295,

    arXiv

  8. [25]

    Tri Dao, Nikos Karampatziakis, H

    URL https://arxiv.org/abs/2307.09288. Tri Dao, Nikos Karampatziakis, H. C. Fast hadamard transform in cuda, with a pytorch interface. URL https://github.com/Dao-AILab/ fast-hadamard-transform. Tseng, A., Chee, J., Sun, Q., Kuleshov, V ., and De Sa, C. Quip#: Even better llm qu...

    arXiv

  9. [26]

    Vargaftik, S., Basat, R

    URLhttps://arxiv.org/abs/2208.08084. Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G., Ben-Itzhak, Y ., and Mitzenmacher, M. Drive: One-bit distributed mean estimation,

    arXiv

  10. [27]

    Vargaftik, S., Basat, R

    URL https:// arxiv.org/abs/2105.08339. Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G., Ben- Itzhak, Y ., and Mitzenmacher, M. Eden: Communication- efficient and robust distributed mean estimation for fed- erated learning,

    arXiv

  11. [28]

    Vaswani, A

    URL https://arxiv.org/ abs/2108.08842. Vaswani, A. Attention is all you need.Advances in Neural Information Processing Systems,

    arXiv

  12. [29]

    Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y ., and Wei, F

    URL https://arxiv.org/ abs/1706.03762. Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y ., and Wei, F. Bitnet: Scaling 1- bit transformers for large language models.arXiv preprint arXiv:2310.11453,

    arXiv

  13. [30]

    Bitnet a4

    Wang, H., Ma, S., and Wei, F. Bitnet a4. 8: 4-bit activations for 1-bit llms.arXiv preprint arXiv:2411.04965,

    arXiv

  14. [31]

    Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization.arXiv preprint arXiv:2403.12422,

    Xi, H., Chen, Y ., Zhao, K., Zheng, K., Chen, J., and Zhu, J. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization.arXiv preprint arXiv:2403.12422,

    arXiv

  15. [32]

    Atom: Low-bit quantization for efficient and accurate llm serving.arXiv preprint arXiv:2310.19102,

    Zhao, Y ., Lin, C.-Y ., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate llm serving.arXiv preprint arXiv:2310.19102,

    arXiv

  16. [33]

    Additional “Trust” Details A.1

    12 Training Accurate LLMs with Low-Bit Weights and Activations A. Additional “Trust” Details A.1. Trust Mask Analysis For the purposes of weight trust masks interpretation, we trained a 30M model over 3B tokens (11,444 iterations at bs=512) with QuEST weights and activations q...

    2000

  17. [35]

    For the MLP block, it uses additional gate projection and SiLU (Elfwing et al.,

    with rotary positional embeddings (Su et al., 2023). For the MLP block, it uses additional gate projection and SiLU (Elfwing et al.,

    2023

  18. [36]

    We kept the MLP intermediate dimension equal to8/3of the hidden size, padding it to 256 for increased kernel compatibility

    activation function. We kept the MLP intermediate dimension equal to8/3of the hidden size, padding it to 256 for increased kernel compatibility. For the AdamW optimizer, we used β1 = 0.90 and β2 = 0.95. We did not apply weight decay to any biases and layer normalizations. Tabl...

    2000

  19. [37]

    As described in Section 4.3, we closely follow the fitting procedure of Hoffmann et al

    Scaling law (5) fit for 3 and 4 bit QuEST with tokens/parameters ratios in{25,50,100}. As described in Section 4.3, we closely follow the fitting procedure of Hoffmann et al. (2022) for the scaling law (5) fitting. Specifically, we copied their grid of initialization given by:...

    2022

  20. [1964]

    URL https: //doi.org/10.1214/aoms/1177703732

    doi: 10.1214/aoms/1177703732. URL https: //doi.org/10.1214/aoms/1177703732. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. InProceedin...

  21. [2009]

    Alistarh, D., Grubic, D., Li, J., Tomioka, R., and V ojnovic, M

    doi: 10.1137/060673096. Alistarh, D., Grubic, D., Li, J., Tomioka, R., and V ojnovic, M. Qsgd: Communication-efficient sgd via gradient quan- tization and encoding.Advances in neural information processing systems, 30,

    doi

  22. [2017]

    Towards end-to- end 4-bit inference on generative large language models

    Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. Towards end-to- end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259,

    arXiv

  23. [2018]

    Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L

    URL https://arxiv.org/abs/ 1803.05457. Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.Advances in Neural Information Processing Systems, 35:30318–30332,

    arXiv

  24. [2019]

    Chee, J., Cai, Y ., Kuleshov, V ., and De Sa, C

    URL https://arxiv.org/abs/ 1911.11641. Chee, J., Cai, Y ., Kuleshov, V ., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36,

    1911 arXiv

  25. [2021]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al

    URL https: //arxiv.org/abs/2104.08758. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  26. [2022]

    Ailon, N

    URLhttps://arxiv.org/abs/2208.11174. Ailon, N. and Chazelle, B. The fast johnson–lindenstrauss transform and approximate nearest neighbors.SIAM Journal on Computing, 39(1):302–322,

    arXiv

  27. [2023]

    L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms.arXiv preprint arXiv:2404.00456,

    arXiv

  28. [2024]

    Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432,

    Bengio, Y ., L´eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432,

    arXiv

  29. [2025]

    Kaushal, A., Vaidhya, T., Mondal, A

    URL https://arxiv.org/abs/2501.12486. Kaushal, A., Vaidhya, T., Mondal, A. K., Pandey, T., Bha- gat, A., and Rish, I. Spectra: Surprising effectiveness of pretraining ternary language models at scale.arXiv preprint arXiv:2407.12327,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.