Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Unified Scaling Laws for Compressed Representations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A compressed representation's capacity is set by how well it reproduces random Gaussian data.

desk verdict The GMSE-based capacity metric is a real idea worth testing, but the paper's core predictive claim is currently supported only by in-sample fits on one architecture and one dataset. read the letter →

arxiv 2506.01863 v1 pith:DRLC5HPW submitted 2025-06-02 cs.LG cs.CL

Andrei Panferov , Alexandra Volkova , Ionut-Vlad Modoranu , Vage Egiazarian , Mher Safaryan , Dan Alistarh This is my paper
classification cs.LGcs.CL
keywords scalinglawsmodelcompressionrepresentationcapacityGaussianMSEquantizationsparsitysparse-quantizedtrainingvector
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish one scaling law that covers any compressed training format, not just a specific quantizer or sparsity level. It claims that every compressed representation $R$ has a capacity number $\rho(R)$, determined by the representation's mean-squared error when fitting random Gaussian data, and that this number predicts training efficiency in the single law $\mathrm{Loss}(N,D) = A\,(N\rho(R))^{-\alpha} + B\,D^{-\beta} + E$. If correct, practitioners could compare formats such as integer, floating-point, sparse, grouped, and vector-quantized weights from a fast Monte Carlo computation instead of training each format. The law also factorizes across compression types, so combined sparse-and-quantized formats can be predicted from their parts.

What carries the argument

The central object is representation capacity $\rho(R)$, a number in $(0,1]$ that acts as an effective-parameter multiplier for a compressed representation $R$. Equation (3) sets $\rho(R) = L\,\tanh\!\bigl(F\,\log^{1/4}(\mathrm{GMSE}(R))\bigr)^C$, with $\mathrm{GMSE}(R)$ the Gaussian mean-squared error of $R$ estimated by Monte Carlo on standard normal inputs; $L$, $F$, and $C$ are fitted once per representation family, with $L$ modelling imperfect convergence and $C$ controlling curvature near $\mathrm{GMSE}=1$. This capacity enters the unified law $\mathrm{Loss}(N,D) = A\,(N\rho(R))^{-\alpha} + B\,D^{-\beta} + E$, whose constants $A,B,\alpha,\beta$ are the standard scaling parameters. The mechanism behind the multiplication of $N$ and compression error is Theorem 1, which bounds the average gradient norm at compressed iterates by $N$ times the average squared compression error, independently of the compression type. The multiplicative factorization of $\rho$ across independent compression axes is what lets compound formats be evaluated from simple ones.

What would settle it

Train two formats with equal Gaussian-MSE but different quantizer structure on the same model and data, and fit their capacity from the loss curves; if the fitted capacities differ, Gaussian-MSE is not sufficient. Alternatively, fit the shared $\rho$ curve on 30M-parameter models and predict loss for the same format at 200M parameters or on another dataset; a systematic prediction error beyond the original fit error would falsify the transfer claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that the parameter efficiency of a compressed representation is not a free fitting constant but an intrinsic property: $\rho(R) = L\,\tanh\!\bigl(F\,\log^{1/4}(\mathrm{GMSE}(R))\bigr)^C$, where $\mathrm{GMSE}(R)$ is the minimal mean-squared error of $R$ when reproducing standard Gaussian data, obtained by a fast Monte Carlo estimate. This capacity sits inside the unified loss law $\mathrm{Loss}(N,D) = A\,(N\rho(R))^{-\alpha} + B\,D^{-\beta} + E$, so a model trained over $R$ behaves like an uncompressed model with $N' = N\rho(R)$ parameters. The fitted capacity curve transfers across bit widths and sparsity levels for the same representation family, extends to vector-quantized lattices, and factorizes across independent compression axes: $\rho(R_1,R_2) = \rho(R_1)\,\rho(R_2)$. The theoretical support is a non-convex convergence bound for Adam with straight-through estimation, whose leading compression-dependent term is the number of parameters $N$ times the average squared error between compressed and uncompressed iterates, matching the multiplicative role $N\rho(R)$ plays in the law.

Load-bearing premise

The load-bearing premise is that a representation's error on random Gaussian data, rather than on real activations or gradients, fully determines its capacity across architectures, datasets, and model sizes.

Editorial extensions

If this is right

  • A single fitted capacity curve for a representation type predicts loss for every bit width, sparsity level, or group size of that type, not just the configuration that was trained.
  • Any compressed model can be mapped to a dense-equivalent parameter count $N\rho(R)$, so standard parameter-data scaling predictions apply without retraining on the compressed format.
  • Because capacity factorizes, the combined effect of weight quantization, weight sparsity, and activation quantization can be computed as a product of individually measured capacities.
  • Formats can be ranked by their Gaussian mean-squared error alone, and the law indicates how close a quantizer is to the optimal capacity for its bit budget.
  • The RMSE-banded backward masking rule increases measured capacity of sparse training compared with magnitude-pruning baselines at the same sparsity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Capacity measured on Gaussian data could become a pre-training format-selection procedure: score any proposed quantizer or sparsity pattern by Monte Carlo GMSE before training a model.
  • Factorization implies a compositional library: tabulate base capacities for integer and floating-point formats, sparsity patterns, and grouping or outlier schemes, and predict arbitrary combinations by multiplication.
  • A natural next check is to log per-step compression error during actual Adam training and compare it with the Gaussian-computed GMSE; that would connect Theorem 1's bound to the fitted law directly.
  • The banded backward-masking heuristic suggests a broader design principle: choose training-time masks to minimize the average squared compression error term in the bound rather than reusing the forward mask.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a unified scaling law for training with compressed representations of the form Loss(N,D)=A(Nρ(R))^{-α}+BD^{-β}+E, where representation capacity ρ(R) is a parametric function of the Gaussian mean-squared error of the representation, Eq. (3). The authors report fits across Llama-style 30M-200M models trained on C4 under scalar quantization, sparsity, sparse-quantized, and vector-quantized formats; they claim that capacity is intrinsic and model-independent, factorizes across compression types, and can be used to compare numerical formats and to design improved sparse-training heuristics (RBBM). The theory component (Theorem 1) bounds Adam convergence with straight-through estimation by N times the time-averaged squared compression error.

Significance. The paper is a useful and ambitious integration of two important lines of work, scaling laws and model compression, and it contains a substantial amount of empirical work: roughly 250 training runs across several compression formats, including vector quantization and hybrid sparse-quantized configurations. The GMSE values are computed independently of training and of the fitted capacity values, so the reported correlations are genuine evidence rather than a definitional artifact. If the central claim were established, the capacity metric would be practically valuable for ranking formats without retraining. However, the current evidence is in-sample and restricted to one architecture and one dataset, and the theory does not directly support the Gaussian-MSE functional form, so the significance is conditional on additional validation.

major comments (4)
  1. [Section 4.1, Eq. (3) and Table 1] The central claim that GMSE predicts capacity is currently supported only by in-sample fits. The parameters L, F, and C in Eq. (3) are fit to the same (GMSE, ρ) points whose fit error is then reported in Table 1; there is no documented held-out split, cross-validation, or out-of-sample prediction test. Figure 3 shows a ranking correlation between GMSE and validation loss, but it does not quantitatively test the claimed functional form. To make the 'prediction' language supportable, the authors should report leave-one-bitwidth or leave-one-sparsity-level fits, or fit on a subset of model sizes and predict the held-out sizes, preferably across an additional architecture and dataset. Without such a test, the fit errors in Table 1 only measure interpolation over the same runs, not predictive power.
  2. [Section 3, Theorem 1] Theorem 1 bounds E[||∇f(bθ)||^2] by N times the time-averaged squared compression error ||bθt-θt||^2 during optimization, plus standard Adam terms. This bound does not involve GMSE(R) measured on random Gaussian data, and no argument is given that static Gaussian-input reconstruction error is equivalent to, or even monotonically related to, the non-Gaussian, non-stationary, trajectory-averaged compression error of the actual weights. The sentence in Section 4.1 that the theorem 'connects to our notion of capacity' therefore overstates the formal support. The authors should either prove a relevant statement connecting GMSE to the optimization trajectory (for example, under unbiased compression or Gaussian weight distributions) or explicitly present the GMSE-capacity relation as an empirical law rather than a theoretically derived one.
  3. [Section 7, first caveat; Section 1] The claim that capacity is an 'intrinsic property of the representation, independent of the model and task' is not supported by the experimental scope. All scaling-law fits are for decoder-only Llama-style transformers trained on C4 in the data-rich regime, as the paper itself concedes in Section 7. This is a load-bearing gap because the word 'intrinsic' means transfer across architectures and datasets. The authors should either provide at least one additional architecture (e.g., a BERT-style encoder, a vision transformer, or an MLP) and one additional dataset at comparable scale, or substantially weaken the claim to 'capacity for Llama-style language models on C4' and adjust the title and abstract accordingly.
  4. [Section 4.3, Figures 6-9] The factorization claim ρ(R1,R2)=ρ(R1)ρ(R2) is presented as a general finding with practical implications, but the evidence shows errors that are not uniformly small. In Figure 6, the absolute error for combinations involving 1-bit and 2-bit quantization reaches values around 0.09, and Section 4.3 excludes 'extreme 2-bit quantization' as a failure case. Since the practical application in Section 5.1 is to predict combined compression configurations without training, the authors should quantify the regime in which the factorization error remains small (in terms of bit width, sparsity, and capacity value) and explain why the 2-bit case is exceptional. Without such a characterization, the statement that 'representation capacity factorizes' is too broad.
minor comments (6)
  1. [Section 3, Theorem 1] The symbol C is used both for the compression map C: R^N -> R^N and for the constant C = 2√(G∞^2 + ε/N) in the theorem statement and its proof; these should be renamed to avoid ambiguity.
  2. [Section 4.1, Eq. (3)] The notation ẽρ(GMSE(R)) in Eq. (3) is not consistently distinguished from ρ(R) in the surrounding text; please state explicitly whether Eq. (3) defines ρ itself or an auxiliary function ẽρ, and use the chosen notation throughout.
  3. [Section 4.3, third bullet] In the third bullet, 'activation sparsity qa' should be 'activation quantization bitwidth qa'; the current text makes the subscript qa inconsistent with the notation used for sparsity.
  4. [Figure 2(b)] The caption says 'relative errors' while the third panel is labeled 'Absolute error'; please align the caption with the panel label.
  5. [Appendix C.2, Table 3] The first row of Table 3, 'Decoupled Independently fitted ρ', does not specify the functional form used for ρ; please add the formula or a reference so the comparison across rows is meaningful.
  6. [Appendix D, Algorithm 1/2] There is a typo in the sentence 'our method is combines ideas'; it should read 'our method combines ideas,' and the notation projgrid in Algorithm 1 should be defined at first use.

Circularity Check

0 steps flagged · score 1.0 of 10

No definitional circularity: GMSE is computed independently of training loss, so the capacity correlation is a falsifiable empirical claim; in-sample fitting is a validation gap, not a circular reduction.

full rationale

The paper's central capacity law is an empirical regression, not a derivation that reduces to its own inputs. The independent variable GMSE(R) is computed by Monte Carlo on standard Gaussian inputs, with no use of the trained losses: Section 4.1 states "The GMSE(R) is easily computable for any representation," and Appendix C.1 confirms that "MSE is computed over standard Gaussian input." The response variable rho(R) is fitted from loss data, and Eq. (3) then fits L, F, C to the (GMSE, rho) pairs. This is an in-sample curve fit: Table 1 reports fit error, and Section 7 concedes only one architecture/dataset, so the paper would be stronger with held-out predictions across architectures. But that is a validation weakness, not circularity, because GMSE is an external quantity that could have failed to correlate with fitted capacity. Theorem 1 is a genuine convergence proof for Adam with compressed iterates; it bounds gradient norms by N times the time-averaged squared compression error of the actual weights. That quantity is not the same as Gaussian GMSE, so Theorem 1 is incomplete support for Eq. (3), but the gap is a missing derivation rather than a closed loop. Self-citations to Frantar et al. [10], QuEST [25], and HIGGS [23] provide the baseline law, quantization tooling, and vector-quantization method, but the load-bearing GMSE-capacity link is tested with the paper's own fits and external Gaussian MSE computations. No uniqueness theorem is imported from prior authors to force the choice of Eq. (3), and the tanh functional form is selected by fitting, not inherited by citation. The factorization results in Section 4.3 are empirical matrix-factorization observations, not consequences of the definitions. Overall, no step in the derivation is equivalent to its input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several fitted quantities: the scaling-law constants A, B, α, β, E, the per-representation capacity values, and the three-parameter GMSE mapping. The paper also relies on the untested assumption that Gaussian-MSE is a universal sufficient statistic for compression effects on training.

free parameters (5)
  • L (capacity saturation limit) = per compression type, not reported
    Saturating limit of ρ in Eq. (3), used to model imperfect high-precision convergence for quantizers such as LSQ.
  • F (capacity curvature scale) = per compression type, not reported
    Scale parameter inside tanh in Eq. (3), fit to (GMSE, ρ) data.
  • C (capacity curvature exponent) = per compression type, not reported
    Curvature exponent in Eq. (3), controlling behavior near GMSE values around 1.
  • Scaling-law constants A, B, α, β, E = not reported
    Fitted to losses across N and D in Eq. (1); with D proportional to N, these five parameters are fit to four model sizes.
  • Per-configuration capacity ρ(R) = reported in matrices, e.g., Figures 6-9
    Each compressed configuration's efficiency is first fitted from training losses; the GMSE relation is then fit to these fitted values.
assumptions (5)
  • domain assumption Chinchilla scaling-law form Loss(N,D) = A·N^{-α} + B·D^{-β} + E (Eq. 2)
    Assumed as the starting framework; all subsequent fits inherit this functional form.
  • domain assumption Data-rich regime with 100 tokens per parameter and model sizes 30M-200M on C4 is sufficient to estimate the law's constants
    Section 3 setting; the paper's fits use four model sizes with D tied to N.
  • standard math Standard assumptions for Adam convergence: L-smoothness, unbiased bounded stochastic gradients, bounded variance (Assumptions 1-3)
    Used in Theorem 1; standard in adaptive-optimization analysis but not verified for LLM training.
  • ad hoc to paper Gaussian input is the right reference distribution for computing GMSE
    Appendix C.1 states Gaussian noise gives the widest MSE spread and best fit; other distributions preserve monotonicity but are not used.
  • ad hoc to paper The first term in Theorem 1 justifies replacing capacity with a function of GMSE
    The theorem bounds gradient norms by average squared compression error during optimization, but the leap from that quantity to GMSE on Gaussian data is asserted, not derived.
invented entities (1)
  • Representation capacity ρ(R)
    purpose: A scalar parameter-efficiency multiplier in Eq. (1) converting N into a dense-equivalent parameter count N'
    A constructed latent quantity fitted from scaling-law curves; its only handles are fits inside this paper, with no external falsifiable prediction yet, for example no experiment outside C4 and Llama-style models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unified Scaling Laws for Compressed Representations." pith.science (2026). https://pith.science/paper/DRLC5HPW

@misc{pith2026250601863,
  author       = {Pith},
  title        = {Pith review of: Unified Scaling Laws for Compressed Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRLC5HPW}},
  note         = {Machine review of arXiv:2506.01863}
}
read the original abstract

Scaling laws have shaped recent advances in machine learning by enabling predictable scaling of model performance based on model size, computation, and data volume. Concurrently, the rise in computational cost for AI has motivated model compression techniques, notably quantization and sparsification, which have emerged to mitigate the steep computational demands associated with large-scale training and inference. This paper investigates the interplay between scaling laws and compression formats, exploring whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations, such as sparse, scalar-quantized, sparse-quantized or even vector-quantized formats. Our key contributions include validating a general scaling law formulation and showing that it is applicable both individually but also composably across compression types. Based on this, our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric -- based on the representation's ability to fit random Gaussian data -- which can robustly predict parameter efficiency across multiple compressed representations. On the practical side, we extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.

Figures

Figures reproduced from arXiv: 2506.01863 by the authors.

Figure 1
Figure 1. Comparison of ρ fits for scaling law forms from [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (a) Scaling law for 2- and 4-dimensional vector quantization. (b) Representation capacity across weight and activation sparsity levels: baseline, factorized prediction, and relative errors. Note the low errors for the factorized predictions, with slight increases at the larger sparsity levels. 4 bits 6 bits 8 bits 10 5 10 3 GMSE E2M1 INT4 E4M1 E3M2 E2M3 INT6 E6M1 E5M2 E4M3 E3M4 E2M5 INT8 GMSE, C4 Val Loss by Floatin… view at source ↗
Figure 3
Figure 3. Comparison of floating point and integer data-types in terms of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Representation capacity ρ(R) versus MSE for (a) group-wise quantization, with markers indicate group counts (color encodes quantization bitwidth), and (b) outlier-aware quantization. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Backward heuristics: (a) forward mask MFW determined by the Tk threshold, (b) backward mask determined by threshold Tp, (c) banded backward mask determined by both Tk and Tp, (d) capacity plot compared to the Magnitude Pruning baseline with MBW = MFW . 5 Applications 5…
Figure 6
Figure 6. Figure 6: Representation capacity coefficients for independent quantization of weights and activations. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Representation capacity coefficients with fit errors in case of sparsity combined with the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Representation capacity fit errors for sparse+quantized weights and quantized activations. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Representation capacity coefficients matrix for sparsity applied with uniform quantization. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 11
Figure 11. Figure 11: Comparison of sparse training methods for Llama-30M. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training

    cs.LG 2025-10 conditional novelty 6.0 of 10

    CAGE, a curvature-aware correction that adds the quantization error to the gradient, reduces loss in low-bit quantization-aware training, letting 3-bit CAGE-trained models match 4-bit baseline-trained models.

Reference graph

Works this paper leans on

44 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [10]

    Frantar, E., Evci, U., Park, W., Houlsby, N., and Alistarh, D. (2025). Compression scaling laws:unifying sparsity and quantization

    work page 2025

  2. [19]

    F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghunathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghunathan, A. (2024). Scaling laws for precision. arXiv preprint arXiv:2411.04330

    arXiv 2024

  3. [1]

    M., and Mendelson, A

    Baskin, C., Liss, N., Schwartz, E., Zheltonozhskii, E., Giryes, R., Bronstein, A. M., and Mendelson, A. (2021). Uniq: Uniform noise injection for non-uniform quantization of neural networks. ACM Transactions on Computer Systems (TOCS), 37(1-4), 1–15

    work page 2021

  4. [2]

    Bengio, Y ., Léonard, N., and Courville, A. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432

    arXiv 2013

  5. [3]

    Chen, C., Shen, L., Huang, H., and Liu, W. (2021). Quantized Adam with Error Feedback. arXiv preprint arXiv:2004.14180

    arXiv 2021

  6. [4]

    Chen, X., Liu, S., Sun, R., and Hong, M. (2019). On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization. International Conference on Learning Representations

    work page 2019

  7. [5]

    Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. (2022). Unified scaling laws for routed language models. International Conference on Machine Learning, pages 4057–4086

    work page 2022

  8. [6]

    Défossez, A., Bottou, L., Bach, F., and Usunier, N. (2022). A simple convergence proof of adam and adagrad. Transactions on Machine Learning Research

    work page 2022

Show all 44 references
  1. [7]

    K., McKinstry, J

    Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. (2019). Learned step size quantization. arXiv preprint arXiv:1902.08153

    2019 arXiv

  2. [8]

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

    2022 arXiv

  3. [9]

    R., Houlsby, N., Alistarh, D., and Evci, U

    Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D., and Evci, U. (2024). Scaling laws for sparsely-connected foundation models. In International Conference on Learning Representations

    2024

  4. [11]

    W., and Keutzer, K

    Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. (2022). A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291–326. Chapman and Hall/CRC

    2022

  5. [12]

    Gibney, E. et al. (2022). How to shrink ai’s ballooning carbon footprint. Nature, 607(7920), 648–648

    2022

  6. [13]

    B., Chakraborty, A., Kostenok, E., Mishin, D., Ha, D., Falsafi, B., Jaggi, M., Liu, M., Oh, Y ., Subramanian, S., and Yazdanbakhsh, A

    Harma, S. B., Chakraborty, A., Kostenok, E., Mishin, D., Ha, D., Falsafi, B., Jaggi, M., Liu, M., Oh, Y ., Subramanian, S., and Yazdanbakhsh, A. (2025). Effective interplay between sparsity and quantization: From theory to practice. In International Conference on Learning Repr...

    2025 arXiv

  7. [14]

    Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. (2021). Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241), 1–124. 10

    2021

  8. [15]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2024

  9. [16]

    Hou, L., Zhang, R., and Kwok, J. T. (2019). Analysis of quantized models. In International Conference on Learning Representations

    2019

  10. [17]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  11. [18]

    Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR). arXiv:1412.6980

    2015 arXiv

  12. [20]

    and Li, P

    Li, X. and Li, P. (2022). Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression. arXiv preprint arXiv:2211.14292

    2022 arXiv

  13. [21]

    Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang, J., Zhao, J., Roy, S., Jin, L., Xiong, Y ., Shi, Y ., Xiao, L., Tian, Y ., Soran, B., Krishnamoorthi, R., Blankevoort, T., and Chandra, V . (2025). Paretoq: Scaling laws in extremely low-bit llm quantization. arXiv preprint arXiv:...

    2025

  14. [22]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations

    2019

  15. [23]

    Malinovskii, V ., Panferov, A., Ilin, I., Guo, H., Richtárik, P., and Alistarh, D. (2025). HIGGS: Pushing the limits of large language model quantization via the linearity theorem. In L. Chiruzzo, A. Ritter, and L. Wang, editors,Proceedings of the 2025 Conference of the Nation...

    2025

  16. [24]

    Modoranu, I.-V ., Safaryan, M., Malinovsky, G., Kurtic, E., Robert, T., Richtarik, P., and Alistarh, D. (2024). Microadam: Accurate adaptive optimization with low space overhead and provable convergence

    2024

  17. [25]

    L., Nikdan, M., and Alistarh, D

    Panferov, A., Chen, J., Tabesh, S., Castro, R. L., Nikdan, M., and Alistarh, D. (2025). Quest: Stable training of llms with 1-bit weights and activations. arXiv preprint arXiv:2502.05003

    2025 arXiv

  18. [26]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. In Proceedings of the 37th International Conference on Machine Learning (ICML), p...

    2020

  19. [27]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of Adam and beyond. arXiv preprint arXiv:1904.09237

    2019 arXiv

  20. [28]

    Robert, T., Safaryan, M., Modoranu, I.-V ., and Alistarh, D. (2024). LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. arXiv preprint arXiv:2410.16103

    2024 arXiv

  21. [29]

    Sardana, N., Portes, J., Doubov, S., and Frankle, J. (2024). Beyond chinchilla-optimal: Ac- counting for inference in language model scaling laws. In International Conference on Machine Learning

    2024

  22. [30]

    Sun, X., Li, S., Xie, R., Han, W., Wu, K., Yang, Z., Li, Y ., Wang, A., Li, S., Xue, J., Cheng, Y ., Tao, Y ., Kang, Z., Xu, C., Wang, D., and Jiang, J. (2025). Scaling laws for floating-point quantization training. arXiv preprint arXiv:2501.02423. 11

    2025 arXiv

  23. [31]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V ., Goyal, ...

    2023 arXiv

  24. [32]

    Zhou, D., Chen, J., Cao, Y ., Yang, Z., and Gu, Q. (2018). On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization. arXiv preprint arXiv:1808.05671

    2018 arXiv

  25. [33]

    and Gupta, S

    Zhu, M. and Gupta, S. (2017). To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878. 12 Appendix Roadmap This appendix provides supporting material organized as follows: • Experimental Setup (Appendix A): Model arch...

    2017 arXiv

  26. [34]

    Quantized weights and activations (Fig. 6)

  27. [35]

    Sparsity + QuEST quantizer (Fig. 7)

  28. [36]

    8), for all combinations (sa, qa, qb) for sparsity sa ∈ [0.25, 0.5, 0.75] and bit widths qa, qb ∈ [2, 4, 6]

    Joint sparse & quantized weights + activations (Fig. 8), for all combinations (sa, qa, qb) for sparsity sa ∈ [0.25, 0.5, 0.75] and bit widths qa, qb ∈ [2, 4, 6]

  29. [37]

    Sparsity + uniform quantizer with maximum absolute value as a scale (Fig. 9). From the factorized representation-capacity matrices we observe the following:

  30. [38]

    The element-wise error of the fitted coefficients ρ (from our scaling law) is of order 10−3–10−2

  31. [39]

    A rank-1 row-column outer product accurately approximates the matrix, confirming the multiplicative property of representation capacity ρ in various scenarios. 13

  32. [40]

    Γtgt − β1 1 − β1 ∆Γtmt−1 2# = −ηE [⟨∇f (θt), Γtgt⟩]| {z } I + ηE ∇f (xt), β1 1 − β1 ∆Γtmt−1 | {z } II + η2L 2 E

    Approximation error remains of the order 10−2, except for the cases of extreme 2-bit quantization, where ρ ≲ 0.1. We explain this gap due to the poorer performance of the optimizer in these extreme compression regimes, which is not taken into account currently by our model (as...

  33. [41]

    This heuristic allows gradients to flow only through the largest parameters by magnitude selected by Top-K, while the low-magnitude parameters will have zero gradient

    fw: the backward mask is simply set to the forward mask: MBW = MF W. This heuristic allows gradients to flow only through the largest parameters by magnitude selected by Top-K, while the low-magnitude parameters will have zero gradient

  34. [42]

    rms: we align the tensor x with the standard normal distribution by dividing x by its root mean square RM S(x) = q 1 n Pn i=1 x2 i , which results in ||x/RM S(x)||2 2 = n. For this heuristic, the user sets a median deviation parameter p ∈ (0, 0.5), which is used to determine t...

  35. [43]

    This banded heuristic determines the backward mask using the thresholdTRM S(x, p) computed for the rms heuristic in conjunction with the Top-K threshold (which we denote by Tk)

    banded-rms (b-rms): the rms heuristic has the property that the absolute values of x that are larger than the threshold TRM S(x, p) will have value 1 in the mask, while the smaller ones will have value 0. This banded heuristic determines the backward mask using the thresholdTR...

  36. [44]

    area-banded-rms (a-b-rms): in the b-rms heuristic we do not have any control over the re- lationship between the Top-K threshold Tk and TRM S(x, p). Let us discuss the two possible cases: (a) Tk < TRM S(x, p) : MBW = ( |x| < Tk) ∨ (TRM S(x, p) < |x|), which means that all valu...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.