Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

CAGE claims that adding the instantaneous quantization error to the optimizer update, with a curvature-aware scaling, makes quantization-aware training converge to a Pareto-optimal point and substantially closes the low-bit accuracy gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:46 UTC pith:U72GGMCA

load-bearing objection Good empirical QAT paper with a theory that doesn't cover the actual algorithm; worth refereeing for the experiments. the 4 major comments →

arxiv 2510.18784 v3 pith:U72GGMCA submitted 2025-10-21 cs.LG

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training

Soroush Tabesh , Mher Safaryan , Andrei Panferov , Alexandra Volkova , Dan Alistarh This is my paper
classification cs.LG
keywords quantization-aware trainingstraight-through estimatorgradient estimationmulti-objective optimizationPareto optimalitylow-bit language modelsconvergence guaranteesquantization error correction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that quantization-aware training can be understood as balancing two objectives: minimizing the task loss and satisfying the quantization constraint. It proposes updating parameters not only with the usual straight-through gradient but also with the current quantization error, scaled by a coefficient, so the training dynamics pull the model toward the quantized set while still minimizing loss. If correct, this closes most of the remaining accuracy gap between low-bit quantized models and full-precision training, with 3-bit pretrained models matching what previously required 4 bits.

Core claim

The central claim is that the quantization error x_t − Q(x_t), which the straight-through estimator normally discards, is exactly the information needed to regularize quantized training. CAGE's update is x_{t+1} = x̃_{t+1} − α λ_t e_t, applied after the base optimizer step, and the paper proves that this dynamics converges ergodically to a λ-Pareto-optimal point, where no small update can improve the loss without hurting the quantization objective. The same correction is shown empirically to roughly halve the quantization-induced accuracy loss in fine-tuning and to make W3A3 pre-training match W4A4 accuracy of the prior best method.

What carries the argument

The machinery is the instantaneous quantization error e_t = x_t − Q(x_t), added to the update as a correction term. It is grounded in a Pareto-optimality condition ∇f(x) + λ(x − Q(x)) = 0, where λ balances loss minimization against quantized-set satisfaction; the error term acts as a stand-in for the gradient of a regularizer that would otherwise require an infeasible proximal step.

Load-bearing premise

The convergence proof relies on the assumption that the quantization error x − Q(x) is the gradient of some smooth function, but the round-to-nearest quantizer used in the experiments is discontinuous, so this assumption does not hold for the implemented algorithm.

What would settle it

On the paper's own toy quadratic f(x) = ½(x − ½)² with Q(x) = ⌊x⌋, compute the Pareto residual ∇f(x) + λ(x − Q(x)) at the point CAGE converges to for a fixed λ. If it does not approach zero, the implemented update does not satisfy the claimed Pareto convergence; if it does, then the smoothness assumption is not necessary on this example.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Compression accuracy loss in QAT fine-tuning is roughly halved relative to the prior best method at similar compute.
  • Pretraining with 3-bit weights and activations reaches the loss that 4-bit training achieved with the prior best method, so low-bit models become a more practical training target.
  • Gains transfer across different optimizer update rules, so the correction behaves as an optimizer-agnostic plug-in.
  • Fitted scaling laws show effective parameter efficiency improves by more than 10% at 4-bit and more than 20% at 2-bit over the prior best QAT method.
  • The convergence result gives an O(1/√T) ergodic rate to a Pareto-optimal point with no non-vanishing quantization-error term, a formal guarantee of a kind previous QAT heuristics lacked.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The correction term can be read as a cheap, diagonal approximation to a second-order regularizer; if that read is right, the same term should transfer to other nondifferentiable constraints such as sparsity or vector quantization.
  • If the empirical gains persist at larger scale, QAT could move from a deployment-time step to the default training procedure, since the method's extra cost is one elementwise addition per parameter per step.
  • The paper's proof assumes a smooth quantizer, while the experiments use a discontinuous round-to-nearest one; a meaningful test is whether replacing the abrupt quantizer with a smooth surrogate preserves the gains, or whether the discontinuity itself is doing the work.
  • A simpler control—adding the quantization error with a fixed size rather than scaling with curvature—would isolate whether the 'curvature-aware' interpretation is necessary or incidental to the observed improvements.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CAGE, a quantization-aware training (QAT) method that augments the straight-through estimator (STE) gradient with a correction term proportional to the instantaneous quantization error e_t = x_t − Q(x_t), either inside or outside the base optimizer's preconditioner. The authors motivate the correction through a multi-objective reformulation of QAT, introduce a λ-Pareto optimality condition, and prove an ergodic convergence theorem (Theorem 1) for an SGD variant under an assumption that the quantization error is the gradient of a smooth potential. Empirically, they report that CAGE improves over the QuEST baseline in synthetic quadratic tasks, MXFP4 fine-tuning of Llama-3.2-3B, and pre-training of Llama-style models from 30M to 800M parameters across W2A2, W3A3, and W4A4, including the headline claim that W3A3 CAGE pre-training matches or beats W4A4 QuEST. The paper also includes scaling-law analysis and comparisons across AdamW, Muon, Shampoo, and SOAP.

Significance. If the empirical results hold, CAGE is a practically valuable and remarkably simple addition to QAT: it consistently lowers validation loss across model sizes, precisions, and optimizers, and the paper provides an official implementation. The extensive pre-training evaluation (up to 800M parameters, multiple seeds, several optimizers) is a real strength, as is the explicit reporting of hyperparameters and ablations. However, the theoretical contribution as stated does not cover the implemented algorithm. Theorem 1 relies on Assumption 3, which is false for the Hadamard round-to-nearest quantizer used in all experiments; the theorem is also only for SGD, not the AdamW algorithm actually evaluated. Thus the paper currently offers a well-engineered heuristic with strong empirical evidence, but the advertised convergence guarantees and the 'curvature-aware' derivation do not apply to the tested method. The gap is central rather than cosmetic, and the authors should either repair the theory or substantially reframe the paper's claims.

major comments (4)
  1. [§3.2, Assumption 3; §4, Quantization pipeline] Assumption 3 requires existence of an Lφ-smooth φ with x−Q(x)=∇φ(x). This forces Q to be continuous (indeed (1+Lφ)-Lipschitz). The quantizer used in §4 — row-wise Hadamard transform followed by symmetric clipping and round-to-nearest — produces an error x−Q(x) that is discontinuous at every rounding and clipping boundary. Hence no such smooth potential exists, and Theorem 1, whose proof (Appendix A) uses φ and the path-independent line integral of I−Q, does not apply to the experiments. The sentence claiming Assumption 3 is 'essentially equivalent to Lipschitz continuity of Q' is also incorrect, since round-to-nearest is not continuous. The paper must either analyze a genuinely smooth surrogate quantizer, prove a version with non-vanishing quantization-error terms, or explicitly label the practical method as heuristic.
  2. [§3.2, Theorem 1 vs §3.3, Algorithm 1] Theorem 1 is stated for the SGD update in Eq. (3), but the algorithm evaluated throughout §4 is Algorithm 1: AdamW with decoupled weight decay, a decoupled CAGE correction, a silence period, and a time-varying λ_t. None of these elements appear in the convergence analysis. Even granting Assumption 3, the theorem does not cover the method whose results are reported. The paper should either extend the analysis to AdamW/decoupled updates or explicitly restrict the theoretical claims to the SGD variant and state that the practical AdamW version is not covered.
  3. [§3.1, Eq. (2); §3.2, Theorem 1] Even if Assumption 3 held, Theorem 1 controls E∥∇f(x̂)+λ(x̂−Q(x̂))∥² at a full-precision iterate x̂. This is a stationarity condition for the weighted-sum surrogate f+λφ, not for the quantized-model objective f(Q(x)) or for the validation loss reported in Figures 3–5 and Table 1. A small Pareto gradient does not imply that Q(x_T) has low loss, especially since Q is discontinuous. The logical link between the convergence theorem and the headline W3A3-vs-W4A4 accuracy result is therefore missing; the authors should state precisely what quantity the theorem controls and argue why it is relevant to QAT accuracy.
  4. [§1, §3.3, Algorithm 1] The method is called 'curvature-aware' and the introduction says the correction 'leverages local second-order information about the loss landscape (i.e., the Hessian).' But the decoupled update used in most experiments (Algorithm 1, line 17) adds only αλ_t e_t, with no Hessian or curvature term. Curvature enters only through the coupled-Adam variant via the preconditioner, which is not the default. The paper should either demonstrate a concrete sense in which λ_t e_t incorporates curvature without Assumption 3, or change the terminology/narrative to match what the algorithm actually computes.
minor comments (5)
  1. [Abstract, §3.1, Assumption 2] Typos and small errors: the abstract has 'can be found over https' instead of 'at'; Assumption 2 has 'F or all iterates'; §3.1 writes 'f(Q(x)' with a missing closing parenthesis. Please proofread.
  2. [Table 1 and §4.3] The table caption says 'final validation perplexity' while the surrounding text says 'validation loss'. These are different metrics; please use consistent terminology and state which quantity is plotted in Figure 5.
  3. [Appendix A, proof of Theorem 1] The line-integral representation is written as φ(x)=∫_{x0}^{x}(I−Q)·dr. This omits the integration constant φ(x0); the correct statement is φ(xT)−φ(x0)=∫_{x0}^{xT}(I−Q)·dr, and the subsequent bound should be justified with an absolute value. As written, the equation is false.
  4. [§4.3, Precision Scaling Law] The eff(P) fits introduce separate free parameters per method and bit-width, and the conclusion that CAGE 'improves parameter efficiency' largely restates the lower validation losses in Table 1. A held-out validation of the scaling-law form or a model-selection comparison would strengthen this claim; otherwise it should be presented as a descriptive summary rather than independent evidence.
  5. [§4.3, Comparison to LOTION] The LOTION comparison is based on the authors' reimplementation, not the official code, and the paper acknowledges this. This is acceptable as a preliminary comparison, but the claim that CAGE outperforms LOTION should be marked as provisional until verified with the official implementation.

Circularity Check

0 steps flagged

No significant circularity: empirical gains are externally benchmarked, and the convergence theorem is a standard descent proof; the main weakness (Assumption 3 mismatch with the implemented quantizer) is a validity issue, not circularity.

full rationale

The paper's central empirical claims (roughly halving fine-tuning quantization loss; W3A3 CAGE matching W4A4 QuEST) are validated against QuEST, LSQ-style prior work, and a BF16 reference on held-out validation sets; these numbers are not fitted parameters renamed as predictions. The theoretical claim is not circular in the prohibited sense: Eq. (2) defines lambda-Pareto stationarity as the zero of the vector field ∇f(x)+λ(x−Q(x)), Eq. (3) is gradient descent on that same field, and Theorem 1 is a standard smooth-descent lemma on F=f+λφ (Appendix A) delivering an explicit O(1/√T) rate and a bounded A3 term. Defining the optimality target as the algorithm's fixed point is conventional in non-convex optimization; the proof does not assume its conclusion. The only self-citation, QuEST (Panferov et al. 2025a), is used as a strong external baseline and base quantizer, not as load-bearing evidence for CAGE's derivation. The real weakness is correctness, not circularity: Section 3.2, Assumption 3 asserts that x−Q(x)=∇φ(x) and calls this 'essentially equivalent to the Lipschitz continuity of the quantization operator Q,' but for the actual row-wise Hadamard + clip + round-to-nearest quantizer of §4, x−Q(x) is discontinuous, so no smooth φ exists; Theorem 1 therefore does not cover the implemented algorithm, and stationarity at full-precision iterates does not directly imply low loss for the quantized model Q(x_T). That is a validity gap to be weighed as correctness risk, but it does not make the derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central method relies on standard smoothness/noise assumptions, an idealized smooth-potential assumption for the quantizer that is not satisfied in experiments, and tuned hyperparameters plus fitted scaling-law constants. The 'Pareto-optimal' target is defined so that the algorithm's fixed points match it, which lowers the independent-content load of the theory.

free parameters (3)
  • CAGE coefficient λ = 2.0 default (swept 1–2.5; λ=5 in optimizer-agnostic runs)
    Controls the strength of the quantization-error correction; tuned via validation perplexity sweeps (Appendix C), not predicted by theory.
  • silence ratio s = 0.9 default (swept 0.8–0.95)
    Fraction of training skipped before the correction ramps up; tuned in Appendix C.
  • scaling-law eff(P) per method/bit-width = e.g., CAGE W4A4 eff≈0.797; QuEST W4A4 eff≈0.733
    Fitted to the loss data via nonlinear least squares to support parameter-efficiency claims; a fitted curve, not a prediction.
axioms (4)
  • domain assumption Loss is Lf-smooth and lower bounded (Assumption 1)
    Standard for analysis, but transformer-scale losses are not guaranteed smooth over the whole parameter space.
  • standard math Stochastic gradients are unbiased with bounded variance (Assumption 2)
    Standard noise assumption in SGD analysis; plausible for mini-batch training.
  • ad hoc to paper Quantization error is the gradient of an Lφ-smooth function φ: x−Q(x)=∇φ(x) (Assumption 3)
    Load-bearing for Theorem 1 and for calling the correction 'curvature-aware'; false for round-to-nearest quantizers used in experiments, so the theorem does not cover the actual algorithm.
  • ad hoc to paper Scaling law L(N,D,P) = A/(N eff(P))^α + B/D^β + E with shared exponents
    Used to define eff(P); fitted rather than derived.
invented entities (2)
  • λ-Pareto-optimal solution no independent evidence
    purpose: Defines the target of CAGE convergence (Eq. 2)
    A new optimality concept internal to the paper; not tied to a measurable external prediction.
  • smooth potential φ for the quantization error no independent evidence
    purpose: Justifies the correction term as the gradient of a regularizer
    No evidence that such φ exists for the actual quantizer; an idealization introduced in Assumption 3.

pith-pipeline@v1.3.0-alltime-deepseek · 16681 in / 13042 out tokens · 117955 ms · 2026-08-04T08:46:29.961016+00:00 · methodology

0 comments
read the original abstract

Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training. To address this, we introduce CAGE (Curvature-Aware Gradient Estimation), a new QAT method that augments the straight-through estimator (STE) gradient with a curvature-aware correction designed to counteract the loss increase induced by quantization. CAGE is derived from a multi-objective view of QAT that balances loss minimization with the quantization constraints, yielding a principled correction term that depends on local curvature information. On the theoretical side, we introduce the notion of Pareto-optimal solutions for quantized optimization, and establish that CAGE yields strong convergence guarantees in the smooth non-convex setting. In terms of implementation, our approach is optimizer-agnostic, but we provide a highly-efficient implementation that leverages Adam statistics. CAGE significantly improves upon the prior state-of-the-art methods in terms of accuracy, for similar computational cost: for QAT fine-tuning, it halves the compression accuracy loss relative to the prior best method, while for QAT pre-training of Llama models, its accuracy for 3-bit weights-and-activations (W3A3) matches the accuracy achieved at 4-bits (W4A4) with the prior best method. The official implementation can be found over https://github.com/IST-DASLab/CAGE .

Figures

Figures reproduced from arXiv: 2510.18784 by Alexandra Volkova, Andrei Panferov, Dan Alistarh, Mher Safaryan, Soroush Tabesh.

Figure 1
Figure 1. Figure 1: Quadratic with non-isotropic Hessian: final validation loss for all methods various condition numbers (lower is better). The loss for CAGE does not include the regularizer term. We find that CAGE consistently reduces stationary error. 0.2 0.0 0.2 PC1 0.8 0.6 0.4 0.2 0.0 0.2 0.4 PC2 3e-06 2e-06 4e-06 5e-06 7e-06 9e-06 1e-05 1e-05 2e-05 3e-05 5e-05 3e-05 6e-05 8e-05 1e-04 1e-04 2e-04 2e-04 3e-04 5e-04 4e-04 … view at source ↗
Figure 3
Figure 3. Figure 3: QAT accuracy on Llama-3.2-3B (Tulu-SFT) for CAGE vs. the state-of-the-art MXFP4 baseline, using QuEST (Panferov et al., 2025a). CAGE consistently improves GSM8K/HellaSwag/WinoGrande results, essentially halving the error due to quantization. Hadamard transform is applied in both cases for outlier mitigation. 8500 8600 8700 8800 8900 9000 9100 Step 3.54 3.56 3.58 3.60 3.62 Validation Loss AdamW Shampoo Muon… view at source ↗
Figure 6
Figure 6. Figure 6: Fitted parameter efficiency eff(P), normalized relative to standard 16-bit, across precisions. CAGE im￾proves effective model capacity over QuEST, peaking near 4-bit [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Illustration of the validation loss for training the 50M Llama model in W4A4 using the QuEST baseline end-to-end (left), focused on the last fraction of steps (middle), and in terms of quantization error (regularizer) values (right). Observe that, while the overall training behavior is similar, CAGE significantly reduces loss for the quantized model in the final part of training. a given parameter budget. … view at source ↗
Figure 8
Figure 8. Figure 8: Pretraining loss comparison between CAGE and LO￾TION on a 100M-parameter Llama-style model trained on C4 under weight-only QAT with AbsMax W4A16 quantization. Both methods follow a similar trajectory early in training, but CAGE continues to improve and attains a substantially lower final loss, while LOTION plateaus. constraints. The key theoretical contribution is the definition of Pareto￾optimal solutions… view at source ↗
Figure 9
Figure 9. Figure 9: Validation perplexity for CAGE under different choices of silence ratio s and regularization coefficient λ on the 50M Llama model with W4A4 quantization. C ABLATION OF CAGE HYPERPARAMETERS We conduct a two-dimensional sweep over the CAGE hyperparameters; the silence ratio s and the regularization coefficient λ [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zero-Shot Quantization via Weight-Space Arithmetic

    cs.CV 2026-04 unverdicted novelty 8.0

    A quantization vector derived from a donor model via weight-space arithmetic can be added to a receiver model to improve post-PTQ Top-1 accuracy by up to 60 points in 3-bit settings without receiver-side QAT or data.

  2. WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points

    cs.LG 2026-05 unverdicted novelty 6.0

    WinQ accelerates quantization-aware training up to 4x and improves sub-4-bit accuracy up to 8.8% by weight interpolation resets and noise-regularized gradients that increase Hessian eigenvalue magnitudes around saddle points.

  3. TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference

    cs.CV 2026-04 unverdicted novelty 4.0

    Tiny NeRV models using capacity scaling, frequency-aware distillation, and low-precision quantization achieve favorable quality-efficiency trade-offs with far fewer parameters and lower computational costs than standard NeRV.

Reference graph

Works this paper leans on

22 extracted references · 21 linked inside Pith · cited by 3 Pith papers

  1. [1]

    K., Hartley, R., and Torr, P

    Ajanthan, T., Dokania, P. K., Hartley, R., and Torr, P. H. S. Proximal mean-field for neural network quantization. arXiv preprint arXiv:1812.04353,

    Pith/arXiv arXiv

  2. [9]

    URL https://arxiv.org/abs/2509. 23202. Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization.arXiv preprint arXiv:1902.08153,

    Pith/arXiv arXiv 1902

  3. [11]

    org/abs/2210.17323

    URL https://arxiv. org/abs/2210.17323. Frantar, E., Evci, U., Park, W., Houlsby, N., and Alistarh, D. Compression scaling laws: Unifying sparsity and quantization,

    Pith/arXiv arXiv

  4. [12]

    S., Love, J., et al

    Gemma, T., Mesnard, T., Hardin, C., Dadashi, R., Bhupati- raju, S., Pathak, S., Sifre, L., Rivi `ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and technology.arXiv preprint arXiv:2403.08295,

    Pith/arXiv arXiv

  5. [13]

    Hou, L., Zhang, R., and Kwok, J

    URL https://arxiv.org/ abs/2203.15556. Hou, L., Zhang, R., and Kwok, J. T. Analysis of Quantized Models.ICLR,

    Pith/arXiv arXiv

  6. [14]

    Jordan, K

    URLhttps://arxiv.org/abs/2503.15748. Jordan, K. Muon: An optimizer for the hidden layers of neu- ral networks. https://kellerjordan.github. io/posts/muon/,

    Pith/arXiv arXiv

  7. [15]

    Krishnamoorthi, R

    URL https://arxiv.org/abs/2005.11035. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper.arXiv preprint arXiv:1806.08342,

    Pith/arXiv arXiv 2005

  8. [16]

    F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A. Scaling laws for precision.arXiv preprint arXiv:2411.04330,

    Pith/arXiv arXiv

  9. [17]

    Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L

    URL https: //arxiv.org/abs/2510.08757. Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, S., Gu, Y ., Malik, S., Graf, V ., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y ., Dasigi, P., and Hajishirzi, H. T ¨ulu 3: Pushing frontiers ...

    arXiv

  10. [18]

    K., Lin, C.-T., and Zach, C

    Le, H., Høier, R. K., Lin, C.-T., and Zach, C. AdaSTE: An Adaptive Straight-Through Estimator to Train Bi- nary Neural Networks.arXiv preprint arXiv:2112.02880,

    Pith/arXiv arXiv

  11. [19]

    Network quantization with element-wise gradient scaling, 2021a

    Lee, J., Kim, D., and Ham, B. Network quantization with element-wise gradient scaling, 2021a. URL https: //arxiv.org/abs/2104.00903. Lee, J., Kim, D., and Ham, B. Network quantization with element-wise gradient scaling. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6448–6457, 2021b. Li, H., De, S., Xu, Z., Studer...

    Pith/arXiv arXiv

  12. [20]

    Liu, L., Dong, C., Liu, X., Yu, B., and Gao, J

    URL https://arxiv.org/abs/2006.07253. Liu, L., Dong, C., Liu, X., Yu, B., and Gao, J. Bridging dis- crete and backpropagation: Straight-through and beyond. Advances in Neural Information Processing Systems, 36: 12291–12311,

    Pith/arXiv arXiv 2006

  13. [21]

    net/forum?id=Bkg6RiCqY7

    URL https://openreview. net/forum?id=Bkg6RiCqY7. Nagel, M., Fournarakis, M., Bondarenko, Y ., and Blankevoort, T. Overcoming oscillations in quantization- aware training, 2022a. URL https://arxiv.org/ abs/2203.11086. Nagel, M., Fournarakis, M., Bondarenko, Y ., and Blankevoort, T. Overcoming oscillations in quantization- aware training. InInternational Co...

    Pith/arXiv arXiv

  14. [22]

    Panferov, A., Chen, J., Tabesh, S., Nikdan, M., and Al- istarh, D

    Accessed 2025-09-21. Panferov, A., Chen, J., Tabesh, S., Nikdan, M., and Al- istarh, D. Quest: Stable training of llms with 1-bit weights and activations. InProceedings of the 42nd In- ternational Conference on Machine Learning (ICML), 2025a. URL https://openreview.net/forum? id=I0Ux2nAN6u. Panferov, A., V olkova, A., Modoranu, I.-V ., Egiazarian, V ., Sa...

    Pith/arXiv arXiv 2025

  15. [2018]

    Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432, 2013a

    Bengio, Y ., L´eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432, 2013a. Bengio, Y ., L´eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation.arXiv preprint arXiv:1308.3432, 2013b. URL...

    Pith/arXiv arXiv

  16. [2019]

    Ajanthan, T., Gupta, K., Torr, P. H. S., Hartley, R., and Dokania, P. K. Mirror Descent View for Neural Network Quantization.arXiv preprint arXiv:1910.08237,

    Pith/arXiv arXiv 1910

  17. [2020]

    Quantized Adam with Error Feedback.arXiv preprint arXiv:2004.14180, 2021a

    Chen, C., Shen, L., Huang, H., and Liu, W. Quantized Adam with Error Feedback.arXiv preprint arXiv:2004.14180, 2021a. Chen, C., Shen, L., Huang, H., and Liu, W. Quantized adam with error feedback, 2021b. URL https://arxiv. org/abs/2004.14180. Darvish Rouhani, B., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, Ritchie amd Hall, M., Klar, J., Chung, ...

    Pith/arXiv arXiv 2004

  18. [2021]

    Proxquant: Quantized neural networks via proximal operators.arXiv preprint arXiv:1810.00861,

    Bai, Y ., Wang, Y .-X., and Liberty, E. Proxquant: Quantized neural networks via proximal operators.arXiv preprint arXiv:1810.00861,

    Pith/arXiv arXiv

  19. [2022]

    Egiazarian, V ., Castro, R

    URLhttps://arxiv.org/abs/2104.09987. Egiazarian, V ., Castro, R. L., Kuznedelev, D., Panferov, A., Kurtic, E., Pandit, S., Marques, A., Kurtz, M., Ashkboos, S., Hoefler, T., and Alistarh, D. Bridging the gap between promise and performance for microscaling fp4 quantiza- tion,

    Pith/arXiv arXiv

  20. [2023]

    Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfon- brener, D., Janson, L., and Kakade, S

    URL https://arxiv.org/abs/2307.09288. Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfon- brener, D., Janson, L., and Kakade, S. M. SOAP: Im- proving and stabilizing shampoo using adam for lan- guage modeling. InThe Thirteenth International Confer- ence on Learning Representations,

    Pith/arXiv arXiv

  21. [2024]

    Climb: Clustering-based iterative data mixture bootstrap- ping for language model pre-training.arXiv preprint arXiv:2504.13161,

    Diao, S., Yang, Y ., Fu, Y ., Dong, X., Su, D., Kliegl, M., Chen, Z., Belcak, P., Suhara, Y ., Yin, H., et al. Climb: Clustering-based iterative data mixture bootstrap- ping for language model pre-training.arXiv preprint arXiv:2504.13161,

    Pith/arXiv arXiv

  22. [2025]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv