pith. sign in

arxiv: 2606.25087 · v1 · pith:42IXXRV5new · submitted 2026-06-23 · 💻 cs.CV

Neural Network Quantization by Learning Low-Loss Subspaces

Vladimir Protsenko , Mikhalina Kharkevich , Alexander Vashchilko , Vladimir Kryzhanovskiy This is my paper

Pith reviewed 2026-06-26 00:03 UTC · model grok-4.3

classification 💻 cs.CV
keywords neural network quantizationlow-loss subspaceslinear paths in weight spacequantization-aware trainingloss landscapedirect quantizationstraight-through estimator
0
0 comments X

The pith

Direct quantization of the midpoint of a learned low-loss subspace matches the accuracy of quantization-aware training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper starts from the observation that well-optimized full-precision networks lie inside connected low-loss regions of the loss surface rather than at isolated points. It constructs linear paths between such points by optimizing them to keep loss low while remaining aware of future quantization. The midpoint of each such path turns out to be a point whose direct quantization recovers nearly the same accuracy as training that explicitly accounts for quantization. Because the construction never discretizes weights or uses a straight-through estimator, it separates the search for a good subspace from the act of quantization itself. A reader would care if this route proves simpler or more stable than current quantization-aware pipelines.

Core claim

Optimizing quantization-aware linear paths in weight space to minimize loss produces a subspace whose midpoint is, by construction, quantization-friendly; quantizing that midpoint directly yields performance comparable to quantization-aware training without employing the straight-through estimator or any explicit discretization step during the optimization.

What carries the argument

Quantization-aware linear paths in weight space, optimized to minimize loss; their midpoint is the central object that is shown to remain quantization-friendly by design.

If this is right

  • Quantized models can be produced without running discretization or straight-through estimators inside the training loop.
  • The same low-loss subspace supplies both the full-precision solution and a ready-to-quantize point.
  • No separate post-training quantization step or fine-tuning is required after the subspace is learned.
  • The approach decouples the discovery of a quantization-friendly weight region from the choice of bit-width or quantization scheme.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same subspace construction might be reused for other forms of model compression such as pruning or low-rank approximation.
  • If the midpoint property holds across different architectures, the method could reduce the need for architecture-specific quantization schedules.
  • Testing the approach on very low bit-widths (2- or 3-bit) would reveal whether the subspace remains quantization-friendly when discretization error grows larger.

Load-bearing premise

Low-loss full-precision solutions lie in connected low-loss subspaces, and linear paths optimized inside those subspaces keep their midpoint friendly to later quantization.

What would settle it

On a standard benchmark such as ImageNet with ResNet-50, if the top-1 accuracy after direct 8-bit quantization of the learned midpoint falls more than 1 percent below the accuracy obtained by standard quantization-aware training, the central claim is falsified.

Figures

Figures reproduced from arXiv: 2606.25087 by Alexander Vashchilko, Mikhalina Kharkevich, Vladimir Kryzhanovskiy, Vladimir Protsenko.

Figure 1
Figure 1. Figure 1: Left panel: The training loss functions Ltask, Rqdist (b = 4) (inset) and Ltotal = Ltask + λqRqdist for ResNet-18/CIFAR-100. Right panels: Density KDE-plots of the distribution of distances between the endpoints dp = |Θ (1) p − Θ (2) p | in the p = "layer3.0.conv2" layer for different epochs (dp, Θ(1),(2) p ∈ R 256×256×3×3 and there are Dp ≈ 6 × 105 values on the "x"-axis). The red vertical line shows dist… view at source ↗
Figure 2
Figure 2. Figure 2: Test accuracy of ResNet-18 on CIFAR-100 as a function of α for a single selected layer (indicated in each panel). For that layer, Θp → (1 − α) Θp(1) + αΘ(2) p , with α ∈ [0, 1], while for all other layers α = 0.5. sharply concentrated around d = s (mid), with a small residual tail for d > s (mid) , and Rqdist becomes negligible. This training dynamic is consistent across experi￾ments (Tabs. 1 to 3). Despit… view at source ↗
Figure 3
Figure 3. Figure 3: Test accuracy of QLS(4-bit train, 4, 5, 6-bit inference) quantized ResNet-18 on CIFAR-100 along the “layer3.0.conv2” subspace. All other layers use α = 0.5 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Upper panels: Test accuracy at higher bitwidths than training. Lower panels: As in Figs. 4a,b, with scale factors directly converted to the powers-of-two. adjustments, yielding significant accuracy gains. As a result, the (QLS → LSQ) pipeline consistently outperforms LSQ-based training alone. Notably, this per￾formance cannot be achieved by LSQ, which starts from a standard FP model, regardless of the choi… view at source ↗
Figure 1
Figure 1. Figure 1: Functional forms of the penalty function F(∆) [PITH_FULL_IMAGE:figures/full_fig_p020_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Fraction of zero weights across layers in W4Afp ResNet-18/CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fraction of zero weights across layers in W3Afp RFDN model [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
read the original abstract

Neural network quantization aims to find a discrete representation of parameters that preserves the performance of a full-precision (FP) model as faithfully as possible. Enforcing discrete constraints perturbs parameters away from a well-optimized minimum, generally resulting in performance degradation. Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape, where the loss maintains nearly the same minimum value. Models sampled from these subspaces are diverse and retain high accuracy. This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance? We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss. We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training. The proposed procedure offers a novel perspective on weight quantization and, in contrast to conventional methods, neither relies on the straight-through estimator nor involves explicit discretization during training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper proposes learning quantization-aware linear paths in weight space, optimized solely to minimize full-precision (FP) loss without using the straight-through estimator (STE) or explicit discretization during training. Building on the existence of connected low-loss subspaces in the FP loss landscape, it claims that the midpoint of the resulting subspace is quantization-friendly by design, such that its direct (post-training) quantization achieves performance comparable to quantization-aware training (QAT).

Significance. If the central claim holds with supporting experiments, the work would provide a conceptually distinct route to quantization that sidesteps STE-related instabilities and discretization during optimization. It would also give a concrete use case for low-loss subspace connectivity results. However, the significance is currently difficult to assess because the manuscript provides no quantitative results, error bars, dataset details, or ablation studies in the available description.

major comments (1)
  1. [Abstract] Abstract: The claim that the midpoint is 'quantization-friendly by design' is not mechanistically justified by the stated procedure. The optimization minimizes FP loss along linear paths; quantization applies a rounding perturbation that is never incorporated (even approximately) into the objective. Low-loss connectivity guarantees that the midpoint retains low FP loss, but supplies no guarantee of reduced sensitivity to that specific perturbation relative to a conventional minimum. This directly undercuts the assertion that direct quantization will match QAT performance.
minor comments (1)
  1. The manuscript should clarify the precise parameterization of the linear paths (e.g., how the two endpoints are initialized and constrained) and the exact loss used for path optimization.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their detailed review and constructive feedback on our work. We provide a point-by-point response to the major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claim that the midpoint is 'quantization-friendly by design' is not mechanistically justified by the stated procedure. The optimization minimizes FP loss along linear paths; quantization applies a rounding perturbation that is never incorporated (even approximately) into the objective. Low-loss connectivity guarantees that the midpoint retains low FP loss, but supplies no guarantee of reduced sensitivity to that specific perturbation relative to a conventional minimum. This directly undercuts the assertion that direct quantization will match QAT performance.

    Authors: We agree with the referee that the optimization procedure minimizes the full-precision loss and does not explicitly incorporate the quantization rounding into the objective function. The low-loss subspace connectivity indeed ensures that the midpoint has low FP loss, but does not automatically imply reduced sensitivity to quantization perturbations. Our assertion that the midpoint is 'quantization-friendly by design' was intended to convey that the learned subspace positions the midpoint in a region where direct quantization performs well, as validated by our experiments. However, we acknowledge that a stronger mechanistic explanation would strengthen the paper. We will revise the abstract to temper the 'by design' language and clarify that the performance comparability is demonstrated empirically. Additionally, we will add discussion in the main text on potential reasons why such subspaces yield quantization-robust points, such as the flatness of the loss landscape along the learned direction potentially mitigating small perturbations like rounding. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation depends on external low-loss subspace literature

full rationale

The paper's central construction optimizes linear paths in weight space to minimize FP loss (explicitly without STE or discretization) and invokes the midpoint's quantization-friendliness via the external claim that low-loss FP solutions lie in connected subspaces. This relies on cited prior studies rather than any internal equation that defines the target property in terms of the optimization output or renames a fitted quantity as a prediction. No self-citation chain, ansatz smuggling, or self-definitional reduction appears in the provided text; the result is therefore not forced by the paper's own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that low-loss solutions form connected subspaces and that linear paths optimized for quantization will place the midpoint in a quantization-friendly location.

axioms (1)
  • domain assumption Low-loss full-precision solutions belong to connected low-loss subspaces of the loss landscape
    Invoked in the abstract as the foundation for the proposed method.

pith-pipeline@v0.9.1-grok · 5722 in / 1128 out tokens · 17278 ms · 2026-06-26T00:03:00.471613+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 33 canonical work pages · 11 internal anchors

  1. [2]

    arXiv preprint arXiv:2404.11769 (2024)

    AskariHemmat, M., Jeddi, A., Hemmat, R.A., Lazarevich, I., Hoffman, A., Sah, S., Saboori, E., Savaria, Y., David, J.P.: Qgen: On the ability to generalize in quantization aware training. arXiv preprint arXiv:2404.11769 (2024)

    work page arXiv 2024

  2. [3]

    Baldi, T., Campos, J., Weng, O., Geniesse, C., Tran, N., Kastner, R., Biondi, A.: Losslandscapeanalysisforreliablequantizedmlmodelsforscientificsensing.arXiv preprint arXiv:2502.08355 (2025)

    work page arXiv 2025

  3. [4]

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

    Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  4. [5]

    Advances in Neural Information Processing Systems32(2019)

    Chen,S.,Wang,W.,Pan,S.J.:Metaquant:Learningtoquantizebylearningtopen- etrate non-differentiable quantization. Advances in Neural Information Processing Systems32(2019)

    2019

  5. [6]

    Advances in neural information processing systems33, 5308–5317 (2020)

    Chmiel,B.,Banner,R.,Shomron,G.,Nahshan,Y.,Bronstein,A.,Weiser,U.,etal.: Robust quantization: One model to rule them all. Advances in neural information processing systems33, 5308–5317 (2020)

    2020

  6. [7]

    In: International conference on machine learning

    Draxler, F., Veschgini, K., Salmhofer, M., Hamprecht, F.: Essentially no barriers in neural network energy landscape. In: International conference on machine learning. pp. 1309–1318. PMLR (2018)

    2018

  7. [8]

    arXiv preprint arXiv:1902.08153 (2019)

    Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned step size quantization. arXiv preprint arXiv:1902.08153 (2019)

    work page arXiv 1902

  8. [9]

    arXiv preprint arXiv:2511.04808 (2025)

    Fan, R., Sandlund, B., Ko, L.M.: Sharp minima can generalize: A loss landscape perspective on data. arXiv preprint arXiv:2511.04808 (2025)

    work page arXiv 2025

  9. [10]

    In: International Conference on Artificial Intelligence and Statistics

    Ferbach, D., Goujaud, B., Gidel, G., Dieuleveut, A.: Proving linear mode connec- tivity of neural networks via optimal transport. In: International Conference on Artificial Intelligence and Statistics. pp. 3853–3861. PMLR (2024)

    2024

  10. [11]

    Sharpness-Aware Minimization for Efficiently Improving Generalization

    Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  11. [12]

    qualitatively characterizing neural network optimization problems

    Frankle, J.: Revisiting" qualitatively characterizing neural network optimization problems". arXiv preprint arXiv:2012.06898 (2020)

    work page arXiv 2012

  12. [13]

    In: International conference on machine learning

    Frankle, J., Dziugaite, G.K., Roy, D., Carbin, M.: Linear mode connectivity and the lottery ticket hypothesis. In: International conference on machine learning. pp. 3259–3269. PMLR (2020)

    2020

  13. [14]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: Gptq: Accurate post- training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  14. [15]

    Topology and Geometry of Half-Rectified Network Optimization

    Freeman, C.D., Bruna, J.: Topology and geometry of half-rectified network opti- mization. arXiv preprint arXiv:1611.01540 (2016) 16 V. Protsenko et al

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  15. [16]

    Advances in neural infor- mation processing systems31(2018)

    Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D.P., Wilson, A.G.: Loss sur- faces, mode connectivity, and fast ensembling of dnns. Advances in neural infor- mation processing systems31(2018)

    2018

  16. [17]

    In: Low-Power Computer Vision, pp

    Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M.W., Keutzer, K.: A sur- vey of quantization methods for efficient neural network inference. In: Low-Power Computer Vision, pp. 291–326. Chapman and Hall/CRC (2022)

    2022

  17. [18]

    Qualitatively characterizing neural network optimization problems

    Goodfellow, I.J., Vinyals, O., Saxe, A.M.: Qualitatively characterizing neural net- work optimization problems. arXiv preprint arXiv:1412.6544 (2014)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  18. [19]

    arXiv preprint arXiv:2510.10693 (2025)

    Ichikawa, Y., Kashiwamura, S., Sakata, A.: High-dimensional learning dy- namics of quantized models with straight-through estimator. arXiv preprint arXiv:2510.10693 (2025)

    work page arXiv 2025

  19. [20]

    Averaging Weights Leads to Wider Optima and Better Generalization

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Aver- aging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  20. [21]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., Kalenichenko,D.:Quantizationandtrainingofneuralnetworksforefficientinteger- arithmetic-only inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2704–2713 (2018)

    2018

  21. [22]

    arXiv preprint arXiv:2410.06020 (2024)

    Javed, S., Le, H., Salzmann, M.: Qt-dog: Quantization-aware training for domain generalization. arXiv preprint arXiv:2410.06020 (2024)

    work page arXiv 2024

  22. [23]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Jiang, J., Meng, Y., Tang, C., Yu, H., Li, Q., Wang, Z., Zhu, W.: Quantization meets ood: Generalizable quantization-aware training from a flatness perspective. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 11937–11946 (2025)

    2025

  23. [24]

    arXiv preprint arXiv:2305.19510 (2023)

    Karhadkar, K., Murray, M., Tseran, H., Montúfar, G.: Mildly overparameterized relu networks have a favorable loss landscape. arXiv preprint arXiv:2305.19510 (2023)

    work page arXiv 2023

  24. [25]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Kim, D., Lee, J., Ham, B.: Distance-aware quantization. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5271–5280 (2021)

    2021

  25. [26]

    Quantizing deep convolutional networks for efficient inference: A whitepaper

    Krishnamoorthi, R.: Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  26. [27]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Lee, J., Kim, D., Ham, B.: Network quantization with element-wise gradient scal- ing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6448–6457 (2021)

    2021

  27. [28]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Lee, T., Seo, K., Yoo, J., Yoon, S.W.: Understanding flatness in generative models: Its role and benefits. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4908–4917 (2025)

    2025

  28. [29]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Li, A., Zhuang, L., Long, X., Yao, M., Wang, S.: Seeking consistent flat minima for better domain generalization via refining loss landscapes. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 15349–15359 (2025)

    2025

  29. [30]

    In: Findings of the association for computational linguistics: EMNLP 2024

    Li, H., Ding, L., Fang, M., Tao, D.: Revisiting catastrophic forgetting in large lan- guage model tuning. In: Findings of the association for computational linguistics: EMNLP 2024. pp. 4297–4308 (2024)

    2024

  30. [31]

    arXiv preprint arXiv:2205.131044(2022)

    Li, T., Huang, Z., Tao, Q., Wu, Y., Huang, X.: Trainable weight averaging for fast convergence and better generalization. arXiv preprint arXiv:2205.131044(2022)

    work page arXiv 2022

  31. [32]

    arXiv preprint arXiv:1909.13144 (2019)

    Li, Y., Dong, X., Wang, W.: Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. arXiv preprint arXiv:1909.13144 (2019)

    work page arXiv 1909

  32. [33]

    In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16

    Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. pp. 41–55. Springer (2020) Neural Network Quantization by Learning Low-Loss Subspaces 17

    2020

  33. [34]

    Sharpness-aware quantization for deep neural networks

    Liu,J.,Cai,J.,Zhuang,B.:Sharpness-awarequantizationfordeepneuralnetworks. arXiv preprint arXiv:2111.12273 (2021)

    work page arXiv 2021

  34. [35]

    Advances in Neural Information Processing Systems 36, 12291–12311 (2023)

    Liu, L., Dong, C., Liu, X., Yu, B., Gao, J.: Bridging discrete and backpropagation: Straight-through and beyond. Advances in Neural Information Processing Systems 36, 12291–12311 (2023)

    2023

  35. [36]

    Relaxed Quantization for Discretized Neural Networks

    Louizos, C., Reisser, M., Blankevoort, T., Gavves, E., Welling, M.: Relaxed quan- tization for discretized neural networks. arXiv preprint arXiv:1810.01875 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  36. [37]

    arXiv preprint arXiv:2505.04877 (2025)

    Ma, L., Ma, J., Zhou, Y., Xie, G., He, Q., Lu, Z.: Learning from loss landscape: Generalizable mixed-precision quantization via adaptive sharpness-aware gradient aligning. arXiv preprint arXiv:2505.04877 (2025)

    work page arXiv 2025

  37. [38]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Nagel, M., Baalen, M.v., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1325–1334 (2019)

    2019

  38. [39]

    A White Paper on Neural Network Quantization

    Nagel, M., Fournarakis, M., Amjad, R.A., Bondarenko, Y., Van Baalen, M., Blankevoort, T.: A white paper on neural network quantization. arXiv preprint arXiv:2106.08295 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  39. [40]

    In: International Conference on Machine Learning

    Nagel, M., Fournarakis, M., Bondarenko, Y., Blankevoort, T.: Overcoming oscil- lations in quantization-aware training. In: International Conference on Machine Learning. pp. 16318–16330. PMLR (2022)

    2022

  40. [41]

    Machine Learning 110(11), 3245–3262 (2021)

    Nahshan, Y., Chmiel, B., Baskin, C., Zheltonozhskii, E., Banner, R., Bronstein, A.M., Mendelson, A.: Loss aware post-training quantization. Machine Learning 110(11), 3245–3262 (2021)

    2021

  41. [42]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Nunez, E., Horton, M., Prabhu, A., Ranjan, A., Farhadi, A., Rastegari, M.: Lcs: Learning compressible subspaces for efficient, adaptive, real-time network compres- sion at inference time. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3818–3827 (2023)

    2023

  42. [43]

    IEEE Transactions on Neural Networks and Learning Systems35(12), 17164–17176 (2023)

    Pei, Z., Yao, X., Zhao, W., Yu, B.: Quantization via distillation and contrastive learning. IEEE Transactions on Neural Networks and Learning Systems35(12), 17164–17176 (2023)

    2023

  43. [44]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Shin, S., Park, J., Boo, Y., Sung, W.: Hlhlp: Quantized neural networks training for reaching flat minima in loss surface. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 5784–5791 (2020)

    2020

  44. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Spallanzani, M., Leonardi, G.P., Benini, L.: Training quantised neural networks with ste variants: the additive noise annealing algorithm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 470–479 (2022)

    2022

  45. [46]

    arXiv preprint arXiv:2007.12927 (2020)

    Von Oswald, J., Kobayashi, S., Meulemans, A., Henning, C., Grewe, B.F., Sacramento, J.: Neural networks with late-phase weights. arXiv preprint arXiv:2007.12927 (2020)

    work page arXiv 2007

  46. [47]

    arXiv preprint arXiv:2210.07171 (2022)

    Wang, Z., Li, J.B., Qu, S., Metze, F., Strubell, E.: Squat: Sharpness-and quantization-aware training for bert. arXiv preprint arXiv:2210.07171 (2022)

    work page arXiv 2022

  47. [49]

    In: International conference on machine learning

    Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.: Model soups: averagingweightsofmultiplefine-tunedmodelsimprovesaccuracywithoutincreas- ing inference time. In: International conference on machine learning. pp. 23965– 23998. PMLR (2022)

    2022

  48. [50]

    Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

    Wu, L., Zhu, Z., et al.: Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv:1706.10239 (2017) 18 V. Protsenko et al

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  49. [51]

    arXiv preprint arXiv:2511.01462 (2025)

    Xia, P., Pang, J., Cai, T.: Efficiently training a flat neural network before it has been quantizated. arXiv preprint arXiv:2511.01462 (2025)

    work page arXiv 2025

  50. [52]

    In: International conference on machine learning

    Xiao,G.,Lin,J.,Seznec,M.,Wu,H.,Demouth,J.,Han,S.:Smoothquant:Accurate and efficient post-training quantization for large language models. In: International conference on machine learning. pp. 38087–38099. PMLR (2023)

    2023

  51. [53]

    arXiv preprint arXiv:2204.12322 (2022)

    Yao, H., Li, P., Cao, J., Liu, X., Xie, C., Wang, B.: Rapq: Rescuing accuracy for power-of-two low-bit post-training quantization. arXiv preprint arXiv:2204.12322 (2022)

    work page arXiv 2022

  52. [54]

    arXiv preprint arXiv:1903.05662 (2019)

    Yin, P., Lyu, J., Zhang, S., Osher, S., Qi, Y., Xin, J.: Understanding straight- through estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662 (2019)

    work page arXiv 1903

  53. [55]

    arXiv preprint arXiv:2412.15846 (2024)

    Yu, C., Yang, S., Zhang, F., Ma, H., Wang, A., Li, E.P.: Improving quantization- aware training of low-precision network via block replacement on full-precision counterpart. arXiv preprint arXiv:2412.15846 (2024)

    work page arXiv 2024

  54. [56]

    arXiv preprint arXiv:2506.10463 (2025)

    Yun, S., Wong, A.: Starting positions matter: A study on better weight initializa- tion for neural network quantization. arXiv preprint arXiv:2506.10463 (2025)

    work page arXiv 2025

  55. [57]

    arXiv preprint arXiv:2206.05916 (2022)

    Zhang, K., Yin, M., Wang, Y.X.: Why quantization improves generalization: Ntk of binary weight neural networks. arXiv preprint arXiv:2206.05916 (2022)

    work page arXiv 2022

  56. [58]

    arXiv preprint arXiv:2406.16988 (2024)

    Zhou, Y., Chen, J., Cao, Q., Schürholt, K., Yang, Y.: Md tree: a model-diagnostic tree grown on loss landscape. arXiv preprint arXiv:2406.16988 (2024)

    work page arXiv 2024

  57. [59]

    Zhou, Z., Sun, Y., Li, B., Wang, J., Wu, H., Wu, L., Yan, J.: How does local landscape geometry evolve in language model pre-training?

  58. [60]

    Neural Network Quantization by Learning Low-Loss Subspaces

    Zimmer, M., Spiegel, C., Pokutta, S.: Sparse model soups: A recipe for improved pruning via model averaging. arXiv preprint arXiv:2306.16788 (2023) Supplemental Material to “Neural Network Quantization by Learning Low-Loss Subspaces” Vladimir Protsenko , Mikhalina Kharkevich , Alexander Vashchilko , and Vladimir Kryzhanovskiy Huawei, Montreal, Canada {pro...

    work page arXiv 2023

  59. [61]

    ResNet-18 & ResNet-50 (TinyImageNet).Models are trained for 300 epochs with a batch size of 128

    The batch size is 64, and Adam is used with the same weight decay as above. ResNet-18 & ResNet-50 (TinyImageNet).Models are trained for 300 epochs with a batch size of 128. The initial learning rate is 0.1 and is decayed to zero using a cosine scheduler. Optimization is performed using SGD with a weight decay of5×10 −4. ResNet-18 & ResNet-50 (ImageNet).Mo...

  60. [62]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 126–135 (2017)

    2017

  61. [63]

    Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

    2012

  62. [64]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  63. [65]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5197–5206 (2015)

    2015

  64. [66]

    Adam: A Method for Stochastic Optimization

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  65. [67]

    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    2009

  66. [68]

    CS 231N7(7), 3 (2015)

    Le, Y., Yang, X., et al.: Tiny imagenet visual recognition challenge. CS 231N7(7), 3 (2015)

    2015

  67. [69]

    In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16

    Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. pp. 41–55. Springer (2020)

    2020

  68. [70]

    International journal of computer vision115, 211–252 (2015)

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International journal of computer vision115, 211–252 (2015)

    2015

  69. [71]

    In: International Conference on Machine Learning

    Wortsman, M., Horton, M.C., Guestrin, C., Farhadi, A., Rastegari, M.: Learning neural network subspaces. In: International Conference on Machine Learning. pp. 11217–11227. PMLR (2021)

    2021

  70. [72]

    In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7

    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse- representations. In: Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7. pp. 711–730. Springer (2012)

    2010