Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Nearly Lossless Adaptive Bit Switching

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single integer-weight model stored at the highest bit-width can switch to any lower precision at inference with nearly lossless accuracy: Double Rounding makes each lower bit a sub-representation of the stored integer.

desk verdict Double Rounding is a genuinely new way to store one integer checkpoint for multiple precisions, but the ALRS update as written cannot be what produced the results. read the letter →

arxiv 2502.01199 v1 pith:XH632Q2H submitted 2025-02-03 cs.CV cs.AI

Haiduo Huang , Zhenhua Liu , Tian Xia , Wenzhe zhao , Pengju Ren This is my paper · ORCID
classification cs.CVcs.AI
keywords doubleroundingadaptivebit-widthswitchingmulti-precisionquantizationmixed-precisionone-shotjointtrainingquantization-awareHessian-awarestochasticbit-switchinglearnedstepsize
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single quantized network, stored as one integer-weight model at the highest bit-width (typically 8-bit), can be switched to any lower bit-width — 6, 4, or 2 bits — at inference time with accuracy almost equal to training each precision separately, removing the need to keep a full-precision model. The load-bearing mechanism is Double Rounding: quantize to the highest bit first, then round a second time to reach the lower bit, so every low-bit weight is a sub-representation of the stored 8-bit integer and one learned scale serves all precisions. To make one-shot joint training stable, the paper adds Adaptive Learning Rate Scaling (ALRS), which rescales each precision's quantization-scale learning rate and closes most of the convergence gap between high and low precisions. For mixed precision, Hessian-Aware Stochastic Bit-switching (HASB) biases bit-width assignment by per-layer Hessian trace and lets sub-networks be extracted by Integer Linear Programming without retraining. If the claims hold, adaptive precision becomes a deployment-time choice instead of a retraining project, and the artifact that travels to the device is four to eight times smaller than today's float-storing baselines.

What carries the argument

Double Rounding (Eqs. 1–3) carries the storage claim: weights are quantized once to the highest bit-width $h$ with one learned scale $s_h$ (symmetric quantization, so the zero point is zero), and any lower bit-width $l$ is obtained by rounding the stored integer a second time, $\tilde{W}_l = \text{clip}(\lfloor \tilde{W}_h / 2^{\Delta} \rceil, -2^{l-1}, 2^{l-1}-1)$ with $\Delta = h-l$, so that the low-bit integer is contained in the high-bit integer and the dequantized value is $\hat{W}_l = \tilde{W}_l \cdot s_h \cdot 2^{\Delta}$. Because the scale relation is a power of two, the division and multiplication are hardware shifts, so switching precision costs almost nothing at runtime. Adaptive Learning Rate Scaling (ALRS, Eqs. 6–7) carries the stabilization claim: it scales the quantization-scale learning rate of each precision by a power of ten fixed by the bit gap (1 for 8-bit, 0.1 for 6-bit, 0.01 for 4-bit), countering the empirically observed order-of-magnitude divergence of low-bit scale gradients during joint training. Hessian-Aware Stochastic Bit-switching (HASB, Algorithm 1) carries the mixed-precision claim: the Hessian matrix trace of each layer sets the probability that the Roulette sampler assigns a high bit-width to that layer, and Integer Linear Programming (Algorithm 2) extracts the Pareto-optimal sub-networks without retraining.

What would settle it

Train MobileNetV2 twice under the {8,6,4,2}-bit schedule, once with the shared high-bit scale and once with an independently learned 2-bit scale, and compare 2-bit top-1 after the same epoch count; if the independent scale recovers a large share of the reported 20-point gap (50.43 vs. 71.14 full precision), then the fixed power-of-two scale, not training competition, is what caps low-bit accuracy on compact models, and the near-lossless claim fails for exactly the architectures where deployment savings matter most.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in Section 3.1, is that rounding twice makes bit-switching nearly lossless while letting the model be stored as the highest-bit integer weight instead of a full-precision weight. Prior methods either kept an FP32 master model with separate quantization parameters per bit-width, or replaced rounding with floor so a single integer model could be shared, accepting accuracy loss at low bits. Double Rounding instead re-rounds the stored high-bit integer $\tilde{W}_h$ down to $\tilde{W}_l$ with a power-of-two scale coupling $s_l = s_h \cdot 2^{\Delta}$, so every low-precision network is literally a sub-representation of one stored integer model; on ImageNet-1K the paper reports accuracy that matches or exceeds prior one-shot multi-precision and mixed-precision methods that store 32-bit models, while storing only 8-bit (or 4-bit) weights. The paper additionally claims to have located the reason joint multi-precision training is unstable — quantization-scale gradients differ across precisions by an order of magnitude, with 2-bit gradients largest — and shows that rescaling per-precision scale learning rates (ALRS) narrows the convergence gap. For mixed precision, it claims that Hessian-trace sensitivity, used to bias stochastic bit-width sampling, plus a direct ILP search, removes the retraining stage and yields a Pareto frontier of bit allocations.

Load-bearing premise

The load-bearing premise is that a single quantization scale and zero point, fit to the highest bit-width, stays near-optimal for every lower bit-width through the fixed power-of-two relation $s_l = s_h \cdot 2^{\Delta}$, so that when the high-bit scale does not match the low-bit weight distribution, the low-bit model begins training with a step size it cannot escape.

Editorial extensions

If this is right

  • Deployment becomes bit-switchable from one artifact: a model stored as 8-bit integers can be served at 6, 4, or 2 bits without reloading weights, because each lower precision is derived from the stored integers by a shift-and-round.
  • Storage drops to a quarter of the FP32 baseline for the 8-bit configuration and to an eighth for the {4,3,2}-bit configuration, with ImageNet-1K accuracy matching or exceeding one-shot baselines that store 32-bit models.
  • Joint multi-precision training costs stay roughly constant as the candidate list grows — ResNet50 on ImageNet costs about 40.8 GPU-hours for four precisions versus 51.6 for training them separately — so more precisions no longer multiply the training budget.
  • The mixed-precision pipeline drops the final retraining or fine-tuning stage: ILP search returns each sub-network in under two minutes from the trained SuperNet, and the searched configurations form a Pareto frontier across bit budgets.
  • The single-model switching transfers beyond classification: COCO detection and instance segmentation with Mask-RCNN backbones, and TinyLlama zero-shot tasks, retain most of their accuracy across the 8/6/4/2-bit settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading of the paper's own Table 1 is that the 'nearly lossless' claim is strained where deployment matters most: MobileNetV2 drops to 50.43 top-1 at 2 bits from 71.14 full precision after joint training, which suggests the shared high-bit scale fits depthwise-heavy compact models poorly; the authors flag this themselves and point to per-layer or per-channel learning-rate scaling as future work
  • An operational consequence the paper motivates but does not measure is that precision becomes a runtime knob on integer hardware: because all lower precisions are shift-and-rounds of the stored integers, a device could drop from 8-bit to 4-bit on the fly to save bandwidth or energy without re-instantiating any weights.
  • Because activations keep per-precision learned scales (Eqs. 4–5), an integer-only deployment still carries several activation-scale tables; extending double rounding to activations with a single stored scale is an open direction the paper leaves implicit.
  • HASB is a quantization-specific way to spend a capacity budget by layer sensitivity, so the same Hessian-trace-biased sampling could plausibly be ported to other once-for-all sub-network choices such as pruning ratios or layer widths, a connection the paper does not draw.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a one-shot multi-precision and mixed-precision quantization framework. It introduces Double Rounding, which quantizes weights by first rounding to a high bit-width and then rounding again to lower bit-widths through a power-of-two scale relation, allowing a single stored integer-weight model to be switched among precisions at inference. It also proposes Adaptive Learning Rate Scaling (ALRS) to stabilize joint training across precisions, and a Hessian-Aware Stochastic Bit-switching (HASB) strategy for mixed-precision SuperNets, with an ILP-based search that avoids retraining. Experiments cover ImageNet-1K classification on ResNet and MobileNetV2, COCO detection/segmentation, and TinyLlama language-model tasks. The central claims are that Double Rounding gives nearly lossless bit-switching while storing only the highest integer precision, and that ALRS and HASB improve over prior one-shot multi-precision and mixed-precision methods.

Significance. If the results hold, the storage-saving property (8-bit instead of FP32 storage with runtime bit-switching) and the low-cost mixed-precision search would be practically useful contributions. The paper includes several strengths: results over multiple architectures and tasks, an ablation showing ALRS helps even when applied to prior methods (Table 7), a Pareto-frontier analysis of HASB, and a public code release. However, the significance is currently limited by a load-bearing technical inconsistency in the ALRS update rule and by the fact that the paper's own low-bit accuracy numbers do not support the 'nearly lossless' phrasing. These issues need to be resolved before the contribution can be evaluated on its merits.

major comments (4)
  1. [Section 3.2, Eq. (6) and Algorithm A.2] The ALRS learning-rate formula as written is not a valid update rule. With the training setup's initial Adam learning rate lambda = 5e-4 (Section 4.1), and with the averaged term (1/L) sum_i min(max_abs(clip_grad(nabla s_i^b, 1.0)), 1.0) lying in [0,1] by construction, Eq. (6) gives lambda_b = eta_b (lambda - [0,1]) < 0 for every precision and every iteration. A negative learning rate applied to the scale optimizer would move the quantization scales in the direction opposite to the gradient, which cannot produce the convergence shown in Figure 3(d) or the accuracies in Tables 1-2. Since Algorithm A.2 explicitly calls this formula, the method as described is internally inconsistent and non-reproducible. The authors need to correct the formula, e.g., by making the gradient statistic a multiplier or a relative adjustment of lambda, and state precisely what clip_grad(., 1.0) means (e.g., clip_grad_value_ versus clip_grad_norm_).
  2. [Section 3.1 and Abstract] The headline claim 'nearly lossless bit-switching' is contradicted by the paper's own results. In Table 1, the {8,6,4,2}-bit MobileNetV2 model loses 20.71 points at w2a2 without KD (50.43 vs. 71.14 FP) and 18.08 points with KD (53.06 vs. 71.14 FP); ResNet18 loses about 3.4 points at w2a2 (66.35/66.84 vs. 69.76 FP), and ResNet50 loses about 3.8 points at w2a2 (72.31/73.78 vs. 76.13 FP). These are substantial accuracy drops, especially for the compact model. The term 'nearly lossless' should either be replaced by a more precise claim (e.g., 'improved accuracy compared to prior multi-precision methods at the same storage budget') or be supported by an explicit accuracy-loss tolerance that these numbers satisfy.
  3. [Appendix A.4, Eqs. (13)-(14)] The gradient derivation does not cover the Double Rounding path. Equations (13)-(14) are the standard LSQ single-rounding gradients for the dequantized value (fW - z)*s or (round((Y - zh)/sh)*sh + zh); they do not differentiate through the second rounding in Eq. (2), i.e., through fW_l = clip(round(fW_h / 2^Delta), ...) and its dequantization cW_l = fW_l * sh * 2^Delta + zh in Eq. (3). The chain rule through the two rounding operations and through the factor 2^Delta is absent, so the exact training objective for the low-bit weight path is unspecified. This is load-bearing because the claimed contribution is precisely the two-step quantization; without the full gradient, the low-bit training procedure cannot be reproduced from the text.
  4. [Section 3.3 and Algorithm A.3] The HASB threshold sigma is introduced but never fully specified. Algorithm A.3 lists 'the threshold of bit-switching: sigma' in the Require block and then updates it as sigma = sigma * (epoch+1)/total_epochs (line 3), but the initial value of sigma is never given. If sigma starts at 0 the update keeps it at 0; if it starts at a positive value the formula is ambiguous about ordering and can push sigma above 1. Since the bit-selection behavior in Algorithm 1 is only invoked when r < sigma, the stochastic schedule is not well defined. The authors should provide the initial value and the intended schedule explicitly.
minor comments (5)
  1. [Section 3.2, Eq. (7)] The mapping from Eq. (7) to the values stated in the text (eta_b = 1, 0.1, 0.01 for 8/6/4-bit) should be written out; currently the reader must decode the even/odd cases to verify that 8-bit uses Delta=0 (eta=1), 6-bit uses Delta=2 (eta=0.1), and 4-bit uses Delta=4 (eta=0.01).
  2. [Section 4.1.1 and Table 1] The MobileNetV2 rows mix two different bit-list settings ({8,6,4}-bit and {8,6,4,2}-bit) in the same block; adding a horizontal separator or a column entry spelling out the bit list for every row would make the table easier to read and would avoid the appearance that the w8a8 numbers are directly comparable.
  3. [Section 3.1, Eqs. (1)-(3)] The notation in Eq. (2) uses the floor-symbol glyph (which is actually the rounding symbol) inconsistently with the text; it should be stated once that round(.) is used in all three equations and that fW_h is an integer tensor before the division by 2^Delta.
  4. [Section 4.3] The LLM experiments in Table 5 show w4a4 and w2a2 accuracies close to chance level for TinyLlama 1.1B (e.g., 36.08/38.46 average vs. 52.99 FP); this should be acknowledged in the main text when claiming that the method 'applies to more recent and complex models', otherwise the reader may over-read the generality claim.
  5. [Algorithm 1 and its note] The note in Algorithm 1 defines n and L only in the caption note rather than in the Require block; moving these definitions into the algorithm would improve clarity.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity in the HASB alignment validation; the central multi-precision accuracy claim rests on external ImageNet comparisons and is not circular.

  1. self definitional [Section 3.3, Algorithm 1; Appendix A.7]
    "Compute bit-switching probability of all candidate bi with pi = bi/∥B∥1; ... These results further validate the effectiveness of the proposed HASB technique."

    The observed 'alignment' of the learned bit allocation with the Hessian trace is built into the method: Algorithm 1 defines the sampling probability for sensitive layers proportionally to bit-width, so high bits are more likely for high-HMT layers by construction, and the ILP objective (Algorithm 2) also contains the HMT term. Appendix A.7 then cites this alignment as validation of HASB, which is a restatement of the design rather than an independent confirmation. The ImageNet accuracy results in Tables 1-3 remain external evidence, so this circularity is limited to the alignment-validity claim.

full rationale

The core of the paper is empirical: Double Rounding and ALRS are evaluated on held-out ImageNet-1K accuracy against external baselines, and no reported accuracy number is fitted into the method. The 'lower precision weight is included in higher precision weight' property follows directly from Eq. (2) by construction, but the paper's substantive claims are the measured accuracies, which are not circular. The ALRS update in Eq. (6) as written would produce negative learning rates at the stated base learning rate of 5e-4, since the averaged clipped-gradient term lies in [0,1]; this is an internal consistency and reproducibility concern, not a circularity. The only genuine circularity found is the Appendix A.7 validation that HASB allocations align with HMT, which is definitionally enforced by Algorithm 1 and Algorithm 2. Because that step is supportive rather than central, the overall circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central method rests on a shared learned scale and a power-of-two relationship between precisions, plus several hand-set hyperparameters for the training stabilizers. No new physical entities are introduced.

free parameters (5)
  • Weight quantization scale s_h per layer = Learned, not reported
    Double Rounding uses a single learned scale s_h for the highest bit-width; lower precisions inherit s_h*2^Delta, so its fitted value controls all precisions (Eq. 1-3).
  • Activation scales s_b and zero points z_b per precision = Learned per precision
    Activations use separately learned scales for each bit-width (Section 3.1, Eq. 4-5), fitted on training data.
  • ALRS scaling factors eta_b = 1, 0.1, 0.01, 0.001 for 8/6/4/2-bit
    Chosen by hand in Eq. 7 as an exponential function of bit difference; no search or theoretical justification is provided.
  • HASB bit-switching threshold sigma = Unspecified initial value, schedule in Algorithm A.3
    The threshold controls how often the Roulette bit-switch is applied; its initial value and final schedule are not specified in the paper.
  • Gradient clipping threshold in ALRS = 1.0
    The clip_grad(.,1.0) inside Eq. 6 is an ad hoc constant that directly enters the learning-rate adjustment.
assumptions (5)
  • standard math Straight-Through Estimator (STE) treats the rounding operation as identity for backpropagation
    Invoked in Section 3.1 for quantization-aware training, cited from Bengio et al. [26].
  • ad hoc to paper A single learned highest-bit scale and zero point can represent weights well at every lower bit width through power-of-two scaling
    Eq. 1-3 force s_l = s_h * 2^Delta; the paper does not prove this shared-scale assumption is near-optimal for low precisions.
  • domain assumption Hessian matrix trace computed on the pretrained FP32 model is a reliable sensitivity signal for bit-width allocation
    HASB (Section 3.3, Algorithm 1) uses HMT from the FP32 model, following HAWQ-V2 [11].
  • domain assumption The pretrained FP32 checkpoint is a valid starting point for all multi- and mixed-precision training
    Section 4.1 initializes all models from an FP32 pretrained model; the paper does not study initialization dependence.
  • ad hoc to paper Updating shared weights after each individual precision's forward pass, with dual optimizers, leads to stable convergence
    Algorithm A.2 uses separate per-precision updates with ALRS; the paper gives only empirical evidence, no convergence analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nearly Lossless Adaptive Bit Switching." pith.science (2026). https://pith.science/paper/XH632Q2H

@misc{pith2026250201199,
  author       = {Pith},
  title        = {Pith review of: Nearly Lossless Adaptive Bit Switching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XH632Q2H}},
  note         = {Machine review of arXiv:2502.01199}
}
read the original abstract

Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-Aware Training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the Double Rounding quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling (ALRS) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our Double Rounding to one-shot mixed precision training and develop a Hessian-Aware Stochastic Bit-switching (HASB) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. We also validate the feasibility of our method on detection and segmentation tasks, as well as on LLMs task. Our codes are available at https://github.com/haiduo/Double-Rounding.

Figures

Figures reproduced from arXiv: 2502.01199 by the authors.

Figure 1
Figure 1. Overview of our proposed lossless adaptive bit-switching strategy. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of four quantization schemes:(from left to right) used in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The statistics of ResNet18 on ImageNet-1K dataset. (a) and (b): The quantization scale [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The HASB stochastic process and Mixed-precision of ResNet18 for {2,4,6,8}-bit. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of HASB and Baseline approaches for Mixed-Precision on ResNet18. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison between different quantization types during quantization-aware training. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The scale gradient statistics of weight of ResNet20 on CIFAR-10 dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The scale gradient statistics of activation of ResNet20 on CIFAR-10 dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: The scale gradient statistics of weight of ResNet18 on ImageNet dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The scale gradient statistics of activation of ResNet18 on ImageNet dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: The scale gradient statistics of weight of ResNet50 on ImageNet dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: The scale gradient statistics of activation of ResNet50 on ImageNet dataset. Note that the [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Layer-Wise Bit-Widths Allocation of Mixed-Precision in ResNet18. (a) Without HASB, [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Layer-Wise Bit-Widths Allocation of Mixed-Precision in ResNet50. (a) Without HASB, [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Layer-Wise Bit-Widths Allocation of Mixed-Precision (3MP) in MobileNet-v2. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages

  1. [1]

    Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,

    S. Zhou, Y . Wu, Z. Ni, X. Zhou, H. Wen, and Y . Zou, “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160, 2016

    arXiv 2016

  2. [2]

    Learned step size quantization,

    S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, “Learned step size quantization,” arXiv preprint arXiv:1902.08153, 2019

    arXiv 1902

  3. [3]

    Adabits: Neural network quantization with adaptive bit-widths,

    Q. Jin, L. Yang, and Z. Liao, “Adabits: Neural network quantization with adaptive bit-widths,” inProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2146–2156

    work page 2020

  4. [4]

    Multiquant: Training once for multi-bit quantization of neural networks,

    K. Xu, Q. Feng, X. Zhang, and D. Wang, “Multiquant: Training once for multi-bit quantization of neural networks,” in IJCAI, L. D. Raedt, Ed. International Joint Conferences on Artificial Intelligence Organization, 7 2022, pp. 3629–3635, main Track. [Online]. Available: https://doi.org/10.24963/ijcai.2022/504

    doi:10.24963/ijcai.2022/504 2022

  5. [5]

    Eq-net: Elastic quantization neural networks,

    K. Xu, L. Han, Y . Tian, S. Yang, and X. Zhang, “Eq-net: Elastic quantization neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 1505–1514

    work page 2023

  6. [6]

    Any-precision deep neural networks,

    H. Yu, H. Li, H. Shi, T. S. Huang, and G. Hua, “Any-precision deep neural networks,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 10 763–10 771

    work page 2021

  7. [8]

    Improved techniques for quantizing deep networks with adaptive bit-widths,

    X. Sun, R. Panda, C.-F. R. Chen, N. Wang, B. Pan, A. Oliva, R. Feris, and K. Saenko, “Improved techniques for quantizing deep networks with adaptive bit-widths,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 957–967

    work page 2024

  8. [9]

    Bit-mixer: Mixed-precision networks with runtime bit-width selection,

    A. Bulat and G. Tzimiropoulos, “Bit-mixer: Mixed-precision networks with runtime bit-width selection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5188–5197

    work page 2021

Show all 40 references
  1. [10]

    Arbitrary bit-width network: A joint layer-wise quantization and adaptive inference approach,

    C. Tang, H. Zhai, K. Ouyang, Z. Wang, Y . Zhu, and W. Zhu, “Arbitrary bit-width network: A joint layer-wise quantization and adaptive inference approach,” 2022. [Online]. Available: https://arxiv.org/abs/2204.09992 11

    2022 arXiv

  2. [11]

    Hawq-v2: Hessian aware trace-weighted quantization of neural networks,

    Z. Dong, Z. Yao, D. Arfeen, A. Gholami, M. W. Mahoney, and K. Keutzer, “Hawq-v2: Hessian aware trace-weighted quantization of neural networks,” Advances in neural information processing systems , vol. 33, pp. 18 518–18 529, 2020

    2020

  3. [12]

    Stochastic quantization for learning accurate low-bit deep neural networks,

    Y . Dong, R. Ni, J. Li, Y . Chen, H. Su, and J. Zhu, “Stochastic quantization for learning accurate low-bit deep neural networks,” International Journal of Computer Vision, vol. 127, pp. 1629–1642, 2019

    2019

  4. [13]

    Towards efficient training for neural network quantization,

    Q. Jin, L. Yang, and Z. Liao, “Towards efficient training for neural network quantization,”arXiv preprint arXiv:1912.10207, 2019

    1912 arXiv

  5. [14]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385

    2015 arXiv

  6. [15]

    Training for multi-resolution inference using reusable quantization terms,

    S. Q. Zhang, B. McDanel, H. Kung, and X. Dong, “Training for multi-resolution inference using reusable quantization terms,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2021, pp. 845–860

    2021

  7. [16]

    Bitwave: Exploiting column-based bit-level sparsity for deep learning acceleration,

    M. Shi, V . Jain, A. Joseph, M. Meijer, and M. Verhelst, “Bitwave: Exploiting column-based bit-level sparsity for deep learning acceleration,” in 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024, pp. 732–746

    2024

  8. [17]

    Self-knowledge distillation with progressive refinement of targets,

    K. Kim, B. Ji, D. Yoon, and S. Hwang, “Self-knowledge distillation with progressive refinement of targets,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6567–6576

    2021

  9. [18]

    Haq: Hardware-aware automated quantization with mixed precision,

    K. Wang, Z. Liu, Y . Lin, J. Lin, and S. Han, “Haq: Hardware-aware automated quantization with mixed precision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8612–8620

    2019

  10. [19]

    Releq: an automatic reinforcement learning approach for deep quantization of neural networks,

    A. Elthakeb, P. Pilligundla, F. Mireshghallah, A. Yazdanbakhsh, S. Gao, and H. Esmaeilzadeh, “Releq: an automatic reinforcement learning approach for deep quantization of neural networks,” in NeurIPS ML for Systems workshop, 2018, 2019

    2018

  11. [20]

    Mixed precision quantization of convnets via differentiable neural architecture search,

    B. Wu, Y . Wang, P. Zhang, Y . Tian, P. Vajda, and K. Keutzer, “Mixed precision quantization of convnets via differentiable neural architecture search,” arXiv preprint arXiv:1812.00090, 2018

    2018 arXiv

  12. [21]

    Single path one-shot neural architecture search with uniform sampling,

    Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y . Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. Springer, 2020, pp. 544–560

    2020

  13. [22]

    Once quantization-aware training: High performance extremely low-bit architecture search,

    M. Shen, F. Liang, R. Gong, Y . Li, C. Li, C. Lin, F. Yu, J. Yan, and W. Ouyang, “Once quantization-aware training: High performance extremely low-bit architecture search,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 5340–5349

    2021

  14. [23]

    Sharpness-aware quantization for deep neural networks,

    J. Liu, J. Cai, and B. Zhuang, “Sharpness-aware quantization for deep neural networks,”arXiv preprint arXiv:2111.12273, 2021

    2021 arXiv

  15. [24]

    Hawq-v3: Dyadic neural network quantization,

    Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y . Wang, M. Mahoney et al., “Hawq-v3: Dyadic neural network quantization,” in International Conference on Machine Learning. PMLR, 2021, pp. 11 875–11 886

    2021

  16. [25]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference,

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. G. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,”CoRR, vol. abs/1712.05877, 2017. [Online]. Available: http://arxiv.org/abs/1712.05877

    2017 arXiv

  17. [26]

    Estimating or propagating gradients through stochastic neurons for conditional computation,

    Y . Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  18. [27]

    Large batch training of convolutional networks,

    Y . You, I. Gitman, and B. Ginsburg, “Large batch training of convolutional networks,” arXiv preprint arXiv:1708.03888, 2017

    2017 arXiv

  19. [28]

    Rethinking differentiable search for mixed-precision neural networks,

    Z. Cai and N. Vasconcelos, “Rethinking differentiable search for mixed-precision neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2349– 2358

    2020

  20. [29]

    Single path one-shot neural architecture search with uniform sampling,

    Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y . Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in European conference on computer vision. Springer, 2020, pp. 544–560

    2020

  21. [30]

    Hawq: Hessian aware quantization of neural networks with mixed-precision,

    Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer, “Hawq: Hessian aware quantization of neural networks with mixed-precision,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 293–302. 12

    2019

  22. [31]

    Ompq: Orthogonal mixed precision quantization,

    Y . Ma, T. Jin, X. Zheng, Y . Wang, H. Li, Y . Wu, G. Jiang, W. Zhang, and R. Ji, “Ompq: Orthogonal mixed precision quantization,” in Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 7, 2023, pp. 9029–9037

    2023

  23. [32]

    A quantization-friendly separable convolution for mobilenets,

    T. Sheng, C. Feng, S. Zhuo, X. Zhang, L. Shen, and M. Aleksic, “A quantization-friendly separable convolution for mobilenets,” in2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2). IEEE, 2018, pp. 14–18

    2018

  24. [33]

    One model for all quantization: A quantized network supporting hot-swap bit-width adjustment,

    Q. Sun, X. Li, Y . Ren, Z. Huang, X. Liu, L. Jiao, and F. Liu, “One model for all quantization: A quantized network supporting hot-swap bit-width adjustment,” arXiv preprint arXiv:2105.01353, 2021

    2021 arXiv

  25. [34]

    Gradient l1 regularization for quantization robustness,

    M. Alizadeh, A. Behboodi, M. van Baalen, C. Louizos, T. Blankevoort, and M. Welling, “Gradient l1 regularization for quantization robustness,” arXiv preprint arXiv:2002.07520, 2020

    2002 arXiv

  26. [35]

    Robust quantization: One model to rule them all,

    B. Chmiel, R. Banner, G. Shomron, Y . Nahshan, A. Bronstein, U. Weiseret al., “Robust quantization: One model to rule them all,” Advances in neural information processing systems, vol. 33, pp. 5308–5317, 2020

    2020

  27. [36]

    Vertical layering of quantized neural networks for heteroge- neous inference,

    H. Wu, R. He, H. Tan, X. Qi, and K. Huang, “Vertical layering of quantized neural networks for heteroge- neous inference,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 15 964–15 978, 2023

    2023

  28. [37]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969

    2017

  29. [38]

    Tinyllama: An open-source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024

    2024

  30. [39]

    Lsq+: Improving low-bit quantization through learnable offsets and better initialization,

    Y . Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak, “Lsq+: Improving low-bit quantization through learnable offsets and better initialization,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 696–697

    2020

  31. [40]

    Slimmable neural networks,

    J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” arXiv preprint arXiv:1812.08928, 2018. 13 A Appendix / supplemental material A.1 Overview In this supplementary material, we present more explanations and experimental results. • First, we provide a det...

    2018 arXiv

  32. [2020]

    Available: https://arxiv.org/abs/2004.05284

    [Online]. Available: https://arxiv.org/abs/2004.05284

    2004 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.