Pith. sign in

REVIEW 4 major objections 6 minor 58 references

MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MLoRQ gives each transformer layer one joint choice: how low-rank, at what bit width, within one memory budget.

desk verdict Joint rank-plus-bit-width search with broad empirical backing; caveats are reproducibility and an unvalidated layer-independence assumption, not fatal flaws. read the letter →

arxiv 2507.09616 v1 pith:ASWQVEES submitted 2025-07-13 cs.LG cs.CV

Ofir Gordon , Ariel Lapid , Elad Cohen , Yarden Yagil , Arnon Netzer , Hai Victor Habi This is my paper · ORCID
classification cs.LGcs.CV
keywords post-trainingquantizationlow-rankapproximationmixedprecisionvisiontransformerstransformercompressionHessian-awareoptimizationadaptiveroundingintegerlinearprogramming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MLoRQ is a post-training transformer compression method that treats low-rank decomposition and mixed-precision quantization as one per-layer decision instead of two competing tricks. Under a fixed weight-memory budget, each linear layer can be kept in direct quantized form or split into two quantized low-rank matrices, each with its own bit width. A two-stage search first filters every layer's options down to a Pareto frontier and then uses integer linear programming to assign ranks and bit widths globally. The paper reports that this joint selection improves accuracy by up to 15% over quantization-only methods at high compression rates, across image classification, object detection, instance segmentation, and language understanding tasks.

What carries the argument

The central object is the unified compressed-weight set $\mathcal{W}_\ell = \mathcal{W}^Q_\ell \cup \mathcal{W}^{LQ}_\ell$, which encodes every layer's legal compression configurations: direct quantization at a chosen bit width, or a low-rank factorization with separate bit widths for the two factors. MLoRQ's machinery is a three-part chain: a Hessian-aware SVD that minimizes an upper bound on the Hessian-weighted reconstruction error, a Pareto-frontier filter that removes compression options dominated in both memory and local error, and an inter-layer integer linear program that selects one Pareto candidate per layer to maximize summed signal-to-quantization-noise ratio under the memory constraint. The optional LoRAda stage adapts the adaptive-rounding objective to the decomposed matrices, choosing round-up or round-down values for quantized $A$ and $B$ while considering model output error.

What would settle it

On a small vision transformer, run MLoRQ, then re-solve the same inter-layer ILP with every layer's SQNR measured while all other layers are in their compressed states instead of floating point; a materially different rank and bit-width assignment would signal that the independence assumption fails. Alternatively, an exhaustive enumeration of all rank and bit-width combinations for that small model at the same memory budget that beats MLoRQ's accuracy would show the two-stage search is not finding the true joint optimum.

Watch

Extended reading notes

Core claim

The paper's central claim is that low-rank approximation and quantization should be selected from a single candidate space, not applied in sequence or with a predetermined split. For a fully connected layer with weight $W$, the compressed options are $\mathcal{W}_\ell = \mathcal{W}^Q_\ell \cup \mathcal{W}^{LQ}_\ell$, meaning the layer is either quantized directly at some bit width or factored as $AB$ with each factor quantized, possibly at different bit widths. The mixed rank-and-precision problem then becomes the assignment of one such option to every layer so that total weight memory stays within a budget and total compression damage is minimized. MLoRQ solves it with a Hessian-weighted SVD that produces low-rank factors, a per-layer Pareto search over error and memory, and a global integer-linear-programming pass that maximizes each layer's signal-to-quantization-noise ratio. An optional low-rank-aware adaptive rounding stage refines the rounding decisions for both factors jointly.

Load-bearing premise

The load-bearing premise is that each layer's compression damage can be measured independently and added together into a global objective; if compressing one layer changes how much another layer's errors matter, the per-layer Pareto filtering and the ILP can settle on a globally suboptimal assignment.

Editorial extensions

If this is right

  • At the most aggressive compression rates, low-bit quantization alone collapses accuracy, while allowing low-rank factors stabilizes the model and recovers a large share of the lost performance.
  • Applying low-rank first and then quantizing the result, or using any fixed split between the two techniques, is dominated by the joint search at the same memory footprint.
  • MLoRQ wraps rather than replaces existing activation-quantization methods: adding it to recently proposed activation error-correction schemes improves those schemes on every architecture tested.
  • The value of the low-rank option grows with layer size; in the reported solutions, larger models receive low-rank assignments in many layers while the smallest model uses them rarely or not at all.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same trend holds for very large linear layers, extending MLoRQ to large language models and vision-language models should widen the accuracy gap at high compression; the authors name this as future work.
  • Because the global objective is modular, swapping signal-to-quantization-noise ratio for a label-free Hessian trace or a task-specific loss would change the allocation without touching the framework, which may matter where SQNR is a weak proxy for task loss.
  • The per-layer rank and bit-width map produced by the integer linear program could also be read as a hardware-scheduling hint: layers chosen for low rank are the ones whose memory and compute profile changes most, and scheduling could exploit that beyond the raw memory budget.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. MLoRQ is a post-training compression framework that jointly searches per-layer low-rank factors and quantization bit-widths for transformer-based models. The method has three stages: (i) an intra-layer search that computes a Hessian-weighted local loss for every rank/bit-width combination and prunes candidates to a Pareto frontier; (ii) an inter-layer stage that casts the global allocation of ranks and bit-widths under a weight-memory constraint as an integer linear program using per-layer SQNR metrics; and (iii) an optional low-rank-aware adaptive rounding step (LoRAda). The paper evaluates MLoRQ on ImageNet classification (ViT/DeiT/Swin models), COCO object detection and instance segmentation, and GLUE with BERT, reporting large improvements at high compression rates and claiming state-of-the-art results. The framework is clearly specified and accompanied by ablations, but the paper's central optimality claim rests on unvalidated independence and diagonality assumptions, and the reported gains are uneven across configurations.

Significance. If the claims hold, MLoRQ would be the first practical joint low-rank and mixed-precision post-training quantization method for transformers, with a modular design that can sit on top of existing activation-quantization schemes. The paper makes useful contributions in the form of a Hessian-aware SVD, a Pareto-frontier candidate reduction, and a rounding procedure specialized for decomposed-and-quantized weights. The algorithmic workflow is described in enough detail to be reimplemented, and the ablation study gives some evidence for the contribution of each component. However, the significance is currently bounded by the absence of direct validation of the two key approximations, by the lack of run-to-run variability information, and by a few configurations where the method's standalone variants are worse than the baselines they are meant to improve.

major comments (4)
  1. [Section 4, Eq. (8)-(9), Eq. (19)] The central "optimal" claim rests on the layer-independence and diagonal-Hessian assumptions that are introduced but never validated. Eq. (8) writes the total loss change as a sum of per-layer quadratic terms, and Section 4.1 further assumes each layer's Hessian is diagonal to define the local loss Lambda_ell in (9). The inter-layer ILP in (19) then maximizes a sum of per-layer SQNR values, each measured with the rest of the model in floating point. Because Section 4.1 discards every candidate that is not Pareto-optimal with respect to the local loss (9), a candidate that is locally dominated but globally important due to cross-layer error compensation is permanently removed before the global stage. I recommend adding a direct validation: on a small model, compare the final ILP assignment against a greedy or exhaustive search over the full candidate set evaluated with the true compressed model, and report the gap. Without such a check, the word "optimal" in Problem Statement 1 overstates what has been demonstrated.
  2. [Section 6.1, Table 1] The text states that the MLoRQ+RepQ and MLoRQ+ERQ results "highlight MLoRQ's standalone effectiveness, independent of dedicated rounding refinement," but Table 1 does not support this. At 9.3% weight memory, MLoRQ+ERQ on ViT-S gives 23.69 top-1 accuracy versus 45.68 for ERQ alone, a drop of about 22 points; MLoRQ+RepQ (17.65) is only 2 points above RepQ-ViT (15.65). The large gains in Table 1 appear only after applying LoRAda. At the 12.5% budget, several final-method improvements over ERQ are smaller than 0.5 points (ViT-S: 69.12 vs 68.91; Swin-S: 80.76 vs 80.74). The paper should report run-to-run variance or repeated evaluations, and should either soften the standalone-effectiveness claim or explain the conditions under which the low-rank search helps before rounding refinement.
  3. [Table 5 and Appendix C] The ablation in Table 5 is not reproducible from the information given. The caption refers to "default settings (Appendix C)," but Appendix C does not state the target weight-memory constraint used for the ablation, nor the activation bit-width, and Table 5 has no column indicating the compression budget for each model. Since this ablation is central to attributing gains to the Hessian-aware loss, the naive rounding replacement, and the SQNR inter-layer objective, the missing experimental setup makes it impossible to interpret the rows. Please add the exact memory budget and all relevant hyperparameters for each ablation row.
  4. [Section 6, Tables 1 and 3] The numerical comparisons are indirect: Table 1 states that all referenced results were taken from [57], and Table 3 states the same for [58]. The paper does not report whether the same model checkpoints, evaluation pipelines, and calibration data were used for the copied baselines, and no error bars or multiple runs are given. Given that several headline margins are below 0.5 percentage points, the claim of state-of-the-art performance would be more convincing with either independently run baselines or a clear statement of the expected variability of the evaluation.
minor comments (6)
  1. [Eq. (2)] The dimensions in Eq. (2) are inconsistent with Eq. (3) and Eq. (11): if W is dy by dx, then A should be dy by r and B should be r by dx, but Eq. (2) states A in R^{dx by r} and B in R^{r by dy}. Please correct this.
  2. [Eq. (5) and Problem Statement 1] The symbol W_ell is used both for the original weight matrix and for the set of compressed options in Eq. (5) and Problem Statement 1; a different symbol for the candidate set would prevent confusion.
  3. [Table 4] The table title says "GELU benchmark" and should read "GLUE benchmark."
  4. [Abstract and Section 6.1] The abstract's "up to 15% performance improvement" is not tied to a specific table or configuration; since Table 1 contains both large gains and near-tie outcomes, please specify the setting in which this number is obtained.
  5. [Appendix E] The interpolation validation in Figure 8 covers a single layer; please report a summary of the interpolation error over all layers or an ablation showing that the ILP solution is insensitive to the interpolation budget k_inf.
  6. [Introduction and Section 6] The statement that MLoRQ "can be seamlessly integrated with most existing quantization algorithms" is supported in the paper only for RepQ and ERQ; please limit the claim to the demonstrated cases or add an additional integration example.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: MLoRQ's optimization is an empirical compression search; the self-cited Hessian tools are ingredients, not predictions of the reported accuracies.

full rationale

The paper's derivation chain is a two-stage search over compression candidates, and the reported results are external benchmark accuracies. The intra-layer objective (Eq. 9) is a Hessian-weighted reconstruction error; the inter-layer objective (Eq. 16/19) maximizes a sum of per-layer SQNR values under a memory constraint. Neither objective is fitted to the final accuracy values, and no 'prediction' in the paper is defined in terms of the quantity it claims to predict. The label-free Hessian and Hessian-MSE are taken from the authors' own EPTQ work [14], and the activation mixed-precision formula comes from the authors' HMQ [16], but these are reusable algorithmic components rather than the paper's central claim; the central claim is that the joint rank/bit-width search improves compression accuracy, which is evaluated against independent baselines on ImageNet, COCO, and GLUE. The layer-independence and Hessian-diagonality assumptions (Eq. 8 and Section 4.1) are correctness risks that could make the search suboptimal, but they are stated assumptions, not circular reductions: Eq. 8 is not used to define the reported accuracy, and the ILP solution is not forced to equal any fitted input. The ablation study (Table 5) tests components independently against a vanilla version and against an external Hessian-based MP objective [9], further indicating that the contributions are evaluated rather than assumed. Therefore, no step in the claimed derivation reduces by construction to its own inputs, and the paper is not circular.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim does not rest on novel physical entities. It does rest on several approximation assumptions (diagonal Hessian, layer independence, Hessian estimator quality) and on hand-chosen hyperparameters. These are standard PTQ assumptions, so the ledger is moderate rather than heavy.

free parameters (6)
  • Quantization scale percentile grid = 0.97, 0.98, 0.99, 0.995, 0.9995, 0.9997, 0.9999, 0.99995, 0.99999, 1.0
    Per-channel quantization ranges are selected by scanning this grid, which is per-layer tuning on calibration data.
  • LoRAda regularization factor lambda = 0.3
    Chosen for all experiments; no sensitivity analysis reported.
  • LoRAda learning rate = 0.3 (0.01 for ViT-S)
    Set per model; no principled selection.
  • Number of calibration images = 1024 (optimization), 32 (Hessian/search)
    Chosen; ablation shows 32 is nearly sufficient, but 1024 is the default.
  • Stretch parameters xi, gamma = 1.1, -0.1
    Taken from AdaRound [32].
  • Bit-width option set B = {2,3,4,6,8}
    Candidate precisions chosen by hand; the central ILP only selects from this set.
assumptions (6)
  • domain assumption Layer-wise independence of compression error: the total change in task loss is the sum of per-layer changes (Eq. 8).
    Used to decompose the global problem into intra-layer and inter-layer stages; no empirical check of cross-layer interaction.
  • domain assumption The weight Hessian H_w_l is diagonal within each layer.
    Section 4.1: 'under the added assumption that Hessian Hwℓ is a diagonal matrix'; enables the element-wise weighting D_l.
  • domain assumption The gradient of the task loss at the pre-trained weights is approximately zero.
    Standard in PTQ; used to drop the first-order term in Eq. (7).
  • domain assumption The Label-Free Hessian approximation of EPTQ [14] accurately estimates the true Hessian.
    The entire Hessian-weighted SVD and quantization parameter selection depend on this approximation from the authors' own prior work.
  • domain assumption SQNR maximization is a valid global objective for mixed-rank-and-precision allocation.
    Section 4.2; supported by an ablation versus Hessian-aware objective, but not a theorem.
  • domain assumption Linear interpolation of per-layer error between neighboring ranks is accurate.
    Appendix E; justified by a first-order Taylor argument and shown for one example layer only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression." pith.science (2026). https://pith.science/paper/ASWQVEES

@misc{pith2026250709616,
  author       = {Pith},
  title        = {Pith review of: MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASWQVEES}},
  note         = {Machine review of arXiv:2507.09616}
}
read the original abstract

Deploying transformer-based neural networks on resource-constrained edge devices presents a significant challenge. This challenge is often addressed through various techniques, such as low-rank approximation and mixed-precision quantization. In this work, we introduce Mixed Low-Rank and Quantization (MLoRQ), a novel method that integrates both techniques. MLoRQ employs a two-stage optimization process to determine optimal bit-width and rank assignments for each layer, adhering to predefined memory constraints. This process includes: (i) an intra-layer optimization that identifies potentially optimal compression solutions out of all low-rank and quantization combinations; (ii) an inter-layer optimization that assigns bit-width precision and rank to each layer while ensuring the memory constraint is met. An optional final step applies a sequential optimization process using a modified adaptive rounding technique to mitigate compression-induced errors in joint low-rank approximation and quantization. The method is compatible and can be seamlessly integrated with most existing quantization algorithms. MLoRQ shows state-of-the-art results with up to 15\% performance improvement, evaluated on Vision Transformers for image classification, object detection, and instance segmentation tasks.

Figures

Figures reproduced from arXiv: 2507.09616 by the authors.

Figure 1
Figure 1. The MLoRQ framework for Mixed Low-Rank and Quantization. (i) Intra-layer search (4.1) for the Pareto frontier optimal candidates set over the local loss and constraint. (ii) Inter-layer search (4.2) to globally assign rank and bit-width to each layer. (iii) Apply complementary quantization algorithms with/without low-rank-aware adaptive rounding. approach [14]. Then, an inter-layer optimization assigns bit-width and… view at source ↗
Figure 2
Figure 2. An analysis of different rank and compression rates: (a) Presenting the MSE of a single fully-connected layer under different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Analysis of joint low-rank and quantization: Comparison between joint optimization and mixed precision only on DeiT-B and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The compression solution for the scenario of 2-bit average weights compression of DeiT-B with MLoRQ, where each layer is [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Number of samples ablation. Note that the number of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Analysis of layer error over different compression rates. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: A comparison of accuracy results when running compression with and without the low-rank option. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the actual per-rank error value and the [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 10
Figure 10. Figure 10: Compression solution for ViT-S and Swin-B with 3-bit average weights and 4-bit activations. Each layer is assigned with either [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Inra-layer Pareto optimization results illustration for the ViT-S model. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 54 canonical work pages

  1. [57]

    ERQ: Error reduction for post-training quanti- zation of vision transformers

    Yunshan Zhong, Jiawei Hu, You Huang, Yuxin Zhang, and Rongrong Ji. ERQ: Error reduction for post-training quanti- zation of vision transformers. In ICML, pages 61664–61680,

    work page

  2. [58]

    Towards accurate post-training quantization of vision transformers via error reduction

    Yunshan Zhong, You Huang, Jiawei Hu, Yuxin Zhang, and Rongrong Ji. Towards accurate post-training quantization of vision transformers via error reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 7, 8 12 Table of Contents: • Related Works can be found in Appendix A. • Extended details regarding the motivation for joint low- ra...

    work page 2025

  3. [1]

    Quantizable transformers: Removing outliers by helping attention heads do nothing

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. NeurIPS, 36,

    work page

  4. [2]

    Cascade r-cnn: Delving into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, pages 6154– 6162, 2018. 7, 9

    work page 2018

  5. [3]

    End- to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End- to-end object detection with transformers. In ECCV, pages 213–229. Springer, 2020. 1

    work page 2020

  6. [4]

    Exploiting linear structure within con- Figure 5

    Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann Le- Cun, and Rob Fergus. Exploiting linear structure within con- Figure 5. Number of samples ablation. Note that the number of samples axis is uniformly spaced for exposition purposes. volutional networks for efficient evaluation. NeurIPS, 27,

    work page

  7. [5]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 4171–4186. Association for Computa- tional Linguisti...

    work page 2019

  8. [6]

    Towards accurate post- training quantization for vision transformer

    Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu. Towards accurate post- training quantization for vision transformer. In ACM MM, pages 5380–5388, 2022. 7, 13

    work page 2022

Show all 58 references
  1. [7]

    Learning to prune deep neural networks via layer-wise optimal brain surgeon

    Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. NeurIPS, 30, 2017. 1

    2017

  2. [8]

    HAWQ: Hessian aware quanti- zation of neural networks with mixed-precision

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Ma- honey, and Kurt Keutzer. HAWQ: Hessian aware quanti- zation of neural networks with mixed-precision. In ICCV, pages 293–302, 2019. 5, 6

    2019

  3. [9]

    Hawq-v2: Hessian aware trace-weighted quantization of neural networks

    Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. In NeurIPS, pages 18518–18529, 2020. 1, 5, 6, 10, 13, 14

    2020

  4. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  5. [11]

    Optq: Accurate quantization for generative pre- trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre- trained transformers. In ICLR, 2022. 1, 7, 8, 13

    2022

  6. [12]

    A survey of quan- tization methods for efficient neural network inference

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quan- tization methods for efficient neural network inference. In 10 Low-power computer vision, pages 291–326. Chapman and Hall/CRC, 2022. 1, 3

    2022

  7. [13]

    Singular value decom- position and least squares solutions

    Gene H Golub and Christian Reinsch. Singular value decom- position and least squares solutions. In Handbook for auto- matic computation: volume II: linear algebra , pages 134–

  8. [14]

    Eptq: Enhanced post-training quantization via hessian-guided network-wise optimization

    Ofir Gordon, Elad Cohen, Hai Victor Habi, and Arnon Netzer. Eptq: Enhanced post-training quantization via hessian-guided network-wise optimization. arXiv preprint arXiv:2309.11531, 2024. 1, 2, 3, 5, 6, 13, 14

    2024 arXiv

  9. [15]

    Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization

    Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Compute...

    2023

  10. [16]

    Hmq: Hardware friendly mixed precision quantization block for cnns

    Hai Victor Habi, Roy H Jennings, and Arnon Netzer. Hmq: Hardware friendly mixed precision quantization block for cnns. In ECCV, pages 448–463, 2020. 1, 9, 14

    2020

  11. [17]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 7, 9

    2017

  12. [18]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In ICLR, 2022. 1, 3, 13

    2022

  13. [19]

    Numerical optimizations for weighted low-rank estimation on language models

    Ting Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou, Yilin Shen, and Hongxia Jin. Numerical optimizations for weighted low-rank estimation on language models. In Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1404–1416, 2022. 9, 13

    2022

  14. [20]

    Dynamic low-rank estimation for transformer-based language models

    Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Dynamic low-rank estimation for transformer-based language models. In The 2023 Confer- ence on Empirical Methods in Natural Language Processing,

    2023

  15. [21]

    Complexity-aware layer-wise mixed-precision schemes with sqnr-based fast analysis

    Hana Kim, Hyun Eun, Jung Hwan Choi, and Ji-Hoon Kim. Complexity-aware layer-wise mixed-precision schemes with sqnr-based fast analysis. IEEE Access, 11:117800–117809,

  16. [23]

    One-shot model for mixed-precision quantization

    Ivan Koryakovskiy, Alexandra Yakovleva, Valentin Buch- nev, Temur Isaev, and Gleb Odinokikh. One-shot model for mixed-precision quantization. In CVPR, pages 7939–7949,

  17. [24]

    Quantizing deep convolu- tional networks for efficient inference: A whitepaper

    Raghuraman Krishnamoorthi. Quantizing deep convolu- tional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. 3

    2018 arXiv

  18. [25]

    BRECQ: Push- ing the limit of post-training quantization by block recon- struction

    Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Push- ing the limit of post-training quantization by block recon- struction. In ICLR, 2021. 5, 7, 8, 13

    2021

  19. [26]

    Repq- vit: Scale reparameterization for post-training quantization of vision transformers

    Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq- vit: Scale reparameterization for post-training quantization of vision transformers. In ICCV, pages 17227–17236, 2023. 1, 2, 5, 7, 8, 13, 14, 18

    2023

  20. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 2

    2014

  21. [28]

    Pd-quant: Post-training quantiza- tion based on prediction difference metric

    Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang, Xinggang Wang, and Wenyu Liu. Pd-quant: Post-training quantiza- tion based on prediction difference metric. In CVPR, pages 24427–24437, 2023. 7

    2023

  22. [29]

    Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers

    Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In CVPR, pages 20321–20330, 2023. 13

    2023

  23. [30]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021. 1

    2021

  24. [31]

    Post-training quantization for vision trans- former

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision trans- former. NeurIPS, 34:28092–28103, 2021. 1, 8, 13

    2021

  25. [32]

    Up or down? adap- tive rounding for post-training quantization

    Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Chris- tos Louizos, and Tijmen Blankevoort. Up or down? adap- tive rounding for post-training quantization. In ICML, pages 7197–7206. PMLR, 2020. 1, 2, 3, 5, 7, 13

    2020

  26. [33]

    Loss aware post-training quantization

    Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. Machine Learning, 110(11):3245–3262, 2021. 3

    2021

  27. [34]

    Compressing pre- trained language models by matrix decomposition

    Matan Ben Noach and Yoav Goldberg. Compressing pre- trained language models by matrix decomposition. InAACL- IJCNLP, pages 884–889, 2020. 2

    2020

  28. [35]

    A practi- cal mixed precision algorithm for post-training quantization

    Nilesh Prasad Pandey, Markus Nagel, Mart van Baalen, Yin Huang, Chirag Patel, and Tijmen Blankevoort. A practi- cal mixed precision algorithm for post-training quantization. arXiv preprint arXiv:2302.05397, 2023. 1, 6

    2023 arXiv

  29. [36]

    OR-Tools, 2024

    Laurent Perron and Vincent Furnon. OR-Tools, 2024. Version v9.11. by Google, Available at: https : / / developers.google.com/optimization/. 6, 14

    2024

  30. [37]

    Probabilistic weather forecasting with machine learn- ing

    Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learn- ing. Nature, 637(8044):84–90, 2025. 1

    2025

  31. [38]

    Lrp-qvit: Mixed- precision vision transformer quantization via layer-wise rel- evance propagation

    Navin Ranjan and Andreas Savakis. Lrp-qvit: Mixed- precision vision transformer quantization via layer-wise rel- evance propagation. arXiv preprint arXiv:2401.11243, 2024. 1, 8, 13

    2024 arXiv

  32. [39]

    Siyu Ren and Kenny Q. Zhu. Low-rank prune-and-factorize for language model compression. In LREC-COLING, pages 10822–10832, 2024. 1, 9, 13

    2024

  33. [40]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 2, 7

    2015

  34. [41]

    Weighted low-rank ap- proximations

    Nathan Srebro and Tommi Jaakkola. Weighted low-rank ap- proximations. In ICML, pages 720–727, 2003. 2, 3, 6 11

    2003

  35. [42]

    Amp-vit: Optimizing vision transformer efficiency with adaptive mixed-precision post- training quantization

    Yu-Shan Tai and An-Yeu Wu. Amp-vit: Optimizing vision transformer efficiency with adaptive mixed-precision post- training quantization. In Proceedings of the Winter Con- ference on Applications of Computer Vision (WACV), pages 6828–6837, 2025. 8

    2025

  36. [43]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In ICML, pages 10347–10357. PMLR, 2021. 1

    2021

  37. [44]

    Mixed precision dnns: All you need is a good parametrization

    Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision dnns: All you need is a good parametrization. In ICLR, 2020. 1, 9

    2020

  38. [45]

    Attention is all you need

    A Vaswani. Attention is all you need. NeurIPS, 2017. 1

    2017

  39. [46]

    GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for ...

    2018

  40. [47]

    SVD- LLM: Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD- LLM: Truncation-aware singular value decomposition for large language model compression. In ICLR, 2025. 1, 3, 13

    2025

  41. [48]

    QDrop: Randomly dropping quantization for extremely low-bit post-training quantization

    Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. QDrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2021. 7, 13

    2021

  42. [49]

    Outlier sup- pression+: Accurate quantization of large language models by equivalent and effective shifting and scaling

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier sup- pression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. InThe 2023 Conference on Empirical Methods in Natural La...

    2023

  43. [50]

    Pytorch image models

    Ross Wightman. Pytorch image models. https : / / github . com / rwightman / pytorch - image - models, 2019. 7

    2019

  44. [51]

    Smoothquant: Accurate and ef- ficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and ef- ficient post-training quantization for large language models. In ICML, pages 38087–38099. PMLR, 2023. 13

    2023

  45. [52]

    Patch- wise mixed-precision quantization of vision transformer

    Junrui Xiao, Zhikai Li, Lianwei Yang, and Qingyi Gu. Patch- wise mixed-precision quantization of vision transformer. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2023. 1, 8, 13

    2023

  46. [53]

    Efficient low-rank back- propagation for vision transformer adaptation

    Yuedong Yang, Hung-Yueh Chiang, Guihong Li, Diana Mar- culescu, and Radu Marculescu. Efficient low-rank back- propagation for vision transformer adaptation. NeurIPS, 36: 14725–14736, 2023. 13

    2023

  47. [54]

    Compressing transformers: fea- tures are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11007– 11015, 2023

    Hao Yu and Jianxin Wu. Compressing transformers: fea- tures are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11007– 11015, 2023. 13

    2023

  48. [55]

    Hessian- aware pruning and optimal neural implant

    Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Se- hoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian- aware pruning and optimal neural implant. InProceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 3880–3891, 2022. 1, 6

    2022

  49. [56]

    Ptq4vit: Post-training quantization for vi- sion transformers with twin uniform quantization

    Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vi- sion transformers with twin uniform quantization. In ECCV, pages 191–207, 2022. 7, 13

    2022

  50. [2024]

    1, 2, 5, 7, 8, 13, 14, 18

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.