Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage 'activation-first, weights-later' scheme puts 4-bit Vision Transformers within about 1-2 points of FP32 accuracy on ImageNet, preserves downstream generalization, and cuts quantization-aware training from days to 1-2 hours.

desk verdict A clever and practical activation-first QAT recipe with a promising basin-preservation probe, but the generalization claim is overstated and the PCA rule is tuned on the test test set—worth refereeing with revisions. read the letter →

arxiv 2506.11784 v1 pith:EBWWTWDE submitted 2025-06-13 cs.CV

Guang Liang , Xinyao Liu , Jianxin Wu This is my paper
classification cs.CV
keywords quantization-awaretrainingvisiontransformerspost-trainingquantizationactivation-firstfeaturemimickingoptimizationbasin4-bitmodelgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GPLQ claims that low-bit (4-bit) quantization of Vision Transformers can be made both accurate and practical by reordering the process: quantize activations first in a single epoch of training while leaving weights in floating point, then quantize the weights with post-training quantization. On the paper's experiments this two-stage scheme puts 4-bit Swin and DeiT models within about 1-2 points of FP32 on ImageNet, keeps or improves average accuracy on five fine-grained downstream tasks, and reduces training time from days to roughly 1-2 hours (about 35 minutes for Swin-T on 8 GPUs) with memory below FP32 training. The paper's two empirical insights are that activation quantization is the harder problem in low-bit ViTs and that staying in the original FP32 optimization basin is what preserves generalization. If these claims hold, GPLQ offers a practical middle path between PTQ's speed-with-accuracy-loss and conventional QAT's accuracy-with-prohibitive-cost.

What carries the argument

The central mechanism is the sequential 'activation-first, weights-later' pipeline, with the PCA feature-mimicking loss as the component that preserves generalization. In Stage 1, per-channel 4-bit activation quantizers with LSQ-learned scales are initialized by percentile-based calibration and trained for one epoch at a low learning rate while weights stay FP32; the loss $L_{PCA}$ (Eq. 2) minimizes the MSE between teacher and student penultimate-layer features projected onto a PCA subspace that captures about 60% of variance. In Stage 2, because activations are already fixed at 4 bits, weight quantization becomes a W32-to-W4 task without coupled activation error: RepQ-ViT quantizes the weights and QwT adds a lightweight linear compensation layer whose parameters are computed in closed form (Eq. 3) from the layer's A4-input outputs before and after weight quantization.

What would settle it

Pre-register the PCA dimension as a fixed multiple of 32 (for example, 256 for a 768-dimensional feature space) without looking at any downstream results, run the full GPLQ pipeline, and evaluate on a held-out set of downstream tasks that were not used when reporting Table 6. If the method no longer matches or beats FP32 on those tasks while ImageNet accuracy stays near 80.4%, then the claim that basin preservation via feature mimicking is what carries generalization would be contradicted.

Watch

Extended reading notes

Core claim

The central discovery is that a full W4A4 ViT does not need joint quantization-aware training of weights and activations. GPLQ's Stage 1 keeps all weights in FP32 and learns only 4-bit activation quantizers for one epoch, with a PCA-based feature-mimicking loss that projects the student's and teacher's penultimate-layer features onto the teacher's principal components (about 60% of cumulative variance) and minimizes their MSE, so the model stays in the original FP32 basin. Stage 2 then quantizes the frozen W32A4 model's weights with percentile-based per-channel PTQ (RepQ-ViT) and compensates the residual error with the closed-form QwT linear layer. The paper presents the basin-retention evidence directly: after GPLQ, the extracted FP32 weights still score 81.1% on ImageNet, nearly the original 81.2%, while a standard QAT method's extracted weights collapse to 68.9%, and GPLQ's average downstream FGVC accuracy (71.18% on Swin-T) exceeds the FP32 model's (70.79%).

Load-bearing premise

The whole method rests on the assumption that a quantized model which still produces nearly the same internal representations as the original model, as judged by a low-dimensional summary holding roughly 60% of the variance, will generalize to new tasks as well as the original; if that summary hides the features that matter for transfer, the promised downstream accuracy disappears.

Editorial extensions

If this is right

  • 4-bit versions of Swin-T/S/B/L and DeiT-S/B reach ImageNet accuracy within roughly 0.8-1.4 points of FP32 (for example, Swin-L at 85.5% versus 86.3% full precision), and their average accuracy on the five FGVC downstream tasks is at or above the FP32 model in most settings.
  • QAT for ViTs becomes a one-epoch, low-memory procedure: Swin-T's Act-QAT takes about 35 minutes on 8 GPUs and remains stable down to a single GPU, and models as large as Swin-B and Swin-L can be quantized without out-of-memory failures that stop other QAT methods.
  • Because activations are fixed before weights are quantized, the residual error in weight PTQ is no longer coupled with activation error, which makes the lightweight QwT compensation unusually effective and keeps the W4A4 model close to the W32A4 stage.
  • The two-stage recipe transfers to object detection: on COCO with Mask R-CNN, GPLQ's W4A4 Swin-T (3x) reaches 0.401 APbox versus 0.361 for RepQ-ViT, even though the detection experiments omit the PCA feature-mimicking loss.
  • The basin-retention experiment (Table 5) suggests that conventional QAT's downstream generalization loss is caused by leaving the FP32 basin, not by quantization itself, so preserving feature structure through feature mimicking is sufficient to retain transferability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If activation-first ordering is the true driver, the same recipe may extend to CNNs and LLMs where activations dominate quantization error; the paper lists these as future directions rather than claims.
  • The PCA dimension is chosen partly by peeking at downstream results (Table 6 is best at 256), so a stronger practical test would pre-specify the dimension as a fixed multiple of 32 without downstream tuning; the method's generality claim would be more believable if that fixed choice still works.
  • Table 9 indicates the one-epoch Act-QAT is data-efficient: about 10% of ImageNet (128k images) yields 79.2% W4A4 accuracy versus 79.8% with the full dataset, which suggests the activation quantizers could be trained on small calibration sets and further cut deployment cost.
  • The paper's internal-FP32 diagnostic (Table 5) could serve as a cheap, standard tool for predicting whether any QAT method will transfer to downstream tasks before running the downstream evaluations themselves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes GPLQ, a two-stage quantization-aware training (QAT) method for Vision Transformers. Stage 1 (Act-QAT) quantizes only activations to 4 bits while keeping weights in FP32, using a PCA-based feature mimicking loss to preserve the FP32 model's optimization basin; stage 2 (Weight-PTQ) quantizes weights with a percentile-based PTQ method followed by QwT compensation. The method is evaluated on ImageNet, five FGVC datasets, and COCO object detection, reporting 4-bit accuracy within 1-2 points of FP32, downstream average accuracy that sometimes exceeds FP32, and training time reduced to 1-2 hours. The core claims are the importance of activation quantization and the need to stay in the original basin for generalization.

Significance. If validated, GPLQ would address a real practical gap: low-bit QAT for ViTs with drastically reduced cost and memory, while retaining or improving transfer. The paper provides useful ablations (Table 3, 4, 10), scales to Swin-L, and includes a resource-constrained GPU study (Table 8). The two-stage activation-first idea is intuitively appealing and builds on the authors' prior TCS and QwT work. However, the evidence base is thin: single runs, no error bars, only two QAT baselines, and a post-hoc selected PCA dimension. The object detection results contradict the generalization claim as stated. The central idea is defensible but not rigorously established.

major comments (4)
  1. [Section 4.3, Table 6] The PCA dimension is chosen post hoc from the same downstream results used to claim improvement. Table 6 shows that Avg Task accuracy peaks at 71.22 with PCA 256 (61.3% variance), while 64 dims (24.1%) gives 70.42 and 512 dims (87.7%) gives 70.80; Table 1 reports 71.18, which is a selected maximum. The stated rule (approximately 60% variance) is not independently validated, and the table itself cannot both choose the dimension and demonstrate the benefit. Please validate on a held-out task or with a pre-registered variance threshold.
  2. [Table 2, Section 4.2] The object detection results undermine the generalization claim. For Swin-T (3x), GPLQ W4A4 AP_box is 0.401 versus 0.460 for FP32, a 12.8% relative drop, in contrast to the FGVC improvements in Table 1. The paper notes that detection did not use PCA feature mimicking; this means the mechanism claimed to preserve generalization is absent in the task where generalization fails. The abstract's 'including object detection' is therefore overstated. Either apply PCA mimicking to detection and show it recovers the gap, or explicitly qualify the generalization claim.
  3. [Tables 1, 3, 4] All experiments are single runs without error bars or multiple seeds. The central claim that GPLQ generalizes at or above FP32 relies on differences of a few tenths of a point (e.g., Swin-T Avg Task 71.18 vs 70.79 in Table 1); these are within typical run-to-run variation and are not statistically supported. Please report mean and standard deviation over at least three seeds for the headline results.
  4. [Table 5, Section 4.3] The basin-retention hypothesis is supported by only one baseline (Q-Var) and the loss curves in Figure 4a come from a single run. The claim that internal FP32 accuracy reflects basin retention, and that basin retention explains downstream generalization, is not established by this evidence. Please provide a quantitative basin measure or multiple examples across methods.
minor comments (3)
  1. [Table 5] The header misspells 'Method' as 'Medthod'; please correct.
  2. [Section 3.3] The statement that GPLQ 'avoids jumping out of the FP32 model's local minima' is an interpretation rather than a measured quantity; consider softening the wording to reflect that this is an inference from the loss curves and internal FP32 accuracy.
  3. [Section 4.2] The paper says detection did not use PCA feature mimicking, which is confusing in light of the generality claim; clarify whether the intended deployment uses PCA mimicking or not for downstream tasks.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: GPLQ is an empirical two-stage composition whose components are ablated in-paper; same-group citations are not load-bearing.

full rationale

The claimed derivation chain is empirical rather than definitional. The activation-first strategy is supported by the independent weight-only vs activation-only PTQ experiment (Figure 2). The PCA feature-mimicking loss in Eq. (2) is a standard feature-distillation objective, and its contribution is tested by the Table 6 ablation (w/o PCA vs w/ PCA), not merely assumed from TCS [37]. The weight-PTQ stage using RepQ-ViT and QwT is similarly ablated in Table 4. The basin-retention argument (Table 5) compares internal FP32 accuracy of GPLQ and a QAT method; even though GPLQ's one-epoch, low-LR activation-only update makes the internal-FP32 closeness partly by design, the downstream results are measured independently with linear probing. The main caveats are methodological rather than circular: the PCA dimension in Table 6 is a hyperparameter sweep without seed/error-bar information, the linear-probing evaluation overlaps with the penultimate-feature matching objective, and the object-detection experiment (Table 2) explicitly does not use PCA feature mimicking, so the abstract's 'including object detection' generalization claim is broader than the evidence. Same-group citations (TCS, QwT, DTL) exist, but each is supported by in-paper experiments or is used as a component whose contribution is isolated, so the citations are not load-bearing for the central empirical claims. No prediction reduces to its input by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method depends on several empirical and representational assumptions: activations dominate 4-bit error, basin preservation transfers, PCA subspace matching suffices, and linear probing measures generalization. These are reasonable but not rigorously established. Free parameters are mostly standard QAT hyperparameters, but the PCA dimension is tuned on the evaluation tasks and the QwT lambda is omitted.

free parameters (6)
  • PCA subspace dimension = 256 for Swin-T, 64 for DeiT-T (based on ~60% cumulative variance, rounded to multiple of 32)
    Chosen after evaluating downstream accuracy across 64/256/512 dimensions (Table 6); 256 gives the best average task accuracy.
  • Act-QAT learning rate = 5e-6
    Fixed for all experiments; no sensitivity analysis reported.
  • Number of Act-QAT epochs = 1
    Core to the 'lightning' claim; no ablation on epoch count is shown.
  • Weight PTQ calibration set size = 512 random ImageNet images
    Set for Stage 2; no ablation on calibration size.
  • PCA variance threshold = ~60%
    Heuristic used to choose the subspace dimension; the paper notes 60% gives a good trade-off but does not justify it independently of downstream peeking.
  • QwT regularization lambda = not reported
    The closed-form compensation in Eq. (3) includes lambda I for stability, but the value is not given, hindering exact replication.
assumptions (6)
  • domain assumption Quantizing activations to 4 bits is the dominant source of accuracy loss, more than weights (Figure 2).
    Motivates activation-first design; measured on 4 models without error bars or a theoretical justification.
  • domain assumption A model that stays in the same optimization basin as the FP32 teacher preserves downstream generalization.
    Supported only by the Q-Variation comparison in Table 5; there is no formal definition or direct measurement of 'basin' beyond internal FP32 accuracy.
  • domain assumption PCA projection with ~60% cumulative variance of penultimate-layer features is a sufficient distillation target.
    Used in Eq. (2); the dimension is selected from downstream accuracy (Table 6), so the choice is not independent of the evaluation.
  • domain assumption Linear probing on 5 FGVC datasets is an adequate proxy for transfer/generalization.
    Evaluation protocol; does not cover full fine-tuning transfer or other task families beyond linear separability.
  • domain assumption The pretrained FP32 model is an appropriate teacher for feature distillation.
    Teacher features form the PCA target; if the teacher is weak or the architecture shifts, the target is mis-specified.
  • standard math Straight-Through Estimator approximates gradients through the round() function in Eq. (1).
    Used to train the scale factor sa; standard in QAT but not formally justified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers." pith.science (2026). https://pith.science/paper/EBWWTWDE

@misc{pith2026250611784,
  author       = {Pith},
  title        = {Pith review of: GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBWWTWDE}},
  note         = {Machine review of arXiv:2506.11784}
}
read the original abstract

Vision Transformers (ViTs) are essential in computer vision but are computationally intensive, too. Model quantization, particularly to low bit-widths like 4-bit, aims to alleviate this difficulty, yet existing Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) methods exhibit significant limitations. PTQ often incurs substantial accuracy drop, while QAT achieves high accuracy but suffers from prohibitive computational costs, limited generalization to downstream tasks, training instability, and lacking of open-source codebase. To address these challenges, this paper introduces General, Practical, and Lightning Quantization (GPLQ), a novel framework designed for efficient and effective ViT quantization. GPLQ is founded on two key empirical insights: the paramount importance of activation quantization and the necessity of preserving the model's original optimization ``basin'' to maintain generalization. Consequently, GPLQ employs a sequential ``activation-first, weights-later'' strategy. Stage 1 keeps weights in FP32 while quantizing activations with a feature mimicking loss in only 1 epoch to keep it stay in the same ``basin'', thereby preserving generalization. Stage 2 quantizes weights using a PTQ method. As a result, GPLQ is 100x faster than existing QAT methods, lowers memory footprint to levels even below FP32 training, and achieves 4-bit model performance that is highly competitive with FP32 models in terms of both accuracy on ImageNet and generalization to diverse downstream tasks, including fine-grained visual classification and object detection. We will release an easy-to-use open-source toolkit supporting multiple vision tasks.

Figures

Figures reproduced from arXiv: 2506.11784 by the authors.

Figure 1
Figure 1. Core advantages of our GPLQ: Generality, Practicality, and Lightning efficiency. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Impact of quantizing weights and activations separately. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of GPLQ: QAT stage first only for activations, then PTQ stage only for weights. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training loss curves and Percentage of explained variance in GPLQ. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. YOLOv8-SMOT: An Efficient and Robust Framework for Real-Time Small Object Tracking via Slice-Assisted Training and Adaptive Association

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A YOLOv8 detector trained on overlapping slices plus an OC-SORT tracker with EMA motion direction and expanded IoU distance penalty achieves 55.205 SO-HOTA on the SMOT4SB public test set.

Reference graph

Works this paper leans on

38 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  2. [2]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InEuropean Conference on Computer Vision, 2014

    2014

  3. [3]

    Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models.arXiv preprint arXiv:2407.11062, 2024

    arXiv 2024

  4. [4]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  5. [5]

    Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile.Advances in Neural Information Processing Systems, 36:9015–9028, 2023

    Peiyan Dong, Lei Lu, Chao Wu, Cheng Lyu, Geng Yuan, Hao Tang, and Yanzhi Wang. Packqvit: Faster sub-8-bit vision transformers via full and packed quantization on the mobile.Advances in Neural Information Processing Systems, 36:9015–9028, 2023

    work page 2023

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  7. [7]

    Learned step size quantization.arXiv preprint arXiv:1902.08153, 2019

    Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization.arXiv preprint arXiv:1902.08153, 2019

    arXiv 1902

  8. [8]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    arXiv 2022

Show all 38 references
  1. [9]

    Quantization without tears.arXiv preprint arXiv:2411.13918, 2024

    Minghao Fu, Hao Yu, Jie Shao, Junjie Zhou, Ke Zhu, and Jianxin Wu. Quantization without tears.arXiv preprint arXiv:2411.13918, 2024

    2024 arXiv

  2. [10]

    Dtl: Disentangled transfer learning for visual recognition

    Minghao Fu, Ke Zhu, and Jianxin Wu. Dtl: Disentangled transfer learning for visual recognition. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12082– 12090, 2024

    2024

  3. [11]

    Knowledge distillation: A survey.International Journal of Computer Vision, 129(6):1789–1819, 2021

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey.International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  4. [12]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Girshick. Mask r-cnn. InProceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017

    2017

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  6. [14]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  7. [15]

    Quantization vari- ation: A new perspective on training transformers with low-bit precision.arXiv preprint arXiv:2307.00331, 2023

    Xijie Huang, Zhiqiang Shen, Pingcheng Dong, and Kwang-Ting Cheng. Quantization vari- ation: A new perspective on training transformers with low-bit precision.arXiv preprint arXiv:2307.00331, 2023

    2023 arXiv

  8. [16]

    Aiqvit: Architecture- informed post-training quantization for vision transformers.arXiv preprint arXiv:2502.04628, 2025

    Runqing Jiang, Ye Zhang, Longguang Wang, Pengpeng Yu, and Yulan Guo. Aiqvit: Architecture- informed post-training quantization for vision transformers.arXiv preprint arXiv:2502.04628, 2025

    2025 arXiv

  9. [17]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. InProceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013. 13

    2013

  10. [18]

    A comprehensive study on quantization techniques for large language models

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. A comprehensive study on quantization techniques for large language models. In2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 224–231. IEEE, 2024

    2024

  11. [19]

    Q-vit: Accurate and fully quantized low-bit vision transformer.Advances in neural information processing systems, 35:34451–34463, 2022

    Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer.Advances in neural information processing systems, 35:34451–34463, 2022

    2022

  12. [20]

    Repq-vit: Scale reparameterization for post-training quantization of vision transformers

    Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17227–17236, 2023

    2023

  13. [21]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  14. [22]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  15. [23]

    Oscillation-free quantization for low-bit vision transformers

    Shih-Yang Liu, Zechun Liu, and Kwang-Ting Cheng. Oscillation-free quantization for low-bit vision transformers. InInternational conference on machine learning, pages 21813–21824. PMLR, 2023

    2023

  16. [24]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  17. [25]

    Post-training quantization for vision transformer.Advances in Neural Information Processing Systems, 34:28092–28103, 2021

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer.Advances in Neural Information Processing Systems, 34:28092–28103, 2021

    2021

  18. [26]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  19. [27]

    Fine- grained visual classification of aircraft.arXiv preprint arXiv:1306.5151, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft.arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  20. [28]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008

    2008

  21. [29]

    A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  22. [30]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012

    2012

  23. [31]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv´e J´egou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021

    2021

  24. [32]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- oth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  25. [33]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 14

    2017

  26. [34]

    Distilling knowledge by mimicking features.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8183–8195, 2021

    Guo-Hua Wang, Yifan Ge, and Jianxin Wu. Distilling knowledge by mimicking features.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8183–8195, 2021

    2021

  27. [35]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  28. [36]

    Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization

    Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. InEuropean conference on computer vision, pages 191–207. Springer, 2022

    2022

  29. [37]

    All you need in knowledge distillation is a tailored coordinate system

    Junjie Zhou, Ke Zhu, and Jianxin Wu. All you need in knowledge distillation is a tailored coordinate system. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22946–22954, 2025

    2025

  30. [38]

    Quantized feature distillation for network quantization

    Ke Zhu, Yin-Yin He, and Jianxin Wu. Quantized feature distillation for network quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11452–11460, 2023. 15

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.