REVIEW 4 major objections 4 minor 52 references
QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection
T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Extreme 4-bit quantization of open-vocabulary detectors is recoverable if the model is quantized stage-by-stage and distilled through text-anchored pairwise similarity matrices.
desk verdict A plausible, well-specified method for 4-bit OVOD quantization, but the empirical support is a single stripped baseline with no code or seeds — worth refereeing, not worth trusting yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Text-anchored pairwise similarity matrices: for each text query c, the method stacks the text embedding t_c with N_c region embeddings assigned by the teacher's task-aligned label assignment (TAL), row-wise L2-normalizes, and forms S_c = X_c X_c^T; the first row/column encodes region-text alignment, the rest encodes region-region relationships, and the student is trained to match the teacher's S_c via Smooth L1. The companion mechanism is curriculum QAT (CQAT): the model is split into functional units and quantized sequentially, here in two stages — backbone first with the neck-head frozen, then neck-head — so that unquantized modules isolate errors and give stable gradients. The two mechani
What would settle it
Hold CR-QAT fixed but replace the teacher's TAL-assigned positive regions per text query with a fixed IoU-based selection on YOLO-World-X; a drop from +4.2 AP over QAT to near zero would confirm that reliable TAL anchors carry the result.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the multi-dimensional relational structure of an OVOD model — the cosine similarities between a text query and the region embeddings assigned to it, and the pairwise similarities among those regions — is the load-bearing quantity that low-bit quantization destroys, and that this structure can be explicitly transferred from a full-precision teacher by building text-anchored pairwise similarity matrices and minimizing their discrepancy. The authors also find that curriculum staging is what makes this distillation effective: applying KD without the stage-by-stage quantization yields almost no gain (+0.4 AP), the curriculum without KD yield
Load-bearing premise
The method's benefit rests on the teacher's TAL-positive regions being reliable anchors for each text query, since noisy assignments would consolidate the wrong relational structure during distillation.
Editorial extensions
If this is right
- Under 4-4-8 (Ch-T-H) quantization, CR-QAT adds +2.5, +3.4, and +4.2 AP over QAT on YOLO-World-M/L/X for LVIS zero-shot, and +4.1, +6.8, and +7.6 AP on COCO.
- Gains concentrate on rare categories: APr improves by +5.2 to +5.7 AP on LVIS-L/X, which the paper takes as evidence that damaged vision-language alignment is being restored.
- The benefit persists across bit-widths and granularities: 4-5-8 (Ch-T-H) yields +0.6 AP and 3-3-8 (Ch-Ch-H) yields +1.3 AP over QAT.
- Embedding-space relational distortion correlates with confidence-space distortion (Spearman ρ=0.76), so preserving inter-region relationships in the embedding carries through to final predictions.
Reading between the lines
- The method is claimed to be architecture-agnostic, but only YOLO-World is tested; a natural test is to apply the same two-stage curriculum and text-anchored distillation to transformer-based OVOD detectors with online cross-modal fusion, where the neck-head grouping may need to be redrawn.
- If the teacher's TAL assignment is itself noisy on rare categories, TRKD could reinforce false region-text relationships; a robust variant might weight the matrix loss by assignment confidence or use multiple positive sets.
- The paper's observation that KD alone fails without the curriculum suggests that for even lower bit-widths (3-bit weight/activation), the curriculum may need more than two stages, or a different freeze schedule; this is an untested extrapolation.
- A deployment-oriented spin-off: instead of recomputing S_c every iteration, cache the teacher's pairwise matrices offline as fixed soft targets, which could cut the distillation memory footprint while retaining most of the relational signal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CR-QAT (also called QATMA in the abstract), a quantization-aware training framework for open-vocabulary object detection. It consists of curriculum QAT (CQAT), which progressively quantizes the backbone then the neck-head, and text-centric relational knowledge distillation (TRKD), which transfers teacher-student region-text and region-region pairwise similarity matrices. Experiments on YOLO-World with 4-4-8-bit weight-activation-attention quantization report consistent AP gains over a plain LSQ QAT baseline on LVIS and COCO zero-shot benchmarks, with relative improvements up to 38.9% and 40.9%.
Significance. If the empirical claims hold, this is a useful first systematic study of extreme low-bit quantization for OVOD models. The method is described in enough detail to reproduce (equations for quantization, curriculum objective, TRKD loss, and hyperparameters), and the ablations are internally coherent. However, the evidence base is currently too narrow: the only QAT comparator is a self-implemented plain-LSQ baseline, no code or seeds are provided, and all results are single runs without variance. The abstract/full-text naming and AP-gain mismatch (QATMA vs CR-QAT; 4.3 vs 4.2) further reduces confidence. The central claim that CR-QAT 'consistently outperforms existing QAT baselines' is therefore not yet established to the standard expected for a venue that publishes comparative empirical claims.
major comments (4)
- [§5.2, Tables 1–2] The only QAT comparator is the authors' own LSQ-based baseline trained with task loss only. The abstract and §1 claim 'consistently outperforms existing QAT baselines', but no existing low-bit QAT method (e.g., LSQ+ [2], QKD [23], Q-DETR [43], or a feature-distillation baseline) is compared. Because CR-QAT adds a curriculum plus two KD losses, the reported +2.5 to +7.6 AP gains could in principle come from generic KD or the curriculum alone. Please add comparisons with at least one published QAT/KD method adapted to YOLO-World, and report the plain-LSQ baseline with the same training budget and also with feature distillation only.
- [Abstract vs §5.2/Table 1] The abstract describes the method as 'QATMA' and reports an AP gain of 'up to 4.3' on LVIS; the full text uses 'CR-QAT' and Table 1 reports +4.2 AP for YOLO-World-X. This is a material inconsistency between the two versions of the paper. It must be reconciled, and the final claim should match the numbers in the tables.
- [§5.4, Tables 3–6] All experiments are single runs with no seeds, error bars, or confidence intervals. Several decisive comparisons are numerically small: Table 4 shows 13.1 vs 13.7 vs 14.3; Table 6 shows 22.0 vs 21.4 for 4-5-8. Without multiple seeds and variance, the ablations and granularity conclusions are not statistically grounded. Please report mean±std over at least three seeds, and release code or a reproducible configuration.
- [§1, §4.1, §5] The paper claims an 'architecture-agnostic' framework and states in §4.1 that the two-stage curriculum applies to 'widely-adopted OVOD architectures (e.g., GLIP, YOLO-World)', yet experiments cover only YOLO-World. This is an overclaim relative to the evidence. Either add a demonstration on a second architecture (e.g., GLIP-T or Grounding DINO) or restrict the claims to YOLO-World and motivate transferability only as future work.
minor comments (4)
- [§4.2, Eq. (10)] Clarify whether the same N_c region embeddings are used for teacher and student, and how regions are selected when the student's TAL assignment differs from the teacher's. The current text says positive regions are determined by the teacher's TAL; if the student uses the same assignment, this should be stated explicitly.
- [§5.1] It is unclear which layers are 'first and last layers' in the YOLO-World instantiation (backbone stem? final head?). Also, state whether the text encoder is kept in FP32 for both teacher and student during training, since it is excluded from quantization.
- [§5.3] The sentence 'distilling multi-scale feature pyramids proves unstable under low-bit settings [50]' has a stray superscript '4' after the citation. Also, footnote 4 says three-stage with neck distillation yields 13.3, which should be incorporated into Table 5 for completeness.
- [§5.4, Fig. 3] The embedding-relation MAE in Fig. 3 is essentially the TRKD objective, so the high correlation with confidence-relation MAE is informative but not an independent confirmation of the method's benefit. This should be noted in the text.
Circularity Check
No circular derivation found; the method's efficacy is evaluated on held-out AP benchmarks rather than its own training objective.
full rationale
CR-QAT's derivation chain—motivating quantization-induced loss of region-text and region-region structure, proposing curriculum QAT (CQAT) and text-centric relational KD (TRKD), and evaluating on LVIS/COCO zero-shot AP—does not reduce to its inputs by construction. The reported AP improvements (Tables 1-2) are measured on external benchmarks (LVIS miniVal fixed AP, COCO val2017) against a QAT baseline, not on the training loss itself. The ablations (Tables 3-5) decompose the contributions of curriculum and KD components and would be meaningless if the result were definitionally forced. The embedding-distortion analyses (Fig. 1, Fig. 3) do track the TRKD objective (Eq. 10), since TRKD directly minimizes the discrepancy between teacher and student pairwise similarity matrices; thus those plots are illustrative of the loss behavior rather than independent confirmation. However, that is not circularity—the central claim of improved AP is independently evaluated. No self-citations are used as load-bearing evidence; references to error accumulation [47] and feature-distillation instability [50] are external prior works. The abstract/full-text naming and AP-gain discrepancy (QATMA vs CR-QAT, 4.3 vs 4.2) is an internal inconsistency that affects reproducibility confidence, but it does not constitute a circular derivation. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- KD loss weight lambda_i =
6.0
- Curriculum stage data split =
1/3 backbone, 2/3 neck-head
- LSQ learning-rate ratio =
0.1 x base learning rate
assumptions (4)
- domain assumption Uniform quantization with the straight-through estimator gives usable gradients for training quantized networks.
- domain assumption The FP32 teacher's text embeddings and TAL positive-region assignments provide reliable relational knowledge to transfer.
- domain assumption Objects365v2 + GQA training with YOLO-World's pretraining pipeline is a valid zero-shot proxy for LVIS/COCO evaluation.
- domain assumption Early-layer quantization error accumulation is the dominant failure mode, motivating stage-by-stage optimization.
Cite this review
Pith. "Pith review of QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection." pith.science (2026). https://pith.science/paper/2Z6RSOCP
@misc{pith2026260305964,
author = {Pith},
title = {Pith review of: QATMA: Quantization-Aware Training with Multimodal Alignment for Open-Vocabulary Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/2Z6RSOCP}},
note = {Machine review of arXiv:2603.05964}
}
read the original abstract
Quantizing open-vocabulary object detection (OVOD) models reduces their memory and computational costs, but extremely low-bit quantization severely degrades both cross-modal (region-text) and intra-modal (region-region) alignments. This multimodal degradation is a unique challenge that prior quantization methods for closed-vocabulary detectors fail to resolve. To overcome this, we propose Quantization-Aware Training with Multimodal Alignment (QATMA), the first multimodal-aware and architecture-agnostic QAT framework tailored for OVOD. QATMA integrates two key components: (i) Curriculum QAT, which partitions the detector by functional roles and progressively expands the quantization scope to suppress error accumulation and ensure stable optimization; and (ii) Text-anchored Pairwise Similarity Distillation, which transfers both region-text and region-region alignments from a full-precision teacher model via pairwise cosine similarities in the joint embedding space. Experimental results on LVIS and COCO zero-shot benchmarks demonstrate that QATMA significantly outperforms existing QAT baselines under extremely low-bit settings, achieving gains of up to 4.3 and 7.6 AP, respectively.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[2]
In: CVPRW
Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., Kwak, N.: LSQ+: Improving low- bit quantization through learnable offsets and better initialization. In: CVPRW. pp. 696–697 (2020)
2020
-
[23]
arXiv preprint arXiv:1911.12491 (2019)
Kim,J.,Bhalgat,Y.,Lee,J.,Patel,C.,Kwak,N.:QKD:Quantization-awareknowl- edge distillation. arXiv preprint arXiv:1911.12491 (2019)
arXiv 1911
-
[43]
In: CVPR
Xu, S., Li, Y., Lin, M., Gao, P., Guo, G., Lü, J., Zhang, B.: Q-DETR: An efficient low-bit quantized detection transformer. In: CVPR. pp. 3842–3851 (2023)
2023
-
[1]
arXiv preprint arXiv:1308.3432 (2013)
Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)
arXiv 2013
-
[3]
In: NeurIPS
Cao, W., Zhang, Y., Gao, J., Cheng, A., Cheng, K., Cheng, J.: PKD: General distillation framework for object detectors via pearson correlation coefficient. In: NeurIPS. pp. 15394–15406 (2022)
2022
-
[4]
In: ECCV
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: ECCV. pp. 213–229 (2020)
2020
-
[5]
In: NeurIPS
Chen, G., Choi, W., Yu, X., Han, T., Chandraker, M.: Learning efficient object detection models with knowledge distillation. In: NeurIPS. vol. 30 (2017)
2017
-
[6]
In: CVPR
Chen, P., Liu, J., Zhuang, B., Tan, M., Shen, C.: AQD: Towards accurate quantized object detection. In: CVPR. pp. 104–113 (2021)
2021
Show all 52 references
-
[7]
In: CVPR
Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., Shan, Y.: YOLO-World: Real-time open-vocabulary object detection. In: CVPR. pp. 16901–16911 (2024)
2024
-
[8]
arXiv preprint arXiv:1805.06085 (2018)
Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.J., Srinivasan, V., Gopalakrish- nan, K.: PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 (2018)
2018 arXiv
-
[9]
In: ICCV Workshops
Choukroun, Y., Kravchik, E., Yang, F., Kisilev, P.: Low-bit quantization of neural networks for efficient inference. In: ICCV Workshops. pp. 3009–3018 (2019)
2019
-
[10]
In: ICLR (2020)
Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned step size quantization. In: ICLR (2020)
2020
-
[11]
In: ICCV
Feng, C., Zhong, Y., Gao, Y., Scott, M.R., Huang, W.: TOOD: Task-aligned one- stage object detection. In: ICCV. pp. 3490–3499 (2021)
2021
-
[12]
In: CVPR
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accu- rate object detection and semantic segmentation. In: CVPR. pp. 580–587 (2014)
2014
-
[13]
In: ICLR (2022)
Gu, X., Lin, T., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: ICLR (2022)
2022
-
[14]
In: CVPR
Gupta, A., Dollár, P., Girshick, R.: LVIS: A dataset for large vocabulary instance segmentation. In: CVPR. pp. 5356–5364 (2019)
2019
-
[15]
In: WACV
Gupta, K., Asthana, A.: Reducing the side-effects of oscillations in training of quantized YOLO networks. In: WACV. pp. 2452–2461 (2024)
2024
-
[16]
In: NeurIPS
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: NeurIPS. vol. 28 (2015)
2015
-
[17]
In: ICCV
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: ICCV. pp. 1389–1397 (2017)
2017
-
[18]
arXiv preprint arXiv:1503.02531 (2015)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
2015 arXiv
-
[19]
In: CVPR
Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: CVPR. pp. 6700–6709 (2019)
2019
-
[20]
In: CVPR
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., Kalenichenko,D.:Quantizationandtrainingofneuralnetworksforefficientinteger- arithmetic-only inference. In: CVPR. pp. 2704–2713 (2018)
2018
-
[21]
In: ICML
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML. pp. 4904–4916 (2021) 16 J. Park et al
2021
-
[22]
Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLOv8.https://github.com/ ultralytics/ultralytics(2023)
2023
-
[24]
In: CVPR
Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., Chang, K., Gao, J.: Grounded language-image pre-training. In: CVPR. pp. 10955–10965 (2022)
2022
-
[25]
In: CVPR
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., Fan, R.: Fully quantized network for object detection. In: CVPR. pp. 2810–2819 (2019)
2019
-
[26]
In: ICCV
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV. pp. 2999–3007 (2017)
2017
-
[27]
In: ECCV
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV. pp. 740–755 (2014)
2014
-
[28]
In: ECCV
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In: ECCV. pp. 38–55 (2024)
2024
-
[29]
In: ECCV
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single shot multibox detector. In: ECCV. pp. 21–37 (2016)
2016
-
[30]
In: ICCV
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV. pp. 10012–10022 (2021)
2021
-
[31]
In: ECCV
Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Doso- vitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al.: Simple open-vocabulary object detection. In: ECCV. pp. 728–755 (2022)
2022
-
[32]
In: ICML
Nagel, M., Amjad, R.A., Van Baalen, M., Louizos, C., Blankevoort, T.: Up or down? adaptive rounding for post-training quantization. In: ICML. pp. 7197–7206 (2020)
2020
-
[33]
In: ICCV
Nagel, M., Baalen, M.v., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. In: ICCV. pp. 1325–1334 (2019)
2019
-
[34]
In: ICML
Nagel, M., Fournarakis, M., Bondarenko, Y., Blankevoort, T.: Overcoming oscilla- tions in quantization-aware training. In: ICML. pp. 16318–16330 (2022)
2022
-
[35]
In: ICLR (2018)
Polino, A., Pascanu, R., Alistarh, D.: Model compression via distillation and quan- tization. In: ICLR (2018)
2018
-
[36]
In: ICML
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)
2021
-
[37]
In: CVPR
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: CVPR. pp. 779–788 (2016)
2016
-
[38]
arXiv preprint arXiv:1804.02767 (2018)
Redmon, J., Farhadi, A.: YOLOv3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
2018 arXiv
-
[39]
In: NeurIPS
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: NeurIPS. pp. 91–99 (2015)
2015
-
[40]
In: ICCV
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A large-scale, high-quality dataset for object detection. In: ICCV. pp. 8430–8439 (2019)
2019
-
[41]
In: ICCV
Tian, Z., Shen, C., Chen, H., He, T.: FCOS: Fully convolutional one-stage object detection. In: ICCV. pp. 9627–9636 (2019)
2019
-
[42]
In: CVPR
Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: CVPR. pp. 7464–7475 (2023) CR-QAT for Open-Vocabulary Object Detection 17
2023
-
[44]
In: CVPR
Zareian, A., Rosa, K.D., Hu, D.H., Chang, S.F.: Open-vocabulary object detection using captions. In: CVPR. pp. 14393–14402 (2021)
2021
-
[45]
In: ICLR (2023)
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.Y.: DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In: ICLR (2023)
2023
-
[46]
In: NeurIPS
Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: GLIPv2: Unifying localization and vision-language under- standing. In: NeurIPS. pp. 36067–36080 (2022)
2022
-
[47]
In: ICCV
Zhang, Y., Dong, Z., Yang, H., Lu, M., Tseng, C.C., Du, Y., Keutzer, K., Du, L., Zhang, S.: QD-BEV: Quantization-aware view-guided distillation for multi-view 3D object detection. In: ICCV. pp. 3825–3835 (2023)
2023
-
[48]
In: CVPR
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., Gao, J.: RegionCLIP: Region-based language-image pretraining. In: CVPR. pp. 16793–16803 (2022)
2022
-
[49]
arXiv preprint arXiv:1606.06160 (2016)
Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., Zou, Y.: DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)
2016 arXiv
-
[50]
In: AAAI
Zhu, K., He, Y.Y., Wu, J.: Quantized feature distillation for network quantization. In: AAAI. vol. 37, pp. 11452–11460 (2023)
2023
-
[51]
In: ICLR (2021)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: Deformable transformers for end-to-end object detection. In: ICLR (2021)
2021
-
[52]
In: CVPR
Zhuang, B., Liu, L., Tan, M., Shen, C., Reid, I.: Training quantized neural networks with a full-precision auxiliary module. In: CVPR. pp. 1488–1497 (2020)
2020
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.