REVIEW 4 major objections 4 minor 46 references
Compressed Feature Quality Assessment: Dataset and Baselines
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper establishes a benchmark for compressed feature quality assessment and shows that MSE, cosine similarity, and CKA fail to consistently measure the semantic distortion introduced by feature compression.
desk verdict A real first benchmark for compressed feature quality, but the ground-truth labels for two of the three tasks measure output drift rather than task-performance drop, so the headline negative result is only solid for classification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the paired dataset with its semantic-distortion labels, together with the evaluation protocol of PLCC and SROCC computed per source feature over its ten compressed versions. The dataset's 300 original features come from DINOv2's intermediate blocks; the four codecs are HM, VTM, a multi-task Hyperprior, and a task-specific Hyperprior; the labels are task-head performance drops. The protocol turns CFQA into a supervised correlation problem: a metric is good if its scores rank the ten distortions the same way the task-level labels do.
What would settle it
Recompute the ground-truth labels using more than one task-head checkpoint or repeated inference, then re-run the PLCC and SROCC evaluation; if the rankings of the ten compressed versions change materially, or if expanding to, say, 50 versions per source shifts the correlations by more than about 0.1, the benchmark's labels are too noisy to support the reported conclusions.
Extended reading notes
Core claim
The central discovery is that semantic distortion caused by feature compression is a measurable but poorly predicted quantity. The authors define true semantic distortion as performance degradation of task heads: softmax rank for classification, mIoU difference for segmentation, and RMSE difference for depth, computed by comparing outputs from compressed and original features. Their benchmark supplies paired feature and distortion data across three tasks and four codecs. Evaluated by PLCC and SROCC, the three candidate metrics have no stable winning behaviour: correlation signs and strengths flip across codecs and tasks, and for segmentation all metrics occasionally show near-zero or even opposite correlation. The paper's conclusion is that conventional signal-similarity measures do not capture the semantic degradation introduced by feature coding, so CFQA needs its own quality estimators.
Load-bearing premise
The benchmark's ground truth assumes that task-head performance drops—softmax rank, mIoU difference, and RMSE difference—are stable, precise measures of semantic distortion, and that ten compressed versions per source suffice for reliable correlation; no variance or significance statistics are reported to back either assumption.
Editorial extensions
If this is right
- Researchers can train or tune CFQA metrics against the released labels without running downstream inference.
- Codec training can use such a metric as a semantic-aware loss, replacing end-to-end task labels.
- Edge-cloud systems could use the metric to decide when to re-encode a feature at a higher bitrate before transmission.
- Baseline results identify cosine similarity as the default starting point for handcrafted codecs, while exposing learning-based codecs as the harder regime.
- The benchmark's three tasks span coarse-to-fine semantics, so a metric that performs well on all three would be a strong candidate for general CFQA.
Reading between the lines
- The softmax rank label for classification can jump from 1 to 1000 with tiny probability changes, so a smoother surrogate such as confidence margin or cross-entropy might yield more stable correlation measurements.
- The same dataset could support a learned CFQA regressor, and reporting its per-codec cross-generalisation would directly test whether any task-agnostic estimator exists.
- Extending labels to video features or to features from other backbones would test whether the findings generalise beyond DINOv2.
- If PLCC and SROCC values are unstable across the ten samples, averaging over more compressed versions per source would settle whether the reported correlations are reliable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper formalizes the task of Compressed Feature Quality Assessment (CFQA), introduces a benchmark dataset of 300 original and 12,000 compressed features derived from three vision tasks (classification, segmentation, depth) and four feature codecs, proposes task-specific semantic distortion labels, and evaluates three baseline metrics (MSE, cosine similarity, CKA) using PLCC and SROCC. The authors conclude that none of the three metrics consistently captures semantic distortion and release the dataset and code.
Significance. The proposed dataset and evaluation protocol are a potentially valuable community resource: if the ground-truth labels are valid, the paper provides the first public CFQA benchmark with paired semantic-distortion labels across multiple tasks and codecs, and the systematic baseline comparison addresses a real gap. The manuscript is an empirical benchmark rather than a derivation, so circularity is not an issue, and the release of data and code is a concrete strength. However, the significance is conditional on the correctness and unambiguous definition of the ground-truth labels, and on the statistical reliability of the reported correlations; the current manuscript does not fully establish either.
major comments (4)
- [Sec. 4.6 vs Sec. 3.1] The definition of ground-truth semantic distortion is internally inconsistent. Sec. 3.1 defines the ground truth as task performance s = A_T(h_T(f_hat)), and the abstract claims the dataset provides 'task-specific performance degradation.' Sec. 4.6, however, operationalizes Seg distortion as 'the mIoU difference between the segmentation masks predicted from f_hat and f' and Dpt distortion as 'the RMSE difference between the depth maps predicted from f_hat and f.' These are output-drift measures, not task-performance degradation against ground truth: a compressed feature that changes an incorrect original prediction into a different incorrect prediction receives a large label, while one that preserves the incorrect prediction receives zero. In contrast, Tables 2 and 3 report standard task-performance numbers (Acc., mIoU, RMSE), and the sign pattern in Table 4 is consistent with task-performance labels. Please disambiguate: if the actual labels are output drift, the abstract's claim is unsupported for two of the three tasks and the conclusions in Sec. 5.5 must be rephrased; if the labels are true task-performance drops, Sec. 4.6 must be rewritten to say 'difference in mIoU with ground truth' and 'difference in RMSE with ground truth.'
- [Sec. 4.6 (Cls label)] For Cls, the distortion label is 'the rank in the softmax function generated from f_hat.' This is not equivalent to task-performance degradation: if the true class has rank 2, the top-1 prediction is still correct, so a conventional accuracy-based performance measure would show no degradation. The paper should either report top-1 accuracy drop as an additional or alternative label, or provide a clear justification for why softmax rank is a valid proxy for semantic utility loss. As written, the 'true semantic distortion' claim for the Cls task is weaker than stated.
- [Sec. 5.2 and Table 4] Per-feature PLCC and SROCC are computed over only 10 rate points per codec. Table 4 then reports averages over the 100 source features, but no standard deviations, confidence intervals, or significance tests are given. With n=10 per correlation, the reported differences are potentially dominated by sampling noise; for example, the task-specific Hyperprior Cls SROCC values range from 0.1303 (MSE) to -0.6084 (cosine), and it is unclear whether these are statistically distinguishable. Please report variance or bootstrap confidence intervals for the Table 4 averages and for the Figure 2 distributions, and add appropriate significance tests (e.g., paired tests across the 100 features).
- [Introduction and Ref. [22]] The Introduction claims 'there is no public dataset that provides compressed features with corresponding task performance across multiple tasks and codecs,' but reference [22] (Feature Coding in the Era of Large Models: Dataset, Test Conditions, and Benchmark) appears to be a closely related public resource by the same group. Please explain how the proposed CFQA dataset differs from [22] and justify the 'first benchmark' claim more carefully; if [22] already provides compressed features with task-performance labels, the novelty claim needs to be narrowed.
minor comments (4)
- [Sec. 4.1] There is a typo in the sentence 'The dataset is designed to support the analysis ofsemantic distortion' — 'ofsemantic' should be 'of semantic.'
- [Table 4 and Sec. 5.4] The paper should state the sign convention for the correlation coefficients explicitly. For similarity metrics such as cosine similarity and CKA, negative PLCC/SROCC values are expected when the ground truth is a distortion measure, while for MSE positive values are expected; without this clarification, readers may misinterpret the signs in Table 4.
- [Figure 2] The axis labels in Figure 2 appear garbled (e.g., '1 0 Correlation'), and the caption does not fully explain how the histograms are constructed from per-feature PLCC values. Please redraw the figure with clear axis labels and describe the rounding/binning procedure in the caption.
- [Sec. 5.1 and reproducibility] The manuscript does not specify how MSE, cosine similarity, and CKA are computed for features of shape 2×4×1611×1536, particularly how the tensor is flattened or pooled before computing these metrics and what kernel/HSIC estimator is used for CKA. Please provide these details in the dataset or supplementary material to support reproducibility.
Circularity Check
Empirical benchmark with direct correlation analysis; no derivation chain reduces to its own inputs.
full rationale
This paper is an empirical benchmark construction and evaluation, not a derivation. The central quantities are: (a) compressed features produced by applying four codecs to extracted DINOv2 features, (b) semantic-distortion labels computed from downstream head outputs on original versus compressed features (Sec. 4.6), and (c) PLCC/SROCC correlations between three hand-defined metrics (MSE, cosine similarity, CKA) and those labels (Sec. 5.2, Table 4). None of these steps fits a parameter whose value is recycled into the evaluation, and no claimed prediction is defined in terms of the metric being evaluated. The only self-citations (e.g., Ref. [22] for codec training protocols, and the authors' prior feature-coding works) are used as methodological background or training recipes, not as load-bearing evidence for the paper's empirical conclusions. A possible validity concern is that the Seg and Dpt labels are prediction-drift measures (mIoU/RMSE difference between masks/depth maps from f_hat and f) rather than task-performance degradation against ground-truth annotations, but that is a question of whether the proxy correctly measures the paper's stated construct, not circularity: the correlation analysis is self-contained with respect to the labels as defined. Therefore no circular step can be exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption Task-specific performance degradation (softmax rank, mIoU difference, RMSE difference) is a valid proxy for true semantic distortion.
- domain assumption DINOv2 middle-block features are a representative testbed for general-purpose feature coding.
- domain assumption The 100 sampled images per task are representative of ImageNet, VOC, and NYUv2 respectively.
- domain assumption Correlations computed over 10 compression levels per feature yield stable estimates of metric quality.
Cite this review
Pith. "Pith review of Compressed Feature Quality Assessment: Dataset and Baselines." pith.science (2026). https://pith.science/paper/VBI55XQL
@misc{pith2026250607412,
author = {Pith},
title = {Pith review of: Compressed Feature Quality Assessment: Dataset and Baselines},
year = {2026},
howpublished = {\url{https://pith.science/paper/VBI55XQL}},
note = {Machine review of arXiv:2506.07412}
}
read the original abstract
The widespread deployment of large models in resource-constrained environments has underscored the need for efficient transmission of intermediate feature representations. In this context, feature coding, which compresses features into compact bitstreams, becomes a critical component for scenarios involving feature transmission, storage, and reuse. However, this compression process inevitably introduces semantic degradation that is difficult to quantify with traditional metrics. To address this, we formalize the research problem of Compressed Feature Quality Assessment (CFQA), aiming to evaluate the semantic fidelity of compressed features. To advance CFQA research, we propose the first benchmark dataset, comprising 300 original features and 12000 compressed features derived from three vision tasks and four feature codecs. Task-specific performance degradation is provided as true semantic distortion for evaluating CFQA metrics. We systematically assess three widely used metrics -- MSE, cosine similarity, and Centered Kernel Alignment (CKA) -- in terms of their ability to capture semantic degradation. Our findings demonstrate the representativeness of the proposed dataset while underscoring the need for more sophisticated metrics capable of measuring semantic distortion in compressed features. This work advances the field by establishing a foundational benchmark and providing a critical resource for the community to explore CFQA. To foster further research, we release the dataset and all associated source code at https://github.com/chansongoal/Compressed-Feature-Quality-Assessment.
Figures
Reference graph
Works this paper leans on
-
[22]
Changsheng Gao, Yifan Ma, Qiaoxi Chen, Yenan Xu, Dong Liu, and Weisi Lin
- [1]
-
[2]
Saeed Ranjbar Alvar and Ivan V. Bajić. 2020. Bit Allocation for Multi-Task Collaborative Intelligence. In ICASSP. 4342–4346. doi:10.1109/ICASSP40776.2020. 9054770
arXiv 2020
- [3]
-
[4]
Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston
Johannes Ballé, David C. Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. 2018. Variational image compression with a scale hyperprior. ArXiv abs/1802.01436 (2018)
arXiv 2018
- [5]
-
[6]
Qiaoxi Chen, Changsheng Gao, and Dong Liu. 2024. End-to-End Learned Scalable Multilayer Feature Compression For Machine Vision Tasks. In ICIP. 1781–1787. doi:10.1109/ICIP51287.2024.10647798
arXiv 2024
-
[8]
Zhuo Chen, Kui Fan, Shiqi Wang, Lingyu Duan, Weisi Lin, and Alex Chichung Kot. 2020. Toward Intelligent Sensing: Intermediate Deep Feature Compression. IEEE Transactions on Image Processing 29 (2020), 2230–2243. doi:10.1109/TIP.2019. 2941660
doi:10.1109/tip.2019 2020
Show all 46 references
-
[9]
Zhuo Chen, Kui Fan, Shiqi Wang, Ling-Yu Duan, Weisi Lin, and Alex Kot. 2019. Lossy Intermediate Deep Learning Feature Compression and Evaluation. In Proceedings of the 27th ACM International Conference on Multimedia (MM ’19) . Association for Computing Machinery, New York, NY,...
2019
-
[10]
Hyomin Choi and Ivan V. Bajić. 2018. Deep Feature Compression for Collaborative Object Detection. In ICIP. 3743–3747. doi:10.1109/ICIP.2018.8451100
2018
-
[11]
Hyomin Choi and Ivan V. Bajić. 2021. Latent-Space Scalability for Multi-Task Col- laborative Intelligence. In ICIP. 3562–3566. doi:10.1109/ICIP42928.2021.9506712
2021
-
[12]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- geNet: A large-scale hierarchical image database. In CVPR. 248–255. doi:10.1109/ CVPR.2009.5206848
2009
-
[13]
Lingyu Duan, Jiaying Liu, Wenhan Yang, Tiejun Huang, and Wen Gao. 2020. Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics. IEEE Transactions on Image Processing 29 (2020), 8680–8695. doi:10.1109/TIP.2020.3016485
2020
-
[14]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
2024 arXiv
-
[15]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. Inter- national journal of computer vision 88 (2010), 303–338
2010
-
[16]
Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen. 2022. Image coding for machines with omnipotent feature learning. In ECCV. Springer, 510–528
2022
-
[17]
Changsheng Gao, Yiheng Jiang, Li Li, Dong Liu, and Feng Wu. 2024. DMOFC: Discrimination Metric-Optimized Feature Compression. In PCS. 1–5. doi:10.1109/ PCS60826.2024.10566361
2024
-
[18]
Changsheng Gao, Yiheng Jiang, Siqi Wu, Yifan Ma, Li Li, and Dong Liu. 2025. IMOFC: Identity-Level Metric Optimized Feature Compression for Identification Tasks. IEEE Transactions on Circuits and Systems for Video Technology 35, 2 (2025), 1855–1869. doi:10.1109/TCSVT.2024.3467124
2025
-
[19]
Changsheng Gao, Zhuoyuan Li, Li Li, Dong Liu, and Feng Wu. 2024. Rethinking the Joint Optimization in Video Coding for Machines: A Case Study. In DCC. 556–556
2024
-
[20]
Changsheng Gao, Shan Liu, Feng Wu, and Weisi Lin. 2025. Cross-architecture uni- versal feature coding via distribution alignment. arXiv preprint arXiv:2506.12737 (2025)
2025 arXiv
-
[21]
Changsheng Gao, Zijie Liu, Li Li, Dong Liu, Xiaoyan Sun, and Weisi Lin. 2025. DT- UFC: Universal Large Model Feature Coding via Peaky-to-Balanced Distribution Transformation. arXiv preprint arXiv:2506.16495 (2025)
2025 arXiv
-
[23]
Sha Guo, Zhuo Chen, Yang Zhao, Ning Zhang, Xiaotong Li, and Lingyu Duan
-
[24]
Robert Henzel, Kiran Misra, and Tianying Ji. 2022. Efficient Feature Compression for the Object Tracking Task. In ICIP. 3505–3509. doi:10.1109/ICIP46576.2022. 9897802
2022
-
[25]
Yuzhang Hu, Sifeng Xia, Wenhan Yang, and Jiaying Liu. 2020. Sensitivity-Aware Bit Allocation for Intermediate Deep Feature Compression. In VCIP. 475–478. doi:10.1109/VCIP49819.2020.9301807
2020
-
[27]
Yeongwoong Kim, Hyewon Jeong, Janghyun Yu, Younhee Kim, Jooyoung Lee, Se Yoon Jeong, and Hui Yong Kim. 2023. End-to-End Learnable Multi-Scale Feature Compression for VCM. IEEE Transactions on Circuits and Systems for Video Technology (2023), 1–1. doi:10.1109/TCSVT.2023.3302858
2023
-
[28]
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. InInternational conference on machine learning. PMLR, 3519–3529
2019
-
[29]
Shibao Li, Chenxu Ma, Yunwu Zhang, Longfei Li, Chengzhi Wang, Xuerong Cui, and Jianhang Liu. 2023. Attention-Based Variable-Size Feature Compression Module for Edge Inference. The Journal of Supercomputing (2023)
2023
-
[30]
Yifan Ma, Changsheng Gao, Qiaoxi Chen, Li Li, Dong Liu, and Xiaoyan Sun. 2024. Feature Compression With 3D Sparse Convolution. In VCIP. 1–5
2024
-
[31]
Kiran Misra, Tianying Ji, Andrew Segall, and Frank Bossen. 2022. Video Feature Compression for Machine Tasks. In ICME. 1–6. doi:10.1109/ICME52920.2022. 9859894
2022
-
[32]
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. 2012. Indoor Segmentation and Support Inference from RGBD Images. In ECCV
2012
-
[33]
Benben Niu, Xiaoran Cao, Ziwei Wei, and Yun He. 2021. Entropy Optimized Deep Feature Compression. IEEE Signal Processing Letters 28 (2021), 324–328. doi:10.1109/LSP.2021.3052097
2021
-
[34]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
2023 arXiv
-
[35]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML. PMLR, 8748–8763
2021
-
[36]
Saurabh Singh, Sami Abu-El-Haija, Nick Johnston, Johannes Ballé, Abhinav Shrivastava, and George Toderici. 2020. End-to-End Learning of Compressible Features. In ICIP. 3349–3353. doi:10.1109/ICIP40778.2020.9190860
2020
-
[37]
Shurun Wang, Shiqi Wang, Wenhan Yang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. 2022. Towards Analysis-Friendly Face Representation With Scalable Feature and Texture Compression. IEEE Transactions on Multimedia 24 (2022), 3169–3181. doi:10.1109/TMM.2021.3094300
2022
-
[38]
Zixi Wang, Fan Li, Yunfei Zhang, and Yuan Zhang. 2023. Low-Rate Feature Compression for Collaborative Intelligence: Reducing Redundancy in Spatial and Statistical Levels. IEEE Transactions on Multimedia (2023), 1–16. doi:10.1109/ TMM.2023.3303716
2023
-
[39]
April 2023
WG2. April 2023. Call for Proposals on Feature Compression for Video Coding for Machines. ISO/IEC JTC 1/SC 29/WG 2, N282 (April 2023)
2023
-
[40]
Ning Yan, Changsheng Gao, Dong Liu, Houqiang Li, Li Li, and Feng Wu. 2021. SSSIC: Semantics-to-Signal Scalable Image Coding With Learned Structural Representations. IEEE Transactions on Image Processing 30 (2021), 8939–8954. doi:10.1109/TIP.2021.3121131
2021
-
[41]
Wenhan Yang, Haofeng Huang, Yueyu Hu, Ling-Yu Duan, and Jiaying Liu. 2024. Video Coding for Machines: Compact Visual Representation Compression for Intelligent Collaborative Analytics. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024), 1–18. doi:10.1109/TPA...
2024
-
[42]
Zhongzheng Yuan, Samyak Rawlekar, Siddharth Garg, Elza Erkip, and Yao Wang
-
[43]
Zhicong Zhang, Mengyang Wang, Mengyao Ma, Jiahui Li, and Xiaopeng Fan
-
[44]
Lingyu Zhu, Binzhe Li, Riyu Lu, Peilin Chen, Qi Mao, Zhao Wang, Wenhan Yang, and Shiqi Wang. 2024. Learned Image Compression for Both Humans and Machines via Dynamic Adaptation. In ICIP. IEEE, 1788–1794
2024
-
[2021]
MSFC: Deep Feature Compression in Multi-Task Network. In ICME. 1–6. doi:10.1109/ICME51207.2021.9428258
2021 arXiv
-
[2022]
In 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval
Feature Compression for Rate Constrained Object Detection on the Edge. In 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval. 1–6. doi:10.1109/MIPR54900.2022.00008
2022
-
[2023]
In Proceedings of the 31st ACM International Confer- ence on Multimedia (Ottawa ON, Canada) (MM ’23)
Toward Scalable Image Feature Compression: A Content-Adaptive and Diffusion-Based Approach. In Proceedings of the 31st ACM International Confer- ence on Multimedia (Ottawa ON, Canada) (MM ’23). Association for Computing Machinery, New York, NY, USA, 1431–1442. doi:10.1145/3581...
-
[2024]
arXiv preprint arXiv:2412.04307 (2024)
Feature Coding in the Era of Large Models: Dataset, Test Conditions, and Benchmark. arXiv preprint arXiv:2412.04307 (2024)
2024 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.