REVIEW 3 major objections 5 minor 62 references
The paper claims that the effective rank of the channel covariance of a single feature map, computed in one forward pass through a frozen pretrained encoder, is a per-image, label-free measure of visual richness—and that pruning training da
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
The effective rank of a deep feature map's channel covariance is a label-free per-image richness score that correlates with human complexity and improves data selection for super-resolution and OCR.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection The per-sample use of effective rank as a richness score is new and the r=0.72 human correlation is the real result; the data-selection gains are modest and the paper oversells one of them, with load-bearing layer averaging that needs a sensitivity analysis. the 3 major comments →
ERank in Latent Space as an Image-Complexity and Richness Measure
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that image richness can be read off the eigenspectrum of a deep feature map's channel covariance. For an image and a layer, the feature map is flattened to a spatial-by-channel matrix, its centered channel covariance is formed, and the normalized eigenvalues define a distribution; exponentiating the Shannon entropy of that distribution gives ERank. This is a smooth, scale- and rotation-invariant relaxation of matrix rank that equals the number of channels when channels are fully decorrelated and collapses to one when they are perfectly correlated. The paper argues that because trained encoders are biased toward low-rank representations, a broad single-image spectrum is i
What carries the argument
ERank is the exponential Shannon entropy of the normalized eigenvalue spectrum of the channel covariance of a feature map: with p_i the normalized eigenvalues, ERank = exp(-sum_i p_i log p_i). This single number turns a raw feature map into a per-image complexity score that is invariant to isotropic scaling and channel rotations, continuous and differentiable, and exactly tied to pairwise channel correlation. The implementation averages ERank over four fixed layers of each encoder (ResNet-18 layers 1-4, CLIP ViT-B/32 blocks 2,5,8,11); this layer averaging is load-bearing, because the paper's own corruption probe shows sign flips at the deepest layers under blur.
Load-bearing premise
The load-bearing premise is that a fixed pretrained encoder and a fixed set of layers, averaged together, rank image richness consistently enough that the ordering transfers across datasets and tasks; if the layer choice or encoder changes, the human correlation and data-selection gains may not persist.
What would settle it
Compute ERank on IC9600 using only a single deepest layer (for example, ResNet-18 layer 4), where the paper's own corruption probe shows sign flips under blur: if the correlation with human complexity drops sharply or the super-resolution/OCR pruning gains vanish, then the reported results are an artifact of the specific layer mixture rather than a stable property of ERank.
If this is right
- If ERank is a valid richness measure, any image dataset can be ordered plain-to-rich with one forward pass per image, no labels, and no extra training.
- Data pruning by ERank improves super-resolution when low-ERank (plain) samples are removed and OCR when high-ERank (cluttered) samples are removed, across pruning ratios up to 50%.
- The measure's failure to help classification, segmentation, and denoising delimits where per-sample richness is the right selection signal: task difficulty must be driven by input richness.
- ERank's r = 0.72 agreement with human complexity annotations suggests it can serve as a cheap substitute or complement for human complexity labeling.
Where Pith is reading between the lines
- Because ERank is differentiable, a natural extension the authors note but do not test is to use it as a training objective or regularizer that rewards decorrelated channel spectra, not just as a pruning filter.
- The layer-dependent sign flips in the corruption probe suggest per-layer ERank profiles could serve as a corruption or domain-shift diagnostic: a mid-layer drop with a deep-layer rise under blur is a signature pattern worth exploiting.
- A testable extension is complexity-routed adaptive computation: use ERank to decide how many network blocks an image needs, since plain images should require fewer layers than visually rich ones.
- The paper tests compact task models; whether ERank-based selection gains persist for larger models and larger-scale pretraining remains open and would be the next empirical check.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the effective rank (ERank) of the channel covariance of an image's deep feature map as a per-sample, label-free measure of visual richness. ERank is computed in a single forward pass through a frozen pretrained encoder (ResNet-18 or CLIP ViT-B/32), by averaging per-layer ERank values over a fixed set of layers. The authors characterize its mathematical properties, show that it correlates with human complexity annotations on IC9600 (r = 0.72), and evaluate it as a data-selection criterion. They report that removing low-ERank samples improves super-resolution and removing high-ERank samples improves OCR, while it does not help classification, segmentation, or denoising. The paper includes a corruption probe, comparisons to unsupervised activation energy (UAE), and a discussion of limitations.
Significance. If the central claim holds, ERank provides a cheap, label-free, per-sample complexity signal with a clear interpretation and plausible applications in data pruning, curriculum learning, and adaptive computation. The paper has several strengths: the measure is simple and computationally efficient; no parameters are fitted to human labels or task outcomes, so the human correlation is a genuine external validation; the authors compare against random removal and report negative results on three tasks, which is informative; and the corruption probe attempts to characterize failure modes. The r = 0.72 correlation with human complexity on IC9600 is a strong quantitative result. However, the validity of the measure as a stable richness ranking depends on the specific choice of layers and their averaging, and the paper does not provide the sensitivity analysis needed to establish that stability. The data-selection claims are also overstated relative to the reported results. These issues are fixable but require additional experiments and careful rewriting.
major comments (3)
- [Abstract, §4.4, Table 3] The layer-averaging choice is load-bearing and appears post hoc. Section 4.1 states the layers were 'fixed a priori' and 'not tuned,' yet Section 4.3 uses Table 1's sign flips (ResNet layer 4 blur +166 vs layer 1 −498; CLIP block 11 −368 vs block 2 −490) to say that these results 'motivate averaging ERank across depths.' This is a post hoc justification of a design choice that directly affects every reported correlation and selection result. Moreover, Table 1's rows do not match Section 4.1: Table 1 lists CLIP blocks 2, 6, 11 and ResNet layers 1, 3, 4, while Section 4.1 specifies blocks 2, 5, 8, 11 and layers 1–4. The paper's own Limitations section concedes that 'results may differ under a systematic layer-selection study,' but no such study is provided. Without per-layer IC9600 correlations or a robustness analysis over layers, encoders, and seeds, the r = 0.72 and the selection gains
- [Table 3, Figures 3a/3b] The claim that 'In both tasks ERank beats random removal across pruning budgets' is false as stated. Table 3, IIIT5K OCR pretraining at the 30% budget: ERank (CLIP) gives 0.224 ± 0.067 vs random 0.285 ± 0.027, i.e., worse than random; ResNet-18 ERank is worse than random at every pretraining budget and at most finetune budgets. The abstract and Section 4.4 need to be corrected to report the specific budgets where ERank helps and where it does not, and to avoid the universal 'beats random' phrasing.
- [§4.3, Figure 2] The super-resolution gains over random are numerically small and their statistical significance is not established. For example, at the 10% pretraining budget, random gives 28.850 ± 0.013 and ERank (CLIP) gives 28.874 ± 0.015; at 50%, 28.688 ± 0.004 vs 28.743 ± 0.013. With only three task-model seeds and no paired significance tests, the claim of consistent improvement 'at every budget' is not supported. The authors should report paired differences, confidence intervals, or additional seeds, and temper the conclusion accordingly.
minor comments (5)
- [Figure 3] The layer lists in Table 1 and Section 4.1 are inconsistent. Please unify them and ensure the reported corruption-probe rows correspond to the layers actually used in scoring.
- [§4.3] The caption says curves are distinguished only by color; this is inaccessible for grayscale printing and color-blind readers. Use distinct line styles or markers.
- [§3.1] The p-values in Table 2 are all reported as '<0.05' or '0.28/0.27'; for correlations on 2000 samples, exact p-values or confidence intervals would be more informative. Also, the p-value for r=0.72 on IC9600 is not given, only 'p<0.05' in the figure caption.
- [§4.4] The paragraph 'Properties of ERank' refers to 'the next section' but Section 3.2 does not exist; the properties are presented in Section 3.1 and Appendices A–B. Please fix the cross-reference.
- [§4.4] In the 'Where ERank helps' paragraph, the phrase 'the visually rich images are the hard ones' is slightly confusing; consider clarifying that high ERank correlates with clutter/noise in OCR, which is the reason for removing high-ERank samples.
Circularity Check
No significant circularity: ERank is an unsupervised, parameter-free feature statistic; the human-label and task validations are independent external correlations, and the layer-averaging design is disclosed rather than fitted to the target labels.
full rationale
ERank is computed from a frozen pretrained encoder with no parameters fitted to IC9600 human labels or to task outcomes (Eq. 2; Sec. 4.1). The layer sets are stated as fixed a priori, and the Limitations section explicitly says the layers were 'fixed a priori and not tuned.' The r=0.72 human-complexity correlation is an external benchmark association, not a curve fit. The corruption probe (Table 1) is characterization, and the sentence 'The sign flips at the deepest layers motivate averaging ERank across depths' is a design rationale, not a validation against the same probe; the human-complexity and data-selection results are independent of that probe. There is no load-bearing self-citation chain: ERank is attributed to Roy and Vetterli, and related measures (RankMe, Vendi Score, UAE) are cited to third parties. The reported layer-label inconsistency between Table 1 (CLIP blocks 2, 6, 11; ResNet layers 1, 3, 4) and Sec. 4.1 (blocks 2, 5, 8, 11; layers 1-4) is a reporting/robustness concern, as is the acknowledged lack of a systematic layer-selection study, but neither makes a prediction equivalent to an input by construction. No circular step is established by the paper's own equations or citations.
Axiom & Free-Parameter Ledger
free parameters (4)
- Layer set for ResNet-18
- Layer set for CLIP ViT-B/32
- Averaging across layers
- Encoder choice
axioms (4)
- domain assumption Pretrained encoders exhibit low-rank bias so that broad channel spectra reflect input richness
- domain assumption Human complexity annotations on IC9600 are a valid ground truth
- domain assumption The fixed layer sets are representative of network depth
- standard math Standard spectral and majorization results (eigenvalues, Shannon entropy, Schur-concavity)
Cite this review
Pith. "Pith review of ERank in Latent Space as an Image-Complexity and Richness Measure." pith.science (2026). https://pith.science/paper/FBWR4GXK
@misc{pith2026260719315,
author = {Pith},
title = {Pith review of: ERank in Latent Space as an Image-Complexity and Richness Measure},
year = {2026},
howpublished = {\url{https://pith.science/paper/FBWR4GXK}},
note = {Machine review of arXiv:2607.19315}
}
abstract
We propose the effective rank (ERank) of the channel covariance of an image's deep feature map as a per-sample, label-free measure of visual richness, computed from a single forward pass through a frozen pretrained encoder. ERank counts how many decorrelated channel directions an image activates, and we characterize its properties, including its behavior under noise. Empirically, ERank orders images from plain to visually rich, correlates with codec bitrate, sharpness, and edge density, and correlates with human complexity annotations on IC9600 with $r = 0.72$. As a data-selection criterion, removing low-ERank samples improves super-resolution and removing high-ERank samples improves OCR, in both pretraining and finetuning, while selection does not help classification, segmentation, or denoising. ERank is thus a cheap richness signal, useful exactly when task difficulty is governed by input richness.
Reference graph
Works this paper leans on
-
[1]
15th European Signal Processing Conference (EUSIPCO) , pages =
Roy, Olivier and Vetterli, Martin , title =. 15th European Signal Processing Conference (EUSIPCO) , pages =
-
[2]
and Olkin, Ingram and Arnold, Barry C
Marshall, Albert W. and Olkin, Ingram and Arnold, Barry C. , title =
-
[3]
and Thomas, Joy A
Cover, Thomas M. and Thomas, Joy A. , title =
-
[4]
, title =
Hill, Mark O. , title =. Ecology , volume =
-
[5]
Oikos , volume =
Jost, Lou , title =. Oikos , volume =
-
[6]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[7]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[8]
Sajjadi, Mehdi S. M. and Bachem, Olivier and Lucic, Mario and Bousquet, Olivier and Gelly, Sylvain , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[9]
Improved precision and recall metric for assessing generative models , booktitle =
Kynk. Improved precision and recall metric for assessing generative models , booktitle =
-
[10]
International Conference on Machine Learning (ICML) , year =
Naeem, Muhammad Ferjad and Oh, Seong Joon and Uh, Youngjung and Choi, Yunjey and Yoo, Jaejun , title =. International Conference on Machine Learning (ICML) , year =
-
[11]
Transactions on Machine Learning Research (TMLR) , year =
Friedman, Dan and Dieng, Adji Bousso , title =. Transactions on Machine Learning Research (TMLR) , year =
-
[12]
arXiv preprint arXiv:1708.08227 , year =
Benhenda, Mostapha , title =. arXiv preprint arXiv:1708.08227 , year =
-
[13]
International Conference on Machine Learning (ICML) , year =
Garrido, Quentin and Balestriero, Randall and Najman, Laurent and LeCun, Yann , title =. International Conference on Machine Learning (ICML) , year =
-
[14]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Agrawal, Kumar Krishna and Mondal, Arnab Kumar and Ghosh, Arna and Richards, Blake , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[15]
International Conference on Learning Representations (ICLR) , year =
Jing, Li and Vincent, Pascal and LeCun, Yann and Tian, Yuandong , title =. International Conference on Learning Representations (ICLR) , year =
-
[16]
Papyan, Vardan and Han, X. Y. and Donoho, David L. , title =. Proceedings of the National Academy of Sciences , volume =
-
[17]
Measuring colorfulness in natural images , booktitle =
Hasler, David and S. Measuring colorfulness in natural images , booktitle =
-
[18]
, title =
Pentland, Alex P. , title =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
-
[19]
An Introduction to
Li, Ming and Vit. An Introduction to
-
[20]
International Conference on Learning Representations (ICLR) , year =
Pope, Phillip and Zhu, Chen and Abdelkader, Ahmed and Goldblum, Micah and Goldstein, Tom , title =. International Conference on Learning Representations (ICLR) , year =
-
[21]
and Zoccolan, Davide , title =
Ansuini, Alessio and Laio, Alessandro and Macke, Jakob H. and Zoccolan, Davide , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[22]
, title =
Toneva, Mariya and Sordoni, Alessandro and des Combes, Remi Tachet and Trischler, Adam and Bengio, Yoshua and Gordon, Geoffrey J. , title =. International Conference on Learning Representations (ICLR) , year =
-
[23]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Paul, Mansheej and Ganguli, Surya and Dziugaite, Gintare Karolina , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[24]
International Conference on Learning Representations (ICLR) , year =
Sener, Ozan and Savarese, Silvio , title =. International Conference on Learning Representations (ICLR) , year =
-
[25]
, title =
Sorscher, Ben and Geirhos, Robert and Shekhar, Shashank and Ganguli, Surya and Morcos, Ari S. , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[26]
International Conference on Learning Representations (ICLR) , year =
Xia, Xiaobo and Liu, Jiale and Yu, Jun and Shen, Xu and Han, Bo and Liu, Tongliang , title =. International Conference on Learning Representations (ICLR) , year =
-
[27]
arXiv preprint arXiv:2303.09540 , year =
Abbas, Amro and Tirumala, Kushal and Simig, D. arXiv preprint arXiv:2303.09540 , year =
-
[28]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Arora, Sanjeev and Cohen, Nadav and Hu, Wei and Luo, Yuping , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[29]
and Bhojanapalli, Srinadh and Neyshabur, Behnam and Srebro, Nati , title =
Gunasekar, Suriya and Woodworth, Blake E. and Bhojanapalli, Srinadh and Neyshabur, Behnam and Srebro, Nati , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[30]
Transactions on Machine Learning Research (TMLR) , year =
Huh, Minyoung and Mobahi, Hossein and Zhang, Richard and Cheung, Brian and Agrawal, Pulkit and Isola, Phillip , title =. Transactions on Machine Learning Research (TMLR) , year =
-
[31]
Scientific Reports , volume =
Facco, Elena and d'Errico, Maria and Rodriguez, Alex and Laio, Alessandro , title =. Scientific Reports , volume =
-
[32]
Physica D: Nonlinear Phenomena , volume =
Grassberger, Peter and Procaccia, Itamar , title =. Physica D: Nonlinear Phenomena , volume =
-
[33]
International Conference on Machine Learning (ICML) , year =
Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and Krueger, Gretchen and Sutskever, Ilya , title =. International Conference on Machine Learning (ICML) , year =
-
[34]
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =
He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , title =. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year =
-
[35]
Visual complexity analysis using deep intermediate-layer features , journal =
Elham Saraee and Mona Jalal and Margrit Betke , keywords =. Visual complexity analysis using deep intermediate-layer features , journal =. 2020 , issn =
2020
-
[36]
IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment , volume =
Feng, Tinglei and Zhai, Yingjie and Yang, Jufeng and Liang, Jie and Fan, Deng-Ping and Shao, Ling and Zhang, Jing and Tao, Dacheng , year =. IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment , volume =
-
[37]
International Conference on Database and Expert Systems Applications (DEXA) , pages =
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning , author =. International Conference on Database and Expert Systems Applications (DEXA) , pages =. 2022 , publisher =
2022
-
[38]
The Twelfth International Conference on Learning Representations (ICLR) , year =
Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning , author =. The Twelfth International Conference on Learning Representations (ICLR) , year =
-
[39]
Transactions on Machine Learning Research (TMLR) , year =
Data Pruning and Neural Scaling Laws: Fundamental Limitations of Score-Based Algorithms , author =. Transactions on Machine Learning Research (TMLR) , year =
-
[40]
Has it Trained Yet? NeurIPS 2022 Workshop , year =
Active Learning is a Strong Baseline for Data Subset Selection , author =. Has it Trained Yet? NeurIPS 2022 Workshop , year =
2022
-
[41]
arXiv preprint arXiv:2505.17799 , year =
A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances , author =. arXiv preprint arXiv:2505.17799 , year =
-
[42]
, title =
Shannon, Claude E. , title =. Bell System Technical Journal , volume =
-
[43]
IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
Canny, John , title =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
-
[44]
International Journal of Computer Vision , volume =
Krotkov, Eric , title =. International Journal of Computer Vision , volume =
-
[45]
Pattern Recognition , volume =
Pertuz, Said and Puig, Domenec and Garcia, Miguel Angel , title =. Pattern Recognition , volume =
-
[46]
, title =
Wallace, Gregory K. , title =. IEEE Transactions on Consumer Electronics , volume =
-
[47]
IEEE Transactions on Image Processing , volume =
Mittal, Anish and Moorthy, Anush Krishna and Bovik, Alan Conrad , title =. IEEE Transactions on Image Processing , volume =
-
[48]
Curriculum Learning , booktitle =
Bengio, Yoshua and Louradour, J. Curriculum Learning , booktitle =
-
[49]
Gradient-Based Learning Applied to Document Recognition , journal =
LeCun, Yann and Bottou, L. Gradient-Based Learning Applied to Document Recognition , journal =
-
[50]
Krizhevsky, Alex , title =
-
[51]
IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =
Agustsson, Eirikur and Timofte, Radu , title =. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =
-
[52]
, title =
Mishra, Anand and Alahari, Karteek and Jawahar, C.V. , title =. British Machine Vision Conference (BMVC) , year =
-
[53]
and Winn, John and Zisserman, Andrew , title =
Everingham, Mark and Van Gool, Luc and Williams, Christopher K.I. and Winn, John and Zisserman, Andrew , title =. International Journal of Computer Vision , volume =
-
[54]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Schuhmann, Christoph and Beaumont, Romain and Vencu, Richard and Gordon, Cade and Wightman, Ross and Cherti, Mehdi and Coombes, Theo and Katta, Aarush and Mullis, Clayton and Wortsman, Mitchell and others , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[55]
IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =
Lim, Bee and Son, Sanghyun and Kim, Heewon and Nah, Seungjun and Lee, Kyoung Mu , title =. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages =
-
[56]
IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
Shi, Baoguang and Bai, Xiang and Yao, Cong , title =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
-
[57]
Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks , booktitle =
Graves, Alex and Fern. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks , booktitle =
-
[58]
arXiv preprint arXiv:1706.05587 , year =
Chen, Liang-Chieh and Papandreou, George and Schroff, Florian and Adam, Hartwig , title =. arXiv preprint arXiv:1706.05587 , year =
-
[59]
IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
Han, Yizeng and Huang, Gao and Song, Shiji and Yang, Le and Wang, Honghui and Wang, Yulin , title =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =
-
[60]
, title =
Huang, Gao and Chen, Danlu and Li, Tianhong and Wu, Felix and van der Maaten, Laurens and Weinberger, Kilian Q. , title =. International Conference on Learning Representations (ICLR) , year =
-
[61]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Wang, Yulin and Huang, Rui and Song, Shiji and Huang, Zeyi and Huang, Gao , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[62]
2026 , eprint=
Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models , author=. 2026 , eprint=
2026
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.