REVIEW 4 major objections 5 minor 52 references
X-Factor: Quality Is a Dataset-Intrinsic Property
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A training set's quality is an intrinsic property of the data—not a joint effect of model, size, or class balance—and it largely predicts how well any architecture will perform.
desk verdict A careful single-dataset experiment with a real cross-model correlation, but the headline claim of architecture-independence rests on excluding the one architecture that disagrees, and the paper needs tightening before the generalization is trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the generation of paired random and 'selected' subsets that spread out achievable performance. Random class-balanced subsets of MNIST cluster so tightly that cross-model correlations are invisible, so the authors introduce four RMSD-based selection rules—uniqueness, global mean, other mean, and own mean—each applied in positive and negative directions, to create 2,000 additional subsets whose performance ranges much more widely. Performance is measured as test error, converted to $Z$-scores within each model and subset size, and then compared across models by Pearson $R^2$ and across classes by Spearman's $\rho$. The RMSD distance (root-mean-square pixel difference between images) is the similarity measure that drives selection without requiring any pretrained probe model.
What would settle it
Train the same controlled class-balanced subsets from several very different datasets—for example natural images, medical images, and text—across the same ten architectures, and compute the pairwise $R^2$ of $Z$-scores across models. If the mean $R^2$ falls well below the roughly 0.82 observed here (or becomes negative) for any of those datasets, the claim that quality is a general intrinsic dataset property would be refuted; if the correlation persists across all of them, the claim would be strongly supported.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that classifier performance is largely determined by which training subset is used, not by which model is trained on it. When errors on 2,500 MNIST subsets are converted to $Z$-scores and compared across ten architectures, pairwise $R^2$ values are high—mean 0.82 excluding the very weak Gaussian naive Bayes baseline—even between deep networks and classical models such as random forests and support-vector machines. Because every subset is perfectly class balanced and sizes span five fixed values, neither size nor class balance can explain the correlation; because the models are architecturally diverse, the correlation is not a quirk of any one model family. The authors further show that the pattern holds at the level of individual digit classes, leading them to propose that dataset quality is an emergent property of the quality of the constituent classes. They conclude that 'quality does exist as a property that is largely but not completely intrinsic to the training set.'
Load-bearing premise
The load-bearing premise is that MNIST subsets are representative of datasets in general; the paper itself states that it uses only a single dataset, so if the cross-model correlation does not replicate on other datasets, the claim that quality is a general intrinsic dataset property would not be supported.
Editorial extensions
If this is right
- A dataset that is good for one architecture will tend to be good for others, so dataset evaluation can be done once rather than re-done for every new model.
- Residual variance left after size, class balance, and architecture is mostly explained by dataset identity, so scaling-law models of performance could incorporate a dataset-quality term.
- Size and class balance are not sufficient to characterize a training set; two datasets matched on both can still differ systematically in difficulty.
- Since quality appears to emerge from constituent classes, curating or modifying datasets at the class level may be an effective lever for overall quality.
- The weak Gaussian naive Bayes exception suggests that a model must have enough expressiveness to reveal dataset quality; quality may be invisible to overly simple models.
Reading between the lines
- If dataset quality is largely intrinsic, then benchmark comparisons between models trained on the same fixed datasets partly reflect the dataset's intrinsic difficulty rather than model merit; controlling for quality could change how leaderboard differences are interpreted.
- A model-free quantitative measure of quality may be discoverable from a dataset's internal statistics alone, which would let practitioners estimate quality before training any model; the paper does not yet propose such a measure.
- The near-identical non-normal distribution of random-subset residuals across all models and sizes hints at a universal fluctuation law for sampling noise; if confirmed on other datasets, it would provide a null model for ordinary dataset-quality variation.
- A direct test of additivity is to combine two high-quality subsets and measure whether the combination is additive, sub-additive, or interfering—the paper lists this as open future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper tests whether training-set quality is an intrinsic property of datasets, independent of dataset size, class balance, and model architecture. Using subsets of MNIST, the authors generate class-balanced random and "selected" subsets via four RMSD-based strategies, train ten models (random forest, SVC, GNB, five MLPs, and two ResNets), and compute pairwise R2 values between models' performance Z-scores. They report a mean pairwise R2 of 0.82 excluding GNB, interpret this as evidence for dataset-intrinsic quality, and explore class-level correlations. The paper acknowledges limitations: only MNIST classification is studied and no quantitative definition of quality is proposed.
Significance. If correct, the finding has substantial practical implications: dataset quality could be a transferable property across model architectures, meaning evaluations of datasets on one model may inform others, independent of size and class balance. The experimental design is a strength: class balance is controlled, subset sizes are matched, selection strategies are based on image statistics (RMSD) rather than model performance, and confidence intervals are bootstrapped. The main empirical result—high cross-model correlations on MNIST subsets—is credible, and the paper's careful control of confounding factors makes the correlation an empirical discovery rather than a circular artifact. However, the generality of the "intrinsic quality" claim is limited by the single dataset, the undefined construct, and the GNB exception, which the paper handles by post hoc exclusion.
major comments (4)
- [Abstract and Section 3] The abstract reports R2=0.79, Figure 1 states R2=0.81, and Section 3 reports a mean R2 of 0.82 (95% CI 0.80-0.84) "excluding GNB." The paper never reports the mean pairwise R2 over all ten models including GNB, nor explains the discrepancy between these three numbers. Because the headline statistic is the paper's main quantitative result, the manuscript should report both the all-model and GNB-excluded means and state which values appear in the abstract and figure.
- [Section 3] The claim that quality is independent of model architecture is load-bearing, yet one of the ten models, Gaussian naive Bayes (GNB), shows low pairwise R2 with all other models. The paper excludes GNB because it is "too simple" (Section 3), but no pre-specified capacity or performance threshold is given; this is a post hoc exclusion. The authors should either report results with GNB included, provide a quantitative criterion for exclusion (e.g., based on the variance or reliability of GNB's Z-scores), or explicitly qualify the architecture-independence claim to exclude low-capacity models.
- [Section 4.1 and Abstract] The paper explicitly notes in Section 4.1 that "the present work is limited by inclusion of only a single dataset (MNIST) and studying only classification tasks," but the abstract and title assert quality as a general property of datasets. The empirical evidence supports the claim only for MNIST subsets. The manuscript should either add at least a second dataset to demonstrate generality or temper the claim to the studied setting.
- [Section 4.1] The paper states that "we have not defined what quality is" and that no quantitative measure of quality is proposed. The operationalization used is the cross-model correlation of performance residuals, which measures agreement between models rather than quality per se. Under this operationalization, the conclusion that "quality does exist as a property" conflates an empirical regularity (cross-model agreement) with a reified construct. The discussion should define quality operationally (e.g., as the shared variance across models) or use more cautious language.
minor comments (5)
- [Section 3 and Figure 3] The Kolmogorov-Smirnov p-value is given as p<10^-5 in the Figure 3 caption and p<10^-6 in the text; please standardize.
- [Section 2.1] "Refered" should be "referred."
- [Appendix A.2] The subsection header "dOwn mean" should be "d: Own mean."
- [Section 4] The phrase "if a dataset results in especially high performance on model" should be "on a model."
- [General] The paper does not provide a data or code availability statement; for reproducibility, consider including a link to the code and data or explicitly stating that they are available upon request.
Circularity Check
No significant circularity: the cross-architecture correlation is an empirical result of a self-contained experiment, and the cited prior work is not load-bearing.
full rationale
The paper's central result is an empirical correlation: subsets selected by image-statistics strategies (RMSD-based, independent of any model's performance) produce Z-scored errors that agree across nine of ten architectures (mean pairwise R2 = 0.82 excluding GNB). Nothing in the construction forces this agreement; the selection rules are defined on pixel distances, not on model outputs, and the Z-score normalization only removes per-model/per-size main effects rather than imposing cross-model rank agreement. The 'quality' construct is an interpretation of the observed common variance, explicitly left unmeasured (Section 4.1 states no quantitative quality metric is proposed or tested), so the conclusion is an inductive inference with stated limitations, not a definitional equivalence. The self-citation [18] motivates the hypothesis and offers an entropic metric, but the current paper neither fits nor validates that metric and lists its absence as a limitation; thus the citation is background, not load-bearing. The post hoc exclusion of GNB and the abstract/body R2 discrepancy are statistical-robustness concerns, not circular reductions, and the paper itself acknowledges GNB as an exception and tempers the conclusion to 'largely but not completely intrinsic.' Remaining limitations (single dataset MNIST, selection strategies mostly worsening performance) bound external generalization but do not make the derivation circular.
Assumptions & free parameters
assumptions (4)
- domain assumption MNIST handwritten digit classification is representative of classification tasks and datasets generally.
- domain assumption Test accuracy on the fixed MNIST test set is a valid measure of dataset quality.
- domain assumption The 10 model architectures span the space of model architectures.
- domain assumption Selection strategies based on pixel RMSD do not introduce a model-specific confound.
invented entities (1)
-
Dataset intrinsic quality (the 'X-factor')
Cite this review
Pith. "Pith review of X-Factor: Quality Is a Dataset-Intrinsic Property." pith.science (2026). https://pith.science/paper/6CGH4FEI
@misc{pith2026250522813,
author = {Pith},
title = {Pith review of: X-Factor: Quality Is a Dataset-Intrinsic Property},
year = {2026},
howpublished = {\url{https://pith.science/paper/6CGH4FEI}},
note = {Machine review of arXiv:2505.22813}
}
abstract
In the universal quest to optimize machine-learning classifiers, three factors -- model architecture, dataset size, and class balance -- have been shown to influence test-time performance but do not fully account for it. Previously, evidence was presented for an additional factor that can be referred to as dataset quality, but it was unclear whether this was actually a joint property of the dataset and the model architecture, or an intrinsic property of the dataset itself. If quality is truly dataset-intrinsic and independent of model architecture, dataset size, and class balance, then the same datasets should perform better (or worse) regardless of these other factors. To test this hypothesis, here we create thousands of datasets, each controlled for size and class balance, and use them to train classifiers with a wide range of architectures, from random forests and support-vector machines to deep networks. We find that classifier performance correlates strongly by subset across architectures ($R^2=0.79$), supporting quality as an intrinsic property of datasets independent of dataset size and class balance and of model architecture. Digging deeper, we find that dataset quality appears to be an emergent property of something more fundamental: the quality of datasets' constituent classes. Thus, quality joins size, class balance, and model architecture as an independent correlate of performance and a separate target for optimizing machine-learning-based classification.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad et al.,Deep learning scaling is predictable, empirically,1712.00409
-
[2]
S. Ahmad and G. Tesauro,Scaling and generalization in neural networks: A case study, in Advances in Neural Information Processing Systems, D. Touretzky, ed., vol. 1, Morgan-Kaufmann, 1988, https://api.semanticscholar.org/CorpusID:15615035
work page 1988
- [3]
-
[4]
A. Althnian, D. AlSaeed, H. Al-Baity, A. Samha, A. B. Dris, N. Alzakari et al.,Impact of Dataset Size on Classification Performance: An Empirical Evaluation in the Medical Domain, Applied Sciences11(2021) 796
work page 2021
-
[5]
E. Caballero, K. Gupta, I. Rish and D. Krueger,Broken neural scaling laws,2210.14891
- [6]
-
[7]
H. S. Seung, H. Sompolinsky and N. Tishby,Statistical mechanics of learning from examples, Phys. Rev. A45(1992) 6056
work page 1992
- [8]
Show all 52 references
-
[9]
Maloney, D
A. Maloney, D. A. Roberts and J. Sully,A Solvable Model of Neural Scaling Laws, 2210.16859
-
[10]
E. J. Michaud, Z. Liu, U. Girit and M. Tegmark,The quantization model of neural scaling, 2303.13506
-
[11]
H. Su, Z. Tian, X. Shen and X. Cai,Unraveling the mystery of scaling laws: Part i, 2403.06563
-
[12]
Zhang,Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit,2405.19398
Z. Zhang,Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit,2405.19398
-
[13]
Bordelon, A
B. Bordelon, A. Atanasov and C. Pehlevan,A dynamical model of neural scaling laws, 2402.01092
-
[14]
Atanasov, J
A. Atanasov, J. A. Zavatone-Veth and C. Pehlevan,Scaling and renormalization in high-dimensional regression,2405.00592
-
[15]
Levi,A simple model of inference scaling laws,2410.16377
N. Levi,A simple model of inference scaling laws,2410.16377
-
[16]
Brill,Neural scaling laws rooted in the data distribution,2412.07942
A. Brill,Neural scaling laws rooted in the data distribution,2412.07942
-
[17]
Riley,Three pitfalls to avoid in machine learning,Nature572(2019) 27
P. Riley,Three pitfalls to avoid in machine learning,Nature572(2019) 27
2019
-
[18]
Couch, R
J. Couch, R. Arnaout and R. Arnaout,Beyond size and class balance: Alpha as a new dataset quality metric for deep learning,2407.15724. 9
-
[19]
Leinster and C
T. Leinster and C. A. Cobbold,Measuring diversity: the importance of species similarity, Ecology93(2012) 477
2012
-
[20]
Reeve, T
R. Reeve, T. Leinster, C. A. Cobbold, J. Thompson, N. Brummitt, S. N. Mitchell et al.,How to partition diversity,1404.6520
-
[21]
Leinster,Entropy and diversity: The axiomatic approach,2012.02113
T. Leinster,Entropy and diversity: The axiomatic approach,2012.02113
2012 arXiv
-
[22]
Houlsby, F
N. Houlsby, F. Huszár, Z. Ghahramani and M. Lengyel,Bayesian Active Learning for Classification and Preference Learning,1112.5745
-
[23]
Y . Gal, R. Islam and Z. Ghahramani,Deep Bayesian Active Learning with Image Data, 1703.02910
-
[24]
P. Ren, Y . Xiao, X. Chang, P.-Y . Huang, Z. Li, B. B. Gupta et al.,A Survey of Deep Active Learning,2009.00236
2009 arXiv
-
[25]
Kadota, H
T. Kadota, H. Hayashi, R. Bise, K. Tanaka and S. Uchida,Deep Bayesian Active-Learning-to-Rank for Endoscopic Image Data,2208.03020
-
[26]
Kirsch, S
A. Kirsch, S. Farquhar, P. Atighehchian, A. Jesson, F. Branchaud-Charron and Y . Gal, Stochastic Batch Acquisition for Deep Active Learning,2106.12059
-
[27]
Kirsch and Y
A. Kirsch and Y . Gal,Unifying Approaches in Data Subset Selection via Fisher Information and Information-Theoretic Quantities,2208.00549
-
[28]
Mindermann, J
S. Mindermann, J. Brauner, M. Razzak, M. Sharma, A. Kirsch, W. Xu et al.,Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt,2206.07137
-
[29]
Jesson, P
A. Jesson, P. Tigas, J. van Amersfoort, A. Kirsch, U. Shalit and Y . Gal,Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data, 2111.02275
-
[30]
Huang, Y
Y . Huang, Y . Mabrouk, G. Gompper and B. Sabass,Sparse inference and active learning of stochastic differential equations from data,2203.11010
-
[31]
C. Riis, F. N. Antunes, F. B. Hüttel, C. L. Azevedo and F. C. Pereira,Bayesian Active Learning with Fully Bayesian Gaussian Processes,2205.10186
-
[32]
Ziatdinov, Y
M. Ziatdinov, Y . Liu, K. Kelley, R. Vasudevan and S. V . Kalinin,Bayesian Active Learning for Scanning Probe Microscopy: from Gaussian Processes to Hypothesis Learning,2205.15458
-
[33]
Khosravani and S
M. Khosravani and S. Zilles,Actively Learning Deep Neural Networks with Uncertainty Sampling Based on Sum-Product Networks,2206.09798
-
[34]
Chinn, R
E. Chinn, R. Arora, R. Arnaout and R. Arnaout,Enrich: Exploiting image similarity to maximize efficient machine learning in medical imaging,Journal of the American Medical Informatics Association30(2023) 1079 [medrxiv.org/content/10.1101/2021.05.22.21257645]
2023 doi
-
[35]
Chinn, R
E. Chinn, R. Arora, R. Arnaout and R. Arnaout,ENRICHing medical imaging training sets enables more efficient machine learning,Journal of the American Medical Informatics Association: JAMIA30(2023) 1079
2023
-
[36]
T. Xie, J. Zhu, G. Ma, M. Lin, W. Chen, W. Yang et al.,Structural-Entropy-Based Sample Selection for Efficient and Effective Learning,arXiv e-prints(2024) arXiv:2410.02268 [2410.02268]
2024 arXiv
-
[37]
Sorscher, R
B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli and A. S. Morcos,Beyond neural scaling laws: beating power law scaling via data pruning,arXiv e-prints(2022) arXiv:2206.14486 [2206.14486]
2022 arXiv
-
[38]
Maharana, P
A. Maharana, P. Yadav and M. Bansal,D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning,arXiv e-prints(2023) arXiv:2310.07931 [2310.07931]
2023 arXiv
-
[39]
Har-Peled and A
S. Har-Peled and A. Kushal,Smaller Coresets for k-Median and k-Means Clustering,Discrete & Computational Geometry37(2007) 3
2007
-
[40]
Killamsetty, X
K. Killamsetty, X. Zhao, F. Chen and R. Iyer,RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning,2106.07760
-
[41]
C. Guo, B. Zhao and Y . Bai,DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning,2204.08499. 10
-
[42]
B. A. Griffin, J. Marks and J. J. Corso,Zero-Shot Coreset Selection: Efficient Pruning for Unlabeled Data,2411.15349
-
[43]
H. Chen, J. Peng, K. Tang, X. Geng and X. Yang,Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimization,2504.14200
-
[44]
Zheng, E
H. Zheng, E. Tsai, Y . Lu, J. Sun, B. R. Bartoldson, B. Kailkhura et al.,ELFS: Label-Free Coreset Selection with Proxy Training Dynamics,2406.04273
-
[45]
K. He, X. Zhang, S. Ren and J. Sun,Deep residual learning for image recognition, 1512.03385
-
[46]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel et al.,Scikit-learn: Machine learning in Python,Journal of Machine Learning Research12(2011) 2825
2011
-
[47]
Paszke, S
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan et al.,Pytorch: An imperative style, high-performance deep learning library,1912.01703
1912 arXiv
-
[48]
LeCun, C
Y . LeCun, C. Cortes, C. Burges et al.,Mnist handwritten digit database, 1998,
1998
-
[49]
D. P. Kingma and J. Ba,Adam: A method for stochastic optimization,1412.6980
-
[50]
Nguyen, R
P. Nguyen, R. Arora, E. D. Hill, J. Braun, A. Morgan, L. M. Quintana et al., greylock: A python package for measuring the composition of complex datasets,2401.00102
-
[51]
Friedman and A
D. Friedman and A. Bousso Dieng,The Vendi Score: A Diversity Evaluation Metric for Machine Learning,arXiv e-prints(2022) arXiv:2210.02410 [2210.02410]
2022 arXiv
-
[52]
look least like
S. Marcel and Y . Rodriguez,Torchvision the machine-vision package of torch, inProceedings of the 18th ACM International Conference on Multimedia, MM ’10, (New York, NY , USA), p. 1485–1488, Association for Computing Machinery, 2010, DOI. 11 A Appendix A.1 Model details Random...
2010
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.