pith. sign in

arxiv: 2604.15538 · v1 · submitted 2026-04-16 · 📊 stat.ML · cs.LG

PRIM-cipal components analysis

Tianhao Liu , Daniel Andr\'es D\'iaz-Pach\'on , J. Sunil Rao This is my paper

Pith reviewed 2026-05-10 09:25 UTC · model grok-4.3

classification 📊 stat.ML cs.LG
keywords bump huntingprincipal component analysisno free lunch theoremselliptical distributionsPRIMvarianceFrobenius norm
0
0 comments X

The pith

For elliptical distributions, peeling along the smallest principal components maximizes variance and Frobenius norm of the retained region while peeling along the largest minimizes them.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that two bump-hunting approaches based on peeling orthogonal dimensions and keeping inter-quantile regions are optimal in opposite ways for elliptical data. One approach using the smallest principal components maximizes the total variance and the Frobenius norm of the covariance matrix of the kept region. The other using the largest principal components minimizes those same quantities. This establishes a no-free-lunch result in unsupervised settings, meaning neither strategy is universally superior. It matters because it justifies developing PRIM-inspired algorithms that deliberately choose to minimize variance or volume depending on the goal of finding rare or common subgroups.

Core claim

For data following an elliptical distribution, when k orthogonal dimensions are peeled from R^d with d at least k and an inter-quantile region of probability 1 minus alpha is retained in each, the total variance and Frobenius norm reach their maximum if the peeled dimensions are the k smallest principal components and their minimum if they are the k largest principal components. These two choices are scientifically meaningful opposites with no universal winner, which motivates the creation of PRIM-based bump-hunting algorithms that operate either by minimizing variance or by minimizing volume.

What carries the argument

Peeling k orthogonal dimensions from R^d and retaining inter-quantile regions of probability 1-alpha per dimension, which extremizes total variance and Frobenius norm depending on selection of the k smallest or k largest principal components.

If this is right

  • Two equally optimal bump-hunting strategies exist that are exact opposites with no universal winner.
  • PRIM-based algorithms can be built to minimize variance by peeling leading principal components or to minimize volume.
  • On data such as Fashion-MNIST, peeling the largest principal components captures multiplicity while peeling the smallest isolates popular styles.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Analysts could run both peeling directions on the same data to surface complementary structures such as both rare and common patterns.
  • The symmetry result may point toward analogous trade-offs in other methods that use orthogonal directions for unsupervised subgroup search.
  • The optimality may hold approximately for data distributions that are close to elliptical.

Load-bearing premise

The data follows an elliptical distribution.

What would settle it

An elliptical dataset where the total variance of the retained inter-quantile region after peeling the k smallest principal components is not strictly larger than after peeling any other set of k orthogonal dimensions would refute the maximality claim.

Figures

Figures reproduced from arXiv: 2604.15538 by Daniel Andr\'es D\'iaz-Pach\'on, J. Sunil Rao, Tianhao Liu.

Figure 1
Figure 1. Figure 1: Level set of a 2-dimensional elliptical distribution. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: T-shirt example. Left) peeling principal directions. Right) peeling [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Log-scale scree plot for the trouser class. Red band: principal region [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Trouser class with naive pettiest selection (raw tail components, no log spectral gap). Left) principal subset. Right) pettiest subset. The pettiest subset is visually indistinguishable from a uniform sample, with no discernible structure separating it from the full data. A. The role of log spectral gap selection A key practical question in applying Algorithms 1 and 2 is how to identify the pettiest compon… view at source ↗
Figure 6
Figure 6. Figure 6: Shirt example. Left) peeling principal directions. Right) peeling [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Sandal example. Left) peeling principal directions. Right) peeling [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 12
Figure 12. Figure 12: Sneaker example. Left) peeling principal directions. Right) peeling [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 11
Figure 11. Figure 11: Bag example. Left) peeling principal directions. Right) peeling [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
read the original abstract

Supervised No Free Lunch Theorems (NFLTs) are well studied, yet unsupervised NFLTs remain underexplored. For elliptical distributions, we prove that there exist two equally optimal, scientifically meaningful bump-hunting strategies that are exact opposites, with no universal winner. Specifically, peeling $k$ orthogonal dimensions from $\mathbb{R}^d$ ($d \ge k$), retaining an inter-quantile region of probability $1-\alpha$ per peeled dimension, maximizes total variance and Frobenius norm when the $k$ smallest principal components (called pettiest components) are selected, and minimizes them when the selected dimensions are the $k$ leading principal components. These optima inspire PRIM-based bump-hunting algorithms either by minimizing variance or by minimizing volume, thereby motivating an NFLT. We test our results on the Fashion-MNIST database, showing that peeling the largest principal components captures multiplicity, while peeling the smallest principal components isolates popular styles.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that for elliptical distributions, there exist two equally optimal but opposite PRIM-based bump-hunting strategies in R^d (d >= k): peeling k orthogonal dimensions while retaining an inter-quantile region of probability 1-alpha per dimension maximizes total variance and Frobenius norm when the k smallest principal components (pettiest components) are chosen, and minimizes them when the k leading principal components are chosen. This establishes an unsupervised No Free Lunch Theorem with no universal winner, inspiring variance-minimizing or volume-minimizing algorithms, which are tested on Fashion-MNIST to show that peeling largest PCs captures multiplicity while smallest PCs isolate popular styles.

Significance. If the central derivation holds, the result provides a clean theoretical trade-off between two scientifically meaningful unsupervised strategies, explicitly showing that variance-maximizing and volume-minimizing peeling are exact opposites under the stated conditions. This contributes to the underexplored area of unsupervised NFLTs and supplies a principled basis for choosing between PRIM variants in high-dimensional settings. The Fashion-MNIST demonstration illustrates the practical distinction, though the strong distributional assumption limits immediate broad applicability.

major comments (2)
  1. [Proof of the main theorem (abstract and theoretical development)] The proof of optimality (referenced in the abstract and presumably detailed in the theoretical sections): the claim that the two strategies are exact opposites for total variance and Frobenius norm of the retained region relies on the elliptical distribution assumption to guarantee that principal components are uncorrelated and that marginal quantiles interact with the quadratic form in the required way. The manuscript must explicitly identify the steps in the derivation where ellipticity is invoked and confirm that the result does not reduce to fitted parameters or circular definitions, as the property need not hold outside this class.
  2. [Empirical validation section] Fashion-MNIST experiment: the application demonstrates the algorithms but supplies no diagnostic (e.g., Mardia's test or QQ-plot against elliptical contours) to verify whether the data satisfy the elliptical distribution required for the theoretical max/min ordering to apply. This leaves the connection between the proved result and the motivating example unanchored.
minor comments (2)
  1. [Notation and setup] Clarify the precise definition of 'pettiest components' and the exact construction of the inter-quantile region (product of marginal central intervals) in the notation section to avoid ambiguity when d > k.
  2. [Abstract and introduction] The abstract states the result holds 'for elliptical distributions'; ensure this qualifier is repeated in the statement of the main theorem and in the discussion of the NFLT motivation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight important clarifications needed around the elliptical assumption and its link to the experiments. We address each point below and have prepared a revised manuscript incorporating the requested changes.

read point-by-point responses

  1. Referee: [Proof of the main theorem (abstract and theoretical development)] The proof of optimality (referenced in the abstract and presumably detailed in the theoretical sections): the claim that the two strategies are exact opposites for total variance and Frobenius norm of the retained region relies on the elliptical distribution assumption to guarantee that principal components are uncorrelated and that marginal quantiles interact with the quadratic form in the required way. The manuscript must explicitly identify the steps in the derivation where ellipticity is invoked and confirm that the result does not reduce to fitted parameters or circular definitions, as the property need not hold outside this class.

    Authors: We agree that the proof relies critically on ellipticity and have revised Section 3 to label each invocation explicitly. Ellipticity is used in two places: (1) to establish that the principal components are uncorrelated (via the property that the covariance matrix is a scalar multiple of the identity after rotation to principal axes), and (2) to ensure that the retained inter-quantile region preserves the quadratic form structure needed for the total variance and Frobenius-norm calculations. The derivation starts from the population definition of elliptical distributions and the spectral theorem; it does not depend on sample estimates or fitted parameters and is therefore not circular. We have added a remark clarifying that the exact max/min ordering is specific to the elliptical class and does not hold for arbitrary distributions. revision: yes

  2. Referee: [Empirical validation section] Fashion-MNIST experiment: the application demonstrates the algorithms but supplies no diagnostic (e.g., Mardia's test or QQ-plot against elliptical contours) to verify whether the data satisfy the elliptical distribution required for the theoretical max/min ordering to apply. This leaves the connection between the proved result and the motivating example unanchored.

    Authors: We acknowledge that Fashion-MNIST does not satisfy ellipticity exactly and that the experiment is therefore illustrative rather than a direct verification of the theorem. In the revision we have added an explicit caveat in Section 4 stating that the theoretical ordering applies only under the elliptical assumption, while the Fashion-MNIST results demonstrate the qualitative behavior of the two PRIM variants (multiplicity capture versus isolation of popular styles) on real data. We also include a brief Mardia skewness test on the leading principal components, which indicates moderate departure from normality but does not alter the observed algorithmic distinction. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is a stated theorem under elliptical assumption with independent content.

full rationale

The paper claims to prove, for elliptical distributions, that peeling the k smallest principal components maximizes total variance and Frobenius norm of the retained inter-quantile region while peeling the k largest minimizes them. This is presented as a mathematical result relying on properties of elliptical symmetry (uncorrelated principal components and specific marginal quantile behavior under the quadratic form). No equations or steps reduce by construction to fitted parameters, self-definitions, or prior self-citations as load-bearing premises. The Fashion-MNIST section is described as a demonstration of the algorithms, not as the source of the optimality claim. The elliptical assumption is explicitly required and external to the derivation itself, satisfying the criteria for a self-contained proof without circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption of elliptical distributions and standard mathematical properties of principal components and inter-quantile regions; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption The underlying data distribution is elliptical.
    Invoked to establish that the two peeling strategies are equally optimal opposites for variance and norm.

pith-pipeline@v0.9.0 · 5464 in / 1177 out tokens · 51029 ms · 2026-05-10T09:25:28.582565+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

  1. [1]

    The famine of forte: Few search problems greatly favor your algorithm,

    G. D. Monta ˜nez, “The famine of forte: Few search problems greatly favor your algorithm,”2017 IEEE International Conference on Systems, 11 Fig. 10. Ankle boot example. Left) peeling principal directions. Right) peeling pettiest directions Fig. 11. Bag example. Left) peeling principal directions. Right) peeling pettiest directions Man, and Cybernetics (SM...

    work page 2017

  2. [2]

    Why machine learning works,

    ——, “Why machine learning works,” Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, PA, May 2017

    work page 2017

  3. [3]

    The Futility of Bias-Free Learning and Search,

    G. D. Monta ˜nez, J. Hayase, J. Lauw, D. Macias, A. Trikha, and J. Vendemiatti, “The Futility of Bias-Free Learning and Search,” in2nd Australasian Joint Conference on Artificial Intelligence (AI 2019), J. Liu and J. Bailey, Eds. Cham: Springer, 2019, pp. 277–288

    work page 2019

  4. [4]

    No Free Lunch Theorems for Search,

    D. H. Wolpert and W. G. MacReady, “No Free Lunch Theorems for Search,” Santa Fe Institute, Tech. Rep. SFI-TR-95-02-010, 1995

    work page 1995

  5. [5]

    No Free Lunch Theorems for Optimization,

    ——, “No Free Lunch Theorems for Optimization,”IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, 1997

    work page 1997

  6. [6]

    The supervised learning no-free-lunch theorems,

    D. H. Wolpert, “The supervised learning no-free-lunch theorems,” inSoft Computing and Industry, R. Roy, M. K ¨oppen, S. Ovaska, T. Furuhashi, Fig. 12. Sneaker example. Left) peeling principal directions. Right) peeling pettiest directions Fig. 13. Trouser example. Left) peeling principal directions. Right) peeling pettiest directions and F. Hoffmann, Eds....

    work page 2002

  7. [7]

    What is important about the No Free Lunch theorems?

    ——, “What is important about the No Free Lunch theorems?” inBlack Box Optimization, Machine Learning and No-Free Lunch Theorems, P. M. Pardalos, V . Rasskazova, and M. N. Vrahatis, Eds. Springer, 2021

    work page 2021

  8. [8]

    K. V . Mardia, J. T. Kent, and J. M. Bibby,Multivariate Analysis. Academic Press, 1979

    work page 1979

  9. [9]

    High Di- mensional Mode Hunting Using Pettiest Component Analysis,

    T. Liu, D. A. D ´ıaz-Pach´on, J. S. Rao, and J.-E. Dazard, “High Di- mensional Mode Hunting Using Pettiest Component Analysis,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4637–4649, April 2023

    work page 2023

  10. [10]

    2011, Neural Comput., 23, 1661, 10.1162/NECO\_a\_00142

    K. Sando and H. Hino, “Modal principal component analysis,”Neural Computation, vol. 32, no. 10, pp. 1901–1935, 2020. [Online]. Available: 12 https://doi.org/10.1162/neco a 01308

    work page doi:10.1162/neco 1901

  11. [11]

    Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms,

    X. Han, K. Rasul, and R. V ollgraf, “Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms,”arXiv, 2017

    work page 2017

  12. [12]

    A characterization of ellipti- cal distributions and some optimality properties of principal components for functional data,

    G. Boente, M. S. Barrera, and D. E. Tyler, “A characterization of ellipti- cal distributions and some optimality properties of principal components for functional data,”Journal of Multivariate Analysis, vol. 131, pp. 254– 264, 2014

    work page 2014

  13. [13]

    K.-T. Fang, S. Kotz, and K. W. Ng,Symmetric Multivariate and Related Distributions. New York: Chapman and Hall/CRC, 1990

    work page 1990

  14. [14]

    Bump hunting in high-dimensional data,

    J. H. Friedman and N. I. Fisher, “Bump hunting in high-dimensional data,”Statistics and Computing, vol. 9, pp. 123–143, 1999

    work page 1999

  15. [15]

    PRIM Analysis,

    W. Polonik and Z. Wang, “PRIM Analysis,”Journal of Multivariate Analysis, vol. 101, no. 3, pp. 525–540, 2010

    work page 2010

  16. [16]

    Local Sparse Bump Hunting,

    J.-E. Dazard and J. S. Rao, “Local Sparse Bump Hunting,”Journal of Computational and Graphical Statistics, vol. 19, no. 4, pp. 900–929, 2010

    work page 2010

  17. [17]

    Breiman, J

    L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone,Classification and Regression Trees, ser. The Wadsworth statistics/probability series. Boca Raton: Chapman and Hall/CRC, 1984

    work page 1984

  18. [18]

    Sparse principal components analysis,

    H. Zou, T. Hastie, and R. Tibshirani, “Sparse principal components analysis,”Journal of Computational and Graphical Statistics, vol. 15, no. 2, pp. 265–286, 2006

    work page 2006

  19. [19]

    Local Sparse Bump Hunting reveals molecular heterogeneity of colon tumors,

    J.-E. Dazard, J. S. Rao, and S. Markowitz, “Local Sparse Bump Hunting reveals molecular heterogeneity of colon tumors,”Statistics in Medicine, vol. 31, no. 11-12, pp. 1203–1220, 2012. [Online]. Available: https://doi.org/10.1002/sim.4389

    work page doi:10.1002/sim.4389 2012

  20. [20]

    Unsupervised Bump Hunting Using Principal Components,

    D. A. D ´ıaz-Pach´on, J.-E. Dazard, and J. S. Rao, “Unsupervised Bump Hunting Using Principal Components,” inBig and Complex Data Analysis: Methodologies and Applications, S. E. Ahmed, Ed. Cham: Springer, 2017, pp. 325–345

    work page 2017

  21. [21]

    Generalized active information: Extensions to unbounded domains,

    D. A. D ´ıaz-Pach´on and R. J. Marks II, “Generalized active information: Extensions to unbounded domains,”BIO-Complexity, vol. 2020, no. 3, pp. 1–6, 2020

    work page 2020

  22. [22]

    Hypothesis testing with active information,

    D. A. D ´ıaz-Pach´on, J. P. S ´aenz, and J. S. Rao, “Hypothesis testing with active information,”Statistics & Probability Letters, vol. 161, p. 108742, 2020

    work page 2020

  23. [23]

    Assessing, testing and estimating the amount of fine-tuning by means of active information,

    D. A. D ´ıaz-Pach´on and O. H ¨ossjer, “Assessing, testing and estimating the amount of fine-tuning by means of active information,”Entropy, vol. 24, no. 10, p. 1323, 2022

    work page 2022

  24. [24]

    A formal framework for knowledge acquisition: Going beyond machine learning,

    O. H ¨ossjer, D. A. D ´ıaz-Pach´on, and J. S. Rao, “A formal framework for knowledge acquisition: Going beyond machine learning,”Entropy, vol. 24, no. 10, p. 1469, 2022

    work page 2022

  25. [25]

    Correcting prevalence estimation for biased sampling with testing errors,

    L. Zhou, D. A. D ´ıaz-Pach´on, C. Zhao, J. S. Rao, and O. H ¨ossjer, “Correcting prevalence estimation for biased sampling with testing errors,”Statistics in Medicine, vol. 42, no. 26, pp. 4713–4737, 2023

    work page 2023

  26. [26]

    Is it possible to know cosmological fine-tuning?

    D. A. D ´ıaz-Pach´on, O. H ¨ossjer, and C. Mathew, “Is it possible to know cosmological fine-tuning?”The Astrophysical Journal Supplement Series, vol. 271, no. 2, p. 56, April 2024

    work page 2024

  27. [27]

    An Information Theoretic Approach to Prevalence Estimation and Missing Data,

    O. H ¨ossjer, D. A. D´ıaz-Pach´on, Z. Chen, and J. S. Rao, “An Information Theoretic Approach to Prevalence Estimation and Missing Data,”IEEE Transactions on Information Theory, vol. 70, no. 5, pp. 3567–3582, 2024

    work page 2024

  28. [28]

    Statistical learning does not always entail knowledge,

    D. A. D ´ıaz-Pach´on, R. Gallegos, O. H ¨ossjer, and J. S. Rao, “Statistical learning does not always entail knowledge,”Bayesian Analysis, 2025

    work page 2025

  29. [29]

    Conserved active information,

    Y . Chen and D. A. D´ıaz-Pach´on, “Conserved active information,”Under review, 2026

    work page 2026

  30. [30]

    Mode hunting through active information,

    D. A. D ´ıaz-Pach´on, J. P. S ´aenz, J. S. Rao, and J.-E. Dazard, “Mode hunting through active information,”Applied Stochastic Models in Business and Industry, vol. 35, no. 2, pp. 376–393, 2019

    work page 2019

  31. [31]

    Eigenvalue Ratio Test for the Number of Factors,

    S. C. Ahn and A. R. Horenstein, “Eigenvalue Ratio Test for the Number of Factors,”Econometrica, vol. 81, no. 3, pp. 1203–1227, 2013

    work page 2013

  32. [32]

    Vershynin,High-Dimensional Probability, 2nd ed

    R. Vershynin,High-Dimensional Probability, 2nd ed. Cambridge: Cambridge University Press, March 2026. Tianhao Liureceived his BS in Physics from Nankai University, China, in 2019, and a MS in Biostatistics from University of Miami, in 2020. He is currently working towards a PhD in Biostatistics at the University of Miami. His research interests are mainly...

    work page 2026