pith. sign in

arxiv: 2605.03283 · v1 · submitted 2026-05-05 · 📊 stat.ML · cs.AI· cs.LG

On the Spectral Structure and Objective Equivalence of Orthogonal Multilabel Fisher Discriminants

Brian Keith-Norambuena , Juan Bekios-Calfa This is my paper

Pith reviewed 2026-05-07 13:26 UTC · model grok-4.3

classification 📊 stat.ML cs.AIcs.LG
keywords multilabel discriminant analysisFisher linear discriminantorthogonal projectionsscatter matricessubspace estimationminimax ratessub-Gaussian concentration
0
0 comments X

The pith

All four multilabel Fisher objectives become equivalent under the total scatter constraint W^T S_t^{ML} W = I_r.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper unifies the algebraic treatment of four common Fisher objective functions for linear discriminant analysis when labels are multilabel and the projection matrix is required to be orthogonal. It proves that these objectives coincide exactly once the total multilabel scatter matrix is used to normalize the solution, while they generally diverge under a plain Stiefel constraint. The same normalization also yields a clean partition of variance and a bound that relates Euclidean distances in the projected space to Hamming distances in the original label space. On the statistical side the work supplies a finite-sample upper bound on subspace recovery error together with a matching minimax lower bound, showing the estimator is nearly optimal once the gap between the r-th and (r+1)-th eigenvalues is taken into account.

Core claim

Under the normalization W^T S_t^{ML} W = I_r every one of the four Fisher objectives (trace-ratio, ratio-trace, and their within-class and between-class variants) returns the same subspace; the multilabel between-class scatter matrix can have rank strictly larger than C-1; the subspace estimation error admits the finite-sample rate O(k_max sqrt(d log d / n) / gap_r) with a matching Omega(sigma^2 d / (n gap_r)) lower bound under sub-Gaussian noise.

What carries the argument

The multilabel total scatter matrix S_t^{ML} together with the normalization constraint W^T S_t^{ML} W = I_r, which simultaneously enforces orthogonality and makes the four objectives algebraically identical.

If this is right

  • The effective dimension of the discriminant subspace can exceed the classical single-label limit of C-1.
  • Any of the four objective formulations can be substituted for the others without changing the recovered subspace once total-scatter normalization is used.
  • The subspace estimator attains a near-minimax rate, improving only by logarithmic and k_max factors over the information-theoretic lower bound.
  • Projected Euclidean distances concentrate around label Hamming distances with high probability.
  • Regularized versions preserve the same spectral structure even when dimension greatly exceeds sample size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Implementations can safely pick the numerically most stable of the four objectives once the total-scatter normalization is adopted.
  • The derived rates suggest that multilabel LDA remains statistically viable in moderately high-dimensional regimes provided the eigenvalue gap is not too small.
  • The label-distance preservation bound could be used to certify that nearest-neighbor rules in the projected space approximate Hamming-nearest-neighbor rules in label space.
  • The same equivalence argument might extend to kernel or deep-feature versions if the corresponding total scatter matrix is substituted into the constraint.

Load-bearing premise

The observations are generated from a linear label-effect model driven by sub-Gaussian noise.

What would settle it

Numerical checks on data drawn from the linear label-effect model showing that the four objectives return visibly different subspaces when the total-scatter normalization is enforced.

read the original abstract

We provide a unified theoretical analysis of Linear Discriminant Analysis with simultaneous multilabel scatter matrix formulations and Stiefel orthogonality constraints. Our contributions span both algebraic structure and statistical guarantees. On the algebraic side, we characterize the rank of the multilabel between-class scatter matrix, showing that the effective discriminant dimensionality can strictly exceed the classical single-label bound of $C-1$; we establish a multilabel partition of variance and prove that all four Fisher objectives are equivalent under the $W^\top S_t^{ML} W = I_r$ constraint while characterizing their divergence under the Stiefel constraint; and we prove a two-sided label-distance preservation bound relating projected distances to Hamming distances in label space. On the statistical side, we establish a finite-sample $O(k_{\max}\sqrt{d\log d/n}/gap_r)$ bound on the subspace estimation error under sub-Gaussian noise with a matching $\Omega(\sigma^2 d/(n\,gap_r))$ minimax lower bound, establishing a near-minimax-optimal rate (matching up to logarithmic and $k_{\max}$ factors) for multilabel discriminant subspace estimation. We further provide high-probability distance concentration, robustness guarantees under label interactions, and a regularization analysis preserving the spectral structure when $d \gg n$. All results are verified numerically on synthetic data generated from the linear label-effect model, covering both the algebraic identities and the multilabel-specific quantities ($k_{\max}$, $\kappa(S_t^{ML})$, $\|\Gamma/n\|_2$, $\Delta_r$) that govern the statistical bounds. The numerical experiments are designed as a sanity check for the theorems rather than as an empirical benchmark; evaluation on real multilabel datasets is left to future work targeting application-oriented venues.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript provides a unified algebraic and statistical analysis of Linear Discriminant Analysis in the multilabel setting with Stiefel orthogonality constraints. Algebraically, it characterizes the rank of the multilabel between-class scatter matrix (showing effective dimensionality can exceed C-1), establishes a multilabel partition of variance, proves equivalence of four Fisher objectives under the W^T S_t^{ML} W = I_r constraint (with divergence under the pure Stiefel constraint), and gives a two-sided label-distance preservation bound. Statistically, it derives a finite-sample upper bound O(k_max sqrt(d log d / n)/gap_r) on subspace estimation error under sub-Gaussian noise, a matching Omega(sigma^2 d/(n gap_r)) minimax lower bound claimed to be near-minimax optimal up to logs and k_max factors, plus high-probability concentration, robustness to label interactions, and a regularization analysis for d >> n. All results are verified via synthetic experiments from a linear label-effect model.

Significance. If the statistical bounds are reconciled to support the near-minimax claim, the paper would make a solid contribution to multilabel discriminant analysis by extending classical LDA results with explicit algebraic equivalences and finite-sample rates. The combination of objective equivalence under a natural constraint, explicit governing quantities (k_max, gap_r), and both upper/lower bounds (even if rates require clarification) would be useful for theory and practice in high-dimensional multilabel settings. The synthetic verification of algebraic identities and bound quantities is a positive feature.

major comments (1)
  1. [Abstract] Abstract: the claimed near-minimax optimality rests on the finite-sample upper bound O(k_max sqrt(d log d / n)/gap_r) and minimax lower bound Omega(sigma^2 d/(n gap_r)) being comparable up to the stated log and k_max factors for the same subspace error metric. These rates differ by a factor on the order of sqrt(d/n) (ignoring constants and logs); when d >> n the lower bound can exceed the upper bound, which is inconsistent. The manuscript invokes a separate regularization analysis for d >> n but does not reconcile the rates or confirm whether the error metric (e.g., Frobenius norm of W-hat - W or sin-Theta distance) is identical in both results. This is load-bearing for the central statistical claim of near-minimax optimality.
minor comments (2)
  1. [Abstract] The abstract and introduction should explicitly define or reference the precise error metric (e.g., which matrix norm or distance) used for both the upper and lower bounds to allow direct comparison.
  2. The numerical section describes experiments as sanity checks for algebraic identities and quantities like k_max, kappa(S_t^{ML}), and Delta_r; adding a brief statement on the range of parameters tested (e.g., values of n, d, C, label density) would improve reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the thorough review and the insightful comment on the statistical bounds. We provide a point-by-point response below and outline the revisions we will make to address the concern.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claimed near-minimax optimality rests on the finite-sample upper bound O(k_max sqrt(d log d / n)/gap_r) and minimax lower bound Omega(sigma^2 d/(n gap_r)) being comparable up to the stated log and k_max factors for the same subspace error metric. These rates differ by a factor on the order of sqrt(d/n) (ignoring constants and logs); when d >> n the lower bound can exceed the upper bound, which is inconsistent. The manuscript invokes a separate regularization analysis for d >> n but does not reconcile the rates or confirm whether the error metric (e.g., Frobenius norm of W-hat - W or sin-Theta distance) is identical in both results. This is load-bearing for the central statistical claim of near-minimax optimality.

    Authors: We thank the referee for pointing out this important inconsistency in the rate comparison. We agree that the upper bound O(k_max sqrt(d log d / n)/gap_r) and the lower bound Omega(sigma^2 d/(n gap_r)) are not directly comparable across all regimes, as their ratio scales with sqrt(n log d / d) (up to constants). This means the near-minimax claim, as phrased, holds only when this factor is bounded by k_max and logs, i.e., when n is sufficiently large relative to d. For the high-dimensional regime d >> n, we indeed rely on the separate regularization analysis provided in the manuscript, which preserves the spectral structure but has its own rate guarantees not claimed to be minimax there. We confirm that both the upper and lower bounds pertain to the same error metric: the sin-Theta distance between the estimated and population subspaces (equivalently, the Frobenius norm of the difference in projection matrices up to constants). We did not explicitly state the regime of validity for the near-minimax optimality in the abstract. To correct this, we will revise the abstract to specify that the near-minimax rate is achieved in the regime where d log d = O(n), and add a clarifying sentence in the introduction regarding the applicable regimes and the consistency of the error metric. We will also ensure the regularization section is cross-referenced appropriately. These changes will be incorporated in the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained from model assumptions

full rationale

The paper derives algebraic identities (rank of multilabel scatter, objective equivalence under W^T S_t^{ML} W = I_r, label-distance bounds) and statistical rates directly from the linear label-effect model with sub-Gaussian noise. The O(k_max sqrt(d log d/n)/gap_r) upper bound and Omega(sigma^2 d/(n gap_r)) minimax lower bound are expressed in explicit model-derived quantities (gap_r, k_max, sigma) rather than fitted parameters or self-referential definitions. Numerical verification uses synthetic data generated from the same model assumptions as a sanity check, which does not feed back into the claims. No load-bearing self-citations, ansatzes smuggled via prior work, or renamings of known results appear in the derivation chain. The central claims remain independent of their own outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on standard sub-Gaussian concentration and a linear generative model for labels; no new free parameters, ad-hoc entities, or non-standard axioms are introduced in the abstract.

axioms (2)
  • domain assumption Data vectors are sub-Gaussian with parameter sigma
    Invoked to obtain the finite-sample upper bound and matching lower bound.
  • domain assumption Labels act linearly on the feature means
    Used to generate the synthetic data that verifies the algebraic and statistical quantities.

pith-pipeline@v0.9.0 · 5628 in / 1430 out tokens · 65118 ms · 2026-05-07T13:26:41.794537+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages

  1. [1]

    D. H. Foley and J. W. Sammon , title =. IEEE Transactions on Computers , year =

    work page

  2. [2]

    Journal of Machine Learning Research , year =

    Jieping Ye , title =. Journal of Machine Learning Research , year =

    work page

  3. [3]

    Pattern Recognition , year =

    Zhong Jin and Jing-Yu Yang and Zhong-Shan Hu and Zhen Lou , title =. Pattern Recognition , year =

    work page

  4. [4]

    Journal of Machine Learning Research , year =

    Jieping Ye and Tao Xiong , title =. Journal of Machine Learning Research , year =

    work page

  5. [5]

    Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence , year =

    Dijun Luo and Chris Ding and Heng Huang , title =. Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence , year =

    work page

  6. [6]

    SIAM Review , year =

    Thanh Ngo and Mohammed Bellalij and Yousef Saad , title =. SIAM Review , year =

    work page

  7. [7]

    Mathematical Programming , year =

    Li Wang and Lei-Hong Zhang and Ren-Cang Li , title =. Mathematical Programming , year =

    work page

  8. [8]

    Pattern Recognition , year =

    Wai-Ki Ching and Delin Chu and Liao-Yu Liao and Xiaoyan Wang , title =. Pattern Recognition , year =

    work page

  9. [9]

    SIAM Journal on Scientific Computing , year =

    Delin Chu and Li-Zhi Goh , title =. SIAM Journal on Scientific Computing , year =

    work page

  10. [10]

    Proceedings of the European Conference on Computer Vision (ECCV) , year =

    Hua Wang and Chris Ding and Heng Huang , title =. Proceedings of the European Conference on Computer Vision (ECCV) , year =

    work page

  11. [11]

    Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , year =

    Shuiwang Ji and Jieping Ye , title =. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , year =

    work page

  12. [12]

    Pattern Recognition Letters , year =

    Chun-Hee Park and Myoung Lee , title =. Pattern Recognition Letters , year =

    work page

  13. [13]

    Neurocomputing , year =

    Jianhua Xu , title =. Neurocomputing , year =

    work page

  14. [14]

    IEEE Transactions on Cybernetics , year =

    Jianhua Xu and Jiale Liu and Jing Yin and Chunyan Sun , title =. IEEE Transactions on Cybernetics , year =

    work page

  15. [15]

    Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence , year =

    Yin Zhang and Zhi-Hua Zhou , title =. Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence , year =

    work page

  16. [16]

    ACM Transactions on Knowledge Discovery from Data , year =

    Yin Zhang and Zhi-Hua Zhou , title =. ACM Transactions on Knowledge Discovery from Data , year =

    work page

  17. [17]

    Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , year =

    Liang Sun and Shuiwang Ji and Jieping Ye , title =. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , year =

    work page

  18. [18]

    Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (CIARP) , year =

    Juan Bekios-Calfa and Brian Keith , title =. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications (CIARP) , year =

    work page

  19. [19]

    Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , year =

    Shuai Zheng and Chris Ding , title =. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , year =

    work page

  20. [20]

    Neurocomputing , year =

    Li Wang and Lei-Hong Zhang and Chungen Shen and Ren-Cang Li , title =. Neurocomputing , year =

    work page

  21. [21]

    Foundations of Intelligent Systems (ISMIS) , year =

    Chang Liu and Xiaowei Dong and Naijie Gu , title =. Foundations of Intelligent Systems (ISMIS) , year =

    work page

  22. [22]

    The Geometry of Algorithms with Orthogonality Constraints , journal =

    Alan Edelman and Tom. The Geometry of Algorithms with Orthogonality Constraints , journal =. 1998 , volume =

    work page 1998

  23. [23]

    Absil and R

    P.-A. Absil and R. Mahony and R. Sepulchre , title =. 2008 , address =

    work page 2008

  24. [24]

    IEEE Transactions on Knowledge and Data Engineering , year =

    Wissam Siblini and Pascale Kuntz and Frank Meyer , title =. IEEE Transactions on Knowledge and Data Engineering , year =

    work page

  25. [25]

    Chemometrics and Intelligent Laboratory Systems , year =

    Sanli Hou and Patrick Riley , title =. Chemometrics and Intelligent Laboratory Systems , year =

    work page

  26. [26]

    1990 , address =

    Keinosuke Fukunaga , title =. 1990 , address =

    work page 1990

  27. [27]

    R. A. Fisher , title =. Annals of Eugenics , year =

    work page

  28. [28]

    Radhakrishna Rao , title =

    C. Radhakrishna Rao , title =. Journal of the Royal Statistical Society, Series B , year =

    work page

  29. [29]

    Johnson and Joram Lindenstrauss , title =

    William B. Johnson and Joram Lindenstrauss , title =. Contemporary Mathematics , year =

    work page

  30. [30]

    Measuring Statistical Dependence with

    Arthur Gretton and Olivier Bousquet and Alex Smola and Bernhard Sch. Measuring Statistical Dependence with. Proceedings of the 16th International Conference on Algorithmic Learning Theory (ALT) , year =

    work page

  31. [31]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson , title =. 2013 , address =

    work page 2013

  32. [32]

    Proceedings of the National Academy of Sciences , year =

    Ky Fan , title =. Proceedings of the National Academy of Sciences , year =

    work page

  33. [33]

    Marshall and Ingram Olkin and Barry C

    Albert W. Marshall and Ingram Olkin and Barry C. Arnold , title =. 2011 , address =

    work page 2011

  34. [34]

    International Journal of Data Warehousing and Mining , year =

    Grigorios Tsoumakas and Ioannis Katakis , title =. International Journal of Data Warehousing and Mining , year =

    work page

  35. [35]

    Tropp , title =

    Joel A. Tropp , title =. Foundations of Computational Mathematics , year =

    work page

  36. [36]

    G. W. Stewart and Ji-Guang Sun , title =. 1990 , address =

    work page 1990

  37. [37]

    Chandler Davis and W. M. Kahan , title =. SIAM Journal on Numerical Analysis , year =

    work page

  38. [38]

    2018 , address =

    Roman Vershynin , title =. 2018 , address =

    work page 2018

  39. [39]

    D. L. Hanson and F. T. Wright , title =. Annals of Mathematical Statistics , year =

    work page

  40. [40]

    Electronic Communications in Probability , year =

    Mark Rudelson and Roman Vershynin , title =. Electronic Communications in Probability , year =

    work page

  41. [41]

    Samworth , title =

    Yi Yu and Tengyao Wang and Richard J. Samworth , title =. Biometrika , year =

    work page

  42. [42]

    Tony Cai and Anru Zhang , title =

    T. Tony Cai and Anru Zhang , title =. Annals of Statistics , year =

    work page

  43. [43]

    Tsybakov , title =

    Alexandre B. Tsybakov , title =. 2009 , address =

    work page 2009

  44. [44]

    Proceedings of the American Mathematical Society , year =

    Alain Pajor and Nicole Tomczak-Jaegermann , title =. Proceedings of the American Mathematical Society , year =

    work page

  45. [45]

    Metric Entropy of Homogeneous Spaces , journal =

    Stanis. Metric Entropy of Homogeneous Spaces , journal =. 1998 , volume =

    work page 1998

  46. [46]

    Tony Cai and Zongming Ma and Yihong Wu , title =

    T. Tony Cai and Zongming Ma and Yihong Wu , title =. Probability Theory and Related Fields , year =

    work page