Variance-Preserving Orthogonal Selection (VPOS): Greedy Feature Selection via Orthogonal Deflation in PCA Loading Space

Baran Koseoglu; Berrin Yanikoglu

Variance-Preserving Orthogonal Selection (VPOS): Greedy Feature Selection via Orthogonal Deflation in PCA Loading Space

T0 review · 5 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a greedy feature-selection algorithm operating on variance-weighted PCA loadings, with orthogonal deflation after each pick, achieves the lowest reconstruction error on all eight benchmarks, with deflation itself respo

desk verdict VPOS is a clean, incremental GKS/pivoted-QR variant on variance-weighted loadings; the theory is right, but the all-eight MSE claim relies on in-sample d selection and a few loose ends in reporting. read the letter →

arxiv 2607.23198 v3 pith:XKR3VCVH submitted 2026-07-25 cs.LG

Baran Koseoglu , Berrin Yanikoglu This is my paper · ORCID

classification cs.LG

keywords unsupervisedfeatureselectionprincipalcomponentanalysisorthogonaldeflationloadingspacegreedyalgorithmcolumnsubsetreconstructionerrorgeneexpression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes VPOS, a greedy unsupervised feature-selection algorithm that works on a weighted PCA loading matrix rather than raw data. After picking a feature, it projects the loading vectors of all remaining features onto the subspace orthogonal to that feature's direction, forcing the next pick to cover a different part of the covariance structure. The paper claims this deflation step is the primary reason VPOS reconstructs the data better than five baselines on all eight benchmarks, with reconstruction error reduced by 10–73% at the same hyperparameter value. It also proves each deflation step reduces the loading matrix rank by exactly one, which guarantees the selected set spans new dimensions. The practical payoff is a fast, single-hyperparameter method for selecting a small interpretable set of original features, which matters for domains like genomics and finance.

What carries the argument

The key objects are the weighted loading embedding B = V_d Λ^{1/2}_d (each row encodes a feature's contribution to the top-d principal components, with squared row norm equal to the variance that feature carries into that subspace) and the null-space deflation operator P⊥_u = I - uu^T / ||u||^2, applied by right-multiplying B. At each step VPOS picks the feature with the largest residual row norm, then removes that feature's direction from all remaining rows. This is a greedy Gram-Schmidt process on the rows of B, and the paper argues it structurally resembles greedy column-subset-selection algorithms with submodularity-style guarantees, while being cheaper than deflating in sample space by

What would settle it

Take any one of the eight datasets, split it into training and held-out halves, select d by the minimum-MSE rule on the training half only, then compute reconstruction MSE on the held-out half for VPOS and for PCA without deflation at that same d. If the deflation-driven MSE reduction largely disappears or reverses, the paper's central attribution of the gain to deflation would be brought into question.

Watch

Extended reading notes

Core claim

The central discovery is that greedy feature selection in the variance-weighted PCA loading space, with null-space deflation after each pick, systematically avoids redundant picks and achieves the lowest reconstruction MSE on all eight datasets tested. The deflation mechanism is isolated by comparing against PCA without deflation at the same d; the paper attributes 10–73% of the MSE reduction to deflation itself, not to the PCA weighting. The supporting theorem (Proposition 1) states that each deflation step reduces the rank of the loading matrix by exactly one, so after k steps the selected features span k independent dimensions of the d-dimensional principal subspace.

Load-bearing premise

The load-bearing premise is that choosing d by the minimum-MSE rule on the same data used for benchmarking does not inflate VPOS's reported advantage; if tuning and evaluation are not separated, the 'lowest MSE on all eight' claim is partly guaranteed by the selection rule rather than the deflation mechanism.

Editorial extensions

If this is right

If VPOS's claims hold, unsupervised feature selection can become both more accurate and much faster than graph-based methods, with 10–140x speedups at scale.
The 10–73% MSE reduction from deflation at matched d implies that static ranking methods (variance thresholding, PCA loading norms) leave substantial reconstruction performance on the table under multicollinearity.
Rank reduction by exactly one per step guarantees the selected set is linearly independent in the loading space, directly addressing the 'clustering trap' where top-k features all track one latent factor.
The minimum-MSE sweep over 5–6 values of d gives a reproducible, label-free hyperparameter rule applicable to new datasets without supervision.
Because VPOS selects original features rather than linear combinations, its output is directly usable in domains where mixing variables is unacceptable (portfolios, assays, biomarkers).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not separate the d-selection data from the evaluation data; a fair reader should expect the reported advantage to shrink if d were chosen on a held-out set. A testable extension is to compare VPOS with d chosen via cross-validation or a fixed heuristic.
The variance-weighting result (VPOS vs. unweighted CSSP on V_d) suggests that eigenvalue weighting is doing part of the work; one could ablate further by testing unweighted B with deflation to quantify how much of the gain comes from weighting alone.
The method's dependence on the linear PCA loading space limits it to linear redundancy; the authors leave open kernel extensions. A natural test is whether a nonlinear embedding with the same deflation logic preserves the gains on datasets with nonlinear correlations.
The Colon Cancer result hints that in D>>N settings the minimum-MSE rule protects against noise by staying conservative in d; this suggests a practical guideline: cap d well below the rank limit, but the paper's own sensitivity tables show kNN accuracy can peak at a different d than MSE, so users optimizing classification should tune separately.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

VPOS is basically pivoted QR / GKS on the variance-weighted loading matrix B = V_d Λ^{1/2}. The authors are upfront about that lineage, and the actual novelty — weighting loadings by sqrt(eigenvalue) and deflating in loading space — is modest but real. Proposition 1 is correct and easy to check: post-multiplying by the projection kills the selected row, so rank drops by exactly one. That part holds up.

What the paper does well: it's clearly written, the sensitivity analysis of d is genuinely informative, and the inclusion of Colon Cancer (D>>N) and the Multi-Sector synthetic are good stress tests. The PCA-no-deflation ablation is a sensible way to isolate the deflation mechanism. The authors also honestly report cases where VPOS hurts kNN accuracy, which is more than many feature-selection papers do.

The soft spots are mostly empirical. The biggest one: d is chosen on the same reconstruction-MSE metric used for the headline comparison. Section 6.1.4 says best-d is the value minimising reconstruction MSE, then Table 1 uses that best-d. So 'lowest MSE on all eight' is partly an in-sample optimum of the evaluation metric. The comparison to baselines is not fully fair either: VPOS gets its own d tuned on the metric, while Laplacian Score and MCFS are fixed at standard hyperparameters. A hold-out split for d selection, or a report of results across the d sweep, would fix this. Without that, the all-eight claim is not as strong as it sounds.

There are also a few concrete inconsistencies. The abstract says deflation reduces MSE by 10–73%, but the numbers in Table 1 give 9.5% for Colon Cancer and 95.6% for Multi-Sector. And on Digits, k=16 while best-d=15, so the algorithm deflates B to zero after 15 steps and the 16th selected feature is undefined per the paper's own Remark 1. That's a small but real bug in the experimental setup.

No error bars or code are provided, and PFA — a directly related baseline — is never compared. These are fixable, but they matter for a paper whose headline is 'lowest on all eight.'

Overall: the core idea is plausible, the theory is correct but elementary, and the empirical claims are not yet cleanly supported. This deserves a serious referee, but the authors need to redo the evaluation with proper d selection and report the range rather than the cherry-picked point.

Who it's for: anyone working on unsupervised feature selection for high-dimensional, interpretability-sensitive problems (genomics, finance). It's a useful incremental addition, not a breakthrough.

Referee Report

5 major / 5 minor

Summary. The paper proposes VPOS, an unsupervised feature-selection method that operates on the variance-weighted PCA loading matrix B = V_d Lambda_d^{1/2}. At each step it selects the remaining feature with the largest row norm and then right-multiplies B by the orthogonal projection onto the complement of the selected feature's loading vector. The authors prove that each deflation step reduces the rank of B by exactly one, connect the greedy procedure to CSSP and submodular maximization, and report experiments on eight datasets claiming the lowest reconstruction MSE in all cases, with deflation accounting for a 10-73% MSE reduction over PCA without deflation at matched d. The central theoretical Proposition 1 is correct under the stated nonzero condition, but the empirical evaluation has a load-bearing protocol problem: d is selected by minimizing the same reconstruction MSE used for benchmarking, on the same data, with no validation split. There is also a concrete inconsistency in the Digits experiment (k=16 > d=15) and discrepancies between the claimed 10-73% range and the numbers in the tables.

Significance. If the empirical claims survive a cleaner protocol, VPOS would be a simple, interpretable, and fast addition to unsupervised feature selection, particularly for block-correlated data where static variance ranking fails. The rank-reduction proposition is elementary but correctly proved, and the complexity analysis is useful. The authors are also appropriately cautious about the submodularity guarantee, explicitly disclaiming a universal approximation result. However, the current manuscript does not yet provide unbiased support for the headline 'lowest MSE on all eight' because the d-selection rule and evaluation metric are not separated, and several reported numbers conflict with the paper's own summary statistics. No code or data splits are provided, which further limits reproducibility assessment.

major comments (5)

[§6.1.4, §6.4, Table 1] Best-d is selected as the value minimising reconstruction MSE over a sweep on the same data used for the benchmark (Section 6.1.4, Table 4), and the headline result 'lowest reconstruction MSE on all eight' (Section 7) is then reported at this best-d. Because the selection criterion is the evaluation metric, the all-eight dominance is in-sample by construction; no hold-out/validation split is used. This also affects the deflation attribution: the PCA (no deflation) baseline is evaluated at a d tuned specifically to VPOS's MSE objective, so the 10-73% reduction is not an unbiased estimate of the deflation mechanism. Please add an independent validation set for d selection, or report both in-sample and out-of-sample MSE, and show that the advantage survives.
[§6.2, Table 1, Algorithm 1, Remark 1] Digits uses k=16 while the sensitivity sweep selects d=15 (§6.4). This violates the k≤d condition in Remark 1. After d steps B=0, so at iteration 16 every candidate score w_j is zero and the argmax in Eq. (7) is not unique; if the selected u is zero, the update in Eq. (5)/(11) divides by ||u||^2 and is undefined. The paper does not define a tie-break or a rule for k>d. Please either restrict experiments to k≤d, add an explicit handling of the zero-embedding case, or explain why Table 1's configuration is valid.
[Abstract, §7, Tables 1-3] The abstract and conclusion state that deflation accounts for a 10-73% MSE reduction at matched d. The reported tables do not support this range: Colon Cancer gives 1 - 0.2732/0.3020 ≈ 9.5%, HighDim gives 1 - 0.5460/0.5572 ≈ 2.0%, and Multi-Sector gives 1 - 0.0150/0.3401 ≈ 95.6% (from Tables 1-3). The claimed interval omits these values. Please correct the summary statistics, or report the per-dataset deflation benefit in a table and base the abstract on those numbers.
[Remark 1, §5.1] Remark 1 states that rank(B)=min(D,d) and that this holds whenever D≥d, and equates rank d with 'no two distinct features have identical loading representations'. Neither statement is correct in general: with D≥d the rows can lie in a lower-dimensional subspace without any two rows being equal (e.g., rows (1,0),(2,0),(3,0) in R^2), and when d exceeds the covariance rank (e.g., d>N-1 in the centered case) B cannot have rank d. The guarantee 'the selected set spans at least k dimensions' should be stated as conditional on rank(B)≥k, not on D≥d or absence of duplicate rows. Please correct the remark and the associated claim.
[§6.2, §6.3, Tables 1-3] All MSE and kNN results are point estimates without error bars or repeated trials, even though MSE for CIFAR-10 and Colon Cancer is averaged over a random sample of 500 features and kNN uses k-fold CV. Several advantages are small (HighDim MSE 0.5460 vs 0.5572, ~2%), so without uncertainty quantification the claim of 'lowest MSE on all eight' is not fully supported. Please report means and standard deviations (or confidence intervals) over multiple random samples/seeds, especially for the close comparisons.

minor comments (5)

[Abstract vs. §7] The abstract says VPOS runs '10-140x faster' than graph-based methods, while the conclusion says '9-210x'. Table 1 shows MCFS on MNIST at 210x and Laplacian Score on CIFAR-10 at about 9x. Use a single, consistent range based on the reported tables.
[§6.1.3, Baselines] The manuscript states that internal sweeps for Laplacian Score and MCFS hyperparameters were verified but not reported. For reproducibility, include these sweeps in an appendix or state fixed hyperparameters more prominently.
[§3.3, Algorithm 1] The division by ||u||^2 in step 11 is undefined if u=0. Even if the experimental protocol always keeps k≤d, the algorithm should specify a zero-norm guard, especially since Remark 1 explicitly discusses B becoming zero.
[§5.1, Proposition 1] The proof of exact rank reduction would be clearer if it noted that the row space of B intersects the kernel of the projection exactly in span(u). As written, 'u maps to 0' alone does not by itself rule out a rank drop of more than one.
[§6.1.1, Datasets] For CIFAR-10, the text says the full 50,000-sample dataset was not accessible; using the OpenML subset is acceptable, but this limitation should be stated in the main text rather than only in the dataset description.

Circularity Check

2 steps flagged · score 6.0 of 10

Best-d is selected on the same reconstruction-MSE metric used for evaluation, making the all-eight MSE claim and 10–73% deflation attribution in-sample; Digits k>d also leaves the final selection undefined.

fitted input called prediction [Section 3.3 (Selection rule for d); Section 6.1.4; Table 4 caption; Section 7]
"Selection rule for d. We adopt a reproducible, single-criterion rule: sweep d over a representative range (5–6 values spanning the plausible signal dimensionality) and select the d that minimises reconstruction MSE on the VPOS-selected features. This criterion is consistent with VPOS’s design objective of maximising variance coverage. The sensitivity analysis in Section 6.4 reports both MSE and kNN accuracy across the sweep; the minimum-MSE rule is the formal selection criterion, and all VPOS results in Table 1 use this best-d."

By construction, 'best-d' is defined as the sweep value minimizing reconstruction MSE — the exact metric later used as the headline result. Therefore 'VPOS with best-d achieves the lowest reconstruction MSE' is, for the VPOS family, an in-sample restatement of the selection rule rather than an out-of-sample prediction. The comparison against other methods at that d is not fully forced, but the reported best-MSE is not independent of the d-selection criterion: the same data, same metric, and same selected-feature MSE are used both to choose d and to score VPOS. No hold-out or validation split is described.
fitted input called prediction [Section 6.1.3 baseline (ii); Abstract / Section 7 conclusion]
"(ii) PCA (no deflation): select top-k by weighted loading norm ∥b_j∥2, identical to VPOS but omitting the deflation step. Uses the same best-d as VPOS (from the sensitivity analysis), so the comparison directly isolates the contribution of the deflation mechanism at matched d. ... The PCA (no deflation) baseline — run at the same best-d — confirms that the deflation mechanism is the primary contributor, with deflation accounting for a 10–73% MSE reduction at matched d."

Because best-d is chosen to minimize VPOS's own reconstruction MSE, evaluating PCA (no deflation) at that same d compares the baseline at a point selected for VPOS's objective, not at its own optimum. The 10–73% reduction attributed to deflation is therefore conditional on a d-selection rule that is already tuned to VPOS; it does not cleanly isolate the deflation mechanism. The conclusion that 'deflation is the primary driver' is an in-sample contrast at a VPOS-tuned d, not an unbiased estimate of the mechanism's marginal benefit.

full rationale

The paper's theoretical core is self-contained: Proposition 1 follows directly from the rank-(d−1) projection and the fact that the selected row maps to zero; no self-citation or imported uniqueness theorem is load-bearing. The circularity is confined to the empirical evaluation protocol. The d hyperparameter is selected by minimizing reconstruction MSE on the same data and same selected-feature MSE that is then reported in Tables 1–3 (Section 3.3, Section 6.1.4, Table 4 caption). Thus the headline 'lowest reconstruction MSE on all eight' is partly constructed by the selection rule, and the PCA (no deflation) comparison at the same best-d does not isolate deflation. This is a fitted-input-called-prediction pattern, not a fully forced tautology, because VPOS must still beat the other methods at the chosen d and the other methods are not tuned on this metric. A separate correctness inconsistency (Digits k=16 vs. best-d=15; Remark 1 says B^(d+1)=0, so the 16th arg max is undefined absent a tie-break) is flagged per the reviewing rule; it is a correctness risk, not itself a circularity. No self-citation chain is used to support the main claim.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on the choice of d, which is fitted to the evaluation metric. The rank-reduction theorem uses only standard linear algebra under stated non-degeneracy conditions. No new particles, forces, or ad hoc entities are introduced.

free parameters (1)

d (PCA dimension) = Wine 5, Breast Cancer 7, Digits 15, MNIST 50, CIFAR-10 120, Colon Cancer 20, HighDim 20, Multi-Sector 3
Selects which principal components define the loading embedding; chosen as the value minimizing reconstruction MSE on the same evaluation data (Section 6.1.4), so it is fitted to the benchmark rather than independently fixed.

assumptions (4)

domain assumption Features are standardized to zero mean and unit variance before covariance computation (Algorithm 1, line 1).
All variance-based claims hold for the correlation matrix rather than raw covariance; this choice is standard but not neutral, especially for genomics data.
domain assumption The top-d eigendecomposition of the feature covariance is a sufficient statistic for feature redundancy; nonlinear redundancy is out of scope.
Section 2.2 and Section 4 define the loading space as linear; the method cannot detect nonlinear structure, and this limitation is acknowledged.
domain assumption For D > N, estimated top-d eigenvectors are assumed to carry signal while trailing components are noise.
The minimum-MSE d rule and the 'conservative d' argument in Section 6.4 rely on this assumption; it is not proven.
standard math u is nonzero and the loading matrix B has full row rank at each selection step for the exact rank-reduction guarantee (Proposition 1).
The paper itself states the u != 0 caveat. This is a standard linear-algebra condition and is not a free parameter.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Variance-Preserving Orthogonal Selection (VPOS): Greedy Feature Selection via Orthogonal Deflation in PCA Loading Space." pith.science (2026). https://pith.science/paper/XKR3VCVH

@misc{pith2026260723198,
  author       = {Pith},
  title        = {Pith review of: Variance-Preserving Orthogonal Selection (VPOS): Greedy Feature Selection via Orthogonal Deflation in PCA Loading Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKR3VCVH}},
  note         = {Machine review of arXiv:2607.23198}
}

abstract

We present Variance-Preserving Orthogonal Selection (VPOS), an unsupervised feature-selection method that performs sequential orthogonal deflation in the variance-weighted principal component analysis (PCA) loading space $\mathbf{V}_d\mathbf{\Lambda}_d^{1/2}$. After each feature is selected, its loading direction is projected out of all remaining candidates, so subsequent selections cover complementary directions of the rank-$d$ covariance approximation while returning original variables. We establish rank-reduction guarantees and a determinant-growth interpretation, and distinguish VPOS from greedy selection on raw data, unweighted eigenvector pivoting, Principal Feature Analysis (PFA), and Principal Variable Selection (PVS). Experiments enforce $k\leq d$, tune method-specific parameters on validation observations, and evaluate on unseen outer folds. Across seven labelled benchmarks, VPOS improves held-out normalised reconstruction error over matched PCA without deflation on every dataset, with reductions of 1--78%. It obtains the lowest mean reconstruction error on Wine, Breast Cancer, and MNIST and is within 1.7% of the lowest error on CIFAR-10 and HighDim. On CIFAR-10, VPOS is approximately 24$\times$ faster than the closely related PVS baseline while incurring a 1.7% reconstruction gap. These results establish VPOS as an efficient covariance-coverage method, particularly when correlated high-dimensional data must be represented by a small set of identifiable original variables.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.