Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

Univariate-Guided Sparse Regression

T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UniLasso, a two-stage method that stacks univariate regressions into a non-negative lasso, matches the lasso's test error while producing sparser models whose signs never flip relative to the univariate fits.

desk verdict Practical, honest two-stage sparse regression method with a real proof gap in Theorem 1. read the letter →

arxiv 2501.18360 v9 pith:A6TGS6NL submitted 2025-01-30 stat.ME

Sourav Chatterjee , Trevor Hastie , Robert Tibshirani This is my paper · ORCID
classification stat.ME MSC 62J07
keywords UniLassosparseregressionunivariatescreeningsignconstraintsstackedsupportrecoveryhigh-dimensionalinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniLasso is a two-stage sparse regression method: first each predictor is regressed on the outcome separately (with leave-one-out fits), then a non-negative lasso is fit using those univariate predictions as features. Because the final coefficient of each variable is its univariate slope times a non-negative weight, the fitted model never flips a sign relative to the simple one-at-a-time regressions, and variables with strong marginal effects tend to get larger final coefficients. The paper argues this buys interpretability and stability without sacrificing accuracy: across simulations and real datasets, UniLasso's test error is usually close to the lasso's while its model is substantially sparser. The theoretical centerpiece is a support-recovery and mean-squared-error consistency theorem that holds under sign-agreement between univariate and multivariate coefficients rather than under the lasso's irrepresentability condition. The authors also show the method extends to GLMs, Cox regression, multiclass problems, and the use of external univariate scores.

What carries the argument

The engine is the non-negative lasso fit on leave-one-out univariate predictions. UniLasso minimizes (1/n) Σ_i (y_i - θ0 - Σ_j θ_j (β̂^{-i}_{0j} + β̂^{-i}_j x_{ij}))^2 + λ Σ_j θ_j subject to θ_j ≥ 0, then sets γ̂_j = β̂_j θ̂_j. The leave-one-out step is what makes small-marginal features drop out: when the univariate correlation of y with a feature is below about $\sqrt$(2/n), the LOO fit's correlation with y tends to be negative, so the non-negativity constraint excludes it. For orthonormal designs the method reduces to a closed-form thresholding function sign(β̂_j)(|β̂_j| - λ/|β̂_j|)_+, which shrinks large coefficients less than small ones, between the shapes of lasso and best-subset selection. The theory also uses the identity β_j/γ_j = 1 + Σ_{k∈S\{j}} (γ_k/γ_j) δ_{k,j}, which shows when sign agreement holds (e.g., when all relevant pairwise covariances have signs compatible with those of the coefficients).

What would settle it

Run 100 replicates of the paper's counter-example: n=100, p=20, x1 ~ N(0,1), x2 = x1 + N(0,1), β=(1,-0.5,0,…,0), noise SD 0.5. If UniLasso's test MSE is consistently about twice the lasso's, the sign-agreement condition is demonstrably load-bearing; if the two methods tie, the claim that sign disagreement causes the failure is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a regression estimator built from univariate information alone can recover the true support of a sparse linear model, provided the marginal and joint effects do not disagree in sign. Theorem 1 states: if sign(β_j) = sign(γ_j) for every active predictor j, the population covariance matrix of active predictors is nonsingular with minimum eigenvalue η, the variables and noise have sub-Gaussian tails, and λ satisfies K1 max_{j∉S}|β_j| ≤ λ ≤ K2, then with probability at least 1 - K4 p n $e^{{-K5 n λ^2}}$ UniLasso sets all inactive coefficients to zero and estimates active coefficients within O(λ) of the truth. This is a support-recovery guarantee for the method, and it deliberately avoids any irrepresentability or mutual-incoherence condition linking active and inactive covariates; the lasso would require such a condition. The empirical work shows that the price of the sign agreement is real but detectable: in the one constructed scenario where the condition fails (two positively correlated predictors with opposite coefficient signs), UniLasso's test MSE is roughly double the lasso's, and cross-validation flags the problem.

Load-bearing premise

The load-bearing premise is that for every truly active predictor, the population univariate regression coefficient has the same sign as the true multivariate coefficient; the paper's own counter-example shows a realistic correlated-feature case where this fails and UniLasso's error roughly doubles.

Editorial extensions

If this is right

  • If the sign-agreement condition holds, UniLasso achieves support recovery in settings where lasso-style irrepresentability conditions fail, because Theorem 1 places no restriction on the relationship between active and inactive covariates.
  • Practitioners who value interpretability get models in which every reported coefficient agrees in sign with the corresponding single-variable regression, and the car-price and proteomics examples show these models are sparser and more stable across train/test splits than lasso's.
  • Because the procedure is a stacking meta-learner, it carries over directly to binomial GLMs, Cox proportional hazards, multiclass one-versus-rest classification, and the setting where only external univariate scores (not raw data) are available.
  • In the counter-example where the sign condition fails, cross-validated error identifies the degradation, and the proposed 'polish' post-processing — lasso on the uniLasso residuals — restores lasso-level error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The sparsity threshold the paper identifies for LOO features (marginal correlations below about sqrt(2/n) tend to flip the LOO correlation negative) suggests a cheap pre-screening rule: discard features whose absolute marginal correlation with y is below sqrt(2/n) before fitting, which could speed up UniLasso considerably.
  • The sign-agreement condition is stated for population coefficients and is not directly checkable from data; a practical diagnostic would be to compare univariate and multivariate signs on a validation split and fall back to lasso (or polish) when violations appear, something the paper only hints at via CV.
  • Since all first-stage fits are univariate, UniLasso may systematically miss predictors that are only useful jointly (e.g., interactions), even when each has near-zero marginal correlation; a multivariate extension using group or pair fits would test whether this limitation is as severe as the counter-example suggests.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes uniLasso, a two-stage regression estimator in which each feature is first replaced by its leave-one-out univariate fitted values, a non-negative lasso is then fit to these values, and final coefficients are formed by multiplying the lasso weights by the univariate slopes. This enforces sign agreement with the marginal regressions and tends to produce sparser models. The main theoretical result, Theorem 1, claims support recovery and an O(lambda) coefficient error under a sign-agreement condition, a lower bound on the covariance of active covariates, sub-Gaussian tails, and a lower bound on lambda relative to the inactive marginal coefficients. The paper also gives sufficient and necessary conditions for the sign-agreement condition (Theorems 2 and 3), proves an oracle-type comparison for the unregularized version uniReg (Theorem 4), derives the orthonormal-design shrinkage function, and reports extensive simulations, real data analyses, GLM/Cox extensions, and an R package.

Significance. The contribution is potentially useful: a simple two-stage method with sign preservation, interpretability, and an explicit connection to stacked regression and the adaptive lasso (Proposition 1), plus a public R implementation. The empirical study is unusually honest: it discloses the 'homecourt' setting designed to favor uniLasso, the counter-example where uniLasso's MSE is about double the lasso's, and the high-SNR setting where lasso wins. If Theorem 1 is repaired, the theoretical contribution is a support-recovery guarantee that does not rely on irrepresentability conditions. The main obstacle is the gap in the proof of Theorem 1 described below; because that gap is local and repairable, I view it as requiring major revision rather than rejection.

major comments (2)
  1. [Appendix B, proof of Theorem 1 (after Eq. (26))] The proof defines eta_hat as the smallest eigenvalue of (sigma_hat_jk)_{j,k in S'} with S'=S union {0} and X_{i,0}=1, and then asserts that its population limit eta_0 is 'equal to the minimum eigenvalue eta of the covariance matrix of (X_j)_{j in S}'. This equality is false without centering: for E[X_j]=mu_j, M=E[(1,X_S)^T(1,X_S)] equals the block matrix [[1, mu^T],[mu, Sigma+mu mu^T]], whose smallest eigenvalue can be far below that of Sigma. For example, with X_1=m+N(0,1), Sigma has minimum eigenvalue 1 but eta_0 is approximately 1/(1+m^2), which can be arbitrarily small. Since the subsequent bound divides by eta_hat - K_9(M+M^2)Q_2, the claimed O(lambda) error with constants depending only on C0,C1,C2,eta,M1,M2,M3,|S| is not established. The theorem can likely be repaired by assuming E[X_j]=0 or by centering the covariates (which leaves the univariate slopes and the final coefficients unchanged), or by replacing Assumption 2 with a lower bound on the augmented Gram matrix E[(1,X_S)^T(1,X_S)].
  2. [Section 7, Theorem 1, Assumption 1] The sign-agreement condition sign(gamma_j)=sign(beta_j) for all j in S is a population-level condition that cannot be checked from data. The paper's own counter-example in Section 8 is a simple two-feature correlated model with opposite coefficient signs; there the condition fails and uniLasso's test MSE is roughly twice the lasso's. The paper honestly labels the assumption 'crucial' and proposes a heuristic polish, but the theoretical guarantee gives no data-dependent way to know whether it applies. Please qualify the abstract and introduction accordingly and discuss whether any finite-sample check (beyond Theorem 2, whose delta_jk are also population quantities) can be provided.
minor comments (5)
  1. [Appendix B, beginning of proof of Theorem 1] The proof says 'constants ... may depend only on C0,C1,C2,theta and |S|', but theta is not defined; it should presumably be eta, and M1,M2,M3 should also be listed.
  2. [Appendix B, Corollary 1] Corollary 1 displays 'P(...) >= 2 exp(...)'; the inequality direction is reversed and should be '<= '.
  3. [Appendix B, proof of Theorem 1, inequalities (20)-(22)] The notation beta_hat_{i,j} and beta_hat_{i,k} is used where beta_hat_j^{-i} and beta_hat_k^{-i} are meant; please define or fix the notation.
  4. [Abstract and Section 8] The abstract's claim that uniLasso 'outperforms Lasso in various settings, particularly in terms of sparsity' should be qualified in light of Section 8, where lasso wins in the high-SNR setting and in the counter-example; the consistent finding is comparable MSE with sparser support in several settings.
  5. [Figures 6 and 7] The vertical axis is labeled 'MSE' in all panels even though some panels report misclassification rates; the caption explains this, but the panels themselves should be labeled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UniLasso's derivation is self-contained, with explicit assumptions rather than conclusions and disclosed simulation designs.

full rationale

The paper's central claim, Theorem 1, is a conditional support-recovery and error bound for the proposed UniLasso estimator. Its proof derives concentration inequalities from sub-Gaussian tail assumptions and bounds the probability of selecting inactive variables and of estimating active coefficients within O(λ). The sign-agreement condition γ_j β_j > 0 for j in S is explicitly stated as an assumption and is acknowledged in the text as 'a crucial one'; it is not a restatement of the conclusion. Theorems 2 and 3 provide separate, proved sufficient and necessary conditions for that sign agreement to hold, so the assumption is not smuggled in as a conclusion. The equivalence to a sign-constrained adaptive lasso (Proposition 1) is established by a direct change of variables and is presented as a derived relationship, not assumed. The sparsity explanation in Section 4 and Appendix D is an algebraic analysis of leave-one-out correlations, not a pre-fitted parameter renamed as a prediction. The 'homecourt' simulation is transparently described as designed to mimic the model fit in UniLasso's second stage ('roughly mimics the model fit by uniLasso in its second stage') and is paired with an explicit counter-example where the method fails, so it is a disclosed design choice rather than circular evidence. Self-citations, such as comparison to Hastie et al. (2015) and the use of glmnet, are contextual or algorithmic and are not load-bearing for the theoretical claims; no uniqueness theorem is imported from the authors' prior work. The paper also explicitly flags its own limitation in Section 9.3: 'We have not been able to prove the LOO version of uniReg ... We leave it as an open question.' This is an honest scope limitation, not circularity. A possible proof gap in Appendix B, where the minimum eigenvalue of the augmented Gram matrix is equated to the covariance minimum eigenvalue without a centering assumption, is a correctness concern about the stated theorem, not a circular reduction: the theorem does not assume its own conclusion. Overall, no step in the derivation chain reduces to its inputs by construction, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theoretical support for the central result rests on the sign-consistency assumption (axiom 2), which is not verifiable from data and is the main fragility. The other assumptions are standard in high-dimensional regression theory. No new physical or mathematical entities are introduced.

assumptions (5)
  • domain assumption The true data-generating model is linear: Y = γ0 + Σ_{j∈S} γ_j X_j + ε, with ε independent of the X_j.
    Invoked at the start of Section 7; standard for sparse regression theory.
  • ad hoc to paper For every j∈S, sign(γ_j) = sign(β_j), where β_j = Cov(Y,X_j)/Var(X_j).
    Theorem 1 assumption 1. This is the paper's sign-consistency condition, which the authors call crucial and which fails in their counter-example.
  • domain assumption The covariance matrix of (X_j)_{j∈S} is nonsingular with minimum eigenvalue η.
    Theorem 1 assumption 2; standard condition.
  • domain assumption Y, ε, and each X_j have sub-Gaussian tails: P(|Z|≥t) ≤ C1 e^{-C2 t^2}.
    Theorem 1 assumption 4; standard concentration condition.
  • domain assumption For Theorem 4, features are i.i.d. N_p(0,Σ) and errors are i.i.d. N(0,σ^2).
    Section 9.3, uniReg theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Univariate-Guided Sparse Regression." pith.science (2026). https://pith.science/paper/A6TGS6NL

@misc{pith2026250118360,
  author       = {Pith},
  title        = {Pith review of: Univariate-Guided Sparse Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6TGS6NL}},
  note         = {Machine review of arXiv:2501.18360}
}
read the original abstract

In this paper, we introduce ``UniLasso'' -- a novel statistical method for sparse regression. This two-stage approach preserves the signs of the univariate coefficients and leverages their magnitude. Both of these properties are attractive for stability and interpretation of the model. Through comprehensive simulations and applications to real-world datasets, we demonstrate that UniLasso outperforms Lasso in various settings, particularly in terms of sparsity and model interpretability. We prove asymptotic support recovery and mean-squared error consistency under a set of conditions different from the well-known irrepresentability conditions for the Lasso. Extensions to generalized linear models (GLMs) and Cox regression are also discussed.

Figures

Figures reproduced from arXiv: 2501.18360 by the authors.

Figure 1
Figure 1. Results for homecourt example: CV and test set prediction error. The dashed vertical lines for each method correspond to the models chosen by CV. MSE-lasso MSE-uniLasso Support-lasso Support-uniLasso Mean 1.098 1.077 7.660 4.790 se 0.005 0.005 0.309 0.142 TPR-lasso TPR-uniLasso FPR-lasso FPR-uniLasso Mean 0.737 0.700 0.135 0.025 se 0.014 0.016 0.012 0.004 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Medium-SNR setting: correlation between the response and the univariate LOO features, versus the correlation between the response and the usual univariate (non-LOO) features. The solid line represents equality. the LOO features, but with a positivity constraint on their coefficients. So even though marginally some LOO features are negatively correlated with the response, this does not guarantee that they will be omi… view at source ↗
Figure 3
Figure 3. Results for car prices data over 50 train-test splits. Shown are test set error, size of the chosen model (support), number of sign change violations relative to the univariate signs, and the stability — the average proportion of common features over all model pairs for each method. 6 Analysis of uniLasso with orthogonal features In this section we derive explicit formula for the uniLasso coefficients in the special… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Coefficient paths for lasso and uniLasso, in a simulated example with 10 orthonormal features. The end of the path represents the least squares fit, and in this case also the univariate coefficients. The paths for features with large absolute univariate coefficients lo…
Figure 5
Figure 5. Figure 5: Shrinkage functions for ridge regression, lasso and uniLasso, in a simulated example with 10 orthonormal features. The last expression can be compared with the similar expression for the lasso in this situation: sign(βˆ j )  |βˆ j | − λ  + . (8) [PITH_FULL_IMAGE:fig…
Figure 6
Figure 6. Figure 6: N = 300, p = 1000: test error (Gaussian) or misclassification rate (Binomial), along with support size. 3. Two-class: We set n = 200, p = 500 with a binary target y. The feature covariance within each class is AR(1) with ρ = 0.8, and the first 20 features are shifted i…
Figure 7
Figure 7. Figure 7: shows the corresponding results for the simulated examples with N = 300 and p = 100. MSE lasso uniLasso uniPolish adapt match 62 64 66 68 70 72 62 64 66 68 70 72 62 64 66 68 70 72 62 64 66 68 70 72 62 64 66 68 70 72 low−SNR MSE Support 1 2 5 10 20 Support MSE lasso uni…
Figure 8
Figure 8. Figure 8: LS vs uniReg on regression datasets from the UCI database. All datasets have n > p. The left plot shows the MSE ratio on the log scale. The right panel shows the support sizes of the chosen models. 9.1 Simulated data In [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Real data examples: Blue lines: relative test error (Gaussian) misclassification rate (Bi￾nomial) or deviance (Cox), with support size superimposed in gold. for each feature there are coefficients for each class, so it is not clear how to replace the feature with a LOO…
Figure 10
Figure 10. Figure 10: Results from an experiment examining the use of external data with uniLasso. The initial training set has 300 observations and we generated 600 additional samples. Shown in blue are the test set MSE for the lasso, and uniLasso, both applied just to the training set; u…
Figure 11
Figure 11. Figure 11: UniLasso polish applied to the counter-example problem of Section 8. The left panel shows the uniLasso solution path, with the dashed vertical line indicating the solution chosen by cross-validation. The right panel shows the polish.uniLasso solution path, which start…
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Car price data: ratio of the achieved test error using CV divided by the minimum test error over the λ path. 1. Fit the p univariate GLMS, and produce the linear predictor functions ˆη i j = βˆ 0j + βˆ jxij . Compute the LOO predictions ˆη −i j for the n training obse…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Likelihood-Free Adaptive Bayesian Inference via Nonparametric Distribution Matching

    stat.ME 2025-05 conditional novelty 7.0 of 10

    ABI estimates the distance between posterior distributions via conditional quantile regression and uses adaptive rejection sampling with generative proposals to produce likelihood-free posterior approximations that co...

  2. Statistical Learning for Heterogeneous Treatment Effects: Pretraining, Prognosis, and Prediction

    stat.ML 2025-05 conditional novelty 5.0 of 10

    Pretraining the R-learner by using the outcome model's active set to weight penalties in the CATE lasso reduces error and raises power when prognostic and predictive factors share support.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [1]

    (1995), ‘Better subset selection using the non-negative garotte’,Technometrics37, 738–

    Breiman, L. (1995), ‘Better subset selection using the non-negative garotte’,Technometrics37, 738–

    work page 1995

  2. [754]

    (1996), ‘Stacked regressions’,Machine Learning24, 51–64

    Breiman, L. (1996), ‘Stacked regressions’,Machine Learning24, 51–64. Candes, E. J., Wakin, M. B. & Boyd, S. P. (2008), ‘Enhancing sparsity by reweighted l1 minimiza- tion’,Journal of Fourier Analysis and Applications14(5), 877–905. Hastie, T., Tibshirani, R. & Wainwright, M. (2015),Statistical Learning with Sparsity: The Lasso and Generalizations, Chapman...

    work page 1996

  3. [942]

    (2006), ‘The adaptive lasso and its oracle properties’,Journal of the American Statistical Association101, 1418–1429

    URL:https://doi.org/10.1214/09-AOS729 Zou, H. (2006), ‘The adaptive lasso and its oracle properties’,Journal of the American Statistical Association101, 1418–1429. Zou, H. & Hastie, T. (2005), ‘Regularization and variable selection via the elastic net’,Journal of the Royal Statistical Society Series B.67(2), 301–320. 35

    doi:10.1214/09-aos729 2006

  4. [2010]

    I: Plenary Lectures and Ceremonies Vols

    (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols. II–IV: Invited Lectures’, World Scientific, pp. 1576–1602. Talagrand, M. (2010),Mean field models for spin glasses: Volume I: Basic examples, Springer Science & Business Media. Wolpert, D. (1992), ‘Stacked generalization’,Neural Networks5, 241–259. Zhang, C.-H. (2010), ‘Nearly unbiased variable ...

    work page 2010

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.