{"id":"69a1f463-4ae2-440a-8d47-15eaf93c8c75","arxiv_id":"2501.18360","paper_version":9,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"UniLasso stacks leave-one-out univariate fits through a non-negative lasso, preserving univariate signs and yielding sparser models than lasso with comparable MSE under a sign-consistency condition.","lead":"This paper introduces UniLasso, a two-stage sparse regression method that first fits simple one-variable models and then combines them with a constrained lasso, guaranteeing the final model's signs match the univariate signs. The method often produces much sparser and more interpretable models than the standard lasso with similar prediction error, but it can fail when a variable's marginal and multivariate signs disagree.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof equates the minimum eigenvalue of the intercept-augmented Gram matrix with that of the covariate covariance; without centering this is false, so the central support-recovery theorem is not established as stated.","rationale":"The reader's structured verdict correctly flags the sign-agreement condition as a serious limitation, and the counter-example shows it can fail in realistic correlated settings. However, the more load-bearing problem for the central theoretical claim is the eigenvalue step in the proof of Theorem 1: it conflates the minimum eigenvalue of the intercept-augmented Gram matrix with that of the covariate covariance matrix. This is not a cosmetic issue; it enters directly into the contraction argument that produces the O(λ) error bound, and the constants are claimed to depend only on the listed quantities, none of which control E[X_S]. Without an additional assumption or a corrected argument, Theorem 1 is not established as stated. The paper has real strengths: a practically motivated two-stage method, reproducible software, extensive simulations, and an unusually honest discussion of limitations including the counter-example and the open question about the LOO version of uniReg. These support conditional acceptance rather than rejection: the proof gap is addressable, and the empirical evidence for sparsity and interpretability is independent of Theorem 1. I therefore keep the reader's CONDITIONAL verdict, while noting that the proof repair should be a required condition.","tokens_in":29363,"tokens_out":6218,"duration_ms":62899,"concrete_test":"Re-derive the eigenvalue step in Appendix B analytically. Compute the minimum eigenvalue of M = [[1, μ^T], [μ, Σ + μμ^T]] for a one-dimensional active set with X_1 = m + N(0,1), μ = m, Σ = 1; verify η_0 ≈ 1/(1 + m^2), which is not the covariance eigenvalue η = 1. Then check whether Theorem 1's proof can be repaired by (i) assuming centered covariates, (ii) adding a lower bound on the augmented Gram matrix, or (iii) allowing constants to depend on ||E[X_S]||. If none of these is added, the theorem is not established as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 1, and its proof in Appendix B contains a step that is not valid as written. After defining S' = S ∪ {0} with X_0 ≡ 1, the proof lets η̂ be the smallest eigenvalue of the sample Gram matrix (σ̂_{j,k})_{j,k∈S'} and then states that its population limit η_0 is 'equal to the minimum eigenvalue η of the covariance matrix of (X_j)_{j∈S}'. This equality holds only if E(X_j) = 0 for all j ∈ S, which is not assumed. For uncentered covariates, M = E[(1, X_S)^T(1, X_S)] = [[1, μ^T], [μ, Σ + μμ^T]], where μ = E[X_S], and its minimum eigenvalue is η_0 = min_{u≠0} [u^T Σ u / ((μ^T u)^2 + ||u||^2)]. This can be arbitrarily small even when the minimum eigenvalue of Σ is bounded below by η: for example, with X_1 = m + N(0,1), η = 1 but η_0 ≈ 1/(1 + m^2), which tends to 0 as m grows. The proof's final error bound divides by (η̂ − small terms), so if η_0 is tiny, the claimed O(λ) error and the constants depending only on C_0, C_1, C_2, η, M_1, M_2, M_3, and |S| do not follow. The theorem may be repairable by adding a lower bound on the augmented Gram matrix or by centering the covariates, but as stated the proof is incomplete. This is more load-bearing than the sign agreement assumption, which the authors explicitly acknowledge and provide a diagnostic/polish for; the eigenvalue step is presented as routine and is not flagged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes uniLasso, a two-stage regression estimator in which each feature is first replaced by its leave-one-out univariate fitted values, a non-negative lasso is then fit to these values, and final coefficients are formed by multiplying the lasso weights by the univariate slopes. This enforces sign agreement with the marginal regressions and tends to produce sparser models. The main theoretical result, Theorem 1, claims support recovery and an O(lambda) coefficient error under a sign-agreement condition, a lower bound on the covariance of active covariates, sub-Gaussian tails, and a lower bound on lambda relative to the inactive marginal coefficients. The paper also gives sufficient and necessary conditions for the sign-agreement condition (Theorems 2 and 3), proves an oracle-type comparison for the unregularized version uniReg (Theorem 4), derives the orthonormal-design shrinkage function, and reports extensive simulations, real data analyses, GLM/Cox extensions, and an R package.","tokens_in":29723,"tokens_out":11124,"duration_ms":103676,"significance":"The contribution is potentially useful: a simple two-stage method with sign preservation, interpretability, and an explicit connection to stacked regression and the adaptive lasso (Proposition 1), plus a public R implementation. The empirical study is unusually honest: it discloses the 'homecourt' setting designed to favor uniLasso, the counter-example where uniLasso's MSE is about double the lasso's, and the high-SNR setting where lasso wins. If Theorem 1 is repaired, the theoretical contribution is a support-recovery guarantee that does not rely on irrepresentability conditions. The main obstacle is the gap in the proof of Theorem 1 described below; because that gap is local and repairable, I view it as requiring major revision rather than rejection.","major_comments":[{"comment":"The proof defines eta_hat as the smallest eigenvalue of (sigma_hat_jk)_{j,k in S'} with S'=S union {0} and X_{i,0}=1, and then asserts that its population limit eta_0 is 'equal to the minimum eigenvalue eta of the covariance matrix of (X_j)_{j in S}'. This equality is false without centering: for E[X_j]=mu_j, M=E[(1,X_S)^T(1,X_S)] equals the block matrix [[1, mu^T],[mu, Sigma+mu mu^T]], whose smallest eigenvalue can be far below that of Sigma. For example, with X_1=m+N(0,1), Sigma has minimum eigenvalue 1 but eta_0 is approximately 1/(1+m^2), which can be arbitrarily small. Since the subsequent bound divides by eta_hat - K_9(M+M^2)Q_2, the claimed O(lambda) error with constants depending only on C0,C1,C2,eta,M1,M2,M3,|S| is not established. The theorem can likely be repaired by assuming E[X_j]=0 or by centering the covariates (which leaves the univariate slopes and the final coefficients unchanged), or by replacing Assumption 2 with a lower bound on the augmented Gram matrix E[(1,X_S)^T(1,X_S)].","section":"Appendix B, proof of Theorem 1 (after Eq. (26))"},{"comment":"The sign-agreement condition sign(gamma_j)=sign(beta_j) for all j in S is a population-level condition that cannot be checked from data. The paper's own counter-example in Section 8 is a simple two-feature correlated model with opposite coefficient signs; there the condition fails and uniLasso's test MSE is roughly twice the lasso's. The paper honestly labels the assumption 'crucial' and proposes a heuristic polish, but the theoretical guarantee gives no data-dependent way to know whether it applies. Please qualify the abstract and introduction accordingly and discuss whether any finite-sample check (beyond Theorem 2, whose delta_jk are also population quantities) can be provided.","section":"Section 7, Theorem 1, Assumption 1"}],"minor_comments":[{"comment":"The proof says 'constants ... may depend only on C0,C1,C2,theta and |S|', but theta is not defined; it should presumably be eta, and M1,M2,M3 should also be listed.","section":"Appendix B, beginning of proof of Theorem 1"},{"comment":"Corollary 1 displays 'P(...) >= 2 exp(...)'; the inequality direction is reversed and should be '<= '.","section":"Appendix B, Corollary 1"},{"comment":"The notation beta_hat_{i,j} and beta_hat_{i,k} is used where beta_hat_j^{-i} and beta_hat_k^{-i} are meant; please define or fix the notation.","section":"Appendix B, proof of Theorem 1, inequalities (20)-(22)"},{"comment":"The abstract's claim that uniLasso 'outperforms Lasso in various settings, particularly in terms of sparsity' should be qualified in light of Section 8, where lasso wins in the high-SNR setting and in the counter-example; the consistent finding is comparable MSE with sparser support in several settings.","section":"Abstract and Section 8"},{"comment":"The vertical axis is labeled 'MSE' in all panels even though some panels report misclassification rates; the caption explains this, but the panels themselves should be labeled.","section":"Figures 6 and 7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a strong, honest paper that is close to being publishable. The gap in the proof of Theorem 1 is local and fixable by centering or by changing the eigenvalue condition, so I recommend major revision rather than rejection. The sign-agreement condition is a genuine scope limitation but is disclosed; the paper would benefit from a more qualified abstract. I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: uniLasso is a practical two-stage stacking method with credible empirical support and an unusually honest limitations section; the catch is that Theorem 1, the main theoretical result, is not established as stated due to a false eigenvalue identity in the proof.\n\nThe methodological core is genuinely new: using leave-one-out univariate fits as features in a non-negative lasso yields sign preservation and strong sparsity, and the LOO variant is not in the adaptive-lasso/garotte/stacking literature. Proposition 1 cleanly reduces the non-LOO version to a sign-constrained adaptive lasso with inverse-univariate weights, which is a useful connection. The simulations and real-data sections are thorough, with software on GitHub/CRAN/Python, and the authors deserve credit for disclosing the counter-example and the high-SNR case where lasso wins.\n\nThe soft spot is the proof of Theorem 1. After defining S' = S ∪ {0} with X_0 = 1, the proof states that the population limit of the minimum eigenvalue of the augmented Gram matrix equals the minimum eigenvalue of the covariance matrix of (X_j)_{j∈S}. That is false without centering. With uncentered predictors, the augmented matrix has a large mean-induced block, and its minimum eigenvalue can be arbitrarily small even when the covariance is well-conditioned; for example, if X_1 = m + N(0,1), the covariance eigenvalue is 1 but the augmented minimum eigenvalue is about 1/(1+m²). Since the final error bound divides by this quantity, the claimed O(λ) recovery does not follow. This is load-bearing, not cosmetic; the theorem may be repairable by centering the covariates or assuming a lower bound on the augmented Gram matrix, but as written it is incomplete. The sign-agreement assumption is a separate, explicitly acknowledged limitation, and the paper's diagnostic/polish does address it.\n\nThe abstract's claim that uniLasso 'outperforms lasso' overstates the body's more careful comparison; MSE is often similar with sparser models, but occasionally much worse.\n\nWho will benefit: practitioners and methodologists working on sparse regression and sign-constrained estimation. The empirical comparisons, the connection to adaptive lasso, and the honest failure-mode discussion make this worth a serious referee. I'd recommend conditional acceptance: fix the eigenvalue argument, qualify the abstract, and state the sign-assumption caveats more prominently in the introduction.","headline":"Practical, honest two-stage sparse regression method with a real proof gap in Theorem 1.","tokens_in":30262,"tokens_out":3100,"would_cite":true,"duration_ms":27985,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J07"],"pacs":[],"model":"deepseek-v4-flash","headline":"UniLasso, a two-stage method that stacks univariate regressions into a non-negative lasso, matches the lasso's test error while producing sparser models whose signs never flip relative to the univariate fits.","keywords":["UniLasso","sparse regression","lasso","univariate screening","sign constraints","stacked regression","support recovery","high-dimensional inference"],"falsifier":"Run 100 replicates of the paper's counter-example: n=100, p=20, x1 ~ N(0,1), x2 = x1 + N(0,1), β=(1,-0.5,0,…,0), noise SD 0.5. If UniLasso's test MSE is consistently about twice the lasso's, the sign-agreement condition is demonstrably load-bearing; if the two methods tie, the claim that sign disagreement causes the failure is not supported.","tokens_in":29121,"feed_emoji":"📊","tokens_out":6829,"duration_ms":58934,"temperature":0.7,"pith_summary":"UniLasso is a two-stage sparse regression method: first each predictor is regressed on the outcome separately (with leave-one-out fits), then a non-negative lasso is fit using those univariate predictions as features. Because the final coefficient of each variable is its univariate slope times a non-negative weight, the fitted model never flips a sign relative to the simple one-at-a-time regressions, and variables with strong marginal effects tend to get larger final coefficients. The paper argues this buys interpretability and stability without sacrificing accuracy: across simulations and real datasets, UniLasso's test error is usually close to the lasso's while its model is substantially sparser. The theoretical centerpiece is a support-recovery and mean-squared-error consistency theorem that holds under sign-agreement between univariate and multivariate coefficients rather than under the lasso's irrepresentability condition. The authors also show the method extends to GLMs, Cox regression, multiclass problems, and the use of external univariate scores.","feed_headline":"UniLasso matches lasso with far sparser models","feed_subtitle":"Sign-guided stacking of univariate fits yields comparable test error, much smaller models, and better stability.","key_machinery":"The engine is the non-negative lasso fit on leave-one-out univariate predictions. UniLasso minimizes (1/n) Σ_i (y_i - θ0 - Σ_j θ_j (β̂^{-i}_{0j} + β̂^{-i}_j x_{ij}))^2 + λ Σ_j θ_j subject to θ_j ≥ 0, then sets γ̂_j = β̂_j θ̂_j. The leave-one-out step is what makes small-marginal features drop out: when the univariate correlation of y with a feature is below about $\\sqrt$(2/n), the LOO fit's correlation with y tends to be negative, so the non-negativity constraint excludes it. For orthonormal designs the method reduces to a closed-form thresholding function sign(β̂_j)(|β̂_j| - λ/|β̂_j|)_+, which shrinks large coefficients less than small ones, between the shapes of lasso and best-subset selection. The theory also uses the identity β_j/γ_j = 1 + Σ_{k∈S\\{j}} (γ_k/γ_j) δ_{k,j}, which shows when sign agreement holds (e.g., when all relevant pairwise covariances have signs compatible with those of the coefficients).","core_discovery":"The central claim is that a regression estimator built from univariate information alone can recover the true support of a sparse linear model, provided the marginal and joint effects do not disagree in sign. Theorem 1 states: if sign(β_j) = sign(γ_j) for every active predictor j, the population covariance matrix of active predictors is nonsingular with minimum eigenvalue η, the variables and noise have sub-Gaussian tails, and λ satisfies K1 max_{j∉S}|β_j| ≤ λ ≤ K2, then with probability at least 1 - K4 p n $e^{{-K5 n λ^2}}$ UniLasso sets all inactive coefficients to zero and estimates active coefficients within O(λ) of the truth. This is a support-recovery guarantee for the method, and it deliberately avoids any irrepresentability or mutual-incoherence condition linking active and inactive covariates; the lasso would require such a condition. The empirical work shows that the price of the sign agreement is real but detectable: in the one constructed scenario where the condition fails (two positively correlated predictors with opposite coefficient signs), UniLasso's test MSE is roughly double the lasso's, and cross-validation flags the problem.","pith_inferences":["The sparsity threshold the paper identifies for LOO features (marginal correlations below about sqrt(2/n) tend to flip the LOO correlation negative) suggests a cheap pre-screening rule: discard features whose absolute marginal correlation with y is below sqrt(2/n) before fitting, which could speed up UniLasso considerably.","The sign-agreement condition is stated for population coefficients and is not directly checkable from data; a practical diagnostic would be to compare univariate and multivariate signs on a validation split and fall back to lasso (or polish) when violations appear, something the paper only hints at via CV.","Since all first-stage fits are univariate, UniLasso may systematically miss predictors that are only useful jointly (e.g., interactions), even when each has near-zero marginal correlation; a multivariate extension using group or pair fits would test whether this limitation is as severe as the counter-example suggests."],"forward_implications":["If the sign-agreement condition holds, UniLasso achieves support recovery in settings where lasso-style irrepresentability conditions fail, because Theorem 1 places no restriction on the relationship between active and inactive covariates.","Practitioners who value interpretability get models in which every reported coefficient agrees in sign with the corresponding single-variable regression, and the car-price and proteomics examples show these models are sparser and more stable across train/test splits than lasso's.","Because the procedure is a stacking meta-learner, it carries over directly to binomial GLMs, Cox proportional hazards, multiclass one-versus-rest classification, and the setting where only external univariate scores (not raw data) are available.","In the counter-example where the sign condition fails, cross-validated error identifies the degradation, and the proposed 'polish' post-processing — lasso on the uniLasso residuals — restores lasso-level error."],"supporting_citations":[{"why":"The adaptive lasso whose univariate-weighted, sign-unconstrained version uniLasso is proved equivalent to (Proposition 1), and against which uniLasso is compared.","marker":"Zou (2006)"},{"why":"Stacked regressions, the meta-learner construction that uniLasso instantiates with univariate learners.","marker":"Breiman (1996)"},{"why":"Proposed stacked generalization, the framework in which uniLasso's two-step procedure is situated as a special case.","marker":"Wolpert (1992)"},{"why":"The non-negative garotte, the closest predecessor, from which uniLasso differs by using univariate LOO fits and p>n applicability.","marker":"Breiman (1995)"},{"why":"Sign-constrained least squares for high-dimensional regression, cited as the relation for the non-negativity constraint in step 2.","marker":"Meinshausen (2012)"},{"why":"Theorem 11.3, the lasso support-recovery result that Theorem 1 is compared against, showing the contrast with irrepresentability.","marker":"Hastie et al. (2015)"},{"why":"Approximate leave-one-out cross-validation estimates used for the LOO fits in the GLM/Cox versions of the algorithm.","marker":"Rad & Maleki (2020)"},{"why":"Iterated reweighted l1 minimization, cited as an alternative sparsity-enhancing scheme related to uniLasso's shrinkage shape.","marker":"Candes et al. (2008)"}],"fun_headline_variants":["UniLasso: sparser models, comparable error","Sign-guided regression matches lasso with sparser models","UniLasso: support recovery without irrepresentability","Sparsity without irrepresentability: UniLasso"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that for every truly active predictor, the population univariate regression coefficient has the same sign as the true multivariate coefficient; the paper's own counter-example shows a realistic correlated-feature case where this fails and UniLasso's error roughly doubles.","fun_headline_variants_meta":{"raw":{"variants":["UniLasso: sparser models, comparable error","Sign-guided regression matches lasso with sparser models","UniLasso: support recovery without irrepresentability","Sparsity without irrepresentability: UniLasso"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001163,"raw_usage":{"total_tokens":4782,"prompt_tokens":882,"completion_tokens":3900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3833}},"tokens_in":498,"tokens_out":3900,"duration_ms":26749,"temperature":1.0,"reasoning_tokens":3833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:46:39.515562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run 100 replicates of the paper's counter-example: n=100, p=20, x1 ~ N(0,1), x2 = x1 + N(0,1), β=(1,-0.5,0,…,0), noise SD 0.5. If UniLasso's test MSE is consistently about twice the lasso's, the sign-agreement condition is demonstrably load-bearing; if the two methods tie, the claim that sign disagreement causes the failure is not supported.","supporting_citations":[{"cited_title":"(1996), ‘Stacked regressions’,Machine Learning24, 51–64","cited_arxiv_id":null,"evidence_quote":"Stacked regressions, the meta-learner construction that uniLasso instantiates with univariate learners."},{"cited_title":"(1995), ‘Better subset selection using the non-negative garotte’,Technometrics37, 738–","cited_arxiv_id":null,"evidence_quote":"The non-negative garotte, the closest predecessor, from which uniLasso differs by using univariate LOO fits and p>n applicability."}],"review_version":1}