Pith. sign in

REVIEW 3 major objections 5 minor 3 references

Random Subset Averaging

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Random Subset Averaging provably attains the optimal squared-loss risk among two-round ensembles, even when inner weights are data-dependent.

desk verdict RSA is a genuinely new two-layer random-subset ensemble with strong simulations, but its central optimality theorem leans on an unproved concentration condition for Mallows weights—worth a serious referee, not a desk reject. read the letter →

arxiv 2512.22472 v2 pith:JBAZIEP3 submitted 2025-12-27 stat.ME

Wenhao Cui , Jie Hu , Haitao Zheng This is my paper
classification stat.ME
keywords ensemblelearningrandomsubsetMallowscriterionmodelaveraginghigh-dimensionalregressionmulticollinearityasymptoticoptimalityforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Random Subset Averaging (RSA), an ensemble predictor for high-dimensional linear models with many correlated covariates. RSA generates many candidate models by independently selecting random subsets of covariates, averages the fitted predictions inside groups, and then combines group predictions with a second round of Mallows weighting; the structure resembles a shallow two-layer neural network. The main claim is that the final predictor is asymptotically optimal in squared loss—its loss converges in probability to the best loss among all convex combinations of the group predictors, even when the first-round weights are themselves learned from data. Under an orthogonal design, RSA's minimal risk matches random subset regression at a fixed selection probability and is strictly smaller than nested model averaging when selection probabilities vary by covariate. If the theory holds, RSA is a practical, tuning-robust alternative to variable selection and equal-weighted ensembles that turns multicollinearity into an advantage.

What carries the argument

Binomial random selection matrices R_m^(ℓ) draw independent Bernoulli masks over covariates, producing M candidate models in each of L groups; each candidate prediction is a least-squares projection onto the selected columns. The two-round Mallows weighting—minimizing ||Y − Σ w_m μ̂_m||² plus 2σ² times the effective model dimension—yields convex weights inside each group and again across groups. The proof treats the data-dependent first-round weights as perturbations of deterministic probability limits w^(ℓ) and shows the perturbation vanishes fast enough to keep the final loss within a factor of the oracle loss.

What would settle it

Run RSA with Mallows first-round weights in a high-dimensional correlated design (K growing with N, ρ=0.9) and compute the ratio of its squared loss to the infimum over all second-round convex weights; if the ratio does not converge to 1 as N increases across many replications, the asymptotic optimality claim fails. A direct check would be to estimate the fourth-moment gap E[||ŵ−w||⁴] for Mallows weights under such designs and see whether it decays at the assumed rate.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a randomized two-round ensemble can be made asymptotically optimal in a way that explicitly accommodates data-dependent first-round weights. The RSA estimator builds L groups of M random-subset least-squares fits, aggregates within each group by Mallows model averaging, then combines the L group predictions by another Mallows step. Theorem 2.1 shows that, under moment and growth conditions and a rate condition on the convergence of the first-round weights, the ratio of RSA's squared loss to the optimal risk over all second-round weights converges to 1 in probability. Theorem 2.2 computes the minimal risk under orthogonal covariates and shows that with co

Load-bearing premise

The proof assumes the first-round weights, which are estimated from the data, converge to fixed limits at a fast enough rate; for the recommended weighting this rate is asserted by analogy rather than proved, and the whole optimality guarantee depends on it.

Editorial extensions

If this is right

  • RSA's loss converges in probability to the infimum risk over second-round convex weights, so with large samples its predictions are as accurate as the best aggregation in its class.
  • Under the orthogonal design, RSA with a common selection probability matches random subset regression asymptotically, while covariate-specific probabilities make it strictly better and also beat nested model averaging.
  • The selection-probability parameter acts as a shrinkage factor; a common p implements ridge-like shrinkage, and covariate-specific p shrinks each coefficient by its signal-to-noise ratio.
  • Because all tuning parameters are chosen by cross-validation, RSA needs no prior ordering or relevance knowledge about covariates.
  • In the reported simulations and the factor-forecasting exercise, RSA with cross-validated tuning yields the lowest or near-lowest out-of-sample error, especially under strong covariate correlation, with competitive training error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the asymptotic optimality argument carries beyond Gaussian linear least squares, RSA could be used as a generic random-subset-and-average wrapper for classification, generalized linear models, or neural network prediction; the main obstacle would be verifying the first-round weight convergence in each new setting.
  • The view of selection probabilities as shrinkage suggests a concrete improvement the paper leaves implicit: estimate covariate-specific probabilities from signal-to-noise ratios rather than a single common p.
  • The paper's reported weak spots—small samples with high correlation and very sparse large-sample settings—point to a hybrid design that falls back to fixed-size subsets or selection-based candidates, which could broaden RSA's applicability without changing its core mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Random Subset Averaging (RSA), a two-layer ensemble method for high-dimensional prediction. Candidate models are built from binomial random subsets of covariates; their predictions are first averaged within groups (uniformly or via Mallows weighting) and then combined across groups by a second Mallows-weighted convex aggregation. The theoretical section claims asymptotic optimality of the final predictor under data-dependent first-round weights (Theorem 2.1) and a lower finite-sample risk bound under orthogonal designs (Theorem 2.2). The empirical part presents an extensive simulation study across sparsity levels, correlation structures, and sample sizes, plus a financial return-forecasting application.

Significance. The RSA architecture is a sensible and potentially useful addition to the model-averaging and random-subspace literature. The paper makes a genuine attempt to handle data-dependent first-round weights, which standard MMA theory does not cover. The simulation study is broad, with MCS-based comparisons across many DGPs, and the empirical application is relevant. However, the central theoretical claims are conditional in ways the authors do not fully disclose: Theorem 2.1 assumes, rather than establishes, the concentration rate of Mallows first-round weights, and Theorem 2.2 characterizes an oracle-tuned lower bound rather than the risk of the feasible CV-selected procedure. These gaps are load-bearing for the abstract's claims of asymptotic optimality and lower finite-sample risk.

major comments (3)
  1. [§2.2, Condition 2 of Theorem 2.1; Remark 2.3; Eq. (2.4)] The main optimality result for the recommended Mallows weighting is not proven. Condition 2 of Theorem 2.1 assumes E[||ŵ(ℓ)−w(ℓ)||⁴|X,R] = O(r⁴_{N,M}), and the proof uses this at multiple points (Eqs. A.1, A.6–A.8). Remark 2.3 asserts r_{N,M} = √(M/N) by analogy with inequality-constrained OLS (Liew 1976), but this analogy is not a derivation: the 'regressors' in Eq. (2.4) are P_{X_R^{(ℓ)}_m}Y, which are random, correlated with the noise, and have a random Gram matrix that can be nearly singular under the high-correlation designs (ρ=0.9) that motivate the paper. Moreover, the Mallows objective in Eq. (2.4) contains σ², which is not specified; if σ² is replaced by an estimate, the optimization changes and the Liew rate no longer applies directly. As stated, Theorem 2.1 covers only trivially data-independent weights such as uniform averaging; the asymptotic optimality of the actual two-rou
  2. [§2.3, Theorem 2.2 and its proof in Appendix A] Theorem 2.2 is stated as a bound on ξ_N = inf_{w∈H_L} Ṟ_N(w) under a constant selection probability p, and under covariate-specific probabilities p_j. However, the proof does not take p or p_j as given; it minimizes the risk expression over p (and over η_k for the varying-p case), see the steps 'Taking the first-order condition with respect to p' and 'optimizing with respect to η_k' in the proof of Theorem 2.2. The displayed asymptotic expressions hold only after plugging in the oracle-optimal p and p_j, which depend on the unknown β and σ². Thus Theorem 2.2 is a bound for an infeasible oracle-tuned RSA, not for the CV-selected p used in the simulations. The comparison in Section 2.3 should be explicitly framed as an oracle comparison; otherwise it overstates the attainable finite-sample risk of the implemented method.
  3. [Eqs. (2.4), (2.6) and Section 2.2] The defining Mallows criteria in Eqs. (2.4) and (2.6) require the error variance σ², but the paper never specifies how σ² is estimated. The method description says tuning parameters are selected by CV, yet σ² is not a tuning parameter in the CV grid; it enters the weighting objective directly. If an estimator σ̂² is plugged in, the objective changes, and the convergence-rate argument in Remark 2.3—which treats the Mallows problem as a standard constrained OLS with known penalty—does not automatically transfer. This is not merely a presentation issue: without a specification of σ̂², the recommended algorithm is not fully defined, and the rate condition in Condition 2 of Theorem 2.1 cannot be verified for any feasible implementation.
minor comments (5)
  1. [Tables 1–2 and B.1–B.3] Some table entries run together, e.g., Table 3 shows '2002.002.64' and Table 4 shows '200 1.981.872.30', making it difficult to identify the sample-size/ K columns and the first method entries. Please fix column spacing throughout.
  2. [Abstract and Section 2.3] The abstract and Section 2.3 say RSA has a 'lower finite-sample risk bound' and 'outperforms' nested MA and RPR/RSR. The theoretical support is for oracle-tuned selection probabilities, not for the CV-tuned procedure used in simulations. Qualifying the language in the abstract and theorem summary would avoid overclaiming.
  3. [Section 4, empirical application] The predictors are orthogonalized before applying RSA, and the text then describes the setting as 'low-correlation' (Section 4 and Appendix B.5.2). Since the paper's motivation is strong covariate correlation, reporting results on the original correlated factors, or at least discussing the effect of orthogonalization on the method's premise, would strengthen the empirical illustration.
  4. [Section 3 and Appendix B.1] No code is provided for the simulations, and the CV grid for (p,M,L) used in Sections 3.2–3.4 is not specified (only the empirical grid is given in Appendix B.5.2). Adding code or a precise grid description would improve reproducibility.
  5. [Lemma 2.3] The optimal P for RPR/RSR is derived treating P as continuous, but P is an integer between 1 and K. The statement could note that the minimum is over continuous relaxation; the integer optimum is within O(1) and does not affect the asymptotic comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 2.1 is a conditional convergence proof and Theorem 2.2 is an oracle risk calculation; neither reduces to its own inputs.

full rationale

The central derivation chain is not circular. Theorem 2.1 states assumptions including Condition 2 (E[||ŵ(ℓ)−w(ℓ)||⁴|X,R]=O(r⁴)) and then proves L_N(ŵ)/inf L_N(w)→1 by bounding differences between the data-dependent predictor and its deterministic-weight proxy (Eqs. A.1–A.13). The rate condition is assumed, not derived from the conclusion or from the paper's own fitted values. Remark 2.3 asserts rather than proves that Mallows weights concentrate at rate √(M/N), citing Liew (1976) by analogy; this is an unsupported gap in the applicability of Theorem 2.1 to the recommended procedure, but it is not circular—the theorem remains a conditional statement. Theorem 2.2 computes the minimal risk of the predictor with optimally chosen weights and selection probabilities, using the true β and σ² (e.g., 'optimizing with respect to η_k yields η_k = ...'). This is an infeasible oracle benchmark (the paper explicitly says 'with properly chosen selection probabilities'), so the later comparison to MA/RPR/RSR is a bound rather than a prediction from fitted parameters. There are no self-citations: the references to Zhang (2021), Liew (1976), Peng & Yang (2022) are independent external works, and no uniqueness claim is imported from the authors' prior papers. The simulations are genuine out-of-sample Monte Carlo evidence. I therefore find no circular step; the main correctness risk is the unproved Mallows-weight concentration rate in Condition 2/Remark 2.3, not circularity.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central theoretical claims rest on a standard linear-model setup plus one paper-specific assumption: fast L4 convergence of data-dependent first-round weights. The oracle selection probabilities are free parameters that depend on the target β and σ². No new physical or structural entity is postulated.

free parameters (5)
  • selection probability p = In theory p* = M/(M-1) · Nβᵀβ/(Nβᵀβ + Kσ²) − 1/(2(M-1)); in practice chosen by CV over a grid.
    Controls expected subset size; the optimal theoretical value depends on unknown β and σ², making the finite-sample bound an oracle bound.
  • candidate models per group M = RSA.opt: CV-selected; RSA.fix: M=30.
    Chosen by 5-fold CV; Theorem 2.1 conditions involve M through r_{N,M}, ξ^{-1} L M² → 0.
  • number of groups L = RSA.opt: CV-selected; RSA.fix: L=30.
    Chosen by CV; affects the condition ξ^{-1} L² → 0.
  • noise variance σ² in Mallows criteria = Not specified; presumably estimated, but the estimator is never stated.
    Equations (2.4) and (2.6) require σ²; the paper does not say how it is obtained, which is a practical free parameter.
  • per-covariate selection probabilities p_j (theoretical variant) = η_k = M/(M-1) · Nβ_k²/(Nβ_k² + σ²) − 1/(2(M-1))
    Used only in the theoretical second claim of Theorem 2.2 to claim superiority over RSR/RPR; these depend on the true β_k and are not implemented.
assumptions (7)
  • domain assumption Homoscedastic linear model y_i = x_i'β + e_i with E(e_i|x_i)=0 and E(e_i²|x_i)=σ² (Eq. 2.1).
    All theoretical results and simulations are confined to this model; Theorem 2.1 and Theorem 2.2 depend on it.
  • standard math Fourth moment E(e_i⁴) < ∞ (Theorem 2.1 Condition 1).
    Standard moment condition used in Mallows averaging proofs, e.g. Lemma 2 in Zhang (2021).
  • ad hoc to paper First-round weights converge: E[||ŵ(ℓ)−w(ℓ)||^4 | X,R] = O(r^4_{N,M}) (Theorem 2.1 Condition 2).
    This is the key unproved premise for data-dependent weights; the paper asserts rates for Mallows/parametric weights in Remark 2.3 but does not derive them under general conditions.
  • standard math Lemmas 1–3 of Zhang (2021) and Theorem 1 of Peng & Yang (2022) are used in the proofs.
    The proof of Theorem 2.1 strings together regression-loss concentration lemmas from Zhang (2021); Theorem 2.1 inherits their validity.
  • domain assumption Orthogonal design X'X = N I_K for Theorem 2.2 and Lemma 2.3.
    The finite-sample lower-bound statements and comparisons to MA/RPR/RSR are derived only under this design; correlated designs are handled only by simulation.
  • domain assumption Independent Bernoulli subsets with common p and independent groups; M,L → ∞ in the asymptotic risk formula.
    The risk formula in Theorem 2.2 integrates over independent subset draws and takes both ensemble sizes to infinity.
  • domain assumption K grows as C N^α with α ∈ [0,1].
    Scaling regime stated in the introduction; used to motivate high-dimensional claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Random Subset Averaging." pith.science (2026). https://pith.science/paper/JBAZIEP3

@misc{pith2026251222472,
  author       = {Pith},
  title        = {Pith review of: Random Subset Averaging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBAZIEP3}},
  note         = {Machine review of arXiv:2512.22472}
}
read the original abstract

We propose a new ensemble prediction method, Random Subset Averaging (RSA), tailored for settings with many covariates, particularly in the presence of strong correlations. RSA constructs candidate models via binomial random subset strategy and aggregates their predictions through a two-round weighting scheme, resulting in a structure analogous to a two-layer neural network. All tuning parameters are selected via cross-validation, requiring no prior knowledge of covariate relevance. We establish the asymptotic optimality of RSA under general conditions, allowing the first-round weights to be data-dependent, and demonstrate that RSA achieves a lower finite-sample risk bound under orthogonal design. Simulation studies demonstrate that RSA consistently delivers superior and stable predictive performance across a wide range of sample sizes, dimensional settings, sparsity levels and correlation structures, outperforming conventional model selection and ensemble learning methods. An empirical application to financial return forecasting further illustrates its practical utility.

Figures

Figures reproduced from arXiv: 2512.22472 by the authors.

Figure 1
Figure 1. Graphical Illustration of RSA. Design of the first layer. To fix ideas, let R = diag(r1, . . . , rK) be a random selection matrix, where each rj is an independent Bernoulli random variable with selection probability pj ∈ [0, 1]. The resulting candidate model is given by Y = XRβR + U, where XR represents a randomly selected subset of the original K covariates. The selection probabilities pj can, in principle, vary ac… view at source ↗
Figure 2
Figure 2. Correlation between original factors We compare RSA against several alternatives: Lasso, SCAD, and MCP as selection-based methods; RSR and RF due to their conceptual similarity to RSA in ensemble prediction; and PMA as a representative model averaging approach for high-dimensional settings. Predictive performance is assessed using the daily MSFE. The training dataset consisting of 332 orthogonalized factors but only… view at source ↗
Figure 3
Figure 3. Mean of MSFE for each period [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references

  1. [1]

    Here, we use the relationship that E [ ∥PXR (ℓ) m Y∥ 4 ] ≤E[∥Y∥ 4] =O(N 2), where the inequality is due to the fact thatPXR (ℓ) m is a projection matrix

    To prove this, we have sup w E [ ∥ˆµRSA(w)−˜µ RSA(w)∥ 2|X,R ] ˜RN(w) ≤ξ−1 N sup w E [ ∥ˆµRSA(w)−˜µ RSA(w)∥ 2|X,R ] =ξ−1 N sup w E  ∥ L∑ ℓ=1 w ℓ M∑ m=1 [ ˆw(ℓ) m −w (ℓ) m ] PXR (ℓ) m Y∥ 2|X,R   ≤ξ−1 N L∑ ℓ=1 √ E[∥ˆw(ℓ)−w (ℓ)∥4|X,R] √E  ( M∑ m=1 ∥PXR (ℓ) m Y∥ 2)2   =O(ξ−1 N r2 N,MLMN) =o(1). Here, we use the relationship that E [ ∥PXR (ℓ) m Y∥ 4...

    2021

  2. [2]

    50 We now consider ( ˆµ(ℓ)−µ )⊤( ˆµ(h)−µ )

    +σ 2ηk( MNβ 2 k Nβ 2 k +σ 2 + 1 2) ] ≍ K∑ k=1 Nβ 2 kσ2 Nβ 2 k +σ 2. 50 We now consider ( ˆµ(ℓ)−µ )⊤( ˆµ(h)−µ ) . The calculation follows similar reasoning: E [( ˆµ(ℓ)−µ )⊤( ˆµ(h)−µ )] =E   M∑ j=1 M∑ q=1 w(ℓ) j w(h) q ( X(A (ℓ) j −I)β+ 1 NXA (ℓ) j X⊤e )( X(A (h) q −I)β+ 1 NXA (h) q X⊤e )  =N K∑ k=1 (1−η 2 k)β2 k +σ 2 K∑ k=1 η2 k. Next, optimizingE[(ˆµR...

    2022

  3. [3]

    poly” refers to polynomially decaying coefficients, while “exp

    = P K. As a result, we have: ξRPR N =ξ RSR N =σ 2P 2 K + 1 K 2 K∑ j=1 Nβ 2 j (K−P) 2. The above risk is minimized when setting P = K ∑K j=1Nβ 2 j Kσ 2+ ∑K j=1Nβ 2 j , leading to the desired result. 52 B Additionally numerical results In this section, we provide additional results from our simulation study and empirical analysis. Section B.1 outlines the i...

    2013

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.