REVIEW 3 major objections 4 minor 42 references
Learning Latent Factors from Diversified Projections and its Applications to Over-Estimated and Weak Factors
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Estimating latent factors by pre-chosen cross-sectional weighted averages is valid even when the working number of factors exceeds the true number, so factor-augmented inference no longer requires consistently counting factors.
desk verdict The over-estimation robustness result is a genuine contribution, but the advertised finite-T guarantee has a real proof gap and one application is asserted without proof. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the affine transformation matrix $H = (1/N)W'B$, which links the diversified projection $\hat f_t = H f_t + (1/N)W'u_t$ to the true latent factors. The argument's core is Proposition A.1: when $R > r$ the gram matrix $(1/T)\hat F'\hat F$ is invertible but its inverse is only of order $O_P(N)$, while $H'((1/T)\hat F'\hat F)^{-1}$ remains well behaved and $H'((1/T)\hat F'\hat F)^{-1}H$ converges to the generalized inverse of its population analogue, so all downstream projections $P_{\hat F}$ behave as if the factor count were correct. The second load-bearing piece is Assumption 2.2, which requires $\mathrm{rank}(H) = r$ and $\nu_{\min}(H) \gg N^{-1/2}$ with $\nu_{\max}(H) \leq C\nu_{\min}(H)$, meaning the user-supplied weights must be sufficiently correlated with all $r$ columns of the loading matrix while still diversifying away the idiosyncratic noise.
What would settle it
Take a two-factor model in which the second factor's loadings are exactly orthogonal to every column of $W$ (for instance, loadings that alternate in sign across units while $W$ is constant); then $\nu_{\min}(H) = 0$, so the projection error bound in Theorem 2.1 diverges rather than vanishes, and a simulation should show $\|P_{\hat F}M - P_F\|$ failing to converge as $N$ grows.
Extended reading notes
Core claim
Under Assumptions 2.1–2.4, Theorem 2.1 establishes that for every bounded $R \geq r$ the projection error satisfies $\|P_{\hat F}M - P_F\| = O_P(N^{-1/2}\nu_{\min}(H)^{-1})$, so the linear space spanned by the diversified-factor estimates asymptotically contains the linear space of the true factors, with $T$ either finite or growing. The estimator's clean identity $\hat f_t = H f_t + (1/N)W'u_t$ reduces the estimation problem to an affine transformation $H = (1/N)W'B$ plus a diversifiable noise term, avoiding eigenvector analysis entirely. The paper then proves that the same $R \geq r$ robustness carries through to factor-augmented forecasting, high-dimensional post-selection inference (including the $r = 0$ case with no factors at all), sparse thresholding estimation of the idiosyncratic covariance, and a test of whether observed factors span the latent factor space.
Load-bearing premise
The load-bearing premise is that the user can supply a set of weights that are independent of the noise yet strongly enough correlated with every one of the true factor loadings, in the sense that the smallest nonzero singular value of the matrix $H = W'B/N$ does not decay faster than $N^{-1/2}$; if any true factor is nearly orthogonal to all the weights, it gets averaged away and the whole construction fails.
Editorial extensions
If this is right
- Out-of-sample forecasts from factor-augmented regressions achieve the rate $O_P(T^{-1/2} + N^{-1/2}\nu_{\min}^{-1})$ without a consistent estimator of $r$, for any bounded $R \geq r$.
- Post-selection inference on a treatment effect in a high-dimensional factor-augmented model is asymptotically normal with valid confidence intervals uniformly over all $0 \leq r \leq R$, including $r = 0$ where no factors exist.
- The thresholded idiosyncratic covariance estimator $\hat\Sigma_u$ is consistent in operator norm at the rate $(\omega_{NT})^{1-q} m_N$ for every $R \geq r$, so over-estimating factors does not corrupt large covariance estimation.
- A specification test of whether observed factors span the latent factor space has a standard normal null distribution, computed with a parametric bootstrap for the variance.
- The special case $r = 0$, $R \geq 1$ shows that extracting 'factors' when the panel is actually weakly dependent is a safe insurance procedure for factor-augmented inference.
Reading between the lines
- If the $r = 0$ insurance result is taken at face value, a practical policy recommendation follows that the paper only hints at: applied researchers should always run factor-augmented post-selection inference with at least one working factor, never pre-testing for the presence of factors.
- The Hadamard deterministic weights make the estimator a fixed linear sketch of the panel with no data-dependent tuning, which connects the method to random-projection and sketching ideas in computational statistics and treats the rank condition $\nu_{\min}(H) \gg N^{-1/2}$ as a coverage-type condition on the loading matrix.
- Because the projection is purely cross-sectional, the estimator is a natural candidate for nonstationary or structurally broken panels, as long as weights can be learned from a pre-period, the setting the moving-window construction in Section 4.2 is designed for.
- The rank condition on $H$ is effectively a demand that the user's weight directions cover the entire $r$-dimensional loading space, so a testable extension would be to check coverage empirically by comparing downstream inferences across several candidate weight matrices and looking for instability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes estimating latent factors in an approximate factor model by cross-sectional projections of the panel onto a user-chosen weight matrix W, yielding \hat f_t = N^{-1} W' x_t. The main theoretical claim, Theorem 2.1, is that when the working number of factors R is at least the true number r, the linear space spanned by the estimated factors consistently contains the space spanned by the true factors, at rate O_P(N^{-1/2} \nu_{\min}(H)^{-1}), and that this holds also for finite T. The paper further develops applications to factor-augmented forecasting, post-selection inference in high-dimensional regressions, thresholded idiosyncratic covariance estimation, a specification test for observed factors, and factor-adjusted false discovery control. Four choices of diversified weights are recommended: loading characteristics, rolling-window PCA loadings, initial transformations, and Hadamard columns. The appendix contains detailed proofs of the main estimation results and the forecast, inference, covariance, and specification-test theorems.
Significance. If the claims hold, the paper is a useful contribution: it offers a computationally trivial estimator of the factor space, it formalizes robustness to over-estimating the number of factors, and it covers several practically important downstream problems. The proofs are unusually detailed, and Proposition A.1 together with Theorem 2.1 provides a rigorous basis for the T, N \to \infty case. The paper is also honest about the fact that the estimator does not estimate the true factors themselves, only their span after an affine transformation. The main reasons I cannot endorse the manuscript as it stands are the gap between the finite-T claim and the proof, and the absence of a formal result for the FDR application.
major comments (3)
- [Theorem 2.1 and Appendix A.2 / Proposition A.1] The finite-T claim in Theorem 2.1 is not supported by the proof. Proposition A.1, on which the proof of Theorem 2.1 relies, is explicitly proved under 'T, N \to \infty', while Theorem 2.1 states 'T is either finite or grows'. More concretely, Proposition A.1(i) asserts that \lambda_{\min}((1/T)\hat F' K \hat F) \ge c/N with probability approaching one, but when T < R the matrix (1/T)\hat F' K \hat F is T \times T and has rank at most T < R, so its smallest eigenvalue is identically zero. The same issue affects the matrix M' \hat F' \hat F M used to define P_{\hat F M} when T < r. Thus the advertised finite-T robustness, repeated in the Abstract, Section 2.4, and the remark after Theorem 2.1, is a proof gap rather than an established result. The theorem should either be restricted to T \to \infty, or a separate argument covering finite T should be supplied.
- [Section 3.5] The factor-adjusted false discovery control is claimed as an application ('Our theories imply the following expansion...'), but no theorem, set of regularity conditions, or proof is given for the FDR control. The expansion displayed in Section 3.5 is stated informally, and it is not shown that the resulting test statistics are sufficiently weakly dependent or that the nominal FDR level is controlled uniformly over R \ge r. Since Section 3.5 is presented as one of the applications of the diversified-factor construction, this is a load-bearing omission for the paper's claims and must be addressed with a formal statement and proof, or the section should be explicitly labeled as heuristic.
- [Assumption 2.2 and Section 4] Assumption 2.2, requiring rank(W'B/N) = r and \nu_{\min}(W'B/N) \gg N^{-1/2}, is the key condition that makes the rates in Theorem 2.1 and all later theorems non-degenerate, but it is not verifiable from the observed panel. The recommended weight choices in Section 4 are given heuristic justification only, and for the Hadamard construction in Section 4.4 no proof is provided that the resulting deterministic W satisfies Assumption 2.2 for a loading matrix B of the assumed form. Because every consistency rate in the paper depends on \nu_{\min}(H), the practical applicability of the method would be substantially strengthened by explicit sufficient conditions on B for at least the deterministic weight choices, or by a discussion of the consequences when Assumption 2.2 fails.
minor comments (4)
- [Table 1] The caption states that Table 1 is 'computed based on one set of simulation replications'; with m = 50 forecast windows, the relative MSE entries therefore have no error bars or replication variability. Statements such as 'DP outperforms under the strong serial correlations' should be softened accordingly.
- [Section 3.5] The notation in Section 3.5 is inconsistent: the text defines \bar f = (1/T)\sum_t \bar f_t and \bar u = (1/T)\sum_t \bar u_t, but the symbols \bar f_t and \bar u_t are not defined before use. This should be cleaned up, for example by writing \hat f_t and \hat u_t.
- [Section 4.2] In the description of the moving-window weights, the periods (I) and (II) are indexed by t = 1,...,T_0 and t = T_0+1,...,T_0+T, but later the independence claim is stated for 't = m+1,...,m+T' with m undefined. The indexing should be made consistent.
- [Equation (2.2)] The statement that a cross-sectional CLT 'is straightforward to verify' under Assumption 2.1 is too casual: Assumption 2.1 does not itself imply a Lindeberg condition or the existence of the limit V. Since (2.2) is motivational rather than used in the main theorems, this is a presentation issue rather than a technical error.
Circularity Check
No circular reduction found: the diversified-factor estimator is an explicit weighted average with user-supplied W, and factor-space consistency is proved from statistical noise bounds, not from fitting the target.
full rationale
The estimator is defined by hat f_t = (1/N)W'x_t = H f_t + (1/N)W'u_t with H = W'B/N, so every application result is an analysis of this explicit decomposition rather than a fitting of the target outcome. Theorem 2.1's bounds (2.6)-(2.7) are rates for the projection error caused by the noise term (1/N)W'u_t, controlled by Assumptions 2.1-2.4 and the rank/non-decay condition on H; no parameter is fitted to y or to the object being predicted, and no fitted value is relabeled as a forecast. The forecast, post-selection, covariance, and specification-test theorems use the same decomposition plus standard regression and thresholding arguments; they do not assume the conclusion. The paper cites several prior works by the authors (Fan et al. 2013, 2015, 2016, 2020; Hansen and Liao 2018), but these appear as benchmarks, model motivation, or standard proof ingredients (e.g., lasso oracle bounds and thresholded covariance rates), not as a uniqueness theorem and not as the source of the estimator's definition; hence they are not load-bearing circularity. I therefore find no circular step. For completeness, two non-circular proof gaps are noted: Theorem 2.1 claims T finite or growing, yet its proof invokes Proposition A.1, which is stated under T,N going to infinity, and Proposition A.1(i) cannot hold when T<R because rank((1/T)hat F'hat F) is at most T < R; additionally, the proof of Proposition A.1 uses an unstated lower bound lambda_min((1/T)UU') >= c0 not present in Assumptions 2.1-2.4. These affect correctness, not circularity, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Working number of factors R =
user-specified bounded integer with R >= r
- Thresholding constant C in covariance estimator =
unspecified large constant
- Lasso penalty constant C =
C > 4 in tau = C sqrt(sigma^2 log N / T)
assumptions (6)
- domain assumption Factor model (1.1) holds with bounded loadings, stationary factors, and E(u_t | f_t) = 0 (Assumption 2.3).
- domain assumption Weak cross-sectional and serial dependence of idiosyncratic errors (Assumption 2.4).
- domain assumption Diversified weights W are bounded, have nonsingular Gram matrix, and are independent of idiosyncratic errors (Assumption 2.1).
- ad hoc to paper Rank condition: rank((1/N)W'B) = r and nu_min((1/N)W'B) >> N^{-1/2} with nu_max <= C nu_min (Assumption 2.2).
- domain assumption Application-specific conditions for forecasts and post-selection inference (Assumptions 3.1-3.4), including stationarity, moment bounds, sparse eigenvalue conditions, and rate restrictions.
- domain assumption For the factor specification test, conditional serial independence, strict sparsity, and a covariance bound (Assumptions 3.5-3.8).
Cite this review
Pith. "Pith review of Learning Latent Factors from Diversified Projections and its Applications to Over-Estimated and Weak Factors." pith.science (2026). https://pith.science/paper/XZQUE7MV
@misc{pith2026190801252,
author = {Pith},
title = {Pith review of: Learning Latent Factors from Diversified Projections and its Applications to Over-Estimated and Weak Factors},
year = {2026},
howpublished = {\url{https://pith.science/paper/XZQUE7MV}},
note = {Machine review of arXiv:1908.01252}
}
read the original abstract
Estimations and applications of factor models often rely on the crucial condition that the number of latent factors is consistently estimated, which in turn also requires that factors be relatively strong, data are stationary and weak serial dependence, and the sample size be fairly large, although in practical applications, one or several of these conditions may fail. In these cases it is difficult to analyze the eigenvectors of the data matrix. To address this issue, we propose simple estimators of the latent factors using cross-sectional projections of the panel data, by weighted averages with pre-determined weights. These weights are chosen to diversify away the idiosyncratic components, resulting in "diversified factors". Because the projections are conducted cross-sectionally, they are robust to serial conditions, easy to analyze and work even for finite length of time series. We formally prove that this procedure is robust to over-estimating the number of factors, and illustrate it in several applications, including post-selection inference, big data forecasts, large covariance estimation and factor specification tests. We also recommend several choices for the diversified weights.
Figures
Reference graph
Works this paper leans on
-
[1]
Ahn, S. and Horenstein, A. (2013). Eigenvalue ratio test for the number of factors. Econometrica 81 1203--1227
work page 2013
-
[2]
Antoniadis, A. and Fan, J. (2001). Regularized wavelet approximations. Journal of the American Statistical Association 96 939--967
work page 2001
-
[3]
Bai, J. (2003). Inferential theory for factor models of large dimensions. Econometrica 71 135--171
work page 2003
- [4]
-
[5]
Bai, J. and Liao, Y. (2017). Inferences in panel data with interactive effects using large covariance matrices. Journal of Econometrics 200 59--78
work page 2017
- [6]
- [7]
-
[8]
Consistent estimation of high-dimensional factor models when the factor number is over-estimated
Barigozzi, M. and Cho, H. (2018). Consistent estimation of high-dimensional factor models when the factor number is over-estimated. arXiv preprint arXiv:1811.00306
work page Pith review arXiv 2018
Show all 42 references
-
[9]
, Chernozhukov, V
Belloni, A. , Chernozhukov, V. and Hansen, C. (2014). Inference on treatment effects after selection among high-dimensional controls. The Review of Economic Studies 81 608--650
2014
-
[10]
and Levina, E
Bickel, P. and Levina, E. (2008). Covariance regularization by thresholding. Annals of Statistics 36 2577--2604
2008
-
[11]
, Ritov, Y
Bickel, P. , Ritov, Y. and Tsybakov, A. (2009). Simultaneous analysis of lasso and dantzig selector. Annals of Statistics 37 1705--1732
2009
-
[12]
Carhart, M. M. (1997). On persistence in mutual fund performance. Journal of Finance 52 57--82
1997
-
[13]
, Pesaran, M
Chudik, A. , Pesaran, M. H. and Tosetti, E. (2011). Weak and strong cross-section dependence and estimation of large panels. The Econometrics Journal 14 C45--C90
2011
-
[14]
and Korajczyk, R
Connor, G. and Korajczyk, R. A. (1986). Performance measurement with the arbitrage pricing theory: A new framework for analysis. Journal of Financial Economics 15 373--394
1986
-
[15]
, Matthias, H
Connor, G. , Matthias, H. and Linton, O. (2012). Efficient semiparametric estimation of the fama-french model and extensions. Econometrica 80 713--754
2012
-
[16]
, Giannone, D
Doz, C. , Giannone, D. and Reichlin, L. (2011). A two-step estimator for large approximate dynamic factor models based on kalman filtering. Journal of Econometrics 164 188--205
2011
-
[17]
Fama, E. F. and French, K. R. (1992). The cross-section of expected stock returns. Journal of Finance 47 427--465
1992
-
[18]
Fan, J. , Ke, Y. , Sun, Q. and Zhou, W.-X. (2019). Farmtest: Factor-adjusted robust multiple testing with approximate false discovery control. Journal of the American Statistical Association 1--29
2019
-
[19]
Fan, J. , Ke, Y. and Wang, K. (2020). Factor-adjusted regularized model selection. Journal of Econometrics
2020
-
[20]
, Liao, Y
Fan, J. , Liao, Y. and Mincheva, M. (2013). Large covariance estimation by thresholding principal orthogonal complements (with discussion). Journal of the Royal Statistical Society, Series B 75 603--680
2013
-
[21]
, Liao, Y
Fan, J. , Liao, Y. and Wang, W. (2016). Projected principal component analysis in factor models. Annals of Statistics 44 219--254
2016
-
[22]
, Liao, Y
Fan, J. , Liao, Y. and Yao, J. (2015). Power enhancement in high dimensional cross-sectional tests. Econometrica 83 1497--1541
2015
-
[23]
, Hallin, M
Forni, M. , Hallin, M. , Lippi, M. and Reichlin, L. (2005). The generalized dynamic factor model: one-sided estimation and forecasting. Journal of the American Statistical Association 100 830--840
2005
-
[24]
, Ossola, E
Gagliardini, P. , Ossola, E. and Scaillet, O. (2016). Time-varying risk premium in large cross-sectional equity data sets. Econometrica 84 985--1046
2016
-
[25]
and Perron, B
Goncalves, S. and Perron, B. (2018). Bootstrapping factor models with cross sectional dependence
2018
-
[26]
and Li s ka, R
Hallin, M. and Li s ka, R. (2007). Determining the number of factors in the general dynamic factor model. Journal of the American Statistical Association 102 603--617
2007
-
[27]
and Liao, Y
Hansen, C. and Liao, Y. (2018). The factor-lasso and k-step bootstrap approach for inference in high-dimensional economic applications. Econometric Theory 1--45
2018
-
[28]
Johnstone, I. M. and Lu, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association 104 682--693
2009
-
[29]
and Sarafidis, V
Juodis, A. and Sarafidis, V. (2020). A linear estimator for factoraugmented fixed-t panels with endogenous regressors. Tech. rep., Monash University, Department of Econometrics and Business Statistics
2020
-
[30]
, Reese, S
Karabiyik, H. , Reese, S. and Westerlund, J. (2017). On the role of the rank condition in cce estimation of factor-augmented panel regressions. Journal of Econometrics 197 60--64
2017
-
[31]
, Urbain, J.-P
Karabiyik, H. , Urbain, J.-P. and Westerlund, J. (2019). Cce estimation of factor-augmented regression models with more factors than observables. Journal of Applied Econometrics 34 268--284
2019
-
[32]
, Korajczyk, R
Kim, S. , Korajczyk, R. A. and Neuhierl, A. (2018). Arbitrage portfolios in large panels. Available at SSRN
2018
-
[33]
, Yang, D
Li, G. , Yang, D. , Nobel, A. B. and Shen, H. (2016). Supervised singular value decomposition and its asymptotic properties. Journal of Multivariate Analysis 146 7--17
2016
-
[34]
Li, H. , Li, Q. and Shi, Y. (2017). Determining the number of factors when the number of factors can increase with sample size. Journal of Econometrics 197 76--86
2017
-
[35]
and Ng, S
Ludvigson, S. and Ng, S. (2007). The empirical risk--return relation: a factor analysis approach. Journal of Financial Economics 83 171--222
2007
-
[36]
and Weidner, M
Moon, R. and Weidner, M. (2015). Linear regression for panel with unknown number of factors as interactive fixed effects. Econometrica 83 1543--1579
2015
-
[37]
Park, B. U. , Mammen, E. , H \"a rdle, W. and Borak, S. (2009). Time series modelling with semiparametric factor dynamics. Journal of the American Statistical Association 104 284--298
2009
-
[38]
Pesaran, H. (2006). Estimation and inference in large heterogeneous panels with a multifactor error structure. Econometrica 74 967--1012
2006
-
[39]
Robinson, P. M. (1988). Root-n-consistent semiparametric regression. Econometrica 56 931--954
1988
-
[40]
and Watson, M
Stock, J. and Watson, M. (2002). Forecasting using principal components from a large number of predictors. Journal of the American Statistical Association 97 1167--1179
2002
-
[41]
, Zhao, Q
Wang, J. , Zhao, Q. , Hastie, T. and Owen, A. B. (2017). Confounder adjustment in multiple hypothesis testing. Annals of statistics 45 1863
2017
-
[42]
and Urbain, J.-P
Westerlund, J. and Urbain, J.-P. (2015). Cross-sectional averages versus principal components. Journal of Econometrics 185 372--377
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.