REVIEW 3 major objections 5 minor 10 references
From Replications to Revelations: Heteroskedasticity-Robust Inference
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Stata's default HC1 standard errors over-reject true nulls, and degree-of-freedom-adjusted methods nearly fix it.
desk verdict A transparent, large-scale simulation study on robust inference with a genuinely new partial-leverage degrees-of-freedom adjustment, but the DGP calibration is the main vulnerability. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the partial leverage $\tilde{h}_{k,i}=\tilde{x}_{k,i}^2/\sum_j \tilde{x}_{k,j}^2$, the normalized squared residual from the Frisch-Waugh-Lovell regression of regressor $k$ on the other regressors. The paper defines the partial-leverage-adjusted sample size $\tilde{n}_k=(\sum_i \tilde{h}_{k,i}^2)^{-1}$, the inverse Herfindahl concentration of partial leverages, and proposes t-tests HC1-PL and HC2-PL that use $\tilde{n}_k-1$ degrees of freedom instead of $n-K$. This quantity carries the argument because it measures how many independent observations effectively inform the variance estimate for $\hat{\beta}_k$; when partial leverages are concentrated in a few observations, the t-distribution is widened accordingly. An appendix derives the adjustment from a Satterthwaite variance-combination approximation, showing that $\tilde{n}_k$ approximates the degrees of freedom of an HC0-based t-statistic under homoskedasticity.
What would settle it
Re-run the Monte Carlo on the same 608 regressions with a different error-variance fitting rule---leave-one-out squared residuals or a spline-based FGLS instead of the selected random-forest candidate---and check whether HC1's average excess still exceeds JK-H's and HC2-PL's by the reported margin.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the degrees of freedom in the t-test, not the standard-error formula alone, drive finite-sample performance of robust inference. In 3,280 test situations drawn from real regressions, HC1 and HC2 with the default $n-K$ degrees of freedom over-reject true nulls on average, while HC3 is conservative. Specifications that replace $n-K$ with data-dependent degrees of freedom---HC2-BM, the jackknife estimator JK-H, and the proposed HC1-PL and HC2-PL---all outperform the HC1--HC4 family, with JK-H and HC2-PL achieving average excess rejection rates of 0.06% and 0.1%, comparable to Monte Carlo noise. The paper also identifies a rare but severe failure mode: 31 test situations where an observation has full leverage and positive partial leverage cause average excess above 20% for every method except JK-H, HC1-PL, and HC2-PL, and shows that assigning the homoskedastic error estimate to such observations instead of zero reduces the excess to about 3%.
Load-bearing premise
The ranking assumes that the random-forest-fitted error variances for the 608 reproducible regressions approximate the true error variances in real economics data, and that those 155 packages represent the broader population; if either assumption fails, the reported ordering of methods could change.
Editorial extensions
If this is right
- If applied researchers switched from the default HC1 specification to HC2-PL or JK-H, the average excess rejection rate in samples up to 1,000 observations would fall from roughly 1.6% to near 0.1%, so many p-values just below 5% would cease to be significant.
- The partial-leverage adjustment needs only FWL residuals, so it is computable in regressions with absorbed fixed effects, where HC2-family hat values are harder to obtain; HC1-PL is thus a practical option for that common setting.
- For the small set of regressions with full leverage and positive partial leverage, all standard methods except JK-H, HC1-PL, and HC2-PL over-reject at rates above 20%, and replacing the zero error estimate with the homoskedastic estimate brings excess down to about 3%.
- Customized degrees of freedom matter more than the standard-error formula: HC2-PL and HC2-BM both use HC2-style variance estimates but outperform HC2 with default degrees of freedom.
- Wild-bootstrap t-tests improve on plain HC1 and HC2, but in the paper's simulations they still show higher average excess than HC2-BM, JK-H, and HC2-PL.
Reading between the lines
- Because the paper's central quantity is the inverse Herfindahl of partial leverages, applying the same construction to cluster partial leverages predicts that a cluster-level $\tilde{G}_k - 1$ degree of freedom will beat the default $G - 1$; the paper notes this extension is planned.
- The paper's DGP selection never validates that the fitted random-forest variances match true error variances; a holdout validation comparing predicted and realized residual variances would tell whether HC1's excess is an artifact of the residual-scaling step.
- The paper averages excess and lack across all test situations; if one instead minimizes worst-case rejection rates, the 31 full-leverage cases suggest reporting $\tilde{n}_k$ for every coefficient as a routine diagnostic, which software could automate.
- The JK-H and HC2-PL average excess figures (0.06% vs 0.1%) differ by less than the 0.09% Monte Carlo noise floor, so their tie may not persist under a larger sample of packages.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper documents that 98.1% of the 40,571 regression commands in 4,420 Stata reproduction packages that specify heteroskedasticity-robust standard errors use Stata's default HC1 specification, and then asks whether this practice leads to poor inference. For 608 regressions from 155 reproduction packages, the author constructs custom Monte Carlo DGPs that keep the original design matrix X, set all coefficients to zero, and generate heteroskedastic normal errors whose standard deviations are estimated by random-forest FGLS candidates and selected by matching the kurtosis of the original OLS residuals. Comparing rejection rates at the 5% level across 3,280 test situations, the paper finds that HC1 and HC2 with default n-K degrees of freedom over-reject, that customized degrees-of-freedom methods (HC2-BM, JK-H, and the new HC1-PL and HC2-PL) have lower average excess, and that JK-H and HC2-PL perform best on average. The paper also recommends a particular treatment of observations with full leverage, namely imputing the homoskedastic error estimate instead of zero for HC2-HC4-type estimators.
Significance. If the Monte Carlo ranking is accepted, the paper provides a valuable large-scale confirmation that Stata's default HC1 can over-reject in small to moderate samples and offers concrete practical guidance in favor of JK-H and HC2-PL. The strengths include the transparency of the Monte Carlo design based on real regressions, the machine-checkable definition of the partial-leverage degrees-of-freedom adjustment, and the systematic evidence on how rarely HC3 and other alternatives are used in practice. The paper also makes falsifiable recommendations and carefully separates excess from lack of rejection rates. However, the significance is conditional on the representativeness of the constructed DGPs, and the current evidence for that representativeness is limited.
major comments (3)
- [Appendix C; Section 2, Eqs. (9)-(10)] The DGP selection criterion does not constrain the variance-leverage relationship that drives the ranking. The true variance of beta_k and the robust variance estimators in Eqs. (9)-(10) are weighted sums of sigma_i^2 times the partial leverage htilde_{k,i}, so the relative performance of HC2, HC3, HC2-BM, JK-H, HC1-PL, and HC2-PL depends on how sigma_i covaries with leverage and partial leverage. The selection procedure in Appendix C, Eqs. (21)-(23), matches only the scalar kurtosis of OLS residuals after scaling to match residual standard deviations, and kurtosis is nearly uninformative about this covariance. The paper should validate that the fitted sigma_i reproduce the observed relationship between absolute residuals (or other variance proxies) and leverages/partial leverages in the original regressions, for example by comparing nonparametric regressions of log |e_i| on h_i and htilde_{k,i} in the original data and in simulated data, and should show that the main ranking is robust to alternative DGP-selection criteria that target this relationship. Without such validation, the favorable showing of JK-H and HC2-PL could be an artifact of the random-forest FGLS specification rather than a property of typical empirical regressions.
- [Table 2, columns (5)-(8)] The recommendation to impute sigma_hat_i = sigma for observations with full leverage rests on only 31 test situations. Of these, 20 have partial-leverage-adjusted sample size tilde n_k < 2, so the rejection rates are extremely dispersed and the average-excess reductions reported in columns (7) and (8) are not statistically reliable. The paper should report the 31 individual rejection rates or at least standard errors or confidence intervals for the averages, and should clearly label the full-leverage imputation recommendation as tentative pending more evidence.
- [Appendix B] The Monte Carlo sample covers only 155 of the 4,420 reproduction packages (608 regressions, 3,280 test situations) and is selected by criteria that include package size, reproducibility within a time limit, and a maximum of four regressions per package. The paper acknowledges that this is disappointing but does not assess how the selected regressions compare with the full population on observable characteristics such as sample size, number of regressors, R-squared, or dummy-variable structure. Because the central claim concerns typical situations encountered in economic analyses, the paper should include a comparison of included and excluded packages or a clear statement of the population to which the conclusions are intended to generalize.
minor comments (5)
- [Section 2] The text after Figure 3 refers to 'Hansen's (2004) jackknife estimator', but the bibliography and the rest of the paper cite Hansen (2024); this should be corrected.
- [Appendix A] The appendix refers to 'Panel A of Table 1' and 'Panel B of Figure 1' when presenting the frequency of standard-error categories, but these panels are in the appendix tables and figures, not the main Table 1 and Figure 1; the cross-references should be renumbered to avoid confusion.
- [Section 3] The paper acknowledges that it does not study the methods of Cattaneo et al. (2018) and Poetscher and Preinerstorfer (2023); given that the latter proposes valid heteroskedasticity-robust testing, the paper's conclusions should be explicitly scoped to the compared methods, and this scope should be stated in the abstract or introduction as well.
- [Appendix C, footnote on random-forest variations] The claim that 'preliminary experiments suggest that these variations have little impact on the main results' is not documented; either provide the supporting experiments or remove the claim.
- [Section 2, full-leverage discussion] The description of Stata's treatment of full leverage relies on 'experiments and personal communications' and is not reproducible from the manuscript; a documented explanation or reference would be helpful.
Circularity Check
No circularity: the partial-leverage degrees-of-freedom adjustment is derived from the design matrix and evaluated on null DGPs, not fitted to the target rejection rates.
full rationale
The paper's derivation chain is self-contained rather than circular. The novel HC1-PL and HC2-PL methods use the partial-leverage-adjusted sample size ntilde_k, defined from the design matrix X alone, as degrees of freedom in the t-test. Appendix E derives ntilde_k as an approximation to the Satterthwaite degrees of freedom for HC0 under homoskedasticity; this is an independent mathematical argument, not a parameter fitted to the Monte Carlo rejection rates. The paper explicitly disclaims this derivation as the sole basis: 'It is clear that this derivation involves several approximations, and we would not propose our degree of freedom adjustment solely based on this result.' The Monte Carlo evaluation uses null DGPs with beta = 0, so the rejection rates are not engineered by construction; the DGP selection in Appendix C matches residual standard deviations and kurtosis, not the ranking of inference methods. The comparison against external benchmarks (Bell and McCaffrey 2002, Hansen 2024) provides independent evidence. The concern that the kurtosis-only DGP selection may not reproduce the true variance-leverage relationship is a representativeness or external-validity limitation, not a circular reduction of the paper's claims to its inputs. No load-bearing self-citation or uniqueness-imported-from-authors pattern is present.
Assumptions & free parameters
free parameters (3)
- Per-regression Monte Carlo DGP candidate and error variances =
Not reported; one of five candidate models selected per regression
- DGP selection distance weights =
0.5 for standardized kurtosis distance, 0.5 for modified distance
- Full-leverage imputation value for sigma_hat_i =
homoskedastic residual standard error sigma_hat
assumptions (5)
- domain assumption The Monte Carlo DGP y = X beta + epsilon with beta=0, epsilon_i ~ N(0, sigma_i^2), and fixed X from original regressions is representative of empirical practice.
- ad hoc to paper Random-forest FGLS candidates can approximate true error heteroskedasticity sufficiently well for method comparison.
- standard math The Satterthwaite approximation gives a t-distribution with degrees of freedom 2 Var(beta_hat)^2 / Var(V_hat).
- domain assumption For observations with full leverage, setting sigma_hat_i = 0 is a valid convention, and setting sigma_hat_i = sigma_hat is more conservative.
- domain assumption The 155 reproduction packages that repbox could successfully process are representative of empirical economics regressions.
Cite this review
Pith. "Pith review of From Replications to Revelations: Heteroskedasticity-Robust Inference." pith.science (2026). https://pith.science/paper/L3T74SDL
@misc{pith2026241114763,
author = {Pith},
title = {Pith review of: From Replications to Revelations: Heteroskedasticity-Robust Inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/L3T74SDL}},
note = {Machine review of arXiv:2411.14763}
}
read the original abstract
Analysing the Stata regression commands from 4,420 reproduction packages of leading economic journals, we find that, among the 40,571 regressions specifying heteroskedasticity-robust standard errors, 98.1% adhere to Stata's default HC1 specification. We then compare several heteroskedasticity-robust inference methods with a large-scale Monte Carlo study based on regressions from 155 reproduction packages. Our results show that t-tests based on HC1 or HC2 with default degrees of freedom exhibit substantial over-rejection. Inference methods with customized degrees of freedom, as proposed by Bell and McCaffrey (2002), Hansen (2024), and a novel approach based on partial leverages, perform best. Additionally, we provide deeper insights into the role of leverages and partial leverages across different inference methods.
Figures
Reference graph
Works this paper leans on
-
[1]
For each original regressionr, we specify a set ofC candidate modelsMr,c indexed byc = 1, ..., C for the distribution of the error term. We only consider candidate models with independently distributed error terms for Monte Carlo samplem satisfying εm,r,c i ∼ N (0, (σr,c i )2). This means each candidate model is fully characterized by the specified vector...
-
[2]
From each candidate model we drawm = 1, ..., Msamples of the error termεm,r,c and compute the corresponding OLS residualsˆεm,r,c
-
[3]
We then compute for each candidate model a distancedr,c between the original OLS residuals ˆεr,o and the set of Monte Carlo residuals{ˆεm,r,c}M m=1
-
[4]
For each original regressionr, we pick that candidate model for the Monte Carlo DGP that has the lowest distancedr,c. One candidate model assumes purely homoskedastic error terms, while all other candidate models estimate error term standard errorsσr,c i using a non-parametric FGLS specification based on random forests. The dependent variable of the rando...
work page 2019
-
[5]
pp. 59–82. • Hansen, B. (2022). “Econometrics”. Princeton University Press. • Hansen, B. (2024). “Jackknife standard errors for clustered regression.” University of Wisconsin. • Huber, Peter J. (1967). "The behavior of maximum likelihood estimates under nonstandard conditions". Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Pro...
work page 2022
-
[6]
Estimatetherestrictedregressionmodelunderthenullhypothesis βk = 0toobtainOLSresiduals er and predicted valuesˆyr.11
-
[7]
Generate bootstrap error terms εb = √ αθer · vb, where αθ is an adjustment based on type θ ∈ {HC1,HC2,HC3} as specified in Section 3. The factor vb is an n × 1 vector of random weights independently drawn from a Rademacher distribution vb i = ( −1 with probability 0.5 1 with probability 0.5 (24)
-
[8]
Form a bootstrap sampleyb = ˆyr + εb and re-estimate the model to obtain the OLS estimator ˆβb k and a corresponding variance estimatorˆV b,η k of type η ∈ {HC1,HC2,HC3}. We then compute the corresponding t-statistic for the null hypothesisβk = 0: tb,η k = ˆβb kq ˆV b,η k (25)
Show all 10 references
-
[9]
Repeat steps 2 and for 3 forB bootstrap replications to construct the bootstrap distribution of the test statistic
-
[10]
11Notably, given the restrictionβk = 0 the restricted OLS residualse are simply the residuals of the OLS regression that leaves out the regressorxk
Calculate the bootstrap p-value as the proportion of bootstrap statisticstb,η k that are as extreme as or more extreme than the test statistictη k from the original regression sample (also computed using a standard error of typeη): p-value = 1 B BX b=1 I(|tb,η k | ≥ |tη k|), (...
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.