{"id":"1c5ab252-44b7-4d54-9248-6ba452f85b1c","arxiv_id":"2411.14763","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Monte Carlo study of 608 real regressions finds that HC1 and HC2 with default degrees of freedom over-reject, and customized degrees-of-freedom methods, including a new partial-leverage approach, perform best.","lead":"This paper examines 4,420 Stata reproduction packages from economics journals and finds that 98.1% of 40,571 regressions with heteroskedasticity-robust standard errors use Stata's default HC1. It then runs large-scale Monte Carlo simulations on 608 real regressions and recommends new degrees-of-freedom adjustments that control false rejection rates better than HC1 or HC2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Kurtosis-only DGP selection in Appendix C does not constrain the variance–leverage relationship, which is exactly what determines the ranking of HC2-BM, JK-H, HC1-PL, and HC2-PL.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, and the identified weakest assumption is precisely the representativeness of the Appendix C DGP calibration. My reading agrees and sharpens the point: the specific unvalidated object is the relationship between error variances and leverages/partial leverages, not merely the marginal distribution of errors. Since the Monte Carlo ranking is the paper's central empirical contribution, and since no replication archive or external validation is provided for the fitted sigma_i patterns, the conditional verdict is appropriate. I considered other possible concerns—the 31 full-leverage test situations, the ad hoc imputation of sigma_i for full-leverage observations, and the representativeness of the 155 packages—but none is as load-bearing as the DGP identification problem. The full-leverage recommendation affects only a tiny subset of test situations, and the package-selection issue is secondary to the internal validity of the Monte Carlo. The proposed concrete test is feasible because the original regressions and residuals are already available from the repbox pipeline, and it directly targets the variance–leverage relationship that determines method ranking. If the ranking survives alternative DGPs matched on the same low-order moments, the central claim would be substantially strengthened. If it does not, the paper's headline recommendation would need to be re-framed as conditional on the random-forest FGLS model. The reader's verdict of CONDITIONAL is therefore unchanged by this stress-test pass.","tokens_in":16361,"tokens_out":3603,"duration_ms":39443,"concrete_test":"On a stratified subsample of the 608 regressions, construct two DGP families per regression: (A) the selected random-forest sigma_i, and (B) sigma_i obtained from a nonparametric regression of log(original OLS residual^2) on h_i and htilde_{k,i}, with both families calibrated to the same residual standard deviation and kurtosis. Run the Monte Carlo with M = 1000 for HC1, HC2, HC3, HC2-BM, JK-H, HC1-PL, and HC2-PL, and compare average excess and lack. If family B materially changes the ranking—for example if HC2-BM no longer ties with JK-H or the PL methods lose their low excess—then kurtosis-only selection does not identify the variance–leverage relationship and the central claim is conditional on the random-forest FGLS choice. If the ranking is stable across both families, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central ranking claim is that customized degrees-of-freedom methods (HC2-BM, JK-H, HC1-PL, HC2-PL) outperform HC1/HC2 with default degrees of freedom. For this claim to be identified, the 608 Monte Carlo DGPs must reproduce the joint distribution of error variances and (partial) leverages found in real regressions. Appendix C selects each DGP from five random-forest FGLS candidates by matching the kurtosis of OLS residuals (equations 21–23), after scaling to match residual standard deviations. Kurtosis is a scalar and is nearly uninformative about how sigma_i varies with h_i or htilde_{k,i}. Equation (9) shows the true variance of beta_k is proportional to sum_i sigma_i^2 htilde_{k,i}, and the relative performance of HC2, HC3, HC2-BM, HC1-PL, and HC2-PL is driven by how much weight high-leverage observations place on noisy residuals. Two DGPs with identical residual sd and kurtosis can have opposite variance–leverage alignments. The paper reports no direct validation that the fitted sigma_i reproduce the observed relationship between absolute residuals and leverages/partial leverages in the original data. Consequently, the favorable showing of HC2-PL and JK-H could be an artifact of the random-forest FGLS specification rather than a property of real empirical regressions. The reader's weakest-assumption critique is correct: the burden is on Appendix C's calibration, not on the internal mathematics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper documents that 98.1% of the 40,571 regression commands in 4,420 Stata reproduction packages that specify heteroskedasticity-robust standard errors use Stata's default HC1 specification, and then asks whether this practice leads to poor inference. For 608 regressions from 155 reproduction packages, the author constructs custom Monte Carlo DGPs that keep the original design matrix X, set all coefficients to zero, and generate heteroskedastic normal errors whose standard deviations are estimated by random-forest FGLS candidates and selected by matching the kurtosis of the original OLS residuals. Comparing rejection rates at the 5% level across 3,280 test situations, the paper finds that HC1 and HC2 with default n-K degrees of freedom over-reject, that customized degrees-of-freedom methods (HC2-BM, JK-H, and the new HC1-PL and HC2-PL) have lower average excess, and that JK-H and HC2-PL perform best on average. The paper also recommends a particular treatment of observations with full leverage, namely imputing the homoskedastic error estimate instead of zero for HC2-HC4-type estimators.","tokens_in":16649,"tokens_out":7581,"duration_ms":73340,"significance":"If the Monte Carlo ranking is accepted, the paper provides a valuable large-scale confirmation that Stata's default HC1 can over-reject in small to moderate samples and offers concrete practical guidance in favor of JK-H and HC2-PL. The strengths include the transparency of the Monte Carlo design based on real regressions, the machine-checkable definition of the partial-leverage degrees-of-freedom adjustment, and the systematic evidence on how rarely HC3 and other alternatives are used in practice. The paper also makes falsifiable recommendations and carefully separates excess from lack of rejection rates. However, the significance is conditional on the representativeness of the constructed DGPs, and the current evidence for that representativeness is limited.","major_comments":[{"comment":"The DGP selection criterion does not constrain the variance-leverage relationship that drives the ranking. The true variance of beta_k and the robust variance estimators in Eqs. (9)-(10) are weighted sums of sigma_i^2 times the partial leverage htilde_{k,i}, so the relative performance of HC2, HC3, HC2-BM, JK-H, HC1-PL, and HC2-PL depends on how sigma_i covaries with leverage and partial leverage. The selection procedure in Appendix C, Eqs. (21)-(23), matches only the scalar kurtosis of OLS residuals after scaling to match residual standard deviations, and kurtosis is nearly uninformative about this covariance. The paper should validate that the fitted sigma_i reproduce the observed relationship between absolute residuals (or other variance proxies) and leverages/partial leverages in the original regressions, for example by comparing nonparametric regressions of log |e_i| on h_i and htilde_{k,i} in the original data and in simulated data, and should show that the main ranking is robust to alternative DGP-selection criteria that target this relationship. Without such validation, the favorable showing of JK-H and HC2-PL could be an artifact of the random-forest FGLS specification rather than a property of typical empirical regressions.","section":"Appendix C; Section 2, Eqs. (9)-(10)"},{"comment":"The recommendation to impute sigma_hat_i = sigma for observations with full leverage rests on only 31 test situations. Of these, 20 have partial-leverage-adjusted sample size tilde n_k < 2, so the rejection rates are extremely dispersed and the average-excess reductions reported in columns (7) and (8) are not statistically reliable. The paper should report the 31 individual rejection rates or at least standard errors or confidence intervals for the averages, and should clearly label the full-leverage imputation recommendation as tentative pending more evidence.","section":"Table 2, columns (5)-(8)"},{"comment":"The Monte Carlo sample covers only 155 of the 4,420 reproduction packages (608 regressions, 3,280 test situations) and is selected by criteria that include package size, reproducibility within a time limit, and a maximum of four regressions per package. The paper acknowledges that this is disappointing but does not assess how the selected regressions compare with the full population on observable characteristics such as sample size, number of regressors, R-squared, or dummy-variable structure. Because the central claim concerns typical situations encountered in economic analyses, the paper should include a comparison of included and excluded packages or a clear statement of the population to which the conclusions are intended to generalize.","section":"Appendix B"}],"minor_comments":[{"comment":"The text after Figure 3 refers to 'Hansen's (2004) jackknife estimator', but the bibliography and the rest of the paper cite Hansen (2024); this should be corrected.","section":"Section 2"},{"comment":"The appendix refers to 'Panel A of Table 1' and 'Panel B of Figure 1' when presenting the frequency of standard-error categories, but these panels are in the appendix tables and figures, not the main Table 1 and Figure 1; the cross-references should be renumbered to avoid confusion.","section":"Appendix A"},{"comment":"The paper acknowledges that it does not study the methods of Cattaneo et al. (2018) and Poetscher and Preinerstorfer (2023); given that the latter proposes valid heteroskedasticity-robust testing, the paper's conclusions should be explicitly scoped to the compared methods, and this scope should be stated in the abstract or introduction as well.","section":"Section 3"},{"comment":"The claim that 'preliminary experiments suggest that these variations have little impact on the main results' is not documented; either provide the supporting experiments or remove the claim.","section":"Appendix C, footnote on random-forest variations"},{"comment":"The description of Stata's treatment of full leverage relies on 'experiments and personal communications' and is not reproducible from the manuscript; a documented explanation or reference would be helpful.","section":"Section 2, full-leverage discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the central question is important, but the main empirical claim depends on the unvalidated DGP-selection step in Appendix C. I recommend major revision rather than rejection because the missing validation is feasible with the existing data and would resolve the main correctness risk."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Kranz paper on heteroskedasticity-robust inference. The headline is that the author has done a large, transparent Monte Carlo comparison using real Stata code from 4,420 reproduction packages, found that 98.1% of robust regressions use HC1, and proposes a new degrees-of-freedom adjustment based on partial leverages (HC1-PL and HC2-PL). The new method is simple, computationally cheap, and seems to work well in the simulations. If the simulation design holds up, that is a practically useful result.\n\nWhat is new: the partial-leverage df adjustment is genuinely new to me; it's distinct from Bell-McCaffrey and Hansen's jackknife. The large-scale design, using 608 regressions from 155 packages, is a meaningful extension of Young (2022). The paper is also unusually transparent about its own limitations; the appendix notes the DGP selection is a heuristic and the repbox toolchain is a work in progress.\n\nWhere I have doubts: the Monte Carlo DGP selection matches the kurtosis of the original OLS residuals, but kurtosis is a scalar and does not pin down how error variances line up with partial leverages. The relative performance of HC2-PL, JK-H, and HC2-BM is exactly what depends on that variance-leverage relationship. So the ranking could be an artifact of the random-forest FGLS family used to generate DGPs. The author does not validate that the fitted sigma_i reproduce the observed variance-leverage pattern in the actual data. That is the load-bearing weakness, and it needs a sensitivity analysis or a different calibration target. The full-leverage recommendation is based on only 31 test situations, which is thin, but the point about setting sigma_i = 0 being a recipe for undercoverage is reasonable. Also, no replication archive is provided yet, and the wild bootstrap results are preliminary.\n\nOn balance, this is a serious, honest paper that deserves a referee. The right referee will ask for evidence that the DGP calibration captures real variance-leverage patterns, and for code/data to check the simulations. But the core descriptive finding and the proposed method are worth engaging with. I'd bring it to reading group.\n\nBest.","headline":"A transparent, large-scale simulation study on robust inference with a genuinely new partial-leverage degrees-of-freedom adjustment, but the DGP calibration is the main vulnerability.","tokens_in":17211,"tokens_out":2110,"would_cite":true,"duration_ms":21784,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J05","62F03","62P20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Stata's default HC1 standard errors over-reject true nulls, and degree-of-freedom-adjusted methods nearly fix it.","keywords":["heteroskedasticity-robust standard errors","HC1","degrees of freedom adjustment","partial leverage","jackknife standard errors","Monte Carlo simulation","reproduction packages","Stata regression commands"],"falsifier":"Re-run the Monte Carlo on the same 608 regressions with a different error-variance fitting rule---leave-one-out squared residuals or a spline-based FGLS instead of the selected random-forest candidate---and check whether HC1's average excess still exceeds JK-H's and HC2-PL's by the reported margin.","tokens_in":16106,"feed_emoji":"📊","tokens_out":12958,"duration_ms":111368,"temperature":0.7,"pith_summary":"This paper asks whether the way applied economists compute heteroskedasticity-robust standard errors produces trustworthy significance tests. Analyzing archived Stata code and data from 4,420 reproduction packages in leading journals, it finds that 98.1% of 40,571 regressions with robust standard errors use the default HC1 specification. The paper then builds Monte Carlo data-generating processes for 608 real regressions from 155 packages, fitting error variances with random forests, and tests 3,280 true zero coefficients at the 5% level. It reports that t-tests based on HC1 or HC2 with default degrees of freedom over-reject on average, while tests with customized degrees of freedom (HC2-BM, JK-H, and the paper's own HC1-PL and HC2-PL) keep rejection rates close to 5%. If the ranking is right, many marginally significant results in published work would not survive a better-chosen test.","feed_headline":"Stata's default robust errors over-reject true nulls in small samples","feed_subtitle":"Analyzing 608 real regressions, it finds degree-of-freedom-adjusted tests cut false rejections to near 5%.","key_machinery":"The central object is the partial leverage $\\tilde{h}_{k,i}=\\tilde{x}_{k,i}^2/\\sum_j \\tilde{x}_{k,j}^2$, the normalized squared residual from the Frisch-Waugh-Lovell regression of regressor $k$ on the other regressors. The paper defines the partial-leverage-adjusted sample size $\\tilde{n}_k=(\\sum_i \\tilde{h}_{k,i}^2)^{-1}$, the inverse Herfindahl concentration of partial leverages, and proposes t-tests HC1-PL and HC2-PL that use $\\tilde{n}_k-1$ degrees of freedom instead of $n-K$. This quantity carries the argument because it measures how many independent observations effectively inform the variance estimate for $\\hat{\\beta}_k$; when partial leverages are concentrated in a few observations, the t-distribution is widened accordingly. An appendix derives the adjustment from a Satterthwaite variance-combination approximation, showing that $\\tilde{n}_k$ approximates the degrees of freedom of an HC0-based t-statistic under homoskedasticity.","core_discovery":"On the paper's own terms, the central discovery is that the degrees of freedom in the t-test, not the standard-error formula alone, drive finite-sample performance of robust inference. In 3,280 test situations drawn from real regressions, HC1 and HC2 with the default $n-K$ degrees of freedom over-reject true nulls on average, while HC3 is conservative. Specifications that replace $n-K$ with data-dependent degrees of freedom---HC2-BM, the jackknife estimator JK-H, and the proposed HC1-PL and HC2-PL---all outperform the HC1--HC4 family, with JK-H and HC2-PL achieving average excess rejection rates of 0.06% and 0.1%, comparable to Monte Carlo noise. The paper also identifies a rare but severe failure mode: 31 test situations where an observation has full leverage and positive partial leverage cause average excess above 20% for every method except JK-H, HC1-PL, and HC2-PL, and shows that assigning the homoskedastic error estimate to such observations instead of zero reduces the excess to about 3%.","pith_inferences":["Because the paper's central quantity is the inverse Herfindahl of partial leverages, applying the same construction to cluster partial leverages predicts that a cluster-level $\\tilde{G}_k - 1$ degree of freedom will beat the default $G - 1$; the paper notes this extension is planned.","The paper's DGP selection never validates that the fitted random-forest variances match true error variances; a holdout validation comparing predicted and realized residual variances would tell whether HC1's excess is an artifact of the residual-scaling step.","The paper averages excess and lack across all test situations; if one instead minimizes worst-case rejection rates, the 31 full-leverage cases suggest reporting $\\tilde{n}_k$ for every coefficient as a routine diagnostic, which software could automate.","The JK-H and HC2-PL average excess figures (0.06% vs 0.1%) differ by less than the 0.09% Monte Carlo noise floor, so their tie may not persist under a larger sample of packages."],"forward_implications":["If applied researchers switched from the default HC1 specification to HC2-PL or JK-H, the average excess rejection rate in samples up to 1,000 observations would fall from roughly 1.6% to near 0.1%, so many p-values just below 5% would cease to be significant.","The partial-leverage adjustment needs only FWL residuals, so it is computable in regressions with absorbed fixed effects, where HC2-family hat values are harder to obtain; HC1-PL is thus a practical option for that common setting.","For the small set of regressions with full leverage and positive partial leverage, all standard methods except JK-H, HC1-PL, and HC2-PL over-reject at rates above 20%, and replacing the zero error estimate with the homoskedastic estimate brings excess down to about 3%.","Customized degrees of freedom matter more than the standard-error formula: HC2-PL and HC2-BM both use HC2-style variance estimates but outperform HC2 with default degrees of freedom.","Wild-bootstrap t-tests improve on plain HC1 and HC2, but in the paper's simulations they still show higher average excess than HC2-BM, JK-H, and HC2-PL."],"supporting_citations":[{"why":"Introduces HC1, HC2, and HC3, the baseline standard-error formulas the paper compares and extends.","marker":"MacKinnon and White (1985)"},{"why":"Proposes the degree-of-freedom adjustment used by HC2-BM, one of the paper's best-performing methods.","marker":"Bell and McCaffrey (2002)"},{"why":"Proposes the jackknife estimator JK-H, a top performer in the Monte Carlo study.","marker":"Hansen (2024)"},{"why":"Popularizes HC2 with adjusted degrees of freedom and the practice of reporting adjusted standard errors.","marker":"Imbens and Kolesar (2016)"},{"why":"Provides the unbiasedness theory for HC2 and the Moore-Penrose convention the paper relies on for full-leverage observations.","marker":"Pustejovsky and Tipton (2017)"},{"why":"Recommends HC3 over HC1 for small samples, the practical claim the paper tests against a large sample of real regressions.","marker":"Long and Ervin (2000)"},{"why":"Introduces HC4, included among the HC-family comparisons.","marker":"Cribari-Neto (2004)"},{"why":"Gives the variance-combination approximation used to motivate the partial-leverage degrees of freedom.","marker":"Satterthwaite (1946)"},{"why":"Establishes consistency of the heteroskedasticity-robust variance estimator, the foundation for all HC methods considered.","marker":"White (1980)"},{"why":"Conducts a large Monte Carlo study based on reproduction packages, the approach this paper scales up.","marker":"Young (2022)"}],"fun_headline_variants":["Default robust errors over-reject; df adjustment fixes","Robust inference: degrees of freedom, not just error formula","Data-driven df tests beat Stata's HC1-HC4 in real regressions","Full leverage breaks robust tests; new df methods save them","JK-H and HC2-PL cut false rejections to near noise levels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes that the random-forest-fitted error variances for the 608 reproducible regressions approximate the true error variances in real economics data, and that those 155 packages represent the broader population; if either assumption fails, the reported ordering of methods could change.","fun_headline_variants_meta":{"raw":{"variants":["Default robust errors over-reject; df adjustment fixes","Robust inference: degrees of freedom, not just error formula","Data-driven df tests beat Stata's HC1-HC4 in real regressions","Full leverage breaks robust tests; new df methods save them","JK-H and HC2-PL cut false rejections to near noise levels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1557,"prompt_tokens":926,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":540}},"tokens_in":542,"tokens_out":631,"duration_ms":24504,"temperature":1.0,"reasoning_tokens":540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:55:31.746122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Monte Carlo on the same 608 regressions with a different error-variance fitting rule---leave-one-out squared residuals or a spline-based FGLS instead of the selected random-forest candidate---and check whether HC1's average excess still exceeds JK-H's and HC2-PL's by the reported margin.","supporting_citations":[],"review_version":1}