When Predictions Become Regressors: A Split-Sample Correction for Biases in Downstream Inference

Nathan Canen; Ted Enamorado

REVIEW 3 major objections 4 minor 32 references

When Predictions Become Regressors: A Split-Sample Correction for Biases in Downstream Inference

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prediction-generated regressors are biased, but a split-sample instrument built from the same data recovers the true coefficient.

desk verdict Split-sample IV is a genuinely new idea and the perfect-match theory is clean, but the feasible nearest-neighbor estimator requires E[ε|Z]=0, which is unlikely in text applications, and the Lin replication matches on the noisy regressor itself. read the letter →

arxiv 2608.02909 v1 pith:PVQULKHI submitted 2026-08-03 econ.EM stat.ML

Nathan Canen , Ted Enamorado This is my paper · ORCID

classification econ.EMstat.ML MSC 62P2062J0562F12

keywords measurementerrorgeneratedregressorsinstrumentalvariablessamplesplittinglargelanguagemodelstext-as-dataattenuationbiasnearest-neighbormatching

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Papers increasingly feed machine-learning predictions—from LLMs, topic models, or classifiers—into regressions as explanatory variables, but these predictions carry measurement error that biases the estimates. This paper proposes a correction that needs no new data: split the sample randomly, produce an independent prediction for each observation's nearest neighbor in the other split, and use that second prediction as an instrument for the first. The paper proves the instrument is valid when prediction errors are independent across observations and mean-zero given the features, and shows in simulations that the IV estimates land near the truth while ordinary least squares stays biased even as sample size grows. In two published re-analyses, the corrected effects are between roughly 30% and 100% larger than the original estimates.

What carries the argument

The load-bearing object is the split-sample instrument pair (Xhat_i, Xhat_instr_i), produced by random sample splitting plus nearest-neighbor matching in feature or embedding space. It works because the covariance factorization Cov(Xhat_i, Xhat_instr_i) = Var(X_i) > 0 while Cov(u_i, Xhat_instr_i) = 0, separating the shared latent signal from independent errors. The randomized split is not a computational convenience; it is what makes the measurement errors across the two measures independent, which is exactly what turns the second prediction into an exogenous instrument.

What would settle it

On a dataset with known true X_i, generate predictions under two error regimes: independent per-document noise and a common shock applied to all predictions, as in the paper's simulation where IV RMSE rises to 1.425 against OLS's 1.502 at high noise. The split-sample IV should recover the truth only in the first regime; if it also fails in the first regime, Assumption 1 or the matching step is the problem.

Watch

Extended reading notes

Core claim

Proposition 1 is the paper's central claim: if the same dataset is split into two independent samples and each observation i is matched to a unit nn(i) in the instrument sample with identical features (Z_nn(i) = Z_i), then the prediction-generated regressor from the matched unit, Xhat_instr_i, is a valid instrument for the original noisy regressor Xhat_i. The covariance argument is that both predictions share the latent X_i, so they are correlated (relevance), while the measurement errors eta_i and eta_nn(i) are independent by Assumption 1 and the random split in Assumption 2, so Cov(u_i, Xhat_instr_i) = 0 (exogeneity). Proposition 2 relaxes perfect matching: exogeneity survives imperfect matches under E[eps_i | Z_i] = 0, and only first-stage strength is affected. The paper also shows that OLS bias from prediction-generated regressors is not classical attenuation—it can inflate as well as deflate estimates—and that the IV estimator is consistent under unique matching.

Load-bearing premise

The correction collapses if prediction errors are not independent across observations—for example, if the LLM or classifier has a systematic bias shared across documents, or errors are correlated through space, time, or networks.

Editorial extensions

If this is right

Researchers who use LLM, topic-model, or classifier outputs as regressors can correct the resulting bias using only the data they already have, by splitting it and instrumenting one prediction with another.
The OLS bias from prediction-generated regressors is not classical attenuation: it can inflate as well as shrink coefficients, so both sign and magnitude corrections are needed.
In simulations, the IV estimator stays centered on the true coefficient while OLS remains biased as the sample grows; IV RMSE falls with sample size in all but the smallest, lowest-noise setting.
In the German Parliament re-analysis, the negative effect of predicted gender on reactions and the positive interaction with speaker gender become 50–100% larger under IV, implying the original published magnitudes understated the role-congruity effect.
In the China re-analysis, the interaction between political risk and the post-2015 period remains positive and significant and is about 30% larger, equivalent to roughly $10,600 in average expenditure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same instrument-validity argument should transfer to any two conditionally independent noisy measures of the same latent construct—two LLM prompts, two models, or two human coders—even though the paper frames the procedure as sample splitting.
Swapping the labels of the two splits and averaging the two IV estimates would use all outcome data, a simple efficiency gain the paper mentions but does not implement or study.
A useful by-product is a specification diagnostic: when Assumption 1 holds, a large gap between OLS and IV signals meaningful measurement error, but when the gap is small one cannot distinguish negligible error from a violation of the independence assumption.
Extending the split-sample instrument to nonlinear downstream models (logit, Poisson, duration) would require exclusion restrictions beyond the linear covariance argument, so the linear setting is likely the safe scope.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The split-sample IV idea is genuinely new and the benchmark theory is correct. The specific construction—split the data, match nearest neighbors on features, use the matched observation's prediction as an instrument—is not in the prior IV or LLM-measurement literature. Proposition 1 is clean, and the simulation design is informative; Table 2 row 2 shows the method honestly degrading when errors are common, which is good practice.

The soft spot is between the perfect-match proposition and the actual algorithm. For imperfect matches (nearest neighbor), the exclusion restriction requires E[ε_i|Z_i] = 0, not just E[ε_i|X_i] = 0. In text applications that is a big ask: the document or Q&A content almost certainly affects the outcome beyond the scalar measure. Ash et al.'s speech content affects reactions beyond the LDA gender score; Lin's Q&A content affects firm spending beyond the BERT risk index. If Z_i matters directly for Y_i, then the matched prediction f(Z_nn(i)) is correlated with ε_i and the instrument fails. The paper states the strengthened assumption in §3.3.1 but does not flag this direct-dependence channel in §3.5.1, where the limitations are listed as correlated measurement error. The simulations cannot detect it because ε_i is independent of Z_i by construction.

Separately, the Lin replication matches on the topic-specific PRIs—the noisy regressor—rather than on features. That violates Assumption 2, which requires the matching rule to be a function of Z only. The paper acknowledges this in footnote 17 but handwaves. The Ash replication at least matches on an SVD of the document-term matrix, though it is still subject to the E[ε|Z] problem. No code or data is provided to verify the replication tables, which matters because the empirical claims rest on those magnitudes.

These are not minor quibbles. The central promise—a cheap fix with no validation data—holds only when measurement errors are idiosyncratic and the scalar measure captures all dependence between Z and Y. That is a narrow setting, and the paper does not state it clearly enough.

I would send this to a serious referee. The idea is novel, the benchmark theory is right, and the simulations are worth having. But it needs substantial revision: either restrict the claims to the perfect-match case or develop the imperfect-match theory under E[ε|Z]=0, redo the Lin matching on raw features, and provide code and data.

Referee Report

3 major / 4 minor

Summary. The paper proposes a split-sample instrumental-variables correction for measurement error in prediction-generated regressors. The idea is to split the data into two independent samples, generate a predicted measure for each observation and for a matched observation in the second sample, and use the matched prediction as an instrument for the original noisy regressor. The paper states two theoretical results (Proposition 1 for perfect matching, Proposition 2 for imperfect matching), provides a nearest-neighbor algorithm, reports Monte Carlo simulations, and re-analyzes two published applications (Ash et al. 2025 on gendered speech; Lin 2025 on political risk). The central claim is that researchers can correct downstream attenuation bias without collecting new data or labels.

Significance. If the required identifying conditions hold, the proposed method is simple, computationally cheap, and potentially valuable: it requires no validation labels, no new data, and only standard IV estimation. The theoretical propositions are clearly stated and the proofs are readable, and the simulation design is a serious attempt to mimic text-as-data settings. However, the feasible estimator is justified only under a strengthened exogeneity condition, E[epsilon_i | Z_i] = 0, that is not implied by the paper's maintained assumption E[epsilon_i | X_i] = 0, and the Lin re-analysis appears to match on the noisy regressor itself, which violates Assumption 2. These issues are central to the paper's empirical claims, so the manuscript needs substantial revision before the advertised scope is supported.

major comments (3)

[Section 3.3.1; Appendix A, Proposition 2] Proposition 2, which is the operative result for the nearest-neighbor algorithm in Section 3.4 and for both applications, establishes exogeneity only under the strengthened condition E[epsilon_i | Z_i] = 0. This is not implied by the maintained assumption E[epsilon_i | X_i] = 0 in equation (3.2), because X_i = f(Z_i) is a scalar function of a potentially much richer Z_i. In the two applications this is not a remote possibility: the content of a speech can affect reactions beyond its LDA-based gender score, and the content of investor-firm Q&As can affect firm spending beyond a BERT-based political-risk index. Section 3.5.1 lists correlated measurement errors and spillovers as limitations but does not identify this direct-dependence channel. The simulations in Section 4 generate epsilon_i independently of Z_i, so they cannot detect failures of E[epsilon_i | Z_i] = 0; the correlated-noise row in Table 2 is a different violation. As written, the paper's central claim is established for the perfect-match benchmark but not for the feasible procedure actually recommended.
[Section 5.2; Assumption 2] The Lin re-analysis matches observations on topic-specific PRIs, which are the noisy prediction-generated regressors themselves, rather than on raw features Z_i. Assumption 2 requires the matching rule to be a function of {Z_i} and the splitting randomization U only. Matching on Xhat makes the match depend on measurement errors eta_i and eta_j, so even if eta_i are independent across observations, the selected pairs can have correlated measurement errors and Cov(Xhat_instr_i, u_i) need not be zero. The statement in Section 5.2 that the topic-specific PRIs are 'a function of the textual data' does not resolve this, because the PRIs are noisy functions that include measurement error. The application should either match on raw text features or on functions of Z_i that are verifiably free of eta, or the analysis should explicitly justify why matching on the noisy regressor does not violate the assumptions.
[Section 4; Table 2] The simulation evidence does not cover the assumption that is actually needed for Proposition 2. In all baseline designs epsilon_i is drawn independently of the entire vector Z_i, so E[epsilon_i | Z_i] = 0 holds by construction, and the nearest-neighbor matching is performed in the embedding space E, which is a deterministic function of Z_i. The 'correlated noise across documents' row in Table 2 perturbs the shared coefficient vector delta, which is a common-shock violation of Assumption 1, not a violation of E[epsilon_i | Z_i] = 0. A simulation in which epsilon_i depends on Z_i (for example, through a component of Z_i not captured by X_i) while E[epsilon_i | X_i] = 0 still holds would directly test the feasibility of the recommended procedure, and such a design should be added.

minor comments (4)

[Abstract; Table 1] The abstract states the approach 'recovers estimates close to the true values... even in relatively small samples,' but Table 1 reports that at n = 5,000 with low noise the IV RMSE (0.248) exceeds the OLS RMSE (0.177). The note acknowledges this, but the abstract should be qualified to reflect the finite-sample trade-off.
[Section 4.1.1, footnote 10] The claim that the violations are 'very mild, under 1% in the Low Noise case' is vague; please report the actual metric (for example, the maximum relative difference between eta_i and tilde-eta_i, or the correlation between the two).
[Section 5.2, Table 4] The IV columns use roughly half the observations of the OLS columns (4,652 versus 9,466), but the paper does not discuss the efficiency loss or the implications for the clustered standard errors beyond reporting them. A brief discussion would help readers assess the cost of the correction.
[Appendix A, proof of Lemma A.1] In equation (A.1) the notation uses the same symbol for the observation and its mean (the numerator and denominator are written with Xhat_i - Xhat_i rather than Xhat_i - bar-Xhat). This is a typographical issue but should be fixed for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's split-sample IV validity claim is a conditional identification theorem, with no fitted parameter or self-citation chain carrying the derivation.

full rationale

The central theoretical claim, Proposition 1, is explicitly conditional: under Assumptions 1-2, Cov(Xhat_i, Xhat_instr_i) != 0 and Cov(u_i, Xhat_instr_i) = 0. The appendix proof expands these covariances and sets each term to zero using exactly the stated independence, mean-zero, and randomization assumptions; relevance follows from Var(X_i)>0. This is a standard identification theorem, not an empirical prediction fitted to data. The simulations use an external DGP with a known beta_1=-2 and compare estimators to that truth; no parameter is calibrated to reverse-engineer the conclusion. The empirical applications re-estimate published specifications and report larger magnitudes, which are falsifiable external claims. The only self-citation, Bosley et al. (2025), is used for the text-simulation design and is not load-bearing for the identification argument. The paper explicitly identifies its key limitation, correlated measurement errors, in Section 3.5.1 and shows in Table 2 Row 2 that a common shock collapses IV RMSE toward OLS RMSE; that is an honest statement of scope, not circularity. Proposition 2's strengthened assumption E[epsilon_i|Z_i]=0 is stated explicitly in Section 3.3.1 and used directly in the proof, rather than being smuggled in or imported from a self-citation. The concerns raised in the skeptic note are about assumption validity and estimator feasibility, not about the derivation reducing to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests almost entirely on independence and exogeneity assumptions, not on fitted constants or new entities. The main fragility is Assumption 1: if prediction errors are correlated across observations or exhibit systematic model bias, the instrument is not exogenous. The Lin reapplication additionally uses matching on the noisy regressor, which is not covered by Assumption 2.

assumptions (5)

domain assumption Assumption 1: (Z_i, eta_i, epsilon_i) are independent across i, and E[eta_i | Z_i] = 0.
This is the core identification condition; it makes measurement errors for the training and instrument measures uncorrelated, so the instrument is exogenous. Stated in Section 3; failures discussed in Section 3.5.1.
domain assumption Assumption 2: the split randomization U is independent of {Z_i, eta_i, epsilon_i}, and the matching rule is a function of {Z_i} and U only.
Guarantees the instrument measure's error is independent of the training sample's error. Violated in the Lin reapplication where matching uses the noisy PRI itself.
domain assumption Outcome error exogeneity: E[epsilon_i | X_i] = 0, and E[epsilon_i | Z_i] = 0 in Proposition 2.
Isolates measurement error as the only source of endogeneity; standard in this literature and stated in Section 3 around equation (3.2).
domain assumption No systematic model error: the prediction model's population counterpart satisfies f(z) = E[Xhat_i | Z_i = z], so eta_i = xi_i.
If the LLM or topic model has systematic bias relative to the true latent variable, the IV cannot correct it; Section 3.1 redefines the target accordingly.
standard math Standard IV asymptotics with unique matches, so each instrument observation is used at most once.
Corollary 3.1 relies on this to invoke standard IV limit theory with i.i.d. data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Predictions Become Regressors: A Split-Sample Correction for Biases in Downstream Inference." pith.science (2026). https://pith.science/paper/PVQULKHI

@misc{pith2026260802909,
  author       = {Pith},
  title        = {Pith review of: When Predictions Become Regressors: A Split-Sample Correction for Biases in Downstream Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PVQULKHI}},
  note         = {Machine review of arXiv:2608.02909}
}

read the original abstract

Prediction-based methods, including Large Language Models (LLMs) and other machine learning techniques, are often used to construct measures of political phenomena that are difficult to quantify directly, such as policy positions in manifestos or emotions expressed on social media. In many applications, these prediction-generated measures are used as explanatory variables in regression models, even though they are measured with error. This leads to biased estimates. In this paper, we propose a simple solution to these biases: instrumental variables constructed from multiple measures created on independent splits of the original data. This approach is theoretically valid, easy to implement, and does not require new data. Through simulations, we show that this approach recovers estimates close to the true values, even in relatively small samples, while the standard approach can produce substantial bias in practice. We illustrate the method by revisiting two applications: whether gendered speech affects legislative outcomes in the German Parliament, and whether political risk influences poverty alleviation programs in China.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 24 canonical work pages

[1]

American Journal of Political Science , volume=

Gender and reactions to speeches in German parliamentary debates , author=. American Journal of Political Science , volume=. 2025 , publisher=

work page 2025
[2]

INFORMS Journal on Data Science , volume=

Achieving reliable causal inference with data-mined variables: A random forest approach to the measurement error problem , author=. INFORMS Journal on Data Science , volume=. 2022 , publisher=

work page 2022
[3]

Management Science , year=

EnsembleIV: Creating Instrumental Variables from Ensemble Learners for Robust Statistical Inference with ML-Generated Variables , author=. Management Science , year=

work page
[4]

Journal of the American statistical Association , volume=

The effect of age at school entry on educational attainment: an application of instrumental variables with moments from two samples , author=. Journal of the American statistical Association , volume=. 1992 , publisher=

work page 1992
[5]

The Review of Economics and Statistics , volume=

Two-sample instrumental variables estimators , author=. The Review of Economics and Statistics , volume=. 2010 , publisher=

work page 2010
[6]

Estimates of the Economic Return to Schooling from a New Sample of Twins , urldate =

Orley Ashenfelter and Alan Krueger , journal =. Estimates of the Economic Return to Schooling from a New Sample of Twins , urldate =

work page
[7]

The Economic Journal , volume=

Emotion and reason in political language , author=. The Economic Journal , volume=. 2022 , publisher=

work page 2022
[8]

Annual Review of Economics , volume=

Large language models: An applied econometric framework , author=. Annual Review of Economics , volume=. 2024 , publisher=

2024

Show all 32 references

[9]

Journal of Political Economy , volume=

Experimenting with measurement error: Techniques with applications to the caltech cohort study , author=. Journal of Political Economy , volume=. 2019 , publisher=

2019
[10]

Science Advances , volume=

How to make causal inferences using texts , author=. Science Advances , volume=. 2022 , publisher=

2022
[11]

, author=

Role congruity theory of prejudice toward female leaders. , author=. Psychological review , volume=. 2002 , publisher=

2002
[12]

Science , volume=

Prediction-powered inference , author=. Science , volume=. 2023 , publisher=

2023
[13]

Advances in Neural Information Processing Systems , volume=

Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models , author=. Advances in Neural Information Processing Systems , volume=
[14]

Sociological Methods & Research , pages=

Correcting the Measurement Errors of AI-Assisted Labeling in Image Analysis Using Design-Based Supervised Learning , author=. Sociological Methods & Research , pages=. 2025 , publisher=

2025
[15]

Econometrica , volume=

Measuring group differences in high-dimensional choices: method and application to congressional speech , author=. Econometrica , volume=. 2019 , publisher=

2019
[16]

2025 , institution=

Inference for Regression with Variables Generated by AI or Machine Learning , author=. 2025 , institution=

2025
[17]

The quarterly journal of economics , volume=

Measuring economic policy uncertainty , author=. The quarterly journal of economics , volume=. 2016 , publisher=

2016
[18]

Available at SSRN 7053778 , year=

Inference with AI-Generated Covariates , author=. Available at SSRN 7053778 , year=
[19]

Frontiers in Political Science , volume=

Large language models and political science , author=. Frontiers in Political Science , volume=. 2023 , publisher=

2023
[20]

arXiv preprint arXiv:2412.06864 , year=

Political-llm: Large language models in political science , author=. arXiv preprint arXiv:2412.06864 , year=

arXiv
[21]

Econometrica , volume=

Deep neural networks for estimation and inference , author=. Econometrica , volume=. 2021 , publisher=

2021
[22]

2025 , publisher=

Large Language Models: A Survey with Applications in Political Science , author=. 2025 , publisher=

2025
[23]

Journal of machine Learning research , volume=

Latent dirichlet allocation , author=. Journal of machine Learning research , volume=
[24]

Available at SSRN 4932600 , year=

Article-Level Slant and Polarization of News Consumption on Social Media , author=. Available at SSRN 4932600 , year=
[25]

2016 , publisher=

Introductory econometrics a modern approach , author=. 2016 , publisher=

2016
[26]

The Journal of Politics , volume=

Addressing Risk by Doing Good: Business Response to Government Policy Initiative , author=. The Journal of Politics , volume=. 2025 , publisher=

2025
[27]

The Annals of Applied Statistics , volume =

Taddy, Matt , title =. The Annals of Applied Statistics , volume =. 2015 , doi =

2015
[28]

and Yang, Eunho and Allen, Genevera I

Inouye, David I. and Yang, Eunho and Allen, Genevera I. and Ravikumar, Pradeep , title =. WIREs Computational Statistics , volume =. doi:https://doi.org/10.1002/wics.1398 , url =. https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/wics.1398 , year =

doi
[29]

, title =

Piantadosi, Steven T. , title =. Psychonomic Bulletin & Review , volume =. 2014 , doi =

2014
[30]

Efficient

LeCun, Yann and Bottou, L. Efficient. Neural Networks: Tricks of the Trade , series =. 1998 , doi =

1998
[31]

Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , title =. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages =. 2019...

2019
[32]

American Political Science Review , volume =

Bosley, Mitchell and Kuzushima, Saki and Enamorado, Ted and Shiraito, Yuki , title =. American Political Science Review , volume =

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

American Journal of Political Science , volume=

Gender and reactions to speeches in German parliamentary debates , author=. American Journal of Political Science , volume=. 2025 , publisher=

work page 2025

[2] [2]

INFORMS Journal on Data Science , volume=

Achieving reliable causal inference with data-mined variables: A random forest approach to the measurement error problem , author=. INFORMS Journal on Data Science , volume=. 2022 , publisher=

work page 2022

[3] [3]

Management Science , year=

EnsembleIV: Creating Instrumental Variables from Ensemble Learners for Robust Statistical Inference with ML-Generated Variables , author=. Management Science , year=

work page

[4] [4]

Journal of the American statistical Association , volume=

The effect of age at school entry on educational attainment: an application of instrumental variables with moments from two samples , author=. Journal of the American statistical Association , volume=. 1992 , publisher=

work page 1992

[5] [5]

The Review of Economics and Statistics , volume=

Two-sample instrumental variables estimators , author=. The Review of Economics and Statistics , volume=. 2010 , publisher=

work page 2010

[6] [6]

Estimates of the Economic Return to Schooling from a New Sample of Twins , urldate =

Orley Ashenfelter and Alan Krueger , journal =. Estimates of the Economic Return to Schooling from a New Sample of Twins , urldate =

work page

[7] [7]

The Economic Journal , volume=

Emotion and reason in political language , author=. The Economic Journal , volume=. 2022 , publisher=

work page 2022

[8] [8]

Annual Review of Economics , volume=

Large language models: An applied econometric framework , author=. Annual Review of Economics , volume=. 2024 , publisher=

2024