{"id":"a93d5987-a2d7-46be-bb78-cafe69bf25c1","arxiv_id":"2608.02909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Two predictions for the same latent concept, generated on independent splits of one dataset, can serve as instruments for each other and reduce measurement-error bias when prediction-generated variables are used as regressors.","lead":"This paper proposes a simple instrumental-variables fix for regressions that use machine learning or LLM-generated predictions as explanatory variables: split the data, generate the same measure on two independent subsamples, and use one predicted measure as an instrument for the other. The method needs no new data and, in simulations and two replications, produces estimates that are larger and closer to the truth than the naive approach.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Nearest-neighbor implementation requires E[ε|Z]=0, not just E[ε|X]=0; without it the instrument is invalid and the empirical claims are unsupported.","rationale":"The paper's headline contribution is that a researcher can fix measurement error without new data. That is true only if either exact matches on Z are available (rare in continuous text settings) or E[ε|Z]=0 holds. The latter is a substantive behavioral assumption about the outcome equation, not about measurement error; it is not part of Assumption 1 and is not defended. Since both empirical applications and the algorithm use approximate matching, the load-bearing theorem is Proposition 2, and its extra condition is the weak link. I agree with the reader that Assumption 1 (idiosyncratic measurement error) is fragile and is conceded by the paper; my concern is distinct because it operates even when measurement errors are perfectly independent. The reader's point about Lin matching on the noisy regressor is related, but the direct-effect channel is more general and affects the Ash application as well. The paper deserves credit for a clean Proposition 1 and for simulation designs that probe correlated errors; however, the nearest-neighbor extension is not fully identified under the stated assumptions. A CONDITIONAL verdict remains appropriate: the authors should either restrict the practical algorithm to exact or coarsened-exact matching on Z, or explicitly adopt and defend E[ε|Z]=0, and then rerun the applications under that condition. Hence I keep the reader's CONDITIONAL verdict, which corresponds to UNCHANGED in the schema, but with the justification shifted toward this additional identifying assumption.","tokens_in":21729,"tokens_out":10222,"duration_ms":95039,"concrete_test":"Run a Monte Carlo with Z=(Z1,Z2), X=Φ(aZ1+bZ2), Y=β0+β1X+ε, where ε=c(Z1−E[Z1|X])+ν and ν is iid N(0,1). Then E[ε|X]=0 by construction but E[ε|Z]≠0. Draw iid η_i and generate Xhat_i=X_i+η_i. Apply the paper's algorithm: random split, nearest-neighbor matching on Z, using Xhat_instr=f(Z_nn)+η_nn as an instrument for Xhat_i, with 500 replications at n=20,000 and the paper's noise levels. If the IV estimator is biased away from β1 or its 95% CI coverage is below nominal, Proposition 2's exogeneity fails exactly when E[ε|Z]≠0. As a control, repeat with exact matching Z_nn=Z_i (or with ε_i independent of Z_i); the IV should be unbiased, isolating the imperfect-match channel. Also rerun the Ash reanalysis with the matching SVD dimensions added as controls; if the IV coefficients move materially, direct dependence of reactions on text is contaminating the instrument.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 1 is internally correct for perfect matches, but the algorithm in §3.4 and both applications rely on nearest-neighbor (imperfect) matching, so Proposition 2 is the operative theorem. Its proof in Appendix A establishes Cov(f(Z_nn), ε_i)=0 by invoking the 'strengthened assumption' E[ε_i | Z_i]=0 (Section 3.3.1). This is not implied by the maintained E[ε_i | X_i]=0 when X_i=f(Z_i): unstructured covariates Z_i can affect Y_i beyond the scalar measure X_i, so ε_i may correlate with Z_i even if it is mean-independent of X_i. Because the matched unit's Z_nn is chosen to be close to Z_i, f(Z_nn) is correlated with Z_i and therefore with ε_i, so the exclusion restriction fails. Text applications are the canonical case: speech content affects reactions beyond the LDA gender score, and Q&A text affects firm spending beyond the BERT-based risk index. The paper flags E[ε|Z]=0 only in passing and lists correlated measurement error, not this direct-dependence channel, as the key limitation in §3.5.1. The simulations in Section 4 use ε_i independent of Z_i, so they cannot detect this failure. Thus the central claim is established for the perfect-match benchmark but not for the feasible estimator actually recommended; Lin's matching on noisy PRIs adds a separate violation of Assumption 2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a split-sample instrumental-variables correction for measurement error in prediction-generated regressors. The idea is to split the data into two independent samples, generate a predicted measure for each observation and for a matched observation in the second sample, and use the matched prediction as an instrument for the original noisy regressor. The paper states two theoretical results (Proposition 1 for perfect matching, Proposition 2 for imperfect matching), provides a nearest-neighbor algorithm, reports Monte Carlo simulations, and re-analyzes two published applications (Ash et al. 2025 on gendered speech; Lin 2025 on political risk). The central claim is that researchers can correct downstream attenuation bias without collecting new data or labels.","tokens_in":1553,"tokens_out":1518,"duration_ms":70423,"significance":"If the required identifying conditions hold, the proposed method is simple, computationally cheap, and potentially valuable: it requires no validation labels, no new data, and only standard IV estimation. The theoretical propositions are clearly stated and the proofs are readable, and the simulation design is a serious attempt to mimic text-as-data settings. However, the feasible estimator is justified only under a strengthened exogeneity condition, E[epsilon_i | Z_i] = 0, that is not implied by the paper's maintained assumption E[epsilon_i | X_i] = 0, and the Lin re-analysis appears to match on the noisy regressor itself, which violates Assumption 2. These issues are central to the paper's empirical claims, so the manuscript needs substantial revision before the advertised scope is supported.","major_comments":[{"comment":"Proposition 2, which is the operative result for the nearest-neighbor algorithm in Section 3.4 and for both applications, establishes exogeneity only under the strengthened condition E[epsilon_i | Z_i] = 0. This is not implied by the maintained assumption E[epsilon_i | X_i] = 0 in equation (3.2), because X_i = f(Z_i) is a scalar function of a potentially much richer Z_i. In the two applications this is not a remote possibility: the content of a speech can affect reactions beyond its LDA-based gender score, and the content of investor-firm Q&As can affect firm spending beyond a BERT-based political-risk index. Section 3.5.1 lists correlated measurement errors and spillovers as limitations but does not identify this direct-dependence channel. The simulations in Section 4 generate epsilon_i independently of Z_i, so they cannot detect failures of E[epsilon_i | Z_i] = 0; the correlated-noise row in Table 2 is a different violation. As written, the paper's central claim is established for the perfect-match benchmark but not for the feasible procedure actually recommended.","section":"Section 3.3.1; Appendix A, Proposition 2"},{"comment":"The Lin re-analysis matches observations on topic-specific PRIs, which are the noisy prediction-generated regressors themselves, rather than on raw features Z_i. Assumption 2 requires the matching rule to be a function of {Z_i} and the splitting randomization U only. Matching on Xhat makes the match depend on measurement errors eta_i and eta_j, so even if eta_i are independent across observations, the selected pairs can have correlated measurement errors and Cov(Xhat_instr_i, u_i) need not be zero. The statement in Section 5.2 that the topic-specific PRIs are 'a function of the textual data' does not resolve this, because the PRIs are noisy functions that include measurement error. The application should either match on raw text features or on functions of Z_i that are verifiably free of eta, or the analysis should explicitly justify why matching on the noisy regressor does not violate the assumptions.","section":"Section 5.2; Assumption 2"},{"comment":"The simulation evidence does not cover the assumption that is actually needed for Proposition 2. In all baseline designs epsilon_i is drawn independently of the entire vector Z_i, so E[epsilon_i | Z_i] = 0 holds by construction, and the nearest-neighbor matching is performed in the embedding space E, which is a deterministic function of Z_i. The 'correlated noise across documents' row in Table 2 perturbs the shared coefficient vector delta, which is a common-shock violation of Assumption 1, not a violation of E[epsilon_i | Z_i] = 0. A simulation in which epsilon_i depends on Z_i (for example, through a component of Z_i not captured by X_i) while E[epsilon_i | X_i] = 0 still holds would directly test the feasibility of the recommended procedure, and such a design should be added.","section":"Section 4; Table 2"}],"minor_comments":[{"comment":"The abstract states the approach 'recovers estimates close to the true values... even in relatively small samples,' but Table 1 reports that at n = 5,000 with low noise the IV RMSE (0.248) exceeds the OLS RMSE (0.177). The note acknowledges this, but the abstract should be qualified to reflect the finite-sample trade-off.","section":"Abstract; Table 1"},{"comment":"The claim that the violations are 'very mild, under 1% in the Low Noise case' is vague; please report the actual metric (for example, the maximum relative difference between eta_i and tilde-eta_i, or the correlation between the two).","section":"Section 4.1.1, footnote 10"},{"comment":"The IV columns use roughly half the observations of the OLS columns (4,652 versus 9,466), but the paper does not discuss the efficiency loss or the implications for the clustered standard errors beyond reporting them. A brief discussion would help readers assess the cost of the correction.","section":"Section 5.2, Table 4"},{"comment":"In equation (A.1) the notation uses the same symbol for the observation and its mean (the numerator and denominator are written with Xhat_i - Xhat_i rather than Xhat_i - bar-Xhat). This is a typographical issue but should be fixed for clarity.","section":"Appendix A, proof of Lemma A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful core idea and the perfect-match result is clean, but the feasible estimator relies on a stronger exogeneity condition than the paper's main assumption, and one application appears to match on the endogenous noisy regressor. I believe the issues are fixable by restating the identifying assumptions, adding the relevant simulations, and revising the Lin application, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe split-sample IV idea is genuinely new and the benchmark theory is correct. The specific construction—split the data, match nearest neighbors on features, use the matched observation's prediction as an instrument—is not in the prior IV or LLM-measurement literature. Proposition 1 is clean, and the simulation design is informative; Table 2 row 2 shows the method honestly degrading when errors are common, which is good practice.\n\nThe soft spot is between the perfect-match proposition and the actual algorithm. For imperfect matches (nearest neighbor), the exclusion restriction requires E[ε_i|Z_i] = 0, not just E[ε_i|X_i] = 0. In text applications that is a big ask: the document or Q&A content almost certainly affects the outcome beyond the scalar measure. Ash et al.'s speech content affects reactions beyond the LDA gender score; Lin's Q&A content affects firm spending beyond the BERT risk index. If Z_i matters directly for Y_i, then the matched prediction f(Z_nn(i)) is correlated with ε_i and the instrument fails. The paper states the strengthened assumption in §3.3.1 but does not flag this direct-dependence channel in §3.5.1, where the limitations are listed as correlated measurement error. The simulations cannot detect it because ε_i is independent of Z_i by construction.\n\nSeparately, the Lin replication matches on the topic-specific PRIs—the noisy regressor—rather than on features. That violates Assumption 2, which requires the matching rule to be a function of Z only. The paper acknowledges this in footnote 17 but handwaves. The Ash replication at least matches on an SVD of the document-term matrix, though it is still subject to the E[ε|Z] problem. No code or data is provided to verify the replication tables, which matters because the empirical claims rest on those magnitudes.\n\nThese are not minor quibbles. The central promise—a cheap fix with no validation data—holds only when measurement errors are idiosyncratic and the scalar measure captures all dependence between Z and Y. That is a narrow setting, and the paper does not state it clearly enough.\n\nI would send this to a serious referee. The idea is novel, the benchmark theory is right, and the simulations are worth having. But it needs substantial revision: either restrict the claims to the perfect-match case or develop the imperfect-match theory under E[ε|Z]=0, redo the Lin matching on raw features, and provide code and data.","headline":"Split-sample IV is a genuinely new idea and the perfect-match theory is clean, but the feasible nearest-neighbor estimator requires E[ε|Z]=0, which is unlikely in text applications, and the Lin replication matches on the noisy regressor itself.","tokens_in":22499,"tokens_out":4446,"would_cite":false,"duration_ms":37984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P20","62J05","62F12"],"pacs":[],"model":"deepseek-v4-flash","headline":"Prediction-generated regressors are biased, but a split-sample instrument built from the same data recovers the true coefficient.","keywords":["measurement error","generated regressors","instrumental variables","sample splitting","large language models","text-as-data","attenuation bias","nearest-neighbor matching"],"falsifier":"On a dataset with known true X_i, generate predictions under two error regimes: independent per-document noise and a common shock applied to all predictions, as in the paper's simulation where IV RMSE rises to 1.425 against OLS's 1.502 at high noise. The split-sample IV should recover the truth only in the first regime; if it also fails in the first regime, Assumption 1 or the matching step is the problem.","tokens_in":21466,"feed_emoji":"📊","tokens_out":7411,"duration_ms":65217,"temperature":0.7,"pith_summary":"Papers increasingly feed machine-learning predictions—from LLMs, topic models, or classifiers—into regressions as explanatory variables, but these predictions carry measurement error that biases the estimates. This paper proposes a correction that needs no new data: split the sample randomly, produce an independent prediction for each observation's nearest neighbor in the other split, and use that second prediction as an instrument for the first. The paper proves the instrument is valid when prediction errors are independent across observations and mean-zero given the features, and shows in simulations that the IV estimates land near the truth while ordinary least squares stays biased even as sample size grows. In two published re-analyses, the corrected effects are between roughly 30% and 100% larger than the original estimates.","feed_headline":"Split-sample predictions fix biased regressions from AI text measures","feed_subtitle":"Use one noisy AI prediction to instrument another — no new data, just independent sample splits.","key_machinery":"The load-bearing object is the split-sample instrument pair (Xhat_i, Xhat_instr_i), produced by random sample splitting plus nearest-neighbor matching in feature or embedding space. It works because the covariance factorization Cov(Xhat_i, Xhat_instr_i) = Var(X_i) > 0 while Cov(u_i, Xhat_instr_i) = 0, separating the shared latent signal from independent errors. The randomized split is not a computational convenience; it is what makes the measurement errors across the two measures independent, which is exactly what turns the second prediction into an exogenous instrument.","core_discovery":"Proposition 1 is the paper's central claim: if the same dataset is split into two independent samples and each observation i is matched to a unit nn(i) in the instrument sample with identical features (Z_nn(i) = Z_i), then the prediction-generated regressor from the matched unit, Xhat_instr_i, is a valid instrument for the original noisy regressor Xhat_i. The covariance argument is that both predictions share the latent X_i, so they are correlated (relevance), while the measurement errors eta_i and eta_nn(i) are independent by Assumption 1 and the random split in Assumption 2, so Cov(u_i, Xhat_instr_i) = 0 (exogeneity). Proposition 2 relaxes perfect matching: exogeneity survives imperfect matches under E[eps_i | Z_i] = 0, and only first-stage strength is affected. The paper also shows that OLS bias from prediction-generated regressors is not classical attenuation—it can inflate as well as deflate estimates—and that the IV estimator is consistent under unique matching.","pith_inferences":["The same instrument-validity argument should transfer to any two conditionally independent noisy measures of the same latent construct—two LLM prompts, two models, or two human coders—even though the paper frames the procedure as sample splitting.","Swapping the labels of the two splits and averaging the two IV estimates would use all outcome data, a simple efficiency gain the paper mentions but does not implement or study.","A useful by-product is a specification diagnostic: when Assumption 1 holds, a large gap between OLS and IV signals meaningful measurement error, but when the gap is small one cannot distinguish negligible error from a violation of the independence assumption.","Extending the split-sample instrument to nonlinear downstream models (logit, Poisson, duration) would require exclusion restrictions beyond the linear covariance argument, so the linear setting is likely the safe scope."],"forward_implications":["Researchers who use LLM, topic-model, or classifier outputs as regressors can correct the resulting bias using only the data they already have, by splitting it and instrumenting one prediction with another.","The OLS bias from prediction-generated regressors is not classical attenuation: it can inflate as well as shrink coefficients, so both sign and magnitude corrections are needed.","In simulations, the IV estimator stays centered on the true coefficient while OLS remains biased as the sample grows; IV RMSE falls with sample size in all but the smallest, lowest-noise setting.","In the German Parliament re-analysis, the negative effect of predicted gender on reactions and the positive interaction with speaker gender become 50–100% larger under IV, implying the original published magnitudes understated the role-congruity effect.","In the China re-analysis, the interaction between political risk and the post-2015 period remains positive and significant and is about 30% larger, equivalent to roughly $10,600 in average expenditure."],"supporting_citations":[{"why":"Supplies the repeated-measurement-as-instrument logic from experiments that the paper adapts to sample splitting.","marker":"Gillen et al. (2019)"},{"why":"Canonical use of multiple measures for the same latent variable as instruments, which motivates the paper's two-measures design.","marker":"Ashenfelter and Krueger (1994)"},{"why":"Provides the two-sample instrumental variables framework that the split-sample estimator builds on.","marker":"Angrist and Krueger (1992)"},{"why":"Extends the two-sample IV framework and underlies the paper's consistency argument with data from independent splits.","marker":"Inoue and Solon (2010)"},{"why":"Supplies the textbook treatment of measurement error bias and the standard IV estimator used throughout the paper.","marker":"Wooldridge (2016)"},{"why":"A prior IV-based correction for ML-generated variables that requires a labeled subset; the paper's method is contrasted with it.","marker":"Yang et al. (2022)"},{"why":"An ensemble-based IV approach for ML-generated variables that also requires labeled data; serves as a comparison point.","marker":"Burtch et al. (2026)"},{"why":"Provides the first empirical application re-analyzed, including the data, specification, and baseline estimates that the IV correction is applied to.","marker":"Ash et al. (2025)"},{"why":"Provides the second empirical application re-analyzed, including the firm-level Political Risk Index and the difference-in-differences baseline.","marker":"Lin (2025)"}],"fun_headline_variants":["Split-sample IV fixes bias when AI predictions become regressors","Independent splits make noisy AI predictions valid instruments","No new data needed: split-sample instruments correct AI prediction bias","Matching on features, split samples: unbiased inference from noisy predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The correction collapses if prediction errors are not independent across observations—for example, if the LLM or classifier has a systematic bias shared across documents, or errors are correlated through space, time, or networks.","fun_headline_variants_meta":{"raw":{"variants":["Split-sample IV fixes bias when AI predictions become regressors","Independent splits make noisy AI predictions valid instruments","No new data needed: split-sample instruments correct AI prediction bias","Matching on features, split samples: unbiased inference from noisy predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1690,"prompt_tokens":926,"completion_tokens":764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":542,"tokens_out":764,"duration_ms":7454,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:56:02.055026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with known true X_i, generate predictions under two error regimes: independent per-document noise and a common shock applied to all predictions, as in the paper's simulation where IV RMSE rises to 1.425 against OLS's 1.502 at high noise. The split-sample IV should recover the truth only in the first regime; if it also fails in the first regime, Assumption 1 or the matching step is the problem.","supporting_citations":[{"cited_title":"Sociological Methods & Research , pages=","cited_arxiv_id":null,"evidence_quote":"Provides the second empirical application re-analyzed, including the firm-level Political Risk Index and the difference-in-differences baseline."}],"review_version":1}