{"id":"281d109d-a360-46e9-b823-ab8d22437068","arxiv_id":"2411.09218","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Random splits of panel data into training and test sets cause temporal and cross-sectional leakage that overstates out-of-sample performance; the correct split depends on whether the task is cross-sectional prediction or sequential forecasting.","lead":"This paper shows that standard machine learning practices, such as random train-test splits, leak future or duplicate information when applied to panel data and inflate measured accuracy. It offers a goal-based framework and guidelines for choosing split and cross-validation strategies in panel data prediction problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; footnote 12's BEA revision assumption is a real caveat but any violation would bias the leakage estimate conservatively, so the central claim stands.","rationale":"The reader's weakest assumption correctly identifies footnote 12 as the most delicate empirical point: the claim that BEA revisions do not leak future information is an assertion about data provenance that is not verified in the paper. I agree that this deserves a more careful limitation statement. However, I do not view it as load-bearing for the central argument. If BEA revisions do contain future information, then both the random-split and time-split forecasting models are contaminated to some degree, and the estimated leakage inflation in Section 4 would be biased downward—making the paper's qualitative conclusion conservative rather than overturned. The conceptual taxonomy in Section 2 and the practical guidelines in Section 3 depend only on the structural distinction between observation-level, unit-level, group-level, and time-level splits, which is definitionally sound. The empirical demonstration is illustrative support, not the foundation of the claim. The remaining weaknesses—single test window, no standard errors around AUC or MSE differences—limit the precision of the quantitative illustration but do not threaten the central claim. I therefore keep the reader's CONDITIONAL verdict unchanged, while noting that the BEA caveat should be stated as a data limitation rather than a resolved factual matter.","tokens_in":22412,"tokens_out":8153,"duration_ms":96220,"concrete_test":"Acquire historical BEA county personal income vintages (e.g., from BEA archives or ALFRED) and rebuild the outcome and predictor panels as of a fixed historical date, say January 2015; then rerun the four split strategies for the forecasting regression and classification tasks. If the current-vintage estimates of the leakage gap (e.g., the XGBoost MSE ratio and the 2009 RF AUC gap) remain nonnegative and similar in magnitude, the footnote 12 assumption holds or the bias is immaterial; if the gap reverses or vanishes, the empirical illustration is contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that random unit-time splits in panel data create temporal and cross-sectional leakage and inflate apparent out-of-sample performance—is conceptually sound and is internally supported by the comparison of split strategies in Section 4. The weakest point is footnote 12, which asserts that BEA revisions do not cause temporal leakage because revisions are driven by methodological improvements rather than future data. If this assumption fails, the time-split 'non-leaked' forecasting models are also contaminated by revised historical values, meaning the reported leakage gap (e.g., the XGBoost MSE ratio above 17% in Section 4.1) would understate, not overstate, the true inflation. Because the bias direction is conservative, the substantive conclusion is not threatened; however, the quantitative magnitudes in Section 4 should be treated as illustrative and could shift if vintage data were used. The lack of uncertainty quantification and the use of a single calendar test window affect precision, not the validity of the organizing framework.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper provides a conceptual framework and empirical illustration of data leakage in supervised machine learning applied to panel data. The authors distinguish two prediction policy problems—cross-sectional prediction and sequential forecasting—and argue that a random split at the unit-time level produces both temporal and cross-sectional leakage, while unit-level and time-level splits each remove one source. They propose practical guidelines for splitting and cross-validation, and illustrate the consequences using a balanced panel of 3,058 U.S. counties (2000–2019) for income growth forecasting and classification, running 480 model configurations. The empirical results show that models affected by leakage achieve better apparent out-of-sample performance, with an XGBoost MSE ratio above 17% in the regression task and an AUC gap of about 0.05 for Random Forest in classification.","tokens_in":22566,"tokens_out":5873,"duration_ms":57000,"significance":"If accepted, the paper fills a real gap in applied econometrics: it provides an accessible organizing framework and concrete recommendations for avoiding leakage in panel ML. The framework is conceptually sound and the distinction between cross-sectional prediction and sequential forecasting is useful. The empirical application is extensive (480 models, public replication package) and demonstrates the direction and rough magnitude of leakage-induced inflation. The claims are falsifiable and the code is available, which strengthens the paper's value for practitioners. The main limitations—lack of uncertainty quantification, a single test window, and reliance on revised BEA data—do not threaten the conceptual core, but they do bound the strength of the quantitative conclusions.","major_comments":[{"comment":"The headline comparisons of 'leaked' versus 'non-leaked' models (e.g., the Random Forest AUC gap of 0.051 and the XGBoost MSE leakage ratio above 17%) are reported without any confidence intervals or significance tests. Footnote 19 states that no suitable group-level ROC test is known and that uncertainty quantification would be computationally demanding, but with 480 models one could compute bootstrap intervals on the differences for matched configurations (e.g., identical predictors and split type) or use permutation tests. As written, the word 'significantly' in the text is informal and the magnitudes are not statistically grounded. This is a load-bearing point for the empirical demonstration, and the manuscript should either add uncertainty quantification or explicitly reframe these as descriptive differences.","section":"Section 4.1, Figure 2, footnote 19"},{"comment":"The comparison of random, county, and state splits does not isolate cross-sectional leakage because the splits change the composition of the training and test sets. A state split introduces new states into the test set, so the observed performance drop could reflect a distribution shift across states (different economic structures) rather than the removal of spatial leakage. The statement that 'all the leaked models dominate the non-leaked ones, indicating the presence of spatial leakage' (p. 20) overreaches. A more controlled design would hold the test units and periods fixed while varying only whether the same or neighboring units appear in training, e.g., by excluding a spatial buffer around test counties rather than splitting by state. This point affects the quantitative support for the cross-sectional leakage claim, although it does not weaken the conceptual argument.","section":"Section 4.2, Figure 4, Tables A.7–A.8"},{"comment":"The claim that BEA historical revisions do not create temporal leakage because revisions are driven by methodological improvements rather than future data is asserted with a general citation but not tested against real-time vintage data. If the assertion is wrong, the 'non-leaked' time-split forecasting models are themselves contaminated by revised historical values, and the reported leakage gaps (e.g., the XGBoost MSE ratio above 17%) would be biased, most likely conservatively. The authors should either verify the assumption using available vintages for the outcome and key predictors, or explicitly state that all quantitative leakage magnitudes should be interpreted as lower bounds.","section":"Section 2, footnote 12"}],"minor_comments":[{"comment":"These tables are missing their column headers and appear garbled (e.g., rows begin with '2448 yes yes 0.0421066 yes 34244 yes' with no model or split-type labels), which makes it impossible to verify the reported model configurations from the tables alone.","section":"Online Appendix Tables A.4 and A.6"},{"comment":"Footnote 22 warns that the Logit classification results are unstable due to class imbalance, yet the Logit models are included in the main headline comparisons in Figures 2 and 3; consider either removing them from the classification figures or reporting a class-balanced metric such as balanced accuracy alongside AUC.","section":"Section 4.1, footnote 22"},{"comment":"The abstract says 'nearly 500 models' but Section 4 states 480 models; make the count consistent.","section":"Abstract and Section 4"},{"comment":"The text says the leakage ratio for the regression problem in 2009 'remains similar' to the all-period ratio, but the reader is not given the numerical value; adding the value to the caption or text would help.","section":"Section 4.1, Figure 3"},{"comment":"The recommendation in Table 2 to split at the unit level for cross-sectional prediction should be reconciled with the later discussion of spatial leakage; the table mentions stratified sampling at a higher level only as a fallback, but the empirical results in Figure 4 show substantial drops under state splits, which suggests the fallback may often be the primary choice for spatially correlated panels.","section":"Table 2 and Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for an applied econometrics journal and the replication package is a strength. I recommend major revision rather than reject because the conceptual framework is sound and the identified issues are fixable. The main risk is that the empirical magnitudes may be overinterpreted without uncertainty quantification; the authors should be encouraged to add bootstrap or permutation-based intervals, even if only for the headline comparisons. The BEA revision concern can be addressed with a clearer caveat or a robustness check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: this is a solid, useful paper that will likely become a standard citation for applied economists doing ML on panel data. The core framework - split by research goal (cross-sectional prediction vs sequential forecasting) and choose unit-level vs time-based splits accordingly - is clear and actionable. What's new is the scale of the empirical demonstration: nearly 500 models on US county data, systematically comparing random unit-time splits, unit splits, group splits, and time splits, with and without contemporaneous predictors. The results show leakage inflates performance in predictable ways, and the 2009 recession exercise is a nice litmus test.\n\nThe paper does a good job with practical guidelines (Table 2) and provides a replication package. The authors are honest about limitations: footnote 19 concedes no statistical tests for the performance gaps, and footnote 22 flags unstable Logit results.\n\nSoft spots, in proportion:\n\n- The quantitative magnitudes (e.g., XGBoost MSE ratio >17%) have no confidence intervals and rest on a single calendar test window (2016-2019). That's fine for illustration, but readers should treat the numbers as suggestive, not precise.\n- The 'first systematic assessment' framing is slightly overreaching. Blocked CV and spatial CV exist in the ML and forecasting literatures; the authors cite some of it but could position this paper more as a synthesis and practical guide than as the first systematic study. That's a rhetoric issue more than a substance issue.\n- The BEA data revision assumption (footnote 12) is a genuine caveat. If revisions use future information, the 'non-leaked' models are also contaminated. The stress-test note is correct that any such bias would likely make the leakage gap understated, not overstated, so the central claim holds. Still, the magnitudes could shift if vintage data were used.\n- There's a mild circularity: leaked models are defined by their split and predictors, and then shown to perform better. The conceptual claim is definitional; what's empirical is the size of the inflation. The paper implicitly knows this, but I'd like the distinction stated more sharply.\n\nWho it's for: applied economists and policy-oriented ML users who need a clear protocol, and journal referees who need a benchmark for what counts as leakage-free validation. I'd send it out for peer review; with a toned-down 'first' claim and a bit more uncertainty quantification (or at least more careful hedging), it will be a genuinely useful contribution. I'd cite it myself.\n\nBest.","headline":"Clear, useful framework for avoiding leakage in panel-data ML; the empirical magnitudes are illustrative, not precise, but the central message is sound and worth publishing.","tokens_in":23046,"tokens_out":3443,"would_cite":true,"duration_ms":33469,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Randomly splitting panel data into train and test sets leaks information and inflates machine-learning performance.","keywords":["data leakage","panel data","machine learning","prediction policy problems","temporal leakage","cross-sectional leakage","cross-validation","forecasting"],"falsifier":"Run the same model comparison using real-time historical vintages of the county income data (the values actually available at each forecast origin) rather than the finalized panel; if the correctly time-split models perform as well as the random-split models, the performance gap attributed to leakage is an artifact of data revision rather than leakage.","tokens_in":22230,"feed_emoji":"📊","tokens_out":7730,"duration_ms":80084,"temperature":0.7,"pith_summary":"This paper argues that applying standard machine-learning practice to panel data—randomly splitting rows into training and testing sets—silently leaks information across both units and time, inflating out-of-sample performance and making models look more useful for policy than they actually are. The authors build an organizing framework that ties the correct split to the type of prediction policy problem: cross-sectional prediction should split at the unit or group level, while sequential forecasting should split non-randomly on time. They demonstrate the stakes with roughly 480 models that predict county income and recession risk across over 3,000 U.S. counties, finding that leaked models beat honest ones by margins large enough to change policy conclusions. The paper's contribution is a set of concrete do's and don'ts so that ML predictions survive contact with genuinely unseen data.","feed_headline":"Random splits leak panel data, inflate ML performance","feed_subtitle":"A 480-model test on U.S. counties shows which train-test splits overstate predictive power and which are honest.","key_machinery":"The paper's central object is a four-way taxonomy of training/testing splits for panel data—random split at the observation (unit-time) level, at the unit level, at the group (e.g., state) level, and non-random split on time—paired with a two-way distinction between cross-sectional prediction policy problems and sequential forecasting policy problems. The taxonomy does the argument's work: each split produces a characteristic combination of temporal and cross-sectional leakage, and the paper shows that the right split is the one whose leakage pattern matches the deployment scenario. The empirical machinery is a controlled comparison of nearly 500 model configurations varying the split, the use of contemporaneous predictors, the inclusion of outcome lags, test-set size, and algorithm, while holding the raw predictors fixed.","core_discovery":"The central claim is that there is no single off-the-shelf way to split panel data; the split must follow the goal. A random split at the observation (unit-time) level always creates both temporal leakage (future observations help train models that then 'predict' the past) and cross-sectional leakage (the same units appear in both training and testing), so the model has already seen the testing set in substance. If the goal is cross-sectional prediction, the right split is at the unit or group level, accepting trend leakage but eliminating unit leakage. If the goal is forecasting, the right split is non-random on time, accepting that all units appear in both sets but ensuring the model never trains on the future. The paper shows empirically that leaked models outperform correctly split models—for example, average AUC for Random Forest classification rises from 0.708 to 0.759 under temporal leakage, and in the 2009 recession year the gap is much larger (AUC 0.442 honest vs 0.692 leaked). For regression, the XGBoost leakage ratio exceeds 17 percent.","pith_inferences":["Beyond the paper, the same leakage logic implies that default random k-fold cross-validation in standard ML libraries is unsafe for panel data; hyperparameter tuning should use blocked or temporal folds to avoid selecting models by their leakage.","A testable extension: the gap between county-level and state-level splits should grow with the strength of spatial autocorrelation, so one could use Moran's I or similar measures to predict when group-level splits matter most.","For readers auditing applied work, the framework suggests a concrete check: demand a holdout on data the model has never seen—either a later time period or a different geography—before trusting policy claims.","If the data-revision assumption is right, the leakage bias in this application is likely a lower bound for settings with stronger common shocks and more persistent unit heterogeneity, since those amplify both temporal and cross-sectional leakage."],"forward_implications":["If researchers use random observation-level splits on panel data, reported AUC/MSE gains will typically overstate real-world performance, and policy targeting built on those models will be less effective than advertised.","For cross-sectional targeting problems, unit-level (or group-level) splits should become the default, with higher-level splits when spatial autocorrelation is suspected.","For forecasting problems, only lagged predictors and non-random time splits are honest; contemporaneous predictors are a form of temporal leakage.","Reporting practices should shift: applied ML papers with panel data should explicitly state the split strategy and how leakage was prevented, as none of the nine policy-targeting papers cited does.","A model that performs suspiciously well on post-break data (like 2009) is likely leaking, not forecasting, so unusually strong out-of-sample results on hard-to-predict events should be treated as a warning sign."],"supporting_citations":[{"why":"Defines prediction policy problems, the framework that determines which split is appropriate.","marker":"Kleinberg et al. (2015)"},{"why":"Documents data leakage as a driver of the reproducibility crisis in ML-based science, motivating the paper.","marker":"Kapoor & Narayanan (2023)"},{"why":"Supplies the forecasting rule that only information available at the forecast origin may be used, which the paper translates into the ban on contemporaneous predictors.","marker":"Petropoulos et al. (2022)"},{"why":"Formalizes leakage in data mining and labels it a top mistake, grounding the paper's definition.","marker":"Kaufman et al. (2012)"},{"why":"Proposes block-sampling cross-validation for panel data that mitigates leakage, the methodological benchmark the paper contrasts with common practice.","marker":"Babii et al. (2023)"},{"why":"Offers a time-series cross-validation with a gap for panel nowcasting, a proper temporal-validation strategy the paper endorses.","marker":"Babii et al. (2024)"},{"why":"States the standard ML firewall principle that training and test sets must be disjoint, which the paper shows panel data violate.","marker":"Mullainathan & Spiess (2017)"}],"fun_headline_variants":["Panel splits leak data, inflate ML scores","How to split panel data without cheating","Random splits overstate ML power in panels","480-model test shows panel leakage costs","Goal-based splits fix panel data leakage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The empirical demonstration assumes that official county income data are not revised using future information, so the 'honest' time-split forecasting models are truly uncontaminated; if revisions do carry future data, the measured inflation from leakage is biased.","fun_headline_variants_meta":{"raw":{"variants":["Panel splits leak data, inflate ML scores","How to split panel data without cheating","Random splits overstate ML power in panels","480-model test shows panel leakage costs","Goal-based splits fix panel data leakage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1380,"prompt_tokens":870,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":486,"tokens_out":510,"duration_ms":6279,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:53:04.617901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same model comparison using real-time historical vintages of the county income data (the values actually available at each forecast origin) rather than the finalized panel; if the correctly time-split models perform as well as the random-split models, the performance gap attributed to leakage is an artifact of data revision rather than leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines prediction policy problems, the framework that determines which split is appropriate."}],"review_version":1}