pith. sign in

arxiv: 2605.23959 · v1 · pith:YNMQDNAYnew · submitted 2026-05-12 · 💱 q-fin.RM · q-fin.TR

When Alpha Disappears: A One-Switch Benchmark for Decision-Time Leakage in Financial Backtests

Fan Zhang , Zhen Li , Sijia Peng , Yu Chen This is my paper

Pith reviewed 2026-06-30 22:36 UTC · model grok-4.3

classification 💱 q-fin.RM q-fin.TR
keywords decision-time leakagefinancial backtestsevaluation benchmarkprotocol-induced inflationmachine learningequity OHLCVwalk-forward testing
0
0 comments X

The pith

A one-switch benchmark shows that decision-time leakage inflates financial backtest metrics only under specific evaluation conventions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a benchmark that measures protocol-induced inflation in financial machine-learning backtests by changing one evaluation rule at a time from a clean t+1-open reference while keeping data, splits, models, and costs fixed. It finds the inflation effect is selective: centered temporal features and same-day-open execution with post-open bar information produce large, stable gains in both predictive accuracy and trading returns, but global normalization, future-informed graphs, and same-day-close execution show little effect across the tested panels and models. A sympathetic reader would care because this isolates which common backtest choices can make models appear profitable when they would not be in live trading without those choices.

Core claim

The benchmark estimates protocol-induced inflation by toggling one evaluation convention at a time around a clean t+1-open reference, while holding the data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention fixed. Across two daily-OHLCV equity panels, six model families, and yearly tests from 2016--2024, inflation is highly selective: centered temporal features and same-day-open execution with post-open daily-bar information cause large and stable increases in both predictive and trading metrics, whereas global normalization, future-informed graph structure, and same-day-close execution are weak in most settings.

What carries the argument

The one-switch benchmark that isolates decision-time leakage by toggling a single evaluation convention around a clean t+1-open reference while fixing all other protocol elements.

If this is right

  • Centered temporal features produce large predictive and trading metric gains when toggled on.
  • Same-day-open execution that includes post-open daily-bar information inflates both predictive and trading metrics in a stable way.
  • Global normalization produces only weak inflation in most tested settings.
  • Future-informed graph structures and same-day-close execution show weak effects on metric inflation across the panels and models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Backtest results that rely on centered features or same-day-open post-open information may drop sharply once those conventions are removed.
  • The benchmark could be applied to other asset classes or higher-frequency data to test whether the same selective pattern holds.
  • Published financial ML papers using the inflating conventions may overstate robustness unless they also report the clean-reference results.

Load-bearing premise

Changing one evaluation convention at a time around the clean reference isolates its leakage effect without hidden interactions from the other fixed elements.

What would settle it

Running the same toggles on a third equity panel or with a different walk-forward scheme and finding that the large inflation from centered features and same-day-open execution no longer appears would falsify the selectivity claim.

Figures

Figures reproduced from arXiv: 2605.23959 by Fan Zhang, Sijia Peng, Yu Chen, Zhen Li.

Figure 1
Figure 1. Figure 1: Same split, different information sets. Chronological order alone does not ensure that the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Controlled benchmark design. Within each pair, the data panel, split, model family, horizon, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Semantic sanity checks on the 2024 Ridge window. Future-suffix perturbations affect [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model and market heterogeneity. Left: U.S. mean LG-SR@5bps by protocol variant for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Yearly LG-SR@5bps for Ridge under EXEC_OPEN. The positive effect is not concentrated in a small number of windows. paired Leakage Gains relative to the clean benchmark. For the two dominant violations, we report the mean yearly LG-SR@5bps, bootstrap 95% confidence intervals, the number of positive years, and a one-sided paired Wilcoxon signed-rank test. Full yearly statistics for the dominant violations ar… view at source ↗
Figure 6
Figure 6. Figure 6: Transaction-cost sensitivity for the Ridge baseline at [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
read the original abstract

We introduce When Alpha Disappears, a paired evaluation benchmark for diagnosing decision-time leakage in financial machine-learning backtests. Rather than treating leakage as a binary property, the benchmark estimates protocol-induced inflation by toggling one evaluation convention at a time around a clean $t{+}1$-open reference, while holding the data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention fixed. Across two daily-OHLCV equity panels, six model families, and yearly tests from 2016--2024, we find that inflation is highly selective: centered temporal features and same-day-open execution with post-open daily-bar information cause large and stable increases in both predictive and trading metrics, whereas global normalization, future-informed graph structure, and same-day-close execution are weak in most settings. The benchmark is diagnostic rather than a claim of tradable alpha, and is intended to make evaluation assumptions, failure modes, and protocol fragility directly measurable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces the 'When Alpha Disappears' benchmark, which diagnoses decision-time leakage in financial ML backtests via one-at-a-time toggles of evaluation conventions around a fixed clean t+1-open reference (holding data panel, walk-forward split, model family, horizon, portfolio rule, and costs fixed). Across two daily-OHLCV equity panels, six model families, and yearly tests 2016-2024, it reports that inflation is highly selective: centered temporal features and same-day-open execution with post-open daily-bar information produce large, stable gains in predictive and trading metrics, while global normalization, future-informed graph structure, and same-day-close execution are weak in most settings. The benchmark is positioned as diagnostic rather than a source of tradable alpha.

Significance. If the selectivity results hold under the controlled protocol, the benchmark supplies a practical, reproducible method for quantifying protocol-induced inflation in quant-finance backtests. The design's emphasis on isolated toggles, multiple panels/models/periods, and explicit reference case is a strength that could help standardize evaluation practices and reduce over-optimism in the field.

major comments (2)
  1. [Methods] Methods (or equivalent section describing the benchmark protocol): the claim that toggles are performed while 'holding the data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention fixed' requires explicit pseudocode or a table enumerating the exact feature-construction and execution rules for the t+1-open reference versus each toggle; without this, readers cannot verify that no unintended leakage was introduced during the 'clean' baseline construction.
  2. [Results] Results section (tables or figures reporting metric changes): the statements of 'large and stable increases' and 'weak in most settings' need accompanying effect-size tables (e.g., mean and std of Sharpe or accuracy deltas across the 9 years) and a clear statement of the statistical test used to classify an effect as 'large' versus 'weak'; the current description supplies no raw numbers or significance thresholds, which is load-bearing for the selectivity conclusion.
minor comments (2)
  1. [Abstract] The abstract states findings but omits any numerical illustration of the reported deltas; adding one or two concrete effect sizes would improve immediate readability.
  2. Notation for the six model families and two equity panels should be defined at first use (or in a table) rather than left implicit.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which will improve the clarity and reproducibility of the manuscript. We address each major comment below.

read point-by-point responses

  1. Referee: [Methods] Methods (or equivalent section describing the benchmark protocol): the claim that toggles are performed while 'holding the data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention fixed' requires explicit pseudocode or a table enumerating the exact feature-construction and execution rules for the t+1-open reference versus each toggle; without this, readers cannot verify that no unintended leakage was introduced during the 'clean' baseline construction.

    Authors: We agree that the current manuscript lacks sufficient detail on the exact rules. In the revised version, we will add a dedicated table (or pseudocode block) in the Methods section that explicitly enumerates the feature-construction steps, execution timing, and information sets for the t+1-open reference case and for each of the toggled variants. This will allow readers to verify the isolation of each toggle. revision: yes

  2. Referee: [Results] Results section (tables or figures reporting metric changes): the statements of 'large and stable increases' and 'weak in most settings' need accompanying effect-size tables (e.g., mean and std of Sharpe or accuracy deltas across the 9 years) and a clear statement of the statistical test used to classify an effect as 'large' versus 'weak'; the current description supplies no raw numbers or significance thresholds, which is load-bearing for the selectivity conclusion.

    Authors: We accept that quantitative support for the selectivity claims is required. The revised manuscript will include new tables reporting, for each toggle, the mean and standard deviation of deltas in key metrics (Sharpe ratio, accuracy, etc.) across the nine yearly periods. We will also state the statistical procedure (paired t-test on yearly deltas, with effect-size thresholds) used to classify effects as large versus weak, including any multiple-testing adjustments. revision: yes

Circularity Check

0 steps flagged

No significant circularity; purely empirical benchmark

full rationale

The paper introduces an empirical one-switch benchmark that measures protocol-induced inflation by toggling single evaluation conventions (e.g., feature centering, execution timing) around a fixed t+1-open reference while holding data panel, splits, models, horizon, and costs constant. No equations, derivations, fitted parameters, or predictions are described; results are reported as observed metric differences across panels, model families, and years. The central claim is a measured selectivity outcome rather than a derived necessity, with no self-citation load-bearing steps or reductions to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no information on free parameters, axioms, or invented entities is provided in the given text.

pith-pipeline@v0.9.1-grok · 5704 in / 1056 out tokens · 28699 ms · 2026-06-30T22:36:11.941293+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages · 2 internal anchors

  1. [1]

    Asset pricing and machine learning: A critical review.Journal of Economic Surveys, 38(1):27–56, 2024

    Matteo Bagnara. Asset pricing and machine learning: A critical review.Journal of Economic Surveys, 38(1):27–56, 2024. URLhttps://doi.org/10.1111/joes.12532

    work page doi:10.1111/joes.12532 2024

  2. [2]

    The probability of backtest overfitting.The Journal of Computational Finance, 20(4):39–69, 2017

    David Bailey, Jonathan Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu. The probability of backtest overfitting.The Journal of Computational Finance, 20(4):39–69, 2017. URL https://doi.org/10.21314/jcf.2016.322

    work page doi:10.21314/jcf.2016.322 2017

  3. [3]

    David H Bailey, Jonathan M Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu. Pseudo- mathematics and financial charlatanism: The effects of backtest overfitting on out-of-sample performance notices of the ams.Notices of the American Mathematical Society, 2014. URL https://doi.org/10.1090/noti1105

    work page doi:10.1090/noti1105 2014

  4. [4]

    The pitfalls of memorization: When memorization hurts generalization

    Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Pascal Vincent. The pitfalls of memorization: When memorization hurts generalization. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=vVhZh9ZpIM

    2025

  5. [5]

    Look-Ahead-Bench: A standardized benchmark of look-ahead bias in point-in-time LLMs for finance,

    Mostapha Benhenda. Look-ahead-bench: a standardized benchmark of look-ahead bias in point-in-time llms for finance.arXiv preprint arXiv:2601.13770, 2026. URL https://doi. org/10.48550/arXiv.2601.13770

    work page doi:10.48550/arxiv.2601.13770 2026

  6. [6]

    The illusion of alpha: Quantifying hidden data leakage in financial machine learning

    Kavya Bhand and Aadi Joshi. The illusion of alpha: Quantifying hidden data leakage in financial machine learning. 2026. URLhttps://doi.org/10.21203/rs.3.rs-9180656/v1

    work page doi:10.21203/rs.3.rs-9180656/v1 2026

  7. [7]

    Implications of data leakage in machine learning preprocessing: a multi-domain investigation

    Mohamed Aly Bouke, Saleh Ali Zaid, and Azizol Abdullah. Implications of data leakage in machine learning preprocessing: a multi-domain investigation. 2024. URL https://doi.org/ 10.21203/rs.3.rs-4579465/v1

    work page doi:10.21203/rs.3.rs-4579465/v1 2024

  8. [8]

    Look-ahead benchmark bias in portfolio performance evaluation.Swiss Finance Institute Research Paper, (08-33), 2008

    Gilles Daniel, Didier Sornette, and Peter Wohrmann. Look-ahead benchmark bias in portfolio performance evaluation.Swiss Finance Institute Research Paper, (08-33), 2008. URL https: //doi.org/10.3905/JPM.2009.36.1.121

    work page doi:10.3905/jpm.2009.36.1.121 2008

  9. [9]

    John Wiley & Sons, 2018

    Marcos Lopez De Prado.Advances in financial machine learning. John Wiley & Sons, 2018

    2018

  10. [10]

    Jean Dessain. Machine learning models predicting returns: Why most popular performance metrics are misleading and proposal for an efficient metric.Expert Systems with Applications, 199:116970, 2022. URLhttps://doi.org/10.1016/j.eswa.2022.116970

    work page doi:10.1016/j.eswa.2022.116970 2022

  11. [11]

    A test of lookahead bias in llm forecasts.arXiv preprint arXiv:2512.23847, 2025

    Zhenyu Gao, Wenxi Jiang, and Yutong Yan. A test of lookahead bias in llm forecasts.arXiv preprint arXiv:2512.23847, 2025. URLhttps://doi.org/10.48550/arXiv.2512.23847

    work page doi:10.48550/arxiv.2512.23847 2025

  12. [12]

    Assessing look-ahead bias in stock return predictions generated by gpt sentiment analysis.Journal of Financial Data Science, 6(1), 2024

    Paul Glasserman and Caden Lin. Assessing look-ahead bias in stock return predictions generated by gpt sentiment analysis.Journal of Financial Data Science, 6(1), 2024. URL https: //doi.org/10.3905/jfds.2023.1.143

    work page doi:10.3905/jfds.2023.1.143 2024

  13. [13]

    Empirical asset pricing via machine learning.The Review of Financial Studies, 33(5):2223–2273, 2020

    Shihao Gu, Bryan Kelly, and Dacheng Xiu. Empirical asset pricing via machine learning.The Review of Financial Studies, 33(5):2223–2273, 2020. URL https://doi.org/10.1093/rfs/ hhaa009

    work page doi:10.1093/rfs/ 2020

  14. [14]

    Data splitting to avoid information leakage with datasail.Nature Communications, 16(1):3337, 2025

    Roman Joeres, David B Blumenthal, and Olga V Kalinina. Data splitting to avoid information leakage with datasail.Nature Communications, 16(1):3337, 2025. URL https://doi.org/10. 1038/s41467-025-58606-8

    2025

  15. [15]

    Leakage and the reproducibility crisis in machine- learning-based science.Patterns, 4(9), 2023

    Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine- learning-based science.Patterns, 4(9), 2023. URL https://doi.org/10.1016/j.patter. 2023.100804

    work page doi:10.1016/j.patter 2023

  16. [16]

    Future is unevenly distributed: Forecasting ability of LLMs depends on what we’re asking

    Chinmay Karkar and Paras Chopra. Future is unevenly distributed: Forecasting ability of LLMs depends on what we’re asking. InAAAI 2026 Workshop on Assessing and Improving Reliability of Foundation Models in the Real World, 2026. URL https://openreview.net/forum?id= zzF5H0kZ8I. 10

    2026

  17. [17]

    Leakage in data mining: Formulation, detection, and avoidance.ACM Transactions on Knowledge Discovery from Data (TKDD), 6(4):1–21, 2012

    Shachar Kaufman, Saharon Rosset, Claudia Perlich, and Ori Stitelman. Leakage in data mining: Formulation, detection, and avoidance.ACM Transactions on Knowledge Discovery from Data (TKDD), 6(4):1–21, 2012. URLhttps://doi.org/10.1145/2382577.2382579

    work page doi:10.1145/2382577.2382579 2012

  18. [18]

    A performance comparison of machine learning models for stock market prediction with novel investment strategy.Plos One, 18(9): e0286362, 2023

    Azaz Hassan Khan, Abdullah Shah, Abbas Ali, Rabia Shahid, Zaka Ullah Zahid, Malik Umar Sharif, Tariqullah Jan, and Mohammad Haseeb Zafar. A performance comparison of machine learning models for stock market prediction with novel investment strategy.Plos One, 18(9): e0286362, 2023. URLhttps://doi.org/10.1371/journal.pone.0286362

    work page doi:10.1371/journal.pone.0286362 2023

  19. [19]

    Adriano Koshiyama and Nick Firoozye. Avoiding backtesting overfitting by covariance- penalties: An empirical investigation of the ordinary and total least squares cases.The Journal of Financial Data Science Fall, 1(4):63–83, 2019. URL https://doi.org/10.3905/jfds. 2019.1.013

    work page doi:10.3905/jfds 2019

  20. [20]

    Time travel is cheating: Going live with deepfund for real-time fund investment benchmarking

    Changlun Li, Yao SHI, Chen Wang, Qiqi Duan, Runke RUAN, Weijie Huang, Haonan Long, Lijun Huang, Nan Tang, and Yuyu Luo. Time travel is cheating: Going live with deepfund for real-time fund investment benchmarking. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2026. URL https: //openreview.net/...

    2026

  21. [21]

    Profit mirage: Revisiting information leakage in llm-based financial agents.arXiv preprint arXiv:2510.07920, 2025

    Xiangyu Li, Yawen Zeng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. Profit mirage: Revisiting information leakage in llm-based financial agents.arXiv preprint arXiv:2510.07920, 2025. URLhttps://doi.org/10.48550/arXiv.2510.07920

    work page doi:10.48550/arxiv.2510.07920 2025

  22. [22]

    Leakage-safe benchmark design for market-stress early warning: An economically credible evaluation

    Ting Liu. Leakage-safe benchmark design for market-stress early warning: An economically credible evaluation. 2026. URLhttps://doi.org/10.20944/preprints202603.1925.v1

    work page doi:10.20944/preprints202603.1925.v1 2026

  23. [23]

    The statistics of sharpe ratios.Financial analysts journal, 58(4):36–52, 2002

    Andrew W Lo. The statistics of sharpe ratios.Financial analysts journal, 58(4):36–52, 2002. URLhttps://doi.org/10.2469/faj.v58.n4.2453

    work page doi:10.2469/faj.v58.n4.2453 2002

  24. [24]

    The memorization problem: Can we trust llms’ economic forecasts?arXiv preprint arXiv:2504.14765, 2025

    Alejandro Lopez-Lira, Yuehua Tang, and Mingyin Zhu. The memorization problem: Can we trust llms’ economic forecasts?arXiv preprint arXiv:2504.14765, 2025. URL https: //doi.org/10.48550/arXiv.2504.14765

    work page doi:10.48550/arxiv.2504.14765 2025

  25. [25]

    Impact of leakage on data harmonization in machine learning pipelines in class imbalance across sites.Neurocomputing, page 133146, 2026

    Nicolás Nieto, Simon B Eickhoff, Christian Jung, Martin Reuter, Kersten Diers, Malte Kelm, Artur Lichtenberg, Federico Raimondo, and Kaustubh R Patil. Impact of leakage on data harmonization in machine learning pipelines in class imbalance across sites.Neurocomputing, page 133146, 2026. URLhttps://doi.org/10.1016/j.neucom.2026.133146

    work page doi:10.1016/j.neucom.2026.133146 2026

  26. [26]

    Spurious Predictability in Financial Machine Learning

    Sotirios D Nikolopoulos. Spurious predictability in financial machine learning.arXiv preprint arXiv:2604.15531, 2026. URLhttps://doi.org/10.48550/arXiv.2604.15531

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.15531 2026

  27. [27]

    Data leakage inflates prediction performance in connectome-based machine learning mod- els.Nature communications, 15(1):1829, 2024

    Matthew Rosenblatt, Link Tejavibulya, Rongtao Jiang, Stephanie Noble, and Dustin Scheinost. Data leakage inflates prediction performance in connectome-based machine learning mod- els.Nature communications, 15(1):1829, 2024. URL https://www.nature.com/articles/ s41467-024-46150-w

    2024

  28. [28]

    Anisha Roy and Dip Roy. Memguard-alpha: Detecting and filtering memorization-contaminated signals in llm-based financial forecasting via membership inference and cross-model disagree- ment.arXiv preprint arXiv:2603.26797, 2026. URL https://doi.org/10.48550/arXiv. 2603.26797

    work page internal anchor Pith review doi:10.48550/arxiv 2026

  29. [29]

    Lookahead bias in pretrained language models

    Suproteem K Sarkar and Keyon Vafa. Lookahead bias in pretrained language models. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. URL https: //openreview.net/forum?id=s6WkKKBgw3

    2025

  30. [30]

    On leakage in machine learning pipelines.arXiv preprint arXiv:2311.04179, 2023

    Leonard Sasse, Eliana Nicolaisen-Sobesky, Juergen Dukart, Simon B Eickhoff, Michael Götz, Sami Hamdan, Vera Komeyer, Abhijit Kulkarni, Juha Lahnakoski, Bradley C Love, et al. On leakage in machine learning pipelines.arXiv preprint arXiv:2311.04179, 2023. URL https://doi.org/10.48550/arXiv.2311.04179. 11

    work page doi:10.48550/arxiv.2311.04179 2023

  31. [31]

    Overview of leakage scenarios in supervised machine learning.Journal of Big Data, 12 (1):135, 2025

    Leonard Sasse, Eliana Nicolaisen-Sobesky, Juergen Dukart, Simon B Eickhoff, Michael Götz, Sami Hamdan, Vera Komeyer, Abhijit Kulkarni, JM Lahnakoski, Bradley C Love, et al. Overview of leakage scenarios in supervised machine learning.Journal of Big Data, 12 (1):135, 2025. URLhttps://doi.org/10.1186/s40537-025-01193-8

    work page doi:10.1186/s40537-025-01193-8 2025

  32. [32]

    The sharpe ratio.Streetwise–the Best of the Journal of Portfolio Management, 3(3):169–85, 1998

    William F Sharpe et al. The sharpe ratio.Streetwise–the Best of the Journal of Portfolio Management, 3(3):169–85, 1998

    1998

  33. [33]

    2009.11189 , archivePrefix =

    Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. Qlib: An ai-oriented quantitative investment platform.arXiv preprint arXiv:2009.11189, 2020. URL https:// arxiv.org/abs/2009.11189. 12 A Broader Impacts This work may have positive impact by improving the reliability and transparency of financial model evaluation. By showing how specific prot...

    work page arXiv 2009