{"id":"d00b67f0-6fc0-4f5e-bd9b-ff7c70a50b2a","arxiv_id":"2506.05764","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On one day of BTC/USDT order book data, Savitzky-Golay smoothing and feature choice helped simple models match or beat deeper neural networks.","lead":"This paper tests whether more complex neural networks really improve short-term cryptocurrency price forecasts from limit order book data, and finds that simple models with careful data smoothing do just as well. The result matters for high-frequency trading because simpler models are faster and easier to interpret.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Centered Savitzky–Golay filter (Eq. 7) leaks up to 1s of future information into test features; reported SG gains likely reflect lookahead, not preprocessing quality.","rationale":"The reader's weakest assumption identifies the central correctness risk. Section 4.2.1 defines a centered Savitzky–Golay window without any causal restriction, and Sections 5 and 6 never clarify that the filter is applied in a leakage-free manner. Since SG is the only non-causal preprocessing step, its consistent across-the-board improvements in Tables 1 and 2 are exactly the signature of future information entering the features. This is a correctness risk, not a matter of differing from consensus: even if the paper's qualitative message is plausible, the reported 1–2% margins and SG gains are not established by the evidence as written. The single-day sample, missing error bars, and acknowledged need for real-time testing are secondary limitations; the leakage issue is decisive. If a causal re-run preserved the SG gains, the central claim would be more credible and could warrant conditional acceptance after multi-day robustness checks, but as written the REJECT verdict stands.","tokens_in":9827,"tokens_out":3003,"duration_ms":35971,"concrete_test":"Re-run Tables 1 and 2 with a causal SG implementation: keep the same centered polynomial fit but shift the output by 10 samples, so that the feature at time t uses only v_{t-20}, ..., v_{t}, or use a one-sided Savitzky–Golay window. Keep the same 80/20 split, model settings, horizons, and evaluation protocol. If the 1000ms binary SG rows fall from roughly 0.72 toward the raw-data levels around 0.65, or if the SG advantage over raw disappears, the headline claim rests on lookahead leakage rather than on preprocessing quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the Savitzky–Golay pipeline is causally valid. Eq. (7) defines the smoothed value at time t as a weighted sum over v_{t-10} through v_{t+10}; the j>0 terms are future observations relative to the feature timestamp. The paper never states that the filter is applied one-sidedly, that the smoothed series is shifted by the window half-width before modeling, or that filtering is performed separately within train and test blocks. If the full 100ms series is filtered before the 80/20 split, every test feature contains information up to 1s after the feature's own timestamp. For the 100ms and 500ms horizons this is direct lookahead; for the 1000ms horizon it spans essentially the entire label interval. This cleanly explains the pattern in Tables 1 and 2: Savitzky–Golay is the only non-causal filter and the only one that consistently improves every model, while the causal Kalman filter often degrades accuracy. The conclusion attributes gains to 'feature engineering and denoising,' but the SG component is precisely the part that can manufacture accuracy by peeking at the future. Because no code or causal-implementation note is provided, the reported SG rows are not trustworthy evidence for real-time deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks six classifiers—logistic regression, XGBoost, CatBoost, CNN+LSTM, CNN+XGBoost, and DeepLOB—on BTC/USDT limit order book snapshots from Bybit sampled at 100 ms. It compares raw data, Kalman-filtered data, and Savitzky–Golay-smoothed data under binary and ternary labeling schemes at 100 ms, 500 ms, and 1000 ms horizons. The central claim is that with data preprocessing and hyperparameter tuning, simpler models (XGBoost, logistic regression) match or exceed deeper neural networks, with faster inference and greater interpretability. The results are presented in four tables reporting F1 scores or accuracy across configurations, plus runtime comparisons.","tokens_in":10113,"tokens_out":4314,"duration_ms":43331,"significance":"If the result holds under a leakage-free protocol, the paper makes a practically useful point: for high-frequency LOB forecasting, input engineering and hyperparameter choice may dominate architectural depth, with implications for latency and interpretability in deployment. The paper uses publicly available data, defines hand-crafted features explicitly, includes multiple model families, and reports training runtimes, which are strengths. However, the empirical evidence is undermined by a likely lookahead bias in the Savitzky–Golay preprocessing, and the claims are made without error bars, significance tests, or multi-day validation. The paper also ships no code, limiting reproducibility of the preprocessing pipeline.","major_comments":[{"comment":"The Savitzky–Golay filter as defined is centered: the smoothed value at time t is a weighted sum of v_{t+j} for j = -10,...,+10, so it uses up to 1 second of future observations. The manuscript never states that the filter is applied one-sidedly, that the smoothed series is shifted by the window half-width before modelling, or that filtering is performed separately within train and test blocks. If the full 100 ms series is filtered before the 80/20 split, every test feature contains information from after the feature timestamp; for the 500 ms and 1000 ms horizons this is direct lookahead. This could explain the pattern in Tables 1 and 2, where Savitzky–Golay is the only filter that consistently improves every model while the causal Kalman filter often degrades accuracy. The authors must rerun the experiments under a clearly causal protocol—for example, a one-sided filter, a shift by the half-window, or filtering applied independently to training and test periods with no future values—and report whether the SG gains survive. As written, the central conclusion that preprocessing/denoising drives performance is not supported.","section":"§4.2.1, Eq. (7)"},{"comment":"The experiments use a single trading day (2025-01-30) and report no confidence intervals, repeated runs, or significance tests. The abstract's claim that simpler models outperform complex neural networks by 1–2% is therefore not statistically grounded; differences of this size across models may be within sampling noise, especially since the test sets are not held constant across configurations (support ranges from 5,442 to 18,336). The authors should add error bars via bootstrapping or repeated train/test splits, and ideally evaluate on additional trading days, to support the global claim.","section":"§3, §5.1–5.2, Tables 1–2"},{"comment":"Prediction horizon and LOB depth are varied simultaneously: the 100 ms rows use a 5-level LOB while the 500 ms and 1000 ms rows use a 40-level LOB. Consequently, the apparent improvement with longer horizons is confounded with the change in input depth, and no clean horizon effect can be extracted. The same confound affects the comparison of filtering methods across rows, because the test sets differ in size and composition. The experimental design should hold depth fixed when varying horizon, or otherwise report a factorial design.","section":"Tables 1 and 2"},{"comment":"The depth comparison for XGBoost is not controlled: the model is tested on 5,442 snapshots at 40 levels but on 18,336 snapshots at 5 levels. The drop in accuracy from 0.715 to 0.580 may reflect the different test sets (different time periods and noise characteristics) rather than LOB depth alone. The conclusion that 'shallower LOB increases coverage but reduces accuracy' requires a matched test set or a per-support analysis.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"There are repeated typos ('Inspired y Sirignano' should be 'Inspired by Sirignano', 'bais' should be 'bias', 'cuased' should be 'caused'). The phrase 'Efficient Market Theorem' should be 'Efficient Market Hypothesis'. The sentence implying that efficiency predicts exactly 50% accuracy conflates market efficiency with unpredictability under a specific loss function.","section":"§2.4"},{"comment":"The text says the models are evaluated with F1 score, but the tables are titled 'Classification ... accuracy' and Section 5.3 uses the word 'accuracy' for the metric that appears to be F1. The metric used in each table should be stated consistently.","section":"§5 and Table 3"},{"comment":"The subsection heading contains a typo: 'Logistic Regregression' should be 'Logistic Regression'.","section":"§5.4"},{"comment":"The Kalman filter parameters Q and R, the Savitzky–Golay window length and polynomial degree, and the ternary epsilon threshold are all described as tuned on a small sample, but the actual values and tuning procedure are not reported. Providing these details, along with the train/validation split and any code repository, would substantiate the 'hyperparameter tuning' claim.","section":"§4.2 and §5"}],"recommendation":"major_revision","confidential_remarks":"The Savitzky–Golay lookahead issue in §4.2.1 is the most serious concern and is directly load-bearing for the paper's central claim. Please ask the authors to re-run the affected experiments under a strictly causal protocol and to report whether the SG improvements persist. If the SG gains vanish, the paper's main conclusion would need substantial revision. The single-day evaluation and lack of uncertainty quantification also need to be addressed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:2506.05764: its headline result is probably an artifact of lookahead. The Savitzky–Golay filter is defined in Eq. (7) with a centered window, j = -10 to +10, so the smoothed value at t contains observations up to one second after t. The paper never says the filter is applied causally, shifted, or separately within train and test. With 100ms snapshots, that is future information in the features. The tables show SG helping every model while the causal Kalman filter often hurts, which is exactly the pattern you'd expect from leakage.\n\nThat said, the paper isn't a waste. It lays out a clean benchmark: six models, two labeling schemes, three horizons, on public Bybit data, with latency numbers. The author is explicit that this is one trading day and that more work is needed. Section 5.4 about sequence length and depth is a reasonable exploratory exercise. The literature review is adequate and the acknowledgment of Kalman's tuning sensitivity is fair.\n\nThe soft spots beyond leakage: only one day (2025-01-30), no error bars or repeated runs, test set size varies by depth (5,442 to 18,336), and there's no code or causal filtering note. Section 5.3's \"accuracy decreases as depth decreases\" is presented as a trade-off but could just reflect sample selection. The binary accuracy numbers around 0.53 for 100ms are close to chance, which is realistic.\n\nProportionally, the leakage is load-bearing. Without a causal SG implementation, the claim that preprocessing beats architecture is not supported. If the author reruns with a one-sided filter or shifts the output, the SG gains might disappear, and then the paper would arguably show the opposite—that model complexity matters little either way.\n\nWho is this for? Someone teaching or debugging lookahead bias would find it useful as a cautionary example. A researcher looking for a solid benchmark of LOB forecasting methods should wait for a revision with causal filters, more days, and proper statistical testing.\n\nMy recommendation: don't desk-reject the author's career, but desk-reject this draft. It's not ready for referees until the leakage is addressed. If the author sends a revised version with a causal pipeline and multiple days, I'd be willing to look again.","headline":"A clear benchmark undone by a centered Savitzky–Golay filter that leaks the future into the features; the main claim is unsupported until the filter is made causal.","tokens_in":10610,"tokens_out":2334,"would_cite":false,"duration_ms":22970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On high-frequency crypto order books, smoothing and feature engineering add more predictive accuracy than neural depth — simple models match or beat deep networks by 1-2%.","keywords":["limit order books","cryptocurrency microstructure","mid-price movement prediction","Savitzky-Golay smoothing","Kalman filtering","XGBoost","DeepLOB","feature engineering vs model complexity"],"falsifier":"Re-run the binary benchmark at the 500 ms horizon with a causal Savitzky-Golay filter — fit the cubic over $\\{v_{t-20},\\ldots,v_t\\}$ or shift the smoothed series ten steps later — and retrain with walk-forward splits so no test window shares a smoothing window with training data. If XGBoost and logistic regression still edge out DeepLOB and CNN+LSTM by 1-2% and the smoothed inputs still beat the raw ones, the central claim holds; if the model gap or the smoothing gain collapses, the reported accuracy came from lookahead.","tokens_in":9600,"feed_emoji":"📈","tokens_out":16468,"duration_ms":144351,"temperature":0.7,"pith_summary":"This paper asks whether the predictive success of deep learning on limit-order-book data comes from architectural depth or from how the inputs are prepared. Benchmarking six models — logistic regression, XGBoost, CatBoost, a CNN+LSTM, a CNN+XGBoost hybrid, and DeepLOB — on 100 ms BTC/USDT snapshots with raw, Kalman-smoothed, and Savitzky-Golay-smoothed inputs, it finds that smoothing improves every model while no deep architecture consistently beats the simple ones. After Savitzky-Golay denoising, XGBoost and logistic regression match or exceed the neural networks by 1-2% in accuracy and F1, with far lower training and inference cost. If the claim holds, the practical lesson is that on noisy high-frequency data, effort spent on filtering, feature engineering, and label construction buys more accuracy than stacking hidden layers, and latency-sensitive strategies can use interpretable models without sacrificing edge.","feed_headline":"Cleaner order-book data beats deeper nets in crypto forecasts","feed_subtitle":"Smoothed inputs let XGBoost and logistic regression match or beat deep nets at a fraction of the compute.","key_machinery":"The load-bearing mechanism is the Savitzky-Golay filter: a local cubic polynomial is fit by least squares over a 21-point centered window ($j=-10,\\dots,10$), so the smoothed value at time $t$ is $\\hat v_t = \\sum_{j=-10}^{10} c_j v_{t+j}$ with fixed convolution weights $c_j$; the paper applies this smoothing to each LOB feature series before scaling. It is the single intervention that lifts accuracy for every model in the benchmark, which is why the model comparison is really a comparison at fixed input quality. The supporting machinery is the shared feature set of the simpler models — first-level and five-level order imbalance, weighted mid-price change, and cumulative book depth — together with the evaluation design that varies label scheme (binary vs ternary), horizon (100 ms, 500 ms, 1 s), and denoising filter (raw, Kalman, Savitzky-Golay).","core_discovery":"The paper's central claim is that, for short-horizon prediction of mid-price moves (the midpoint of the best bid and ask) on cryptocurrency limit order books, data preprocessing and hyperparameter tuning contribute more to accuracy than architectural depth. On a benchmark of six models over BTC/USDT snapshots, no deep network consistently beats the baselines: after Savitzky-Golay smoothing, XGBoost and logistic regression marginally outperform DeepLOB and the CNN+LSTM variants by 1-2% in accuracy and F1 while training in a fraction of the time, and the same smoothing improves every model, whereas Kalman filtering with fixed noise covariances can degrade performance below the raw baseline. The paper also documents two secondary findings: using all 40 book levels rather than 5 raises binary accuracy from about 0.58 to 0.715 but cuts the usable sample by more than half, and feeding ten consecutive snapshots instead of one adds roughly 2% accuracy for both XGBoost and logistic regression at several times the training cost. The paper is explicit that these results come from a single trading day of offline experiments, leaving cross-day robustness and live deployment open.","pith_inferences":["The paper's own equations suggest an upper bound rather than a guaranteed edge: because the smoothing window is centered, a reader cannot rule out that part of the measured gain comes from features that peek at future snapshots, so the numbers should be treated as best-case until a causal filter is tested.","If the flat model ordering generalizes, published deep-learning gains on public LOB benchmarks deserve re-examination against a tuned, denoised tree baseline; some reported 'architectural' improvements may be preprocessing effects any model would pick up.","A nearby question the paper leaves open is whether preprocessing dominance transfers to event-driven LOB data and thinner books, where noise and imbalance signals have different shapes; the answer would decide whether the finding is about crypto LOBs specifically or about noisy high-frequency data generally."],"forward_implications":["Model complexity is a second-order factor: adding a third convolutional block to a CNN+LSTM pipeline (the DeepLOB configuration) produces no consistent accuracy gain over simpler variants, so the paper's recipe of filter, tune, and simple model is the benchmark to beat.","Denoising choice dominates model choice: Savitzky-Golay smoothing improves accuracy across the board, while Kalman filtering with fixed noise variances can fall below the raw-data baseline, meaning preprocessing choices deserve as much tuning attention as hyperparameters.","There is a quantifiable coverage-accuracy trade-off: 40-level books reach 0.715 binary accuracy but cover only 5,442 usable test snapshots, while 10- and 5-level books raise coverage to more than 18,000 and drop accuracy to about 0.58, so system builders must choose an operating point.","Sequence context is a modest 2% gain: feeding ten consecutive 100 ms snapshots instead of one raises F1 by about 2% for both XGBoost and logistic regression, at several times the training time but still only minutes on a laptop."],"supporting_citations":[{"why":"Supplies the handcrafted LOB features (weighted mid-prices, quantity imbalances) and the linear-classifier baseline framing the paper's simpler models build on.","marker":"Kercheval & Zhang (2015)"},{"why":"Provides the FI-2010 benchmark dataset and the 48-64% logistic-regression accuracy range the paper's results are compared against.","marker":"Ntakaris et al. (2018)"},{"why":"Credited in the text with pioneering the CNN+LSTM DeepLOB-style architecture that serves as the deep benchmark.","marker":"Tsantekidis et al. (2018)"},{"why":"Defines the local-polynomial smoothing procedure whose application is the paper's main driver of accuracy gains.","marker":"Savitzky & Golay (1964)"},{"why":"Motivates the epsilon threshold tuning and inverse-frequency loss weighting used for the ternary labels.","marker":"Sirignano & Cont (2019)"},{"why":"Supplies the per-day z-score normalization precedent that the paper extends by adding its two smoothing filters.","marker":"Tsantekidis et al. (2017)"},{"why":"Supplies the Kalman-filter smoothing approach used as the second denoising pipeline.","marker":"Wu et al. (2021), cited in text as Wu et al. (2011)"},{"why":"Precedent for the CNN+CatBoost hybrid, where convolutional feature maps feed a CatBoost classifier.","marker":"Bileki et al. (2022)"}],"fun_headline_variants":["Crypto forecasts: data cleaning beats model depth","Order-book data preprocessing beats extra neural layers","Simpler models match deep nets with better crypto inputs","Inputs beat architecture in crypto order-book prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gains rest on a smoothing filter that, as written, uses future snapshots to clean each data point, and the paper does not show it was applied causally or that the train/test split prevented that future information from leaking into the test set.","fun_headline_variants_meta":{"raw":{"variants":["Crypto forecasts: data cleaning beats model depth","Order-book data preprocessing beats extra neural layers","Simpler models match deep nets with better crypto inputs","Inputs beat architecture in crypto order-book prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2821,"prompt_tokens":1025,"completion_tokens":1796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1737}},"tokens_in":641,"tokens_out":1796,"duration_ms":14170,"temperature":1.0,"reasoning_tokens":1737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:14:00.692358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the binary benchmark at the 500 ms horizon with a causal Savitzky-Golay filter — fit the cubic over $\\{v_{t-20},\\ldots,v_t\\}$ or shift the smoothed series ten steps later — and retrain with walk-forward splits so no test window shares a smoothing window with training data. If XGBoost and logistic regression still edge out DeepLOB and CNN+LSTM by 1-2% and the smoothed inputs still beat the raw ones, the central claim holds; if the model gap or the smoothing gain collapses, the reported accuracy came from lookahead.","supporting_citations":[{"cited_title":"Using Deep Learning for price prediction by exploiting stationary limit order book features","cited_arxiv_id":"1810.09965","evidence_quote":"Credited in the text with pioneering the CNN+LSTM DeepLOB-style architecture that serves as the deep benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the local-polynomial smoothing procedure whose application is the paper's main driver of accuracy gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the epsilon threshold tuning and inverse-frequency loss weighting used for the ternary labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the per-day z-score normalization precedent that the paper extends by adding its two smoothing filters."}],"review_version":1}