{"id":"cfa25302-87a6-44d9-b80d-22699c25a9ee","arxiv_id":"2504.17664","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An empirical comparison of classifiers for financial time series that argues for big data, but the headline result uses in-sample evaluation and the cited model architecture is not the one implemented.","lead":"This preprint compares small and big data approaches for classifying financial time series, running simple backtests on Strike, Bitcoin, and EUR/USD data. It argues that big data approaches matter, but its main backtest evaluates the model on the exact data it was trained on, so the reported 20% return is not evidence of predictive skill.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20% return in Section 3.3.2 is computed on the same 100,000 samples used for training (Appendix E), so it does not demonstrate out-of-sample predictive skill and cannot support the claim that the model identified profitable patterns.","rationale":"The reader's weakest-assumption diagnosis is correct and precisely matches the appendix code. The 20% return is computed from predictions on the training set itself, so the central claim of profitable pattern identification is unsupported by the reported experiment. This is not a disagreement with consensus; it is an internally invalid evaluation protocol. The concrete test would settle whether any out-of-sample skill exists. Since the reader already recommended REJECT and this analysis confirms that judgment, the verdict remains unchanged.","tokens_in":27089,"tokens_out":2194,"duration_ms":24115,"concrete_test":"Re-run the Appendix E experiment with a strict chronological split: fit the scaler and train the network only on df.iloc[:80_000] (or another prespecified train fraction), then compute predictions and cumulative strategy returns exclusively on the unseen df.iloc[80_000:] segment, comparing against the market and a random strategy over the same test window. If the test-period cumulative return is near zero, negative, or not significantly above the random strategy across several random seeds, then the reported 20% return is an in-sample artifact rather than evidence of predictive skill.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that a ConvTimeNet model achieved a 20% cumulative return on EUR/USD minute data, outperforming a random strategy while the market stayed flat (Section 3.3.2). The evaluation in Appendix E undermines this claim: line 22 takes df.iloc[:100_000] as the entire dataset; line 30 fits the StandardScaler on all of these samples; line 38 constructs a DataLoader over the full dataset with shuffle=True; lines 68-76 train the model for 100 epochs on this full dataset; line 91 then computes predictions on the same full DataLoader; and lines 94-101 use those predictions to compute strategy returns and cumulative returns. There is no train/test split, no temporal holdout, and no walk-forward validation. The model is therefore evaluated on the exact data it was trained on, and the 20% figure is an in-sample fit. A flexible convolutional network trained for 100 epochs on 100,000 samples can trivially memorize training labels, especially with a 3-class target derived from return percentiles; the resulting in-sample cumulative return has no bearing on whether the model would generalize to unseen data. The text's own Section 2.3.2 acknowledges this issue for the small-data experiments ('it is often the result of overfitting'), but the same issue is not controlled for in the headline big-data result. Because every headline conclusion rests on this 20% figure, the evaluation protocol is the load-bearing weak point. A secondary concern is that the implemented network (Appendix E lines 40-60) is a simple two-layer Conv1d with adaptive average pooling, not the ConvTimeNet architecture with deformable patch embedding described in Section 3.2.1, but the in-sample evaluation alone is sufficient to invalidate the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares small-data and big-data machine learning approaches for multivariate financial time series classification, using EUR/USD and cryptocurrency datasets. It describes traditional models (XGBoost, SVM, LSTM) and a deep learning model attributed to ConvTimeNet, and reports trading-simulation results in terms of cumulative returns against market and random strategies. The headline result is a 20% cumulative return in Section 3.3.2 from a ConvTimeNet-style model trained on 100,000 minute-level EUR/USD samples. The paper concludes that big data can be beneficial for identifying profitable patterns, though it also discusses overfitting in small data and market adaptation.","tokens_in":27385,"tokens_out":5253,"duration_ms":47792,"significance":"If the 20% out-of-sample return were valid, the paper would offer a noteworthy demonstration that convolutional models can extract tradeable signals from high-frequency FX data. The paper is transparent in providing full code appendices, which allow the evaluation protocol to be checked directly. However, the central empirical claim is not established, because the reported performance is computed on the training set; the significance of the contribution therefore rests on an evaluation that provides no out-of-sample evidence.","major_comments":[{"comment":"The headline 20% cumulative return is computed on the same data used for training. Appendix E line 22 defines df_min as the first 100,000 samples; line 30 fits the StandardScaler on this full set; line 38 creates a DataLoader for the full dataset; lines 68–76 train the model for 100 epochs on this dataset; line 91 computes predictions on the same DataLoader; and lines 94–101 derive strategy returns and cumulative returns from those predictions. There is no train/test split, temporal holdout, or walk-forward validation. The claim in Section 3.3.2 that the model 'was able to identify profitable patterns in the EUR/USD data' is therefore unsupported; the 20% figure reflects in-sample training accuracy, not predictive skill.","section":"§3.3.2, Appendix E"},{"comment":"The small-data experiments use the same in-sample evaluation: after hyperparameter selection with TimeSeriesSplit, Appendix B line 108 calls best_model.predict(features_scaled) on the full feature set, and line 110 forms strategy returns on the same data. Consequently, Figure 13 and the small-data backtests in Figures 6 and 7 are based on training-set predictions. The paper explicitly acknowledges in Section 2.3.2 that the small-data outperformance 'is often the result of overfitting,' but the identical protocol flaw is not controlled in the big-data experiment, so the claimed contrast between small and big data is not established.","section":"§2.3.2, Appendix B"},{"comment":"The implemented network does not correspond to the ConvTimeNet architecture described in Section 3.2.1 and cited to [Cheng et al., 2024]. The code defines a plain two-layer 1D CNN with batch normalization and adaptive average pooling; it contains no deformable patch embedding and no fully convolutional blocks with deepwise and pointwise convolutions. Thus the experiments evaluate a generic CNN baseline, not the architecture named in the paper, and any results cannot be attributed to ConvTimeNet.","section":"Appendix E, lines 40–60"},{"comment":"The comparison against a random strategy is based on a single random draw of signals with no repetitions, error bars, or significance test. Given the high noise level of minute-level FX returns, the observed difference between the model and one random realization could arise by chance; the manuscript provides no statistical evidence that the model's performance exceeds a random baseline.","section":"§3.3.2, Appendix E line 96"}],"minor_comments":[{"comment":"The GARCH model is cited to [Engle and Ng, 1982]; the standard reference for GARCH is Engle (1982), and the authors may want to correct the citation.","section":"§1.1.1"},{"comment":"Figure 13 is referenced in the text but its construction is described only through code in Appendix B; the text should state the dataset sizes and models shown.","section":"§2.3.2, Figure 13"},{"comment":"The data source for the EUR/USD dataset is not described in the main text; the paper should state the provider, period, and feature definitions.","section":"§3.3.1"},{"comment":"The random strategy in Appendix A uses choices {0,1} while Appendix E uses {-1,0,1}; the difference should be explained or reconciled.","section":"Appendices A and E"},{"comment":"The code listing contains several corrupted imports and repeated lines; a clean, executable version of the appendices would improve reproducibility.","section":"Appendix E"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be a working note rather than a completed research article. The central experiment lacks any out-of-sample evaluation, and the code shows the model is scored on its training data. If the authors can rerun the experiments with a proper temporal split and report honest out-of-sample performance, the paper could be reconsidered, but the current version does not support its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: the paper is a review of standard methods (SVM, LSTM, ConvTimeNet) with two small illustrative experiments and one big-data experiment. The big-data experiment is the load-bearing piece: Section 3.3.2 reports a 20% cumulative return and claims the model identified profitable patterns. But Appendix E trains on the full 100,000 samples (line 22) and then computes predictions on that same DataLoader (line 91). No holdout, no walk-forward, no error bars. The 20% is in-sample fit. That is not a minor flaw; it invalidates the central empirical claim.\n\nCredit where it's due: the paper reads cleanly. The background on SVM and LSTM is accurate, the small-data sections correctly warn that good backtest performance on tiny samples is often overfitting, and the code is included in the appendices. If I were teaching an intro ML-for-finance course, the small-data example and the heatmap of performance versus dataset size would be a usable illustration. The heatmap showing performance deteriorating as data increases is actually a decent sanity check, though it undercuts the abstract's claim that big data is valuable.\n\nThe soft spots beyond the leakage: the implemented network in Appendix E is a two-layer Conv1d, not the ConvTimeNet architecture with deformable patch embedding described in Section 3.2.1, so the name is misleading. The dataset is not provided or identified beyond 'data.csv', so the numbers cannot be reproduced. The label thresholds are computed from the full dataset, an additional leakage. And the conclusion wanders into adaptive markets and non-Euclidean spaces without connecting to the experiment. The paper's own evidence argues against the thesis, and the thesis is asserted, not tested.\n\nI agree with the reader's rejection. This is not a substantive research contribution. It is a competent student-level survey with a seriously flawed evaluation. I would not cite it. I would not bring it to a reading group except perhaps as a cautionary example. A serious referee is not needed; the load-bearing flaw is visible from the appendix alone, and the novelty is negligible. Desk reject.","headline":"A survey with an invalid key experiment: the 20% return is an in-sample artifact, so the paper's central claim collapses, though the exposition has some pedagogical value.","tokens_in":27973,"tokens_out":3086,"would_cite":false,"duration_ms":30060,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that large multivariate financial datasets favor fully convolutional deep classifiers over classical models, reporting a 20% EUR/USD strategy return on flat markets.","keywords":["multivariate time series classification","financial time series","EUR/USD","ConvTimeNet","small data versus big data","support vector machines","deep learning","backtesting"],"falsifier":"Train ConvTimeNet on the first 80,000 minute-level EUR/USD samples and freeze the model, then compute strategy returns on the last 20,000 samples; if the 20% excess return shrinks to the random or market baseline, the paper's claim that the model identifies exploitable patterns is unsupported.","tokens_in":26867,"feed_emoji":"📈","tokens_out":7165,"duration_ms":62975,"temperature":0.7,"pith_summary":"This paper is trying to establish that the choice of classifier for multivariate financial time series should depend on whether the dataset is small or large. On small data, the author argues, models like SVMs and XGBoost produce impressive-looking backtests that are actually overfitting, illustrated by a Strike cryptocurrency experiment that fails to transfer to Bitcoin. On large data, the paper claims a deep fully convolutional model (ConvTimeNet) trained on 100,000 minute-level EUR/USD samples with 200 features achieves a 20% cumulative strategy return while the market is flat and beats a random strategy. If true, this would mean convolutional classifiers can extract tradeable short-term direction from high-frequency FX data.","feed_headline":"ConvNet strategy posts 20% gain on flat EUR/USD data","feed_subtitle":"Paper pits ConvTimeNet against small-data models and random trades on 100,000 minute-level samples.","key_machinery":"The central mechanism is a three-class labeling scheme combined with a fully convolutional deep network. Returns are shifted and split at the 33rd and 67th percentiles into -1, 0, +1 classes; the network, implemented in Appendix E as a two-block 1D CNN with batch normalization, ReLU, adaptive average pooling, dropout, and a linear head, is trained with cross-entropy loss and the Adam optimizer. Strategy returns multiply the predicted signal by the realized return and are accumulated to produce the comparison curves against market and random strategies. The text describes the model as ConvTimeNet, with deformable patch embedding and fully convolutional blocks, and stresses the O(n·k) complexity of convolutions versus O($n^{2}$) attention as the scaling advantage.","core_discovery":"The paper's central claim is that the right tool for financial time series classification shifts with data scale: on small datasets, classical models such as SVMs and XGBoost can appear to beat the market but are really overfitting, while on big datasets a fully convolutional network can capture exploitable structure. Applying a ConvTimeNet-style architecture to 100,000 minute-level EUR/USD samples with 200 features, the author reports a cumulative strategy return of 20% while the market stays flat, and takes this as evidence that the model found profitable short-term patterns. The paper also reports that in the small-data regime, final cumulative returns deteriorate as the number of observations grows from 500 to about 1,700, which it reads as a decreasing signal-to-noise ratio rather than a reason to prefer more data.","pith_inferences":["Appendix E trains on all 100,000 EUR/USD samples and then computes the strategy return on the same samples, so the 20% figure is an in-sample statistic; the paper does not report a temporal holdout, and that is the test that would turn the claim into evidence of out-of-sample skill.","The random-strategy comparison is a single random draw; averaging many random seeds would show whether the model's edge is wider than the noise of random trading.","Because the 33rd and 67th percentile thresholds are computed on the full dataset, the labels may embed future information; computing the thresholds in a rolling or walk-forward manner would be a direct robustness check.","The Appendix E code implements a simple two-convolution network rather than the deformable-patch ConvTimeNet described in the text, so the reported result should be attributed to the implemented architecture."],"forward_implications":["Big-data convolutional classifiers can in principle generate positive strategy returns on minute-level FX data even when the underlying market is flat, which is the paper's main evidence for learnable structure.","Small-data financial backtests should be regarded as overfit unless validated on data the model has not seen, because the same workflow that looks strong on Strike fails on Bitcoin.","Convolutional architectures scale better than attention-based ones for high-frequency multivariate series, making them a practical choice as data volume grows.","More observations do not automatically improve classification accuracy; in the small-data experiments, performance falls as the sample grows, suggesting signal quality matters as much as quantity."],"supporting_citations":[{"why":"Supplies the ConvTimeNet architecture that the big-data EUR/USD experiment is built on.","marker":"[Cheng et al., 2024]"},{"why":"Provides the deep-learning direction-classification approach and the up-to-68% accuracy benchmark the paper contrasts with classical methods.","marker":"[Dixon et al., 2017]"},{"why":"Gives the trend-based labeling method used to construct the three-class target from returns.","marker":"[Wu et al., 2020]"},{"why":"Defines the big-data setting and the small-data generalization problem that motivates the comparison.","marker":"[Goldstein et al., 2021]"},{"why":"Supplies the unified deep-learning framework for time series classification that frames the model zoo.","marker":"[Ismail Fawaz et al., 2019]"},{"why":"Raises the adaptive-markets caveat that shapes the conclusion about whether more data keeps helping.","marker":"[Lo, 2004]"}],"fun_headline_variants":["ConvNet returns 20% on flat EUR/USD, big data key","Scale matters: CNN hits 20% on flat market, SVMs overfit","Big data unlocks 20% gain with ConvTimeNet on EUR/USD","ConvTimeNet: 20% on flat EUR/USD, small data misleading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result assumes that computing trading returns from predictions on the training set says something about future performance, since the model is trained and evaluated on the same 100,000 samples.","fun_headline_variants_meta":{"raw":{"variants":["ConvNet returns 20% on flat EUR/USD, big data key","Scale matters: CNN hits 20% on flat market, SVMs overfit","Big data unlocks 20% gain with ConvTimeNet on EUR/USD","ConvTimeNet: 20% on flat EUR/USD, small data misleading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1651,"prompt_tokens":750,"completion_tokens":901,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":366,"completion_tokens_details":{"reasoning_tokens":816}},"tokens_in":366,"tokens_out":901,"duration_ms":7654,"temperature":1.0,"reasoning_tokens":816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:33:28.168675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ConvTimeNet on the first 80,000 minute-level EUR/USD samples and freeze the model, then compute strategy returns on the last 20,000 samples; if the 20% excess return shrinks to the random or market baseline, the paper's claim that the model identifies exploitable patterns is unsupported.","supporting_citations":[{"cited_title":"ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis","cited_arxiv_id":"2403.01493","evidence_quote":"Supplies the ConvTimeNet architecture that the big-data EUR/USD experiment is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the deep-learning direction-classification approach and the up-to-68% accuracy benchmark the paper contrasts with classical methods."},{"cited_title":"S., and Ye, M","cited_arxiv_id":null,"evidence_quote":"Defines the big-data setting and the small-data generalization problem that motivates the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the unified deep-learning framework for time series classification that frames the model zoo."}],"review_version":1}