{"id":"ce100a02-857d-4c5c-b374-93cb66aebd13","arxiv_id":"2412.19932","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"Hidformer matches classic neural network baselines on 128-day stock close price forecasts for six large-cap stocks, with no clear advantage after accounting for evaluation flaws.","lead":"The authors apply Hidformer, a Transformer variant for long time series, to daily stock price forecasting and report it roughly matches or slightly beats CNN, RNN, LSTM, and DNN baselines. The paper is a straightforward application study, but its evaluation has data leakage and missing baselines that limit what can be concluded.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-sample min-max normalization in Eq. (1) leaks validation-period statistics into training, so the reported out-of-sample gains of Hidformer are not trustworthy.","rationale":"The reader's weakest assumption identifies the same load-bearing issue I would stress: full-sample normalization in Eq. (1) includes the validation period, making the reported results look-ahead contaminated. Section 4.1 explicitly says 'the entire sample,' and Section 5.1 confirms that the last 5% of the data is the validation set, so this is not a misreading. The contamination affects every number in Tables 2-4, which are the only quantitative support for the paper's central claim that Hidformer outperforms or matches existing basic models. Even if the effect is numerically small, the validity of the out-of-sample interpretation is at stake. Other issues such as marginal metric differences, lack of transaction costs, absence of a buy-and-hold baseline, and no released code/data are real but secondary; they would weaken the practical claim, whereas the normalization leakage undermines the measurement itself. I find no independent support in the paper that would offset this: there is no formal verification, no reproducibility artifact, and the visual analytics are illustrative rather than quantitative. My recommendation is therefore not to change the reader's REJECT verdict. The concrete test of retraining with training-only normalization statistics would settle whether the concern lands, because it directly removes the leakage while keeping the rest of the experimental setup intact.","tokens_in":100,"tokens_out":3899,"duration_ms":101811,"concrete_test":"Re-run the complete pipeline with Eq. (1) replaced by xmin/xmax computed only from the first 95% training window (or with an expanding/rolling window that never peeks past the current prediction point), keeping all hyperparameters and the validation split otherwise identical. Compare Table 2 row Hidformer vs CNN and Table 4 net values. If Hidformer no longer outperforms or matches on MAE/MSE/MAPE, or if its net-value advantage shrinks or reverses, the look-ahead normalization is the cause and the central claim fails. As an additional check, reserve a final holdout segment untouched by model selection and report metrics on it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines normalization in Eq. (1) with xmax and xmin taken from 'the entire sample.' Section 5.1 then splits the data so the last 5% (31/12/2021 to 31/12/2023) is the validation set used for evaluation. Thus the scaling constants used to build every training window are computed from the same future period the model is later asked to predict. This leaks distributional information about the validation period into both the input features and the normalized targets during training. The reported MAE/MSE/MAPE values in Table 2 and the net values in Tables 3-4 are therefore not clean out-of-sample measurements; they depend on future data. The same validation split is also used for model selection ('we selected the model chosen as the best during training'), so the final numbers are selected, not purely predictive. Because the central claim is an empirical superiority/matching claim, this contamination is directly load-bearing: if the normalization were redone with training-only statistics, the rankings and significance levels could change.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies Hidformer, an existing hierarchical dual-tower Transformer architecture for long-term time series, to daily stock price forecasting. Using daily open/high/low/close/adjusted-close/volume data for six consumer-goods stocks from 1980 to 2023, the authors train Hidformer and CNN/RNN/LSTM/DNN baselines on a 128-day-input/128-day-output moving-window task and evaluate them with MAE, MSE, MAPE, a sign-based trading strategy, and visual inspection. The central claim, stated in Section 1, is that Hidformer outperforms or matches the existing baselines. The experiments use a chronological 95/5 split, with the last 5% of data treated as a validation set.","tokens_in":7714,"tokens_out":5271,"duration_ms":55496,"significance":"If the experimental results were clean, the paper would offer a modest data point on the applicability of a recent Transformer-style architecture to daily stock price forecasting. Its strengths include a clearly specified set of hyperparameters, five independent runs, and evaluation through both error metrics and a trading backtest. However, the evaluation protocol contains a full-sample normalization look-ahead and uses the same validation window for model selection and final scoring, so the reported numbers cannot currently be interpreted as out-of-sample evidence. The paper does not provide code or data, and it makes no theoretical or architectural contribution beyond applying an existing model, so its value is entirely empirical. Because the empirical evaluation is contaminated, the significance of the claimed finding is not assessable at present.","major_comments":[{"comment":"Equation (1) normalizes the data using xmax and xmin computed from 'the entire sample.' Section 5.1 then designates the last 5% of the data (31/12/2021 to 31/12/2023) as the validation set. Consequently, the scaling constants used to build every training window are derived from the future period the model is later asked to predict. This leaks validation-period statistics into both the training inputs and the normalized targets, so the MAE/MSE/MAPE values in Table 2 and the net values in Tables 3–4 are not clean out-of-sample measurements. The normalization should be recomputed using statistics from the training portion only (or an expanding window), and all reported metrics and trading results should be regenerated under that protocol.","section":"§4.1, Eq. (1), with §5.1"},{"comment":"The same 5% validation window is used both for model selection ('we selected the model chosen as the best during training') and for computing all reported evaluation metrics. No held-out test set or nested resampling procedure is described. The reported errors and trading returns are therefore selected on the same data used for early stopping, which can inflate apparent performance. The authors should introduce a true held-out test period or a time-series cross-validation scheme, and report metrics separately for model selection and final evaluation.","section":"§5.1 and §5.2"},{"comment":"The text repeatedly states that Hidformer is 'significantly' better than the baselines with 'P-values lower than 0.01,' but it does not report the Mann-Whitney U test procedure, the exact p-values, the number of pairwise comparisons, or any correction for multiple testing. This matters because the differences in Table 2 are within one standard error on several metrics (e.g., MAE 0.159 vs. 0.160 for CNN; MSE 0.040 vs. 0.039 for CNN), so the significance claims are under-supported as reported. The authors should provide the full test setup, exact p-values for each metric and pairwise model comparison, and an appropriate multiplicity adjustment.","section":"§5.4, Tables 2–4"},{"comment":"The trading strategy in Eq. (2) uses a one-day-ahead prediction ŷ_{t+1}, but the model is trained to output Ty = 128 future close prices. It is not explained how ŷ_{t+1} is extracted from the multi-horizon output (e.g., the first horizon only, the last horizon, or an aggregation). This choice directly affects the backtest returns in Tables 3 and 4, so the authors should specify and justify the extraction rule and, ideally, evaluate all horizons consistently.","section":"§5.3, Eq. (2)"}],"minor_comments":[{"comment":"Table 3 reports mean net values without standard errors, while Table 4 includes standard errors for the same quantities; reporting errors consistently across both tables would aid interpretation.","section":"§5.4, Tables 3–4"},{"comment":"The figure captions do not identify which stock or time interval is shown, and the plots lack clear axis labels and legends; adding these would make the visual analysis reproducible and interpretable.","section":"§5.5, Figures 4–6"},{"comment":"The term 'validation set' is used for the final evaluation data, but no test set is defined; using 'test set' for the final evaluation would avoid ambiguity about model selection.","section":"Throughout"},{"comment":"Figure 3 is captioned as 'Hidformer model overwiew. [16]' but the source reference appears to be to the original Hidformer paper through citation [8]; the caption should cite the original source and clarify whether the figure is reproduced or adapted.","section":"§4.2, Figure 3"},{"comment":"Several references are incomplete (e.g., [11] and [14] lack full bibliographic details such as publishers or page ranges); the reference list should be brought to journal style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the evaluation protocol, rather than the model, drives the positive conclusions. The full-sample normalization and the shared validation/model-selection split are fixable by re-running the experiments with a proper training-only scaling and a held-out test period, so I do not recommend outright rejection. However, if the re-run changes the rankings or the significance pattern, the paper's central claim will need to be substantially reframed, and the authors should be prepared for that possibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies Hidformer, an existing time-series transformer, to stock index forecasting with a 128-day horizon. That is a legitimate new application, and the authors compare fairly against CNN, RNN, LSTM, and DNN, reporting standard errors over five runs and a simple trading backtest. The writing is clear, and the model description is competent.\n\nThe soft spots are load-bearing. Equation (1) normalizes using min/max over the entire sample, and the last 5% of data (2022–2023) is held out as validation. So the scaling constants for training inputs are computed from the future period the model is later asked to predict. That is look-ahead bias. The same 5% split is also used both for model selection and for the reported metrics, with no held-out test set. The backtest ignores transaction costs and has no buy-and-hold baseline, so the net-value results are hard to interpret.\n\nThe claimed superiority is marginal even in the paper's own Table 2: Hidformer's MAE is 0.159 vs CNN's 0.160, its MSE is 0.040 vs CNN's 0.039 (slightly worse), and MAPE is tied at 66. The Mann-Whitney tests are mentioned but the details are sparse, so the significance claims are not fully supported.\n\nThis is not a careless or deceptive paper; the normalization is stated openly. But the central empirical claim is not supported by the experimental design as reported. If the authors fixed the normalization and used a proper train/validation/test split, they might end up with a modest application study. As is, it is mainly a teaching example of look-ahead bias. No code or data is provided, so reproducibility is limited.\n\nI would not send this to peer review as is. The flaw is easy to spot and it is fatal to the main claim. If the authors redo the evaluation with training-only normalization and a real test set, the revised version would be worth another look.","headline":"Full-sample normalization and validation reuse make Hidformer's reported edge in stock forecasting unverifiable.","tokens_in":8291,"tokens_out":2510,"would_cite":false,"duration_ms":25222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hidformer, a hierarchical dual-tower Transformer, predicts 128-day stock price paths with error and trading-return results that match or beat CNN, RNN, LSTM, and DNN baselines on six consumer-goods stocks.","keywords":["stock price forecasting","Transformer","Hidformer","time series prediction","technical analysis","deep learning","financial time series","trading strategy"],"falsifier":"Recompute all experiments with min-max normalization fitted only on the training portion (or on a rolling window that never sees the validation period), and compare Hidformer's MAE, MSE, MAPE, and net values against the same baselines; if Hidformer no longer has the lowest MAE or the best net values, the central 'outperforms or matches' claim fails for a genuinely out-of-sample setup.","tokens_in":7275,"feed_emoji":"📈","tokens_out":7107,"duration_ms":61704,"temperature":0.7,"pith_summary":"The paper asks whether Hidformer—a Transformer-style architecture designed for long-term time-series forecasting—can be carried over to stock price prediction. The authors train it on 128 days of daily open, high, low, close, adjusted close, and volume data from six large consumer-goods stocks, and test whether it can forecast the close price over the next 128 days. Their central claim is that Hidformer either outperforms or matches CNN, RNN, LSTM, and DNN baselines on standard error metrics and on a simple long/short trading strategy, with many differences reported as statistically significant. In the headline numbers, Hidformer posts the lowest MAE (0.159) and a MAPE (66) tied with the best baselines, while its MSE (0.040) is essentially tied with CNN's 0.039. If the claim holds, Transformer-style forecasting becomes a credible, practical option for multi-step financial prediction and trading support.","feed_headline":"Hidformer matches or beats classic nets on 128-day stock forecasts","feed_subtitle":"Hidformer posts the lowest MAE and competitive backtest returns across six consumer stocks.","key_machinery":"The load-bearing object is Hidformer, a hierarchical dual-tower Transformer that feeds the input through a time-domain encoder with recursive attention and a frequency-domain encoder with linear attention, splitting the series into tokens and merging the segments after each block; the merged outputs feed an MLP decoder that produces all 128 forecast days in one pass. It is paired with a weighted MSE loss (weights running from $T_y=128$ down to 1) and a moving-window scheme in which the last 128 days of prices and volumes predict the next 128 close prices. Statistical significance is assessed with a Mann-Whitney U test across the five runs.","core_discovery":"On the paper's own terms, the discovery is that a hierarchical dual-tower Transformer—Hidformer—can be adapted to multi-step stock price forecasting with performance at least comparable to, and on several metrics better than, standard deep learning baselines. Hidformer processes the 128-day input through two encoder towers: one in the time domain with recursive attention, one in the frequency domain with linear attention, merging multi-scale segments before an MLP decoder outputs 128 future close prices. Across five independent training runs on six consumer-goods stocks, the model achieves the lowest MAE (0.159), a MAPE of 66 tied with CNN and DNN, and an MSE of 0.040, marginally above CNN's 0.039. The paper also reports that a long/short rule driven by Hidformer's one-day-ahead prediction produces net values that are often significantly higher ($P<0.01$) or similar to the baselines over a two-year validation period, and that visual analysis shows it tracking upward, downward, and mixed trends. The authors interpret this as empirical validation of Hidformer's adaptability to financial time series and its potential as a decision-support tool.","pith_inferences":["A consequence the authors leave implicit is that the full-sample min-max normalization makes the validation numbers not strictly out-of-sample; refitting the scaler on the training window only would tell whether the reported MAE advantage survives a genuinely forward-looking setup.","Because MSE still favors CNN (0.039 vs 0.040), a user whose loss function penalizes large errors more heavily might prefer CNN; the choice of metric, not just the model, determines the practical winner.","The paper's own future-work suggestion of per-sector decoders points to a testable extension: training one decoder per industry (or conditioning on a sector vector) could prevent the encoders from learning contradictory formations and improve cross-industry generalization.","The 128-day horizon with a single MLP decoder could be combined with uncertainty estimation (for example, ensembles or quantile heads) to produce prediction intervals for risk management; the paper does not report intervals, but the architecture does not preclude them."],"forward_implications":["For practitioners, Hidformer offers a workable alternative to CNN, RNN, LSTM, and DNN when the goal is a 128-day close-price forecast, with no extra financial feature engineering beyond min-max scaling.","Because the decoder emits all 128 future days at once, the model can sketch a trend path rather than only the next tick, which is what makes it useful for human decision-making.","The reported net-value improvements imply that a simple sign-of-prediction long/short strategy on consumer-goods stocks could earn higher or similar backtested returns than the same strategy driven by classical networks.","The statistical significance claims (Mann-Whitney U, $P<0.01$ on many comparisons) mean the observed differences are unlikely to be artifacts of a single training run.","The two-tower frequency/time design appears to transfer from the original long-term series benchmarks (Weather, Traffic, Electricity) to financial data, supporting the general claim that Transformer-style attention is competitive for financial time series."],"supporting_citations":[{"why":"Defines the Hidformer architecture with two-tower time/frequency encoders, segment-and-merge, and recursive/linear attention that this paper adapts.","marker":"[8]"},{"why":"Supplies the long-sequence forecasting lineage and the MLP-style decoder used to generate all 128 future days in one operation.","marker":"[16]"},{"why":"Provides the canonical Transformer attention mechanism that Hidformer modifies.","marker":"[12]"},{"why":"Gives the stock-index Transformer prediction setting, the comparison with CNN/RNN/LSTM, and the long/short net-value trading strategy the paper reuses.","marker":"[13]"},{"why":"Defines the CNN/RNN/LSTM/DNN baseline families and the survey context motivating hybrid and Transformer approaches.","marker":"[5]"},{"why":"Motivates the technical-analysis premises and the six daily quote components (open, high, low, close, adjusted close, volume) used as inputs.","marker":"[10]"},{"why":"Documents earlier Transformer adaptations for time series (locality and memory bottleneck fixes) that frame the architecture's design choices.","marker":"[7]"}],"fun_headline_variants":["Hidformer posts lowest MAE in 128-day stock test","Hidformer edges classic nets on stock forecasts","Transformer variant tops stock price prediction","Hidformer outperforms baselines in stock forecast","Hidformer leads stock forecast accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the minimum and maximum values used to scale the data may be computed from the entire sample, including the final 5% validation period; if the scaler is restricted to training data only, every reported error and trading return could change.","fun_headline_variants_meta":{"raw":{"variants":["Hidformer posts lowest MAE in 128-day stock test","Hidformer edges classic nets on stock forecasts","Transformer variant tops stock price prediction","Hidformer outperforms baselines in stock forecast","Hidformer leads stock forecast accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1363,"prompt_tokens":940,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":556,"tokens_out":423,"duration_ms":4931,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:45:56.210958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute all experiments with min-max normalization fitted only on the training portion (or on a rolling window that never sees the validation period), and compare Hidformer's MAE, MSE, MAPE, and net values against the same baselines; if Hidformer no longer has the lowest MAE or the best net values, the central 'outperforms or matches' claim fails for a genuinely out-of-sample setup.","supporting_citations":[{"cited_title":"Expert Systems with Appli- cations 239, 122412 (Apr 2024)","cited_arxiv_id":null,"evidence_quote":"Defines the Hidformer architecture with two-tower time/frequency encoders, segment-and-merge, and recursive/linear attention that this paper adapts."},{"cited_title":"A Survey of Forex and Stock Price Prediction Using Deep Learning","cited_arxiv_id":"2103.09750","evidence_quote":"Defines the CNN/RNN/LSTM/DNN baseline families and the survey context motivating hybrid and Transformer approaches."},{"cited_title":"New York Institute of Finance (1999)","cited_arxiv_id":null,"evidence_quote":"Motivates the technical-analysis premises and the six daily quote components (open, high, low, close, adjusted close, volume) used as inputs."}],"review_version":1}