{"id":"0b3b9037-5741-4dce-91e0-41154897dcf1","arxiv_id":"2507.16548","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"Transformer models with the MADL loss are claimed to beat LSTM and buy-and-hold on six assets, but the non-differentiable loss makes the described training impossible.","lead":"This paper tests whether a Transformer model trained with a custom direction-based loss, MADL, beats LSTM and buy-and-hold on six daily-traded assets. The authors report higher risk-adjusted returns for the Transformer in most cases, but the loss function is not differentiable, so the training procedure as described cannot actually update the model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MADL's zero gradient makes the described training loop unable to update weights, so the central empirical claim lacks a credible mechanism.","rationale":"The paper's central contribution is the use of MADL as a training loss for Transformer and LSTM models in algorithmic trading. The reader's weakest assumption identifies that sign(R_i·R̂_i) has zero derivative with respect to R̂_i almost everywhere, making the described backpropagation procedure unable to update weights. This is exactly the same load-bearing concern I find. It is confirmed by the paper's own text: §3.4 describes a standard training loop with weight updates based on the lowest loss value, §3.5 states that the model minimizes MADL, and the conclusion admits non-differentiability without explaining how it was handled. Since no surrogate or straight-through estimator is described, the empirical results in Tables 2 and 3 lack a credible mechanism by which the MADL objective shaped the model parameters. This is not a matter of disagreeing with the existing literature; it is an internal consistency problem with the reported methodology. The verdict of REJECT is appropriate, and the path to revision is to specify and test a differentiable surrogate or to use MADL only for validation and checkpointing. No other concern is needed to justify the rejection.","tokens_in":8798,"tokens_out":2590,"duration_ms":27282,"concrete_test":"Reproduce the §3.4 training setup in Keras/TensorFlow with the exact loss of Eq. (4) and log the gradient norm after the first optimizer step, or compare model weights before and after one epoch, on any one asset. If gradient norms are exactly zero and weights are unchanged while an MSE-trained control updates normally, the MADL loss as written cannot drive learning; the paper must then specify the actual differentiable objective used to produce Tables 2–3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusions claim that Transformer models trained with MADL outperform LSTM and Buy&Hold. The load-bearing condition is that MADL can actually train the networks in the Keras/TensorFlow loop described in §3.4. Eq. (4) defines MADL = (1/N) Σ (-1)·sign(R_i·R̂_i)·|R_i|. For fixed observed R_i, the derivative of sign(R_i·R̂_i) with respect to R̂_i is zero almost everywhere because sign is piecewise constant, and |R_i| does not depend on R̂_i; therefore ∂MADL/∂R̂_i = 0 almost everywhere. A standard backpropagation pass with this loss produces zero gradient, so no weight updates occur and the model is never actually trained by minimizing MADL. The conclusion even acknowledges 'the problem of its non-differentiability in certain areas ([14])', but the paper never states which surrogate, straight-through estimator, or two-stage procedure was used. If none was used, the reported strategy results cannot be produced by the described training procedure. This is an internal inconsistency, not merely a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the Mean Absolute Directional Loss (MADL), introduced in the authors' earlier work, as the training objective for Transformer and LSTM models that generate long/short signals on daily returns for three equities and three cryptocurrencies. A walk-forward procedure with a multi-year out-of-sample period is used, and strategy performance is summarized by annualized return, volatility, drawdown, and several information-ratio variants. The central claim, stated in the abstract and conclusions, is that Transformer models trained with MADL produce significantly better risk-adjusted returns than LSTM models and Buy&Hold in almost every asset considered.","tokens_in":9016,"tokens_out":4738,"duration_ms":51419,"significance":"If the central empirical claim were credible, the paper would offer useful evidence on Transformer architectures for algorithmic investment strategies, with strengths including a multi-asset design, long out-of-sample periods, walk-forward validation, and comparison against both a classical benchmark (Buy&Hold) and a strong baseline (LSTM). The paper also makes a methodological point about selecting loss functions for financial forecasting. However, the reported results depend on the described training procedure actually minimizing MADL, and that mechanism is not credible as written; moreover, the 'significantly better' language is unsupported by statistical inference, and the strategy returns appear to be gross of transaction costs despite high turnover. These issues place the main contribution in doubt.","major_comments":[{"comment":"The loss in Eq. (4) has zero gradient with respect to the predicted return almost everywhere. Because sign(R_i * R_hat_i) is piecewise constant, its derivative with respect to R_hat_i is zero except at the single point R_hat_i = 0, and abs(R_i) does not depend on R_hat_i, so ∂MADL/∂R_hat_i = 0 almost everywhere. In a standard Keras/TensorFlow backpropagation loop with the Adam optimizer, as described in §3.4, this produces zero weight updates, meaning the models are not trained by minimizing MADL. The paper notes in the Conclusions that MADL has 'the problem of its non-differentiability in certain areas', but it never specifies a surrogate gradient, a straight-through estimator, or a two-stage training procedure. The reported strategy results therefore lack a credible learning mechanism as described; the authors must either define and implement a differentiable version of the loss, provide code or pseudocode showing the actual training objective, and rerun the experiments, or substantially weaken the claims.","section":"§3.5, Eq. (4); §3.4; Conclusions"},{"comment":"The claim that Transformer results are 'significantly better' than LSTM results is not supported by any statistical test. Tables 2 and 3 report point estimates for a single walk-forward path, with no confidence intervals, no bootstrap or Diebold-Mariano tests of return differences, and no correction for multiple comparisons across six assets. Without such procedures, the word 'significantly' is unjustified. The authors should either add appropriate significance tests for the reported performance metrics or replace 'significantly' with descriptive language such as 'consistently higher in the sample'.","section":"Abstract; §4, Tables 2-3"},{"comment":"The reported strategy performance appears to be gross of transaction costs, yet the strategies execute many trades: nTrades ranges from about 1031 to 1723 over the out-of-sample period, and the daily signal construction implies potentially frequent position changes. For the claim that these strategies yield 'abnormal risk-adjusted returns' in a realistic setting, costs are a first-order concern. The authors should state whether the reported returns are net of transaction costs and, if not, provide a sensitivity analysis under realistic cost assumptions for equities and cryptocurrencies.","section":"§4, Tables 2-3; §3.6"}],"minor_comments":[{"comment":"The data description says Ethereum starts on '2015-08-07', but this is likely a typo for Litecoin; additionally, the table header 'EHT' should be 'ETH', and the table note contains a duplicated 'Note:'.","section":"§3.3, Table 3"},{"comment":"'ChatGTP' should be 'ChatGPT'.","section":"§1"},{"comment":"The description of the training window is inconsistent: §3.3 says the test set size is '252/365 days' and Table 1 says 'Train/test size 252/252', while §3.4 describes an expanding window that grows 'up to 4 years'. Please clarify whether the training window expands without bound or is capped, and state the exact units (trading days versus calendar days).","section":"§3.3, §3.4, Table 1"},{"comment":"In Eq. (10), the notation 'ARC' should probably be 'aRC' for consistency with Eq. (5). Also, the bullet list in §3.3 mentions 'IR, IR, IR' where three distinct information ratios are presumably meant.","section":"§3.6, Eq. (10)"},{"comment":"References [12] and [2] each appear twice in the sentence listing common flaws in backtesting studies; please remove the duplicates.","section":"§2"},{"comment":"The notation in Eq. (4) mixes 'abs(R_i)' with the text; it would be clearer to use |R_i| throughout. The behavior at R_i = 0 or R_hat_i = 0 is also undefined and should be specified.","section":"§3.5, Eq. (4)"}],"recommendation":"reject","confidential_remarks":"The fundamental problem is that the described training procedure cannot, as written, learn from MADL, and the 'significantly better' claim is not backed by statistical tests. Fixing the training requires a new loss formulation or surrogate and a full rerun of the experiments, which is beyond a normal revision. I would encourage the authors to resubmit after addressing these issues; the multi-asset walk-forward design is a useful framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper takes the authors' MADL loss, applies it to a Transformer, and compares against LSTM and buy-and-hold on six assets with an expanding-window out-of-sample design. That is a reasonable thing to try, and the OOS design is more careful than much of this literature. But the core training claim is broken as written: MADL in Eq. (4) is (1/N) sum -sign(R_i*R_hat_i)*abs(R_i). For fixed observed R_i, the gradient with respect to R_hat_i is zero almost everywhere, so Adam has nothing to update. The paper never specifies a surrogate, straight-through estimator, or two-stage procedure. The conclusions even mention non-differentiability \"in certain areas\" as future work, citing [14], but the results tables treat the model as trained by MADL. This is not a stylistic issue; the reported equity curves cannot be produced by the described pipeline.\n\nWhat the paper does well: it targets real methodological pitfalls in backtesting—walk-forward validation, long out-of-sample periods, two asset classes, and some overfitting controls—and the six-asset comparison is more extensive than many LSTM-vs-Transformer studies. The novelty is modest: MADL is the authors' own from [14,15], and Transformer versus LSTM is a common comparison. This is the first combination of MADL with a Transformer on these assets, but it is an incremental extension.\n\nSecondary problems, which would matter even after fixing the gradient issue: \"significantly better\" appears with no significance tests; the six assets were chosen partly for long history and index membership, introducing survivorship bias; there are no transaction costs despite 1000+ trades per strategy; and no code or data are released. The reference list also has some duplicated entries, minor in itself.\n\nThe central concern raised in the stress-test note holds up. This is an internal inconsistency, not a disagreement about methodology choices. I would not send this to referees in its current form. The right next step is for the authors to specify a differentiable surrogate for MADL (or use it only as a validation metric), report seed variability and cost-adjusted returns, and release code. If they do that, the empirical comparison could be informative. As it stands, this is a useful cautionary example for a reading group, not a citable result.","headline":"The paper applies the authors' MADL loss to Transformers on six assets with a long walk-forward out-of-sample setup, but the loss as written has zero gradient, so the described training loop cannot update the network and the headline results are not credible as reported.","tokens_in":9566,"tokens_out":4961,"would_cite":false,"duration_ms":54342,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformer models trained with a directional loss beat LSTM and buy-and-hold on six assets.","keywords":["Deep Learning","Neural Networks","LSTM","Algorithmic Investment Strategies","Loss Function","Transformer","Mean Absolute Directional Loss","Walk-forward validation"],"falsifier":"Compute $\\partial \\mathrm{MADL}/\\partial \\hat{R}_i$: it is zero for every $\\hat{R}_i$ except at $\\hat{R}_i = 0$. Then train a small network using exactly the described setup with MADL as the loss and Adam as the optimizer; if the weights remain unchanged after several epochs, the reported trading results cannot be produced by learning under the loss as written.","tokens_in":8582,"feed_emoji":"📈","tokens_out":15501,"duration_ms":123468,"temperature":0.7,"pith_summary":"This paper claims that the choice of training loss function is as important as model architecture in machine-learning trading systems. The paper applies a recently proposed Mean Absolute Directional Loss (MADL) to Transformer models, and reports that the resulting strategies outperform both LSTM models and a simple buy-and-hold benchmark on six assets: three equities (JP Morgan, S&P 500, Exxon Mobil) and three cryptocurrencies (Bitcoin, Ethereum, Litecoin). The out-of-sample period spans eight to nine years under a walk-forward expanding-window procedure. If the claim is right, training forecasts with an objective that directly rewards correct direction and scales with realized move size is a concrete way to improve risk-adjusted returns in algorithmic trading.","feed_headline":"Transformer with directional loss beats LSTM and buy-and-hold","feed_subtitle":"Over 8-9 years, out-of-sample tests on stocks and crypto show higher risk-adjusted returns.","key_machinery":"The central object is the Mean Absolute Directional Loss (MADL), defined as $\\mathrm{MADL} = \\frac{1}{N}\\sum_{i=1}^{N} (-1)\\,\\mathrm{sign}(R_i\\,\\hat{R}_i)\\,|R_i|$, where $R_i$ is the realized return and $\\hat{R}_i$ the model's predicted return for interval $i$. The loss rewards a correct directional call with the absolute size of the realized move and punishes a wrong call the same way, so minimizing it is equivalent to maximizing the profit of a strategy that goes long or short according to the predicted sign. The paper uses MADL as the training objective inside a standard gradient-based deep-learning loop with the Adam optimizer, together with a walk-forward expanding-window scheme that retrains on a growing window and predicts one year ahead.","core_discovery":"The paper's central empirical claim is that Transformer models trained with MADL generate trading signals whose risk-adjusted performance exceeds both LSTM and buy-and-hold on every asset tested. Across six assets, the Transformer attains the highest values on all reported information ratios, which annualize return and penalize volatility, drawdown, and loss duration. For example, on Bitcoin the Transformer attains a drawdown-adjusted information ratio (IR**) of 5.301, versus 1.939 for LSTM and 1.287 for buy-and-hold. The paper attributes the edge to the loss function: because MADL penalizes wrong directional forecasts in proportion to the magnitude of the realized move, the model learns to prioritize large correct bets rather than minimizing squared error.","pith_inferences":["A critical reading of Eq. (4) suggests that the reported training may have actually used a differentiable surrogate of the sign function, such as a straight-through estimator or a soft sign; if so, the paper's central comparison is really about that surrogate, not about MADL as written.","The two architectures are compared under different hyperparameters (learning rate 0.5 vs 0.01, 300 vs 50 epochs, different regularization), so the 'Transformer versus LSTM' conclusion conflates architecture choice with tuning; an equal-budget comparison would isolate the loss-function effect.","A natural follow-up experiment would train both models with a differentiable approximation of MADL and with mean squared error, holding all other settings fixed, to see whether the directional loss itself, rather than the choice of model, drives the reported outperformance.","The policy implications about regulation discussed in the paper may be premature until the non-differentiability issue and the hyperparameter imbalance are resolved."],"forward_implications":["If the claim holds, loss-function design becomes a first-order decision in building machine-learning trading strategies: aligning the objective with the trading rule matters as much as model architecture.","The results would support the view that Transformers, despite being designed for natural language, can be competitive on financial time series when paired with a suitable objective.","The walk-forward, expanding-window evaluation with multi-year out-of-sample periods offers a template for reducing overfitting claims in algorithmic trading research.","MADL could in principle be applied to other model classes and asset classes, extending the comparison beyond LSTM versus Transformer."],"supporting_citations":[{"why":"Introduces the MADL loss function that the paper uses as the training objective.","marker":"[14]"},{"why":"Further develops and empirically tests MADL for algorithmic investment strategies, supplying the version used here.","marker":"[15]"},{"why":"Provides the Transformer architecture with self-attention that the paper adapts to time-series forecasting.","marker":"[18]"},{"why":"Provides the LSTM architecture used as the comparison baseline.","marker":"[9]"},{"why":"Raises the challenge that Transformers may not beat simple linear models on time series, the position this paper engages with.","marker":"[23]"},{"why":"Supplies the performance metrics (aRC, aSD, MD, IR) used to evaluate the strategies.","marker":"[27]"},{"why":"Also supplies performance metric definitions used in the study.","marker":"[11]"}],"fun_headline_variants":["MADL loss gives Transformer edge over LSTM and hold","Transformer with directional loss tops LSTM, buy-and-hold","Directional loss boosts Transformer returns vs LSTM","Transformer plus MADL beats LSTM and buy-and-hold","MADL-trained Transformer wins over LSTM and holding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training procedure assumes MADL has a usable gradient with respect to the predicted return, but because the sign function has zero derivative almost everywhere, the loss as written cannot update network weights through ordinary backpropagation.","fun_headline_variants_meta":{"raw":{"variants":["MADL loss gives Transformer edge over LSTM and hold","Transformer with directional loss tops LSTM, buy-and-hold","Directional loss boosts Transformer returns vs LSTM","Transformer plus MADL beats LSTM and buy-and-hold","MADL-trained Transformer wins over LSTM and holding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1114,"prompt_tokens":788,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":404,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":404,"tokens_out":326,"duration_ms":3258,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:07:07.356031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $\\partial \\mathrm{MADL}/\\partial \\hat{R}_i$: it is zero for every $\\hat{R}_i$ except at $\\hat{R}_i = 0$. Then train a small network using exactly the described setup with MADL as the loss and Adam as the optimizer; if the weights remain unchanged after several epochs, the reported trading results cannot be produced by learning under the loss as written.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Raises the challenge that Transformers may not beat simple linear models on time series, the position this paper engages with."},{"cited_title":"´Slepaczuk, P","cited_arxiv_id":null,"evidence_quote":"Supplies the performance metrics (aRC, aSD, MD, IR) used to evaluate the strategies."},{"cited_title":"Kijewski, R","cited_arxiv_id":null,"evidence_quote":"Also supplies performance metric definitions used in the study."}],"review_version":1}