{"id":"0242a030-6701-45e1-b27e-a24c79141628","arxiv_id":"2508.18921","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"LSTM networks with skewed Student's t distributions can produce distributional forecasts of equity returns that are competitive with GARCH for Value-at-Risk, though calibration failures and selective reporting temper the result.","lead":"A study tests whether deep neural networks can forecast the full probability distribution of daily stock index returns, not just the average. It finds LSTM with a skewed Student's t distribution matches classical GARCH models for Value-at-Risk, but the reported calibration is mixed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PIT calibration results in Table 2 reject uniform PIT for 17 of 18 model-index combinations, undermining the claim of 'accurate distributional forecasts.'","rationale":"The reader correctly identifies the GARCH comparison as under-specified; however, the more direct threat to the central claim comes from the paper's own calibration diagnostics. The PIT p-values reported in Table 2 are not merely a missing comparison—they are internal evidence against 'accurate distributional forecasts.' When a majority of configurations reject uniformity, the probabilistic forecasts are statistically misspecified. The paper's statement that 'several configurations achieve p-values above conventional significance thresholds' is misleading because only one or two of the 18 reported combinations do so. This is a load-bearing concern because it attacks the first part of the central claim directly, and it requires no external benchmark to make the point. I therefore propose a concrete test that uses the published code to verify the calibration pattern across all six indices. If the pattern holds, the authors must soften the abstract and conclusion. The GARCH comparison issue remains, but it is secondary: the VaR results might still justify 'competitive' even if distributional accuracy is overstated, so the paper can be conditionally accepted with revisions to the accuracy claims and a fuller GARCH baseline.","tokens_in":8621,"tokens_out":5301,"duration_ms":45186,"concrete_test":"Run the published code (github.com/jmichankow/deep_learning_probability) for all six indices and compute PIT values for every model-distribution combination on the test set. Then: (1) Report the proportion of combinations with a calibrated PIT test (e.g., Anderson-Darling or chi-square with fixed bins) p>0.05, and (2) for any rejection, report the maximum absolute deviation from the uniform CDF with a 95% confidence band. If fewer than half of the combinations pass, or if the LSTM-SSTD fails calibration on more than one index, the claim of 'accurate distributional forecasts' should be revised to 'VaR exceedance rates comparable to GARCH' and the abstract amended accordingly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the neural networks produce accurately calibrated predictive distributions. The paper's own PIT p-values (Table 2) show the opposite: for the three indices reported, 17 of 18 model-distribution combinations reject uniformity at the 5% level. The headline LSTM-SSTD model has PIT p-values of 0.0309 (S&P), 2.41e-07 (Nikkei), and 5.08e-06 (KOSPI). Only CNN-SSTD for S&P (p=0.1144) and CNN-SSTD for Nikkei (p=0.0476) do not clearly reject. Moreover, DAX, WIG, and BOVESPA are absent from the distributional evaluation table, so the claim that the models 'provide accurate distributional forecasts' across six global markets is not supported by the reported evidence. Because a miscalibrated predictive distribution cannot be called accurate, the abstract's first claim is internally contradicted. The authors should either weaken this claim to 'VaR exceedance rates are competitive with GARCH' or provide a calibration analysis that demonstrates the deviations are practically negligible (e.g., via effect sizes or a test with known power).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 1D CNN and LSTM architectures for forecasting the full probability distribution of daily financial returns, using negative log-likelihood losses for Normal, Student's t, and skewed Student's t distributions. The models are evaluated with expanding-window walk-forward validation on six equity indices, with LPS, CRPS, and PIT as distributional metrics and VaR/ES backtests as risk-management metrics. The paper claims that deep learning models provide accurate distributional forecasts, that LSTM with skewed Student's t performs best, and that the models are competitive with classical GARCH models for VaR estimation.","tokens_in":9007,"tokens_out":7149,"duration_ms":66279,"significance":"If the claims were validated, the paper would provide a useful benchmark for deep-learning-based distributional forecasting in financial risk management and would extend the probabilistic deep-learning literature to daily equity returns. The strengths are the use of proper scoring rules, a walk-forward evaluation design, multiple asset classes, and the availability of source code. However, the paper's own calibration evidence undermines the central claim of 'accurate distributional forecasts': 17 of 18 reported PIT tests reject uniformity at the 5% level, including those for the preferred LSTM-SSTD model on two of three reported indices. In addition, the claim of superior performance rests on selecting the best model from the same test set without multiple-comparison correction, and the GARCH comparison is reported without estimation details or statistical tests. The empirical framework is reasonable and the code availability is a plus, but the current mismatch between the abstract/conclusion and the reported calibration statistics is substantial.","major_comments":[{"comment":"The PIT calibration results are presented as evidence supporting the abstract's claim of 'accurate distributional forecasts,' but the table reports rejection of uniformity for 17 of 18 model-index combinations at the 5% level. The headline LSTM-SSTD model has PIT p-values of 0.0309 (S&P 500), 2.41e-07 (Nikkei 225), and 5.08e-06 (KOSPI). Only CNN-SSTD for S&P 500 (p=0.1144) does not reject. Moreover, Table 2 reports only three of the six indices, while the abstract claims evaluation across six global markets; DAX, WIG, and BOVESPA are absent from the distributional evaluation, despite Figure 2 showing DAX PIT histograms without p-values. The claim of well-calibrated distributional forecasts is therefore contradicted by the paper's own reported evidence. The authors should either weaken the claim to 'VaR exceedance rates are competitive' or provide a calibration analysis with effect sizes,","section":"Section 5.1, Table 2"},{"comment":"The 'best' model is selected from the same test data on which its performance is reported, and no multiple-comparison correction is applied. Table 2 identifies LSTM-SSTD as best by lowest LPS/CRPS, but no significance tests (e.g., Diebold-Mariano or a test for equal predictive ability) are provided for the differences. Similarly, Table 5 selects the 'Best NN Model' and 'Best GARCH' per index and per VaR level after inspecting the results, so the reported exceedance rates are post hoc selections and are not statistically protected. This makes the headline claim 'LSTM-SSTD performs best' and the broader viability claim weaker than stated. The authors should report all models' results without selection, correct for multiple testing, or use a genuinely separate holdout for model choice.","section":"Sections 5.1-5.2, Tables 2-5"},{"comment":"The comparison with classical GARCH models is not supported by the reported evidence. No estimation details, distributional assumptions beyond the labels (G, AP, E, GJR), or diagnostics for the GARCH baselines are given; no Kupiec or Christoffersen test results are reported for the GARCH models; and only four of the six indices appear in Table 5. Without these details, the claim that deep learning models 'perform competitively with classical GARCH models' cannot be verified. At minimum, the authors should provide the full set of GARCH results with the same backtests and p-values, and clarify why WIG and BOVESPA are excluded from this comparison.","section":"Section 5.3, Table 5"},{"comment":"The PIT p-values in Table 2 are central to the calibration claim, but the paper never specifies which statistical test was applied to the PIT sequence (e.g., Kolmogorov-Smirnov, Anderson-Darling, Berkowitz). With 2,487 out-of-sample forecasts, even small deviations from uniformity can produce very small p-values. The test statistic and a discussion of effect size are necessary for the reader to interpret whether the rejections are practically significant. The exact repetition of p=2.41e-07 across multiple rows also suggests a possible reporting artifact; please report exact p-values and test statistics.","section":"Section 3.5, Eq. (20)"}],"minor_comments":[{"comment":"The moments for the skewed Student's t distribution are stated as E(X)=φσ and Var(X)=(γ−φ²)σ², omitting the location parameter μ. If μ is the location, the mean should be μ+φσ; if μ is the mean, the parameterization in Eq. (11) and the NLL in Eq. (16) are inconsistent. Please clarify the exact parameterization used in the implementation.","section":"Section 3.2, Eqs. (9)-(10)"},{"comment":"The walk-forward procedure is underspecified. Please clarify how the training/validation split evolves with the expanding window, how the 33% validation fraction is applied, and how the sequence length of 10 interacts with the minimum window length. Also state whether early stopping is applied per window and how the optimizer and learning-rate schedules are handled across retraining.","section":"Section 4, Table 1"},{"comment":"The repeated PIT value 2.41e-07 for several model-index pairs is suspicious; even if the test is the same, exact equality across models with different forecast distributions is unlikely. Please report the test statistic and full precision.","section":"Table 2"},{"comment":"The note says 'Bolded = closest to theoretical tolerance, underlined = correct Kupiec test, * = correct Christoffersen test,' but the table as typeset does not show bold or underline, and the asterisk notation is ambiguous. Please reformat the table so the notation is legible and consistent.","section":"Table 3"},{"comment":"Table 4 uses commas as decimal separators (e.g., 1,263296), inconsistent with Tables 2, 3, and 5. Also, 'Chrisotff.' is a typo for 'Christoffersen.' Please standardize notation.","section":"Table 4"},{"comment":"The caption and text describe Figure 2 as showing DAX PIT histograms and claim the LSTM-SSTD is closest to uniform, but no p-values or test statistics are reported for DAX. Please add quantitative calibration results for all indices shown.","section":"Section 5.1, Figure 2"},{"comment":"The text says 'we apply the Fernandez and Steel [23], [5] transformation'; this is confusing because [5] and [23] are listed as different references. Please correct the citation to avoid ambiguity.","section":"Section 3.3, References [5],[23]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core empirical contribution is potentially useful, but the gap between the abstract's claims and the reported calibration statistics is wide. The PIT results in Table 2 are the paper's own evidence and they contradict the 'accurate distributional forecasts' claim. The viability of the paper will depend on whether the authors can either present a calibration analysis that shows the deviations are practically negligible or substantially weaken the claims to focus on VaR exceedance performance. The lack of statistical testing for model comparisons and the incomplete GARCH baselines are also fixable in revision. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an incremental but competently executed study. It applies well-known distributional forecasting techniques (NLL-trained CNNs/LSTMs with Normal, Student's t, and skewed Student's t) to six equity indices, evaluates with LPS/CRPS/PIT, and compares VaR exceedance rates against GARCH variants. The code is on GitHub, and the walk-forward setup is sensible. For VaR, the results are plausibly competitive with GARCH, and the paper does a decent job of showing that neural networks are a usable tool rather than a breakthrough.\n\nThe soft spots are real and mostly concentrated in the calibration claims. The stress-test note holds up: Table 2 reports PIT p-values for only three indices, and across those, 17 of 18 model--distribution combinations reject uniformity at 5%. The headline LSTM-SSTD model, which the paper champions, has p-values of 0.031 (S&P), 2.4e-07 (Nikkei), and 5.1e-06 (KOSPI). Only CNN-SSTD for S&P (and marginally Nikkei) is well calibrated. DAX, WIG, and BOVESPA are absent from the distributional evaluation entirely. So the abstract's first sentence—'accurate distributional forecasts'—is not supported by the reported evidence. The VaR claim is more defensible, but the paper should either weaken the distributional claim or provide a calibration analysis that demonstrates the deviations are practically negligible.\n\nA few smaller issues: there are no statistical tests for LPS/CRPS differences between models, and the 'best model' is selected on the test set without multiple-comparison corrections. The GARCH comparison is also thin—no estimation details or distributional metrics for the baselines, just VaR exceedance rates. These are fixable in revision. I would not call any of this fatal; the VaR finding is useful and the experiments are reproducible, but the current framing oversells what the data show.\n\nWho should read it: practitioners wanting a sanity check that NNs can sit next to GARCH for VaR, and referees who want to push the authors on calibration and statistical rigor. It deserves peer review, because the empirical setup is sound enough and the question is relevant. I would not cite it in my own work, but I might put it on a reading group list as a cautionary example of claims versus evidence.","headline":"A competent, reproducible application of known probabilistic forecasting methods to equity indices, but the calibration evidence in its own Table 2 contradicts the abstract's claim of 'accurate distributional forecasts.'","tokens_in":9396,"tokens_out":1433,"would_cite":false,"duration_ms":15322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep learning matches GARCH for value-at-risk on six major equity indexes, with LSTM and skewed Student-t leading.","keywords":["probabilistic forecasting","deep learning","LSTM","1D CNN","skewed Student's t","Value-at-Risk","GARCH comparison","distributional calibration"],"falsifier":"Re-run the walk-forward experiment on one index, say the S&P 500, with the same expanding-window protocol, fit a standard GARCH-t by maximum likelihood, and compare its 5% and 1% VaR exceedance counts and Kupiec test p-values with the values reported in Table 5; if a correctly implemented GARCH baseline deviates materially from the reported 'best GARCH' exceedance rates, the paper's headline competitiveness claim would not hold.","tokens_in":8563,"feed_emoji":"📈","tokens_out":5166,"duration_ms":50182,"temperature":0.7,"pith_summary":"This paper claims that deep neural networks can forecast the full probability distribution of daily financial returns, not just point predictions. Using custom negative log-likelihood losses, 1D CNN and LSTM models directly output the parameters of Normal, Student's t, and skewed Student's t distributions. Tested on six global equity indices over more than two decades, the models produce accurate distributional forecasts and VaR exceedance rates close to theoretical levels, competitive with classical GARCH specifications. The LSTM with a skewed Student's t distribution performs best overall, suggesting that combining sequential memory with heavy-tailed, asymmetric distributions is a viable route for risk management.","feed_headline":"Deep learning matches GARCH for value-at-risk on six indexes","feed_subtitle":"An LSTM forecasting skewed Student-t return distributions gives accurate, well-calibrated risk measures.","key_machinery":"The engine is a conditional density network: an LSTM or 1D CNN reads the last 10 daily returns and outputs the parameters—location, scale, degrees of freedom, and skewness—of a predictive distribution. Training minimizes the negative log-likelihood of the observed return under that distribution. For the skewed Student's t case, a piecewise skewing transformation turns a symmetric t density into an asymmetric one, so the network learns heavy tails and asymmetry directly from data. This parameter-output design lets VaR and Expected Shortfall be read off analytically from the predicted quantiles.","core_discovery":"The central claim is that a neural network trained to minimize the negative log-likelihood of a parametric return distribution can serve as a practical distributional forecaster for financial risk. On daily returns of the S&P 500, BOVESPA, DAX, WIG, Nikkei 225, and KOSPI, the best configuration—an LSTM outputting skewed Student's t parameters—achieves the lowest Log Predictive Score and CRPS, better PIT calibration than Normal or symmetric t variants, and 5%/1% VaR exceedance rates (e.g., 4.50%/0.84% on the S&P 500) that satisfy Kupiec and Christoffersen tests. Against GARCH benchmarks, the best neural network exceedance rates are comparable or closer to the nominal rate, supporting the pape","pith_inferences":["The paper does not separate calm from crisis periods in its evaluation; testing the same LSTM-SSTD recipe on stress subsets (e.g., 2008, 2020) would reveal whether the learned tail asymmetry is stable or regime-dependent.","Since LSTM consistently edges out CNN on daily data, sequence memory seems to matter more than local feature extraction at this frequency; intraday data, where local patterns dominate, is a natural test of whether CNN closes the gap.","A direct ablation varying only the distribution family while holding architecture fixed would isolate how much of the calibration gain comes from the skewed Student-t shape rather than the network structure."],"forward_implications":["Risk managers can use LSTM-SSTD predictive distributions to compute VaR and Expected Shortfall directly from the forecasted parameters, without separate volatility modeling.","The skewed Student's t specification improves calibration over Normal and symmetric t, so the extra tail and asymmetry parameters carry real forecasting information rather than overfitting noise.","Neural distributional forecasts are competitive enough to serve as a benchmark or replacement for GARCH-based VaR systems across diverse equity markets.","The CNN's near-competitive performance with lower computational cost points to a practical trade-off for high-frequency risk applications."],"supporting_citations":[{"why":"Supplies the proper scoring rule framework and the CRPS decomposition used to evaluate distributional forecast accuracy.","marker":"[7]"},{"why":"Provides the Kupiec test for whether observed VaR exceedance rates match the nominal level.","marker":"[10]"},{"why":"Provides the Christoffersen conditional coverage test for VaR forecast independence and correct coverage.","marker":"[3]"},{"why":"Supplies the Expected Shortfall backtest used to validate tail forecasts beyond VaR.","marker":"[12]"},{"why":"Underpins the skewing transformation used to build the skewed Student's t likelihood.","marker":"[5]"},{"why":"Defines the GARCH model variants used as classical econometric benchmarks.","marker":"[20]"},{"why":"Provides the precedent of distributional neural networks for electricity price forecasting that motivates the approach.","marker":"[11]"}],"fun_headline_variants":["Skewed-t LSTM beats GARCH on six indexes","LSTM with skewed-t tops GARCH for VaR","Deep net rivals GARCH using skewed t","LSTM edges GARCH on six market risk forecasts","Neural net with skewed t matches GARCH"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported competitiveness with GARCH depends on the GARCH baselines being correctly specified, estimated on the same expanding windows, and directly comparable to the neural network exceedance rates, but the paper gives no estimation details or diagnostics for those baselines.","fun_headline_variants_meta":{"raw":{"variants":["Skewed-t LSTM beats GARCH on six indexes","LSTM with skewed-t tops GARCH for VaR","Deep net rivals GARCH using skewed t","LSTM edges GARCH on six market risk forecasts","Neural net with skewed t matches GARCH"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1333,"prompt_tokens":737,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":481,"tokens_out":596,"duration_ms":6918,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:04:52.347709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the walk-forward experiment on one index, say the S&P 500, with the same expanding-window protocol, fit a standard GARCH-t by maximum likelihood, and compare its 5% and 1% VaR exceedance counts and Kupiec test p-values with the values reported in Table 5; if a correctly implemented GARCH baseline deviates materially from the reported 'best GARCH' exceedance rates, the paper's headline competitiveness claim would not hold.","supporting_citations":[{"cited_title":"and Raftery, A","cited_arxiv_id":null,"evidence_quote":"Supplies the proper scoring rule framework and the CRPS decomposition used to evaluate distributional forecast accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Kupiec test for whether observed VaR exceedance rates match the nominal level."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Christoffersen conditional coverage test for VaR forecast independence and correct coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Expected Shortfall backtest used to validate tail forecasts beyond VaR."},{"cited_title":"and Steel, M","cited_arxiv_id":null,"evidence_quote":"Underpins the skewing transformation used to build the skewed Student's t likelihood."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GARCH model variants used as classical econometric benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the precedent of distributional neural networks for electricity price forecasting that motivates the approach."}],"review_version":1}