{"id":"a7caa16f-38aa-46d7-84e2-80dc6c70dfed","arxiv_id":"2501.01278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LSTM-MDNs give adequate but not superior Value-at-Risk forecasts, performing better in high-volatility periods while still lagging GARCH on volatility reactivity.","lead":"This finance study compares Long Short-Term Memory mixture density networks (LSTM-MDNs) with three standard Value-at-Risk models on three stock indices. It finds the neural networks sometimes beat simpler models during turbulent markets, but are not reliably better and depend heavily on large datasets and careful initialization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'not superior' conclusion rests on an asymmetric comparison: benchmarks are re-estimated daily on rolling 250-day windows, while LSTM-MDNs are trained once and never refit; periodic refitting could change the conclusion.","rationale":"The reader's weakest assumption identifies the same asymmetry, and I agree that it is the main threat to the central claim. The paper's own Section 5.3 acknowledges that refitting is the natural remedy, which indicates the authors view the fixed-training protocol as a limiting choice. The strongest claim is comparative ('not superior'), so the evaluation protocol must not systematically handicap one family: a model that is never refit over two years is not given the same opportunity as a model whose parameters are re-estimated every day. This is not a dispute with the backtesting math; it is a question of whether the empirical comparison supports the general comparative conclusion. The proposed test would settle it by making the updating scheme symmetric. The seed-selection under-specification and missing uncertainty quantification are secondary; they could be fixed in a revision but are not the main threat. The reader's conditional verdict is appropriate, and I do not move it.","tokens_in":37299,"tokens_out":5860,"duration_ms":58954,"concrete_test":"Implement a rolling refit version of NNet1-3 exactly as proposed in Section 5.3: start with the same pre-evaluation training set, then every 20 trading days retrain on all available data up to that day (expanding window, same hyperparameters and seed protocol), and produce daily one-day-ahead VaR forecasts for the six index/period evaluations. Rerun the Kupiec and Christoffersen tests. The concern lands if, for example, the EURO STOXX 50 calm-period overshoots move from 0% toward the nominal 1% with UC p-values above 5%, or if the S&P 500 calm-period NNet1 independence violation disappears; more generally, if the number of NN joint-test passes increases materially relative to Tables 4.1-4.6. If periodic refitting leaves the calm-period results basically unchanged, the 'not superior' claim survives this criticism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim ('LSTM-MDNs are adequate but not superior') is evaluated under a protocol that is asymmetric in a way that disadvantages the neural networks. In Section 3.2, all benchmark forecasts (HS, CMM, GARCH) are produced by re-estimating parameters daily on a rolling 250-day window, so every forecast uses only the most recent 250 returns. In contrast, Section 3.4 and the model description in Section 3.3.2 show that each LSTM-MDN is trained once on a fixed pre-evaluation set and then applied unchanged for the entire two-year evaluation window (18 models total, one per index/period/architecture). The measured performance difference therefore conflates model architecture with updating policy. The calm-period failures (e.g., EURO STOXX 50 2017-2018, Table 4.3) could be driven by the network being unable to adapt to the recent regime, not by an inherent deficiency of LSTM-MDNs. The turbulent-period successes could reflect the training set already containing high-volatility episodes. The paper itself identifies this limitation in Section 5.3: 'Mitigating the data requirements could be achieved with regularly refitting the neural networks on all data which is available at time point t.' Since the central claim is about capability relative to established models, a static-training comparison does not settle it. If periodic refitting improves calm-period coverage and pass rates, the correct conclusion would be that LSTM-MDNs are not superior under a deliberately static implementation, not that they lack competitive capability. Secondary issues (best-of-three seed selection under-specified in Section 3.3.2, missing uncertainty intervals on Table 5.2 correlations) reinforce the need for caution but are less decisive than the updating asymmetry.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript implements three Long Short-Term Memory Mixture Density Network (LSTM-MDN) variants for one-day-ahead 99% Value-at-Risk forecasting: a two-component network, a two-component network with an L2-regularized loss, and a three-component network, the last being described as new in this context. These are compared with historical simulation, the constant mean model, and GARCH(1,1) on three stock indices (FTSE 100, S&P 500, EURO STOXX 50) over two two-year evaluation windows, one calm (2017-2018) and one turbulent (2021-2022). Forecast adequacy is assessed with Kupiec's unconditional coverage test, Christoffersen's independence test, and the joint conditional coverage test, and volatility-clustering behaviour is examined through correlations between VaR forecasts and rolling realized volatility. The paper concludes that LSTM-MDNs produce adequate risk forecasts in some settings, can capture volatility clustering in a manner comparable to GARCH, but are not clearly superior to established benchmarks and are hampered by training-data requirements and sensitivity to weight initialization.","tokens_in":37621,"tokens_out":5957,"duration_ms":61208,"significance":"If the central comparison were conducted under a fully symmetric and reproducible protocol, the study would provide a useful independent benchmark for a recent deep-learning approach in a standard risk-management evaluation framework. The manuscript has several strengths: it uses a conventional train/validation/test split, applies well-credited backtesting tests, evaluates three architectures including a novel three-component MDN, and makes its code and data available through a repository. The correlation analysis with rolling volatility, while indirect, is a concrete attempt to measure a mechanism often discussed only qualitatively. However, the significance of the headline conclusion is substantially reduced by an asymmetric estimation protocol and by incomplete reporting of the seed-selection procedure; these issues need to be resolved before the comparative claims can be accepted.","major_comments":[{"comment":"The comparison of forecast accuracy is asymmetric. The benchmark models (HS, CMM, GARCH) are re-estimated daily on a rolling 250-day window (Section 3.2), whereas each LSTM-MDN is trained once on a fixed pre-evaluation set and is then used unchanged for the entire two-year evaluation period (Sections 3.3.2 and 3.4, with training-set sizes in Table 5.1). The reported differences therefore conflate model architecture with updating policy, so the abstract's claim that LSTM-MDNs are 'adequate but not superior' is not fully supported by the evidence. The manuscript itself acknowledges this in Section 5.3 ('Mitigating the data requirements could be achieved with regularly refitting the neural networks on all data which is available at time point t'), but leaves this as future work. A fair comparison requires either a refitting schedule for the neural networks or a fixed-window benchmark; without this, the central comparative conclusion remains open.","section":"Section 3.2 vs. Sections 3.3.2 and 3.4; Table 5.1"},{"comment":"The best-of-three initialization procedure is not described precisely enough to support the conclusions drawn about initialization sensitivity. It is not stated whether the 'seed providing the best results on average' is selected on the validation set or on the evaluation period, and no results for the non-selected seeds or any measure of seed-to-seed variability are reported. Given that Section 5.1.3 identifies weight initialization as a key driver of model performance and unreliability, the reader cannot tell whether the presented backtests reflect a typical model or an optimistic selection. Please specify the selection metric (e.g., lowest validation loss) and report sensitivity to the three seeds for at least one index-period pair.","section":"Section 3.3.2, 'Weight initialization'; Section 5.1.3"},{"comment":"The conclusion that LSTM-MDNs capture volatility clustering 'in a comparable way as the GARCH models' is stronger than the evidence in Table 5.2 appears to justify. The reported Pearson correlations between rolling volatility (d=5) and VaR forecasts are generally lower for the neural networks than for GARCH, and some entries are negative (S&P 500 NNet1 turbulent period: -0.240; EURO STOXX 50 NNet2 calm period: -0.245), indicating a failure to track volatility in those cases. Simple correlation with rolling realized volatility is also not a direct test of clustering dynamics; it does not establish that the VaR forecasts react to volatility in a manner equivalent to GARCH. A more direct comparison of conditional-volatility responses, or a statistical test of the difference in correlations, is needed before the volatility-clustering claim can be accepted.","section":"Section 5.1.2, Table 5.2"}],"minor_comments":[{"comment":"The Kupiec POF statistic as printed contains a typographical error: the term (α̂/1)^{I(α)} should be (α̂/(1-α))^{I(α)}, since the expected violation proportion under the null is 1-α. The p-values in Tables 4.1-4.6 appear to be computed from the correct formula, so this is a presentation issue, but it should be corrected to avoid misleading readers.","section":"Equation (2.20)"},{"comment":"The text states that the networks contain 6 nodes in the LSTM layer and 10 nodes in the Dense layer, but the architecture boxes for NNet 1, NNet 2, and NNet 3 all list 'Dense-Layer [12 nodes]'. These numbers should be reconciled.","section":"Section 3.3.2"},{"comment":"The forecast plots do not label the benchmark-model VaR curves; adding a legend identifying V aR_HS, V aR_CMM, and V aR_GARCH alongside the NNet curves would make the visual comparisons in Section 5.1.1 much easier to verify.","section":"Chapter 4, Figures 4.1-4.6"},{"comment":"The evaluation runs six index-period combinations and several tests per model at a 5% significance level, but no multiple-testing correction is discussed. If the six evaluations are treated as independent replications this is acceptable, but the manuscript should state this explicitly rather than implicitly treating each individual p-value as decisive.","section":"Section 2.4 and Chapter 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a Master's thesis that has been posted to arXiv without evidence of external peer review beyond the university supervisors. The incremental novelty is modest: the only new architectural element is the three-component MDN, and the evaluation is small in scale. However, the paper is clearly written and the code appears to be shared, so with a corrected evaluation protocol and more careful claims it could serve as a useful reference point for practitioners considering LSTM-MDNs for VaR. The main risk is that the headline 'not superior' conclusion is, as it stands, an artifact of comparing daily-refitted benchmarks against once-trained neural networks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the punchline: this paper is a careful, honest empirical comparison that reproduces what the prior literature already found — LSTM-MDNs are adequate but not superior to GARCH for 99% daily VaR. Its only genuinely new element, a three-component mixture, does not change that picture. If you're looking for a decisive new result, this isn't it. If you want a clean, reproducible data point on how static neural-network VaR models behave in calm versus turbulent periods, this is a solid one.\n\nWhat it does well: the backtesting design is standard and correctly implemented — Kupiec POF, Christoffersen independence, joint conditional coverage — on three indices over two clearly separated two-year windows. The authors are unusually candid about limitations: they explicitly note the need for regular refitting (Section 5.3), the reliance on large training sets, and instability under different weight initializations. The code is available on GitHub, which matters. And the paper scopes its claim to \"as it is used in this thesis,\" which is exactly the right scope.\n\nThe soft spot the reader flagged — benchmarks re-estimated daily on rolling 250-day windows while networks are trained once — is real but not damning. Because the conclusion is already that LSTM-MDNs show limited promise, the asymmetry works against the networks. A refitted version might do better, and the paper says so itself. So the conclusion is conservative, not inflated. The bigger reproducibility issue is the \"best-of-three\" seed selection: the paper lists seeds but doesn't define what \"best\" means, and choosing the best on average over the evaluation period could introduce selection bias. That needs specification. The missing uncertainty intervals on the correlation table are minor.\n\nThe 3-component variant is really a parameter scan over K, and the paper honestly reports no striking difference. So novelty is thin. But the work is transparent, the math and data are solid, and the limitations are clearly stated. I'd send this out for peer review; the authors will likely be asked to address the updating asymmetry and seed-selection details, but those are fixable, and the empirical record is worth having on the table.","headline":"A clean, honest empirical comparison that confirms the prior result that LSTM-MDNs don't beat GARCH for 99% daily VaR, with the 3-component variant adding little; the updateding asymmetry is real but the conclusion is conservative, not inflated.","tokens_in":38200,"tokens_out":2901,"would_cite":false,"duration_ms":31080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62M10","62P05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper tests whether LSTM-MDNs can compete with established Value-at-Risk models, finding them viable but not superior and dependent on large training sets and weight initialization.","keywords":["Long Short-Term Memory","Mixture Density Network","Value-at-Risk","backtesting","volatility clustering","GARCH","neural network risk forecasting","tail risk"],"falsifier":"Re-run the six evaluations with each LSTM-MDN refit every 20 trading days on all data available up to that day; if the neural networks then pass the conditional coverage test in most calm-period cases and still match GARCH in turbulent periods, the paper's conclusion that LSTM-MDNs offer only modest potential for practical use would be overturned.","tokens_in":37088,"feed_emoji":"📉","tokens_out":7967,"duration_ms":70342,"temperature":0.7,"pith_summary":"The paper asks whether Long Short-Term Memory mixture density networks (LSTM-MDNs) can produce Value-at-Risk forecasts that compete with established models, and whether they can capture volatility clustering in daily returns. Using three stock indices and two two-year evaluation windows, it backtests three LSTM-MDN designs against historical simulation, the constant mean model, and GARCH at the 99% daily confidence level. The results show a mixed picture: the networks underperform in the calm 2017/2018 period, outperform the benchmarks in the turbulent 2021/2022 period, and track volatility clustering about as well as GARCH, though with weaker correlation. The paper concludes that LSTM-MDNs give adequate risk forecasts but are not practically ready without better initialization and larger training sets.","feed_headline":"LSTM-MDN risk forecasts beat benchmarks only in turbulent markets","feed_subtitle":"Backtests on three indices show they capture volatility clustering but need refitting plus more data for real use.","key_machinery":"The central object is the LSTM mixture density network: an LSTM hidden layer that reads a sequence of daily returns and feeds a mixture-density output layer, which returns the component weights, means, and standard deviations $(\\pi_k,\\mu_k,\\sigma_k)$ of a $K$-component Gaussian mixture for the next day's return. Two-component and three-component variants are trained by minimizing the negative log-likelihood of the mixture, with one variant adding an L2 penalty on the $\\pi$'s. One-day-ahead VaR is then the empirical $\\alpha$-quantile of 100,000 Monte Carlo losses drawn from the predicted mixture. The recurrent hidden state is the mechanism expected to carry information about past volatility, which is how the network accounts for volatility clustering.","core_discovery":"On the paper's own terms, the central claim is that LSTM-MDNs are a workable but not superior alternative to GARCH for daily 99% Value-at-Risk, with performance strongly shaped by market regime, training-set size, and random initialization. Across six backtests, the neural networks pass the conditional coverage tests in only 4 of 9 calm-period cases but in 8 of 9 turbulent-period cases, while GARCH remains the most consistently reliable single model. The LSTM mechanism does give the networks a real ability to react to short-term volatility shifts, mainly through time-varying location and variance parameters rather than through the mixture weights, and the paper is the first to test a three-component mixture in this setting. The paper also identifies two blockers to practical use: forecasts depend heavily on which random seed initializes the weights, and performance improves when the training set is large.","pith_inferences":["A fairer comparison would refit the neural networks periodically on an expanding window, as the benchmarks are refitted daily; doing so would likely change both the calm-period and turbulent-period results.","A testable extension is to refit every 20 trading days and re-run the six backtests; if the calm-period failures disappear, the paper's 'modest potential' verdict would be a property of static training rather than of the architecture.","Because volatility clustering is carried by $\\mu$ and $\\sigma$ rather than by $\\pi$, a simpler recurrent model emitting a time-varying variance might capture much of the same benefit at lower cost.","The best-of-three seed selection used in training likely paints an optimistic picture; reporting the full distribution of outcomes across seeds would show how much of the result is luck of the draw."],"forward_implications":["LSTM-MDNs do not displace GARCH for daily 99% VaR on the tested indices; a simpler GARCH(1,1) with GED innovations is at least as accurate and much cheaper to run.","The networks' strongest results appear in high-volatility periods with larger training sets, so any practical deployment should be conditioned on market regime and data availability.","The three-component mixture is a viable addition to VaR forecasting and in some evaluations reacts to volatility shifts with the largest amplitude of the three architectures.","The regularized two-component loss does not deliver the clear accuracy gain reported in earlier work, so its benefit is not confirmed here.","Weight initialization is a first-order practical issue: different initial seeds can produce materially different risk forecasts."],"supporting_citations":[{"why":"It supplies the LSTM-MDN approach for VaR forecasting, the regularized loss term, and the emphasis on large training sets that the paper adapts and tests.","marker":"[Arimond et al., 2020]"},{"why":"It is the prior two-component LSTM-MDN comparison with mean-variance and historical simulation that the paper builds on, including findings of conservative forecasts and the ReLU-activation explanation.","marker":"[Karlsson Lille and Saphir, 2021]"},{"why":"It provides a GARCH-LSTM hybrid benchmark and evidence on neural-network superiority that motivates the paper's comparison with GARCH.","marker":"[Buczynski and Chlebus, 2023]"},{"why":"It compares LSTM networks with GARCH configurations and highlights reactivity to volatility shifts, the property the paper tests by correlation analysis.","marker":"[Ormaniec et al., 2022]"},{"why":"It introduces mixture density networks, the output architecture that returns parameters of a conditional mixture distribution.","marker":"[Bishop, 1994]"},{"why":"It defines the GARCH model used as the principal benchmark and as the reference for volatility-clustering behavior.","marker":"[Bollerslev, 1986]"},{"why":"It supplies the proportion-of-failures test used to evaluate unconditional coverage of the VaR forecasts.","marker":"[Kupiec, 1995]"},{"why":"It supplies the independence and conditional-coverage tests used in the backtesting procedure.","marker":"[Christoffersen, 1998]"},{"why":"It introduces the LSTM architecture whose hidden state the paper relies on to capture temporal dependence in returns.","marker":"[Schmidhuber et al., 1997]"}],"fun_headline_variants":["LSTM-MDN VaR beats GARCH only during turbulent periods","Neural VaR model edges out benchmarks only in market stress","LSTM-MDNs capture volatility clustering but need big data","Turbulent markets: LSTM-MDN forecasts outperform benchmarks","LSTM-MDN risk forecasting: regime-dependent, needs refits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats one fixed training of each LSTM-MDN as comparable to benchmark models that are re-estimated every day on a rolling 250-day window, so the reported underperformance in calm periods could stem from this unequal updating rather than from the network itself.","fun_headline_variants_meta":{"raw":{"variants":["LSTM-MDN VaR beats GARCH only during turbulent periods","Neural VaR model edges out benchmarks only in market stress","LSTM-MDNs capture volatility clustering but need big data","Turbulent markets: LSTM-MDN forecasts outperform benchmarks","LSTM-MDN risk forecasting: regime-dependent, needs refits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000983,"raw_usage":{"total_tokens":4198,"prompt_tokens":998,"completion_tokens":3200,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":3112}},"tokens_in":614,"tokens_out":3200,"duration_ms":22104,"temperature":1.0,"reasoning_tokens":3112,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:30:49.435926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six evaluations with each LSTM-MDN refit every 20 trading days on all data available up to that day; if the neural networks then pass the conditional coverage test in most calm-period cases and still match GARCH in turbulent periods, the paper's conclusion that LSTM-MDNs offer only modest potential for practical use would be overturned.","supporting_citations":[{"cited_title":"Neural Networks and Value at Risk","cited_arxiv_id":"2005.01686","evidence_quote":"It supplies the LSTM-MDN approach for VaR forecasting, the regularized loss term, and the emphasis on large training sets that the paper adapts and tests."},{"cited_title":"and Saphir, D","cited_arxiv_id":null,"evidence_quote":"It is the prior two-component LSTM-MDN comparison with mean-variance and historical simulation that the paper builds on, including findings of conservative forecasts and the ReLU-activation explanation."},{"cited_title":"and Chlebus, M","cited_arxiv_id":null,"evidence_quote":"It provides a GARCH-LSTM hybrid benchmark and evidence on neural-network superiority that motivates the paper's comparison with GARCH."},{"cited_title":"Estimating value at risk: LSTM vs. GARCH","cited_arxiv_id":"2207.10539","evidence_quote":"It compares LSTM networks with GARCH configurations and highlights reactivity to volatility shifts, the property the paper tests by correlation analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the proportion-of-failures test used to evaluate unconditional coverage of the VaR forecasts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the independence and conditional-coverage tests used in the backtesting procedure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduces the LSTM architecture whose hidden state the paper relies on to capture temporal dependence in returns."}],"review_version":1}