{"id":"de699376-59c5-47e4-a95c-69398893fc6f","arxiv_id":"2411.15674","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying the quantile check loss to BD-LSTM, ED-LSTM, and Conv-LSTM gives multi-step forecasts with 5th-95th percentile bands at similar RMSE to the standard models on crypto and benchmark data.","lead":"Quantile regression, which predicts low and high percentiles instead of just a single value, is added to three deep learning forecasters for Bitcoin, Ethereum, and benchmark time series. The quantile versions produce 5th-95th percentile bands at about the same error as the plain versions, though the evaluation uses a random train-test split that is questionable for time series.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random 80:20 split in §3.5 leaks future information into training, so the reported accuracy parity and the comparison with Wu et al. [44] do not yet support the central claim; a temporal re-split test would settle it.","rationale":"The paper's central claim is that adding quantile loss to standard deep forecasters delivers valid quantile bands at no cost in point accuracy and with better volatility handling. All empirical support for that claim sits on the cryptocurrency experiments and the comparison with Wu et al. [44]. The random 80:20 split in Section 3.5 is not a minor detail: with overlapping sliding windows, a random split mixes periods, so some training samples cover test-period observations and the reported RMSEs are optimistic in an uncontrolled way. This is exactly the assumption the reader flagged, and I agree with that identification. The paper is transparent about the split and about the absence of quantile accuracy measures, but transparency does not make the evaluation valid for the stated forecasting task. The release of code and data is genuine independent support, and the relative claim (quantile versions roughly match standard versions) is plausible, so a temporal re-split is the right next step. My concern does not move the reader's verdict: conditional acceptance, pending a temporal-split re-analysis and proper quantile scoring, remains appropriate.","tokens_in":23014,"tokens_out":5905,"duration_ms":52303,"concrete_test":"Using the provided GitHub code, re-run the Bitcoin and Ethereum experiments with a strict temporal split: train on the first 80% of each date-ordered series and test on the final 20%, keeping architectures, hyperparameters, window sizes, and the 30-seed protocol fixed. Report median RMSE with 95% CI and, for the quantile models, empirical interval coverage and pinball loss at the five quantiles. If quantile-versus-standard parity and the comparison with Wu et al.'s reported 0.0373 survive, the central claim is supported; if RMSEs degrade or parity breaks, the random split is responsible for the headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 ('we created the training dataset using a split that was randomly selected, i.e. 80:20 ratio') is the load-bearing weakness. The cryptocurrency data are converted into overlapping sliding windows (input length d=6, output length m=5); a random split of these windows mixes periods, so training windows can contain values from the test interval and test-period observations can appear as training targets. Reported RMSEs in Tables 2–5 therefore measure interpolation over a shuffled pool, not true multi-step-ahead out-of-sample forecasting. The leakage also inflates the apparent volatility robustness: the authors deliberately randomize so that COVID-era volatility appears in both training and test, which is why the model appears to 'handle' it. Section 5 compares Quantile-ED-LSTM's Bitcoin multivariate RMSE of 0.0112 with Wu et al.'s 0.0373, attributing the gap only to the 80:20 vs 70:30 ratio; that comparison is uncontrolled because the split mechanics differ, and the random split invalidates the baseline comparison. A related gap is conceded in the Discussion ('we had no indication on how accurate our quantile predictions are'): no calibration, coverage, or pinball scoring is reported, so the 'handles volatility more effectively' claim is unsupported even apart from leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces quantile-regression variants of BD-LSTM, ED-LSTM, and Conv-LSTM for multi-step ahead time series prediction, replacing the standard loss with the quantile check loss and reporting RMSE at five quantiles (0.05, 0.25, 0.5, 0.75, 0.95). The models are evaluated on Bitcoin, Ethereum, Sunspot, Mackey-Glass, and Lorenz datasets, with univariate and multivariate input strategies. The central claims are that integrating a quantile loss function with deep learning provides additional quantile predictions without a loss in point-forecast accuracy relative to conventional models, and that the quantile models handle volatility more effectively.","tokens_in":23268,"tokens_out":6622,"duration_ms":53746,"significance":"If fully supported, the paper would provide a practical recipe for adding quantile outputs to standard deep forecasting models at negligible cost, with useful open-source code, reproducible 30-run experiments, and direct comparisons across several architectures. The accuracy-parity result is encouraging and the paper is clearly written in terms of model variants and loss functions. However, the current evaluation protocol measures interpolation on a randomly shuffled window pool rather than genuine multi-step ahead forecasting, and the absence of any probabilistic calibration metric leaves the central uncertainty-quantification claim undemonstrated. The paper also contains apparent data inconsistencies in the Ethereum tables that must be resolved before the empirical conclusions can be accepted.","major_comments":[{"comment":"The random 80:20 train-test split described in Section 3.5 leaks future information into the training set. Because the data are first converted into overlapping windows with input length d=6 and output length m=5 (Section 3.3), a random split of these windows places post-test observations in training and pre-test observations in testing. The reported RMSE values in Tables 2, 4, and 6 are therefore interpolation errors on a shuffled window pool, not errors of genuine multi-step-ahead forecasting. This also explains, rather than validates, the claimed volatility robustness: Section 3.5 explicitly randomizes so that COVID-era volatility appears in both training and test sets. The accuracy-parity claim and the volatility-handling claim should be re-evaluated with a temporal split (for example, the last 20% of the series as test) and with volatility-period-specific metrics.","section":"Section 3.5"},{"comment":"The comparison with Wu et al. [44] is not controlled. The paper contrasts the multivariate Quantile-ED-LSTM Bitcoin RMSE of 0.0112 with the 0.0373 reported by Wu et al. and attributes the difference to the 80:20 versus 70:30 split ratio. However, the two studies also differ in split mechanics (random window split versus the procedure used in [44]), in test-set composition, and potentially in preprocessing and hyperparameters, even though Section 3.3 relies on [44] for hyperparameters. The 0.0112 figure is itself produced under the leaking random split of the previous comment. This comparison should be removed or replaced with a head-to-head evaluation on identical temporal splits and preprocessing.","section":"Section 5"},{"comment":"The Ethereum results contain apparent data errors. The multivariate ED-LSTM row in Table 4 (mean 0.0113; steps 0.0101, 0.0112, 0.0118, 0.0119, 0.0117) is identical to the multivariate ED-LSTM row for Bitcoin in Table 2, which is implausible for a different dataset. In addition, Table 5 reports a 0.5-quantile RMSE of 0.0137 for multivariate Quantile-ED-LSTM on Ethereum, whereas Table 4 reports a mean RMSE of 0.0126 for the same model and strategy; these two numbers should coincide because Section 3.2 states the median quantile is taken as the point prediction. The Ethereum-based conclusions in Section 4.1 and the rankings in Table 7 should be rechecked after correcting these entries.","section":"Tables 4 and 5"},{"comment":"The authors concede that 'We had no indication on how accurate our quantile predictions are.' Indeed, the paper never reports calibration, empirical coverage, pinball score, or quantile-crossing diagnostics for the estimated 5%-95% intervals; Tables 3 and 5 only report RMSE per quantile, which is not a proper scoring rule for distributional forecasts. Consequently, the Abstract's claim that the quantile model 'has the ability to handle volatility more effectively' is unsupported on its own terms, independent of the data-splitting issue. The paper should add proper probabilistic forecast evaluation (e.g., empirical coverage of the 5%-95% interval, interval score or pinball loss, and a high-volatility-period subsample analysis).","section":"Section 5 (Discussion)"}],"minor_comments":[{"comment":"Equation (2) appears to contain a typesetting error: the indicator function is written as '1u<0' without an explicit I(·), and the parentheses in ρτ(u) = u(τ− 1u<0)) are unbalanced.","section":"Section 2.1, Eq. (2)"},{"comment":"The quantile notation is inconsistent: Section 2.1 uses τ, while Eq. (3) in Section 3.2 uses q; please unify the symbols.","section":"Sections 2.1 and 3.2"},{"comment":"The sentence 'we use 64 filters with a kernel size of 2' appears twice in the same paragraph; please remove the duplicate.","section":"Section 3.5"},{"comment":"The caption contains the typo 'not explicitly suing in the recurrent neural network' instead of 'not explicitly shown in the recurrent neural network.'","section":"Figure 3 caption"},{"comment":"Table 6 reports only steps 2, 5, 8, and 10 despite the text describing 10-step horizons; please clarify whether these are representative steps and why intermediate steps are omitted.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful central idea and reproducible code, but the random-split protocol and the missing calibration evaluation are load-bearing for the stated claims. The apparent duplication of the Bitcoin ED-LSTM row in the Ethereum table is a serious data-integrity issue that the authors must resolve. I would not recommend rejection outright, as the framework itself is sound and the technical issues are fixable with a temporal re-split, corrected tables, and proper probabilistic evaluation metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one thing well: it gives a clean, reproducible implementation of quantile loss on three standard deep learning forecasters (BD-LSTM, ED-LSTM, Conv-LSTM), runs each 30 times, reports confidence intervals, and shows that the quantile versions roughly match the point accuracy of the classic versions. The code and data are public, and the internal comparison is consistent. That part earns credit.\n\nThe problem is the evaluation. Section 3.5 says the train/test split was randomly selected at 80:20, deliberately mixing periods so that volatile episodes like the COVID crash land in both training and test. For time series with overlapping sliding windows, this is textbook leakage: training windows can contain values from the test interval, and test windows can include training observations as targets. The reported RMSEs in Tables 2-5 measure how well the model interpolates a shuffled pool, not how it forecasts unseen future points. That invalidates the accuracy magnitudes and the comparison to Wu et al. in Section 5, where the authors attribute their 0.0112 vs 0.0373 gap to the 80:20 vs 70:30 split without acknowledging that the split mechanics differ fundamentally.\n\nThe volatility claim is even less supported. No coverage analysis, pinball score, or any other quantile calibration metric is reported. The Discussion actually concedes this: \"we had no indication on how accurate our quantile predictions are.\" That is not a minor omission; it is the central claim of the paper. The quantile bands could be wildly miscalibrated and nothing in the paper would detect it.\n\nMinor issues: the hyperparameters are taken from the authors' own prior work, which is fine and standard practice, though it means the comparison to Wu et al. is not the only uncontrolled element. The benchmark datasets (sunspot, Mackey-Glass, Lorenz) are useful but they are chaotic or pseudo-periodic, not representative of financial volatility, so they don't rescue the main result.\n\nWho is this for? Practitioners who want a ready-made template for adding quantile output heads to LSTM-style forecasters. That is a legitimate use, but the numbers in the paper should not be cited as evidence until the split is fixed. The paper deserves a serious referee: it is reproducible, the methods are clearly described, and the fix is straightforward. I would recommend peer review with a mandatory request for a temporal split (e.g., rolling-origin) and a proper quantile evaluation (coverage, pinball, interval score) before any acceptance.","headline":"Plausible internal comparison but the random 80:20 split leaks future information, so the headline forecasting claim is not yet supported; a temporal re-split and quantile calibration check could fix it.","tokens_in":23815,"tokens_out":1539,"would_cite":false,"duration_ms":16328,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Swapping the loss function of LSTM and CNN forecasters for quantile loss yields interval predictions without sacrificing point accuracy.","keywords":["quantile regression","deep learning","time series prediction","multi-step ahead","uncertainty quantification","cryptocurrency","LSTM","CNN"],"falsifier":"Re-run the same models with a strict chronological split (train on the earliest 80% of dates, test on the most recent 20%), keep all other hyperparameters fixed, and compare the quantile models' median RMSE and their 90% interval coverage to the baselines. If the quantile models no longer match the baselines, or if coverage falls far below 90%, the paper's central claim would be contradicted.","tokens_in":22797,"feed_emoji":"📈","tokens_out":5066,"duration_ms":40919,"temperature":0.7,"pith_summary":"This paper tries to show that adding quantile regression to standard deep learning forecasters produces multi-step ahead predictions with uncertainty intervals at no cost in point accuracy. The authors build quantile versions of bidirectional, encoder-decoder, and convolutional LSTM models, trained with the asymmetric check loss instead of mean squared error, and test them on Bitcoin, Ethereum, and three benchmark time series. They report that the quantile models match and sometimes beat their conventional counterparts, and that the median output of a quantile ED-LSTM is the most accurate and stable predictor. If true, this means practitioners can get calibrated-looking risk bands from the same architectures they already use, without a separate uncertainty model.","feed_headline":"Quantile loss adds uncertainty bands with no accuracy cost","feed_subtitle":"LSTM and CNN forecasters trained on quantile loss match point accuracy while emitting 5th–95th percentile ranges.","key_machinery":"The central object is the quantile (check) loss function $\\rho_\\tau(u) = u(\\tau - \\mathbb{1}_{u<0})$, applied to the output layer of standard recurrent and convolutional architectures. For each prediction horizon the network emits one value per target quantile (0.05, 0.25, 0.5, 0.75, 0.95), and the median ($\\tau = 0.5$) is used as the point forecast. The loss rewards under-prediction for high $\\tau$ and over-prediction for low $\\tau$, so the trained network's outputs spread into a conditional quantile band. This machinery is what carries the claim: it converts a single-output forecaster into a multi-quantile forecaster with no architectural change beyond the loss and output dimension.","core_discovery":"The central claim is that integrating a quantile loss function with deep learning provides additional predictions for selected quantiles without a loss in prediction accuracy compared to conventional deep learning models, and that the resulting quantile models handle volatility more effectively. On Bitcoin and Ethereum daily close prices, the quantile encoder-decoder LSTM achieves the best or tied-best mean RMSE across five-step horizons, while also emitting 5th, 25th, 75th, and 95th percentile trajectories. The paper's stated objective is not to beat the existing models but to show that the quantile versions can provide uncertainty information 'for free' — matching point accuracy while adding a distributional view. The authors frame this as a form of extreme-value forecasting because the outer quantiles bound the range of possible outcomes.","pith_inferences":["A chronological train/test split would likely be a stricter test; given the paper's random split, the reported RMSEs likely mix interpolation and extrapolation, so the parity claim may not survive out-of-sample.","The paper does not measure quantile calibration (e.g., empirical coverage of the 90% interval); a coverage check on a temporal holdout would directly test whether the bands are trustworthy.","The volatility-handling claim rests on tighter confidence intervals across runs; a sharper test would compare quantile-model errors during known high-volatility subperiods against baseline errors.","The framework extends naturally to data imputation and climate extremes, as the authors note, but those uses would require enforcing non-negativity constraints the paper admits it omitted."],"forward_implications":["Quantile loss can be dropped into BD-LSTM, Conv-LSTM, and ED-LSTM forecasters to produce 5th–95th percentile prediction bands while keeping point RMSE effectively unchanged.","The median output of the quantile ED-LSTM is the most accurate and stable predictor on the cryptocurrency datasets and benchmarks tested.","The framework offers a simple frequentist route to uncertainty quantification in multi-step ahead forecasting, complementing Bayesian neural networks.","Practitioners in volatile domains such as crypto trading can obtain risk ranges directly from a single trained network instead of running ensemble or Bayesian methods.","The ranking across architectures (ED-LSTM best, BD-LSTM least robust) persists in the quantile versions, suggesting architecture choice dominates loss-function choice."],"supporting_citations":[{"why":"Introduces quantile regression and the check loss function that the paper applies to deep learning outputs.","marker":"[1]"},{"why":"Supplies the Bitcoin and Ethereum datasets, window sizes, and baseline results the paper compares against.","marker":"[44]"},{"why":"Supplies the benchmark time series (Sunspot, Mackey-Glass, Lorenz), hyperparameter conventions, and the multi-step ahead evaluation methodology.","marker":"[47]"},{"why":"Provides the sequence-to-sequence encoder-decoder LSTM architecture used by the ED-LSTM models.","marker":"[78]"},{"why":"Provides the convolutional LSTM architecture used by the Conv-LSTM models.","marker":"[79]"}],"fun_headline_variants":["Free uncertainty bands from deep time series models","Quantile loss: more forecast info, zero accuracy loss","Deep nets get quantile ranges without sacrificing point error","Extra quantile forecasts at no accuracy cost, new study","Uncertainty bands come free with deep learning forecasters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a randomly selected 80:20 split of a time series produces a valid test set for multi-step ahead forecasting; because training samples can come from after the test period, the reported RMSE measures interpolation within the same time span rather than genuine forward prediction.","fun_headline_variants_meta":{"raw":{"variants":["Free uncertainty bands from deep time series models","Quantile loss: more forecast info, zero accuracy loss","Deep nets get quantile ranges without sacrificing point error","Extra quantile forecasts at no accuracy cost, new study","Uncertainty bands come free with deep learning forecasters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000126,"raw_usage":{"total_tokens":1096,"prompt_tokens":915,"completion_tokens":181,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":104}},"tokens_in":531,"tokens_out":181,"duration_ms":2775,"temperature":1.0,"reasoning_tokens":104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:01:29.632278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same models with a strict chronological split (train on the earliest 80% of dates, test on the most recent 20%), keep all other hyperparameters fixed, and compare the quantile models' median RMSE and their 90% interval coverage to the baselines. If the quantile models no longer match the baselines, or if coverage falls far below 90%, the paper's central claim would be contradicted.","supporting_citations":[{"cited_title":"Review of deep learning models for crypto price prediction: implementation and evaluation","cited_arxiv_id":"2405.11431","evidence_quote":"Supplies the Bitcoin and Ethereum datasets, window sizes, and baseline results the paper compares against."},{"cited_title":"Sutskever, O","cited_arxiv_id":null,"evidence_quote":"Provides the sequence-to-sequence encoder-decoder LSTM architecture used by the ED-LSTM models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the convolutional LSTM architecture used by the Conv-LSTM models."}],"review_version":1}