{"id":"55ce2a29-721b-4647-8519-c90a6d11b1ae","arxiv_id":"2412.13878","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmarking study comparing five quantum forecasting models with classical baselines on two real data sets, finding no quantum advantage over classical models.","lead":"This paper benchmarks five quantum machine learning models against classical baselines for one-step-ahead time series forecasting on two real data sets, finding that classical models achieve the best accuracy overall, with quantum models reaching comparable results on one data set. A smart generalist might read it for evidence on whether quantum methods currently offer practical forecasting advantages over mature classical approaches.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The negative result rests on an unverified fairness assumption: unequal hyperparameter budgets and unreported tuning ranges for LSTM, plus absent code/data, leave the classical-vs-quantum comparison open.","rationale":"The reader's weakest assumption—fairness of hyperparameter optimization—is the right starting point, and I agree it is load-bearing. My stress-test sharpens it: Section III.C gives the configuration counts for QLSTM/QDBM (108) and QNN/QNN Ising (48) but omits the LSTM's count, so the comparison budget for the winning classical model is unverifiable. I also flag a second, potentially more serious leakage risk: the paper says only that both data sets were min-max normalized to [0,1] without specifying whether the transformation was fit on training folds only. If the global range (including test periods) was used, all evaluated models see future information, which undermines the benchmark as a forecasting study. The paper explicitly acknowledges the unequal budgets and the QDBM exclusion, so these are not hidden flaws, but they remain unresolved. Because the central claim is a negative result about relative performance, the burden is on the authors to show the comparison is fair and reproducible. The lack of code, data, and a precise description of the LSTM search space justifies the reader's UNVERDICTED verdict; I do not see grounds to move it. My agreement is partial because I extend the fairness concern with the normalization-leakage issue and focus on the missing LSTM budget rather than only the quantum-model budgets.","tokens_in":7269,"tokens_out":6693,"duration_ms":65022,"concrete_test":"Obtain the code and data (or request them from the authors) and rerun the benchmark under a fixed 108-configuration budget per model with matched hyperparameter-range widths, and with normalization fitted only on each training fold. If LSTM still achieves lower test MAE than the best quantum model on pasta, and ARIMA/Last Value still leads on Apple, the central claim survives; if any quantum model ties or beats the classical leader, the conclusion as stated is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract and Section V) is that the best classical models outperformed the best quantum models. For this to be meaningful, each model must have received comparable tuning effort. Section III.C explicitly states that the number of hyperparameter configurations was unequal—108 for QLSTM and QDBM, 48 for QNN and QNN Ising—and that grid-search ranges were chosen by hand. The number of configurations for the classical LSTM, which wins on the pasta data, is not reported, so the reader cannot verify that it did not receive a systematically larger or better-matched budget. If LSTM's search space was materially wider or better aligned to the data, the observed gap could reflect tuning effort rather than model capability. In addition, the paper does not state whether min-max normalization parameters were estimated on training data only; if the global range including test windows was used, there is information leakage. The absence of code and data makes these checks impossible from the text. The post-hoc exclusion of <1% divergent QDBM predictions (Section IV) is a smaller issue but further complicates a fair comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks five quantum machine learning models (QNN, QNN Ising, QDBM, QRC, QLSTM) against three classical baselines (Last Value, ARIMA, LSTM) for one-step-ahead univariate forecasting on two real datasets: pasta sales and Apple stock. The models are trained with temporal k-fold cross-validation, early stopping, and grid-search hyperparameter optimization, and all quantum circuits are simulated noiselessly. The main finding is that the best classical models outperform the best quantum models on both datasets, that most quantum models achieve comparable results, and that relative performance is strongly dataset-dependent. The paper also reports that hyperparameter choice has a smaller effect than model choice.","tokens_in":7574,"tokens_out":4281,"duration_ms":38709,"significance":"If the result is correct, this is a useful contribution to the emerging benchmarking literature for quantum machine learning. The study compares several quantum architectures on the same protocol with repeated runs, error bars, temporal k-fold cross-validation, and real data, and it explicitly avoids the common single-model versus single-classical-baseline comparison. Its main value is as a point of empirical reference: it suggests no accuracy advantage for current quantum models on these simple univariate tasks, and it highlights dataset dependence. The conclusions are empirical and not derived from fitted parameters, so circularity is not a concern; the burden lies on the fairness, completeness, and reproducibility of the experimental comparison.","major_comments":[{"comment":"The central claim that classical models outperform quantum models depends on comparable tuning effort, but the manuscript does not provide the full hyperparameter search spaces. It states that QLSTM and QDBM had 108 configurations, the QNN models 48, and that the remaining models were manually optimized, yet it never reports the number of configurations or grid ranges for the classical LSTM, which is the best model on the pasta data. If LSTM were given a wider or better-matched search space, the observed gap could reflect tuning effort rather than model capability. Please report the complete grid for every model, including LSTM and ARIMA, and either equalize the number of configurations or explicitly discuss the consequences of the imbalance.","section":"Section III.C"},{"comment":"The text states that 'Both data sets were min-max normalized to between 0 and 1,' but it does not say whether the normalization parameters (the minimum and maximum) are estimated on the training portion of each fold only. If the global range that includes validation or test windows is used, future information leaks into the training features and can differentially inflate or deflate model performance. The authors should state the exact normalization procedure and, if global normalization was used, re-run the benchmarks with training-only normalization.","section":"Section III.A"},{"comment":"The data-splitting protocol is not fully specified. The text describes a first k-fold iteration with 500 data points (450 train, 50 validation), then shifting by 50 time steps, but it is unclear how the three validation folds and the three test folds are arranged. Are the test folds temporally after the validation folds, are they disjoint from them, and are the window sizes identical? This ambiguity prevents the reader from verifying that model selection and final evaluation use separate data. Please define the exact construction of the three validation and three test folds, including whether they overlap and how the best hyperparameters are transferred from validation to testing.","section":"Section III.B.1"},{"comment":"The exclusion of a small percentage (<1%) of divergent QDBM predictions from the figure and, presumably, from the reported MAE is a fairness issue. If the divergent predictions are extreme values, removing them improves the QDBM's reported performance, and the effect on both the mean and the variance should be quantified. Please report the number of excluded predictions per fold, the criterion used to define divergence, and the MAE with and without these predictions included.","section":"Section IV"},{"comment":"The manuscript contains no code or data availability statement, and the hyperparameter ranges are not listed in a table or appendix. For an empirical benchmark whose main conclusion is a negative result about relative model performance, this lack of detail makes it impossible for a reader to verify the fairness of the comparison or to reproduce the results. I would request a code/data release, or at minimum a detailed appendix with all hyperparameter grids, random seeds, per-fold results, and the exact training configuration for every model.","section":"General (reproducibility)"}],"minor_comments":[{"comment":"The text says 'The hyperparameter ranges can be seen in Figure 5,' but Figure 5 plots MAE across configurations rather than listing the actual ranges; a separate table of the grid ranges for each model is needed.","section":"Section III.C / Figure 5"},{"comment":"The MSE formula is written as MSE = 1/n * sum(y - y_hat)^2, but the indices over which n is computed are not defined; please specify whether the sum is over forecast horizon, batch elements, or both.","section":"Section III.B.3"},{"comment":"The abstract mentions that 'for one data set two quantum models outperformed the classical ARIMA model,' but Section IV does not identify which two models or on which dataset; please name them explicitly when the results are described.","section":"Abstract / Section IV"},{"comment":"The statement that 'QNN and QNN Ising models had similar performance ... more gates do not necessarily improve the performance' is interesting, but it would be strengthened by reporting the number of parameters or gates in each model, since the paper otherwise leaves parameter counts for other work.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a plausible and potentially useful negative benchmarking result, but the absence of code/data and the incomplete description of the hyperparameter search spaces and data-splitting protocol make the central fairness assumption unverifiable. This is the kind of issue that should be fixed before the paper is used as a reference point by the community. The paper fits the scope of the journal if revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best to know: this is a sober, multi-model benchmark of five quantum forecasting approaches against classical baselines on two real datasets, and the headline result—classical wins, dataset dependence is strong—matches the broader QML benchmarking literature. The paper is honest about most of its limitations, and it gives the field a useful negative data point. It is not a breakthrough, and it does not need to be.\n\nWhat's genuinely useful: the breadth of models (gate-based QNNs, QLSTM, QRC, plus an annealing-inspired QDBM) and the consistent experimental protocol: k-fold CV with temporal block splitting, early stopping, repeated runs, and a clear separation between validation (HPO) and test folds. The observation that model choice matters more than hyperparameter tuning, and that relative ranking flips between datasets, is worth having on record. The paper's caution—no claim of advantage, explicit call for benchmarking like Bowles et al.—is commendable.\n\nWhere it gets soft: the central classical-vs-quantum comparison assumes comparable tuning effort, and that assumption is not verifiable from the manuscript. QLSTM and QDBM get 108 configurations, the two QNNs 48, and the classical LSTM's configuration count is not reported at all. The grid-search ranges are only in a figure we can't see, so we can't check whether LSTM's search space was wider or better matched. The paper explicitly acknowledges the unequal numbers, which is honest, but it doesn't tell us whether the imbalance favors the classical side. Also, the min-max normalization is described in one sentence ('Both data sets were min-max normalized') with no statement about fitting on training windows only; if the global range was used, that's leakage. And the <1% divergent QDBM predictions are 'left out of the Figure'—it's unclear if they were also excluded from the reported averages, which would be a red flag. None of these points is fatal on its own; together they mean the headline numeric gap is not yet fully supported.\n\nBottom line: this is a paper for QML forecasting specialists and for referees who want a concrete, if incomplete, benchmark to cite. It deserves peer review rather than desk rejection, provided the authors can supply code, data, and a precise accounting of tuning budgets and normalization. Without those, it reads as a solid technical report, not a settled benchmark.","headline":"A useful but under-documented negative benchmark: classical beats quantum in two forecasting tasks, yet unequal tuning budgets and missing artifacts leave the fairness assumption open.","tokens_in":7976,"tokens_out":2612,"would_cite":false,"duration_ms":22673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper benchmarks five quantum machine learning models against classical baselines for one-step-ahead univariate time-series forecasting and finds that the best classical models achieve lower mean absolute error on both datasets, while…","keywords":["quantum machine learning","time-series forecasting","benchmarking","quantum neural networks","quantum reservoir computing","quantum Boltzmann machine","QLSTM","hyperparameter optimization"],"falsifier":"A re-benchmark that gives every model the same number of hyperparameter configurations—for example, 108 for the QNN and QNN Ising models—and the same hand-picked range widths, on the same data and folds, would settle whether the reported classical advantage is an artifact of unequal tuning budgets; if a quantum model then achieves lower MAE than the best LSTM on both datasets, the paper's central conclusion would be overturned.","tokens_in":7085,"feed_emoji":"⚛️","tokens_out":6876,"duration_ms":56136,"temperature":0.7,"pith_summary":"This paper asks whether quantum machine learning models can beat established classical methods at the simplest forecasting task: predicting the next value of a univariate time series. It benchmarks five quantum approaches—two gate-based quantum neural networks, a quantum deep Boltzmann machine, a quantum reservoir computer, and a quantum LSTM—against last-value, ARIMA, and LSTM baselines on two real datasets, using temporal-order-preserving k-fold cross-validation, early stopping, and grid-search hyperparameter optimization. The central finding is that the best classical models achieve lower mean absolute error than the best quantum models on both datasets, although most quantum models land close to classical performance and two quantum models beat classical ARIMA on the pasta dataset. The authors conclude that current quantum models offer no accuracy advantage for this task and that relative model performance is strongly dataset-dependent.","feed_headline":"Quantum models match—not beat—classical forecasters in new benchmark","feed_subtitle":"A controlled comparison on real sales and stock data finds no accuracy advantage from current quantum methods.","key_machinery":"The load-bearing machinery is the benchmarking protocol: a single pipeline that applies the same k-fold splits (three folds for hyperparameter selection, three for testing), early stopping, ten repeated runs per configuration, and grid-search hyperparameter optimization to every model, with mean absolute error as the final comparison metric. The quantum models are hybrid quantum-classical systems simulated noiselessly—small variational circuits or an annealer-based energy model embedded in classical training loops—so the comparison isolates the model architecture's forecasting accuracy rather than hardware performance.","core_discovery":"On the paper's own terms, the discovery is a negative result with a nuance: after extensive hyperparameter optimization, the best classical models (the LSTM on the pasta data and the trivial last-value model on the Apple data) outperformed the best quantum models on both datasets. Most quantum models nevertheless achieved comparable accuracy, and the relative ranking of the quantum models flipped between datasets—QRC was best on Apple stock and worst on pasta, while QNN Ising showed the opposite pattern. Two quantum models beat classical ARIMA on the pasta data, but none beat the best classical model anywhere. The authors read this as evidence that current quantum machine learning provides no accuracy advantage for one-step-ahead univariate forecasting, and that claims of advantage must be dataset-specific rather than general.","pith_inferences":["Because the study covers only single-step univariate forecasting, its negative result does not speak to multivariate or multi-step problems, where quantum circuit expressivity could matter more; testing that is a natural next step.","The unequal hyperparameter budgets (108 configurations for QLSTM and QDBM versus 48 for the QNNs) leave open the possibility that a re-run with equal budgets would narrow or change the gap.","All gate-based runs were noiseless simulations; evaluating on real quantum hardware, which the paper leaves for future work, could shift relative rankings in either direction.","The paper's finding that architecture choice outweighs tuning effort suggests that future benchmarking should prioritize broad model-family search over fine-grained hyperparameter optimization."],"forward_implications":["The best classical model beat the best quantum model on both datasets, so current quantum approaches do not improve one-step-ahead univariate forecasting accuracy in these settings.","Model choice mattered more than hyperparameter tuning: the spread in validation MAE across model types was larger than the spread across hyperparameter configurations within a model.","Performance did not transfer across datasets: the quantum reservoir computer was the best quantum model on Apple stock data but the worst on pasta sales, while the QNN Ising model showed the opposite.","On the Apple stock data, even the last-value baseline beat every machine learning model, classical or quantum, suggesting the daily series is effectively unpredictable with the features used.","On the pasta sales data, two quantum models outperformed classical ARIMA, but the classical LSTM still had the lowest error."],"supporting_citations":[{"why":"Supplies the methodological rationale that thorough benchmarking across multiple models is essential for fair assessment of quantum machine learning.","marker":"[25]"},{"why":"Source of the first quantum neural network architecture and the Apple-stock forecasting comparison.","marker":"[21]"},{"why":"Source of the variational-quantum-circuit QLSTM architecture adapted for this benchmark.","marker":"[31]"},{"why":"Source of the quantum deep Boltzmann machine adapted here for time-series forecasting.","marker":"[29]"},{"why":"Defines quantum reservoir computing with a fixed circuit reservoir and linear readout, basis of the QRC model.","marker":"[30]"},{"why":"Defines the classical LSTM that serves as the key deep-learning baseline.","marker":"[27]"},{"why":"Supplies automatic ARIMA hyperparameter selection for the classical baseline.","marker":"[37]"},{"why":"Introduced the pasta sales dataset used in the benchmark.","marker":"[32]"}],"fun_headline_variants":["Quantum forecasting: no advantage over classical models","Classical models beat quantum in forecasting benchmark","Quantum time-series models can't surpass classical baselines","Benchmark finds quantum no better than classical forecasting","Quantum vs classical forecasting: no quantum edge found"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison is fair, meaning the hyperparameter search spaces and training effort for quantum and classical models are comparable enough; the paper itself notes that the number of configurations differed (108 for QLSTM and QDBM, 48 for the QNNs) and the ranges were hand-picked.","fun_headline_variants_meta":{"raw":{"variants":["Quantum forecasting: no advantage over classical models","Classical models beat quantum in forecasting benchmark","Quantum time-series models can't surpass classical baselines","Benchmark finds quantum no better than classical forecasting","Quantum vs classical forecasting: no quantum edge found"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1167,"prompt_tokens":838,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":454,"tokens_out":329,"duration_ms":3775,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:40:09.536691+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A re-benchmark that gives every model the same number of hyperparameter configurations—for example, 108 for the QNN and QNN Ising models—and the same hand-picked range widths, on the same data and folds, would settle whether the reported classical advantage is an artifact of unequal tuning budgets; if a quantum model then achieves lower MAE than the best LSTM on both datasets, the paper's central conclusion would be overturned.","supporting_citations":[{"cited_title":"Emmanoulopoulos and Sofija Dimoska","cited_arxiv_id":null,"evidence_quote":"Source of the first quantum neural network architecture and the Apple-stock forecasting comparison."},{"cited_title":"Quantum long short-term memory","cited_arxiv_id":null,"evidence_quote":"Source of the variational-quantum-circuit QLSTM architecture adapted for this benchmark."},{"cited_title":"Oberoi, and Pooya Ronagh","cited_arxiv_id":null,"evidence_quote":"Source of the quantum deep Boltzmann machine adapted here for time-series forecasting."},{"cited_title":"Temporal information processing on noisy quantum computers","cited_arxiv_id":null,"evidence_quote":"Defines quantum reservoir computing with a fixed circuit reservoir and linear readout, basis of the QRC model."},{"cited_title":"Long short-term memory","cited_arxiv_id":null,"evidence_quote":"Defines the classical LSTM that serves as the key deep-learning baseline."},{"cited_title":"Smith et al","cited_arxiv_id":null,"evidence_quote":"Supplies automatic ARIMA hyperparameter selection for the classical baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the pasta sales dataset used in the benchmark."}],"review_version":1}