REVIEW 5 major objections 4 minor 37 references
Benchmarking Quantum Models for Time-series Forecasting
T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper benchmarks five quantum machine learning models against classical baselines for one-step-ahead univariate time-series forecasting and finds that the best classical models achieve lower mean absolute error on both datasets, while…
desk verdict A useful but under-documented negative benchmark: classical beats quantum in two forecasting tasks, yet unequal tuning budgets and missing artifacts leave the fairness assumption open. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the benchmarking protocol: a single pipeline that applies the same k-fold splits (three folds for hyperparameter selection, three for testing), early stopping, ten repeated runs per configuration, and grid-search hyperparameter optimization to every model, with mean absolute error as the final comparison metric. The quantum models are hybrid quantum-classical systems simulated noiselessly—small variational circuits or an annealer-based energy model embedded in classical training loops—so the comparison isolates the model architecture's forecasting accuracy rather than hardware performance.
What would settle it
A re-benchmark that gives every model the same number of hyperparameter configurations—for example, 108 for the QNN and QNN Ising models—and the same hand-picked range widths, on the same data and folds, would settle whether the reported classical advantage is an artifact of unequal tuning budgets; if a quantum model then achieves lower MAE than the best LSTM on both datasets, the paper's central conclusion would be overturned.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a negative result with a nuance: after extensive hyperparameter optimization, the best classical models (the LSTM on the pasta data and the trivial last-value model on the Apple data) outperformed the best quantum models on both datasets. Most quantum models nevertheless achieved comparable accuracy, and the relative ranking of the quantum models flipped between datasets—QRC was best on Apple stock and worst on pasta, while QNN Ising showed the opposite pattern. Two quantum models beat classical ARIMA on the pasta data, but none beat the best classical model anywhere. The authors read this as evidence that current quantum machine learning provides no accuracy advantage for one-step-ahead univariate forecasting, and that claims of advantage must be dataset-specific rather than general.
Load-bearing premise
The load-bearing premise is that the comparison is fair, meaning the hyperparameter search spaces and training effort for quantum and classical models are comparable enough; the paper itself notes that the number of configurations differed (108 for QLSTM and QDBM, 48 for the QNNs) and the ranges were hand-picked.
Editorial extensions
If this is right
- The best classical model beat the best quantum model on both datasets, so current quantum approaches do not improve one-step-ahead univariate forecasting accuracy in these settings.
- Model choice mattered more than hyperparameter tuning: the spread in validation MAE across model types was larger than the spread across hyperparameter configurations within a model.
- Performance did not transfer across datasets: the quantum reservoir computer was the best quantum model on Apple stock data but the worst on pasta sales, while the QNN Ising model showed the opposite.
- On the Apple stock data, even the last-value baseline beat every machine learning model, classical or quantum, suggesting the daily series is effectively unpredictable with the features used.
- On the pasta sales data, two quantum models outperformed classical ARIMA, but the classical LSTM still had the lowest error.
Reading between the lines
- Because the study covers only single-step univariate forecasting, its negative result does not speak to multivariate or multi-step problems, where quantum circuit expressivity could matter more; testing that is a natural next step.
- The unequal hyperparameter budgets (108 configurations for QLSTM and QDBM versus 48 for the QNNs) leave open the possibility that a re-run with equal budgets would narrow or change the gap.
- All gate-based runs were noiseless simulations; evaluating on real quantum hardware, which the paper leaves for future work, could shift relative rankings in either direction.
- The paper's finding that architecture choice outweighs tuning effort suggests that future benchmarking should prioritize broad model-family search over fine-grained hyperparameter optimization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper benchmarks five quantum machine learning models (QNN, QNN Ising, QDBM, QRC, QLSTM) against three classical baselines (Last Value, ARIMA, LSTM) for one-step-ahead univariate forecasting on two real datasets: pasta sales and Apple stock. The models are trained with temporal k-fold cross-validation, early stopping, and grid-search hyperparameter optimization, and all quantum circuits are simulated noiselessly. The main finding is that the best classical models outperform the best quantum models on both datasets, that most quantum models achieve comparable results, and that relative performance is strongly dataset-dependent. The paper also reports that hyperparameter choice has a smaller effect than model choice.
Significance. If the result is correct, this is a useful contribution to the emerging benchmarking literature for quantum machine learning. The study compares several quantum architectures on the same protocol with repeated runs, error bars, temporal k-fold cross-validation, and real data, and it explicitly avoids the common single-model versus single-classical-baseline comparison. Its main value is as a point of empirical reference: it suggests no accuracy advantage for current quantum models on these simple univariate tasks, and it highlights dataset dependence. The conclusions are empirical and not derived from fitted parameters, so circularity is not a concern; the burden lies on the fairness, completeness, and reproducibility of the experimental comparison.
major comments (5)
- [Section III.C] The central claim that classical models outperform quantum models depends on comparable tuning effort, but the manuscript does not provide the full hyperparameter search spaces. It states that QLSTM and QDBM had 108 configurations, the QNN models 48, and that the remaining models were manually optimized, yet it never reports the number of configurations or grid ranges for the classical LSTM, which is the best model on the pasta data. If LSTM were given a wider or better-matched search space, the observed gap could reflect tuning effort rather than model capability. Please report the complete grid for every model, including LSTM and ARIMA, and either equalize the number of configurations or explicitly discuss the consequences of the imbalance.
- [Section III.A] The text states that 'Both data sets were min-max normalized to between 0 and 1,' but it does not say whether the normalization parameters (the minimum and maximum) are estimated on the training portion of each fold only. If the global range that includes validation or test windows is used, future information leaks into the training features and can differentially inflate or deflate model performance. The authors should state the exact normalization procedure and, if global normalization was used, re-run the benchmarks with training-only normalization.
- [Section III.B.1] The data-splitting protocol is not fully specified. The text describes a first k-fold iteration with 500 data points (450 train, 50 validation), then shifting by 50 time steps, but it is unclear how the three validation folds and the three test folds are arranged. Are the test folds temporally after the validation folds, are they disjoint from them, and are the window sizes identical? This ambiguity prevents the reader from verifying that model selection and final evaluation use separate data. Please define the exact construction of the three validation and three test folds, including whether they overlap and how the best hyperparameters are transferred from validation to testing.
- [Section IV] The exclusion of a small percentage (<1%) of divergent QDBM predictions from the figure and, presumably, from the reported MAE is a fairness issue. If the divergent predictions are extreme values, removing them improves the QDBM's reported performance, and the effect on both the mean and the variance should be quantified. Please report the number of excluded predictions per fold, the criterion used to define divergence, and the MAE with and without these predictions included.
- [General (reproducibility)] The manuscript contains no code or data availability statement, and the hyperparameter ranges are not listed in a table or appendix. For an empirical benchmark whose main conclusion is a negative result about relative model performance, this lack of detail makes it impossible for a reader to verify the fairness of the comparison or to reproduce the results. I would request a code/data release, or at minimum a detailed appendix with all hyperparameter grids, random seeds, per-fold results, and the exact training configuration for every model.
minor comments (4)
- [Section III.C / Figure 5] The text says 'The hyperparameter ranges can be seen in Figure 5,' but Figure 5 plots MAE across configurations rather than listing the actual ranges; a separate table of the grid ranges for each model is needed.
- [Section III.B.3] The MSE formula is written as MSE = 1/n * sum(y - y_hat)^2, but the indices over which n is computed are not defined; please specify whether the sum is over forecast horizon, batch elements, or both.
- [Abstract / Section IV] The abstract mentions that 'for one data set two quantum models outperformed the classical ARIMA model,' but Section IV does not identify which two models or on which dataset; please name them explicitly when the results are described.
- [Section IV] The statement that 'QNN and QNN Ising models had similar performance ... more gates do not necessarily improve the performance' is interesting, but it would be strengthened by reporting the number of parameters or gates in each model, since the paper otherwise leaves parameter counts for other work.
Circularity Check
No circularity: the paper's claims are empirical benchmark observations on held-out test data, not derivations from fitted inputs or self-citations.
full rationale
This paper is an empirical benchmarking study, not a derivation chain. The central claim that the best classical models outperformed the best quantum models is a measured result on test data that was held out from model selection: three k-folds are used for hyperparameter optimization and three additional k-folds are used for testing (Section III.B.1), and the best hyperparameter configuration is chosen by validation MAE and then evaluated on the test folds (Section III.C). No model output is defined in terms of the target result, and no fitted parameter is relabeled as a prediction. The QRC readout is fit by linear regression on training data and evaluated on held-out data, which is standard supervised learning rather than circularity. The paper's model descriptions cite external prior work (QNN from [21], QDBM from [29], QRC from [30], QLSTM from [31]) and do not invoke the authors' own prior results as load-bearing evidence. Concerns such as unequal hyperparameter budgets across models, unreported LSTM tuning ranges, and the unspecified timing of min-max normalization are fairness and reproducibility limitations, not circularity, because they do not make any outcome equal to an input by construction. Therefore no circular step is present.
Assumptions & free parameters
Cite this review
Pith. "Pith review of Benchmarking Quantum Models for Time-series Forecasting." pith.science (2026). https://pith.science/paper/NW53WYTV
@misc{pith2026241213878,
author = {Pith},
title = {Pith review of: Benchmarking Quantum Models for Time-series Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/NW53WYTV}},
note = {Machine review of arXiv:2412.13878}
}
read the original abstract
Time series forecasting is a valuable tool for many applications, such as stock price predictions, demand forecasting or logistical optimization. There are many well-established statistical and machine learning models that are used for this purpose. Recently in the field of quantum machine learning many candidate models for forecasting have been proposed, however in the absence of theoretical grounds for advantage thorough benchmarking is essential for scientific evaluation. To this end, we performed a benchmarking study using real data of various quantum models, both gate-based and annealing-based, comparing them to the state-of-the-art classical approaches, including extensive hyperparameter optimization. Overall we found that the best classical models outperformed the best quantum models. Most of the quantum models were able to achieve comparable results and for one data set two quantum models outperformed the classical ARIMA model. These results serve as a useful point of comparison for the field of forecasting with quantum machine learning.
Figures
Reference graph
Works this paper leans on
-
[1]
Time series analysis: forecasting and control
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control . John Wiley & Sons, 2015
2015
-
[2]
Time-series forecasting with deep learning: a survey
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A , 379(2194):20200209, 2021
work page 2021
-
[3]
Rieffel, Pedram Roushan, Nicholas Rubin, Daniel Sank, Kevin J
Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando Brandao, David Buell, Brian Burkett, Yu Chen, Jimmy Chen, Ben Chiaro, Roberto Collins, William Courtney, Andrew Dunsworth, Edward Farhi, Brooks Foxen, Austin Fowler, Craig Michael Gidney, Marissa Giustina, Rob Graff, Keith Guerin, Steve Hab...
work page 2019
-
[4]
Peter W. Shor. Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer. SIAM Journal on Computing, 26(5):1484–1509, 1997
1997
-
[5]
Quantum algorithm for linear systems of equations
Aram W Harrow, Avinatan Hassidim, and Seth Lloyd. Quantum algorithm for linear systems of equations. Physical review letters , 103(15):150502, 2009
2009
-
[6]
Quantum machine learning
Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, 549(7671):195–202, 2017
2017
-
[7]
Quantum machine learning: from nisq to fault tolerance
Yunfei Wang and Junyu Liu. Quantum machine learning: from nisq to fault tolerance. arXiv preprint arXiv:2401.11351 , 2024
arXiv 2024
-
[8]
The power of quantum neural networks
Amira Abbas, David Sutter, Christa Zoufal, Aur ´elien Lucchi, Alessio Figalli, and Stefan Woerner. The power of quantum neural networks. Nature Computational Science , 1(6):403–409, 2021
work page 2021
Show all 37 references
-
[9]
Expressibility and entangling capability of parameterized quantum circuits for hy- brid quantum-classical algorithms
Sukin Sim, Peter D Johnson, and Al ´an Aspuru-Guzik. Expressibility and entangling capability of parameterized quantum circuits for hy- brid quantum-classical algorithms. Advanced Quantum Technologies , 2(12):1900070, 2019
2019
-
[10]
Ex- pressive power of parametrized quantum circuits
Yuxuan Du, Min-Hsiu Hsieh, Tongliang Liu, and Dacheng Tao. Ex- pressive power of parametrized quantum circuits. Phys. Rev. Research, 2:033125, Jul 2020
2020
-
[11]
Generalization in quantum machine learning from few training data
Matthias C Caro, Hsin-Yuan Huang, Marco Cerezo, Kunal Sharma, Andrew Sornborger, Lukasz Cincio, and Patrick J Coles. Generalization in quantum machine learning from few training data. Nature communi- cations, 13(1):4919, 2022
2022
-
[12]
Balakrishnan, Umasree Mariappan, Pagadala Geetha Manikanta Raghavendra, Pallela Karthikeya Reddy, Rayavarapu Lak- shmi Narasimha Dinesh, and Shaik Bugganapalli Jabiulla
D. Balakrishnan, Umasree Mariappan, Pagadala Geetha Manikanta Raghavendra, Pallela Karthikeya Reddy, Rayavarapu Lak- shmi Narasimha Dinesh, and Shaik Bugganapalli Jabiulla. Quantum neural network for time series forecasting: Harnessing quantum computing’s potential in predicti...
2023
-
[13]
Michał Siemaszko, Adam Buraczewski, B. L. Saux, and Magdalena Stobi’nska. Rapid training of quantum recurrent neural networks. Quantum Machine Intelligence , 5:1–16, 2022
2022
-
[14]
Recurrent quantum neural networks
Johannes Bausch. Recurrent quantum neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1368–1379. Curran Associates, Inc., 2020
2020
-
[15]
Learning temporal data with a variational quantum recurrent neural network
Yuto Takaki, Kosuke Mitarai, Makoto Negoro, Keisuke Fujii, and Masahiro Kitagawa. Learning temporal data with a variational quantum recurrent neural network. Physical Review A , 103(5):052414, 2021
2021
-
[16]
Learning quantum processes with memory–quantum recurrent neural networks
Dmytro Bondarenko, Robert Salzmann, and Viktoria-S Schmiesing. Learning quantum processes with memory–quantum recurrent neural networks. arXiv preprint arXiv:2301.08167 , 2023
2023 arXiv
-
[17]
So- riano, and Roberta Zambrini
Pere Mujal, Rodrigo Mart ´ınez-Pe˜na, Gian Luca Giorgi, Miguel C. So- riano, and Roberta Zambrini. Time-series quantum reservoir computing with weak and projective measurements. npj Quantum Information , 9(16), 2023
2023
-
[18]
Quantum reservoir computing implementations for classical and quantum problems
Adam Burgess and Marian Florescu. Quantum reservoir computing implementations for classical and quantum problems. arXiv preprint arXiv:2211.08567, 2022
2022 arXiv
-
[19]
Quantum reservoir computing with repeated measurements on superconducting devices
Toshiki Yasuda, Yudai Suzuki, Tomoyuki Kubota, Kohei Nakajima, Qi Gao, Wenlong Zhang, Satoshi Shimono, Hendra I Nurdin, and Naoki Yamamoto. Quantum reservoir computing with repeated measurements on superconducting devices. arXiv preprint arXiv:2310.06706 , 2023
-
[20]
Time series forecasting with quantum machine learning architectures
Mayra Alejandra Rivera-Ruiz, Andres Mendez-Vazquez, and Jos´e Mauricio L ´opez-Romero. Time series forecasting with quantum machine learning architectures. In Obdulia Pichardo Lagunas, Juan Mart´ınez-Miranda, and Bella Mart ´ınez Seis, editors, Advances in Computational Intell...
2022
-
[21]
Emmanoulopoulos and Sofija Dimoska
D. Emmanoulopoulos and Sofija Dimoska. Quantum machine learning in finance: Time series forecasting. 2022
2022
-
[22]
Ferreira-Martins, Samurai Brito QC Ware Corp, Irif Universit’e Paris Cit’e, Cnrs, and Ita’u Unibanco
Sohum Thakkar, Skander Kazdaghli, Natansh Mathur, Iordanis Kereni- dis, Andre J. Ferreira-Martins, Samurai Brito QC Ware Corp, Irif Universit’e Paris Cit’e, Cnrs, and Ita’u Unibanco. Improved financial forecasting via quantum machine learning. Quantum Mach. Intell., 6:27, 2023
2023
-
[23]
El Amine Cherrat, S. Sridhar Raj, Iordanis Kerenidis, Abhishek Shekhar, Ben Wood, John Dee, Shouvanik Chakrabarti, Richard Chen, Dylan Herman, Shaohan Hu, Pierre Minssen, Ruslan Shaydulin, Yue Sun, Romina Yalovetzky, and Marco Pistoia. Quantum deep hedging. ArXiv, abs/2303.16585, 2023
2023 arXiv
-
[24]
The potential of quantum techniques for stock price prediction
Naman Srivastava, Gaurang Belekar, Neel Shahakar, and Aswath Babu H. The potential of quantum techniques for stock price prediction. 2023 IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) , pages 1–7, 2023
2023
-
[25]
Better than classical? the subtle art of benchmarking quantum machine learning models
Joseph Bowles, Shahnawaz Ahmed, and Maria Schuld. Better than classical? the subtle art of benchmarking quantum machine learning models. ArXiv, abs/2403.07059, 2024
2024 arXiv
-
[26]
Brockwell and Richard A
Peter J. Brockwell and Richard A. Davis. Introduction to Time Series and Forecasting. Springer Texts in Statistics. Springer International Publishing, 2016
2016
-
[27]
Long short-term memory
Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9:1735–80, 12 1997
1997
-
[28]
Learning to forget: Continual prediction with lstm
Felix Gers, J ¨urgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. Neural computation , 12:2451–71, 10 2000
2000
-
[29]
Oberoi, and Pooya Ronagh
Daniel Crawford, Anna Levit, Navid Ghadermarzy, Jaspreet S. Oberoi, and Pooya Ronagh. Reinforcement learning using quantum boltzmann machines, 2019
2019
-
[30]
Temporal information processing on noisy quantum computers
Jiayin Chen, Hendra I Nurdin, and Naoki Yamamoto. Temporal information processing on noisy quantum computers. Physical Review Applied, 14(2):024065, 2020
2020
-
[31]
Quantum long short-term memory
Samuel Yen-Chi Chen, Shinjae Yoo, and Yao-Lung L Fang. Quantum long short-term memory. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8622–8626. IEEE, 2022
2022
-
[32]
Paolo Mancuso, Veronica Piccialli, and Antonio M. Sudoso. A machine learning approach for forecasting hierarchical time series. Expert Systems with Applications , 182:115102, 2021
2021
-
[33]
On over-fitting in model selection and subsequent selection bias in performance evaluation
Gavin Cawley and Nicola Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research , 11:2079–2107, 07 2010
2010
-
[34]
Time series nested cross-validation, May 2018
Courtney Cochrane. Time series nested cross-validation, May 2018
2018
-
[35]
Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55–69
Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55–69. Springer, 2002
2002
-
[36]
Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance
Cort J Willmott and Kenji Matsuura. Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate research, 30(1):79–82, 2005
2005
-
[37]
Smith et al
Taylor G. Smith et al. pmdarima: Arima estimators for Python, 2017–. [Online; accessed ¡today¿]
2017
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.