REVIEW 3 major objections 6 minor 16 references
CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read CleanQRL provides single-script implementations of quantum reinforcement learning algorithms, each with a classical counterpart, so that QRL comparisons are no longer tangled in divergent codebases.
desk verdict A useful single-file QRL library with honest adoption of CleanRL, but the paper's central benchmarking claim is not supported by any in-paper numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the single-script agent file with three fixed parts: a make_env function, an Agent class inheriting from a standard module interface, and a training function. For quantum agents two further pieces are added: an observation wrapper that normalizes inputs to account for qubit count and gate periodicity, and a parametrized_quantum_circuit function containing the ansatz. The ansatz itself is the generic circuit with encoding, variational, and entangling blocks, with trainable scaling and rotation parameters; it defines the parametrized circuit as a module so that training logic is identical to the classical case. This structure is what lets one script carry the entire algorithm and lets quantum-classical differences be isolated to the approximator.
What would settle it
Re-run one included configuration, for example the quantum DQN agent on CartPole, from the released code with fixed random seeds and compare the logged learning curve with the published report. If the run does not reproduce, or if merely rescaling the observation wrapper flips the agent from learning to failing, the claimed consistent, ready-to-use comparison foundation is not as stable as stated.
Extended reading notes
Core claim
The central claim is that a consistent foundation for QRL is achievable by porting the single-file implementation philosophy of classical RL to quantum agents. In CleanQRL, the classical and quantum versions of an algorithm share the same logic; the quantum agent replaces the neural-network module with a parametrized quantum circuit embedded in the same module interface, so the only change is the function approximator. The included ansatzes follow a standard encoding, variational, and entangling block design, observations are wrapped for normalization, and gradient estimation cost in circuit executions is reported alongside sample efficiency. If this holds, researchers can modify one file, run one script, and compare their results against a stable baseline rather than reimplementing every subroutine.
Load-bearing premise
The library's value rests on the assumption that its default hyperparameters and generic quantum circuit designs perform well enough on the included environments to make the benchmarks meaningful; the paper notes that no extensive hyperparameter search was performed and that such circuits can be prone to training failures known as barren plateaus.
Editorial extensions
If this is right
- A researcher can take a quantum DQN or policy-gradient script, swap the ansatz, and re-run the benchmark without touching the training logic, which makes algorithmic variants cheap to test.
- Classical-versus-quantum comparisons produced on this substrate isolate the effect of the function approximator, because the two versions differ only there.
- Reported results include both environment-step sample efficiency and circuit-execution cost, reflecting the parameter-shift rule's linear gradient cost, so practitioners can judge quantum overhead directly.
- The bundled configuration files and logging setup give each published benchmark a reproducible starting point, inviting community-contributed hyperparameters.
- Tutorial integrations for graph-based, Hamiltonian-based, maze, and noise-model approaches demonstrate that new QRL ideas can be dropped into the framework without restructuring.
Reading between the lines
- Beyond the paper's claims: if the single-file convention becomes the community default, QRL papers may converge on shared baselines and shrink implementation-driven variance.
- Beyond the paper's claims: because no extensive hyperparameter search was run and generic ansatzes risk barren plateaus, the reported classical-versus-quantum curves are best read as feasibility demonstrations rather than performance verdicts.
- Beyond the paper's claims: the logged circuit-execution counts could feed a standardized 'quantum cost per unit of reward' metric, treating circuit executions as a first-class resource.
- Beyond the paper's claims: the observation wrapper is likely where most environment-specific tuning happens; varying only that wrapper across continuous-state environments would reveal how much of the observed gap is encoding, not algorithm.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CleanQRL, a Python library that provides single-file implementations of several quantum reinforcement learning (QRL) algorithms together with classical counterparts, following the design philosophy of CleanRL. The library uses PennyLane and PyTorch for the variational quantum circuits, supports gymnasium and selected jumanji environments, integrates Ray Tune for distributed hyperparameter search, and uses Weights & Biases for experiment logging. The manuscript describes the implementation architecture, the parametrized quantum circuits used, the tutorial examples, and benchmarks that are presented as external Weights & Biases reports. The central claim, stated in the Conclusion, is that CleanQRL establishes a consistent foundation for QRL algorithms and enables more nuanced comparisons between quantum algorithms and between classical and quantum algorithms.
Significance. If the library performs as described, CleanQRL would be a useful community resource: the single-file approach is genuinely valuable for readability, the inclusion of matched classical and quantum scripts addresses a recurring comparability problem in QRL research, the tutorials for graph-based, Hamiltonian-based, and noise-model approaches broaden the scope, and the public repository and documentation are strengths. However, the paper's main benchmarking claim is not substantiated by any quantitative evidence inside the manuscript: the only performance figure is a low-resolution screenshot of an external Weights & Biases report, and the text explicitly says that no extensive hyperparameter search was performed. Since the paper itself stresses that QRL is highly sensitive to hyperparameters and that the provided ansatzes are only generic examples, the reported comparisons cannot currently support the conclusion that CleanQRL enables meaningful benchmarks. The manuscript therefore needs a major revision to either supply in-paper evaluation evidence or adjust the claims to describe an infrastructure contribution rather than an established benchmarking foundation.
major comments (3)
- [Section V, Fig. 4] The central benchmarking claim is not supported by evidence inside the paper. The only quantitative result is a screenshot from an external Weights & Biases report, and the text states that no extensive hyperparameter search was performed. Because Section III also calls QRL 'highly sensitive to hyperparameters,' the classical-vs-quantum comparisons shown in Fig. 4 can reflect the accidental quality of the default configurations rather than any robust property of the algorithms. Please include in the manuscript benchmark tables or comparable figures with fixed seeds, means and standard deviations over multiple independent runs, environment steps to solution, and circuit-execution counts, together with the exact hyperparameter values used. If such an evaluation is not feasible in the paper, the claims should be explicitly limited to 'infrastructure for benchmarking' and the current conclusion should be softened.
- [Section III-A] The claim that CleanQRL establishes a consistent foundation and enables nuanced comparisons rests on the representative quality of the default ansatzes. The paper warns that the hardware-efficient ansatzes are 'just examples of simple and generic ansatzes that may require modifications' and that they are known to suffer from barren plateaus, yet the benchmark reports appear to use those defaults without documenting per-environment adaptations. Since ansatz choice strongly affects QRL performance, the paper should either justify that the defaults are adequate for the evaluated environments (for example, by comparing against the original Skolik et al. setup or by reporting the performance after a minimal user adaptation) or explicitly state that all comparisons are illustrative and not a benchmark of the algorithms themselves.
- [Section III, Implementation] The library's correctness is not evidenced in the manuscript. The paper presents the scripts as 'high-quality' implementations but provides no unit tests, smoke tests, or validation against known results from the original QRL publications for any of the implemented algorithms. Since the main contribution is a software artifact, some form of validation is needed to substantiate the quality claim: for example, expected reward curves on standard environments, a cross-check with the corresponding CleanRL classical baselines, or a comparison with published results for the same circuits and environments. At minimum, a reproducibility and validation statement should be added to the paper or appendix.
minor comments (6)
- [Abstract] The phrase 'at the interception between quantum computing and machine learning' should be 'intersection' rather than 'interception.'
- [Throughout] Use consistent capitalization for proper names: 'Weights & Biases' rather than 'weights&biases,' and 'PennyLane' and 'PyTorch' rather than 'pennylane' and 'pytorch.'
- [Fig. 4] The screenshot-based figure is hard to read and does not include error bars, confidence intervals, or axis labels that are clearly legible; please replace it with vector plots showing means and variances over seeds.
- [Section V] There are typographical issues in the text: 'parameter-shift rule ,' and 'open RL benchmark ,' both contain stray spaces before commas.
- [Section III] The paper describes the library as 'single-file' but then admits exceptions for replay buffers and jumanji interfaces; the terminology should be qualified (e.g., 'single-file for the core algorithm') to avoid overstating the design.
- [Section III and Section IV] A compact table listing the implemented algorithms, the corresponding file names, the supported environments, and the tutorial topics would greatly improve the navigability of the paper and would help reviewers verify the claimed coverage.
Circularity Check
No circular derivation: CleanQRL is an implementation/benchmarking library paper with no mathematical prediction chain; its self-citations are not load-bearing.
full rationale
This paper contains no derivation chain that could reduce to its own inputs. Its central claim is that CleanQRL provides single-file QRL implementations and a consistent benchmarking workflow; the supporting evidence is the library artifacts, external weights-and-biases reports, and configuration files, not a fitted model or a theorem. The variational ansatz is taken from the external work of Skolik et al. (2022) and is explicitly presented as an example, with the paper warning that these generic ansatzes may require modifications. The only author-overlapping citations, Kruse et al. (2024) and Lorenz et al. (2025), are used respectively as a tutorial example and as a general call for benchmarking; neither is invoked to justify a formal result or to forbid alternatives. The stated absence of an extensive hyperparameter search and the warning about barren plateaus weaken the benchmarking claim as a validity matter, but this is not circularity: no benchmark value in the paper is produced by fitting a parameter and then relabeling it as a prediction.
Assumptions & free parameters
assumptions (2)
- domain assumption The ansatzes from Skolik et al. (2022) are effective function approximators for the tested QRL environments.
- domain assumption The software stack (PennyLane, PyTorch, Ray Tune, Weights and Biases) works as expected and the single-file structure is maintained across all scripts.
Cite this review
Pith. "Pith review of CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms." pith.science (2026). https://pith.science/paper/Q62BGK5N
@misc{pith2026250707593,
author = {Pith},
title = {Pith review of: CleanQRL: Lightweight Single-file Implementations of Quantum Reinforcement Learning Algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q62BGK5N}},
note = {Machine review of arXiv:2507.07593}
}
read the original abstract
At the interception between quantum computing and machine learning, Quantum Reinforcement Learning (QRL) has emerged as a promising research field. Due to its novelty, a standardized and comprehensive collection for QRL algorithms has not yet been established. Researchers rely on numerous software stacks for classical Reinforcement Learning (RL) as well as on various quantum computing frameworks for the implementation of the quantum subroutines of their QRL algorithms. Inspired by the CleanRL library for classical RL algorithms, we present CleanQRL, a library that offers single-script implementations of many QRL algorithms. Our library provides clear and easy to understand scripts that researchers can quickly adapt to their own needs. Alongside ray tune for distributed computing and streamlined hyperparameter tuning, CleanQRL uses weights&biases to log important metrics, which facilitates benchmarking against other classical and quantum implementations. The CleanQRL library enables researchers to easily transition from theoretical considerations to practical applications.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
[Bergholm et al., 2018] Bergholm, V ., Izaac, J., Schuld, M., Gogolin, C.,
work page 2018
-
[2]
S., Alonso-Linaje, G., AkashNarayanan, B., Asadi, A., et al
Ahmed, S., Ajith, V ., Alam, M. S., Alonso-Linaje, G., AkashNarayanan, B., Asadi, A., et al. (2018). Pennylane: Automatic differentiation of hybrid quantum-classical computations. arXiv preprint arXiv:1811.04968 . [Biewald, 2020] Biewald, L. (2020). Experiment tracking with weights and biases. Software available from wandb.com. [Bonnet et al., 2023] Bonne...
arXiv 2018
-
[3]
Kalloniatis, T., et al. (2023). Jumanji: a diverse suite of scalable reinforce- ment learning environments in jax. arXiv preprint arXiv:2306.09884 . [Bou et al., 2023] Bou, A., Bettini, M., Dittert, S., Kumar, V ., Sodhani, S.,
arXiv 2023
-
[4]
Yang, X., De Fabritiis, G., and Moens, V . (2023). Torchrl: A data-driven decision-making library for pytorch. arXiv preprint arXiv:2306.00577 . [Bowles et al., 2024] Bowles, J., Ahmed, S., and Schuld, M. (2024). Better than classical? the subtle art of benchmarking quantum machine learning models. arXiv preprint arXiv:2403.07059 . [Brockman et al., 2016]...
arXiv 2023
-
[5]
Bellemare, M. G. (2018). Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110 . [Chen et al., 2020] Chen, S. Y .-C., Yang, C.-H. H., Qi, J., Chen, P.-Y .,
arXiv 2018
-
[6]
Ma, X., and Goan, H.-S. (2020). Variational quantum circuits for deep reinforcement learning. IEEE access, 8:141007–141024. [Dr˘agan et al., 2022] Dr ˘agan, T.-A., Monnet, M., Mendl, C. B., and Lorenz, J. M. (2022). Quantum reinforcement learning for solving a stochastic frozen lake environment and the impact of quantum architecture choices. arXiv preprin...
work page Pith review arXiv 2020
-
[7]
Goan, H.-S. (2022). Unentangled quantum reinforcement learning agents in the openai gym. arXiv preprint arXiv:2203.14348 . [Huang et al., 2022] Huang, S., Dossa, R. F. J., Ye, C., Braga, J.,
arXiv 2022
-
[8]
Chakraborty, D., Mehta, K., and Ara ˜Aˇsjo, J. G. (2022). Cleanrl: High- quality single-file implementations of deep reinforcement learning algo- rithms. Journal of Machine Learning Research , 23(274):1–18. [Huang et al., 2024] Huang, S., Gallou ´edec, Q., Felten, F., Raffin, A., Dossa, R. F. J., Zhao, Y ., Sullivan, R., Makoviychuk, V ., Makoviichuk, D.,...
work page 2022
Show all 16 references
-
[9]
N., Nikulin, A., Hu, X., Liu, T., Choi, J., and Yi, B
Mehta, K., Chakraborty, D., KG, A., Charraut, V ., Ye, C., Liu, Z., Alegre, L. N., Nikulin, A., Hu, X., Liu, T., Choi, J., and Yi, B. (2024). Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning. arXiv preprint arXiv:2402.03046 . [Javadi-Abhari et al...
2024 arXiv
-
[10]
J., Lishman, J., Gacon, J., Martiel, S., Nation, P
Wood, C. J., Lishman, J., Gacon, J., Martiel, S., Nation, P. D., Bishop, L. S., Cross, A. W., et al. (2024). Quantum computing with qiskit. arXiv preprint arXiv:2405.08810. [Jerbi et al., 2021] Jerbi, S., Gyurik, C., Marshall, S., Briegel, H., and Dun- jko, V . (2021). Paramet...
2024 arXiv
-
[11]
Lorenz, J. M. (2024). Hamiltonian-based quantum reinforcement learn- ing for neural combinatorial optimization. In 2024 IEEE International Conference on Quantum Computing and Engineering (QCE) , volume 1, pages 1617–1627. IEEE. [Kwak et al., 2021] Kwak, Y ., Yun, W. J., Jung, ...
2024
-
[12]
Cerezo, M. (2024). A review of barren plateaus in variational quantum computing. arXiv preprint arXiv:2405.00781 . [Liang et al., 2017] Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gonzalez, J., Goldberg, K., and Stoica, I. (2017). Ray rllib: A composable and scala...
2024 arXiv
-
[13]
Schopfer, F., Barbaresco, F., Kurowski, K., van der Schoot, W., Strohm, T., Senellart, J., et al. (2025). Systematic benchmarking of quantum computers: status and recommendations. arXiv preprint arXiv:2503.04905. [Lu et al., 2022] Lu, C., Kuba, J., Letcher, A., Metz, L., Schro...
2025 arXiv
-
[14]
Hartmann, M. (2023). Quantum policy gradient algorithm with optimized action decoding. In International Conference on Machine Learning , pages 24592–24613. PMLR. [Muldal et al., 2019] Muldal, A., Doron, Y ., Aslanides, J., Harley, T., Ward, T., and Liu, S. (2019). dm env: A py...
2023
-
[15]
Dunjko, V . (2023a). Equivariant quantum circuits for learning on weighted graphs. npj Quantum Information , 9(1):47. [Skolik et al., 2022] Skolik, A., Jerbi, S., and Dunjko, V . (2022). Quantum agents in the gym: a variational quantum algorithm for deep q-learning. Quantum, 6...
2023 arXiv
-
[16]
Zhang, M., Su, Y ., Su, H., and Zhu, J. (2022). Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1–6. [Wu et al., 2025] Wu, S., Jin, S., Wen, D., Han, D., and Wang, X. (2025). Quantum reinforcement learning in co...
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.