Dragonfly: a modular deep reinforcement learning library

Amirhossein Bateni; Elie Hachem; Jonathan Viquerat; Paul Garnier

REVIEW 4 major objections 4 minor 12 references

Dragonfly: a modular deep reinforcement learning library

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A modular deep reinforcement learning library claims that JSON-driven configuration plus a factory pattern makes experimentation fast, and that its standard agents perform competitively on common benchmarks.

desk verdict A solid modular RL library whose abstract overclaims a benchmark comparison the paper never actually performs. read the letter →

arxiv 2505.03778 v2 pith:7QQNY3IN submitted 2025-04-30 cs.LG

Jonathan Viquerat , Paul Garnier , Amirhossein Bateni , Elie Hachem This is my paper · ORCID

classification cs.LG

keywords deepreinforcementlearningmodularlibraryJSONserializationfactorypatternbenchmarkcomparisonstaterepresentationparallelbootstrappingseparableenvironments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dragonfly is a deep reinforcement learning library built around a simple claim: if every component — agent, policy, value network, loss, optimizer, environment — is registered in a string-keyed factory and configured through a single JSON file, then experimental work reduces to editing that file instead of maintaining code. On top of this architecture the library provides specialised tools for CPU-heavy numerical simulations: parallel-environment bootstrapping that preserves on-policy training beyond the usual limit, built-in state-representation learning (PCA and an autoencoder), and a separable-environment trainer that splits an $n$-action control problem into $n$ one-action problems. The paper reports learning curves for PPO, DQN, DDPG, TD3 and SAC on standard benchmark environments and concludes that Dragonfly's performance compares favorably with the literature. If that performance claim holds, the library is a practical open-source alternative for researchers who want quick ablations and parameter sweeps without rewriting code.

What carries the argument

The load-bearing mechanism is the factory pattern driven by JSON serialization: every class registers a constructor under a string key, and a .json configuration file, parsed into SimpleNamespace objects, selects which key to instantiate for each object type. This makes configuration the only interface between the user and the algorithm, so changing an agent, loss, or network is a data-file edit rather than a code edit. A second named mechanism is bootstrapping termination, which handles the broken on-policy assumption when many parallel environments are unrolled; the paper shows that with this technique performance is maintained up to 16 parallel environments and only slightly degrades up to 32.

What would settle it

Run each reported agent with a fixed transition budget across multiple random seeds (say 5 to 10) on the same environments, using an independent reference implementation with default hyperparameters; then compare seed-averaged learning curves. If Dragonfly's curves lie clearly outside the reference range on any task, or if its scores do not reach published values when settings match standard defaults, the favorable-comparison claim is false. The paper's curves contain no error bars and no baseline runs, so the comparison is currently unfalsified rather than confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is architectural: RL experimentation can be reorganised around JSON-serialised configuration plus a factory pattern, so swapping algorithmic building blocks requires no code changes. Each object type (agents, losses, networks, buffers, trainers) has its own string-keyed registry, and a .json file determines which block is instantiated; the same file can drive parameter sweeps. The paper further claims that on standard gym-style, physics-based, and flow-control benchmarks the included PPO, DQN, DDPG, TD3 and SAC agents 'compare favorably with the literature,' and that three CPU-oriented features — parallel bootstrap termination, PCA/autoencoder state-representation learning, and separable-environment training — are either necessary or beneficial for simulation-heavy use cases.

Load-bearing premise

The performance claim assumes that the reported curves were produced with standard hyperparameters, wrappers, and terminations rather than favorable runs; nothing in the paper verifies that, because no baselines, seed averages, or config files are shown.

Editorial extensions

If this is right

With the JSON factory design, a user can change algorithm components and run parameter sweeps by editing one file, which makes ablations and experiment reuse cheaper.
Parallel bootstrapping lets CPU-bound on-policy agents train with 4 to 8 times more simultaneous environments than the vanilla limit, reducing wall-clock time for simulations.
Built-in PCA or autoencoder representation learning can condense high-dimensional observations and, in the reported flow-control case, outperform an agent fed raw observations.
The separable-environment trainer converts one environment with $n$ actions into $n$ environments with one action, increasing sample count per action and stabilising learning for high-dimensional control.
If performance is as reported, Dragonfly is a credible open-source production alternative to existing RL libraries for standard benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not run a same-seed, same-budget head-to-head against reference implementations; that comparison would convert 'compares favorably with the literature' from a claim into a measurement.
Because all configuration lives in JSON files, Dragonfly could naturally emit a reproducibility artifact for every run; the paper does not describe such auditability, but it follows directly from the architecture.
The separation of the trainer from the agent suggests that future algorithms could be added without touching existing agents; the paper mentions this flexibility but does not demonstrate a third-party algorithm integration.
If SRL's benefit generalises beyond flow control, PCA-style preprocessing could become a default preprocessing step for high-dimensional physical-control tasks; the paper's evidence is a single environment, so this remains speculative.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: if someone treats this as a library release with demos, it's fine; if they treat the abstract's 'compares favorably with the literature' as a measured claim, it isn't supported. The code is real, the modular design is sensible, and the bootstrapping trick, SRL, and separable-env trainer are useful additions for CPU-heavy flow-control RL. Credit where due: features are attributed to earlier papers by the same group, and the library is open source.

The soft spot is the benchmark section. Figures 6-8 show learning curves for standard agents on gym/MuJoCo/beacon, but there are no curves from CleanRL, stable-baselines3, or original algorithm papers overlaid; no seed averaging or confidence intervals; no hyperparameters, wrappers, or termination settings; no configuration files or commit hash. Without these, the reader cannot tell if the runs are typical or cherry-picked. The stress-test note is right: the claim is not false, just unsupported by the evidence in the paper. The bootstrapping feature also changes the effective data distribution, and the benchmark doesn't say whether it was enabled in Figures 6-8.

I would not call this a fatal flaw. For a library paper, the value is in the tooling and the feature demonstrations, and the curves at least show the agents learn on these environments. If the authors reframe the abstract claim as 'these agents achieve reasonable performance' and add a short benchmark protocol with seeds and configs, the paper would be solid. As is, it deserves a serious referee, but the referee should push for a real comparison.

I'd bring it to a reading group only if someone there works on flow-control RL; otherwise it's a skim. I wouldn't cite it in my own work unless I adopted the library.

Referee Report

4 major / 4 minor

Summary. The paper introduces Dragonfly, an open-source deep reinforcement learning library built around a JSON-based configuration system, a factory pattern, and SimpleNamespace-based parameter handling. It implements standard agents (PPO, DQN, DDPG, TD3, SAC) and claims a modular architecture that eases experimentation. The library also features parallel-environment bootstrapping to preserve on-policiness under many parallel workers, a state representation learning module (PCA and auto-encoder), and a separable-environment trainer that decomposes high-dimensional action spaces into independent 1-D subproblems. The authors evaluate the library on three benchmark families (gymnasium, MuJoCo, and their own beacon flow-control benchmark) and report learning curves in Figures 6-8. The central claim, stated in the abstract and introduction, is that Dragonfly's performance on standard agents and common benchmarks 'compares favorably with the literature.'

Significance. If the performance claim were properly substantiated, Dragonfly could be a useful open-source alternative to CleanRL and Stable-Baselines3, particularly for CPU-intensive simulation environments where parallel bootstrapping and state representation learning are relevant. The modular design, the inclusion of code snippets for the factory pattern, and the explicit handling of JSON-driven swapping and parameter sweeps are genuinely useful contributions for the RL engineering community. The paper also demonstrates that the agents do learn on the tested environments, which is a minimum requirement for a working library. However, the significance of the performance claim is currently limited by the lack of a quantitative comparison against reference implementations. The paper's strengths are its reproducible architecture description, concrete code examples, and the demonstrated functionality of advanced features such as bootstrapped parallel environments and separable action spaces.

major comments (4)

[Section 4, Figures 6-8] The central claim of the abstract and introduction—that Dragonfly's performance 'compares favorably with the literature'—is not supported by the evidence presented. Figures 6-8 show only Dragonfly's own learning curves, with no overlaid baseline curves from CleanRL, Stable-Baselines3, or the original algorithm papers. There are no aggregate statistics such as mean final return over seeds, confidence intervals, or training budgets. Without any quantitative comparison to existing implementations, the claim as stated fails; the paper needs to define a comparison protocol and present baseline runs under identical conditions.
[Section 4, benchmark protocol] The reproducibility of the benchmark results is insufficient. The paper does not report the exact hyperparameters, JSON configuration files, environment wrappers, termination settings, number of seeds, random seed values, or transition budgets used in Figures 6-8. Since deep RL is highly sensitive to these choices, the reader cannot verify whether the plotted curves are typical runs, best-of-many runs, or produced under non-standard settings. At minimum, the paper should provide the configuration files and seed-averaged results with variance or error bars.
[Section 3.4 and Section 4] The bootstrapping termination technique described in Section 3.4 modifies the on-policy property of PPO and is presented as a non-default feature. The benchmark section does not state whether this feature was enabled in the runs that produced Figures 6-8. If it was used, the comparison to standard literature baselines would be confounded; if it was not used, the impact of the feature on the benchmark results is unclear. The authors should explicitly disclose which library features were active for each reported curve.
[Section 3.5, Figure 4] The state representation learning module introduces a free parameter, the latent space dimension, that strongly affects performance: Figure 4a shows that a dimension of 10 leads to no learning at all, while a dimension of 300 outperforms the standard agent. The paper suggests using explained variance to choose this dimension, but this guidance is informal and is demonstrated only on the authors' own beacon environment. For a general-purpose library claim, the authors should either provide a principled selection rule or report sensitivity analyses on the standard benchmarks.

minor comments (4)

[Section 3.3] The word 'avaiable' in the description of the buffer is a typo and should be 'available'.
[Figure 5 caption] The note 'the reward is not computed in the same manner as in [10]' is vague; the authors should specify exactly how the reward computation differs from the referenced work.
[Section 1] The word 'litterature' is a typo; it should be 'literature'.
[Section 2.2] The factory class code snippet catches its own ValueError and then re-raises a new ValueError with a custom message; this pattern is confusing and could be simplified to a single raise statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Dragonfly's performance claim is empirical and not derived from its inputs.

full rationale

The paper is a software library description, not a derivation chain. The central claim ('Its performance on standard agents using common benchmarks compares favorably with the literature') is supported, or intended to be supported, by direct learning curves in Section 4 (Figures 6-8). These curves are measurements of agent returns over transitions, not predictions obtained by fitting parameters to target values, so they cannot reduce to their inputs by construction. The bootstrapping termination and separable-environment features are explicitly reproduced from prior work by the same authors ([8], [10]) and the benchmark suite includes the authors' beacon library ([9]), but these are implemented features and one of three benchmark families; they are not premises from which the performance claim is derived. The main weakness is evidentiary rather than circular: Section 4 reports no reference-library curves, no seed averages, no confidence intervals, and no configuration files, so the 'compares favorably with the literature' claim is not actually substantiated in the text. That is a reproducibility/comparability problem, not circularity. No load-bearing argument in the paper is equivalent to its own inputs by definition or by self-citation.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central performance claim depends on undisclosed hyperparameters and on the correctness of TensorFlow and the standard RL algorithms. The bootstrapping and separability features are taken from self-cited prior work, so they are not independently re-derived here. No new physical entities are introduced.

free parameters (2)

Agent hyperparameters for benchmarks = Not reported
Learning rates, batch sizes, and other algorithm settings used in Figures 6-8 are not given, making the favorable comparison claim non-reproducible.
SRL latent space dimension = 300 (example in Figure 4)
For the ppo-pca demonstration, latent space dimension of 300 is chosen by the authors based on explained variance; other dimensions give worse performance.

assumptions (2)

domain assumption TensorFlow 2 provides correct gradients and training primitives
The entire training pipeline depends on TensorFlow behavior; this is not proven in the paper.
domain assumption Standard RL algorithm definitions (PPO, DQN, etc.) as implemented in Dragonfly match canonical versions
No pseudocode or formal specification is given; the favorable comparison assumes the implementations are equivalent to reference implementations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dragonfly: a modular deep reinforcement learning library." pith.science (2026). https://pith.science/paper/7QQNY3IN

@misc{pith2026250503778,
  author       = {Pith},
  title        = {Pith review of: Dragonfly: a modular deep reinforcement learning library},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QQNY3IN}},
  note         = {Machine review of arXiv:2505.03778}
}

read the original abstract

Dragonfly is a deep reinforcement learning library focused on modularity, in order to ease experimentation and developments. It relies on a json serialization that allows to swap building blocks and perform parameter sweep, while minimizing code maintenance. Some of its features are specifically designed for CPU-intensive environments, such as numerical simulations. Its performance on standard agents using common benchmarks compares favorably with the literature.

Figures

Figures reproduced from arXiv: 2505.03778 by the authors.

**Figure 1.** General architecture of the library. 2.2 Implementation details The backbone of the present library is composed of three components: (i) a .json serialization of the different components, (ii) the use of SimpleNamespace instances to store the parameters of each object, and (iii) a factory pattern. The .json parsing relies on the standard json module, and exploits the SimpleNamespace iterablebased constructor to pro… view at source ↗

**Figure 2.** Score curves obtained for different number of parallel environments on the shkadov-v0 environment from the beacon benchmark [9]. The agent is updated with 4 full trajectories. (Top) With no bootstrapping, the learning rate and the final performance decrease significantly beyond 4 parallel environemnts (Bottom) With bootstrapping, performance is maintained exactly up to 16 parallel environments, and with a very slig… view at source ↗

**Figure 3.** State representation learning setup in the DRL learning loop. The traditional observation feedback is replaced by a modified feedback loop, in which observations are transformed in a latent representation before being fed to the agent. representation learning aims to process these observations upstream of the DRL agent, by learning a modified or a low-dimensional representation of the original data, called latent re… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Comparing ppo with ppo-pca on the shkadov-v0 environment. (Top left) The ppopca overperforms the standard ppo algorithm for an observation vector of size 1000 corresponding to the entire information of the considered environment. Performance varies based on the chose …

**Figure 5.** Figure 5: Comparing traditional ppo with separable ppo on the shkadov-v0 environment with 5−, 10− and 20− dimensional action spaces. Note that the reward is not computed in the same manner as in [10]. Doing so, the problem is simplified by (i) reducing the observation space dime…

**Figure 6.** Figure 6: Performance comparison on the gym benchmark. 4 Performance and benchmarks The agents implementations are tested on three different litterature benchmarks: (i) the gymnasium environments [11], (ii) the mujoco benchmark [12], and (iii) the beacon benchmark [9], the latte…

**Figure 7.** Figure 7: Performance comparison on the mujoco benchmark. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Performance comparison on the beacon benchmark. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2018

work page 2018
[2]

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602, 2013

arXiv 2013
[3]

Silver, J

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge.Nature, 550, 2017

work page 2017
[4]

Pinto, M

L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning.arXiv preprint arXiv:1710.06542, 2017

arXiv 2017
[5]

Kendall, J

A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and A. Shah. Learning to drive in a day.arXiv preprint arXiv:1807.00412, 2018

arXiv 2018
[6]

Huang, R

S. Huang, R. F. J. Dossa, C. Ye, and J. Braga. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms, 2021

work page 2021
[7]

Raffin, A

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Re- liable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1– 8, 2021

work page 2021
[8]

Viquerat and E

J. Viquerat and E. Hachem. Parallel bootstrap-based on-policy deep reinforcement learning for continuous flow control applications, 2023

work page 2023

Show all 12 references

[9]

Viquerat, P

J. Viquerat, P. Meliga, P. Jeken-Rico, and E. Hachem. Beacon, a lightweight deep reinforcement learning benchmark library for flow control.Applied Sciences, 14(9), 2024

2024
[10]

Belus, J

V. Belus, J. Rabault, J. Viquerat, Z. Che, E. Hachem, and U. Reglade. Exploiting locality and translationalinvariancetodesigneffectivedeepreinforcementlearningcontrolofthe1-dimensional unstable falling liquid film.AIP Advances, 9:125014, 2019

2019
[11]

Towers, A

M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, H. Tan, and O. G. Younis. Gymnasium: A standard interface for reinforcement learning environments, 2024

2024
[12]

Todorov, T

E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. 10

2012

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

R. S. Sutton and A. G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2018

work page 2018

[2] [2]

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602, 2013

arXiv 2013

[3] [3]

Silver, J

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge.Nature, 550, 2017

work page 2017

[4] [4]

Pinto, M

L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning.arXiv preprint arXiv:1710.06542, 2017

arXiv 2017

[5] [5]

Kendall, J

A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J.-M. Allen, V.-D. Lam, A. Bewley, and A. Shah. Learning to drive in a day.arXiv preprint arXiv:1807.00412, 2018

arXiv 2018

[6] [6]

Huang, R

S. Huang, R. F. J. Dossa, C. Ye, and J. Braga. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms, 2021

work page 2021

[7] [7]

Raffin, A

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Re- liable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1– 8, 2021

work page 2021

[8] [8]

Viquerat and E

J. Viquerat and E. Hachem. Parallel bootstrap-based on-policy deep reinforcement learning for continuous flow control applications, 2023

work page 2023