{"id":"2ce5ad12-a303-4f17-a943-aac2da102efa","arxiv_id":"2505.03778","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dragonfly is a modular open-source deep RL library with JSON-based configuration, parallel-environment bootstrapping, and built-in state representation learning, and it reports favorable benchmark performance.","lead":"This paper presents Dragonfly, an open-source deep reinforcement learning library that uses JSON files to configure and swap agents, networks, and training components. It is aimed at CPU-intensive simulation users such as flow control researchers who need to run many parallel environment workers with minimal code changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark section does not substantiate the 'favorable vs literature' claim: no baselines, seeds, or hyperparameters are reported.","rationale":"The reader's weakest_assumption identifies precisely the gap I find most load-bearing: Section 4's benchmark protocol lacks hyperparameters, seed averaging, baseline runs from other libraries, and configuration files. The central claim is an empirical performance comparison, and the paper provides no operational definition of 'compares favorably with the literature.' The figures are single curves over transitions, with no error bars and no reference curves. In deep RL, a single training run is not a reliable indicator of algorithm performance, and missing hyperparameters make the results non-reproducible and potentially non-representative. The library features described in Sections 2 and 3—modular JSON serialization, factories, parallel bootstrapping, SRL, separable environments—are concrete engineering contributions and are not called into question by this concern. The issue is limited to the benchmark evidence for the headline performance claim. The reader's CONDITIONAL verdict is appropriate: the claim could be supported by adding the missing comparison protocol, but as presented it is not established. I therefore keep the verdict unchanged rather than moving to accept or reject, and I agree with the reader's identification of the weakest assumption.","tokens_in":5899,"tokens_out":1931,"duration_ms":22783,"concrete_test":"Reproduce one representative comparison, e.g., HalfCheetah-v4 with PPO: run Dragonfly and Stable-Baselines3 (or CleanRL) using identical network sizes, learning rates, batch sizes, and other hyperparameters, for at least 5 seeds each, and report mean ± standard error learning curves and final performance. Publish the exact JSON configs, the commit hash of Dragonfly, and the environment wrapper options used. If Dragonfly's curves fall within the noise band of the reference implementation and the configs match standard practice, the 'compares favorably' claim is supported; otherwise it should be qualified or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and introduction claim that Dragonfly's performance on standard agents and common benchmarks 'compares favorably with the literature,' but Section 4 presents only raw learning curves (Figures 6–8) with no quantitative comparison to any reference implementation. No curves from CleanRL, Stable-Baselines3, or the original algorithm papers are overlaid, and no aggregate statistics such as mean final return over seeds or confidence intervals are given. RL is notoriously sensitive to hyperparameters, environment wrappers, reward scaling, and termination conditions; without the exact JSON configuration files, the number of seeds, the random seed values, and the wall-clock or transition budgets, the reader cannot determine whether the plotted curves are typical, best-of-many, or produced under nonstandard settings. Figure 2 also illustrates that the library includes a bootstrapping termination method that changes the effective on-policy assumption, yet the benchmark section does not state whether this or other non-default features were used in Figures 6–8. Because the central claim is explicitly a performance comparison, the absence of a defined comparison protocol is load-bearing: the claim is not false, but it is currently unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Dragonfly, an open-source deep reinforcement learning library built around a JSON-based configuration system, a factory pattern, and SimpleNamespace-based parameter handling. It implements standard agents (PPO, DQN, DDPG, TD3, SAC) and claims a modular architecture that eases experimentation. The library also features parallel-environment bootstrapping to preserve on-policiness under many parallel workers, a state representation learning module (PCA and auto-encoder), and a separable-environment trainer that decomposes high-dimensional action spaces into independent 1-D subproblems. The authors evaluate the library on three benchmark families (gymnasium, MuJoCo, and their own beacon flow-control benchmark) and report learning curves in Figures 6-8. The central claim, stated in the abstract and introduction, is that Dragonfly's performance on standard agents and common benchmarks 'compares favorably with the literature.'","tokens_in":6087,"tokens_out":2951,"duration_ms":29970,"significance":"If the performance claim were properly substantiated, Dragonfly could be a useful open-source alternative to CleanRL and Stable-Baselines3, particularly for CPU-intensive simulation environments where parallel bootstrapping and state representation learning are relevant. The modular design, the inclusion of code snippets for the factory pattern, and the explicit handling of JSON-driven swapping and parameter sweeps are genuinely useful contributions for the RL engineering community. The paper also demonstrates that the agents do learn on the tested environments, which is a minimum requirement for a working library. However, the significance of the performance claim is currently limited by the lack of a quantitative comparison against reference implementations. The paper's strengths are its reproducible architecture description, concrete code examples, and the demonstrated functionality of advanced features such as bootstrapped parallel environments and separable action spaces.","major_comments":[{"comment":"The central claim of the abstract and introduction—that Dragonfly's performance 'compares favorably with the literature'—is not supported by the evidence presented. Figures 6-8 show only Dragonfly's own learning curves, with no overlaid baseline curves from CleanRL, Stable-Baselines3, or the original algorithm papers. There are no aggregate statistics such as mean final return over seeds, confidence intervals, or training budgets. Without any quantitative comparison to existing implementations, the claim as stated fails; the paper needs to define a comparison protocol and present baseline runs under identical conditions.","section":"Section 4, Figures 6-8"},{"comment":"The reproducibility of the benchmark results is insufficient. The paper does not report the exact hyperparameters, JSON configuration files, environment wrappers, termination settings, number of seeds, random seed values, or transition budgets used in Figures 6-8. Since deep RL is highly sensitive to these choices, the reader cannot verify whether the plotted curves are typical runs, best-of-many runs, or produced under non-standard settings. At minimum, the paper should provide the configuration files and seed-averaged results with variance or error bars.","section":"Section 4, benchmark protocol"},{"comment":"The bootstrapping termination technique described in Section 3.4 modifies the on-policy property of PPO and is presented as a non-default feature. The benchmark section does not state whether this feature was enabled in the runs that produced Figures 6-8. If it was used, the comparison to standard literature baselines would be confounded; if it was not used, the impact of the feature on the benchmark results is unclear. The authors should explicitly disclose which library features were active for each reported curve.","section":"Section 3.4 and Section 4"},{"comment":"The state representation learning module introduces a free parameter, the latent space dimension, that strongly affects performance: Figure 4a shows that a dimension of 10 leads to no learning at all, while a dimension of 300 outperforms the standard agent. The paper suggests using explained variance to choose this dimension, but this guidance is informal and is demonstrated only on the authors' own beacon environment. For a general-purpose library claim, the authors should either provide a principled selection rule or report sensitivity analyses on the standard benchmarks.","section":"Section 3.5, Figure 4"}],"minor_comments":[{"comment":"The word 'avaiable' in the description of the buffer is a typo and should be 'available'.","section":"Section 3.3"},{"comment":"The note 'the reward is not computed in the same manner as in [10]' is vague; the authors should specify exactly how the reward computation differs from the referenced work.","section":"Figure 5 caption"},{"comment":"The word 'litterature' is a typo; it should be 'literature'.","section":"Section 1"},{"comment":"The factory class code snippet catches its own ValueError and then re-raises a new ValueError with a custom message; this pattern is confusing and could be simplified to a single raise statement.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The performance comparison is presented as a central contribution of the paper, but the current manuscript does not include any baseline runs or seed averaging, which is a serious omission for a machine-learning software paper. The inclusion of the authors' own beacon benchmark is not inherently problematic, but it should be clearly separated from the claim of comparison with the broader literature. The paper would be strengthened by adding a rigorous benchmarking section with baselines from established libraries, a clear statement of the experimental setup, and public configuration files. The library itself appears functional and the modular design is interesting, so the result is not fatally flawed; the benchmarking needs to be brought up to the standard of the field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if someone treats this as a library release with demos, it's fine; if they treat the abstract's 'compares favorably with the literature' as a measured claim, it isn't supported. The code is real, the modular design is sensible, and the bootstrapping trick, SRL, and separable-env trainer are useful additions for CPU-heavy flow-control RL. Credit where due: features are attributed to earlier papers by the same group, and the library is open source.\n\nThe soft spot is the benchmark section. Figures 6-8 show learning curves for standard agents on gym/MuJoCo/beacon, but there are no curves from CleanRL, stable-baselines3, or original algorithm papers overlaid; no seed averaging or confidence intervals; no hyperparameters, wrappers, or termination settings; no configuration files or commit hash. Without these, the reader cannot tell if the runs are typical or cherry-picked. The stress-test note is right: the claim is not false, just unsupported by the evidence in the paper. The bootstrapping feature also changes the effective data distribution, and the benchmark doesn't say whether it was enabled in Figures 6-8.\n\nI would not call this a fatal flaw. For a library paper, the value is in the tooling and the feature demonstrations, and the curves at least show the agents learn on these environments. If the authors reframe the abstract claim as 'these agents achieve reasonable performance' and add a short benchmark protocol with seeds and configs, the paper would be solid. As is, it deserves a serious referee, but the referee should push for a real comparison.\n\nI'd bring it to a reading group only if someone there works on flow-control RL; otherwise it's a skim. I wouldn't cite it in my own work unless I adopted the library.","headline":"A solid modular RL library whose abstract overclaims a benchmark comparison the paper never actually performs.","tokens_in":6600,"tokens_out":1839,"would_cite":false,"duration_ms":19614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modular deep reinforcement learning library claims that JSON-driven configuration plus a factory pattern makes experimentation fast, and that its standard agents perform competitively on common benchmarks.","keywords":["deep reinforcement learning","modular library","JSON serialization","factory pattern","benchmark comparison","state representation learning","parallel bootstrapping","separable environments"],"falsifier":"Run each reported agent with a fixed transition budget across multiple random seeds (say 5 to 10) on the same environments, using an independent reference implementation with default hyperparameters; then compare seed-averaged learning curves. If Dragonfly's curves lie clearly outside the reference range on any task, or if its scores do not reach published values when settings match standard defaults, the favorable-comparison claim is false. The paper's curves contain no error bars and no baseline runs, so the comparison is currently unfalsified rather than confirmed.","tokens_in":5701,"feed_emoji":"🧩","tokens_out":7276,"duration_ms":70314,"temperature":0.7,"pith_summary":"Dragonfly is a deep reinforcement learning library built around a simple claim: if every component — agent, policy, value network, loss, optimizer, environment — is registered in a string-keyed factory and configured through a single JSON file, then experimental work reduces to editing that file instead of maintaining code. On top of this architecture the library provides specialised tools for CPU-heavy numerical simulations: parallel-environment bootstrapping that preserves on-policy training beyond the usual limit, built-in state-representation learning (PCA and an autoencoder), and a separable-environment trainer that splits an $n$-action control problem into $n$ one-action problems. The paper reports learning curves for PPO, DQN, DDPG, TD3 and SAC on standard benchmark environments and concludes that Dragonfly's performance compares favorably with the literature. If that performance claim holds, the library is a practical open-source alternative for researchers who want quick ablations and parameter sweeps without rewriting code.","feed_headline":"One JSON file swaps RL agents, networks, and losses","feed_subtitle":"The paper says the modular design makes parameter sweeps and CPU-heavy simulation experiments fast to set up and reuse.","key_machinery":"The load-bearing mechanism is the factory pattern driven by JSON serialization: every class registers a constructor under a string key, and a .json configuration file, parsed into SimpleNamespace objects, selects which key to instantiate for each object type. This makes configuration the only interface between the user and the algorithm, so changing an agent, loss, or network is a data-file edit rather than a code edit. A second named mechanism is bootstrapping termination, which handles the broken on-policy assumption when many parallel environments are unrolled; the paper shows that with this technique performance is maintained up to 16 parallel environments and only slightly degrades up to 32.","core_discovery":"On the paper's own terms, the central discovery is architectural: RL experimentation can be reorganised around JSON-serialised configuration plus a factory pattern, so swapping algorithmic building blocks requires no code changes. Each object type (agents, losses, networks, buffers, trainers) has its own string-keyed registry, and a .json file determines which block is instantiated; the same file can drive parameter sweeps. The paper further claims that on standard gym-style, physics-based, and flow-control benchmarks the included PPO, DQN, DDPG, TD3 and SAC agents 'compare favorably with the literature,' and that three CPU-oriented features — parallel bootstrap termination, PCA/autoencoder state-representation learning, and separable-environment training — are either necessary or beneficial for simulation-heavy use cases.","pith_inferences":["The paper does not run a same-seed, same-budget head-to-head against reference implementations; that comparison would convert 'compares favorably with the literature' from a claim into a measurement.","Because all configuration lives in JSON files, Dragonfly could naturally emit a reproducibility artifact for every run; the paper does not describe such auditability, but it follows directly from the architecture.","The separation of the trainer from the agent suggests that future algorithms could be added without touching existing agents; the paper mentions this flexibility but does not demonstrate a third-party algorithm integration.","If SRL's benefit generalises beyond flow control, PCA-style preprocessing could become a default preprocessing step for high-dimensional physical-control tasks; the paper's evidence is a single environment, so this remains speculative."],"forward_implications":["With the JSON factory design, a user can change algorithm components and run parameter sweeps by editing one file, which makes ablations and experiment reuse cheaper.","Parallel bootstrapping lets CPU-bound on-policy agents train with 4 to 8 times more simultaneous environments than the vanilla limit, reducing wall-clock time for simulations.","Built-in PCA or autoencoder representation learning can condense high-dimensional observations and, in the reported flow-control case, outperform an agent fed raw observations.","The separable-environment trainer converts one environment with $n$ actions into $n$ environments with one action, increasing sample count per action and stabilising learning for high-dimensional control.","If performance is as reported, Dragonfly is a credible open-source production alternative to existing RL libraries for standard benchmarks."],"supporting_citations":[{"why":"Contrast target for the modularity claim: single-file RL implementations that Dragonfly positions itself against, motivating the factory/JSON design.","marker":"[6]"},{"why":"Another reference point for reliable RL implementations; cited to distinguish Dragonfly's modular pattern from scalable single-file agents.","marker":"[7]"},{"why":"Supplies the parallel bootstrapping termination technique that Dragonfly uses to keep on-policy training valid with 4-8x more parallel environments.","marker":"[8]"},{"why":"Supplies the flow-control benchmark environments (e.g., shkadov-v0) used to test parallel bootstrapping, state-representation learning, and separable environments.","marker":"[9]"},{"why":"Source of the separable-environment approach that splits an n-action environment into n one-action environments.","marker":"[10]"},{"why":"Provides the standard gym-style benchmark environments used in the first set of performance comparisons.","marker":"[11]"},{"why":"Provides the physics-based continuous-control benchmark environments used in the second set of performance comparisons.","marker":"[12]"}],"fun_headline_variants":["JSON config swaps RL blocks without code edits","One JSON file remixes agents, losses, networks in RL","RL building blocks swap via JSON, no code changes needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance claim assumes that the reported curves were produced with standard hyperparameters, wrappers, and terminations rather than favorable runs; nothing in the paper verifies that, because no baselines, seed averages, or config files are shown.","fun_headline_variants_meta":{"raw":{"variants":["JSON config swaps RL blocks without code edits","One JSON file remixes agents, losses, networks in RL","RL building blocks swap via JSON, no code changes needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000848,"raw_usage":{"total_tokens":3587,"prompt_tokens":744,"completion_tokens":2843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":360,"completion_tokens_details":{"reasoning_tokens":2802}},"tokens_in":360,"tokens_out":2843,"duration_ms":18318,"temperature":1.0,"reasoning_tokens":2802,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:00:08.334456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each reported agent with a fixed transition budget across multiple random seeds (say 5 to 10) on the same environments, using an independent reference implementation with default hyperparameters; then compare seed-averaged learning curves. If Dragonfly's curves lie clearly outside the reference range on any task, or if its scores do not reach published values when settings match standard defaults, the favorable-comparison claim is false. The paper's curves contain no error bars and no baseline runs, so the comparison is currently unfalsified rather than confirmed.","supporting_citations":[{"cited_title":"Huang, R","cited_arxiv_id":null,"evidence_quote":"Contrast target for the modularity claim: single-file RL implementations that Dragonfly positions itself against, motivating the factory/JSON design."},{"cited_title":"Raffin, A","cited_arxiv_id":null,"evidence_quote":"Another reference point for reliable RL implementations; cited to distinguish Dragonfly's modular pattern from scalable single-file agents."},{"cited_title":"Viquerat and E","cited_arxiv_id":null,"evidence_quote":"Supplies the parallel bootstrapping termination technique that Dragonfly uses to keep on-policy training valid with 4-8x more parallel environments."},{"cited_title":"Viquerat, P","cited_arxiv_id":null,"evidence_quote":"Supplies the flow-control benchmark environments (e.g., shkadov-v0) used to test parallel bootstrapping, state-representation learning, and separable environments."},{"cited_title":"Belus, J","cited_arxiv_id":null,"evidence_quote":"Source of the separable-environment approach that splits an n-action environment into n one-action environments."},{"cited_title":"Towers, A","cited_arxiv_id":null,"evidence_quote":"Provides the standard gym-style benchmark environments used in the first set of performance comparisons."},{"cited_title":"Todorov, T","cited_arxiv_id":null,"evidence_quote":"Provides the physics-based continuous-control benchmark environments used in the second set of performance comparisons."}],"review_version":1}