{"id":"b7c90d64-e45e-4dba-ab25-44149e154bfd","arxiv_id":"1908.04127","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A survey of deep reinforcement learning for fluid mechanics, plus a new result where DRL agents recover the same optimal control-cylinder positions as an adjoint method on a square-cylinder flow.","lead":"This paper reviews how deep reinforcement learning is used to control and optimize fluid flows, and it adds a new demonstration on a square-cylinder test case. A generalist reader might care because it shows a learning-based method matching classical adjoint optimization on a laminar flow benchmark, with code released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.7's reward is a 17-step (or single-action) drag coefficient; without evidence that this transient reward has the same argmax as the steady drag used by the adjoint reference, the 'same optimal positions' claim is not established.","rationale":"I read the paper as a review with an illustrative new experimental section; the review portion is informative, and the authors provide code, which is a genuine reproducibility asset. The strongest and most novel claim is the Section 3.7 comparison between DRL and the adjoint-based results of Meliga et al. (2014). The load-bearing assumption is that the DRL reward, built from drag coefficients evaluated after 17 time steps or after a single action, has the same optimizer as the steady or time-averaged drag that the adjoint method targets. The paper never documents the simulation horizon in physical time, nor does it compare the short-horizon reward with a converged drag value at the same positions. Without that, the reported agreement could reflect an accidental alignment of transient objectives rather than a genuine match with the adjoint optimum. This concern is internal to the new result and is not a matter of disagreeing with consensus; it is a question of whether the reward signal measures what the comparison claims to measure. The proposed grid test would settle the issue directly by comparing the two objective landscapes. Because the concern is real but not disqualifying for the paper as a whole, the appropriate action remains conditional acceptance pending clarification, which matches the reader's verdict.","tokens_in":17794,"tokens_out":4070,"duration_ms":47851,"concrete_test":"Use the provided fenics-DRL code to evaluate the reward and the long-time averaged drag at a grid of control-cylinder positions (e.g., 9x9 covering the reported optimum region) for Re=40 and Re=100. For each position, compute (i) the original short-horizon reward as in training and (ii) the time-averaged CD over a fully developed periodic state after the initial transient has decayed. If the argmax of the long-time averaged drag differs from the DRL-reported positions or from the short-horizon reward argmax, the central comparison fails. Additionally, rerun training with 3-5 seeds and report the spread of final positions; if the reported positions are within seed-to-seed variability of the adjoint optimum, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The new result in Section 3.7 rests on the reward rt = C0_D - C_D being a faithful proxy for the objective optimized in Meliga et al. (2014). The adjoint reference computes sensitivities of aerodynamic forces in the established (steady or time-periodic) flow, whereas the DRL reward is evaluated over episodes of 17 time steps (A3C agents 1.1/2.1/3.1) or a single action (PPO agents 1.2/2.2/3.2). The paper does not report the physical time step, the horizon in convective time units, or any check that the drag sampled at the end of an episode coincides with the quasi-steady drag at the proposed cylinder position. After a sudden displacement of the control cylinder, the force on both cylinders evolves over a transient set by vortex convection; a reward based on the first 17 steps can advantage positions whose favorable effect appears early but vanishes once the wake reorganizes. Because the agent is trained to maximize this truncated-horizon reward rather than steady drag, agreement of the resulting optima with adjoint optima is not guaranteed. For the sequential agents, the reward is also history-dependent, since the cylinder moves every step, so per-step CD is not a pointwise function of the current position. The claim 'the same optimal positions were found' is stated without error bars, number of seeds, or a quantitative comparison of the full reward landscape. The available code would allow the needed check, but the manuscript alone does not establish that the comparison target is the same.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a review of deep reinforcement learning (DRL) applied to fluid mechanics. The first part introduces RL/DRL basics (MDPs, DQN, policy gradients, actor-critic methods, TRPO/PPO), and the second part surveys representative applications: fish swimming and collective swimming, fluid-rigid-body control, microfluidic flow shaping, and active flow control of a cylinder wake. The final application section (§3.7) contains an original experiment in which six DRL agents (A3C and PPO, with and without autoencoder-based state compression) are trained to position a small control cylinder near a square cylinder at Re = 10, 40, and 100. The authors report that the DRL-optimal positions match those obtained by the adjoint-based sensitivity analysis of Meliga et al. (2014), and use this as evidence that DRL can reproduce classical gradient-based optimization results without gradient information.","tokens_in":18124,"tokens_out":7995,"duration_ms":78778,"significance":"The review component is useful and generally accurate: it collects algorithm choices, reward functions, and CFD coupling strategies in one place and is written accessibly for newcomers. The code for §3.7 is made publicly available, which supports reproducibility. If the agreement with the adjoint benchmark is robustly established, the §3.7 result would be a valuable demonstration that DRL can match a classical gradient-based method on a laminar optimization task without using gradients, while also benefiting from transfer learning and parallel multi-environment training. However, the evidence currently supplied is insufficient to establish that specific claim, so the significance of the paper is conditional on additional verification.","major_comments":[{"comment":"The central claim that \"the same optimal positions were found\" depends on the reward r_t = C_D^0 - C_D being a valid proxy for the steady-state drag objective optimized by the adjoint method in Meliga et al. (2014). According to Table 3, agents 1.1/2.1/3.1 act over 17 time steps and agents 1.2/2.2/3.2 take a single action, but the manuscript does not report the physical time step, the horizon in convective time units, or any check that the drag at the end of an episode coincides with the quasi-steady drag at the proposed position. Because the flow past a displaced cylinder relaxes over a vortex-convection time scale, a truncated-horizon reward can advantage positions whose early transient benefit disappears once the wake reorganizes; for the sequential agents the reward is also history-dependent because the cylinder moves at each step. To support the comparison, the authors should evaluate the learned positions with longer rollouts or steady-state drag calculations and show that the transient reward used in training has the same optimum as the steady drag.","section":"§3.7, Table 3"},{"comment":"The agreement with Meliga et al. (2014) is asserted qualitatively. The paper gives no numerical coordinates of the DRL-optimal positions, no drag coefficients at those positions, no distance or tolerance criterion for \"same\", no number of independent training runs, and no error bars. Figure 11 simply shows six final configurations without overlaying the adjoint reference positions. This is a load-bearing omission for the paper's main new result; the authors should provide a quantitative comparison and define the notion of agreement, or downgrade the claim from \"same optimal positions\" to a qualitative demonstration.","section":"§3.7, Figure 11"},{"comment":"No learning curves, reward traces, or convergence diagnostics are reported for any of the six agents, despite the text citing training times and transfer-learning speedups. Since DRL training is stochastic, the absence of multiple seeds and of a reward-convergence check means the reader cannot assess whether the positions in Figure 11 represent converged optima or single trajectories. At minimum, report the reward evolution and repeat the training with at least three seeds for one representative configuration.","section":"§3.7, training convergence"}],"minor_comments":[{"comment":"The citation \"American Institute of Aeronautics and Astronautics, ed. 2008 Morphing Airfoils with Four Morphing Parameters\" is not a citable paper; the underlying conference contribution and authors should be identified.","section":"Table 2 / References"},{"comment":"The mapping between the two architectures and the agent labels is confusing: Table 3 lists agents 1.1/2.1/3.1 as \"Control over 17 time steps\" and agents 1.2/2.2/3.2 as \"Direct optimization\", while the text describes the PPO single-action architecture first and the A3C sequential architecture second. Please make the algorithm/architecture correspondence explicit.","section":"§3.7, Table 3"},{"comment":"There are frequent copyediting issues, including missing spaces between citations (e.g., \"Goodfellow et al. (2017) Strang (2019) Bottou et al. (2018)\"), inconsistent title capitalization in the bibliography, and \"time station\" in Table 1; a careful proofreading pass is needed.","section":"§2.2.1 and throughout"},{"comment":"The closing claim that DRL methods \"proved to be robust when exposed to possible numerical noise\" is stronger than what the reviewed single-realization studies and the new experiment show; please qualify it.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The review portion is solid and publishable on its own; the obstacle is §3.7, where the paper makes a benchmark-validation claim that the reported evidence does not yet support. I would be willing to see a revision with added diagnostics. In addition, Sections 3.5-3.7 are largely based on the authors' own prior work and repositories; this is not improper, but the editor may want the authors to state clearly, relative to Rabault et al. (2019) and Rabault & Kuhnle (2019), what the genuinely new contribution is. The incomplete AIAA reference should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the review half is genuinely useful: it gives a compact, accurate tour of DRL algorithms and of the existing DRL-for-fluid-mechanics literature, including reward shaping and coupling choices. Second, the new result in Section 3.7 is a real experiment but its headline claim is not yet established. The reward is a short-horizon drag coefficient, and the paper never shows that this proxy has the same argmax as the steady drag that Meliga et al.'s adjoint method optimizes.\n\nWhat is new and worth credit: Garnier et al. apply PPO and A3C to place a small control cylinder behind a square cylinder, using autoencoder features and transfer learning across Reynolds numbers, and compare against Meliga et al. (2014). The code is on GitHub, which is a real reproducibility asset. The review's summaries of Rabault et al., Novati et al., Verma et al., and the microfluidics work are accurate and appropriately detailed.\n\nThe soft spots are in the experimental section. The stress-test note is fair: for the sequential A3C agents the episode is 17 time steps; for the PPO agents it is a single action. There is no statement of the physical time step, no convective time horizon, and no check that the drag at the end of an episode is close to quasi-steady. Since moving the cylinder triggers a transient, a position that looks good after 17 steps may not be good once the wake reorganizes. For the sequential agents the per-step drag depends on the path, not just on the current cylinder position, so the reward is not a pointwise function of position. That alone makes \"the same optimal positions were found\" a qualitative claim, not a demonstrated one. On top of that, there are no multiple seeds, no error bars, no convergence diagnostics, and the comparison with Meliga is visual rather than quantitative. The conclusion that DRL \"proved to be robust\" to numerical noise is unsupported by the experiment. None of this kills the paper; it just means the experimental contribution is a proof-of-concept, not a validation.\n\nThe review content alone is worth publishing, and the experimental section is worth refereeing rather than desk-rejecting. A serious referee should ask for the missing horizon and seed statistics, and for a quantitative comparison of the reward landscape. With those additions the paper would be a solid benchmark. I'd bring it to a reading group because the review saves time and the Section 3.7 discussion is a good starting point for a methods seminar.","headline":"Useful review plus a new DRL-vs-adjoint control-cylinder comparison; the review is worth reading, but the new experiment's headline claim is not yet established because the reward is a short-horizon drag proxy.","tokens_in":18659,"tokens_out":3573,"would_cite":true,"duration_ms":34740,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep reinforcement learning can serve as a gradient-free substitute for classical adjoint methods in fluid-flow optimization, and the paper demonstrates this by finding the same optimal control-cylinder positions as an adjoint method at…","keywords":["deep reinforcement learning","fluid mechanics","flow control","shape optimization","adjoint method","drag reduction","transfer learning","autoencoder"],"falsifier":"Repeat the same optimization with episodes long enough for the wake to reach a quasi-steady state, or with a reward based on time-averaged drag over many shedding periods. If the final cylinder positions no longer match the adjoint-based optimum, the short-horizon reward is not a faithful proxy; if they still match, the proxy is sound.","tokens_in":17624,"feed_emoji":"🌊","tokens_out":9441,"duration_ms":89478,"temperature":0.7,"pith_summary":"Deep reinforcement learning has begun to be applied to fluid mechanics problems that are nonlinear and high-dimensional, and this paper reviews the state of that field. Its own contribution is a concrete comparison: a DRL agent is trained to position a small control cylinder so as to reduce the drag on a square cylinder in laminar flow. With a reward equal to the drag reduction, and with observations supplied through an autoencoder-compressed velocity field, the agent recovered the same optimal positions as a classical adjoint sensitivity method at Reynolds numbers 40 and 100, and produced coherent results at Re=10. The authors also show that transfer learning between Reynolds numbers and parallel multi-environment training substantially cut the learning time. They conclude that DRL is a viable gradient-free tool for both flow control and shape optimization, though high-Reynolds and high-dimensional action spaces remain untested.","feed_headline":"Reinforcement learning finds same flow optimum as adjoint method","feed_subtitle":"A drag-reduction agent trained without gradients matched a classical adjoint optimization on a square-cylinder flow.","key_machinery":"The argument is carried by a coupled training loop: a CFD solver supplies velocity fields, a convolutional autoencoder compresses each field from about 8000 to 70 features, and a policy network receives those features together with the last two cylinder positions and the Reynolds number, and outputs an action, either a small displacement or a direct position. The reward is the instantaneous drag reduction, $r_t = C_D^0 - C_D$, the same quantity the adjoint method differentiates. The two policy-gradient algorithms used, A3C and PPO, are standard DRL choices: A3C is an asynchronous actor-critic method and PPO is a trust-region-style method with clipped updates. Transfer learning, reusing agents trained at a lower Reynolds number as initializations for higher ones, is what turns a five-hour training run into ten-hour runs for four agents on a single CPU.","core_discovery":"On the test case of a square main cylinder with a movable smaller control cylinder in an incompressible laminar flow, the paper's finding is that deep reinforcement learning converges to the same optimum as the classical adjoint method. Two architectures are used: a sequential A3C agent that moves the cylinder in small steps over 17 time steps, and a direct PPO agent that chooses a position in one step, at Reynolds numbers 10, 40, and 100. Both share the same reward, the difference between the drag of the main cylinder alone and the total drag of the two cylinders, so a positive reward means net drag reduction. The agents' observations include the last two positions, the Reynolds number, and neural-network-encoded features of the velocity field. Comparing with the adjoint-based sensitivity results, the authors report the same optimal positions at Re=40 and Re=100, and coherent results at Re=10, which they read as evidence that DRL can perform optimization without gradient information.","pith_inferences":["If the short-horizon reward proxy holds, the same validation protocol, DRL against an adjoint baseline, could be carried to flows where adjoint gradients are hard to obtain, such as turbulent wakes; the episode length would then need to be checked carefully.","Because the reward gives only a scalar drag difference, the agent may have converged to the same local optimum as the adjoint method rather than the global one; starting the direct agent from several initial positions would tell.","The autoencoder's compression of the velocity field from 8000 to 70 dimensions suggests DRL could be driven by experimental flow images instead of simulated fields, a step the paper does not test.","At higher Reynolds numbers the drag reduction alone may be too noisy a signal, so a time-averaged or multi-objective reward could be necessary before DRL can match adjoint results outside laminar flow."],"forward_implications":["DRL can serve as a gradient-free alternative to adjoint-based optimization in laminar flow problems where computing sensitivities is expensive or impossible.","Transfer learning lets an agent trained at one Reynolds number initialize training at another, reducing computational cost compared with starting from scratch.","Parallel multi-environment training scales nearly perfectly with the number of simulations, so wall-clock training time drops substantially.","The same reward function supports both sequential control and direct one-shot optimization, giving a unified interface for control and design.","Autoencoder-based state compression allows DRL to handle high-dimensional fluid fields with small policy networks, which is relevant to experimental or large-scale CFD settings."],"supporting_citations":[{"why":"supplies the adjoint-based sensitivity results and optimal control-cylinder positions that form the comparison baseline for the DRL agents.","marker":"Meliga et al. (2014)"},{"why":"establishes the CFD-DRL coupling and active flow-control setup from which the square-cylinder drag-reduction test case is derived.","marker":"Rabault et al. (2019)"},{"why":"provides the multi-environment parallel training and transfer-learning procedures used to train the six agents.","marker":"Rabault & Kuhnle (2019)"},{"why":"demonstrates the convolutional autoencoder feature extraction from velocity fields that is reused to compress the state space.","marker":"Ma et al. (2018)"},{"why":"defines the benchmark laminar flow around a cylinder at Re=100 that sets the physical conditions of the test case.","marker":"Schäfer et al. (1996)"},{"why":"supplies the FEniCS finite-element CFD solver that implements the incompressible Navier-Stokes simulation.","marker":"Alnæs et al. (2015)"}],"fun_headline_variants":["DRL matches adjoint method on cylinder flow","Deep RL finds same optimum as adjoint method","Gradient-free RL matches classical adjoint in fluids","RL matches adjoint optimization without gradients","Reinforcement learning equals adjoint on flow control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward is measured over very short episodes, 17 time steps for one set of agents and a single action for the other, so everything leans on the assumption that such short-horizon drag readings stand in for the steady-state drag that the adjoint method optimizes.","fun_headline_variants_meta":{"raw":{"variants":["DRL matches adjoint method on cylinder flow","Deep RL finds same optimum as adjoint method","Gradient-free RL matches classical adjoint in fluids","RL matches adjoint optimization without gradients","Reinforcement learning equals adjoint on flow control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3878,"prompt_tokens":908,"completion_tokens":2970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":2899}},"tokens_in":524,"tokens_out":2970,"duration_ms":24559,"temperature":1.0,"reasoning_tokens":2899,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:49:46.318502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the same optimization with episodes long enough for the wake to reach a quasi-steady state, or with a reward based on time-averaged drag over many shedding periods. If the final cylinder positions no longer match the adjoint-based optimum, the short-horizon reward is not a faithful proxy; if they still match, the proxy is sound.","supporting_citations":[{"cited_title":"Physics of Fluids 26 , 26,104101","cited_arxiv_id":null,"evidence_quote":"supplies the adjoint-based sensitivity results and optimal control-cylinder positions that form the comparison baseline for the DRL agents."}],"review_version":1}