{"id":"bd727dd5-507c-40a9-8a39-a06b53dd1ede","arxiv_id":"2411.15902","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In simulated 2D turbulence, microswimmers trained with a Vicsek-style flocking action plus adversarial Q-learning reach a target more often than non-flocking smart swimmers in most parameter regimes tested.","lead":"This paper tests whether adding a flocking rule to reinforcement-learning navigation helps microswimmers reach a target in turbulent water. In computer simulations, flocking-aided 'smart flockers' reach the target more often than non-flocking smart swimmers and naive swimmers for most of the swimming speeds and flow conditions tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward shaping in Eq. (8) doubles the reward specifically for the flocking action, so the SF advantage may reflect the bonus rather than any navigational benefit of flocking; a neutral-reward control is needed.","rationale":"The reader's weakest assumption correctly identifies the reward shaping in Eq. (8) as the most likely source of the reported SF advantage. My stress-test concentrates on that single equation because it is the one place where the comparison between SF and SS is designed to favor flocking: the action A5 receives twice the reward exactly when it leads to progress. In Q-learning, such a reward bias will cause the agent to prefer A5 even if the flocking direction carries no additional navigational information. Therefore, the paper's internal evidence, such as Fig. 6(a) showing many swimmers choosing A5, is not an independent confirmation of the headline claim; it is a logical consequence of the reward function. The transfer-learning selection of the best Q-matrix is a real but secondary concern, and the absence of error bars and public code weakens reproducibility, but neither is as structurally decisive as the reward-shaping issue. The proposed neutral-reward control is inexpensive and would settle whether flocking per se is useful. Since the authors could plausibly run this control and the rest of the paper remains informative, the conditional verdict is appropriate; I do not see grounds to reject outright, but the central claim should not be accepted without the control.","tokens_in":11322,"tokens_out":2805,"duration_ms":28725,"concrete_test":"Retrain smart flockers with Eq. (8) replaced by rSF = tildesr for every action, keeping the 5-action set, all hyperparameters in Table I, and the same adversarial slave protocol unchanged. Then compare NSF - NSS and NSF - NNS over the same 10 initial conditions and parameter sweeps. If the SF advantage persists at comparable magnitude, the flocking action itself is beneficial; if it shrinks or vanishes, the advantage is attributable to the doubled reward. As a complementary check, train smart swimmers with a fifth action chosen as a random or target-independent direction but with the same doubled-reward rule; if they match SF, the bonus, not flocking, is responsible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that smart flockers (SF) outperform naive and smart swimmers because flocking itself aids path planning. But Eq. (8) defines rSF(t) = 2*tildesr only when the chosen action is the flocking action A5 AND the swimmer has made progress toward the target; otherwise rSF = tildesr. Because this reward enters directly into the Bellman update, Eq. (5), the Q-values for A5 are inflated exactly in the states where progress is occurring. The learned preference for A5, visible in Fig. 6(a) and the Q-matrix panels of Fig. 4, is therefore a direct consequence of the reward design, not independent evidence that flocking helps navigation. The comparison SF vs. SS is not a controlled test: SF has an extra action and a doubled reward for that action, while SS has neither. If the factor 2 were removed, the reported SF advantage could shrink or disappear. A secondary issue is the transfer-learning protocol: the Q-matrix is selected from the best-performing initial condition (IC-1) before evaluating, which mixes training performance with test selection. The primary load-bearing concern, however, is the reward shaping in Eq. (8), because it is structurally guaranteed to favor A5.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a reinforcement-learning strategy for microswimmer navigation in 2D homogeneous isotropic turbulence that adds a Vicsek-type flocking action to the adversarial Q-learning algorithm of Ref. [1]. It compares naive swimmers (NS), smart swimmers (SS), and smart flockers (SF) by the cumulative number of swimmers reaching a central target, and reports that SF can outperform both NS and SS for optimized hyperparameters, especially after a transfer-learning step. The paper also includes a brief comparison of SF/SS/NS with the surfing strategy of Monthiller et al. [16].","tokens_in":11630,"tokens_out":4123,"duration_ms":38126,"significance":"If substantiated, the result would extend reinforcement-learning navigation to interacting swimmers and suggest that flocking provides a useful inductive bias for path planning in turbulence. The manuscript builds on an established baseline and uses direct numerical simulation of 2D turbulence, and the authors state that data and scripts are available upon request. However, the reported SF advantage is not yet convincing because the reward function in Eq. (8) structurally favors the flocking action, the transfer-learning demonstration selects the Q-matrix from the best-performing initial condition, and the simulation results lack error bars and ensemble averaging. These issues must be resolved before the significance of the claim can be assessed.","major_comments":[{"comment":"The reward in Eq. (8) doubles r_tilde_SF when r_tilde_SF > 0 and the chosen action is the flocking action A5; otherwise r_SF = r_tilde_SF. Because this reward enters directly into the Bellman update in Eq. (5), the Q-values for A5 are inflated exactly in states where progress toward the target is occurring. The learned preference for the flocking action, visible in Fig. 6(a) and the yellow band in Fig. 4, is therefore at least partly a consequence of the reward design rather than independent evidence that flocking aids navigation. The comparison between SF and SS is not a controlled test because SF has an extra action and a doubled reward for that action, whereas SS has neither. To support the central claim, the authors must include a control in which the same reward is given to all actions (or a double reward is assigned to a non-flocking action) and show that SF still outperforms SS.","section":"II.D.3, Eq. (8)"},{"comment":"The text states that \"we select the Q-matrix, which we obtain from the best-performing case [Fig. 4 (k) with initial condition IC1]\" and then evaluate this Q-matrix on the ten initial conditions in Figs. 7-9. Because IC-1 is one of the few initial conditions for which SF already outperforms SS without transfer learning (Fig. 3(c)), selecting it post hoc and then reporting SF superiority on all ICs mixes training performance with test selection. This is a selection-on-the-test-set problem. The authors should pre-specify the Q-matrix (for example, from a validation set) or use a proper cross-validation procedure, and report results for all Q-matrices, not only the best one.","section":"III, transfer-learning protocol"},{"comment":"All results are based on single simulation runs per initial condition, with no ensemble averaging or error bars. The differences plotted in Fig. 3(c) are positive for only 3 of 10 initial conditions, and the claimed superiority after transfer learning relies on the selective protocol described above. Without multiple runs per initial condition and error bars (or at least a demonstration that the observed differences exceed run-to-run fluctuations), the count-based evidence is insufficient to support the general claim that SF outperform SS and NS.","section":"III, Figs. 3, 5, 7-10"}],"minor_comments":[{"comment":"The sentence \"we use o = o_SS and o = o_SF for smart microswimmers and smart microswimmers, respectively\" should read \"smart microswimmers and smart microflockers, respectively\".","section":"II.D.2, after Eq. (6)"},{"comment":"The condition \"aSF = ⟨p̂⟩\" is not consistent with the definition A5 ≡ arctan(⟨p̂_j⟩_n), which is an angle; please clarify whether the reward condition refers to the action index (A5) or the alignment direction.","section":"II.D.3, Eq. (8)"},{"comment":"The Vicsek update in Eq. (3) is used with ζ_i(t) = 0, with the justification that turbulence supplies noise. Since the original Vicsek model relies on explicit noise for ordering, this assumption should be tested or discussed in more detail, for example by showing that the ordering behavior is qualitatively similar with small nonzero ζ.","section":"II.C"},{"comment":"The claim that \"we have verified that our results do not depend sensitively on the type of forcing by using either Kolmogorov forcing or random forcing\" is not supported by any figure or table in the manuscript. Please provide this comparison or remove or qualify the claim.","section":"II.A"},{"comment":"The notation for NSS, NSF, and NNS is used inconsistently, with superscripts in some places and subscripts in others; please unify the notation.","section":"Figs. 3 and 5, notation"},{"comment":"The flowchart mentions a \"Q_F-matrix\" that is not defined in the text; please introduce the symbol or explain the notation.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The novelty of the paper is incremental, but the topic is of interest to the soft-matter and active-matter communities. The most urgent issue is the reward shaping in Eq. (8), which may fully explain the reported flocking advantage; a neutral-reward control is essential. The transfer-learning selection is also a clear methodological flaw that should be corrected. The lack of error bars would be acceptable for a letter if the effect were large and consistent, but here it is not. If the authors can supply the controls and a pre-registered transfer-learning protocol, the paper could become publishable after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper combines two existing ingredients—adversarial Q-learning for microswimmers in turbulence (from Ref. [1]) and Vicsek-type flocking—by adding one extra action (A5) to the action set and a tailored reward bonus. That is a legitimate, modest extension, and the study is cleanly set up: direct numerical simulation of 2D turbulence, ten initial conditions, a fixed target, and a comparison against naive swimmers, smart swimmers, and the surfing strategy of Monthiller et al. I appreciate that the authors test several values of swimming speed and alignment time, and that they openly discuss the transfer-learning protocol. The paper is readable and honest about its own scope.\n\nThe problem is in Eq. (8). The reward for a smart flocker is doubled whenever the flocking action A5 is chosen and the swimmer has made progress toward the target. That reward directly feeds the Bellman update, so the Q-values for A5 are inflated exactly in the states where progress is occurring. Unsurprisingly, Fig. 6(a) shows that many swimmers choose the flocking action, and the Q-matrix panels in Fig. 4 show A5 dominating. This is an artifact of the reward design, not independent evidence that flocking helps navigation. The SF-vs-SS comparison is not a controlled test: SF has both an extra action and a doubled reward for that action, while SS has neither. The authors do not run a neutral-reward control (e.g., doubling the reward for any action, or no bonus at all). If they did, the reported SF advantage might shrink or vanish. The transfer-learning section has a secondary issue: the Q-matrix is selected from the best-performing initial condition (IC-1) before evaluating on the others, which mixes training performance with test selection. Missing error bars and single runs per initial condition also make the quantitative claims hard to assess.\n\nThat said, this is not a sloppy paper. The reward shaping is explicit, not hidden, and the authors appear aware that they are encouraging flocking. The extension is natural and the simulations are reasonable for an incremental study. The central claim, however, is load-bearing and currently unsupported. Who is this for? Researchers working on RL-based navigation of microswimmers or microbots in turbulence. They would find a useful baseline and a clear warning about reward design, but not a demonstrated flocking advantage.\n\nRecommendation: send to peer review, but make the referee request a neutral-reward control, pre-registered transfer-learning selection, uncertainty estimates, and shared code/data. Without those, the headline claim should not be accepted.","headline":"A plausible incremental extension of adversarial Q-learning adds a Vicsek-style flocking action, but the headline claim is confounded by a reward function that double-counts the flocking action, so the advantage of 'smart flockers' is not yet established.","tokens_in":12138,"tokens_out":1269,"would_cite":false,"duration_ms":13800,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["47.27.-i"],"model":"deepseek-v4-flash","headline":"Adding a Vicsek-style flocking action to adversarial Q-learning lets more microswimmers reach a fixed target in two-dimensional turbulent flow, the paper claims.","keywords":["microswimmers","turbulent flows","path planning","reinforcement learning","Q-learning","Vicsek model","flocking","transfer learning"],"falsifier":"Train smart flockers with the reward of Eq. (8) replaced by a neutral version that pays the same reward regardless of which action is chosen, keeping the state space, hyperparameters, and transfer-learning protocol fixed; if the flockers no longer surpass smart swimmers, the claim that flocking itself aids navigation fails. As a second check, freeze the Q-matrix from a fixed training epoch of initial condition IC-1 before any evaluation is run, instead of selecting the best-performing case, and verify that the flocking advantage still holds across all ten initial conditions.","tokens_in":11139,"feed_emoji":"🎯","tokens_out":21890,"duration_ms":171893,"temperature":0.7,"pith_summary":"Microscopic swimmers in a turbulent flow are buffeted unpredictably, so steering to a fixed target is a hard navigation problem. This paper claims that flocking helps: adding a Vicsek-model-style action — steer toward the average heading of nearby swimmers — to an adversarial Q-learning controller raises the number of microswimmers that reach a fixed target in two-dimensional turbulence. In that controller, a 'master' swimmer is trained by being rewarded for outdoing a naive 'slave' that always heads toward the target. Without transfer learning, the flocking swimmers beat the four-action smart swimmers in three of ten turbulent initial conditions; after transfer learning, using the Q-matrix trained on the best-performing initial condition, they surpass both naive and smart swimmers over a wide range of speeds and alignment timescales, except at the highest speed, where the naive strategy wins. If the claim holds, coordination among swimmers becomes a usable lever for microrobot navigation and targeted delivery in unsteady flows.","feed_headline":"Flocking helps microswimmers reach targets in turbulent flows","feed_subtitle":"A Vicsek-style stay-together action added to Q-learning gets more swimmers to the target in 2D turbulence","key_machinery":"The load-bearing piece is the expanded action set of the smart flockers: the four target-relative directions plus $A_5$, the Vicsek alignment direction computed as the argument of the vector sum of neighbours' swimming directions within $R_0 = 0.2$. The state space is the same 12-state discretisation used by the baseline — three vorticity bins crossed with four bins for the angle between the target direction and the swimming direction — and the Q-table is updated by Bellman's equation with $\\epsilon$-greedy exploration. What makes the new action stick is the reward shaping of Eq. (8): a swimmer that chooses the flocking action while its paired naive slave gains on the target receives double the reward it would otherwise get. The adversarial master–slave pairing from Ref. [1] supplies the distance-based reward signal, and transfer learning freezes the Q-matrix trained on initial condition IC-1 for reuse on all ten initial conditions without exploration.","core_discovery":"The central claim is that one extra action changes the outcome: a microswimmer that can choose to align its heading with the average direction of its neighbours reaches the target more reliably than a swimmer choosing among the four target-relative directions alone. The paper extends the adversarial Q-learning method of Ref. [1] by enlarging the action set from $\\{\\hat{T},\\,-\\hat{T},\\,\\hat{T}_\\perp,\\,-\\hat{T}_\\perp\\}$ to include the flocking action $A_5 = \\arctan(\\langle \\hat{p}_j\\rangle_n)$, the mean direction of all neighbours within radius $R_0 = 0.2$, which is the alignment rule of the Vicsek model. The reward is reshaped for flockers: when the flocking action is chosen and the paired naive slave has gained on the target, the reward is doubled (Eq. (8)). The results are: without transfer learning, flockers outperform smart swimmers in three of ten turbulent initial conditions; with transfer learning, in which the Q-matrix trained on the best-performing initial condition (IC-1) is reused, flockers surpass both naive and smart swimmers over the entire simulation window for normalised alignment timescales $\\tilde{B} = 0.4$–$6.0$ and speeds $\\tilde{V}_s = 0.5$–$3.0$, with the naive strategy winning only at the highest speed. In the transfer-learning regime, the flocking action becomes the dominant maximal-value action in the learned Q-matrix, which the paper offers as evidence that the flocking direction is being used, not merely tried.","pith_inferences":["Reward-shaping control: because Eq. (8) doubles the reward when the flocking action is chosen during progress, the learned preference for $A_5$ may mostly reflect the bonus; a neutral control paying the same reward for every action would separate reward-driven from physics-driven benefit.","Bonus scaling: varying the multiplier (e.g., scaling $\\tilde{r}_{\\rm SF}$ by $\\beta = 1, 2, 4$ when the flocking action accompanies progress) and measuring the performance curve would show whether the advantage scales with the incentive, a signature of reward-driven rather than hydrodynamic benefit.","Flow-type sensitivity: the claim is established for statistically homogeneous and isotropic turbulence; a natural stress test is the same action set in flows with persistent coherent vortices, such as von Kármán vortex streets, where trapping in eddies is the dominant failure mode.","Decentralised control: the paper trains a single shared Q-matrix used by all swimmers; the relevant regime for swarms of autonomous microrobots is fully decentralised learning, where each swimmer updates only its own table from its own experience, and whether the flocking advantage survives there is untested."],"forward_implications":["Smart flockers reach the target in larger numbers than naive and smart swimmers when the hyperparameters are optimised, so a flocking action can improve, not just coexist with, adversarial Q-learning navigation.","Transfer learning consolidates the advantage: a Q-matrix trained on one turbulent initial condition and reused elsewhere makes flockers surpass both baselines for the entire simulation window.","The flocking advantage holds across alignment timescales $\\tilde{B} = 0.4$–$6.0$ and swimming speeds $\\tilde{V}_s = 0.5$–$2.0$, but breaks down at the highest speed tested ($\\tilde{V}_s = 3.0$), where always pointing at the target wins.","Without transfer learning, the benefit is initial-condition-dependent: flockers beat smart swimmers in only three of ten turbulent fields, so the flocking action pays off in some flow configurations more than in others.","Against 'surfer' navigation, which follows local flow gradients, every trained strategy beats surfers at large surf-time values, while naive swimmers match surfers at small ones."],"supporting_citations":[{"why":"Supplies the adversarial master–slave Q-learning method and the smart-swimmer baseline that the paper extends with a flocking action.","marker":"[1]"},{"why":"Defines the Vicsek alignment rule — steer toward the mean heading of neighbours within a radius — that becomes the flocking action $A_5$.","marker":"[20]"},{"why":"Models Vicsek-type flocks in a steady Taylor–Green vortical flow, the setting the paper generalises to fully turbulent flow.","marker":"[23]"},{"why":"Provides the Q-learning update (Bellman equation) and $\\epsilon$-greedy exploration used to train the Q-matrices.","marker":"[34]"},{"why":"Introduces reinforcement-learning navigation for smart microswimmers, the strategy family the paper builds on and compares against.","marker":"[10]"},{"why":"Defines the surfing navigation strategy (following local flow gradients) used as the comparison in Fig. 10.","marker":"[16]"},{"why":"Presents an actor-critic reinforcement-learning approach for point-to-point navigation in 2D turbulent flows, an alternative baseline.","marker":"[11]"}],"fun_headline_variants":["Flocking action boosts microswimmer navigation in 2D turbulence","Vicsek-style alignment improves Q-learning for microswimmers","Transfer learning plus flocking beats naive microswimmer strategies","Flocking microswimmers reach targets better in turbulent flows","Adding a flocking move ups microswimmer success in turbulence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported edge of flocking rests on the reward rule of Eq. (8), which pays double for choosing the flocking action whenever the paired naive slave has made progress toward the target, and the paper includes no control that pays the same reward for all actions; if the learned preference for the flocking action mainly mirrors that bonus, the advantage could shrink or disappear when the bonus is removed.","fun_headline_variants_meta":{"raw":{"variants":["Flocking action boosts microswimmer navigation in 2D turbulence","Vicsek-style alignment improves Q-learning for microswimmers","Transfer learning plus flocking beats naive microswimmer strategies","Flocking microswimmers reach targets better in turbulent flows","Adding a flocking move ups microswimmer success in turbulence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3695,"prompt_tokens":1010,"completion_tokens":2685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":2600}},"tokens_in":626,"tokens_out":2685,"duration_ms":18529,"temperature":1.0,"reasoning_tokens":2600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:45:26.823912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train smart flockers with the reward of Eq. (8) replaced by a neutral version that pays the same reward regardless of which action is chosen, keeping the state space, hyperparameters, and transfer-learning protocol fixed; if the flockers no longer surpass smart swimmers, the claim that flocking itself aids navigation fails. As a second check, freeze the Q-matrix from a fixed training epoch of initial condition IC-1 before any evaluation is run, instead of selecting the best-performing case, and verify that the flocking advantage still holds across all ten initial conditions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the adversarial master–slave Q-learning method and the smart-swimmer baseline that the paper extends with a flocking action."},{"cited_title":"Gustavsson, F","cited_arxiv_id":null,"evidence_quote":"Provides the Q-learning update (Bellman equation) and $\\epsilon$-greedy exploration used to train the Q-matrices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces reinforcement-learning navigation for smart microswimmers, the strategy family the paper builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the surfing navigation strategy (following local flow gradients) used as the comparison in Fig. 10."},{"cited_title":"Reddy, A","cited_arxiv_id":null,"evidence_quote":"Presents an actor-critic reinforcement-learning approach for point-to-point navigation in 2D turbulent flows, an alternative baseline."}],"review_version":1}