Can flocking aid the path planning of microswimmers in turbulent flows?

Akanksha Gupta; Jaya Kumar Alageshan; Kolluru Venkata Kiran; Rahul Pandit

REVIEW 3 major objections 6 minor 39 references

Can flocking aid the path planning of microswimmers in turbulent flows?

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a Vicsek-style flocking action to adversarial Q-learning lets more microswimmers reach a fixed target in two-dimensional turbulent flow, the paper claims.

desk verdict A plausible incremental extension of adversarial Q-learning adds a Vicsek-style flocking action, but the headline claim is confounded by a reward function that double-counts the flocking action, so the advantage of 'smart flockers' is not yet established. read the letter →

arxiv 2411.15902 v1 pith:6UKFZMYA submitted 2024-11-24 physics.flu-dyn

Akanksha Gupta , Jaya Kumar Alageshan , Kolluru Venkata Kiran , Rahul Pandit This is my paper · ORCID

classification physics.flu-dyn PACS 47.27.-i

keywords microswimmersturbulentflowspathplanningreinforcementlearningQ-learningVicsekmodelflockingtransfer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Microscopic swimmers in a turbulent flow are buffeted unpredictably, so steering to a fixed target is a hard navigation problem. This paper claims that flocking helps: adding a Vicsek-model-style action — steer toward the average heading of nearby swimmers — to an adversarial Q-learning controller raises the number of microswimmers that reach a fixed target in two-dimensional turbulence. In that controller, a 'master' swimmer is trained by being rewarded for outdoing a naive 'slave' that always heads toward the target. Without transfer learning, the flocking swimmers beat the four-action smart swimmers in three of ten turbulent initial conditions; after transfer learning, using the Q-matrix trained on the best-performing initial condition, they surpass both naive and smart swimmers over a wide range of speeds and alignment timescales, except at the highest speed, where the naive strategy wins. If the claim holds, coordination among swimmers becomes a usable lever for microrobot navigation and targeted delivery in unsteady flows.

What carries the argument

The load-bearing piece is the expanded action set of the smart flockers: the four target-relative directions plus $A_5$, the Vicsek alignment direction computed as the argument of the vector sum of neighbours' swimming directions within $R_0 = 0.2$. The state space is the same 12-state discretisation used by the baseline — three vorticity bins crossed with four bins for the angle between the target direction and the swimming direction — and the Q-table is updated by Bellman's equation with $\epsilon$-greedy exploration. What makes the new action stick is the reward shaping of Eq. (8): a swimmer that chooses the flocking action while its paired naive slave gains on the target receives double the reward it would otherwise get. The adversarial master–slave pairing from Ref. [1] supplies the distance-based reward signal, and transfer learning freezes the Q-matrix trained on initial condition IC-1 for reuse on all ten initial conditions without exploration.

What would settle it

Train smart flockers with the reward of Eq. (8) replaced by a neutral version that pays the same reward regardless of which action is chosen, keeping the state space, hyperparameters, and transfer-learning protocol fixed; if the flockers no longer surpass smart swimmers, the claim that flocking itself aids navigation fails. As a second check, freeze the Q-matrix from a fixed training epoch of initial condition IC-1 before any evaluation is run, instead of selecting the best-performing case, and verify that the flocking advantage still holds across all ten initial conditions.

Watch

Extended reading notes

Core claim

The central claim is that one extra action changes the outcome: a microswimmer that can choose to align its heading with the average direction of its neighbours reaches the target more reliably than a swimmer choosing among the four target-relative directions alone. The paper extends the adversarial Q-learning method of Ref. [1] by enlarging the action set from $\{\hat{T},\,-\hat{T},\,\hat{T}_\perp,\,-\hat{T}_\perp\}$ to include the flocking action $A_5 = \arctan(\langle \hat{p}_j\rangle_n)$, the mean direction of all neighbours within radius $R_0 = 0.2$, which is the alignment rule of the Vicsek model. The reward is reshaped for flockers: when the flocking action is chosen and the paired naive slave has gained on the target, the reward is doubled (Eq. (8)). The results are: without transfer learning, flockers outperform smart swimmers in three of ten turbulent initial conditions; with transfer learning, in which the Q-matrix trained on the best-performing initial condition (IC-1) is reused, flockers surpass both naive and smart swimmers over the entire simulation window for normalised alignment timescales $\tilde{B} = 0.4$–$6.0$ and speeds $\tilde{V}_s = 0.5$–$3.0$, with the naive strategy winning only at the highest speed. In the transfer-learning regime, the flocking action becomes the dominant maximal-value action in the learned Q-matrix, which the paper offers as evidence that the flocking direction is being used, not merely tried.

Load-bearing premise

The reported edge of flocking rests on the reward rule of Eq. (8), which pays double for choosing the flocking action whenever the paired naive slave has made progress toward the target, and the paper includes no control that pays the same reward for all actions; if the learned preference for the flocking action mainly mirrors that bonus, the advantage could shrink or disappear when the bonus is removed.

Editorial extensions

If this is right

Smart flockers reach the target in larger numbers than naive and smart swimmers when the hyperparameters are optimised, so a flocking action can improve, not just coexist with, adversarial Q-learning navigation.
Transfer learning consolidates the advantage: a Q-matrix trained on one turbulent initial condition and reused elsewhere makes flockers surpass both baselines for the entire simulation window.
The flocking advantage holds across alignment timescales $\tilde{B} = 0.4$–$6.0$ and swimming speeds $\tilde{V}_s = 0.5$–$2.0$, but breaks down at the highest speed tested ($\tilde{V}_s = 3.0$), where always pointing at the target wins.
Without transfer learning, the benefit is initial-condition-dependent: flockers beat smart swimmers in only three of ten turbulent fields, so the flocking action pays off in some flow configurations more than in others.
Against 'surfer' navigation, which follows local flow gradients, every trained strategy beats surfers at large surf-time values, while naive swimmers match surfers at small ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Reward-shaping control: because Eq. (8) doubles the reward when the flocking action is chosen during progress, the learned preference for $A_5$ may mostly reflect the bonus; a neutral control paying the same reward for every action would separate reward-driven from physics-driven benefit.
Bonus scaling: varying the multiplier (e.g., scaling $\tilde{r}_{\rm SF}$ by $\beta = 1, 2, 4$ when the flocking action accompanies progress) and measuring the performance curve would show whether the advantage scales with the incentive, a signature of reward-driven rather than hydrodynamic benefit.
Flow-type sensitivity: the claim is established for statistically homogeneous and isotropic turbulence; a natural stress test is the same action set in flows with persistent coherent vortices, such as von Kármán vortex streets, where trapping in eddies is the dominant failure mode.
Decentralised control: the paper trains a single shared Q-matrix used by all swimmers; the relevant regime for swarms of autonomous microrobots is fully decentralised learning, where each swimmer updates only its own table from its own experience, and whether the flocking advantage survives there is untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper combines two existing ingredients—adversarial Q-learning for microswimmers in turbulence (from Ref. [1]) and Vicsek-type flocking—by adding one extra action (A5) to the action set and a tailored reward bonus. That is a legitimate, modest extension, and the study is cleanly set up: direct numerical simulation of 2D turbulence, ten initial conditions, a fixed target, and a comparison against naive swimmers, smart swimmers, and the surfing strategy of Monthiller et al. I appreciate that the authors test several values of swimming speed and alignment time, and that they openly discuss the transfer-learning protocol. The paper is readable and honest about its own scope.

The problem is in Eq. (8). The reward for a smart flocker is doubled whenever the flocking action A5 is chosen and the swimmer has made progress toward the target. That reward directly feeds the Bellman update, so the Q-values for A5 are inflated exactly in the states where progress is occurring. Unsurprisingly, Fig. 6(a) shows that many swimmers choose the flocking action, and the Q-matrix panels in Fig. 4 show A5 dominating. This is an artifact of the reward design, not independent evidence that flocking helps navigation. The SF-vs-SS comparison is not a controlled test: SF has both an extra action and a doubled reward for that action, while SS has neither. The authors do not run a neutral-reward control (e.g., doubling the reward for any action, or no bonus at all). If they did, the reported SF advantage might shrink or vanish. The transfer-learning section has a secondary issue: the Q-matrix is selected from the best-performing initial condition (IC-1) before evaluating on the others, which mixes training performance with test selection. Missing error bars and single runs per initial condition also make the quantitative claims hard to assess.

That said, this is not a sloppy paper. The reward shaping is explicit, not hidden, and the authors appear aware that they are encouraging flocking. The extension is natural and the simulations are reasonable for an incremental study. The central claim, however, is load-bearing and currently unsupported. Who is this for? Researchers working on RL-based navigation of microswimmers or microbots in turbulence. They would find a useful baseline and a clear warning about reward design, but not a demonstrated flocking advantage.

Recommendation: send to peer review, but make the referee request a neutral-reward control, pre-registered transfer-learning selection, uncertainty estimates, and shared code/data. Without those, the headline claim should not be accepted.

Referee Report

3 major / 6 minor

Summary. This paper proposes a reinforcement-learning strategy for microswimmer navigation in 2D homogeneous isotropic turbulence that adds a Vicsek-type flocking action to the adversarial Q-learning algorithm of Ref. [1]. It compares naive swimmers (NS), smart swimmers (SS), and smart flockers (SF) by the cumulative number of swimmers reaching a central target, and reports that SF can outperform both NS and SS for optimized hyperparameters, especially after a transfer-learning step. The paper also includes a brief comparison of SF/SS/NS with the surfing strategy of Monthiller et al. [16].

Significance. If substantiated, the result would extend reinforcement-learning navigation to interacting swimmers and suggest that flocking provides a useful inductive bias for path planning in turbulence. The manuscript builds on an established baseline and uses direct numerical simulation of 2D turbulence, and the authors state that data and scripts are available upon request. However, the reported SF advantage is not yet convincing because the reward function in Eq. (8) structurally favors the flocking action, the transfer-learning demonstration selects the Q-matrix from the best-performing initial condition, and the simulation results lack error bars and ensemble averaging. These issues must be resolved before the significance of the claim can be assessed.

major comments (3)

[II.D.3, Eq. (8)] The reward in Eq. (8) doubles r_tilde_SF when r_tilde_SF > 0 and the chosen action is the flocking action A5; otherwise r_SF = r_tilde_SF. Because this reward enters directly into the Bellman update in Eq. (5), the Q-values for A5 are inflated exactly in states where progress toward the target is occurring. The learned preference for the flocking action, visible in Fig. 6(a) and the yellow band in Fig. 4, is therefore at least partly a consequence of the reward design rather than independent evidence that flocking aids navigation. The comparison between SF and SS is not a controlled test because SF has an extra action and a doubled reward for that action, whereas SS has neither. To support the central claim, the authors must include a control in which the same reward is given to all actions (or a double reward is assigned to a non-flocking action) and show that SF still outperforms SS.
[III, transfer-learning protocol] The text states that "we select the Q-matrix, which we obtain from the best-performing case [Fig. 4 (k) with initial condition IC1]" and then evaluate this Q-matrix on the ten initial conditions in Figs. 7-9. Because IC-1 is one of the few initial conditions for which SF already outperforms SS without transfer learning (Fig. 3(c)), selecting it post hoc and then reporting SF superiority on all ICs mixes training performance with test selection. This is a selection-on-the-test-set problem. The authors should pre-specify the Q-matrix (for example, from a validation set) or use a proper cross-validation procedure, and report results for all Q-matrices, not only the best one.
[III, Figs. 3, 5, 7-10] All results are based on single simulation runs per initial condition, with no ensemble averaging or error bars. The differences plotted in Fig. 3(c) are positive for only 3 of 10 initial conditions, and the claimed superiority after transfer learning relies on the selective protocol described above. Without multiple runs per initial condition and error bars (or at least a demonstration that the observed differences exceed run-to-run fluctuations), the count-based evidence is insufficient to support the general claim that SF outperform SS and NS.

minor comments (6)

[II.D.2, after Eq. (6)] The sentence "we use o = o_SS and o = o_SF for smart microswimmers and smart microswimmers, respectively" should read "smart microswimmers and smart microflockers, respectively".
[II.D.3, Eq. (8)] The condition "aSF = ⟨p̂⟩" is not consistent with the definition A5 ≡ arctan(⟨p̂_j⟩_n), which is an angle; please clarify whether the reward condition refers to the action index (A5) or the alignment direction.
[II.C] The Vicsek update in Eq. (3) is used with ζ_i(t) = 0, with the justification that turbulence supplies noise. Since the original Vicsek model relies on explicit noise for ordering, this assumption should be tested or discussed in more detail, for example by showing that the ordering behavior is qualitatively similar with small nonzero ζ.
[II.A] The claim that "we have verified that our results do not depend sensitively on the type of forcing by using either Kolmogorov forcing or random forcing" is not supported by any figure or table in the manuscript. Please provide this comparison or remove or qualify the claim.
[Figs. 3 and 5, notation] The notation for NSS, NSF, and NNS is used inconsistently, with superscripts in some places and subscripts in others; please unify the notation.
[Appendix A] The flowchart mentions a "Q_F-matrix" that is not defined in the text; please introduce the symbol or explain the notation.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (8) doubles the reward specifically for the flocking action, so the learned preference for flocking and the reported SF advantage may be manufactured by the reward design; a neutral-reward control is needed.

fitted input called prediction [Sec. II D 3, Eq. (8), and Sec. III, Fig. 6(a) discussion]
""To encourage flocking action, we define the final reward for SF to be rSF (t) = ( 2˜rSF , if ˜rSF > 0 & aSF = ⟨ˆp⟩ , ˜rSF , otherwise . (8)" ... "Fig. 6(a) clearly demonstrates that a significant number of microswimmers have chosen flocking, signifying the effectiveness of this action.""

The reward in Eq. (8) is a designed input that doubles the progress reward only when the flocking action A5 is selected and progress is positive. Since the Q-learning update in Eq. (5) feeds this reward directly into Q(s, A5), the Q-values for A5 are inflated exactly in the states where progress occurs. The observed preference for A5 in Fig. 6(a), offered as evidence that flocking is effective, is therefore a consequence of the reward definition, not independent evidence. The SF-versus-SS comparison is uncontrolled: SF has both the extra action A5 and the doubled reward for A5, while SS has neither. If the factor 2 were removed or the bonus were applied to all actions, the reported SF advantage could shrink or vanish.

full rationale

The paper's central comparison is SF versus SS versus NS. The structural difference between SF and SS in the learning algorithm is (i) the extra flocking action A5 and (ii) the reward in Eq. (8), which doubles rtilde when A5 is chosen and progress is positive. The Q-learning update Eq. (5) uses this reward directly, so the inflated Q-values for A5 and the resulting action distribution in Fig. 6(a) follow from the reward design by construction. The paper interprets the high frequency of A5 as "signifying the effectiveness of this action," which is circular: the reward was engineered to make A5 attractive. No control is run with A5 available but without the reward bonus, nor with the same bonus applied to all actions, so the SF advantage cannot be cleanly attributed to flocking per se. The transfer-learning protocol also selects the Q-matrix from the best-performing initial condition (IC-1) before evaluating, which is an outcome-based selection; this is a methodological concern but not itself a circular step. The adversarial-Q-learning framework and the flow solver are otherwise self-contained and are not circular; Ref. [1] is a genuine external prior implementation. However, because the load-bearing claim of flocking advantage is entangled with the reward shaping by construction, the circularity score is elevated.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central comparison depends on the hand-chosen reward multiplier, the arbitrary state discretization, the interaction radius, the capture radius, and several modeling assumptions about turbulence, passive swimmers, and the Vicsek control action. The most consequential free parameter is the reward multiplier in Eq. (8), which structurally encourages the flocking action.

free parameters (4)

Flocking reward multiplier (factor 2 in Eq. 8) = 2.0
Chosen by hand to encourage the flocking action A5; directly shapes the learned policy and the SF versus SS comparison, with no sensitivity analysis.
Vorticity state threshold ω0 = 2.0
Arbitrary discretization of the flow vorticity into three states; changes the state space and therefore the Q-learning problem.
Flocking interaction radius R0 = 0.2 (about 0.03L)
Described as an illustrative value; defines the neighborhood for the flocking action A5 and affects how often flocking is available.
Target capture radius rδ = 0.062
Defines what counts as reaching the target and therefore sets the success counts used in all comparisons.

assumptions (5)

domain assumption The 12-state discretization (vorticity sign plus angle to target) is a sufficient Markov state for optimal navigation control.
Q-learning is applied over this coarse state space; if important flow information is lost, the learned policy is suboptimal. Introduced in Sec. II.D and Fig. 2.
domain assumption Microswimmers are passive tracers with self-propulsion and no back-reaction on the flow.
Eq. (2) couples swimmers to the fluid one-way; real swimmers would perturb the flow. Standard simplification, but load-bearing for the comparison.
domain assumption A 256x256 DNS of 2D homogeneous isotropic turbulence with linear friction and forcing is representative of the intended application.
The flow is statistically steady 2D HIT at moderate resolution; the paper states forcing-type insensitivity but does not show the data. Section II.A.
ad hoc to paper Vicsek alignment without explicit noise (ζ=0) is a valid model of flocking in turbulence.
Eq. (3) sets noise to zero and relies on turbulence to supply disorder; this choice is motivated by convenience and affects the action A5.
domain assumption The adversarial reward based on a reinitialized 'slave' swimmer's distance change is a good proxy for genuine progress toward the target.
Eq. (7) uses the slave distance difference; this proxy is inherited from Ref. [1] and is not independently validated here.

invented entities (1)

Flocking action A5 = arctan(⟨p̂_j⟩_n)
purpose: Adds Vicsek-type alignment as an explicit RL action so agents can choose to point along the mean heading of neighbors within radius R0.
The paper provides no external experimental or theoretical evidence that this control action improves navigation; its apparent benefit is demonstrated only inside the reward-shaping scheme.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can flocking aid the path planning of microswimmers in turbulent flows?." pith.science (2026). https://pith.science/paper/6UKFZMYA

@misc{pith2026241115902,
  author       = {Pith},
  title        = {Pith review of: Can flocking aid the path planning of microswimmers in turbulent flows?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UKFZMYA}},
  note         = {Machine review of arXiv:2411.15902}
}

read the original abstract

We show that flocking of microswimmers in a turbulent flow can enhance the efficacy of reinforcement-learning-based path-planning of microswimmers in turbulent flows. In particular, we develop a machine-learning strategy that incorporates Vicsek-model-type flocking in microswimmer assemblies in a statistically homogeneous and isotropic turbulent flow in two dimensions (2D). We build on the adversarial-reinforcement-learning of Ref.~\cite{alageshan2020machine} for non-interacting microswimmers in turbulent flows. Such microswimmers aim to move optimally from an initial position to a target. We demonstrate that our flocking-aided version of the adversarial-reinforcement-learning strategy of Ref.~\cite{alageshan2020machine} can be superior to earlier microswimmer path-planning strategies.

Figures

Figures reproduced from arXiv: 2411.15902 by the authors.

**Figure 2.** FIG. 2: (a) Discretization of the vorticity field [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** FIG. 3: Plots versus the nondimensionalised time [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: FIG. 4: Plots versus the normalised time of the maximal value in a row of the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: FIG. 5: Bar charts of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: FIG. 6: Bar charts of [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: FIG. 7: Plots versus the nondimensionalised time [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: FIG. 8: Plots versus the nondimensionalised time [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: FIG. 9: Plots versus the nondimensionalised time [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: FIG. 10: Plots versus the nondimensionalised time [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 11.** Figure 11: FIG. 11: Flowchart with the sequence of processes [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages

[1]

The ϵ-greedy method In this method [34] a probability distribution decides the control direction and balances exploration and ex- ploitation in Q-learning. For our problem we choose [1] the probability distribution function (PDF) P[ ˆ oi(si)] = ϵg Na + (1 − ϵg)δ(ˆ oi(si) − ˆ omax) , ˆ omax := argmax a∈AQ(s, a) , (4) where Na is the number of actions, whic...

work page
[16]

E. M. Barrows, Animal behavior desk reference: a dic- tionary of animal behavior, ecology, and evolution (CRC press, 2011)

work page 2011
[2]

We must consider two such matrices, namely, QSS and QSF , with ˆoSS := argmaxa∈ASS QSS (s, aSS ) or ˆoSF := argmaxa∈ASF QSF (s, aSF ) , (6) in Eq

Q-learning scheme We use Bellman’s equation, which is a Markov-decision process, i.e., for a given Q-matrix Q(s(t), ˆo(t)) 7→ (1 − λ) Q(s(t), ˆo(t)) + λ r(t) + γ max ˆa Q(s(t + ∆t), ˆa) ,(5) where λ and γ are learning parameters that need to be set [see Table I]. We must consider two such matrices, namely, QSS and QSF , with ˆoSS := argmaxa∈ASS QSS (s, aS...

work page
[3]

[1], which employs ad- versarial Q learning for every smart microswimmers SS , by using a similar learning scheme for every smart mi- croflocker SF

Adversarial Q learning and Rewards We build on the work of Ref. [1], which employs ad- versarial Q learning for every smart microswimmers SS , by using a similar learning scheme for every smart mi- croflocker SF . In this scheme, we associate one na ¨ ıve mi- croswimmer N S, referred to as the slave [1], per SS and SF , referred to as the master [1]; we u...

work page
[4]

J. K. Alageshan, A. K. Verma, J. Bec, and R. Pandit, Ma- chine learning strategies for path-planning microswim- mers in turbulent flows, Physical Review E 101, 043110 (2020)

work page 2020
[5]

R. D. Deegan, O. Bakajin, T. F. Dupont, G. Huber, S. R. Nagel, and T. A. Witten, Capillary flow as the cause of ring stains from dried liquid drops, Nature 389, 827 (1997)

1997
[6]

L. I. Zaichik, V. M. Alipchenkov, and E. G. Sinaiski, Particles in turbulent flows (John Wiley & Sons, 2008)

work page 2008
[7]

C. K. Schmidt, M. Medina-S´ anchez, R. J. Edmondson, and O. G. Schmidt, Engineering microrobots for tar- geted cancer therapies from a medical perspective, Na- ture Communications 11, 5618 (2020)

work page 2020

Show all 39 references

[8]

J. K. Mills and D. Needham, Targeted drug delivery, Ex- pert Opinion on Therapeutic Patents 9, 1499 (1999)

1999
[9]

Hassanzadeh, F

P. Hassanzadeh, F. Atyabi, and R. Dinarvand, The sig- nificance of artificial intelligence in drug delivery system design, Advanced drug delivery reviews 151, 169 (2019)

2019
[10]

Y. H. Bae and K. Park, Targeted drug delivery to tumors: myths, reality and possibility (2011)

2011
[11]

Reddy, A

G. Reddy, A. Celani, T. J. Sejnowski, and M. Vergassola, Learning to soar in turbulent environments, Proceedings of the National Academy of Sciences 113, E4877 (2016)

2016
[12]

Reddy, J

G. Reddy, J. Wong-Ng, A. Celani, T. J. Sejnowski, and M. Vergassola, Glider soaring via reinforcement learning in the field, Nature 562, 236 (2018)

2018
[13]

Colabrese, K

S. Colabrese, K. Gustavsson, A. Celani, and L. Biferale, Flow navigation by smart microswimmers via rein- forcement learning, Physical review letters 118, 158004 (2017)

2017
[14]

Biferale, F

L. Biferale, F. Bonaccorso, M. Buzzicotti, P. Clark Di Leoni, and K. Gustavsson, Zermelo’s problem: Op- timal point-to-point navigation in 2d turbulent flows us- ing reinforcement learning, Chaos: An Interdisciplinary Journal of Nonlinear Science 29, 103138 (2019)

2019
[15]

Calascibetta, L

C. Calascibetta, L. Biferale, F. Borra, A. Celani, and M. Cencini, Taming lagrangian chaos with multi- objective reinforcement learning, The European Physical Journal E 46, 1 (2023)

2023
[17]

D. B. Dusenbery, Living at micro scale: the unexpected physics of being small (Harvard University Press, 2009)

2009
[18]

G. H. Wadhams and J. P. Armitage, Making sense of it all: bacterial chemotaxis, Nature Reviews Molecular Cell Biology 5, 1024–1037 (2004)

2004
[19]

Monthiller, A

R. Monthiller, A. Loisy, M. A. R. Koehl, B. Favier, and C. Eloy, Surfing on turbulence: A strategy for planktonic navigation, Phys. Rev. Lett. 129, 064502 (2022)

2022
[20]

ten Hagen, F

B. ten Hagen, F. K¨ ummel, R. Wittkowski, D. Takagi, H. L¨ owen, and C. Bechinger, Gravitaxis of asymmetric self-propelled colloidal particles, Nature Communications 5, 4829 (2014)

2014
[21]

M. C. Marchetti, J.-F. Joanny, S. Ramaswamy, T. B. Liverpool, J. Prost, M. Rao, and R. A. Simha, Hydrody- namics of soft active matter, Reviews of modern physics 85, 1143 (2013)

2013
[22]

Cavagna, A

A. Cavagna, A. Cimarelli, I. Giardina, G. Parisi, R. San- tagati, F. Stefanini, and M. Viale, Scale-free correlations in starling flocks, Proceedings of the National Academy of Sciences 107, 11865 (2010)

2010
[23]

Vicsek, A

T. Vicsek, A. Czir´ ok, E. Ben-Jacob, I. Cohen, and O. Shochet, Novel type of phase transition in a system of self-driven particles, Physical review letters 75, 1226 (1995)

1995
[24]

D. Jang, J. Jeong, H. Song, and S. K. Chung, Targeted drug delivery technology using untethered microrobots: A review, Journal of Micromechanics and Microengineer- ing 29, 053002 (2019)

2019
[25]

Z. Zou, Y. Liu, A. C. Tsang, Y.-N. Young, and O. S. Pak, Adaptive micro-locomotion in a dynamically changing 9 FIG. 6: Bar charts of NSF at the end of our simulation for initial condition IC-1 showing distributions (a) over actions the possible actions aSF and (b) the states S...

2024
[26]

A. W. Baggaley, Model flocks in a steady vortical flow, Physical Review E 91, 053019 (2015)

2015
[27]

Monthiller, A

R. Monthiller, A. Loisy, M. A. Koehl, B. Favier, and C. Eloy, Surfing on turbulence: a strategy for planktonic navigation, Physical Review Letters 129, 064502 (2022)

2022
[28]

Perlekar, S

P. Perlekar, S. S. Ray, D. Mitra, and R. Pandit, Persis- tence problem in two-dimensional fluid turbulence, Phys. Rev. Lett. 106, 054501 (2011)

2011
[29]

S. A. Orszag, On the elimination of aliasing in finite- difference schemes by filtering high-wavenumber compo- nents., Journal of Atmospheric Sciences 28, 1074 (1971)

1971
[30]

Canuto, M

C. Canuto, M. Hussaini, and A. Quarteroni, Ta zang spectral methods in fluid dynamics, Series in Computa- tional Physics (1988)

1988
[31]

S. M. Cox and P. C. Matthews, Exponential time dif- ferencing for stiff systems, Journal of Computational Physics 176, 430 (2002)

2002
[32]

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction (MIT press, 2018)

2018
[33]

Beintema, A

G. Beintema, A. Corbetta, L. Biferale, and F. Toschi, Controlling rayleigh–b´ enard convection via reinforcement learning, Journal of Turbulence 21, 585 (2020)

2020
[34]

Gustavsson, F

K. Gustavsson, F. Berglund, P. Jonsson, and B. Mehlig, Preferential sampling and small-scale clustering of gy- rotactic microswimmers in turbulence, Physical review letters 116, 108104 (2016)

2016
[35]

J. Qiu, W. Huang, C. Xu, and L. Zhao, Swimming strat- egy of settling elongated micro-swimmers by reinforce- ment learning, Science China Physics, Mechanics & As- tronomy 63, 284711 (2020)

2020
[36]

Mui˜ nos-Landin, A

S. Mui˜ nos-Landin, A. Fischer, V. Holubec, and F. Ci- chos, Reinforcement learning with artificial microswim- mers, Science Robotics 6, eabd9285 (2021)

2021
[37]

C. J. Watkins and P. Dayan, Technical note: Q-learning, Machine Learning 8, 279 (1992)

1992
[38]

X. Kong, B. Xin, F. Liu, and Y. Wang, Revisiting the master-slave architecture in multi-agent deep reinforce- ment learning, arXiv preprint arXiv:1712.07305 (2017)

2017 arXiv
[39]

Kumar, H

N. Kumar, H. Soni, and S. Ramaswamy, Flocking at a distance in active granular matter, Nature Communica- tions 5, 10.1038/ncomms5688 (2014). Appendix A: Pseudocode We present the pseudocode for calculating the reward after choosing the best possible action from the epsilon- gr...

2014 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

The ϵ-greedy method In this method [34] a probability distribution decides the control direction and balances exploration and ex- ploitation in Q-learning. For our problem we choose [1] the probability distribution function (PDF) P[ ˆ oi(si)] = ϵg Na + (1 − ϵg)δ(ˆ oi(si) − ˆ omax) , ˆ omax := argmax a∈AQ(s, a) , (4) where Na is the number of actions, whic...

work page

[2] [16]

E. M. Barrows, Animal behavior desk reference: a dic- tionary of animal behavior, ecology, and evolution (CRC press, 2011)

work page 2011

[3] [2]

We must consider two such matrices, namely, QSS and QSF , with ˆoSS := argmaxa∈ASS QSS (s, aSS ) or ˆoSF := argmaxa∈ASF QSF (s, aSF ) , (6) in Eq

Q-learning scheme We use Bellman’s equation, which is a Markov-decision process, i.e., for a given Q-matrix Q(s(t), ˆo(t)) 7→ (1 − λ) Q(s(t), ˆo(t)) + λ r(t) + γ max ˆa Q(s(t + ∆t), ˆa) ,(5) where λ and γ are learning parameters that need to be set [see Table I]. We must consider two such matrices, namely, QSS and QSF , with ˆoSS := argmaxa∈ASS QSS (s, aS...

work page

[4] [3]

[1], which employs ad- versarial Q learning for every smart microswimmers SS , by using a similar learning scheme for every smart mi- croflocker SF

Adversarial Q learning and Rewards We build on the work of Ref. [1], which employs ad- versarial Q learning for every smart microswimmers SS , by using a similar learning scheme for every smart mi- croflocker SF . In this scheme, we associate one na ¨ ıve mi- croswimmer N S, referred to as the slave [1], per SS and SF , referred to as the master [1]; we u...

work page

[5] [4]

J. K. Alageshan, A. K. Verma, J. Bec, and R. Pandit, Ma- chine learning strategies for path-planning microswim- mers in turbulent flows, Physical Review E 101, 043110 (2020)

work page 2020

[6] [5]

R. D. Deegan, O. Bakajin, T. F. Dupont, G. Huber, S. R. Nagel, and T. A. Witten, Capillary flow as the cause of ring stains from dried liquid drops, Nature 389, 827 (1997)

1997

[7] [6]

L. I. Zaichik, V. M. Alipchenkov, and E. G. Sinaiski, Particles in turbulent flows (John Wiley & Sons, 2008)

work page 2008

[8] [7]

C. K. Schmidt, M. Medina-S´ anchez, R. J. Edmondson, and O. G. Schmidt, Engineering microrobots for tar- geted cancer therapies from a medical perspective, Na- ture Communications 11, 5618 (2020)

work page 2020