pith. sign in

arxiv: 2605.12689 · v1 · pith:Z5N2KXTBnew · submitted 2026-05-12 · 💻 cs.RO

3D RL-DWA: A Hybrid Reinforcement Learning and Dynamic Window Approach for Goal-Directed Local Navigation in Multi-DoF Robots

Chiara Castellani , Enrico Turco , Domenico Prattichizzo This is my paper

Pith reviewed 2026-05-14 20:17 UTC · model grok-4.3

classification 💻 cs.RO
keywords reinforcement learningdynamic window approachdeformable microrobots3D navigationvascular simulationlocal planningsparse point cloudshybrid control
0
0 comments X

The pith

A hybrid RL and DWA controller improves deformation and path completion for microrobots in 3D constrained spaces.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents a hybrid controller that merges reinforcement learning with the dynamic window approach to steer deformable microrobots through tight three-dimensional environments. The system processes sparse point cloud data to choose both movement and shape changes, with the goal of reaching a target while maximizing the volume the robot occupies. In a simulated vascular network, the combined method produced higher deformation and near-perfect path success rates across 1080 trials compared with pure reinforcement learning or model-based planning alone. These results held during training and extended to new, unseen layouts under limited sensing. The work focuses on showing that classical local planning can usefully constrain and improve learned policies for high-degree-of-freedom shape-changing robots.

Core claim

The central claim is that integrating reinforcement learning with a Dynamic Window Approach-based local planner significantly enhances both deformation and navigation capabilities of high-degree-of-freedom deformable microrobots compared to pure RL and model-based methods, consistently achieving high deformation and near-perfect path completion in a simulated vascular network while maintaining robust performance in unseen scenarios.

What carries the argument

The RL-DWA hybrid controller, which uses reinforcement learning to select actions inside the dynamic window approach constraints so the microrobot can adjust both its motion and its shape from sparse point cloud inputs.

If this is right

  • The hybrid controller achieves consistently high deformation while navigating during training.
  • Near-perfect path completion rates hold even when the robot encounters new vascular layouts not seen in training.
  • The approach delivers better deformation and navigation than either pure reinforcement learning or traditional model-based planners alone.
  • Robust performance persists under sparse sensory conditions in complex three-dimensional spaces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hybrid learning-plus-classical planners may reduce the amount of real-world data needed to achieve reliable navigation in shape-changing systems.
  • The same structure could be tested on other multi-DoF robots that must alter form to pass through narrow passages.
  • Future trials could measure how contact forces and sensor noise in physical setups alter the policies learned in simulation.

Load-bearing premise

The simulated vascular network and sparse point-cloud sensor model are representative enough of real contact forces, dynamics, and sensing noise that performance gains transfer to physical robots.

What would settle it

Deploy the trained hybrid controller on a physical deformable microrobot inside a real vascular phantom and measure whether deformation levels and path completion rates match or exceed the simulation results under comparable goal and obstacle conditions.

Figures

Figures reproduced from arXiv: 2605.12689 by Chiara Castellani, Domenico Prattichizzo, Enrico Turco.

Figure 1
Figure 1. Figure 1: Proposed framework for adaptive 3D local navigation. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Deformations of the robot along the three axes of its [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training and test scenario. Yellow spheres placed at [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Learning curves of each control strategy, showing the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of quantitative metrics. Median values and [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
read the original abstract

In this paper, we present a novel hybrid approach that combines Reinforcement Learning (RL) with Dynamic Window Approach (DWA) for adaptive 3D local navigation of high-degree-of-freedom robotic systems. Our method leverages sparse point cloud data to dynamically adjust both the motion and the shape of a deformable microrobot, enabling the system to navigate toward a goal in complex, constrained environments while maximizing the occupied volume. We evaluate our framework in a simulated vascular network. Experimental results, based on 1080 trials, indicate that integrating RL with a DWA-based local planner significantly enhances both deformation and navigation capabilities compared to a pure RL and a model-based methods. In particular, the proposed autonomous controller consistently achieves high deformation and near-perfect path completion during training and maintains robust performance in unseen scenarios. These findings highlight the potential of hybrid planning strategies for efficient and adaptive 3D navigation under sparse sensory conditions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes a hybrid 3D RL-DWA controller that combines reinforcement learning with the dynamic window approach to enable goal-directed local navigation for high-DoF deformable microrobots. Using sparse point-cloud observations, the method simultaneously optimizes robot shape (deformation) and motion in constrained 3D environments. Evaluation in a simulated vascular network across 1080 trials shows the hybrid controller achieving higher deformation and near-perfect path completion than pure RL and model-based baselines, with maintained performance in unseen scenarios.

Significance. If the simulation results transfer, the hybrid RL-DWA formulation provides a practical way to fuse learned deformation policies with model-based local planning for sparse-sensing navigation tasks. This could be relevant for medical microrobotics and other high-DoF systems operating under partial observability, where pure RL struggles with local constraints and pure DWA lacks adaptive shape control.

major comments (2)
  1. [Abstract and Evaluation] Abstract and Evaluation section: the claim of statistically significant improvement over baselines rests on 1080 trials, yet no error bars, confidence intervals, hypothesis tests, or training-seed variance are reported, so the magnitude and reliability of the reported gains cannot be assessed.
  2. [Simulation setup and results] Simulation setup and results: the central claim that the hybrid controller 'maintains robust performance in unseen scenarios' is supported only by trials inside a single simulated vascular network with a sparse point-cloud sensor model; without any analysis of how contact forces, tissue compliance, fluid drag, or sensor noise compare to physical conditions, the performance advantage may be an artifact of the simulator rather than a property of the controller.
minor comments (2)
  1. [Method] Clarify the precise state representation passed to the RL policy and the exact form of the reward that balances deformation volume against goal progress and collision avoidance.
  2. [Method] Specify the DWA parameter ranges and how they are modulated by the RL output; the integration point between the learned policy and the local planner is described at a high level but lacks implementation equations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address the two major comments point-by-point below, clarifying the scope of our simulation study while committing to improvements in statistical reporting and discussion of limitations.

read point-by-point responses

  1. Referee: [Abstract and Evaluation] Abstract and Evaluation section: the claim of statistically significant improvement over baselines rests on 1080 trials, yet no error bars, confidence intervals, hypothesis tests, or training-seed variance are reported, so the magnitude and reliability of the reported gains cannot be assessed.

    Authors: We agree that the current presentation lacks sufficient statistical detail to fully support claims of improvement. In the revised manuscript we will add error bars, 95% confidence intervals, and aggregated results across multiple training seeds (with variance reported) in the Evaluation section and associated figures. This will allow readers to assess both the magnitude and reliability of the performance differences. revision: yes

  2. Referee: [Simulation setup and results] Simulation setup and results: the central claim that the hybrid controller 'maintains robust performance in unseen scenarios' is supported only by trials inside a single simulated vascular network with a sparse point-cloud sensor model; without any analysis of how contact forces, tissue compliance, fluid drag, or sensor noise compare to physical conditions, the performance advantage may be an artifact of the simulator rather than a property of the controller.

    Authors: The evaluation is explicitly a simulation study; 'unseen scenarios' denotes novel goal locations and path segments within the same vascular network model. We acknowledge that no direct comparison to physical contact forces, tissue compliance, fluid drag, or sensor noise is provided. In revision we will expand the Discussion to detail the simulator's modeling assumptions for these phenomena and to state clearly that physical transfer remains an open question for future work. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical comparisons are independent of fitted inputs

full rationale

The paper reports performance from 1080 direct simulation trials comparing the hybrid RL-DWA controller against pure RL and model-based baselines. No equations, parameter fits, or self-citations are invoked to derive the claimed gains in deformation or path completion; results are presented as raw experimental outcomes. The derivation chain consists solely of training and evaluation procedures whose outputs are not algebraically forced by their inputs. This is the standard non-circular case for an empirical robotics paper.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The approach rests on standard assumptions of reinforcement learning for continuous control and the geometric validity of the dynamic window approach; no new free parameters or invented entities are introduced in the abstract.

axioms (2)
  • domain assumption Reinforcement learning policies trained in simulation can produce effective continuous control for shape and motion of a deformable body.
    Invoked to justify training the RL component on the simulated vascular task.
  • domain assumption The dynamic window approach can be extended to select both velocity and deformation commands without violating kinematic constraints.
    Required for the hybrid planner to operate on a shape-changing robot.

pith-pipeline@v0.9.0 · 5467 in / 1267 out tokens · 47683 ms · 2026-05-14T20:17:07.159888+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 1 internal anchor

  1. [1]

    Path planning techniques for mobile robots: Review and prospect,

    L. Liu, X. Wang, X. Yang, H. Liu, J. Li, and P. Wang, “Path planning techniques for mobile robots: Review and prospect,”Expert Systems with Applications, vol. 227, p. 120254, 2023

    work page 2023

  2. [2]

    Laugier and R

    C. Laugier and R. Chatila,Autonomous navigation in dynamic environ- ments. Springer, 2007, vol. 35

    work page 2007

  3. [3]

    Fast motion planning for high-dof robot systems using hierarchical system identification,

    B. Jia, Z. Pan, and D. Manocha, “Fast motion planning for high-dof robot systems using hierarchical system identification,” in2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 5140–5147

    work page 2019

  4. [4]

    A rrt based path planning scheme for multi-dof robots in unstructured environments,

    M. Kang, Q. Chen, Z. Fan, C. Yu, Y . Wang, and X. Yu, “A rrt based path planning scheme for multi-dof robots in unstructured environments,” Computers and Electronics in Agriculture, vol. 218, p. 108707, 2024

    work page 2024

  5. [5]

    The dynamic window approach to collision avoidance,

    D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,”IEEE Robotics & Automation Magazine, vol. 4, no. 1, pp. 23–33, 1997

    work page 1997

  6. [6]

    Deep reinforcement learning based mobile robot navigation: A review,

    K. Zhu and T. Zhang, “Deep reinforcement learning based mobile robot navigation: A review,”Tsinghua Science and Technology, vol. 26, no. 5, pp. 674–691, 2021

    work page 2021

  7. [7]

    Improvement of dynamic window approach using reinforcement learning in dynamic environments,

    J. Kim and G.-H. Yang, “Improvement of dynamic window approach using reinforcement learning in dynamic environments,”International Journal of Control, Automation and Systems, vol. 20, 08 2022

    work page 2022

  8. [8]

    Td3 based collision free motion planning for robot navigation,

    H. Liu, Y . Shen, C. Zhou, Y . Zou, Z. Gao, and Q. Wang, “Td3 based collision free motion planning for robot navigation,” in2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE). IEEE, 2024, pp. 247–250

    work page 2024

  9. [9]

    Dynamic adaptive dynamic window approach,

    M. Dobrevski and D. Sko ˇcaj, “Dynamic adaptive dynamic window approach,”IEEE Transactions on Robotics, vol. 40, pp. 3068–3081, 2024

    work page 2024

  10. [10]

    Combined three dimensional locomotion and deformation of functional ferrofluidic robots,

    F. Xinjian, Y . Zhang, Z. Wu, H. Xie, L. Sun, and Z. Yang, “Combined three dimensional locomotion and deformation of functional ferrofluidic robots,”Nanoscale, vol. 15, 11 2023

    work page 2023

  11. [11]

    Deformable ferrofluid microrobot with omnidi- rectional self-adaptive mobility,

    Y . Ji, C. Gan, Y . Dai, X. Bai, Z. Zhu, L. Song, W. Luyao, H. Chen, J. Zhong, and L. Feng, “Deformable ferrofluid microrobot with omnidi- rectional self-adaptive mobility,”Journal of Applied Physics, vol. 131, p. 64701, 02 2022

    work page 2022

  12. [12]

    A survey on swarm microrobotics,

    L. Yang, J. Yu, S. Yang, B. Wang, B. J. Nelson, and L. Zhang, “A survey on swarm microrobotics,”IEEE Transactions on Robotics, vol. 38, no. 3, pp. 1531–1551, 2022

    work page 2022

  13. [13]

    A comprehensive review on autonomous navigation,

    S. Nahavandi, R. Alizadehsani, D. Nahavandi, S. Mohamed, N. Mohajer, M. Rokonuzzaman, and I. Hossain, “A comprehensive review on autonomous navigation,”ACM Comput. Surv., vol. 57, no. 9, May

    work page

  14. [14]

    Available: https://doi.org/10.1145/3727642 1

    [Online]. Available: https://doi.org/10.1145/3727642

    work page doi:10.1145/3727642

  15. [15]

    Actor-critic model predictive control,

    A. Romero, Y . Song, and D. Scaramuzza, “Actor-critic model predictive control,” in2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 14 777–14 784

    work page 2024

  16. [16]

    Dwa- 3d: A reactive planner for robust and efficient autonomous uav navigation in confined environments,

    J. Bes, J. Dendarieta, L. Riazuelo, and L. Montano, “Dwa- 3d: A reactive planner for robust and efficient autonomous uav navigation in confined environments,”Robotics and Autonomous Systems, vol. 195, p. 105196, 2026. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0921889025002933

    work page 2026

  17. [17]

    3d global dynamic window approach for navigation of autonomous underwater vehicles,

    I. Tusseyeva, S.-G. Kim, and Y .-G. Kim, “3d global dynamic window approach for navigation of autonomous underwater vehicles,”Interna- tional Journal of Fuzzy Logic and Intelligent Systems, vol. 13, 06 2013

    work page 2013

  18. [18]

    Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships,

    C. Wu, W. Yu, G. Li, and W. Liao, “Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships,”Ocean Engineering, vol. 284, p. 115208, 2023

    work page 2023

  19. [19]

    Dwa-rl: Dynamically feasible deep reinforcement learning policy for robot navigation among mobile obstacles,

    U. Patel, N. K. S. Kumar, A. J. Sathyamoorthy, and D. Manocha, “Dwa-rl: Dynamically feasible deep reinforcement learning policy for robot navigation among mobile obstacles,” in2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 6057–6063

    work page 2021

  20. [20]

    Reinforcement based mo- bile robot path planning with improved dynamic window approach in unknown environment,

    L. Chang, L. Shan, C. Jiang, and Y . Dai, “Reinforcement based mo- bile robot path planning with improved dynamic window approach in unknown environment,”Autonomous robots, vol. 45, no. 1, pp. 51–76, 2021

    work page 2021

  21. [21]

    Uav path planning employing mpc-reinforcement learning method considering collision avoidance,

    M. Ramezani, H. Habibi, H. V ooset al., “Uav path planning employing mpc-reinforcement learning method considering collision avoidance,” arXiv preprint arXiv:2302.10669, 2023

    work page arXiv 2023

  22. [22]

    Reinforcement learning- based nmpc for tracking control of asvs: Theory and experiments,

    A. B. Martinsen, A. M. Lekkas, and S. Gros, “Reinforcement learning- based nmpc for tracking control of asvs: Theory and experiments,” Control Engineering Practice, vol. 120, p. 105024, 2022

    work page 2022

  23. [23]

    Adaptive pattern and motion control of magnetic microrobotic swarms,

    J. Yu, L. Yang, X. Du, H. Chen, T. Xu, and L. Zhang, “Adaptive pattern and motion control of magnetic microrobotic swarms,”IEEE Transactions on Robotics, vol. 38, no. 3, pp. 1552–1570, 2022

    work page 2022

  24. [24]

    Hierarchical planning with deep rein- forcement learning for 3d navigation of microrobots in blood vessels,

    Y . Yang, M. Bevan, and B. Li, “Hierarchical planning with deep rein- forcement learning for 3d navigation of microrobots in blood vessels,” Advanced Intelligent Systems, vol. 4, 09 2022

    work page 2022

  25. [25]

    Autonomous navigation of magnetic microrobots with improved planning and control in complex environments,

    Y . Liu, H. Wang, X. Wu, J. Qu, X. Liu, and Q. Fan, “Autonomous navigation of magnetic microrobots with improved planning and control in complex environments,”IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 2421–2432, 2025

    work page 2025

  26. [26]

    Cargo capture and transport by colloidal swarms,

    Y . Yang and M. A. Bevan, “Cargo capture and transport by colloidal swarms,”Science Advances, vol. 6, no. 4, p. eaay7679, 2020

    work page 2020

  27. [27]

    Blood vessel [3d model],

    FablabNDMC, “Blood vessel [3d model],” https://sketchfab.com/ 3d-models/blood-vessel-09f09ebc66e14e2dabebbe9514a7a88a, 2025, accessed: 2025-09-11

    work page 2025

  28. [28]

    Proximal Policy Optimization Algorithms

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [29]

    Reducing cognitive load in teleoperating swarms of robots through a data-driven shared control approach,

    E. Turco, C. Castellani, V . Bo, C. Pacchierotti, D. Prattichizzo, and T. L. Baldi, “Reducing cognitive load in teleoperating swarms of robots through a data-driven shared control approach,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 4731–4738

    work page 2024

  30. [30]

    Real-time imaging and tracking of microrobots in tissues using ultrasound phase analysis,

    S. Pane, V . Iacovacci, E. Sinibaldi, and A. Menciassi, “Real-time imaging and tracking of microrobots in tissues using ultrasound phase analysis,”Applied Physics Letters, vol. 118, no. 1, 2021

    work page 2021

  31. [31]

    Automated in vivo navigation of magnetic-driven microrobots using oct imaging feedback,

    D. Li, D. Dong, W. Lam, L. Xing, T. Wei, and D. Sun, “Automated in vivo navigation of magnetic-driven microrobots using oct imaging feedback,”IEEE Transactions on Biomedical Engineering, vol. 67, no. 8, pp. 2349–2358, 2020

    work page 2020