Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Decentralized Consensus Inference-based Hierarchical Reinforcement Learning for Multi-Constrained UAV Pursuit-Evasion Game

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-level reinforcement learning framework, CI-HRL, lets UAV swarms act on a consensus view of a pursuit-evasion game inferred from partial local observations, and reports better cooperative evasion and target-coverage performance than…

desk verdict Solid empirical MARL engineering, but the central policy-gradient theorem is mathematically wrong and the paper should not be cited as is. read the letter →

arxiv 2506.18126 v1 pith:VDNO54L2 submitted 2025-06-22 cs.AI

Xiang Yuming , Li Sizhao , Li Rongpeng , Zhao Zhifeng , Zhang Honggang This is my paper · ORCID
classification cs.AI
keywords hierarchicalreinforcementlearningmulti-agentpursuit-evasiongameUAVswarmconsensusinferenceformationcontrolpolicydistillationcommunication-limitedMARL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a hierarchical reinforcement learning design can solve a multi-constrained pursuit-evasion task—where a UAV swarm must cover multiple target zones while evading a pursuer under limited communication—better than flat multi-agent baselines. The proposed CI-HRL splits the problem: a high-level policy picks anchor points based on a consensus of the global situation, while a low-level policy handles formation, navigation, and obstacle avoidance. In multi-agent particle simulations, CI-HRL reports the best overall high-level reward among the baselines (RH = -281.56), the highest task reward tied with one baseline (Rt = 107.37), and the lowest count of dangerous events (E = 4.46). High-fidelity software-in-the-loop experiments show the swarm adaptively splitting and re-forming, with message-similarity evidence that agents converge on shared target intentions.

What carries the argument

The load-bearing mechanism is the ConsMAC module: each agent runs a GRU-like memory over its own message and observation, weights neighbor messages with a learnable distance encoder and multi-head attention, and feeds the aggregated latent vector to a global estimator that is supervised to reconstruct the global state (training loss $\|\hat{g}_i^{(t)} - g_i^{(t)}\|^2$). The same latent vector becomes the communicated message and the high-level policy's consensus input. On the low level, AT-M alternately trains a formation-and-navigation module and an obstacle-avoidance module, then fine-tunes their combination, and policy distillation merges per-formation teacher policies into one student policy. Theorem 1 supplies the gradient through the nested policies: $\nabla_{\Theta_H} J_{\mathrm{high}} = \mathbb{E}[\nabla_{\Theta_H} \ln \pi_H(p_a|s) Q(s,u) + \nabla_{p_a} \ln \pi_L(u|s,p_a) \nabla_{\Theta_H} \pi_H(p_a|s) Q(s,u)]$, showing how the high-level policy can be updated when low-level actions mediate the environment interaction.

What would settle it

In the same MPE task, retrain ConsMAC with the global-state label replaced by random noise while keeping the rest of CI-HRL unchanged; if the high-level reward stays near -281.56 and message similarity remains high, the consensus mechanism is not what carries the result, whereas a clear drop would confirm the label's role.

Watch

Extended reading notes

Core claim

The central claim is that consensus can be learned and used hierarchically: rather than transmitting raw observations or black-box latent vectors, each agent runs a supervised module, ConsMAC, that aggregates neighbor messages and is trained to predict the global state, producing a shared message that anchors subsequent decisions. This high-level consensus then sets temporary target points for a low-level policy, AT-M, which is trained by alternating formation-only and obstacle-avoidance-only stages and then distilled into a single adaptive-formation controller. The paper reports that this arrangement outperforms state-of-the-art multi-agent baselines when each is paired with the same low-level controller, and that the advantage persists under different adversary strategies and larger swarm sizes.

Load-bearing premise

The high-level consensus module is trained by supervised regression against a global state label that must be available during training; if that label is missing, noisy, or delayed in a real deployment, the consensus inference and the policy built on it have no training signal, and the paper does not evaluate that failure mode.

Editorial extensions

If this is right

  • A single distilled low-level policy covers formation patterns for 3 to 8 agents, so a deployed swarm can lose or gain members without switching control models.
  • High-level decisions are made every 10 time steps and communicated as a 64-dimensional message, shrinking per-step communication overhead relative to raw-observation sharing.
  • Robustness results in software-in-the-loop show formation and navigation completion staying near 78% under 8 m/s wind and 80% under 0.8 m sensing deviation, supporting the framework's practicality beyond idealized particle environments.
  • Per-agent navigation and evasion rewards remain stable when the swarm grows from 8 to 15 agents, suggesting the consensus mechanism scales beyond its training configuration.
  • Stable task and evasion rewards across several adversary policies indicate the high-level policy generalizes across pursuer behaviors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending beyond the paper: the supervised global-state label in ConsMAC could be replaced by a self-supervised consistency loss across agents, which would test whether consensus alone, rather than the oracle label, drives the performance gain.
  • The cosine-similarity analysis of transmitted messages suggests a deployable confidence proxy: message similarity could gate communication frequency or trigger formation switching in real time.
  • Because the high-level action space is restricted to nine discrete anchor points, a continuous-anchor variant might reveal whether discretization limits performance in denser obstacle fields.
  • The reported comparison with a weaker curriculum-learning low-level policy hints that the high-level policy partially compensates for a weaker low-level controller; a direct cross-controller transfer experiment would quantify this modularity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CI-HRL, a two-level decentralized MARL framework for a cooperative evasion and formation coverage (CEFC) task in a multi-UAV pursuit-evasion game. The high-level policy selects anchor points using ConsMAC, a communication module trained to infer a global state from neighbor messages, while the low-level policy uses Alternative Training-based MAPPO (AT-M) and policy distillation to provide formation, navigation, and obstacle-avoidance control. The experiments include MPE simulations and ROS/Gazebo/PX4 software-in-the-loop tests, with comparisons against MAPPO, HASAC, TarMAC, MASIA, NVIF, and variants of CI-HRL. The paper reports that CI-HRL achieves the best high-level reward, task reward, and lowest dangerous-event count in Table V, and demonstrates adaptive formation switching and message-similarity evidence of consensus in SITL.

Significance. If the empirical results are reproducible and statistically supported, the paper makes a useful engineering contribution: it demonstrates a hierarchical MARL architecture with communication-limited decentralized execution, adaptive formation switching, and extensive SITL validation including wind, sensing deviation, and heterogeneous hardware deployment. The ablation study in Sec. V-C2 and the robustness tests in Tables VI-VIII are more extensive than typical for this venue. However, the theoretical foundation is not sound as written: Theorem 1's gradient formula is mathematically invalid, and the main performance tables report single-run averages without variance or statistical tests. The contribution is therefore promising but needs substantive correction before the central claims can be assessed.

major comments (4)
  1. [Sec. IV-C3, Theorem 1 and Appendix B, Eq. (24)] The claimed gradient formula is not a valid policy gradient. In Eq. (24), the proof applies ∇ΘH to πL(u|s',pa) inside F(u|s')=Σpa πL(u|s',pa)πH(pa|s'), producing the spurious chain-rule term ∇pa ln πL(u|s,pa) ∇ΘH πH(pa|s). Since pa is sampled from πH and is not a differentiable function of ΘH, this term has no counterpart in the policy gradient theorem; the correct gradient is E_{pa∼πH,u∼πL}[∇ΘH ln πH(pa|s) Q(s,u)], without the additional term. The extra term is generally nonzero, so Eq. (14) is biased. The proof also assumes pa is re-sampled at every environment step, whereas Sec. V-A2 and Algorithm 2 freeze pa for 10 low-level steps; the single-step policy gradient theorem does not directly apply to that temporally abstracted MDP. Because Theorem 1 is presented as the basis of the high-level RL update, this point is load-bearing: either provide a correct gradient derivation for the temporal-abstract setting, or state explicitly that the implementation uses the standard MAPPO gradient without the extra term and remove Theorem 1.
  2. [Sec. IV-C2, Eq. (11) and Algorithm 2, line 8] In ConsMAC-A, the supervised label g_i(t) is set to the anchor points {p_a,j(t) of all agents}, but those anchor points are generated at Algorithm 2 line 8 by the very high-level policy πH being trained. The regression target is therefore self-referential: the loss trains the encoder to predict the policy's own outputs rather than a ground-truth global state. This does not by itself invalidate the empirical comparisons, but it changes the interpretation of the results from 'consensus inference about an objective global state' to 'a learned communication protocol for sharing intended anchors.' Please clarify the role of the label, and ideally evaluate with a non-self-generated label (e.g., true global positions or observations as in ConsMAC-O) to support the consensus claim.
  3. [Tables III, V, VI, VII, VIII] The empirical comparisons report single-run averages over 50 episodes without standard deviations, confidence intervals, or statistical tests. For example, the main advantage over CI-HRL-w-CL-M in Table V (RH = -281.56 vs -320.69) and the task-reward tie with HASAC+AT-M (Rt = 107.37 vs 105.70) may be within run-to-run noise. Since the central claim is empirical superiority, the paper should report multiple seeds with variance statistics and, where feasible, significance tests.
  4. [Algorithm 2 vs Sec. V-A2] Algorithm 2 lines 6-13 compute p_a,i(t) at every time step t, but Sec. V-A2 states that anchor points are generated by the high-level policy every 10 steps. This discrepancy matters for the correctness of Theorem 1 and for the semantics of the high-level decision interval; the pseudocode and the text should be aligned.
minor comments (5)
  1. [Eq. (17)] The inner summation index appears as 'k* TR' instead of a summation over qualifying groups k*; please fix the notation.
  2. [Table III] The MADDPG rows report '-' for the RL and C columns without explanation; please provide the missing values or state why they are unavailable.
  3. [Ref. [27]] The word 'onine' should be 'online'.
  4. [Notation and figures] The text uses 'UA V', 'UAV', and 'UAVs' inconsistently; also, several figures contain stray '/uni000000...' artifacts in axis labels and captions that should be cleaned before publication.
  5. [Fig. 4(b)] The caption says a random agent drops out every 200 time steps, while the text says 'some randomly selected UAVs are assumed to be no longer observed' with no timing; please align the description.

Circularity Check

1 steps flagged · score 3.0 of 10

ConsMAC-A's supervised 'global state' labels are the anchor points sampled by the very high-level policy that consumes ConsMAC's messages; the rest of the pipeline is externally evaluated.

  1. self definitional [Sec. IV-C2, Eqs. (11)-(12)]
    "We leverage global state g(t) i ∈ s(t) as the label for supervised learning ... LConsMAC(Ψ) = Ei,t[ ||ˆg(t) i − g(t) i ||2 ]. Notably, the specific choice of global state g(t) i could be rather flexible such as the anchor points of all agents g(t) i ={p(t) a,j|∀j∈N} ... denoted as ConsMAC-A and ConsMAC-O, respectively. ... p(t) a,i∼F θHS (E(t) oi, m(t+1) i)."

    In ConsMAC-A, the variant used for the final CI-HRL results, the supervised label g_i is {p_a,j}, the anchor points sampled by the high-level selector. By Eq. (12), p_a,i is sampled as a function of the ConsMAC message m_i+1, which is generated by Eqs. (7)-(10) and trained by Eq. (11). Hence the target that ConsMAC is trained to reconstruct is itself a function of ConsMAC's own output passed through the policy under training. The 'global state' is therefore not an external ground truth for this variant; minimizing Eq. (11) mainly enforces self-consistency between the consensus encoder and the current high-level policy. This is a partial self-definitional loop: the paper also evaluates ConsMAC-O with external observation labels, and the benchmark comparisons are not fitted.

full rationale

The low-level policy pipeline (AT-M plus policy distillation) is externally evaluable: the teacher policies are trained against environment rewards and the distilled student is compared with MADDPG, ORCA-F, CL-M, and AT-M variants on formation stability, navigation, and collision metrics, so there is no circularity there. The high-level RL objective is also trained on environment rewards and compared against MAPPO, HASAC, TarMAC, MASIA, NVIF using the same low-level policy, which provides independent evidence for the central performance claim. The only exhibited self-referential loop is ConsMAC-A's label choice: Eq. (11)'s regression target is, by construction, the output of Eq. (12), which itself consumes ConsMAC's message. That is a partial self-definitional issue, not a fitted parameter renamed as a prediction. Self-citations in the related-work and background sections are not load-bearing. Theorem 1's gradient formula is a mathematical-validity concern rather than a circularity concern, so it does not raise the circularity score.

Assumptions & free parameters 9 free parameters · 7 assumptions · 0 invented entities

The framework relies on the CTDE assumption that global state labels are available during training, on hand-tuned reward weights and environment constants, and on the policy gradient theorem. No invented physical entities are introduced; ConsMAC and AT-M are algorithmic components rather than new entities.

free parameters (9)
  • Low-level reward weights (ωf, ωn, ωc) = (15, 4, 100)
    Chosen by hand to balance formation, navigation, and collision; Table II. Performance comparisons are sensitive to these weights.
  • High-level reward weights (ωt, ωn, ωe) = (10, 0.1, 100)
    Chosen by hand; Table II. The high-level objective in Sec. IV-C1 uses these.
  • Formation lag coefficient ωl = 0.3
    Used in formation reward (15); set manually in Table IX.
  • Urgency decay factor ωd = 0.003
    Controls target urgency decay in (19); set manually in Table IX.
  • Safety distances δa,e, δa,c, δs = (2 m, 0.5 m, 0.2 m)
    Alert and minimum safety distances in evasion and collision rewards; Table IX.
  • Collision constants ωcr1, ωcr2, C1, C2 = (24, 8, 3, 1)
    Shape the collision penalty in (22); Table IX.
  • Anchor point action grid = {-8, 0, 8} x {-8, 0, 8}
    Hand-chosen discrete high-level action space; limits achievable targets and strongly shapes policy behavior; Sec. V-A2.
  • Observation/communication range δobs = 3 m
    Defines the neighbor set ξ_i in Sec. III-A2; hand-chosen environment parameter.
  • High-level decision interval = 10 time steps
    Anchor points are regenerated every 10 steps; this temporal abstraction is not modeled in Theorem 1; Sec. V-A2.
assumptions (7)
  • standard math Policy gradient theorem and GAE are valid for the hierarchical policy
    Invoked in Appendix B to prove Theorem 1; standard result but applied to a setting where anchor actions are sampled less frequently than low-level actions.
  • domain assumption The CEFC problem is a Dec-POMDP with shared global reward R
    Model in Sec. III-A2; all agents optimize the same weighted reward, which requires a cooperative objective design.
  • domain assumption Fixed-altitude 2D flight (uz ≡ 0) is sufficient for the task
    Stated in Sec. III-A1; no altitude planning, so results do not transfer to 3D obstacle fields.
  • domain assumption Global state labels g_i are available during training
    Eq. (11) uses s(t) as the supervised label for ConsMAC; standard CTDE but central to the consensus training signal.
  • ad hoc to paper The hand-tuned reward weights encode the intended task priorities
    Weights in Table II are chosen by the authors and not justified by any calibration procedure; performance claims are relative to this reward.
  • domain assumption The low-level policy trained with random anchors remains competent when coupled with the learned high-level anchors
    Low-level is trained in Sec. IV-B and fixed before high-level fine-tuning; if the anchor distribution shifts, the hierarchy could fail.
  • domain assumption A single PPO adversary is a sufficient training opponent
    Sec. III-A2 uses a default PPO adversary; robustness to stronger adaptive adversaries is deferred to future work in Sec. VI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Consensus Inference-based Hierarchical Reinforcement Learning for Multi-Constrained UAV Pursuit-Evasion Game." pith.science (2026). https://pith.science/paper/VDNO54L2

@misc{pith2026250618126,
  author       = {Pith},
  title        = {Pith review of: Decentralized Consensus Inference-based Hierarchical Reinforcement Learning for Multi-Constrained UAV Pursuit-Evasion Game},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDNO54L2}},
  note         = {Machine review of arXiv:2506.18126}
}
read the original abstract

Multiple quadrotor unmanned aerial vehicle (UAV) systems have garnered widespread research interest and fostered tremendous interesting applications, especially in multi-constrained pursuit-evasion games (MC-PEG). The Cooperative Evasion and Formation Coverage (CEFC) task, where the UAV swarm aims to maximize formation coverage across multiple target zones while collaboratively evading predators, belongs to one of the most challenging issues in MC-PEG, especially under communication-limited constraints. This multifaceted problem, which intertwines responses to obstacles, adversaries, target zones, and formation dynamics, brings up significant high-dimensional complications in locating a solution. In this paper, we propose a novel two-level framework (i.e., Consensus Inference-based Hierarchical Reinforcement Learning (CI-HRL)), which delegates target localization to a high-level policy, while adopting a low-level policy to manage obstacle avoidance, navigation, and formation. Specifically, in the high-level policy, we develop a novel multi-agent reinforcement learning module, Consensus-oriented Multi-Agent Communication (ConsMAC), to enable agents to perceive global information and establish consensus from local states by effectively aggregating neighbor messages. Meanwhile, we leverage an Alternative Training-based Multi-agent proximal policy optimization (AT-M) and policy distillation to accomplish the low-level control. The experimental results, including the high-fidelity software-in-the-loop (SITL) simulations, validate that CI-HRL provides a superior solution with enhanced swarm's collaborative evasion and task completion capabilities.

Figures

Figures reproduced from arXiv: 2506.18126 by the authors.

Figure 1
Figure 1. An illustration of the CEFC task, wherein the UAV swarm (in blue [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of CI-HRL. High-level Policy, in Section IV-C, generates anchor point p (t) a,i: (a) The ConsMAC Module: Process local state z (t) i to infer global state g (t) i and generate the consensus message m (t+1) i , (b) Communication process for M (t) nei,i under limited range; Low-level Policy, in Section IV-B, outputs specific actions u (t) i for task execution: (c) Policy distillation process for adaptive … view at source ↗
Figure 3
Figure 3. An illustration of the pre-defined formations corresponding to different [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Simulation results of adaptive formation. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison to other communication methods. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The ablation results of ConsMAC-A. 0 100 200 300 400 500 Epoch 1000 900 800 700 600 500 400 300 200 R H CI-HRL CI-HRL-DT Episode [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 9
Figure 9. Figure 9: Four typical cooperation scenarios in one episode of SITL. (a), (b), (c), and (d) stand for snapshots at high-level decision steps [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Task overview in Gazebo Simulator for SITL. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Transmitted hidden states m for high-level decision steps and related similarity matrix analysis. (a) Hidden states similarity matrix variation for one episode. (b)∼(e) Illustration of transmitted hidden states and similarity matrices at Steps 1, 19, 50, and 79. TABLE…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 52 canonical work pages

  1. [1]

    Meta reinforcement learning for strategic IoT deployments coverage in disaster-response UA V swarms,

    M. Dhuheir , et al. , “Meta reinforcement learning for strategic IoT deployments coverage in disaster-response UA V swarms,” inIEEE Glob. Comm. Conf., Kuala Lumpur, Malaysia, 2023, pp. 6159–6164

    work page 2023

  2. [2]

    A distributed control framework for a team of unmanned aerial vehicles for dynamic wildfire tracking,

    H. X. Pham , et al. , “A distributed control framework for a team of unmanned aerial vehicles for dynamic wildfire tracking,” in IEEE/RSJ Int. Conf. Intell. Robots Syst., Vancouver, Canada, 2017, pp. 6648–6653

    work page 2017

  3. [3]

    Transfer reinforcement learning for multi-agent pursuit- evasion differential game with obstacles in a continuous environment,

    P. Hu , et al. , “Transfer reinforcement learning for multi-agent pursuit- evasion differential game with obstacles in a continuous environment,” Asian J. Cont. , vol. 26, pp. 2125–2140, 2024

    work page 2024

  4. [4]

    Consensus, cooperative learning, and flocking for multiagent predator avoidance,

    Z. Young , et al. , “Consensus, cooperative learning, and flocking for multiagent predator avoidance,” J. Adv. Robot. Syst. , vol. 17, no. 5, pp. 1–19, Sep. 2020

    work page 2020

  5. [5]

    Multi robot surveillance and planning in limited communication environments,

    V . Kedege , et al. , “Multi robot surveillance and planning in limited communication environments,” in Int. Conf. Age. Artif. Intell. , online, 2022, pp. 139–147

    work page 2022

  6. [6]

    Decentralized function approximated Q-learning in multi-robot systems for predator avoidance,

    R. Konda , et al. , “Decentralized function approximated Q-learning in multi-robot systems for predator avoidance,” IEEE Robot. Autom. Lett. , vol. 5, no. 4, pp. 6342–6349, 2020

    work page 2020

  7. [7]

    A pursuit-evasion game on a real-city virtual simulation platform based on multi-agent reinforcement learning,

    Z. Zhang , et al., “A pursuit-evasion game on a real-city virtual simulation platform based on multi-agent reinforcement learning,” in Chinese Cont. Conf., Tianjin, China, 2023, pp. 6018–6023

    work page 2023

  8. [8]

    Multiagent formation control and dynamic obstacle avoidance based on deep reinforcement learning,

    Z. Yuan , et al. , “Multiagent formation control and dynamic obstacle avoidance based on deep reinforcement learning,” IEEE Trans. Ind. Inform., vol. early access, pp. 1–11, 2025

    work page 2025

Show all 56 references
  1. [9]

    Multi-agent actor-critic for mixed cooperative- competitive environments,

    R. Lowe , et al. , “Multi-agent actor-critic for mixed cooperative- competitive environments,” Adv. Neural Inf. Process. Syst. , vol. 30, pp. 1–16, 2017

    2017

  2. [10]

    Collision-avoiding flocking with multiple fixed-wing UA Vs in obstacle-cluttered environments: A task-specific curriculum- based MADRL approach,

    C. Yan , et al. , “Collision-avoiding flocking with multiple fixed-wing UA Vs in obstacle-cluttered environments: A task-specific curriculum- based MADRL approach,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 35, no. 8, pp. 10 894–10 908, 2023

    2023

  3. [11]

    NVIF: Neighboring variational information flow for cooperative large-scale multiagent reinforcement learning,

    J. Chai , et al. , “NVIF: Neighboring variational information flow for cooperative large-scale multiagent reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 35, no. 12, pp. 17 829–17 841, 2024

    2024

  4. [12]

    Monotonic value function factorisation for deep multi- agent reinforcement learning,

    T. Rashid , et al., “Monotonic value function factorisation for deep multi- agent reinforcement learning,” J. Mach. Learn. Res. , vol. 21, no. 1, pp. 7234–7284, 2020

    2020

  5. [13]

    Efficient multi-agent communication via self-supervised information aggregation,

    C. Guan , et al., “Efficient multi-agent communication via self-supervised information aggregation,” Adv. Neural Inf. Process. Syst., vol. 35, pp. 1– 13, Nov 2022

    2022

  6. [14]

    Maximum entropy heterogeneous-agent reinforcement learning,

    J. Liu , et al. , “Maximum entropy heterogeneous-agent reinforcement learning,” in Proc. Int. Conf. Learn. Repre., Vienna, Austria, May 2024, pp. 1–12

    2024

  7. [15]

    Proximal policy optimization algorithms,

    J. Schulman , et al. , “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017

    2017 arXiv

  8. [16]

    The surprising effectiveness of ppo in cooperative multi- agent games,

    C. Yu , et al. , “The surprising effectiveness of ppo in cooperative multi- agent games,” Adv. Neural Inf. Process. Syst., vol. 35, pp. 24 611–24 624, Nov 2022

    2022

  9. [17]

    Multi-UA V behavior-based formation with static and dynamic obstacles avoidance via reinforcement learning,

    Y . Xie , et al. , “Multi-UA V behavior-based formation with static and dynamic obstacles avoidance via reinforcement learning,” arXiv:2410.18495, 2024

    2024 arXiv

  10. [18]

    Relative distributed formation and obstacle avoidance with multi-agent reinforcement learning,

    Y . Yan, et al. , “Relative distributed formation and obstacle avoidance with multi-agent reinforcement learning,” in Proc. IEEE Int. Conf. Robot. Automat., Philadelphia, PA, USA, May 2022, pp. 1661–1667

    2022

  11. [19]

    Hierarchical and stable multiagent reinforcement learning for cooperative navigation control,

    Y . Jin, et al., “Hierarchical and stable multiagent reinforcement learning for cooperative navigation control,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 1, pp. 90–103, 2021

    2021

  12. [20]

    Self-critical alternate learning based semantic broadcast communication,

    Z. Lu , et al. , “Self-critical alternate learning based semantic broadcast communication,” arXiv:2312.01423, 2023

    2023 arXiv

  13. [21]

    A survey of multi-agent deep reinforcement learning with communication,

    C. Zhu , et al., “A survey of multi-agent deep reinforcement learning with communication,” in Proc. 23rd Int. Conf. Auton. Agents Multiagent Syst., Auckland, New Zealand, 2024, p. 2845–2847

    2024

  14. [22]

    Consensus in multi-agent systems: A review,

    A. Amirkhani , et al. , “Consensus in multi-agent systems: A review,” Artif. Intell. Review , vol. 55, no. 5, pp. 3897–3935, Jun 2022

    2022

  15. [23]

    Data-based optimal consensus control for multiagent systems with policy gradient reinforcement learning,

    X. Yang , et al. , “Data-based optimal consensus control for multiagent systems with policy gradient reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 33, no. 8, pp. 3872–3883, 2021

    2021

  16. [24]

    Learning multiagent communication with back- propagation,

    S. Sukhbaatar , et al. , “Learning multiagent communication with back- propagation,” Adv. Neural Inf. Process. Syst. , vol. 29, p. 2252–2260, 2016

    2016

  17. [25]

    TarMAC: Targeted multi-agent communication,

    A. Das , et al., “TarMAC: Targeted multi-agent communication,” inProc. 36th Int. Conf. Mach. Learn. , California, USA, Jun 2019, pp. 1–9

    2019

  18. [26]

    Communication in multi-agent reinforcement learning: Intention sharing,

    W. Kim , et al. , “Communication in multi-agent reinforcement learning: Intention sharing,” in Proc. Int. Conf. Learn. Repre. , online, Oct 2020, pp. 1–15

    2020

  19. [27]

    ToM2C: Target-oriented multi-agent communication and cooperation with theory of mind,

    Y . Wang, et al. , “ToM2C: Target-oriented multi-agent communication and cooperation with theory of mind,” in Proc. Int. Conf. Learn. Repre., onine, Apr 2022, pp. 1–17

    2022

  20. [28]

    Consensus learning for cooperative multi-agent rein- forcement learning,

    Z. Xu , et al. , “Consensus learning for cooperative multi-agent rein- forcement learning,” in Proc. AAAI Conf. Artif. Intell. , vol. 37, no. 10, Washington, USA, 2023, pp. 11 726–11 734

    2023

  21. [29]

    Hierarchical reinforcement learning: A comprehensive survey,

    S. Pateria , et al., “Hierarchical reinforcement learning: A comprehensive survey,” ACM Computing Surveys , vol. 54, no. 5, pp. 1–35, 2021

    2021

  22. [30]

    Ase: Large-scale reusable adversarial skill embed- dings for physically simulated characters,

    X. B. Peng , et al. , “Ase: Large-scale reusable adversarial skill embed- dings for physically simulated characters,” ACM Trans. Graph., vol. 41, no. 4, pp. 1–17, 2022

    2022

  23. [31]

    Autonomous obstacle avoidance and target tracking of UA V based on deep reinforcement learning,

    G. Xu , et al. , “Autonomous obstacle avoidance and target tracking of UA V based on deep reinforcement learning,” J. Intell. Robot. Syst. , vol. 104, no. 4, pp. 60–79, 2022

    2022

  24. [32]

    Game of drones: Multi-UA V pursuit-evasion game with online motion planning by deep reinforcement learning,

    R. Zhang , et al. , “Game of drones: Multi-UA V pursuit-evasion game with online motion planning by deep reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 34, no. 10, pp. 7900–7909, 2023

    2023

  25. [33]

    Large scale pursuit-evasion under collision avoidance using deep reinforcement learning,

    H. Yang , et al. , “Large scale pursuit-evasion under collision avoidance using deep reinforcement learning,” in IEEE/RSJ Int. Conf. Intell. Robots Syst., Detroit, USA, 2023, pp. 2232–2239

    2023

  26. [34]

    Multi-agent cooperative pursuit-defense strategy against one single attacker,

    Z. Deng , et al., “Multi-agent cooperative pursuit-defense strategy against one single attacker,” IEEE Robot. Autom. Lett. , vol. 5, no. 4, pp. 5772– 5778, 2020

    2020

  27. [35]

    Feudal reinforcement learning,

    P. Dayan , et al. , “Feudal reinforcement learning,” Adv. Neural Inf. Process. Syst., vol. 5, pp. 1–8, 1992

    1992

  28. [36]

    Hierarchical reinforcement learning with the maxq value function decomposition,

    T. G. Dietterich, “Hierarchical reinforcement learning with the maxq value function decomposition,” J. Artif. Intell. Res. , vol. 13, pp. 227– 303, 2000

    2000

  29. [37]

    The option-critic architecture,

    P.-L. Bacon , et al., “The option-critic architecture,” in Proc. AAAI Conf. Artif. Intell., vol. 31, 2017, p. 1726–1734

    2017

  30. [38]

    Learning multi-level hierarchies with hindsight,

    A. Levy , et al. , “Learning multi-level hierarchies with hindsight,” arXiv:1712.00948, 2017

    2017 arXiv

  31. [39]

    Px4: A node-based multithreaded open source robotics framework for deeply embedded platforms,

    L. Meier , et al., “Px4: A node-based multithreaded open source robotics framework for deeply embedded platforms,” in Proc. IEEE Int. Conf. Robot. Automat., Seattle, USA, 2015, pp. 6235–6240

    2015

  32. [40]

    Modelling, identification and control of a quadrotor helicopter,

    T. Bresciani, “Modelling, identification and control of a quadrotor helicopter,” MSc theses, 2008

    2008

  33. [41]

    Long-term tracking of evasive urban target based on intention inference and deep reinforcement learning,

    P. Yan , et al. , “Long-term tracking of evasive urban target based on intention inference and deep reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 35, no. 11, pp. 16 886–16 900, 2024

    2024

  34. [42]

    Flexible formation control using hausdorff distance: A multi-agent reinforcement learning approach,

    C. Pan , et al. , “Flexible formation control using hausdorff distance: A multi-agent reinforcement learning approach,” in Euro. Sign. Proc. Conf., Glasgow, UK, Aug 2022, pp. 972–976

    2022

  35. [43]

    High-dimensional continuous control using gen- eralized advantage estimation,

    J. Schulman , et al. , “High-dimensional continuous control using gen- eralized advantage estimation,” in Proc. Int. Conf. Learn. Repre. , May 2016, pp. 1–17

    2016

  36. [44]

    Policy distillation,

    A. A. Rusu , et al., “Policy distillation,” in Proc. Int. Conf. Learn. Repre., San Juan, Puerto Rico, May 2016, pp. 1–13

    2016

  37. [45]

    Attention is all you need,

    A. Vaswani , et al., “Attention is all you need,” Adv. Neural Inf. Process. Syst., vol. 30, pp. 1–15, Dec 2017

    2017

  38. [46]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. Koenig , et al., “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in IEEE/RSJ Int. Conf. Intell. Robots Syst., vol. 3, Sendai, Japan, 2004, pp. 2149–2154 vol.3

    2004

  39. [47]

    Recent progress on 3gpp 5g positioning,

    Y . Wang, et al., “Recent progress on 3gpp 5g positioning,” in IEEE Veh. Tech. Conf., 2023, pp. 1–6

    2023

  40. [48]

    Precise payload delivery via unmanned aerial vehicles: An approach using object detection algorithms,

    A. Vadduri , et al. , “Precise payload delivery via unmanned aerial vehicles: An approach using object detection algorithms,” arXiv preprint arXiv:2310.06329, 2023

    2023 arXiv

  41. [49]

    Formation control with collision avoidance through deep reinforcement learning using model-guided demonstration,

    Z. Sui , et al., “Formation control with collision avoidance through deep reinforcement learning using model-guided demonstration,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 32, no. 6, pp. 2358–2372, 2020

    2020

  42. [50]

    Xtdrone: A customizable multi-rotor UA Vs simulation platform,

    K. Xiao , et al. , “Xtdrone: A customizable multi-rotor UA Vs simulation platform,” in Int. Conf. Robot. Autom. Sci. , online, 2020, pp. 55–61

    2020

  43. [51]

    Efficient and scalable reinforcement learning for large- scale network control,

    C. Ma , et al. , “Efficient and scalable reinforcement learning for large- scale network control,” Nat. Mach. Intell., vol. 6, no. 9, pp. 1006–1020, 2024. 16

    2024

  44. [52]

    An attention mechanism and adaptive accuracy triple- dependent maddpg formation control method for hybrid uavs,

    J. Wu , et al. , “An attention mechanism and adaptive accuracy triple- dependent maddpg formation control method for hybrid uavs,” IEEE Trans. Intell. Trans. Syst. , vol. 25, no. 9, pp. 11 648–11 663, 2024

    2024

  45. [53]

    Robust deep reinforcement learning through adversar- ial attacks and training: A survey,

    L. Schott , et al., “Robust deep reinforcement learning through adversar- ial attacks and training: A survey,” arXiv:2403.00420, 2024

    2024 arXiv

  46. [54]

    Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers,

    L. Yuan , et al. , “Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers,” in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 10, Washington, USA, 2023, pp. 11 753–11 762

    2023

  47. [55]

    Policy gradient methods for reinforcement learning with function approximation,

    R. S. Sutton , et al., “Policy gradient methods for reinforcement learning with function approximation,” Adv. Neural Inf. Process. Syst. , vol. 12, pp. 1–7, 1999

    1999

  48. [56]

    Learning to utilize shaping rewards: A new approach of reward shaping,

    Y . Hu, et al. , “Learning to utilize shaping rewards: A new approach of reward shaping,” Adv. Neural Inf. Process. Syst. , vol. 33, pp. 15 931– 15 941, 2020

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.