RECORD
Distributed Learning of Decentralized Control Policies for Articulated Mobile Robots
Robotics · Submitted 24 Jan 2019
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our primary contribution shows individual agents in the A3C algorithm can be defined by independently controlled portions of the robot's body, thus enabling distributed learning on a single robot for efficient hardware implementation.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
State-of-the-art distributed algorithms for reinforcement learning rely on multiple independent agents, which simultaneously learn in parallel environments while asynchronously updating a common, shared policy. Moreover, decentralized control architectures (e.g., CPGs) can coordinate spatially distributed portions of an articulated robot to achieve system-level objectives. In this work, we investigate the relationship between distributed learning and decentralized control by learning decentralized control policies for the locomotion of articulated robots in challenging environments. To this end, we present an approach that leverages the structure of the asynchronous advantage actor-critic (A3C) algorithm to provide a natural means of learning decentralized control policies on a single articulated robot. Our primary contribution shows individual agents in the A3C algorithm can be defined by independently controlled portions of the robot's body, thus enabling distributed learning on a single robot for efficient hardware implementation. We present results of closed-loop locomotion in unstructured terrains on a snake and a hexapod robot, using decentralized controllers learned offline and online respectively. Preprint of the paper submitted to the IEEE Transactions in Robotics (T-RO) journal in October 2018, and accepted for publication as a regular paper in May 2019.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | State-of-the-art distributed algorithms for reinforcement learning rely on multiple independent agents, which simultaneously learn in parallel environments while asynchronously updating a common, shared policy. | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| s1 | Moreover, decentralized control architectures (e.g., CPGs) can coordinate spatially distributed portions of an articulated robot to achieve system-level objectives. | background 1.00 | 0.05 | 0.03 | asserted 0.83 |
| s2 | In this work, we investigate the relationship between distributed learning and decentralized control by learning decentralized control policies for the locomotion of articulated robots in challenging environments. | contribution 0.93 | 0.58 | 0.07 | asserted 0.57 |
| s3 | To this end, we present an approach that leverages the structure of the asynchronous advantage actor-critic (A3C) algorithm to provide a natural means of learning decentralized control policies on a single articulated robot. | method 0.89 | 0.72 | 0.08 | asserted 0.87 |
| s4 | Our primary contribution shows individual agents in the A3C algorithm can be defined by independently controlled portions of the robot's body, thus enabling distributed learning on a single robot for efficient hardware implementation. | contribution 1.00 | 0.91 | 0.87 | asserted 1.00 |
| s5 | We present results of closed-loop locomotion in unstructured terrains on a snake and a hexapod robot, using decentralized controllers learned offline and online respectively. | result 0.81 | 0.96 | 0.11 | asserted 1.00 |
| s6 | Preprint of the paper submitted to the IEEE Transactions in Robotics (T-RO) journal in October 2018, and accepted for publication as a regular paper in May 2019. | background 0.84 | 0.15 | 0.02 | asserted 0.92 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
State-of-the-art distributed algorithms for reinforcement learning rely on multiple independent agents, which simultaneously learn in parallel environments while asynchronously updating a common, shared policy. Moreover, decentralized control architectures (e.g., CPGs) can coordinate spatially distributed portions of an articulated robot to achieve system-level objectives. In this work, we investigate the relationship between distributed learning and decentralized control by learning decentralized control policies for the locomotion of articulated robots in challenging environments. To this end, we present an approach that leverages the structure of the asynchronous advantage actor-critic (A3C) algorithm to provide a natural means of learning decentralized control policies on a single articulated robot. Our primary contribution shows individual agents in the A3C algorithm can be defined by independently controlled portions of the robot's body, thus enabling distributed learning on a single robot for efficient hardware implementation. We present results of closed-loop locomotion in unstructured terrains on a snake and a hexapod robot, using decentralized controllers learned offline and online respectively. Preprint of the paper submitted to the IEEE Transactions in Robotics (T-RO) journal in October 2018, and accepted for publication as a regular paper in May 2019.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 8 sentences without a confident label
Uses the method or tool of 3 papers
-
To overcome this problem, we rely on experience replay [ref] that helps decorrelating the agents' experiences with time during training, similar to our recent multi-robot construction work [ref] .
Prioritized Experience Replay · Introduction · 0.99
-
We used an entropy-based loss function introduced in [ref] to update the policy, $π(a_t | s_t;Ψ_A)$:
Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU · Actor-Critic Network · 1.00
-
During training, we compute the gradients of both functions and optimize the functions using the Adam Optimizer [ref] .
Adam: A Method for Stochastic Optimization · Actor-Critic Network · 1.00
Cites as support 1 paper
-
Stochastic policies are usually more robust to the many sources of noise and uncertainty that are integral to any hardware training and deployment [ref] .
Reinforcement Learning with Deep Energy-Based Policies · Introduction · 0.97
Cites as background 2 papers
-
The policy is generally represented by a function, mapping the current state of an agent to the optimal action the agent should enact; this function is very often approximated by a neural network [ref] .
Playing Atari with Deep Reinforcement Learning · Introduction · 0.96
-
many recent works have focused on decentralized policy learning [ref] , where agents each learn their own policy, which should encompass a measure of agent cooperation at least during training.
Counterfactual Multi-Agent Policy Gradients · Multi-Agent Reinforcement Learning (MARL) · 0.86
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.