REVIEW 18 cited by
The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at \url{https://github.com/marlbenchmark/on-policy}.
Forward citations
Cited by 18 Pith papers
-
Multi-Agent Reinforcement Learning for SLA-Aware Network Slicing in UAV-Enabled MEC
A predictive MAPPO framework with an SLA-aware reward improves slice SLA stability in UAV-enabled MEC, approaching an informed oracle in simulations.
-
Offline Nash Solvers Meet Online Tree Search in Multi-Agent Games on Graphs
Primitive-Guided Tree Search combines offline exact Nash solutions of 1v1/2v1 subgames with online SM-MCTS to produce coordinated multi-agent pursuit policies that outperform PSRO and neural MCTS baselines.
-
ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning
ASALT uses observation-level and state-level adapters to align mismatched dimensionalities into a shared embedding for transferring actors and critics in MARL, showing improved sample efficiency and reduced negative t...
-
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
A robust semi-Markov RL agent with MILP feasibility projection and Wasserstein ambiguity set achieves $1.22M net profit on an NYC EV simulator with zero feeder violations, outperforming heuristic and other RL baselines.
-
Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking
Robustness to varied vehicle topologies, not partial observability or coordination, is the dominant obstacle for MARL in C-V2X spectrum sharing.
-
Towards Microgrid Resilience Enhancement via Mobile Power Sources and Repair Crews: A Multi-Agent Reinforcement Learning Approach
A hierarchical multi-agent reinforcement learning method coordinates mobile power sources and repair crews to restore loads after extreme events in coupled power-transport networks.
-
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.
-
COLMAR: Cooperative View Policy Learning for Multi-Agent Active 3D Reconstruction
A shared PPO policy with overlap-aware rewards improves multi-agent active 3D reconstruction coverage and accuracy in simulated indoor scenes.
-
Multi-Agent Reinforcement Learning for SLA-Aware Network Slicing in UAV-Enabled MEC
Predictive MAPPO with D3QL mobility forecasts reduces slice SLA violation probability and duration in UAV-MEC simulations versus greedy, GA, and random baselines, nearing an informed oracle when forecasts are accurate.
-
Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination
HetNet achieves 5.84% to 707.65% performance gains and 200x bandwidth reduction over baselines in heterogeneous multi-agent robot teams via graph-attention networks and binarized messaging.
-
Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning
Multi-agent RL produces robot policies that form balanced supporting shapes under arbitrary objects for cooperative transport and generalize across cluttered scenes and complex geometries.
-
Enhancing the MADDPG Algorithm for Multi-Agent Learning via Action Inference and Importance Sampling
Action inference and geometric importance sampling enhance MADDPG, yielding better stability, cooperation, and exploration efficiency on the discrete Predator-Prey benchmark.
-
Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering
A decoupled estimator combining gated dynamics learning and recursive Kalman filtering improves robustness of pre-trained MARL policies under stale observations and message loss.
-
Coordination Architecture Shapes Continuous Demand Response Outcomes in Building Districts
In a 25-building district simulation, the hybrid MPC-SAC architecture delivered the strongest balance of load tracking accuracy (4.8% NMBE), thermal comfort (16.8% exceedance), and lowest spatial variability compared ...
-
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
PD-RSAC, a distributionally robust SAC variant with GCN encoder and MILP constraint projection, reports $1.22M net profit on an NYC taxi-based EV simulator while achieving zero feeder violations, outperforming heurist...
-
Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
Self-supervised multi-agent goal-reaching, where each agent independently learns a contrastive critic of its own observations, achieves cooperation and exploration in sparse-reward MARL tasks where standard baselines fail.
-
$\alpha$-fair heterogeneous agent reinforcement learning
Introduces α-fair HATRPO and HAPPO algorithms that integrate α-fairness into HATRL via a weighted advantage function while claiming to preserve convergence to Nash equilibria.
-
Multi-agent Reinforcement Learning for Robotized Coral Reef Sample Collection
An RL controller for coral sample collection, trained in Unity, is transferred zero-shot to a physical BlueROV2 using real-time underwater motion capture to drive the digital twin.
Discussion (0). Sign in to comment.