Pith. sign in

REVIEW 5 cited by

Massively Parallel Methods for Deep Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1507.04296 v2 pith:J54V66PV submitted 2015-07-15 cs.LG cs.AIcs.DCcs.NE

Massively Parallel Methods for Deep Reinforcement Learning

Arun Nair , Praveen Srinivasan , Sam Blackwell , Cagdas Alcicek , Rory Fearon , Alessandro De Maria , Vedavyas Panneershelvam , Mustafa Suleyman
show 6 more authors
Charles Beattie Stig Petersen Shane Legg Volodymyr Mnih Koray Kavukcuoglu David Silver
This is my paper
classification cs.LG cs.AIcs.DCcs.NE
keywords distributedgamesarchitecturedeeplearningparallelalgorithmbehaviour
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

    cs.LG 2019-11 accept novelty 8.0

    MuZero matches or exceeds AlphaZero-level performance in Go, Chess, Shogi and sets a new state of the art on 57 Atari games by learning a model that directly supports planning rather than reconstructing full environme...

  2. Placeto: Learning Generalizable Device Placement Algorithms for Distributed Machine Learning

    cs.LG 2019-06 unverdicted novelty 7.0

    Placeto learns generalizable RL policies for device placement via iterative improvements and graph embeddings, needing up to 6.1x fewer steps than prior methods and applying to unseen graphs without retraining.

  3. Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

    cs.LG 2026-07 unverdicted novelty 6.0

    Under smoothness assumptions, stale rollouts in async GRPO bias the surrogate gradient by O(S * eta) and yield the stability bound eta << min(R_batch / (S * G_upd), R_crit / (T * G_upd)).

  4. Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

    cs.LG 2026-07 unverdicted novelty 5.0

    Stale rollouts introduce O(S * eta) surrogate-gradient bias in async GRPO, yielding stability condition eta << min{R_batch / (S * G_upd), R_crit / (T * G_upd)} under smoothness assumptions.

  5. ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information

    cs.LG 2026-06 unverdicted novelty 4.0

    ASymPO normalizes token losses by average current-policy negative log-probability to restore zero-sum balance in asynchronous LLM RL without behavior information.