pith. sign in

arxiv: 1803.03835 · v1 · pith:CIAMJRBCnew · submitted 2018-03-10 · 💻 cs.LG

Kickstarting Deep Reinforcement Learning

Simon Schmitt , Jonathan J. Hudson , Augustin Zidek , Simon Osindero , Carl Doersch , Wojciech M. Czarnecki , Joel Z. Leibo , Heinrich Kuttler
show 3 more authors
Andrew Zisserman Karen Simonyan S. M. Ali Eslami
This is my paper
classification 💻 cs.LG
keywords agentkickstartingagentsperformancestudenttrainingallowkickstarted
0
0 comments X
read the original abstract

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  2. Red Teaming Language Models with Language Models

    cs.CL 2022-02 conditional novelty 7.0

    One language model can generate diverse test cases to automatically uncover tens of thousands of harmful behaviors, including offensive replies and privacy leaks, in a large target language model.

  3. Proximal Policy Distillation

    cs.LG 2024-07 conditional novelty 6.0

    PPD integrates PPO into policy distillation so the student collects and uses its own rewards, yielding better sample efficiency and robustness than standard student-distill or teacher-distill on ATARI, Mujoco, and Pro...

  4. Attentive Multi-Task Deep Reinforcement Learning

    cs.LG 2019-07 unverdicted novelty 6.0

    Attention mechanism dynamically groups task knowledge at state granularity in multi-task DRL to enable positive transfer and avoid negative transfer, matching or exceeding prior methods with fewer parameters.