Pith. sign in

REVIEW 9 cited by

Simplifying Deep Temporal Difference Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.04811 v6 pith:EYTSET24 submitted 2024-07-05 cs.LG

Matteo Gallici , Mattie Fellows , Benjamin Ellis , Bartomeu Pou , Ivan Masmitja , Jakob Nicolaus Foerster , Mario Martin This is my paper
classification cs.LG
keywords bufferoff-policyq-learningreplayalgorithmdeeplargenetwork
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Q-learning played a foundational role in the field reinforcement learning (RL). However, TD algorithms with off-policy data, such as Q-learning, or nonlinear function approximation like deep neural networks require several additional tricks to stabilise training, primarily a large replay buffer and target networks. Unfortunately, the delayed updating of frozen network parameters in the target network harms the sample efficiency and, similarly, the large replay buffer introduces memory and implementation overheads. In this paper, we investigate whether it is possible to accelerate and simplify off-policy TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms without the need for a target network or replay buffer, even with off-policy data. Empirically, we find that online, parallelised sampling enabled by vectorised environments stabilises training without the need for a large replay buffer. Motivated by these findings, we propose PQN, our simplified deep online Q-Learning algorithm. Surprisingly, this simple algorithm is competitive with more complex methods like: Rainbow in Atari, PPO-RNN in Craftax, QMix in Smax, and can be up to 50x faster than traditional DQN without sacrificing sample efficiency. In an era where PPO has become the go-to RL algorithm, PQN reestablishes off-policy Q-learning as a viable alternative.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Replacing the hard target-copy in DQN with a Q-value-sensitivity-weighted merge of the last K network copies yields competitive Atari performance, but only marginally beats architecture-matched baselines.

  2. TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning

    cs.MA 2026-02 unverdicted novelty 6.0 of 10

    TABX is a JAX-based, GPU-accelerated, configurable multi-agent battle simulator that lets researchers vary units, terrain, and physics to benchmark cooperative MARL algorithms.

  3. Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks

    cs.RO 2025-07 conditional novelty 6.0 of 10

    The paper introduces MTBench, a GPU-accelerated benchmark for massively parallel multi-task RL, and reports experiments suggesting on-policy methods outperform off-policy baselines while value learning limits MTRL per...

  4. Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    OIR relabels failed trajectories via an LLM into open-ended instructions and trains a unified instruction-following policy, outperforming PQN and ELLM on Craftax.

  5. The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    In PPO, scaling data by adding parallel environments improves final performance and network stability more than scaling rollout length, across Atari, Procgen, and Isaac Gym.

  6. Universal Value-Function Uncertainties

    cs.LG 2025-05 conditional novelty 6.0 of 10

    UVU measures value-function uncertainty by the TD-trained prediction error between an online network and a fixed random target, and in the infinite-width NTK limit this error exactly matches ensemble variance.

  7. Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Bayesian deep Q-learning exhibits a cold posterior effect, caused partly by misspecified Gaussian priors, and Laplace or meta-learned priors improve performance.

  8. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

  9. FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control

    cs.RO 2025-05 conditional novelty 4.0 of 10

    FastTD3, a tuned TD3 with parallel environments and large-batch distributional critic updates, solves humanoid benchmark tasks in under 3 hours on one GPU.

Pith tools