Pith. sign in

REVIEW 10 cited by

Dropout Q-Functions for Doubly Efficient Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.02034 v2 pith:6RLUIZWF submitted 2021-10-05 cs.LG cs.AI

Takuya Hiraoka , Takahisa Imagawa , Taisei Hashimoto , Takashi Onishi , Yoshimasa Tsuruoka This is my paper
classification cs.LGcs.AI
keywords efficiencyredqdropoutefficientq-functionssamplecomputationalcomputationally
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    SOPE uses an actor-aligned OPE signal on a held-out validation split to dynamically stop offline stabilization phases in online RL, improving performance up to 45.6% and cutting TFLOPs up to 22x on 25 Minari tasks.

  2. Learning to Play Piano in the Real World

    cs.RO 2025-03 unverdicted novelty 7.0 of 10

    A Sim2Real2Sim learning pipeline enables a real-world dexterous robot to play piano pieces including Happy Birthday and Ode to Joy with an average F1-score of 0.881.

  3. Debiased Model-based Representations for Sample-efficient Continuous Control

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DR.Q debiases model-based representations for Q-learning by maximizing mutual information between state-action and next-state representations and applying faded prioritized experience replay, achieving competitive or ...

  4. SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

    cs.LG 2026-05 conditional novelty 6.0 of 10

    SOPE dynamically controls offline training length in online RL using actor-aligned OPE on validation data to stop when benefits saturate, achieving up to 45.6% better performance and 22x less computation on Minari tasks.

  5. Reinforcement learning entangling operations on spin qubits

    quant-ph 2025-08 conditional novelty 6.0 of 10

    An RL agent learns robust CNOT entangling pulses for realistically modeled GaAs singlet-triplet qubits and matches a gradient-based benchmark in simulation.

  6. Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model

    cs.LG 2025-07 conditional novelty 6.0 of 10

    ReMix cuts reinforcement finetuning rollout volume by 30x to 450x on math reasoning by mixing historical and on-policy data with a convex KL constraint and a mid-training switch to on-policy updates.

  7. ReBRAC-v2: The Return of the King

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.

  8. Distributional Value Estimation Without Target Networks for Robust Quality-Diversity

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    QDHUAC is a distributional, target-free QD-RL method that enables stable high-UTD training and competitive performance on Brax locomotion tasks using far fewer environment steps than prior approaches.

  9. Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 5.0 of 10

    LoRA applied to critics in SAC and FastTD3 reduces critic loss and yields best or competitive policy performance on most evaluated tasks.

  10. Deep Reinforcement Learning: From First Principles to Reasoning Models

    eess.SY 2026-07 unverdicted novelty 1.0 of 10

    A textbook survey of deep reinforcement learning, from Bellman foundations to DQN, PPO, MuZero, offline RL, and reasoning models, with UAV/SD-WAN examples throughout.

Pith tools