Pith. sign in

REVIEW 3 major objections 5 minor 71 references

An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Offline reinforcement learning in factorisable action spaces can be made practical and scalable by decomposing global actions into per-sub-action utilities, which reduces overestimation bias and lets adapted offline methods match or beat…

desk verdict A solid, honest first benchmark for offline RL with factorised actions, with the caveat that all data comes from factorised policies. read the letter →

arxiv 2411.11088 v1 pith:GPFERAVK submitted 2024-11-17 stat.ML cs.LG

Alex Beeson , David Ireland , Giovanni Montana This is my paper · ORCID
classification stat.MLcs.LG
keywords offlinereinforcementlearningfactorisedactionspacesvaluedecompositionDecQNoverestimationbiasconservativeQ-learningbenchmarksdiscretecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that offline reinforcement learning can work well in factorisable discrete action spaces—where a global action is a combination of independent sub-actions—by representing values with the DecQN decomposition rather than treating every combination as a separate atomic action. The authors argue that factorising turns many out-of-distribution global actions into in-distribution sub-actions, which should lower the overestimation bias that plagues offline Q-learning. To test this, they adapt four established offline methods (BCQ, CQL, IQL, and one-step RL) to the factorised setting and evaluate them on a new benchmark of maze and continuous-control tasks with datasets of varying quality. Their results show these factorised offline methods generally outperform behaviour cloning and plain DecQN, extract expert-level policies from medium-expert data in many tasks, and scale more gracefully than atomic DQN-CQL as the number of bins grows.

What carries the argument

The central object is the DecQN value-decomposition: each sub-action space $i$ has its own utility function $U^i_{\theta_i}(s,a^i)$, and the global action value is the mean of these utilities, $Q_\theta(s,a)=\frac{1}{N}\sum_i U^i_{\theta_i}(s,a^i)$. This identity carries the argument by shrinking the number of values to learn from $\prod_i n_i$ to $\sum_i n_i$ and by making sub-action coverage the relevant statistic for overestimation bias, since a global action can be out-of-distribution while its constituent sub-actions are in-distribution. The offline adaptations (DecQN-BCQ, DecQN-CQL, DecQN-IQL, DecQN-OneStep) all apply their regularisation at the sub-action level, preserving this decomposition while constraining or penalising value estimates.

What would settle it

Train both DecQN-CQL and atomic DQN-CQL on a task where the reward depends on a strong nonlinear interaction between two sub-actions (for example, a reward that is positive only when both sub-actions match) using a medium-expert dataset; if the factorised method cannot recover expert behaviour while the atomic method does, the claimed bias-reduction advantage fails in exactly the regime the paper concedes is its limit.

Watch

Extended reading notes

Core claim

The paper's central claim is that the DecQN value-decomposition, $Q_\theta(s,a) = \frac{1}{N}\sum_{i=1}^N U^i_{\theta_i}(s,a^i)$, is a sound foundation for offline RL in factorisable action spaces because it converts the value-learning problem from a product of sub-actions into a sum, and because a global action that is absent from the dataset can still be composed of sub-actions that are present. That coverage advantage reduces the expected overestimation bias in target values compared with an atomic representation, especially when datasets are small or action spaces large. The authors support this with a noise model in which in-distribution errors are smaller than out-of-distribution errors, and they demonstrate empirically that DecQN-CQL retains performance and low Q-value error as the number of bins increases while DQN-CQL degrades sharply, and that factorised offline methods generally beat behavioural cloning and plain DecQN across their new benchmark.

Load-bearing premise

Everything rests on the assumption that the true Q-value of a global action is well approximated by the average of per-sub-action utilities; if sub-actions interact strongly, that average can be systematically wrong, and the factorised offline methods inherit the error.

Editorial extensions

If this is right

  • Using the same dataset, switching from an atomic to a factorised action representation can reduce expected overestimation bias, which is the main failure mode in offline RL.
  • Factorised offline methods (DecQN-CQL, DecQN-IQL, DecQN-OneStep) consistently match or beat behaviour cloning, and in many tasks they extract expert-level policies from medium-expert datasets.
  • As the number of sub-actions per dimension grows, DecQN-CQL stays near expert performance while atomic DQN-CQL degrades and its training time and GPU memory explode.
  • On low-quality or mixed datasets for complex control tasks, all factorised methods still improve over behaviour cloning, but their gap to expert performance remains visible, pointing to a limit of the approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit is that factorisation should help most when datasets are small and sub-action coverage is high; this can be tested by sweeping dataset size and bin count while holding the task fixed.
  • The same bias-reduction logic suggests factorised offline methods could transfer to high-dimensional discrete control or healthcare dosing, but only in regimes where inter-action dependence is weak; strongly interacting settings would need a decomposition that models interactions rather than averaging.
  • The random-medium-expert results indicate a performance ceiling on complex tasks; combining factorisation with uncertainty estimation or ensembles, as has been done in continuous offline RL, is a natural next step that the paper does not itself test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies offline reinforcement learning in factorised discrete action spaces. It adapts four offline algorithms (BCQ, CQL, IQL, OneStep) to the DecQN value-decomposition framework, introduces a benchmark suite of datasets built from Maze and DeepMind Control tasks, and reports experiments showing that the factorised offline methods generally outperform behaviour cloning and plain DecQN, extract near-expert policies from medium-expert datasets, and scale more gracefully than atomic DQN-CQL in a two-task case study. The authors also provide a theoretical argument, supported by a simulation in Appendix A, that factorised representations can reduce overestimation bias for out-of-distribution actions.

Significance. If the empirical claims hold beyond the current benchmark, this is a useful contribution: it is the first systematic study of offline RL in factorisable action spaces, it provides a new benchmark with open code and datasets, and it reports careful engineering details such as compute time, GPU usage, and a 10-seed check. The paper is also commendably explicit about the limitations of value decomposition (Sections 4.1-4.2). However, the central empirical claim is not fully established because all benchmark datasets are collected by factorised DecQN/REValueD policies, so the experiments cannot separate the benefit of factorised value decomposition from the benefit of a factorised behaviour policy. The absence of atomic or non-factorised data-generation controls is the main weakness.

major comments (3)
  1. [§6, Appendix C] All benchmark datasets are collected by policies trained with DecQN/REValueD (Section 6, Appendix C), i.e. the behaviour policy is factorised by construction and each sub-action is selected independently. The per-sub-action behaviour cloning in DecQN-BCQ/IQL/OneStep and the per-sub-action CQL regularisation are therefore matched exactly to the data-generating process. The empirical gains reported in Figures 5-7 and Tables 7-9 may reflect this matched inductive bias rather than a general benefit of factorised offline RL. The paper does not include any dataset collected by an atomic or non-factorised policy (e.g. DQN or a policy with correlated sub-actions), which is precisely the regime where the coverage argument of Section 4 is most fragile. Please add such control datasets for at least a subset of tasks, or substantially qualify the central empirical claim.
  2. [Table 2, Appendix D] The hyperparameters τ, α, and λ (Table 2) are selected by evaluating the final policy on the same environment/task used for the reported results. While the paper forbids variation across datasets within a task, the selection still uses test-task performance. This can inflate the apparent advantage of the tuned offline methods over behaviour cloning, which does not have these method-specific hyperparameters, and it weakens the cross-method comparisons (e.g. the claimed CQL versus IQL/OneStep differences in Figures 6-7). Please report a sensitivity analysis or select hyperparameters on a validation split using off-policy evaluation, or at least discuss the magnitude of this effect.
  3. [§4, Eqs. (4)-(7), Appendix A] The theoretical bias-reduction argument assumes that both the approximate and the true Q-function decompose exactly as in Eq. (2). Section 4.2 acknowledges that this can fail under strong sub-action interactions, but the paper does not measure the decomposition error on the benchmark tasks, so it is unclear whether the reduction in target-difference bias (Eqs. 6-7) outweighs the approximation error in practice. I suggest adding a Monte Carlo estimate of the true Q-values (as in Appendix F) and reporting the mean squared error of the mean-of-utilities approximation on several tasks, to support the claim that 'for the same dataset we can potentially reduce overestimation bias moving from an atomic action representation to a factorised one' (Section 4).
minor comments (5)
  1. [Eq. (7)] In the second maximum of Eq. (7), the error term is written as \epsilon_in for out-of-distribution sub-actions; this should be \epsilon_out.
  2. [Algorithm 4] The OneStep target in Algorithm 4 mixes state and next-state indices: y = r + 1/N \sum_i \sum_{a_i} \pi^i_{\phi_i}(a_i|s) U^i_{\hat\theta_i}(s', a'_i) should sum over next-state sub-actions a'_i and evaluate \pi_{\phi_i}(a'_i|s').
  3. [Table 7] Table 7 contains a stray '± ± ± ±' in the Maze (Actuators=12) row headers, and the table caption for Table 3 has a typo ('c' at the end of the heading).
  4. [Section 6] The paper states that datasets are made available, but the URL points only to the code repository; please provide a direct link to the datasets for reproducibility.
  5. [Section 4] The notation \epsilon_in(s, a_in^i) and \epsilon_out(s, a_out^i) in Eqs. (6)-(7) is introduced informally; a one-sentence definition would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the overestimation-bias argument is a conditional mathematical derivation from stated assumptions, and the empirical benchmark, while self-generated, is not fitted to its own conclusions.

full rationale

The paper's central theoretical claim in Section 4 is a conditional derivation rather than a circular one. It assumes value estimates carry noise and that true Q-values decompose as in Equation (2), then compares the maximum over atomic-action errors with the mean of per-sub-action maxima. The claimed reduction in overestimation follows mathematically from the stated decomposition and from the coverage property that global out-of-distribution actions can contain in-distribution sub-actions; it is not a fit to the target result. The assumptions are explicitly flagged as approximate in Sections 4.1-4.2 and 8, including the drug-interaction failure mode, so the paper does not present the decomposition as an established fact. Appendix A provides an independent simulation of the bias/variance tradeoff under uniform noise, deriving the same comparison from closed forms rather than from fitted outputs. The self-citations to Ireland and Montana (REValueD) supply a parameter-free target-difference identity and an online-effectiveness observation; the present paper re-derives the relevant equations and runs its own ablation in Appendix H, so those self-citations are not load-bearing. The benchmark datasets are generated by DecQN/REValueD policies, which is a potential covariate-shift or matched-inductive-bias confound for the empirical comparisons, but it is not a circularity of the derivation chain: no prediction is fitted to the data it later 'predicts', and behavioural-cloning baselines provide an external comparator. No step in the paper's derivation reduces by construction to its own inputs, and the limitations the authors state are consistent with a formative, conditional investigation rather than a forced self-referential result.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the decomposition assumption, the noise model, and tuned hyperparameters. These are acknowledged in the paper, but they limit generalisation to strongly coupled action spaces.

free parameters (5)
  • BCQ relative threshold τ = per task values in Table 2, range 0.05-0.75
    Chosen by grid search over {0.025,0.05,0.1,0.25,0.5,0.75} per environment/task (Appendix D); affects which sub-actions are allowed in target values.
  • CQL conservative coefficient α = per task values in Table 2, range 0.25-2
    Chosen by grid search over {0.25,0.5,1,2}; controls conservatism in DecQN-CQL loss (Eq. 8).
  • IQL expectile τ (β in Table 2) = 0.5 in Table 2; search range 0.5-0.8
    Expectile for state-value regression in Algorithm 3; tuned per environment/task.
  • IQL/OneStep balance λ = per task values in Table 2, range 1-50
    Weights RL advantage vs behavioural cloning in policy extraction for DecQN-IQL and DecQN-OneStep; grid searched.
  • Network/training hyperparameters = learning rate 3e-4, batch size 256, hidden size 512, Polyak μ=0.005
    Hand-chosen constants (Appendix D) that affect all methods; not tuned per task, but the results depend on them.
assumptions (5)
  • domain assumption True global Q-function decomposes as the mean of per-sub-action utility functions (Eq. 2).
    Invoked in Section 4 before Eq. 5; the paper's own Sections 4.1-4.2 state it holds only under weak inter-action dependence and approximately factorisable rewards.
  • domain assumption Function-approximation errors for in-distribution actions are smaller than for out-of-distribution actions, and can be treated as additive noise.
    Section 4, Eqs. 4-7; essential to the claim that factorisation reduces overestimation bias.
  • ad hoc to paper For the Appendix A analysis, errors are i.i.d. uniform U(-b,b) in-distribution and U(-kb,kb) out-of-distribution.
    Used to derive closed-form bounds E[Z_dec]≤E[Z_dqn]; not justified empirically for real tasks.
  • domain assumption The benchmark tasks are suitably amenable to factorisation to test the decomposition assumption.
    Section 6 selects maze and DMC tasks that are inherently factorisable; results may not transfer to strongly coupled action spaces.
  • standard math Bellman optimality backup and Q-learning bootstrap provide valid learning targets.
    Section 2.1; standard background used by all algorithms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces." pith.science (2026). https://pith.science/paper/GPFERAVK

@misc{pith2026241111088,
  author       = {Pith},
  title        = {Pith review of: An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPFERAVK}},
  note         = {Machine review of arXiv:2411.11088}
}
read the original abstract

Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offline is mitigating overestimation bias in value estimates for state-action pairs absent from data. Whilst numerous approaches have been proposed in recent years, these tend to focus primarily on continuous or small-scale discrete action spaces. Factorised discrete action spaces, on the other hand, have received relatively little attention, despite many real-world problems naturally having factorisable actions. In this work, we undertake a formative investigation into offline reinforcement learning in factorisable action spaces. Using value-decomposition as formulated in DecQN as a foundation, we present the case for a factorised approach and conduct an extensive empirical evaluation of several offline techniques adapted to the factorised setting. In the absence of established benchmarks, we introduce a suite of our own comprising datasets of varying quality and task complexity. Advocating for reproducible research and innovation, we make all datasets available for public use alongside our code base.

Figures

Figures reproduced from arXiv: 2411.11088 by the authors.

Figure 1
Figure 1. In this simple example there are N = 3 sub-action dimensions, each with two actions {↑, ↓}. In-distribution and out-of-distribution actions/sub-actions are highlighted in green and red, respectively. For a particular state, the dataset contains two global actions. Under atomic representation only actions which match those in the dataset are in-distribution. Under fac￾torised representation, individual sub-actions wh… view at source ↗
Figure 2
Figure 2. Examples of maze environment with different numbers of actuators. The star represents [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Comparisons of performance (left) and computation time (right) for DQN-CQL and [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Comparison of performance for DQN-CQL and DecQN-CQL on the Maze task with [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison on maze task for varying numbers of actuators. For presentation [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison across benchmark for [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison for dog-trot for ni ∈ {3, 10, 30, 50, 75, 100}. Each approach is reasonably resilient to increases in the number of bins, although for “random-medium-expert” datasets extracting a good policy appears to become more challenging as n gets very larg…
Figure 8
Figure 8. Figure 8: Comparing the expectation and variance of target differences under DQN and DecQN [PITH_FULL_IMAGE:figures/full_fig_p036_8.png]
Figure 9
Figure 9. Figure 9: Comparison of estimated errors in Q-values for “cheetah-run-medium-expert” dataset for [PITH_FULL_IMAGE:figures/full_fig_p037_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 52 canonical work pages

  1. [1]

    Uncertainty-based offline reinforcement learning with diversified Q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified Q-ensemble . Advances in neural information processing systems, 34: 0 7436--7447, 2021

    work page 2021

  2. [2]

    Model-based offline planning

    Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Conference on Learning Representations, 2020

    work page 2020

  3. [3]

    Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning

    Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2022

    2022

  4. [4]

    Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning

    Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning. Machine Learning, 113 0 (1): 0 443--488, 2024

    work page 2024

  5. [5]

    Offline RL without off-policy evaluation

    David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. Advances in neural information processing systems, 34: 0 4933--4946, 2021

    work page 2021

  6. [6]

    Learning action representations for reinforcement learning

    Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learning action representations for reinforcement learning. In International conference on machine learning, pages 941--950. PMLR, 2019

    work page 2019

  7. [7]

    Q-transformer: Scalable offline reinforcement learning via autoregressive Q-functions

    Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive Q-functions . In Conference on Robot Learning, pages 3909--3928. PMLR, 2023

    work page 2023

  8. [8]

    The dynamics of reinforcement learning in cooperative multiagent systems

    Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998 0 (746-752): 0 2, 1998

    work page 1998

Show all 71 references
  1. [9]

    Deep visual reasoning - learning to predict action sequences for assembly tasks

    Driess, Ha, and Toussaint. Deep visual reasoning - learning to predict action sequences for assembly tasks. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4645--4651. IEEE, 2020

    2020

  2. [10]

    Value function factorization with dynamic weighting for deep multi-agent reinforcement learning

    Wei Du, Shifei Ding, Lili Guo, Jian Zhang, Chenglong Zhang, and Ling Ding. Value function factorization with dynamic weighting for deep multi-agent reinforcement learning. Information Sciences, 615: 0 191--208, 2022

    2022

  3. [11]

    Deep reinforcement learning in large discrete action spaces

    Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679, 2015

    2015 arXiv

  4. [12]

    Growing action spaces

    Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. In International Conference on Machine Learning, pages 3040--3051. PMLR, 2020

    2020

  5. [13]

    D4RL : Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL : Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  6. [14]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 20132--20145, 2021

    2021

  7. [15]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587--1596. PMLR, 2018

    2018

  8. [16]

    Benchmarking batch deep reinforcement learning algorithms

    Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708, 2019 a

    1910 arXiv

  9. [17]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052--2062. PMLR, 2019 b

    2019

  10. [18]

    Q-learning for robot control

    Chris Gaskett. Q-learning for robot control. PhD thesis, Australian National University, 2002

    2002

  11. [19]

    Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters

    Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35: 0 18267--18281, 2022

    2022

  12. [20]

    Evaluating reinforcement learning algorithms in observational health settings

    Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv:1805.12298, 2018

    2018 arXiv

  13. [21]

    Learning pseudometric-based action representations for offline reinforcement learning

    Pengjie Gu, Mengchen Zhao, Chen Chen, Dong Li, Jianye Hao, and Bo An. Learning pseudometric-based action representations for offline reinforcement learning. In International Conference on Machine Learning, pages 7902--7918. PMLR, 2022

    2022

  14. [22]

    Efficient solution algorithms for factored MDPs

    Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman. Efficient solution algorithms for factored MDPs . Journal of Artificial Intelligence Research, 19: 0 399--468, 2003

    2003

  15. [23]

    Addressing extrapolation error in deep offline reinforcement learning

    Caglar Gulcehre, Sergio G \'o mez Colmenarejo, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, Nando de Freitas, et al. Addressing extrapolation error in deep offline reinforcement learning. Offline Reinforcement Learning Workshop at ...

    2020

  16. [24]

    Rl unplugged: A suite of benchmarks for offline reinforcement learning

    Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio G \'o mez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Proces...

    2020

  17. [25]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018

    2018

  18. [26]

    Distributed prioritized experience replay

    Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018

    2018

  19. [27]

    Learning and planning in complex action spaces

    Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pages 4476--4486. PMLR, 2021

    2021

  20. [28]

    Revalued: Regularised ensemble value-decomposition for factorisable Markov decision processes

    David Ireland and Giovanni Montana. Revalued: Regularised ensemble value-decomposition for factorisable Markov decision processes. In The Twelfth International Conference on Learning Representations, 2023

    2023

  21. [29]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902--9915. PMLR, 2022

    2022

  22. [30]

    Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pa...

    2018

  23. [31]

    Efficient reinforcement learning in factored MDPs

    Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored MDPs . In IJCAI, volume 16, pages 740--747, 1999

    1999

  24. [32]

    MOReL : Model-based offline reinforcement learning

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL : Model-based offline reinforcement learning. Advances in neural information processing systems, 33: 0 21810--21823, 2020

    2020

  25. [33]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  26. [34]

    Al Sallab, Senthil Yogamani, and Patrick Pérez

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 0 (6): 0 4909--4926, 2022

    2022

  27. [35]

    Offline reinforcement learning with Fisher divergence critic regularization

    Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774--5783. PMLR, 2021 a

    2021

  28. [36]

    Offline reinforcement learning with implicit Q-Learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-Learning . In International Conference on Learning Representations, 2021 b

    2021

  29. [37]

    Multi-agent reinforcement learning as a rehearsal for decentralized planning

    Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190: 0 82--94, 2016

    2016

  30. [38]

    Stabilizing off-policy Q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019

    2019

  31. [39]

    Conservative Q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  32. [40]

    Batch reinforcement learning

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Springer, pages 45--73, 2012

    2012

  33. [41]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  34. [42]

    Efficient large-scale fleet management via multi-agent deep reinforcement learning

    Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1774--1783, 2018

    2018

  35. [43]

    Reinforcement learning for clinical decision support in critical care: comprehensive review

    Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, and Mengling Feng. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet research, 22 0 (7): 0 e18477, 2020

    2020

  36. [44]

    Action-quantized offline reinforcement learning for robotic skill learning

    Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action-quantized offline reinforcement learning for robotic skill learning. In 7th Annual Conference on Robot Learning, 2023

    2023

  37. [45]

    Benchmarking reinforcement learning algorithms on real-world robots

    A Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning, pages 561--591. PMLR, 2018

    2018

  38. [46]

    Discrete sequential prediction of continuous actions for deep RL

    Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL . arXiv preprint arXiv:1705.05035, 2017

    2017 arXiv

  39. [47]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  40. [48]

    Anti-exploration by random network distillation

    Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pages 26228--26244. PMLR, 2023

    2023

  41. [49]

    Factored action spaces in deep reinforcement learning

    Thomas Pierrot, Valentin Mac \'e , Jean-Baptiste Sevestre, Louis Monier, Alexandre Laterre, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Factored action spaces in deep reinforcement learning. 2021

    2021

  42. [50]

    Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33: 0 10199--10210, 2020 a

    2020

  43. [51]

    Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 0 (178): 0 1--51, 2020 b

    2020

  44. [52]

    Leveraging factored action spaces for off-policy evaluation

    Aaman Peter Rebello, Shengpu Tang, Jenna Wiens, and Sonali Parbhoo. Leveraging factored action spaces for off-policy evaluation. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023

    2023

  45. [53]

    Is bang-bang control all you need? solving continuous control with Bernoulli policies

    Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with Bernoulli policies. Advances in Neural Information Processing Systems, 34: 0 27209--27...

    2021

  46. [54]

    Solving continuous control via Q-learning

    Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via Q-learning . In The Eleventh International Conference on Learning Representations, 2022

    2022

  47. [55]

    Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning

    Sahil Sharma, Aravind Suresh, Rahul Ramesh, and Balaraman Ravindran. Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning. arXiv preprint arXiv:1705.07269, 2017

    2017 arXiv

  48. [56]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017

    2017 arXiv

  49. [57]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  50. [58]

    Overcoming model bias for robust offline deep reinforcement learning

    Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104: 0 104366, 2021

    2021

  51. [59]

    Leveraging factored action spaces for efficient offline reinforcement learning in healthcare

    Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems, 35: 0 34272--34286, 2022

    2022

  52. [60]

    Discretizing continuous action space for on-policy optimization

    Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the aaai conference on artificial intelligence, volume 34, pages 5981--5988, 2020

    2020

  53. [61]

    Action branching architectures for deep reinforcement learning

    Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. In Proceedings of the aaai conference on artificial intelligence, volume 32, 2018

    2018

  54. [62]

    Issues in using function approximation for reinforcement learning

    Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, volume 255, page 263. Hillsdale, NJ, 1993

    1993

  55. [63]

    dm\_control: Software and tasks for continuous control

    Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, 2020

    2020

  56. [64]

    Q-learning in enormous action spaces via amortized approximate maximization

    Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enormous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116, 2020

    2001 arXiv

  57. [65]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  58. [66]

    RORL : Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL : Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems, 2022

    2022

  59. [67]

    Reinforcement learning in healthcare: A survey

    Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55 0 (1): 0 1--36, 2021 a

    2021

  60. [68]

    MOPO : Model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO : Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 0 14129--14142, 2020

    2020

  61. [69]

    COMBO : Conservative offline model-based policy optimization

    Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO : Conservative offline model-based policy optimization. Advances in neural information processing systems, 34: 0 28954--28967, 2021 b

    2021

  62. [70]

    Deep reinforcement learning for page-wise recommendations

    Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference on recommender systems, pages 95--103, 2018

    2018

  63. [71]

    PLAS : Latent action space for offline reinforcement learning

    Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS : Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719--1735. PMLR, 2021

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.