Pith. sign in

REVIEW 3 major objections 5 minor 94 references

Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simulation-trained policy is certified for real-world deployment by measuring how much an ensemble of critics disagrees on a small target-domain dataset.

desk verdict A practical deployment gate for domain randomization with strong experiments, but the central safety claim is not supported by the paper's own theory. read the letter →

arxiv 2507.06111 v1 pith:KWILOZRS submitted 2025-07-08 cs.LG cs.RO

Mohamad H. Danesh , Maxime Wabartha , Stanley Wu , Joelle Pineau , Hsiu-Chin Lin This is my paper
classification cs.LGcs.RO
keywords uncertainty-awarereinforcementlearningout-of-distributiondetectiondomainrandomizationensemblecriticsofflinesim-to-realtransferdeploymentsafetygatequadrupedlocomotion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a method called UARL that makes domain randomization safer by adding an uncertainty check before a policy is allowed to leave simulation. UARL trains an ensemble of critics so that their disagreement is low on in-distribution states and high on out-of-distribution states, then uses a small target-domain dataset only for evaluation. If the critics' variance on this dataset exceeds a threshold, the algorithm expands the randomization range and fine-tunes in simulation, repeating until the variance falls below the threshold. The paper argues that this gate blocks deployment of policies that would fail under real-world conditions, and it reports supporting evidence on MuJoCo benchmarks and on the ANYmal-D quadruped, including a zero-shot hardware transfer after the gate certified the policy.

What carries the argument

The argument is carried by an ensemble of critic networks trained with a diversity objective adapted from repulsive-location ensemble learning. Each critic is fit to Bellman targets on the nominal dataset, while a diversity term pushes each critic away from its own Bellman target on a repulsive dataset collected under wider randomization, making the critics agree in-distribution and disagree out-of-distribution. A deployment gate compares the critics' variance on the target-domain dataset $D_t$ against a threshold $\tau$, and a curriculum progressively expands the randomization range whenever the variance is too high. The theoretical backbone is a Lipschitz analysis showing that a nonzero shift in domain parameters forces a proportional change in the Q-function fixed point, which the paper uses to argue that out-of-distribution dynamics are detectable through ensemble variance.

What would settle it

Introduce a target-domain shift that is invisible to the randomized parameters: keep mass, friction, and noise in-distribution but add sensor latency, actuator delay, or communication loss to the real system, then check whether the variance gate passes while deployment performance collapses. A cheaper proxy test is to collect $D_t$ only from a narrow region of the target dynamics, such as flat-floor rollouts, while the deployment surface is slippery, and see whether the gate certifies a policy that then fails.

Watch

Extended reading notes

Core claim

The central claim is that critic-ensemble variance computed on a limited target-domain dataset $D_t$ serves as an effective proxy for policy performance in the target environment $E_t$. Specifically, low variance on $D_t$ indicates that the target dynamics fall within the simulation coverage established by progressive domain randomization, while high variance flags out-of-distribution conditions. The policy is iteratively fine-tuned on progressively more randomized simulated environments until the variance drops below a threshold $\tau$, at which point deployment is allowed. The paper supports this claim with experiments showing that UARL consistently separates in-distribution from out-of-distribution critic variance across MuJoCo environments and randomized parameters, that the variance gate correlates with deployment returns, and that the gate vetoed an earlier ANYmal policy before certifying a zero-shot transfer that succeeded on three real-world surfaces.

Load-bearing premise

The safety guarantee rests on the assumption that how much the learned value estimators disagree on a small sample of target-domain data tells you whether the policy will actually behave safely in that domain.

Editorial extensions

If this is right

  • Deployment can be gated using critic variance on a small target-domain dataset, blocking policies whose uncertainty exceeds a threshold and preventing unsafe real-world trials.
  • The progressive randomization curriculum can stop once the variance gate is satisfied, reducing over-randomization and improving sample efficiency relative to training on the full expanded range from the start.
  • UARL can be layered onto existing offline RL algorithms such as CQL, AWAC, and TD3BC, and also onto an on-policy PPO setup for a quadruped, improving OOD detection without degrading nominal locomotion performance.
  • Under an abrupt distribution shift during an episode, only the UARL critics show a sharp variance spike, enabling online detection of the shift while off-dynamics baselines remain flat.
  • A zero-shot sim-to-real transfer on the ANYmal-D quadruped passed the variance gate after expanding the mass randomization interval, and the robot tracked commands on concrete, carpet, and epoxy-resin-coated terrazzo surfaces without failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variance gate is a reliable safety proxy, the same ensemble could serve as a continuous runtime monitor that halts operation when critic disagreement spikes during deployment, an extension the paper only partially explores.
  • The reliability of the gate likely depends on how representative $D_t$ is of the true deployment conditions; a natural testable extension is to vary the size, coverage, and collection policy of $D_t$ and measure how often the gate's verdict matches actual deployment safety.
  • Since the paper proves only that low variance is necessary, not sufficient, for in-distribution dynamics, a stronger certification might combine the variance gate with a conservative Q-value or trajectory-filtering check, a direction the authors leave implicit.
  • The single-parameter-at-a-time curriculum could be extended to multi-parameter randomization where the variance signal is used to identify which parameter most needs expansion, potentially reducing the number of fine-tuning iterations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes UARL, an offline-to-sim-to-real RL framework that trains an ensemble of critics with a diversity loss on nominal and repulsive datasets, maintains a balancing replay buffer, and gates deployment on the ensemble variance computed over a limited target-domain dataset D_t. The central claim is that low critic variance on D_t certifies that the policy is safe to deploy in the target environment. The method is instantiated on CQL, AWAC, and TD3BC, evaluated on five MuJoCo environments across three randomized dynamics parameters, and demonstrated on an ANYmal-D quadruped with a zero-shot hardware transfer. Theoretical support is attempted in App. B through value-gap bounds, a variance decomposition, and a convergence theorem.

Significance. If the central deployment-gate claim were sound, UARL would be a practically valuable contribution: it addresses a real gap in domain randomization and off-dynamics RL, ships a public code link, provides unusually detailed appendices, and presents a real-hardware ANYmal experiment with a certified zero-shot transfer. However, the load-bearing claim that low ensemble variance on D_t certifies safety is not established by the paper's own formal results. Eq. (17) leaves the shared-bias term unbounded, Prop. B.4 explicitly concedes that low variance is insufficient, and the only direct empirical support is a three-phase negative correlation in Fig. 5 and a single zero-shot trial. The method is a plausible engineering heuristic, but it is not currently a safety certificate.

major comments (3)
  1. [Sec. 4.3, Eq. (17), Prop. B.4] The deployment gate treats sigma^2(D_t) < tau as sufficient evidence for safe deployment, but the paper's own bias-variance decomposition in App. B.3 gives E[(Q_t - Q)^2] = sigma^2 + (mu - Q_t)^2. Low variance removes only the disagreement term; the shared-bias term (mu - Q_t)^2 is never bounded by any theorem in the paper. Prop. B.4 explicitly states that low variance is necessary but not sufficient to conclude that E_t is in-distribution without full coverage of the parameter space. Prop. 4.3 and Thm. B.7 bound the value gap in terms of ||phi - phi_t||, not in terms of ensemble variance, so they do not close this gap. The only direct evidence is the negative correlation over three fine-tuning phases in Fig. 5 and a single ANYmal zero-shot trial; this does not rule out the concrete failure mode in which all critics agree on a common biased extrapolation for OOD target points. The safety claim should be weakened to a necessary-condition gate, or supplemented with an explicit bound on (mu - Q_t)^2 under stated coverage assumptions.
  2. [Sec. 4.2, Eq. (8); App. B.4.3, Eq. (29)] There is an internal contradiction in the balancing replay buffer. Eq. (8) assigns weight sigma^{-2} to repulsive samples from D' and sigma^2 to nominal samples from D, so higher-variance OOD samples receive lower weight. This contradicts the prose in Sec. 4.2 ('increasing sampling probability for high-uncertainty states from diverse simulations') and the statement in App. B.4.3 that 'high-variance (OOD) samples should receive larger weights in the Bellman update.' If the implementation follows Eq. (8), then the BRB ablation in Fig. 20 is not testing the mechanism described in the text; if the implementation follows the text, then Eq. (8) and Prop. B.9 are incorrect. The authors must resolve this discrepancy and rerun the ablation under the intended weighting.
  3. [App. B.4.4, Thm. B.10; Assum. B.8] Theorem B.10 is stated as a guarantee that weighted fitted-Q iteration recovers an eta-optimal target policy with polynomial samples and iterations, but the proof is only described as 'standard fitted Q-iteration analysis' and the key sample-complexity constant C_1 is left unspecified, with its dependence on L_R, R_max, L_T, epsilon_F, and the state/action dimensions deferred to future work. More importantly, the bias-reduction result in Prop. B.9 depends on Assum. B.8, which postulates a positive covariance between critic variance and parameter distance to the target. That assumption is essentially the property the deployment gate needs to establish, and the paper provides no mechanism by which Eq. (7) guarantees it. Thus the formal apparatus in App. B does not prove the safety claim; it restates a key part of the claim as an assumption.
minor comments (5)
  1. [Def. 4.2, Sec. 5.1] The symbol tau is used both for the KL-divergence threshold tau_KL in Def. 4.2 and for the deployment variance threshold in Sec. 4.3 and Sec. 5.1; a distinct notation such as tau_var would avoid ambiguity.
  2. [Fig. 4 caption] The caption contains typos ('incosistent', 'seprates') and does not explain how the rows (fine-tuning iterations) are labeled; please clarify the row/column semantics.
  3. [Alg. 1, lines 6 and 13] The variance formula uses Q^(0) in line 6 and Q^(i+1) in line 13, but the notation is not defined; please specify that the variance is averaged over the ensemble and over the samples in D_t.
  4. [App. E.7, Fig. 28] The text refers to ground friction as the randomized parameter, but the Fig. 28 legend says 'joint friction'; please make the parameter naming consistent.
  5. [Table 1, App. C] The table reports UARL memory of ~4 GB against a baseline of ~2 GB, which is a 100% increase, but the table and text say '50%'; please correct the percentage or the memory values.

Circularity Check

1 steps flagged · score 4.0 of 10

OOD detection evaluation partly restates the Eq. 7 training objective on repulsive data; the central safety claim retains independent held-out support.

  1. fitted input called prediction [Section 4.1 (Eq. 7) and Section 5.1, Fig. 4]
    "The diversity term LRL div in Eq. 7 encourages each Qi to diverge from its own Bellman target on the repulsive dataset D′... Each row represents a fine-tuning iteration with an expanded ID range by multiplying the agent’s mass vector by a constant: 1x→5x→10x. Clear “OOD-awareness” places blue curves below and red curves above the black curve; UARL-AWAC does so consistently."

    The repulsive dataset D′ for a given iteration is collected from the newly expanded randomization range (e.g., 5x mass). Eq. 7 explicitly trains the ensemble to have high variance on D′ by penalizing each critic for matching its own Bellman target on D′. The OOD detection evaluation in Fig. 4 then labels exactly these newly expanded ranges (5x in the first row, 10x in the second) as 'OOD' and reports that UARL produces high variance on them. For those data points, the high variance is a direct consequence of the training loss, not a prediction; the experiment is partially restating the objective the critics were optimized against.

full rationale

The central safety claim — that critic ensemble variance on a target-domain dataset Dt proxies policy performance in Et — is validated with held-out target-domain rollouts (Fig. 5) and a real-robot ANYmal zero-shot transfer trial, so it has independent grounding. However, the OOD detection experiments in Sec. 5.1 partially evaluate the training objective on the repulsive data used to fit it: Eq. 7 explicitly drives high variance on D′, and Fig. 4 then reports high variance on those same newly expanded ranges as evidence of OOD detection. This is a mild fitted-input-called-prediction pattern. The theory (Eq. 17, Props. B.3–B.4) does not establish that low variance implies small bias, but that is a correctness gap rather than circularity, and the paper itself concedes the insufficiency of low variance. Self-citations to the authors' prior DENN work are normal and not load-bearing in a circular way, since DENN is an independent published method with its own validation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method rests on several unproven or heuristically chosen elements: the diversity hyperparameters, the deployment threshold, the manual randomization ranges, and assumptions about critic calibration and target-domain representativeness.

free parameters (5)
  • lambda (diversity coefficient) = adaptive, ~10% of total loss
    Keeps Ldiv at about 10% of the total loss; the OOD detection behavior depends on this balance.
  • delta (diversity scale) = 1e-2
    Chosen from DENN's recommended range [1e-3, 1e-1/2]; controls repulsive penalty sharpness.
  • tau (deployment threshold) = 95th percentile (or 5th percentile) of ID critic variances
    Set from 100 ID rollouts; determines whether deployment is allowed.
  • Randomization expansion schedule = e.g., Ant mass 1x,5x,10x,15x; friction increments vary per environment
    Manually specified in App. D.2; the curriculum's success is sensitive to these ranges.
  • Ensemble size N = 2 critics
    Default from baselines; a larger ensemble would change variance estimates.
assumptions (5)
  • domain assumption Assum. 4.4: Ensemble critics optimize Eq. 7 until loss stabilizes, implying low variance on nominal D and high variance on repulsive D'.
    The OOD detection property of the ensemble depends on this assumed convergence behavior.
  • standard math Assum. B.5: Reward and transition kernel are Lipschitz; critic class approximates Q within epsilon_F.
    Used to prove the operator perturbation bound (Lem. B.6) and value error bound (Thm B.7).
  • ad hoc to paper Assum. B.8: Covariance between critic variance and parameter distance to target is at least rho > 0.
    This is needed for the bias-reduction claim of the weighted Bellman operator (Prop. B.9); no empirical verification is given.
  • domain assumption D_t is a representative proxy for the target domain E_t.
    Stated in Sec. 4 before Def. 4.1; the deployment gate's reliability depends on this.
  • domain assumption Definition 4.2: Coverage set C defined by KL divergence; OOD means phi_t not in C.
    This formalizes ID/OOD, but the proposition in Eq. 5 uses phi_t not in Phi instead, a mismatch.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation." pith.science (2026). https://pith.science/paper/KWILOZRS

@misc{pith2026250706111,
  author       = {Pith},
  title        = {Pith review of: Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWILOZRS}},
  note         = {Machine review of arXiv:2507.06111}
}
read the original abstract

Deploying reinforcement learning (RL) policies in real-world involves significant challenges, including distribution shifts, safety concerns, and the impracticality of direct interactions during policy refinement. Existing methods, such as domain randomization (DR) and off-dynamics RL, enhance policy robustness by direct interaction with the target domain, an inherently unsafe practice. We propose Uncertainty-Aware RL (UARL), a novel framework that prioritizes safety during training by addressing Out-Of-Distribution (OOD) detection and policy adaptation without requiring direct interactions in target domain. UARL employs an ensemble of critics to quantify policy uncertainty and incorporates progressive environmental randomization to prepare the policy for diverse real-world conditions. By iteratively refining over high-uncertainty regions of the state space in simulated environments, UARL enhances robust generalization to the target domain without explicitly training on it. We evaluate UARL on MuJoCo benchmarks and a quadrupedal robot, demonstrating its effectiveness in reliable OOD detection, improved performance, and enhanced sample efficiency compared to baselines.

Figures

Figures reproduced from arXiv: 2507.06111 by the authors.

Figure 1
Figure 1. Overview of UARL. Blue shows our contributions while orange is adapted from existing RL methods. Agent processes nominal and repulsive datasets through RL and diversity terms. The verification module assesses policy safety in the target domain to avoid deploying policies in OOD scenarios. Furthermore, repetitively testing policies trained through DR on physical hardware introduces safety concerns, as the policies ma… view at source ↗
Figure 2
Figure 2. Conceptual visualization of nom￾inal E and repulsive E ′ environments. Our goal is to train a policy that can be deployed in an unknown target domain Et without directly refining or validating in the target domain. We assume that we have access to a limited target domain dataset Dt, where Dt could act as a proxy for the target domain Et, allowing us to evaluate policy performance on Et without the risk of unsafe dep… view at source ↗
Figure 3
Figure 3. Conceptual visu￾alization of state space ex￾pansion from E0 (initial) to E1, . . . , E3 by increas￾ing randomization. $E_t$ denotes the theoretical tar￾get environment; $E_t’$ is where the agent performs ef￾fectively. If the disagreement of the critics is above τ , we expand the randomized parameter to reach to E2 from E1 and collect the new repulsive dataset D2. We combine D0 and D1 as the nominal datasets using ou… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Critic variance across 100 rollouts in the Ant-v4 environment for PBRL, RORL, EDAC, DARL, and AWAC-based methods. The randomized hyperparameter is agent mass. Each column represents an algorithm, and each row represents a fine-tuning iteration with an expanded ID range…
Figure 5
Figure 5. Figure 5: Normalized critic variance (left) and deployment [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Critic ensemble variance for nominal , repulsive and target domain trajectories before (left) and after (middle) one fine-tuning step. The uncertainty gate (black dashed line) moves to accommodate the new nominal interval. Right: ANYmal-D executing the zero-shot transf…
Figure 7
Figure 7. Figure 7: The OOD detection results for AWAC and UARL-AWAC initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]
Figure 8
Figure 8. Figure 8: The OOD detection results for CQL and UARL-CQL initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]
Figure 9
Figure 9. Figure 9: The OOD detection results for AWAC and UARL-AWAC initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: The OOD detection results for CQL and UARL-CQL initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: The OOD detection results for AWAC and UARL-AWAC initial noise scale (top) and friction [PITH_FULL_IMAGE:figures/full_fig_p037_11.png]
Figure 12
Figure 12. Figure 12: The OOD detection results for CQL and UARL-CQL initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p038_12.png]
Figure 13
Figure 13. Figure 13: The OOD detection results for AWAC and UARL-AWAC initial noise scale (top) and friction [PITH_FULL_IMAGE:figures/full_fig_p039_13.png]
Figure 14
Figure 14. Figure 14: The OOD detection results for CQL and UARL-CQL initial noise scale (top) and friction coefficient [PITH_FULL_IMAGE:figures/full_fig_p040_14.png]
Figure 15
Figure 15. Figure 15: Offline training (1st iteration) performance, showing average return during training across the randomized parameters. Curves are smoothed for clarity. E.2 Overall Policy Performance This subsection presents an extended analysis of UARL’s performance across five MuJoC…
Figure 16
Figure 16. Figure 16: Offline training (1st iteration) performance in Hopper-v4 (top), Swimmer-v4 (middle), and Walker2d-v4 (bottom) environments, showing average return during training across three randomized hy￾perparameters. 0 10 20 30 6000 6500 7000 7500 Average Return Initial Noise Sc…
Figure 17
Figure 17. Figure 17: Fine-tuning (2nd iteration) performance in Ant-v4 (top) and HalfCheetah-v4 (bottom) environments, showing average return during fine-tuning across three randomized parameters. 43 [PITH_FULL_IMAGE:figures/full_fig_p043_17.png]
Figure 18
Figure 18. Figure 18: Fine-tuning (2nd iteration) performance in Hopper-v4 (top), Swimmer-v4 (middle), and Walker2d-v4 (bottom) environments, demonstrating average return during fine-tuning across three randomized hyperparame￾ters. the gains observed in the 2nd iteration. For instance, in …
Figure 19
Figure 19. Figure 19: Fine-tuning (3rd iteration) performance in various environments, demonstrating average return during fine-tuning across three randomized hyperparameters. In order, from top to bottom: Ant-v4, HalfCheetah-v4, Hopper-v4, Swimmer-v4, Walker2d-v4. 45 [PITH_FULL_IMAGE:fig…
Figure 20
Figure 20. Figure 20: The impact of balancing the replay buffer (BRB) during the initial fine-tuning for all randomized [PITH_FULL_IMAGE:figures/full_fig_p046_20.png]
Figure 21
Figure 21. Figure 21: Performance in HalfCheetah-v4 under domain shift. While all methods degrade when transitioning from source (friction coefficient=0.5) to target domain (friction coefficient=0.8), UARL maintains competitive performance despite training exclusively on source domain data…
Figure 22
Figure 22. Figure 22: OOD detection in HalfCheetah-v4. When friction coefficient changes abruptly at step 650 and stays (grey area), only UARL reliably detects the distribution shift through critic variance spikes that exceed the 95% ID CI (black lines). Baselines fail to recognize the OOD…
Figure 23
Figure 23. Figure 23: Performance sensitivity to hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p051_23.png]
Figure 24
Figure 24. Figure 24: Performance sensitivity to hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p052_24.png]
Figure 25
Figure 25. Figure 25: Isaac Gym running 4096 parallel ANYmal simulations. Each agent receives randomized dynamics for domain randomization curriculum, enabling massive state transitions’ on one GPU. E.7 Additional ANYmal Details Thus far we have evaluated UARL on MuJoCo tasks with low-dime…
Figure 26
Figure 26. Figure 26: Mean episode return (smoothed) for PPO and UARL-PPO on ANYmal, across different randomized [PITH_FULL_IMAGE:figures/full_fig_p054_26.png]
Figure 27
Figure 27. Figure 27: Critic variance across 100 rollouts in the ANYmal environment for PPO and UARL-PPO methods. The randomized hyperparameter is the added base mass. Each column represents an algorithm, and each row represents a fine-tuning iteration with an expanded ID range shown in th…
Figure 28
Figure 28. Figure 28: Critic variance across 100 rollouts in the ANYmal environment for PPO and UARL-PPO methods. The randomized hyperparameter is the ground friction. Each column represents an algorithm, and each row represents a fine-tuning iteration with an expanded ID range shown in th…
Figure 29
Figure 29. Figure 29: Safety-critical failure modes resulting from mis-specified domain randomization bounds: (a) over [PITH_FULL_IMAGE:figures/full_fig_p056_29.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

94 extracted references · 55 canonical work pages

  1. [1]

    MIT press, 2018

    Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction. MIT press, 2018

    2018

  2. [2]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    arXiv 2005

  3. [3]

    Reinforcement learning in robotics: A survey

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013

    2013

  4. [4]

    Toward self-driving processes: A deep reinforcement learning approach to control.AIChE journal, 65(10):e16689, 2019

    Steven Spielberg, Aditya Tulsyan, Nathan P Lawrence, Philip D Loewen, and R Bhushan Gopaluni. Toward self-driving processes: A deep reinforcement learning approach to control.AIChE journal, 65(10):e16689, 2019

    2019

  5. [5]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey

    Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020

    2020

  6. [6]

    Out-of-distribution dynamics detection: Rl-relevant benchmarks and results.arXiv preprint arXiv:2107.04982, 2021

    Mohamad H Danesh and Alan Fern. Out-of-distribution dynamics detection: Rl-relevant benchmarks and results.arXiv preprint arXiv:2107.04982, 2021

    arXiv 2021

  7. [7]

    Off-dynamics reinforcement learning: Training for transfer with domain classifiers

    Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2021

    2021

  8. [8]

    Robust dynamic programming.Mathematics of Operations Research, 30(2):257–280, 2005

    Garud N Iyengar. Robust dynamic programming.Mathematics of Operations Research, 30(2):257–280, 2005

    2005

Show all 94 references
  1. [9]

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble

    Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. InConference on Robot Learning, pages 1702–1712. PMLR, 2022

    2022

  2. [10]

    Adaptive policy learning for offline-to-online reinforcement learning

    Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11372–11380, 2023

    2023

  3. [11]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30...

    2017

  4. [12]

    Pal, and Liam Paull

    Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization. InProceedings of the Conference on Robot Learning, volume 100, pages 1162–1176. PMLR, 30 Oct–01 Nov 2020

    2020

  5. [13]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012

    2012

  6. [14]

    Towards a generic solution for inspection of industrial sites

    Marco Hutter, Remo Diethelm, Samuel Bachmann, Peter Fankhauser, Christian Gehring, Vassilios Tsounis, Andreas Lauber, Fabian Guenther, Marko Bjelonic, Linus Isler, et al. Towards a generic solution for inspection of industrial sites. InField and Service Robotics: Results of th...

    2018

  7. [15]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. InInternational Conference on Learning Representations, 2022

    2022

  8. [16]

    Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.arXiv preprint arXiv:1907.00456, 2019

    Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.arXiv preprint arXiv:1907.00456, 2019

    1907 arXiv

  9. [17]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132–20145, 2021. 10

    2021

  10. [18]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. InAdvances in Neural Information Processing Systems, volume 33, pages 1179–1191. Curran Associates, Inc., 2020

    2020

  11. [19]

    Uncertainty-based of- fline reinforcement learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based of- fline reinforcement learning with diversified q-ensemble. InAdvances in Neural Information Processing Systems, volume 34, pages 7436–7447. Curran Associates, Inc., 2021

    2021

  12. [20]

    Iteratively refined behavior regularization for offline reinforcement learning

    Yi Ma, HAO Jianye, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2023

    2023

  13. [21]

    Offline reinforcement learning with OOD state correction and OOD action suppression

    Yixiu Mao, Cheems Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with OOD state correction and OOD action suppression. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  14. [22]

    Model-Bellman inconsistency for model-based offline reinforcement learning

    Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-Bellman inconsistency for model-based offline reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edi- tors,Proceedin...

    2023

  15. [23]

    Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning

    Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. InInternational Conference on Learning Representations, 2022

    2022

  16. [24]

    Rorl: Ro- bust offline reinforcement learning via conservative smoothing.Advances in neural information processing systems, 35:23851–23866, 2022

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing.Advances in neural information processing systems, 35:23851–23866, 2022

    2022

  17. [25]

    When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning

    Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599–36612, 2022

    2022

  18. [26]

    Cross-domain policy adaptation via value-guided data filtering

    Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  19. [27]

    Cross-domain policy adaptation by capturing representation mismatch

    Jiafei Lyu, Chenjia Bai, Jing-Wen Yang, Zongqing Lu, and Xiu Li. Cross-domain policy adaptation by capturing representation mismatch. InForty-first International Conference on Machine Learning, 2024

    2024

  20. [28]

    Unsolved problems in ml safety.arXiv preprint arXiv:2109.13916, 2021

    Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety.arXiv preprint arXiv:2109.13916, 2021

    2021 arXiv

  21. [29]

    Learning dexterous in-hand manipulation.The International Journal of Robotics Research, 39(1):3–20, 2020

    OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation.The International Journal of Robotics Research, 39(1):3–20, 2020

    2020

  22. [30]

    Network randomization: A simple technique for generalization in deep reinforcement learning

    Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. InInternational Conference on Learning Representations, 2020

    2020

  23. [31]

    Learning domain randomization distributions for training robust locomotion policies

    Melissa Mozian, Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Learning domain randomization distributions for training robust locomotion policies. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6112–6117. IEEE, 2020

    2020

  24. [32]

    A Markovian Decision Process.Indiana University Mathematics Journal, 6(4):679–684, 1957

    Richard Bellman. A Markovian Decision Process.Indiana University Mathematics Journal, 6(4):679–684, 1957. 11

    1957

  25. [33]

    Handling black swan events in deep learning with diversely extrapolated neural networks

    Maxime Wabartha, Audrey Durand, Vincent François-Lavet, and Joelle Pineau. Handling black swan events in deep learning with diversely extrapolated neural networks. InProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 2140...

    2020

  26. [34]

    Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning

    Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  27. [35]

    Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  28. [36]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  29. [37]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasi...

    2023

  30. [38]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational Conference on Machine Learning, pages 1861–1870. PMLR, 2018

    2018

  31. [39]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational Conference on Machine Learning, pages 1587–1596. PMLR, 2018

    2018

  32. [40]

    Corl: Research-oriented deep offline reinforcement learning library

    Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. In3rd Offline RL Workshop: Offline RL as a ”Launchpad”, 2022

    2022

  33. [41]

    Efficient online reinforcement learning with offline data

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. InInternational Conference on Machine Learning, pages 1577–1594. PMLR, 2023

    2023

  34. [42]

    Odrl: A benchmark for off-dynamics reinforcement learning

    Jiafei Lyu, Kang Xu, Jiacheng Xu, Mengbei Yan, Jingwen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, and Xiu Li. Odrl: A benchmark for off-dynamics reinforcement learning. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  35. [43]

    Darl: distance-aware uncertainty estimation for offline reinforcement learning

    Hongchang Zhang, Jianzhun Shao, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Darl: distance-aware uncertainty estimation for offline reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11210–11218, 2023

    2023

  36. [44]

    Open rl benchmark: Comprehensive tracked experiments for reinforcement learning.arXiv preprint arXiv:2402.03046, 2024

    Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learning.arXiv ...

    2024 arXiv

  37. [45]

    Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger

    Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C. Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger. Anymal - a highly mobile and dynamic quadruped...

    2016

  38. [46]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. InConference on Robot Learning, pages 91–100. PMLR, 2022. 12

    2022

  39. [47]

    Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots.Science Robotics, 4(26):eaau5872, 2019

    2019

  40. [48]

    REvolveR: Continuous evolutionary models for robot-to-robot policy transfer

    Xingyu Liu, Deepak Pathak, and Kris Kitani. REvolveR: Continuous evolutionary models for robot-to-robot policy transfer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Confer- ence o...

    2022

  41. [49]

    Proto: Iterative policy regularized offline-to-online reinforcement learning.arXiv preprint arXiv:2305.15669, 2023

    Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning.arXiv preprint arXiv:2305.15669, 2023

    2023 arXiv

  42. [50]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. InProceedings of the 36th International Conference on Machine Learning, volume 97, pages 2052–2062. PMLR, 09–15 Jun 2019

    2019

  43. [51]

    An optimistic perspective on offline reinforcement learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. InInternational Conference on Machine Learning, pages 104–114. PMLR, 2020

    2020

  44. [52]

    Tree-based batch mode reinforcement learning.Journal of Machine Learning Research, 6, 2005

    Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning.Journal of Machine Learning Research, 6, 2005

    2005

  45. [53]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  46. [54]

    Batch reinforcement learning

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pages 45–73. Springer, 2012

    2012

  47. [55]

    Critic regularized regression.Advances in Neural Information Processing Systems, 33:7768–7778, 2020

    Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression.Advances in Neural Information Processing Systems, 33:7768–7778, 2020

    2020

  48. [56]

    Revisiting the minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [57]

    Behavior regularized offline reinforcement learning.arXiv preprint arXiv:1911.11361, 2019

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  50. [58]

    Keep doing what worked: Behavior modelling priors for offline reinforcement learning

    Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. InInternational Conference on L...

    2020

  51. [59]

    Offline reinforcement learning with fisher divergence critic regularization

    Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. InProceedings of the 38th International Conference on Machine Learning, volume 139, pages 5774–5783. PMLR, 18–24 Jul 2021

    2021

  52. [60]

    Uncertainty weighted actor-critic for offline reinforcement learning

    Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 11...

    2021

  53. [61]

    Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization

    Kun LEI, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  54. [62]

    Enoto: Improving offline-to-online reinforcement learning with q-ensembles

    Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Enoto: Improving offline-to-online reinforcement learning with q-ensembles. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 5563–5571. Interna- ti...

    2024

  55. [63]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. InInternational Conference on Machine Learning, pages 27042–27059. PMLR, 2022

    2022

  56. [64]

    Actor-critic alignment for offline-to-online reinforcement learning

    Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. InProceedings of the 40th International Conference on Machine Learning, volume 202, pages 40452–40474. PMLR, 23–29 Jul 2023

    2023

  57. [65]

    Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  58. [66]

    Policy expansion for bridging offline-to-online reinforcement learning

    Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  59. [67]

    Albrecht, and Amos Storkey

    Trevor McInroe, Adam Jelley, Stefano V . Albrecht, and Amos Storkey. Planning to go out-of- distribution in offline-to-online reinforcement learning. InReinforcement Learning Conference (RLC), 2024

    2024

  60. [68]

    A comprehensive survey on safe reinforcement learning

    Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015

    2015

  61. [69]

    Consideration of risk in reinforcement learning

    Matthias Heger. Consideration of risk in reinforcement learning. InMachine Learning Proceed- ings 1994, pages 105–111. Elsevier, 1994

    1994

  62. [70]

    Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005

    2005

  63. [71]

    Safe offline reinforcement learning with feasibility-guided diffusion model

    Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations, 2024

    2024

  64. [72]

    Enhancing efficiency of safe reinforcement learning via sample manipulation

    Shangding Gu, Laixi Shi, Yuhao Ding, Alois Knoll, Costas Spanos, Adam Wierman, and Ming Jin. Enhancing efficiency of safe reinforcement learning via sample manipulation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  65. [73]

    Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    2020

  66. [74]

    Causally aligned curriculum learning

    Mingxuan Li, Junzhe Zhang, and Elias Bareinboim. Causally aligned curriculum learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  67. [75]

    Au- tomated curriculum learning for neural networks

    Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Au- tomated curriculum learning for neural networks. Ininternational conference on machine learning, pages 1311–1320. Pmlr, 2017

    2017

  68. [76]

    MIT press, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep learning. MIT press, 2016

    2016

  69. [77]

    Robust training with ensemble consensus

    Jisoo Lee and Sae-Young Chung. Robust training with ensemble consensus. InInternational Conference on Learning Representations, 2020

    2020

  70. [78]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. InAdvances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  71. [79]

    Improving robustness and calibration in ensembles with diversity regularization

    Hendrik Alexander Mehrtens, Camila González, and Anirban Mukhopadhyay. Improving robustness and calibration in ensembles with diversity regularization. InDAGM German Conference on Pattern Recognition, pages 36–50. Springer, 2022

    2022

  72. [80]

    Maximizing overall diversity for improved uncertainty estimates in deep ensembles.Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4264–4271, Apr

    Siddhartha Jain, Ge Liu, Jonas Mueller, and David Gifford. Maximizing overall diversity for improved uncertainty estimates in deep ensembles.Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4264–4271, Apr. 2020. 14

    2020

  73. [81]

    Improving adversarial robustness via promoting ensemble diversity

    Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. InInternational Conference on Machine Learning, pages 4970–4979. PMLR, 2019

    2019

  74. [82]

    Webb, Henry W

    Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luján, and Gavin Brown. A unified theory of diversity in ensemble learning.Journal of Machine Learning Research, 24(359):1–49, 2023

    2023

  75. [83]

    Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization. InAdvances in Neural Information Processing Systems, 2022

    2022

  76. [84]

    Noise contrastive priors for functional uncertainty

    Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. InUncertainty in Artificial Intelligence, pages 905–914. PMLR, 2020

    2020

  77. [85]

    Deep exploration via bootstrapped dqn.Advances in neural information processing systems, 29, 2016

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn.Advances in neural information processing systems, 29, 2016

    2016

  78. [86]

    Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning

    Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. InInternational Conference on Machine Learning, pages 6131–6141. PMLR, 2021

    2021

  79. [87]

    Accurate uncertainty estimation and decomposition in ensemble learning.Advances in Neural Information Processing Systems, 32, 2019

    Jeremiah Liu, John Paisley, Marianthi-Anna Kioumourtzoglou, and Brent Coull. Accurate uncertainty estimation and decomposition in ensemble learning.Advances in Neural Information Processing Systems, 32, 2019

    2019

  80. [88]

    Disentangling epistemic and aleatoric uncertainty in reinforcement learning.arXiv preprint arXiv:2206.01558, 2022

    Bertrand Charpentier, Ransalu Senanayake, Mykel Kochenderfer, and Stephan Günnemann. Disentangling epistemic and aleatoric uncertainty in reinforcement learning.arXiv preprint arXiv:2206.01558, 2022

    2022 arXiv

  81. [89]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

    2015

  82. [90]

    Robustness

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. 15 Appendix Table of Contents A Related Work 17 B Theoretical Analysis of UARL 20 B.1 Definitions and Notations . . . ....

    2017 arXiv

  83. [91]

    |R(s, a)−R(s′, a′)| ≤LR (s, a)−(s′, a′) ,∀(s, a),(s ′, a′)∈S×A,(19) and satisfies|R(s, a)| ≤Rmax

    The rewardR(s, a)isL R–Lipschitzin(s, a), i.e. |R(s, a)−R(s′, a′)| ≤LR (s, a)−(s′, a′) ,∀(s, a),(s ′, a′)∈S×A,(19) and satisfies|R(s, a)| ≤Rmax

  84. [92]

    We acknowledge that real-world contact dynamics can violate global Lipschitz continuity

    For any two domain parametersϕ 1, ϕ2 ∈Φ, W1 Tϕ1 (· |s, a), Tϕ2 (· |s, a) ≤L T ∥ϕ1 −ϕ 2∥,∀(s, a)∈S×A,(20) where W1 is the 1-Wasserstein distance. We acknowledge that real-world contact dynamics can violate global Lipschitz continuity. In such cases one must either restrict to l...

  85. [93]

    distance

    The critic class F can uniformly approximate the true Q-function up to error ϵF . Defining the projection operator ΠF [f] = arg min g∈F ∥f−g∥ ∞,(21) we require ∥Qπ ϕ −Π F [Qπ ϕ]∥∞ ≤ϵ F ,(22) meaning there exists some ˜Q∈Fwith∥Q π ϕ − ˜Q∥∞ ≤ϵ F . Assum. B.5 ensures both the Bel...

  86. [94]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.