Pith. sign in

REVIEW 2 major objections 4 minor 43 references

Mitigating Goal Misgeneralization via Minimax Regret

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training to minimize worst-case regret prevents goal misgeneralization, while maximizing expected value does not.

desk verdict A solid, well-scoped theory-plus-experiments paper on minimax regret and goal misgeneralization, held back only by an abstract that overstates the full-observability caveat the authors themselves document. read the letter →

arxiv 2507.03068 v2 pith:2L7AMY4M submitted 2025-07-03 cs.LG

Karim Abdel Sadek , Matthew Farrugia-Roberts , Usman Anwar , Hannah Erlebach , Christian Schroeder de Witt , David Krueger , Michael Dennis This is my paper · ORCID
classification cs.LG
keywords goalmisgeneralizationminimaxexpectedregretunsupervisedenvironmentdesigndomainrandomizationproxydistributionshiftreinforcementlearninggeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the standard reinforcement learning objective of maximizing expected value can produce policies that silently pursue a proxy goal rather than the designer's true goal when the two diverge only in rare situations. It formalizes this as a proxy-distinguishing distribution shift and proves that approximately optimizing the maximum expected value (MEV) objective allows such misgeneralization whenever the training fraction of distinguishing levels is small. In contrast, it proves that approximately optimizing the minimax expected regret (MMER) objective is immune: every approximately MMER policy is approximately optimal under the deployment goal. The paper then shows empirically in procedurally-generated grid-worlds that domain randomization, an MEV-based method, misgeneralizes at low rates of distinguishing levels, while regret-based unsupervised environment design methods are more robust, especially with more flexible adversaries and better regret estimators. If correct, this makes MMER a principled and practical target for training agents that use their capabilities toward the intended goal.

What carries the argument

The central object is the expected regret identity G_R(π; Λ) = max_{π'} V_R(π'; Λ) − V_R(π; Λ), which holds only because policies are conditioned on the level θ in a fully observable underspecified MDP. This identity lets the minimax expected regret objective be analyzed as a distribution-free robustness property: the adversary can always concentrate on the levels where a proxy-pursuing policy loses the most. The paper also introduces the proxy-distinguishing distribution shift (a pair of training and deployment distributions over levels, with different fractions α and β of distinguishing levels), and uses it to state the possibility theorem for MEV and the impossibility theorem for MMER in a single formal setting.

What would settle it

Run a partially observable variant of CHEESE IN THE CORNER where the cheese position is hidden until the mouse is adjacent, train with ACCEL plus the oracle-latest regret estimator, and measure deployment return on distinguishing levels; a return below the optimal level minus the optimization threshold would disprove the claim that MMER-based training is robust to goal misgeneralization.

Watch

Extended reading notes

Core claim

The central claim is Theorem 2: for any proxy-distinguishing distribution shift in a fully observable underspecified MDP, every policy that is approximately optimal under the minimax expected regret objective is also approximately optimal under the true goal on the deployment distribution, and hence does not goal misgeneralize. Theorem 1 shows the contrast: if the training distribution contains distinguishing levels with probability at most the optimization threshold ε, then some approximately MEV-optimal policy is proxy-optimal on deployment but βC-suboptimal under the true goal. The paper also demonstrates empirically that regret-based UED methods amplify rare distinguishing levels in the training buffer, which is the mechanism that steers the policy back to the true goal, and that domain randomization fails exactly where the theory predicts. The guarantee degrades gracefully under partial observability: an irreducible regret gap enters the bound, and the paper flags this as a limitation of standard MMER.

Load-bearing premise

The core guarantee assumes the policy can observe which level it is in; once levels are only partially observable, an irreducible regret gap can let goal misgeneralization slip through.

Editorial extensions

If this is right

  • Any approximately MMER policy is approximately optimal under any deployment distribution over levels in the specified space, so regret-based training is provably robust to goal misgeneralization in fully observable settings.
  • Domain randomization will misgeneralize whenever the training fraction of distinguishing levels α is at most the optimization tolerance ε; experiments show failure at α as high as 1e-2 to 1e-1.
  • Regret-based UED adversaries amplify rare distinguishing levels in the training buffer, which is the mechanism that corrects goal pursuit; more flexible adversaries and more accurate regret estimators strengthen this effect.
  • An adversary with unrestricted edit operations can prevent goal misgeneralization even when α=0, showing that the limitation is access to distinguishing levels rather than the MMER objective itself.
  • In partially observable environments, an irreducible regret gap can enter the MMER bound and break the guarantee; the paper proposes a lexicographic refinement to address this.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the theory holds, MMER offers a training objective that does not require manually tuning the proportion of rare, goal-revealing situations, since the adversary discovers them automatically; this could replace the practice of hand-crafting α for safety-critical tasks.
  • The KEYS AND CHESTS failure with the max-latest regret estimator suggests that the practical bottleneck is regret estimation rather than the MMER objective, so advances in value-network-based or sample-based maximum-return estimators could close most of the remaining gap.
  • The paper's broader-impact argument extends naturally: as RL agents become more capable, regret-maximizing adversaries also become more capable of finding distinguishing situations, so MMER-style training should become more, not less, effective at keeping advanced systems aligned with the intended goal.
  • A testable extension would be to run the same grid-world tasks with the level identity only partially observable, comparing ACCEL with the oracle-latest estimator against domain randomization; the paper's Section E predicts that the UED advantage shrinks or reverses as irreducible regret grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper formalizes goal misgeneralization as a proxy-distinguishing distribution shift over underspecified MDPs. It proves that under approximate maximum expected value (MEV) optimization, a proxy-pursuing policy can be near-optimal on the training distribution when the fraction of distinguishing levels is at most the optimization threshold (Theorem 1), while under approximate minimax expected regret (MMER) optimization every near-optimal policy is near-optimal on the deployment distribution (Theorem 2), under the paper's stated assumption that policies are conditioned on the level and the environment is fully observable. The paper then compares domain randomization with two regret-based unsupervised environment design (UED) methods, PLR⊥ and ACCEL, using both a sample-based and an oracle regret estimator, in three procedurally generated grid-world environments. The empirical results show that domain randomization often misgeneralizes when distinguishing levels are rare, while UED methods frequently prevent this, with notable exceptions reported in Section 7.4.

Significance. If the claims are read within their stated assumptions, the paper makes a valuable theoretical contribution: it gives the first formal treatment of goal misgeneralization as a distribution-shift phenomenon and cleanly proves that MMER has a robustness guarantee that MEV lacks. The proofs in Section A are straightforward and correct, and the paper is unusually transparent: it reports the failure of ACCEL with the max-latest estimator in KEYS AND CHESTS, includes extensive ablations in Appendices J through N, and explicitly derives a partial-observability weakening in Section E.2. The formal setting of proxy-distinguishing distribution shifts is likely to be reused by later work. The main weakness is that the advertised claims in the abstract and conclusion are stated more broadly than the theorems support, because Theorem 2 relies on full observability and level-conditioned policies and the experiments are all in fully observable environments.

major comments (2)
  1. [Section 5.2, Eq. (2); Section E.2; Abstract; Section 8] Theorem 2's robustness guarantee is carried by Eq. (2), which holds only when policies are conditioned on the level and the environment is fully observable (Propositions 1 and 2). The paper's own Section E.2 shows that under partial observability the conclusion degrades to the bound epsilon + g(Lambda_Deploy), and the aliasing example in Section E.1 exhibits a gap g equal to the full value range. The abstract's claim that goal misgeneralization is possible under MEV 'but not the MMER objective' and the conclusion's 'provably robust' are therefore unqualified overstatements. Please carry the level-conditioned/full-observability qualification into the abstract and the statement of Theorem 2, and move the POMDP caveat from Section E.2 into at least a main-text remark.
  2. [Section 6.1 and Section F] All three experimental environments are fully observable: Section F states that all goal-relevant information, including cheese, dish, key, and chest positions, is encoded in the observation channels. Consequently the empirical study does not test the regime in which Theorem 3's irreducible-regret gap can make the MMER guarantee vacuous. The abstract's claim that regret-based UED methods 'are more robust to goal misgeneralization' is presented without this scope restriction. Please state explicitly in the main text and abstract that the empirical results concern fully observable environments only, and note that the partial-observability case remains open.
minor comments (4)
  1. [Section 5.2, Remarks] The remark that 'a slightly modified bound holds for partially observable environments' understates the severity of the degradation: in the aliasing example of Section E.1, Theorem 3's additive gap can equal the full value range. Please refer explicitly to Eq. (12) and Theorem 3 when making this remark.
  2. [Section 7.4 and Figure 4] In KEYS AND CHESTS, ACCEL with the max-latest regret estimator performs worse than domain randomization (Figure 4, Section 7.4). The paper reports this honestly, but the abstract's summary that 'current regret-based UED methods are more robust to goal misgeneralization' should name this exception or be softened to 'in most environments and configurations tested'.
  3. [Section K, Theorem 4 proof] The proof asserts without derivation that c(theta, alpha) >= V^R(pi*; theta-prime) for theta not in the adversary's support and theta-prime in the support. Please add a sentence explaining that alpha equals the per-level value of pi* on the MMEV adversary's support and that c(theta, alpha) >= alpha by Definition 12.
  4. [Section 3, Eq. (2)] Equation (2) is introduced as an identity before it is proved in Section B; adding a forward reference to Proposition 1 would help the reader understand why this nontrivial identity is available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 2 is derived from definitions and an explicitly proved identity, with scope limitations disclosed in Section E.

full rationale

The central theoretical claim (Theorem 2) is proved in Appendix A using only the definitions of expected return, expected regret, approximate MMER, and the identity in Eq. (2). That identity is not imported from prior work; it is proved in Appendix B via Propositions 1 and 2, which rely on the explicit assumption that policies are conditioned on the level theta. The proof of Theorem 2 then converts a regret bound into an expected-return bound by substitution into Eq. (2), and the final step that minimum regret vanishes is also derived from Eq. (2). No fitted parameter is renamed as a prediction, and no self-cited uniqueness theorem or ansatz is used as the load-bearing premise. Self-citations such as Dennis et al. (2020) and Langosco et al. (2022) supply context, baselines, and environment provenance, but the theoretical results do not depend on them. The partial-observability restriction is explicitly disclosed in Section E.2, where the authors state that the irreducible regret gap g(Lambda_Deploy) weakens the guarantee and that this 'is a limitation of standard MMER.' Even if the abstract's unqualified phrasing overstates the scope of the fully-observable theorem, that is a scope/accuracy concern rather than circular reasoning. The empirical study is also self-contained: DR, PLR, and ACCEL are compared on procedurally generated environments, and the regret estimators are either sample-based or based on exact environment solutions, not fitted to the deployment results. Overall, the derivation chain is independent of its conclusions, so no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central theorems rest on standard finite UMDP assumptions plus two domain assumptions: level-observing policies allowing the regret identity, and the ability to train in distinguishing levels with reliable true-goal reward. The empirical section adds hand-chosen hyperparameters but no fitted constants. No invented entities such as new forces or particles are introduced.

free parameters (3)
  • PPO hyperparameters = learning rate 5e-5, clip 0.1, GAE lambda 0.95, entropy 1e-3 (1e-2 for KEYS AND CHESTS)
    Hand-chosen training settings (Appendix G.1); entropy coefficient changes across environments and can influence how easily a proxy policy is found.
  • UED configuration = buffer size 4096, rank prioritization, temperature 0.1, staleness 0.1, replay rate 0.33 (PLR) or 0.5 (ACCEL)
    Hand-chosen adversarial training settings; affect how strongly rare distinguishing levels are amplified (Sections 6.2 and G.1).
  • ACCEL edit distribution = 12 elementary edits, classification-preserving in main text
    Hand-chosen mutation operator; Section J shows results depend strongly on whether edits can introduce distinguishing levels.
assumptions (4)
  • domain assumption UMDP is finite: Theta, S, A finite; discount gamma in (0,1); goals normalized to [0,1].
    Section 3. Used in Theorem 1's value bounds and to make argmax/argmin sets well-defined.
  • domain assumption Policies are level-conditioned, pi: Theta x S -> Delta(A), and the environment is fully observable, so max and expectation commute (Proposition 2).
    Section 3 and Eq. (2). This is the load-bearing premise of Theorem 2. Section E shows a partial-observability caveat with an irreducible regret gap.
  • domain assumption Distinguishing levels are present in Theta and accessible during training; true-goal reward is reliable in them.
    Section 4.3. If deployment-distinguishing levels are absent from the training level space, no method can correct the proxy goal.
  • standard math Approximate optimization is modeled as arg-eps-max and arg-eps-min over policy sets.
    Definitions 4 and 5. This is the modeling device for finite optimization power in the theorems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Goal Misgeneralization via Minimax Regret." pith.science (2026). https://pith.science/paper/2L7AMY4M

@misc{pith2026250703068,
  author       = {Pith},
  title        = {Pith review of: Mitigating Goal Misgeneralization via Minimax Regret},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2L7AMY4M}},
  note         = {Machine review of arXiv:2507.03068}
}
read the original abstract

Safe generalization in reinforcement learning requires not only that a learned policy acts capably in new situations, but also that it uses its capabilities towards the pursuit of the designer's intended goal. The latter requirement may fail when a proxy goal incentivizes similar behavior to the intended goal within the training environment, but not in novel deployment environments. This creates the risk that policies will behave as if in pursuit of the proxy goal, rather than the intended goal, in deployment -- a phenomenon known as goal misgeneralization. In this paper, we formalize this problem setting in order to theoretically study the possibility of goal misgeneralization under different training objectives. We show that goal misgeneralization is possible under approximate optimization of the maximum expected value (MEV) objective, but not the minimax expected regret (MMER) objective. We then empirically show that the standard MEV-based training method of domain randomization exhibits goal misgeneralization in procedurally-generated grid-world environments, whereas current regret-based unsupervised environment design (UED) methods are more robust to goal misgeneralization (though they don't find MMER policies in all cases). Our findings suggest that minimax expected regret is a promising approach to mitigating goal misgeneralization.

Figures

Figures reproduced from arXiv: 2507.03068 by the authors.

Figure 1
Figure 1. Maximum expected value and minimax expected regret vs. goal misgeneralization. A mouse searches a maze for cheese that is usually located in the top-left corner. There is a proxy goal (“go to the corner”) that mostly incentivizes the same optimal behavior as the true goal (“go to the cheese”). (Left): Standard RL methods that approximately maximize expected value/return could find a policy that behaves as if pursuin… view at source ↗
Figure 2
Figure 2. Illustration and examples of non-distinguishing and distinguishing levels. A level θ ∈ Θ can be classified as non-distinguishing or 0-distinguishing (Definitions 1 and 2). (1st row): Possible relationships between sets Π⋆ = arg maxπ∈Π V R(π; θ) and Π˜ ⋆ = arg maxπ∈Π V R˜ (π; θ). (2nd row): Example levels for a navigation environment. Yellow and orange arrows show optimal behaviors for the true goal (“go to the chees… view at source ↗
Figure 3
Figure 3. Example procedurally-generated non-distinguishing/distinguishing levels. The agent’s [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Distribution shift performance for various training distributions. Average return over 512 steps for an evaluation batch of 256 distinguishing levels sampled from Λ Deploy = ΛDistg.. High performance indicates policies generalizing as intended; low performance indicate…
Figure 5
Figure 5. Figure 5: Rate at which adversary plays distinguishing levels. We plot the proportion of adver￾sarially sampled levels classified as distinguishing across training for T environment steps. The di￾agonal represents the proportion from the underlying training distribution Λ Train …
Figure 6
Figure 6. Figure 6: Performance on CHEESE IN THE CORNER levels with varying cheese position. For each training configuration, we evaluate the trained policy (first of 8 seeds) on a batch of 122 levels with shared wall layout and mouse spawn position but different cheese positions. We indi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 36 canonical work pages

  1. [1]

    A simple environment for showing mesa misalignment

    Matthew Barnett. A simple environment for showing mesa misalignment. Alignment Forum, September 2019. URL https://www.alignmentforum.org/posts/AFdRGfYDWQqmkdhFq

    work page 2019

  2. [2]

    Foerster

    Michael Beukman, Samuel Coward, Michael Matthews, Mattie Fellows, Minqi Jiang, Michael Dennis, and Jakob N. Foerster. Refining minimax regret for unsupervised environment design. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.\ 3637--3657. PMLR, 2024

    work page 2024

  3. [3]

    JAX: composable transformations of Python + NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas , Skye Wanderman-Milne , and Qiao Zhang. JAX: composable transformations of Python + NumPy programs, 2018. URL http://github.com/jax-ml/jax

    work page 2018

  4. [4]

    Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Maria Elisabeth Bechtle, Feryal Behbahani, Stephanie C. Y. Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando D...

    work page 2024

  5. [5]

    Techniques for optimizing worst-case performance

    Paul Christiano. Techniques for optimizing worst-case performance. AI Alignment (Blog), February 2018. URL https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99

    work page 2018

  6. [6]

    Quantifying generalization in reinforcement learning

    Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 1282--1289. PMLR, 2019

    work page 2019

  7. [7]

    Leveraging procedural generation to benchmark reinforcement learning

    Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 2048--2056. PMLR, 2020

    work page 2020

  8. [8]

    Russell, Andrew Critch, and Sergey Levine

    Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart J. Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems 33, pp.\ 13049--13061. Curran Associates, Inc., 2020

    work page 2020

Show all 43 references
  1. [9]

    IMPALA : Scalable distributed deep- RL with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA : Scalable distributed deep- RL with importance weighted actor-learner architectures. In Proceedings of ...

    2018

  2. [10]

    Wichmann

    Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2: 0 665--673, 2020

    2020

  3. [11]

    Recurrent world models facilitate policy evolution

    David Ha and J\" u rgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31, pp.\ 2450--2462. Curran Associates, Inc., 2018

    2018

  4. [12]

    Russell, and Anca Dragan

    Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems 30, pp.\ 6765--6774. Curran Associates, Inc., 2017

    2017

  5. [13]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference on Learning Representations. OpenReview, 2020

    2020

  6. [14]

    Mastering diverse control tasks through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640 0 (8059): 0 647--653, 2025

    2025

  7. [15]

    Towards an empirical investigation of inner alignment

    Evan Hubinger. Towards an empirical investigation of inner alignment. Alignment Forum, September 2019. URL https://www.alignmentforum.org/posts/2GycxikGnepJbxfHT

    2019

  8. [16]

    Foerster, Edward Grefenstette, and Tim Rockt\" a schel

    Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob N. Foerster, Edward Grefenstette, and Tim Rockt\" a schel. Replay-guided adversarial environment design. In Advances in Neural Information Processing Systems 34, pp.\ 1884--1897. Curran Associates, Inc., 2021 a

    2021

  9. [17]

    Prioritized level replay

    Minqi Jiang, Edward Grefenstette, and Tim Rockt \"a schel. Prioritized level replay. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 4940--4950. PMLR, 2021 b

    2021

  10. [18]

    A survey of zero-shot generalisation in deep reinforcement learning

    Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rockt \"a schel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 0 201--264, 2023

    2023

  11. [19]

    RMA : Rapid motor adaptation for legged robots

    Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA : Rapid motor adaptation for legged robots. In Proceedings of Robotics: Science and Systems XVII, 2021

    2021

  12. [20]

    Sharkey, Jacob Pfau, and David Krueger

    Lauro Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.\ 12004--12019. ...

    2022

  13. [21]

    Liu, Behzad Haghgoo, Annie S

    Evan Z. Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th International Conference on Machine Learning, ...

    2021

  14. [22]

    DrEureka: language model guided sim-to-real transfer

    Yecheng Jason Ma, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. DrEureka: language model guided sim-to-real transfer. In Proceedings of Robotics: Science and Systems XX, 2024

    2024

  15. [23]

    Isaac gym: High performance GPU based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Proceedings of the Neural ...

    2021

  16. [24]

    Foerster

    Nathan Monette, Alistair Letcher, Michael Beukman, Matthew Thomas Jackson, Alexander Rutherford, Alexander David Goldie, and Jakob N. Foerster. An optimisation framework for unsupervised environment design. Reinforcement Learning Journal, 2025. To appear

    2025

  17. [25]

    Robot learning from randomized simulations: A review

    Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot learning from randomized simulations: A review. Frontiers in Robotics and AI, 9: 0 799893, 2022

    2022

  18. [26]

    The alignment problem from a deep learning perspective

    Richard Ngo, Lawrence Chan, and S\"oren Mindermann. The alignment problem from a deep learning perspective. In 12th International Conference on Learning Representations. OpenReview, 2024

    2024

  19. [27]

    Solving Rubik's Cube with a robot hand

    OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zh...

    1910 arXiv

  20. [28]

    Foerster, Edward Grefenstette, and Tim Rockt \"a schel

    Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob N. Foerster, Edward Grefenstette, and Tim Rockt \"a schel. Evolving curricula with regret-based environment design. Preprint arXiv:2203.01302 [cs.LG], 2022

    2022 arXiv

  21. [29]

    Sim-to-real transfer of robotic control with dynamics randomization

    Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, pp.\ 3803--3810. IEEE, 2018

    2018

  22. [30]

    No regrets: Investigating and improving regret approximations for curriculum discovery

    Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. In Advances in Neural Information Processing Systems 37, pp.\ 16071--16101. Curran Associa...

    2024

  23. [31]

    Leonard J. Savage. The theory of statistical decision. Journal of the American Statistical Association, 46 0 (253): 0 55--67, 1951

    1951

  24. [32]

    Mastering Atari, Go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Natur...

    2020

  25. [33]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. Preprint arXiv:1506.02438 [cs.LG], 2015

    2015 arXiv

  26. [34]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Preprint arXiv:1707.06347 [cs.LG], 2017

    2017 arXiv

  27. [35]

    Goal misgeneralization: Why correct specifications aren't enough for correct goals

    Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. Preprint arXiv:2210.01790 [cs.LG], 2022

    arXiv

  28. [36]

    Addressing goal misgeneralization with natural language interfaces

    Giulio Starace. Addressing goal misgeneralization with natural language interfaces. Master's thesis, University of Amsterdam, 2023

    2023

  29. [37]

    Miguel Suau, Matthijs T. J. Spaan, and Frans A. Oliehoek. Bad habits: Policy confounding and out-of-trajectory generalization in reinforcement learning. Reinforcement Learning Journal, 4: 0 1711--1732, 2024

    2024

  30. [38]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017

    2017

  31. [39]

    Danesh, Nguyen X

    Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, and Benjamin Plaut. Getting by goal misgeneralization with a little help from a mentor. In The First Workshop on Safe & Trustworthy Agents, 2024. Workshop at NeurIPS 2024. Preprint arXiv:2410.21052 [cs.LG]

    2024 arXiv

  32. [40]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In 13th International Conference on Learning Representations. OpenReview, 2025

    2025

  33. [41]

    On the foundation of distributionally robust reinforcement learning

    Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. On the foundation of distributionally robust reinforcement learning. Preprint arxiv:2311.09018 [cs.LG], 2023

    2023 arXiv

  34. [42]

    Sohoni, Hongyang R

    Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher R \'e . Correct-N-Contrast: a contrastive approach for improving robustness to spurious correlations. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed...

    2022

  35. [43]

    Consequences of misaligned AI

    Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI . In Advances in Neural Information Processing Systems 33, pp.\ 15763--15773. Curran Associates, Inc., 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.