Pith. sign in

REVIEW 3 major objections 7 minor 42 references

An Optimisation Framework for Unsupervised Environment Design

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Unsupervised environment design is reformulated as a nonconvex-strongly-concave game with the first provable convergence guarantee for zero-sum scores, and the practical variant outperforms prior methods on three benchmarks.

desk verdict A genuine step forward for UED theory and practice, but the regret convergence guarantee needs a deterministic-dynamics assumption before it covers what it claims. read the letter →

arxiv 2505.20659 v2 pith:KXU3KZWP submitted 2025-05-27 cs.LG

Nathan Monette , Alistair Letcher , Michael Beukman , Matthew T. Jackson , Alexander Rutherford , Alexander D. Goldie , Jakob N. Foerster This is my paper · ORCID
classification cs.LG
keywords unsupervisedenvironmentdesignreinforcementlearningrobustnessminimaxoptimisationconvergenceguaranteelearnabilityentropyregularisation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to put unsupervised environment design on a firmer optimisation footing. It reformulates UED as a minimax game over an entropy-regularised expected score, which is strongly concave in the adversary's sampling distribution, and proves that a two-timescale stochastic gradient descent-ascent algorithm converges to an $\epsilon$-stationary policy when the score is zero-sum, such as regret or negative return. That gives UED its first convergence guarantee that does not presuppose convergence. The paper also derives a generalised learnability score for arbitrary deterministic RL environments and packages the method into a practical algorithm, NCC, which outperforms prior UED baselines on Minigrid, XLand-Minigrid, and Craftax. If correct, the work converts UED from a heuristic training loop into an optimisation problem with a certificate of local optimality.

What carries the argument

The central object is the entropy-regularised expected-score objective $f(x,y) = y^T s(\pi_x,\Lambda) + \alpha H(y)$, whose entropy term enforces strong concavity in the adversary's distribution $y$ and keeps training levels diverse. The argument runs through two-timescale stochastic gradient descent-ascent, with the agent learning slowly ($\eta_x = \Theta(\alpha^2/\ell^3)$) and the adversary faster ($\eta_y = \Theta(1/\ell)$), and uses the strong-concavity to apply a known best-iterate convergence theorem for nonconvex-strongly-concave minimax problems. The generalised learnability score $s(\pi_x,\lambda) = \sigma_\lambda \mathcal{N}(\mu_\lambda|\mu,\sigma^2)$ (level-return standard deviation weighted by a Gaussian in the mean return) is the practical device that lets the method apply to non-binary environments.

What would settle it

Run Algorithm 1 with its exact theoretical configuration (static level set, REINFORCE estimator, regret score) on a two-level gridworld whose optimal returns are known, and record the norm $\|\nabla_x \max_y f(x,y)\|$ at each iteration; if it does not fall below $\epsilon$ within the iteration bound of Theorem 5.1, the convergence theorem is refuted.

Watch

Extended reading notes

Core claim

The central discovery is a reformulation and a convergence theorem. The UED objective is written as $f(x,y) = y^T s(\pi_x, \Lambda) + \alpha H(y)$, where $y$ is a categorical distribution over a finite level set $\Lambda$, $s$ is the score vector (e.g., negative return or regret), and $H$ is the entropy. Because the entropy term makes $f$ $\alpha$-strongly concave in $y$, the game becomes nonconvex-strongly-concave. With a $\zeta$-greedy, $L$-Lipschitz, $K$-smooth policy, the paper proves that the best iterate $x^*$ returned by two-timescale stochastic gradient descent-ascent satisfies $\|\nabla_x \max_y f(x^*, y)\| < \epsilon$ in $O(\Delta \ell^3/(\alpha^2 \epsilon^2) + 2\ell^3/(\alpha \epsilon^4))$ iterations (Theorem 5.1). The resulting point is an $\epsilon$-approximate first-order Nash equilibrium. The paper also proposes a generalised learnability score $s(\pi_x,\lambda) = \sigma_\lambda \mathcal{N}(\mu_\lambda|\mu,\sigma^2)$ that extends the binary-outcome learnability of prior work to arbitrary deterministic environments.

Load-bearing premise

The load-bearing premise is that the simplified setting analysed in Theorem 5.1—static finite level set, REINFORCE gradient estimator, and zero-sum score—represents real UED well enough that the convergence guarantee transfers to the practical method; Section 6 of the paper concedes the practical algorithm instead uses PPO, a dynamic buffer, and general-sum score functions.

Editorial extensions

If this is right

  • Zero-sum UED (regret or negative return) now has a polynomial iteration bound to reach an $\epsilon$-first-order-Nash point, so future analyses can assume convergence rather than only assert properties conditional on it.
  • The entropy-regularised adversary yields an explicit distribution over levels, so the level sampler is directly optimisable by gradients instead of heuristics.
  • The generalised learnability score extends sample-for-learnability to any deterministic RL environment, removing the binary-outcome restriction of prior work.
  • The two-timescale rate separation gives practical guidance: the adversary's learning rate should dominate the agent's, and the temperature $\alpha$ controls exploration of the level space.
  • If the framework holds, UED becomes a principled component of robust RL pipelines with a certificate of local optimality, rather than an ad-hoc curriculum heuristic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entropy regularisation trick is general: any objective that is linear in a distribution over tasks could be regularised the same way, so the framework likely extends beyond UED to curriculum learning and dataset selection.
  • Because the theorem covers only Algorithm 1, the practical NCC's empirical wins on XLand-Minigrid and Craftax are not yet backed by the convergence proof; bridging that gap may require analysing PPO-like updates or general-sum score functions.
  • The generalised learnability score's Gaussian weighting over mean returns could suppress levels with unusually low or high mean return even when they are learnable; a controlled comparison on environments with skewed return distributions would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper reformulates UED as a nonconvex-strongly-concave minimax game by adding entropy regularization to the adversary's level distribution over a finite static level set. It proposes Algorithm 1, a two-timescale stochastic gradient descent-ascent method, and proves (Theorem 5.1) an epsilon-stationary best-iterate convergence guarantee under Assumptions 1 and 2, using Proposition 1 for variance and smoothness bounds and invoking Lin et al. (2020). The paper then introduces a practical variant, NCC (Algorithm 2), using PPO, a dynamic level buffer, and a generalized learnability score, and reports experiments on Minigrid, XLand-Minigrid, and Craftax showing improvements over PLR, DR, and SFL baselines.

Significance. If the convergence theorem were sound, it would provide UED's first rigorous convergence guarantee for a zero-sum formulation, a meaningful advance over prior saddle-point-existence arguments. The paper is transparent about the gap between Algorithm 1 and the practical NCC, and the empirical study covers standard benchmarks with 10 seeds and includes robustness (alpha-CVaR) evaluations. The generalized learnability score is a useful practical contribution. However, the theorem's applicability to regret in stochastic environments is currently not established.

major comments (3)
  1. [Section 5.1, Eq. (10), Proposition 1] The regret estimator is not unbiased. Regret is defined in Section 2.3 as J(pi*_lambda, lambda) - J(pi_x, lambda), where J is expected discounted return. The estimator \hat s(pi_x, lambda_i) = max_tau R(tau, lambda_i) - (1/M) sum_j R(tau_ij, lambda_i) has expectation E[max_tau R(tau, lambda)] - J(pi_x, lambda), which equals regret only if E[max_tau R] = J(pi*_lambda, lambda). This equality fails under stochastic transitions, which Assumption 1 does not exclude, because the maximum over trajectories is a maximum of sampled returns rather than the optimal expected value; with finite M it also fails in deterministic environments unless the max is an exact oracle over all trajectories. Since Lin et al. (2020) requires unbiased stochastic gradients, Theorem 5.1 as stated does not cover the zero-sum regret objective. Please add an explicit deterministic-environment assumption (or an exact optimal-return oracle), state the resulting scope, or replace the estimator with an unbiased one.
  2. [Section 5.1, Assumption 2] A hard zeta-greedy policy is piecewise constant and not K-smooth, yet the proof of Proposition 1 relies on bounds such as ||nabla_x log pi_x|| <= L/zeta and ||nabla^2_x log pi_x|| <= K/zeta + L^2/zeta^2, which require differentiability and Lipschitz/smoothness of the policy. Please restate Assumption 2 as, for example, a smooth stochastic policy whose action probabilities are bounded below by zeta (a 'zeta-soft' policy), and verify that all bounds in Appendix A hold under that definition.
  3. [Sections 4.2, 6, Algorithm 2] The practical NCC violates the theorem's assumptions (PPO instead of REINFORCE, dynamic buffer, any score function, any optimizer), as Table 1 acknowledges. Section 6.2 further concedes that general-sum score functions such as learnability have no convergence guarantees. Consequently, the 'provably convergent' claim attaches only to Algorithm 1; the abstract's phrase 'providing stronger theoretical guarantees for practical settings' and the Section 1 statement 'a gradient-based algorithm that is provably convergent' should be scoped explicitly to Algorithm 1 to avoid overclaiming. This is a presentation/scope issue, but it is load-bearing for how readers interpret the empirical results.
minor comments (7)
  1. [Abstract] The abstract says 'outperforming prior methods in a number of environments', but the cover-page summary says 'two of three environments'; Figure 3 shows NCC-Learn and Gen-SFL comparable in Craftax, so please make the abstract consistent with the actual results.
  2. [Section 2.4, Eq. (11)] The generalization of learnability in Eq. (11) uses the level-wise standard deviation sigma_lambda, while the text says 'standard error values'; please align the terminology with the formula.
  3. [Algorithm 2] The call \hat G(x_t, y_t; s, \tilde s) is underspecified because Eq. (10) defines \hat G with arguments (x, y) only; it is unclear whether the merged score vector \tilde s or the batch score vector s is used in the adversary update.
  4. [Appendix B.1] The note 'We write details for the dynamic buffer in maroon' is not visible in the provided manuscript; please use a different cue, such as a labeled comment or a separate paragraph.
  5. [Table 1] In the row 'Score Function Only zero-sum (i.e. Regret and −J)', the 'i.e.' should be 'e.g.', since regret and negative return are examples of zero-sum scores, not an exhaustive list.
  6. [Section 5.1, Algorithm 1] The 'Best-iterate policy parameters x*' returned by Algorithm 1 lacks a selection criterion; Lin et al.'s guarantee is existential over iterates, so please state how the best iterate is identified or acknowledge that the guarantee is not constructive.
  7. [Section 6.2] The sentence 'We note that the baselines ... also lack convergence guarantees' is a fair caveat but should not be phrased as a justification; the absence of guarantees in baselines does not address the lack of guarantees for general-sum NCC.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the convergence guarantee is imported from an external theorem, and no fitted input is relabeled as a prediction.

full rationale

The paper's central result, Theorem 5.1, is an application of Lin et al. (2020, Theorem 4.5) to a newly constructed entropy-regularized minimax objective. The entropy term is explicitly added to make the objective strongly concave in the adversary's strategy, and Proposition 1 proves the needed variance, Lipschitzness, and smoothness bounds from Assumptions 1 and 2 rather than assuming them. The score functions used in the objective (negative return and regret) are defined independently of the empirical evaluation metrics, and the generalized learnability score is constructed as a variance-like heuristic, not fitted to the paper's experimental outcomes. Self-citations to prior UED work (Dennis et al., Jiang et al., Rutherford et al.) are contextual motivation and baseline descriptions, not load-bearing for the proof; the proof's external dependency is Lin et al. and standard policy-gradient results. The paper also explicitly discloses that the practical NCC algorithm breaks the theoretical assumptions (Section 6 and Table 1), which is an honest limitation rather than a circular move. Any concern about the regret estimator's unbiasedness in stochastic environments is a correctness or assumption-satisfaction issue, not a case of the derivation reducing to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper's central result is assembled from an external optimization theorem plus new modeling choices (entropy regularization, truncated simplex, zero-sum scores). The practical method adds heuristics and deliberately breaks the assumptions, so the ledger separates theoretical inputs from the practical score functions. No new entities are postulated; the generalized learnability score is a function, not an entity.

free parameters (4)
  • alpha (entropy temperature) = 0.05 for Minigrid and Craftax, annealed as alpha divided by the cube root of t+1 in Craftax; 0 for XLand-Minigrid
    Sets strong concavity constant of the adversary objective in Theorem 5.1; when alpha=0 in XLand the convergence guarantee does not apply.
  • eta_x and eta_y (learning rates) = eta_x 0.001/0.0001/0.0001 and eta_y 0.1/0.01/0.01 for Minigrid/XLand/Craftax
    Hand-tuned per environment; the required two-timescale separation is an input to Lin et al. (2020).
  • Level buffer sizes |Lambda|, |Lambda'|, batch |lambda| = |Lambda|=4000 on all; |Lambda'|=256/8192/0; |lambda|=256/8192/1024
    Chosen by hand for compute/performance, not derived from theory; the dynamic buffer itself violates the static-Lambda assumption.
  • xi (simplex truncation)
    Appears in Assumption 2 and in the Lipschitz/smoothness bounds of Proposition 1; no value is given because it is a proof device, and practice uses the un-truncated simplex.
assumptions (6)
  • standard math Lin et al. (2020) Theorem 4.5: two-timescale stochastic GDA finds an epsilon-stationary point for nonconvex-strongly-concave objectives
    The convergence theorem is a direct application; the result is not re-derived.
  • domain assumption Assumption 1: finite level count N, finite horizon T, bounded state/reward spaces
    Used throughout Proposition 1 to bound variance, Lipschitzness, and smoothness; excludes infinite or continuous level spaces.
  • domain assumption Assumption 2: zeta-greedy L-Lipschitz K-smooth policy and xi-truncated simplex
    Required for the gradient and Hessian bounds; not verified for the PPO policies used in the experiments.
  • domain assumption Zero-sum score condition (s = -J or Reg) for the guarantee
    Theorem 5.1 applies only when the score is zero-sum with negative return; learnability and PVL are explicitly outside this regime.
  • ad hoc to paper Entropy regularization alpha H(y) is added by the authors
    It is the device that creates strong concavity; it changes the UED objective from standard minimax regret, so the guarantee is for the regularized game.
  • ad hoc to paper Generalized learnability heuristic s = sigma_lambda N(mu_lambda | mu, sigma^2)
    Introduced as a practical score function in Eq. 11; its choice is justified empirically in Appendix B.4, not by the optimization theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Optimisation Framework for Unsupervised Environment Design." pith.science (2026). https://pith.science/paper/KXU3KZWP

@misc{pith2026250520659,
  author       = {Pith},
  title        = {Pith review of: An Optimisation Framework for Unsupervised Environment Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXU3KZWP}},
  note         = {Machine review of arXiv:2505.20659}
}
read the original abstract

For reinforcement learning agents to be deployed in high-risk settings, they must achieve a high level of robustness to unfamiliar scenarios. One method for improving robustness is unsupervised environment design (UED), a suite of methods aiming to maximise an agent's generalisability across configurations of an environment. In this work, we study UED from an optimisation perspective, providing stronger theoretical guarantees for practical settings than prior work. Whereas previous methods relied on guarantees if they reach convergence, our framework employs a nonconvex-strongly-concave objective for which we provide a provably convergent algorithm in the zero-sum setting. We empirically verify the efficacy of our method, outperforming prior methods in a number of environments with varying difficulties.

Figures

Figures reproduced from arXiv: 2505.20659 by the authors.

Figure 1
Figure 1. A visual representation of our training loop, which has simultaneous updates for the agent [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Mean solve rates with standard error bars on Minigrid, a common UED testbed. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Performance on more difficult benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: α-CVaR evaluation performance on Minigrid. Robustness Evaluation We also perform the α-CVaR evaluation protocol from Rutherford et al. (2024), which evaluates policies from 10 seeds per method on the α% worst-case levels which are still solvable. We find that in XLand-…
Figure 5
Figure 5. Figure 5: Empirical comparison between the different instantiations of NCC. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Analysis of Generalised Learnability Score function on Minigrid. The black lines represent [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Omitted α-CVaR Robustness Evaluation curves. B.6 Compute Time [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: DR: Sampled levels at halfway through training (top row) and the end of training (bottom [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: SFL: Highest learnability scoring levels at halfway through training (top row) and the end [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: NCC-Learn: Highest weighted levels at halfway through training (top row) and the end [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021

    work page 2021

  2. [2]

    Clutr: curriculum learning via unsupervised task representation learning

    Abdus Salam Azad, Izzeddin Gur, Jasper Emhoff, Nathaniel Alexis, Aleksandra Faust, Pieter Abbeel, and Ion Stoica. Clutr: curriculum learning via unsupervised task representation learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    work page 2023

  3. [3]

    Refining minimax regret for unsupervised environment design

    Michael Beukman, Samuel Coward, Michael Matthews, Mattie Fellows, Minqi Jiang, Michael Dennis, and Jakob Foerster. Refining minimax regret for unsupervised environment design. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    work page 2024

  4. [4]

    JAX : composable transformations of P ython+ N um P y programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  5. [5]

    Accelerated algorithms for constrained nonconvex-nonconcave min-max optimization and comonotone inclusion

    Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Accelerated algorithms for constrained nonconvex-nonconcave min-max optimization and comonotone inclusion. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning...

    work page 2024

  6. [6]

    Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J Terry. Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neur...

    work page 2023

  7. [7]

    Adversarial environment design via regret-guided diffusion models

    Hojun Chung, Junseo Lee, Minsoo Kim, Dohyeong Kim, and Songhwai Oh. Adversarial environment design via regret-guided diffusion models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 63715--63746. Curran Associates, Inc., 2024. URL https://proce...

    work page 2024

  8. [8]

    Jaxued: A simple and useable ued library in jax, 2024

    Samuel Coward, Michael Beukman, and Jakob Foerster. Jaxued: A simple and useable ued library in jax, 2024. URL https://arxiv.org/abs/2403.13091

    arXiv 2024

Show all 42 references
  1. [9]

    Last-iterate convergence: Zero-sum games and constrained min-max optimization, 2020

    Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization, 2020. URL https://arxiv.org/abs/1807.04252

    2020

  2. [10]

    Emergent complexity and zero-shot transfer via unsupervised environment design

    Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advance...

    2020

  3. [11]

    Lucas, and Stefano V

    Samuel Garcin, James Doran, Shangmin Guo, Christopher G. Lucas, and Stefano V. Albrecht. Dred: Zero-shot transfer in reinforcement learning via data-regularised environment design. JMLR.org, 2024

    2024

  4. [12]

    Mirror learning: A unifying framework of policy optimisation

    Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confere...

    2022

  5. [13]

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic

    Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33 0 (1): 0 147--180, 2023. doi:10.1137/20M1387341. URL https://...

    2023 doi

  6. [14]

    Replay-guided adversarial environment design

    Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rockt\" a schel. Replay-guided adversarial environment design. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Pro...

    2021

  7. [15]

    Prioritized level replay

    Minqi Jiang, Edward Grefenstette, and Tim Rockt \"a schel. Prioritized level replay. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 4940--4950. PMLR, 18--...

    2021

  8. [16]

    u ttler, Edward Grefenstette, Tim Rockt\

    Minqi Jiang, Michael Dennis, Jack Parker-Holder, Andrei Lupu, Heinrich K\" u ttler, Edward Grefenstette, Tim Rockt\" a schel, and Jakob Foerster. Grounding aleatoric uncertainty for unsupervised environment design. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and...

    2022

  9. [17]

    Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning...

    2020

  10. [18]

    Learning equilibria in adversarial team markov games: A nonconvex-hidden-concave min-max optimization problem

    Fivos Kalogiannis, Jingming Yan, and Ioannis Panageas. Learning equilibria in adversarial team markov games: A nonconvex-hidden-concave min-max optimization problem. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural...

    2024

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  12. [20]

    Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes

    Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, and xiao wang. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Art...

    2021

  13. [21]

    Enhancing the hierarchical environment design via generative trajectory modeling, 2024

    Dexun Li and Pradeep Varakantham. Enhancing the hierarchical environment design via generative trajectory modeling, 2024. URL https://arxiv.org/abs/2310.00301

    2024 arXiv

  14. [22]

    Tiada: A time-scale adaptive algorithm for nonconvex minimax optimization

    Xiang Li, Junchi YANG, and Niao He. Tiada: A time-scale adaptive algorithm for nonconvex minimax optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=zClyiZ5V6sL

    2023

  15. [23]

    On gradient descent ascent for nonconvex-concave minimax problems

    Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp...

    2020

  16. [24]

    Craftax: A lightning-fast benchmark for open-ended reinforcement learning

    Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning. In International Conference on Machine Learning ( ICML ) , 2024

    2024

  17. [25]

    Kinetix: Investigating the training of general agents through open-ended physics-based control tasks

    Michael Matthews, Michael Beukman, Chris Lu, and Jakob Nicolaus Foerster. Kinetix: Investigating the training of general agents through open-ended physics-based control tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview....

    2025

  18. [26]

    Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile

    Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra(-gradient) mile. In International Conference on Learning Representations, 2019. URL https:/...

    2019

  19. [27]

    Stable recurrent models

    John Miller and Moritz Hardt. Stable recurrent models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=Hygxb2CqKm

    2019

  20. [28]

    Robust reinforcement learning

    Jun Morimoto and Kenji Doya. Robust reinforcement learning. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000. URL https://proceedings.neurips.cc/paper_files/paper/2000/file/e8dfff4676a47048d6f0c4ef899...

    2000

  21. [29]

    J.F. Nash. Non-cooperative games. Annals of Mathematics, 54 0 (2): 0 286--295, 1951

    1951

  22. [30]

    XL and-minigrid: Scalable meta-reinforcement learning environments in JAX

    Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Viacheslav Sinii, Artem Agarkov, and Sergey Kolesnikov. XL and-minigrid: Scalable meta-reinforcement learning environments in JAX . In Intrinsically-Motivated and Open-Ended Learning Workshop, NeurIPS2023, 2023. URL https://o...

    2023

  23. [31]

    Solving a class of non-convex min-max games using iterative first order methods

    Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neu...

    2019

  24. [32]

    Evolving curricula with regret-based environment design, 2023

    Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design, 2023. URL https://arxiv.org/abs/2203.01302

    2023 arXiv

  25. [33]

    Robust optimization over multiple domains

    Qi Qian, Shenghuo Zhu, Jiasheng Tang, Rong Jin, Baigui Sun, and Hao Li. Robust optimization over multiple domains. AAAI Press, 2019. ISBN 978-1-57735-809-1. doi:10.1609/aaai.v33i01.33014739. URL https://doi.org/10.1609/aaai.v33i01.33014739

    2019 doi

  26. [34]

    No regrets: Investigating and improving regret approximations for curriculum discovery

    Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds....

    2024

  27. [35]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  28. [36]

    Hessian aided policy gradient

    Zebang Shen, Alejandro Ribeiro, Hamed Hassani, Hui Qian, and Chao Mi. Hessian aided policy gradient. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Resear...

    2019

  29. [37]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. IEEE Press, 2017. doi:10.1109/IROS.2017.8202133. URL https://doi.org/10.1109/IROS.2017.8202133

    2017

  30. [38]

    P roximal C urriculum for R einforcement L earning A gents

    Georgios Tzannetos, B\'arbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. P roximal C urriculum for R einforcement L earning A gents. Transactions of Machine Learning Research (TMLR), 2023

    2023

  31. [39]

    Lipschitz regularity of deep neural networks: analysis and efficient estimation

    Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran A...

    2018

  32. [40]

    Albrecht, and Andrew Nystrom

    Maciej Wiatrak, Stefano V. Albrecht, and Andrew Nystrom. Stabilizing generative adversarial networks: A survey, 2020. URL https://arxiv.org/abs/1910.00927

    2020 arXiv

  33. [41]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8 0 (3–4): 0 229–256, May 1992. ISSN 0885-6125. doi:10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696

    1992 doi

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.