Pith. sign in

REVIEW 4 major objections 4 minor 115 references

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Converting both actor and critic objectives to classification terms that ignore the current scale of returns lets a single Transformer-based agent learn and adapt across many tasks without labels.

desk verdict A solid, well-validated empirical paper showing two-hot critic classification plus binary advantage-filtered actor updates substantially improve label-free multi-task meta-RL; the scale-invariance mechanism is plausible but not fully isolated. read the letter →

arxiv 2411.11188 v1 pith:NMPZT3KR submitted 2024-11-17 cs.LG

Jake Grigsby , Justin Sasek , Samyak Parajuli , Daniel Adebi , Amy Zhang , Yuke Zhu This is my paper · ORCID
classification cs.LG
keywords meta-reinforcementlearningmulti-taskreinforcementin-contexttransformersscaleinvariancetwo-hotvalueclassificationadvantage-weightedimitationlong-termmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the "multi-task barrier" in meta-reinforcement learning is largely a problem of uneven return scales, and that it can be removed without task labels by making both learning objectives scale-resistant. The proposed agent, built on a Transformer policy that conditions on trajectory context, replaces value regression with two-hot classification over symlog-compressed return bins and replaces the Q-weighted policy update with a binary advantage-filtered imitation objective. Because neither loss is scaled by the size of current rewards, tasks with high absolute returns no longer dominate the gradient. Across Meta-World ML45, Multi-Task POPGym, Multi-Game Procgen, Multi-Game Atari, and BabyAI, the scale-resistant combination improves multi-task performance and sample efficiency relative to the same architecture trained with standard actor-critic losses. A sympathetic reading takes the paper's central claim to be that scale invariance is a first-order design requirement for adaptive memory-based agents, not that these are the only useful properties of the two losses.

What carries the argument

The load-bearing mechanism is a pair of scale-resistant classification updates. The critic's two-hot classification converts a scalar TD target $y_t$ into probabilities over $B$ fixed bins through $\mathrm{twohot}_B(\mathrm{symlog}(y_t))$, with a symlog transform $\mathrm{symlog}(y)=\mathrm{sign}(y)\ln(|y|+1)$ compressing a wide return range into a bounded label space and symexp recovering the scalar. This makes the loss depend on bin-count resolution rather than on the absolute size of returns. The actor's binary advantage filter turns policy improvement into imitation of actions whose one-step advantage estimate is positive, so the update is a maximum-likelihood classification over a dynamically selected subset of the replay buffer. The shared Transformer consumes trajectories of observations, actions, rewards, and reset signals, so task identity must be inferred from context rather than provided; an ensemble of critics and multiple discount factors are used as in the AMAGO baseline. The machinery's role is to keep the optimization process insensitive to each task's current return scale while preserving the memory and adaptation behavior of sequence-model RL.

What would settle it

A decisive test would train the same agent on a set of tasks whose return scales are deliberately equalized and compare the scale-resistant losses to standard regression; if the classification losses still win, their benefit is not scale invariance. The complementary check is to multiply the rewards of a single task by a large constant and observe whether Q-dependent losses degrade much faster than the scale-resistant ones.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single Transformer-based meta-RL agent can learn and adapt across qualitatively different tasks without task labels if its actor and critic updates depend on Q-values only through classification. Concretely, the critic loss is $L_{\text{Critic-Ind}}(t) = -\mathrm{twohot}_B(y_t)^T \log Q_B(h_t, a_t)$, where the TD target $y_t$ is mapped with a symlog transform into two adjacent bins, and the actor loss is $L_{\text{Actor-Ind}}(t) = -\mathbf{1}\{Q(h_t,a_t)-\mathbb{E}_{a'\sim\pi(h_t)}[Q(h_t,a')] > 0\}\log\pi(a_t\mid h_t)$, so the policy imitates only actions whose estimated advantage over the current policy is positive. Both losses keep the one-step TD structure and can be computed in parallel over a context window, but they do not grow or shrink with the absolute magnitude of returns in any task. The paper reports that this pair of updates more than doubles ML45 success over the original RL2 result at 100M timesteps, matches MuZero's reported 50M-timestep result with a simpler one-step Q-learning, improves 8 of 10 unclipped Atari games, and remains largely unaffected by deliberately rescaling rewards in Procgen by factors of 10 and 100. The paper reads this evidence as supporting the hypothesis that multi-task training is bottlenecked by return-scale imbalance.

Load-bearing premise

The load-bearing premise is that the two losses' measured gains come from ignoring return scale; the paper does not isolate that property from other benefits of two-hot classification and advantage filtering, so if those other benefits do most of the work, the scale-barrier mechanism would be wrong even if the recipe still works.

Editorial extensions

If this is right

  • The same Transformer architecture, with no task-specific heads or labels, can be trained across heterogeneous task suites; the paper demonstrates this on ML45, 27 POPGym tasks, 16 Procgen games, 10 Atari games, and 68 BabyAI task configurations.
  • Because the losses do not depend on return scale, deliberate reward rescaling (Coinrun rewards multiplied by 100 and Climber rewards divided by 10) barely hurts the scale-resistant agent, while the Q-dependent baseline loses most of its performance in the unaffected games.
  • Scale-resistant critics and actors improve not just aggregate scores but coverage: in ML45 the number of manipulation skills mastered roughly doubles compared with Q-dependent updates and the original RL2 reference.
  • Long-context memory settings benefit as well: in the Procgen memory mode and two-episode BabyAI evaluations, the agent improves from the first to the second attempt on unseen levels and held-out tasks, showing adaptation and memory can coexist with multi-task training.
  • The recipe reduces RL updates to two supervised-learning-style classification heads, pointing to a simpler path for scaling online RL with recycled data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the scale-invariance mechanism is the true driver, the same two losses should transfer to non-Transformer sequence models such as recurrent or state-space policies; this is not tested in the paper and would be a direct way to check the claim's scope.
  • The paper's binary actor filter means the policy update is automatically ignoring a changing fraction of the replay buffer; an untested corollary is that the method should be robust to strongly imbalanced replay data without explicit rebalancing, since it learns from the positive-advantage subset.
  • The dependent-dependent baseline uses a global, task-agnostic normalization layer rather than the per-task normalization PopArt originally proposed, so the reported gap may overstate the value of scale invariance relative to a label-free per-task solution; comparing against an oracle that recovers task identity from context would clarify the margin.
  • A clean single-task control with artificially separated return scales could separate scale-invariance from other benefits of two-hot classification such as representation learning and noisy-target robustness; the paper does not run this control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a modification to the AMAGO Transformer-based meta-RL agent that replaces the standard scale-dependent actor and critic losses (Eqs. 1-2) with scale-insensitive classification objectives: a two-hot symlog value classifier for the critic (Eq. 3) and an advantage-filtered imitation loss for the actor (Eq. 4). The authors evaluate all four combinations of dependent/independent actor and critic losses on Meta-World ML45, multi-task POPGym, multi-game Procgen, multi-game Atari, and BabyAI, reporting that the fully independent combination improves performance and sample efficiency in most settings. They conclude that scale-invariant objectives break a 'multi-task barrier' in online meta-RL without requiring task labels.

Significance. If the mechanism claim holds, the paper offers a simple, label-free recipe for scaling online meta-RL to heterogeneous task mixtures, with code and per-task learning curves provided in the appendix. The main empirical contribution, namely consistent gains of the fully independent update across five benchmarks, is meaningful and well documented. However, the causal attribution to return-scale invariance is not isolated from other properties of the two loss changes, and the dependent baseline may be weaker than a proper scale-normalized control. These issues leave the title-level claim under-supported, although the practical value of the loss combination is credible.

major comments (4)
  1. [Section 3, Eqs. (3)-(4); Figure 4; Appendix A] The central claim that the gains come from decoupling optimization from return scale is not established by the 2x2 ablation. Each loss change alters two things at once: the loss functional form and its sensitivity to Q-scale. Two-hot classification (Eq. 3) is known to improve representation learning and robustness to noisy targets even in single-task RL, as the paper itself notes by citing reference [34], and advantage-filtered imitation (Eq. 4) adds an off-policy stabilization effect. The paper does not include a control that isolates scale-invariance, such as training the dependent losses with per-task or inferred-task return normalization, or comparing the independent update against a task-agnostic normalization that removes global scale differences. Without such a control, the results support the weaker claim that classification-style losses help multi-task meta-RL, but not the stronger claim that scale invariance is the causal mechanism.
  2. [Appendix A; Section 4, Figure 6] The dependent baseline uses a global, task-agnostic PopArt layer, as stated in Appendix A, yet the paper cites per-task PopArt as the standard multi-task solution in Section 2. A global layer normalizes the aggregate scale but leaves per-task relative imbalances intact, so the comparison may overstate the advantage of scale-resistant losses over a stronger label-free baseline. The reward-rescaling experiment in Figure 6 demonstrates that the dependent update is more sensitive to artificially injected scale differences, but it does not quantify the natural return-scale imbalance in Meta-World, Atari, or BabyAI. The paper should either add a stronger normalization baseline or present evidence that natural return-scale imbalance is the active bottleneck in those domains; otherwise the conclusion that scale invariance 'should be a priority' in Section 5 overreaches.
  3. [Section 4, Figures 4-6, 9-10] Statistical claims such as 'significant progress' and 'more than double' rest on error bars that are the minimum and maximum over three or four independent trials, and Figure 6 uses only two trials. The paper should report means with standard errors or confidence intervals, state the number of seeds explicitly, and, where possible, include a paired significance test across tasks or seeds. This is necessary to support the strength of the headline empirical claims, especially in comparisons to prior methods such as RL2-PPO and MuZero that use different architectures, budgets, and hyperparameters.
  4. [Appendix A, Value Classification Details] The paper explicitly states that the trade-offs among bin count B, bin limits, and the use of symlog are 'underexplored in the results of this work.' Because the two-hot critic requires choosing B and the return bounds, and because Figure 7 shows that the multi-task Procgen result depends on the number of frequently used labels, the manuscript should include at least one multi-task sensitivity analysis for these choices or justify the fixed configuration more strongly. Without this, the 'simple and scalable' claim is qualified by an unstudied hyperparameter dependence.
minor comments (4)
  1. [Section 4, Meta-World ML45] The comparisons to RL2-PPO, MuZero, and HTrMRL are gathered from prior papers with different architectures, budgets, and hyperparameters; the text should label these as reference curves rather than implying a controlled comparison, especially when stating that the proposed method 'more than doubles' the RL2 success rate.
  2. [Figures 1 and 10] There are typographical errors in the manuscript: Figure 1 contains 'agentso' and Figure 10 is titled 'Multli-Task BabyAI' instead of 'Multi-Task BabyAI'.
  3. [Section 3, Eq. (4)] The statement that the independent actor loss 'becomes IL on expert datasets' is imprecise: the binary filter is based on a learned advantage estimate, not on expert labels, so the imitation is only approximate and depends on the quality of the critic.
  4. [Appendix A, Base RL Details] The use of a global PopArt layer for the dependent baseline is an important design choice that should be highlighted in the main text rather than only in the appendix, because it directly affects the interpretation of the headline ablation in Figure 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's claims are empirical evaluations of externally-sourced loss modifications, not fits or self-citations dressed as predictions.

full rationale

AMAGO-2 is an empirical systems paper: it takes two known loss modifications (two-hot/symlog value classification from C51/DreamerV3-style work, and binary-filtered AWR from CRR-Binary-Mean) and measures their effect on multi-task/meta-RL benchmarks. There is no derivation chain in which an output quantity is defined in terms of the claimed result. The scale-resistance of the losses is shown analytically in Figure 3 by plotting the loss as a function of relative target error (Eqs. 1 vs. 3-4), but that only establishes the intended property of the loss functional; whether that property helps multi-task learning is tested empirically against held-out tasks and reference baselines, not derived from the property itself. The bin counts and wide symlog limits are per-domain hyperparameters, selected and reported (Appendix A), not fitted to the reported benchmark scores, so no fitted input is renamed as a prediction. The self-citations to AMAGO [57] provide the base Transformer actor-critic and engineering details (ensemble critics, multi-gamma updates), which is ordinary use of prior work by the same group; the novel contribution is the loss-family ablation on external benchmarks, which does not reduce to those citations. The paper also explicitly acknowledges the concurrent alternative explanation in [34] that classification losses help via representation learning and robustness to noisy targets, and it labels its own scale-invariance story as a hypothesis ('We will be exploring the impact of classification losses...'), so the mechanism is not smuggled in as a forced conclusion. The Appendix A admission that the dependent baseline uses a global task-agnostic PopArt layer is a legitimate baseline-strength concern, but it is a confound, not circularity. Nothing in the manuscript equates a target with an input, fits a parameter to a target and then calls it a prediction, or imports a deciding uniqueness theorem from the authors' prior work. The central empirical claim stands independently of the citations.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or physical postulates. The central claim relies on tuned hyperparameters (bin resolution) and on domain assumptions about the sufficiency of reward signals for task inference and about the current scale imbalance being the core bottleneck. These assumptions are reasonable for the evaluated benchmarks but are not proven.

free parameters (3)
  • Critic output bins B = Meta-World 128, POPGym 64, Procgen Easy 128, Procgen Memory-Hard 128, Atari 128, BabyAI 32
    Tuned per domain; sets the resolution of the two-hot value classifier and is central to the scale-invariance mechanism.
  • Symlog bounds (Rlow, Rhigh) = (-1e5, 1e5)
    Chosen wide to avoid tuning; defines the label space and affects how relative errors are mapped to loss.
  • Learning rate = 5e-4 for Meta-World, 1e-4 for other domains
    Standard optimization hyperparameter, set per domain and reported in Table 1.
assumptions (3)
  • domain assumption Uneven return scales across tasks are a primary cause of the multi-task optimization barrier.
    This is the paper's core hypothesis, supported indirectly by the Procgen reward-rescaling experiment but not proven as the unique mechanism.
  • domain assumption Transformer sequence models can perform in-context task inference from trajectories.
    Inherited from AMAGO and prior meta-RL; the paper does not re-derive this.
  • standard math Two-hot classification with symlog preserves comparable loss across return scales.
    Demonstrated numerically in Figure 3, not a formal theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers." pith.science (2026). https://pith.science/paper/NMPZT3KR

@misc{pith2026241111188,
  author       = {Pith},
  title        = {Pith review of: AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMPZT3KR}},
  note         = {Machine review of arXiv:2411.11188}
}
read the original abstract

Language models trained on diverse datasets unlock generalization by in-context learning. Reinforcement Learning (RL) policies can achieve a similar effect by meta-learning within the memory of a sequence model. However, meta-RL research primarily focuses on adapting to minor variations of a single task. It is difficult to scale towards more general behavior without confronting challenges in multi-task optimization, and few solutions are compatible with meta-RL's goal of learning from large training sets of unlabeled tasks. To address this challenge, we revisit the idea that multi-task RL is bottlenecked by imbalanced training losses created by uneven return scales across different tasks. We build upon recent advancements in Transformer-based (in-context) meta-RL and evaluate a simple yet scalable solution where both an agent's actor and critic objectives are converted to classification terms that decouple optimization from the current scale of returns. Large-scale comparisons in Meta-World ML45, Multi-Game Procgen, Multi-Task POPGym, Multi-Game Atari, and BabyAI find that this design unlocks significant progress in online multi-task adaptation and memory problems without explicit task labels.

Figures

Figures reproduced from arXiv: 2411.11188 by the authors.

Figure 1
Figure 1. Task Spaces in RL Generalization. Meta-RL agentso adapt to dense variations of a core task. Multi-Task RL overcomes optimization challenges of learning from isolated tasks. Scalable ideas from both areas allow us to extend adaptive agents towards increasingly general behavior. A truly general agent would also be able to play entirely different games. We will use the term “task" to refer to this kind of multi-game ge… view at source ↗
Figure 2
Figure 2. Transformer-based Actor-Critic Architecture. We study multi-task adaptation without task labels by building on the flexible memory-based framework in which sequence models optimize standard RL objectives across trajectory inputs (Section 2). Agents observe trajectory slices from a replay buffer with a context length of up to l timesteps (τt−l:t). The information re￾vealed between timesteps (oi , ai−1, ri−1, di−1) is… view at source ↗
Figure 3
Figure 3. Scale-Resistant Value Regression. We plot the value of the standard critic loss (Eq. 1) as a function of the relative prediction error of the TD target (y) across four orders of magnitude (left). Y-axes are self-normalized according to the largest displayed value. Two-Hot classification (Eq. 4) maps the same relative error to similar loss values across the different absolute return scales of each task (right). We ca… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Meta-World ML45 Train Task Results. (Left) Coverage of the 45 manipulation skills measured by an adaptation horizon success rate ≥ 2/3. (Center) Average success rate over tasks, variants, and 3-episode rollouts. Reference scores for MuZero and RL2 -PPO are gathered fro…
Figure 5
Figure 5. Figure 5: Multi-Task POPGym Results. Re￾turns are normalized by single-task experts trained for 15M timesteps. Error bars indicate the maxi￾mum and minimum returns across three trials. Multi-Task POPGym. Meta-World is de￾signed as a meta-learning benchmark, but its adaptation do…
Figure 6
Figure 6. Figure 6: Return Scales in Multi-Game Procgen. We evaluate over 2 episodes in unseen levels after optimizing default or rescaled reward functions. Scores are averaged over two model checkpoints and converted to the default scale. Error bars indicate the difference between two 27…
Figure 7
Figure 7. Figure 7: Learning from Rescaled Returns. (Left) Procgen test level returns normalized across games according to the standard benchmark scale [28]. (Center) Frequencies of the critic classifica￾tion label index with the highest probability throughout training. We trim the y-axis…
Figure 8
Figure 8. Figure 8: Multi-Task Procgen in Memory-Hard Mode. We measure policy performance across the two episodes of its adaptation window. Results are averaged over 30M frames in unseen test levels. Inspired by the success of the “Ind. / Ind.” learning update in the easy distribution of …
Figure 9
Figure 9. Figure 9: Multi-Game Atari Without Reward Clipping. We train a single policy on 10 games simultaneously. Results are plotted relative to human performance (left axis) and the raw scale of returns (right axis). Error bars denote the maximum and minimum over four random trials. Mu…
Figure 10
Figure 10. Figure 10: Multli-Task BabyAI. Results are the average over four training seeds and plotted according to the median, mean, and interquartile range over the task set. PickupDistDebug-v0 PickupLoc-v0 PickupDist-v0 OpenDoorDebug-v0 Pickup-v0 Unlock-v0 GoToImpUnlock-v0 UnblockPickup…
Figure 11
Figure 11. Figure 11: In-Context BabyAI. We measure the average return of “Ind. / Ind.” agents by attempt in unseen tasks and/or layouts (variants). Results are arbitrarily sorted in order of increasing second￾episode return and are the average of four training seeds and 10M evaluation tim…
Figure 12
Figure 12. Figure 12: Scale-Resistant Regression. We plot the critic loss as a function of the relative prediction error of the value target (y) across several orders of magnitude. Standard MSE (top left) assigns significantly different weights to the same relative inaccuracy across tasks …
Figure 13
Figure 13. Figure 13: Bin Counts in POPGym. We compare two label space sizes (B) with the same Rlow and Rhigh limits on two single-task POPGym environments [27]. 0 2 4 6 Timesteps 1e6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Average Return With Symlog B = 32 B = 16 B = 4 B = 2 0 2 4 6 Timesteps 1e6…
Figure 14
Figure 14. Figure 14 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Imbalanced Procgen Datasets. We measure the inflow of experience to the replay buffer by game. Climber accounts for much of our early training data but the buffer is rebalanced as policies improve. One detail is that selecting a random game between resets does not acc…
Figure 16
Figure 16. Figure 16: One-Shot Multi-Task POPGym. Plotted on the raw scale of returns in each environment. Error bars denote the maximum and minimum results over three trials. Single-Task reference scores at 15M timesteps indicate the best of 14 sequence model backbones trained by PPO in t…
Figure 17
Figure 17. Figure 17: Two-Episode Multi-Task BabyAI. Held-out test tasks are highlighted in red. Error bars denote the maximum and minimum value of four random trials. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Three-Episode Meta-World ML45 Learning Curves (Part 1/7) 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Three-Episode Meta-World ML45 Learning Curves (Part 2/7) 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Three-Episode Meta-World ML45 Learning Curves (Part 3/7) 27 [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Three-Episode Meta-World ML45 Learning Curves (Part 4/7) 28 [PITH_FULL_IMAGE:figures/full_fig_p028_21.png]
Figure 22
Figure 22. Figure 22: Three-Episode Meta-World ML45 Learning Curves (Part 5/7) 29 [PITH_FULL_IMAGE:figures/full_fig_p029_22.png]
Figure 23
Figure 23. Figure 23: Three-Episode Meta-World ML45 Learning Curves (Part 6/7) 30 [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Three-Episode Meta-World ML45 Learning Curves (Part 7/7) 31 [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

115 extracted references · 26 canonical work pages

  1. [34]

    Stop re- gressing: Training value functions via classification for scalable deep rl

    Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop re- gressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024

    arXiv 2024

  2. [1]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  3. [2]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  4. [3]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020

    2020

  5. [4]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03874, 2021. URL https://api.semanticscholar. org/CorpusID:232134851

    arXiv 2021

  6. [5]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2019

  7. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877– 1901, 2020

    1901

  8. [7]

    General-purpose in-context learning by meta-learning transformers

    Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022

    arXiv 2022

Show all 115 references
  1. [8]

    A survey of meta-reinforcement learning

    Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023

    2023 arXiv

  2. [9]

    Learning to reinforcement learn

    Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016

    2016 arXiv

  3. [10]

    Rl 2: Fast reinforcement learning via slow reinforcement learning

    Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016

    2016 arXiv

  4. [11]

    Meta reinforcement learning as task inference

    Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424, 2019

    1905 arXiv

  5. [12]

    Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability

    Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in neural information processing systems, 34:25502–25515, 2021

    2021

  6. [13]

    Fast adaptation via meta reinforcement learning

    Luisa Zintgraf. Fast adaptation via meta reinforcement learning. PhD thesis, University of Oxford, 2022. 11

    2022

  7. [14]

    On the effectiveness of fine-tuning versus meta-reinforcement learning

    Mandi Zhao, Pieter Abbeel, and Stephen James. On the effectiveness of fine-tuning versus meta-reinforcement learning. Advances in Neural Information Processing Systems, 35:26519– 26531, 2022

    2022

  8. [15]

    Investigating multi-task pretraining and generalization in reinforcement learning

    Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https:// openreview....

    2023

  9. [16]

    Probing transfer in deep reinforcement learning without task engineering

    Andrei Alex Rusu, Sebastian Flennerhag, Dushyant Rao, Razvan Pascanu, and Raia Hadsell. Probing transfer in deep reinforcement learning without task engineering. In Conference on Lifelong Learning Agents, pages 1231–1254. PMLR, 2022

    2022

  10. [17]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  11. [18]

    Parameterizing non-parametric meta- reinforcement learning tasks via subtask decomposition

    Suyoung Lee, Myungsik Cho, and Youngchul Sung. Parameterizing non-parametric meta- reinforcement learning tasks via subtask decomposition. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum? id=JX6UloWrmE

    2023

  12. [19]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vin...

    2022

  13. [20]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2023

    2023

  14. [21]

    Jack of all trades, master of some, a multi-purpose transformer agent.arXiv preprint arXiv:2402.09844, 2024

    Quentin Gallouédec, Edward Beeching, Clément Romac, and Emmanuel Dellandréa. Jack of all trades, master of some, a multi-purpose transformer agent.arXiv preprint arXiv:2402.09844, 2024

    2024 arXiv

  15. [22]

    Causes and cures for interference in multilingual translation

    Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. Causes and cures for interference in multilingual translation. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

    2023

  16. [23]

    Do current multi-task optimization methods in deep learning even help? Advances in neural information processing systems, 35:13597–13609, 2022

    Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? Advances in neural information processing systems, 35:13597–13609, 2022

    2022

  17. [24]

    A survey on multi-task learning.IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609, 2021

    Yu Zhang and Qiang Yang. A survey on multi-task learning.IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609, 2021

    2021

  18. [25]

    Multi-task deep reinforcement learning with popart

    Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796–3803, 2019

    2019

  19. [26]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  20. [27]

    POP- Gym: Benchmarking partially observable reinforcement learning

    Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POP- Gym: Benchmarking partially observable reinforcement learning. In The Eleventh Interna- tional Conference on Learning Representations, 2023. URL https://openreview.net/ forum?id=chDrutUTs0K

    2023

  21. [28]

    Leveraging procedural generation to benchmark reinforcement learning

    Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020. 12

    2020

  22. [29]

    The arcade learning environment: An evaluation platform for general agents

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013

    2013

  23. [30]

    BabyAI: First steps towards grounded language learning with a human in the loop

    Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations,

  24. [31]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  25. [32]

    Critic regularized regression

    Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768–7778, 2020

    2020

  26. [33]

    Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions

    Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Ku- mar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning , pages 3909–3928. ...

    2023

  27. [35]

    Offline actor-critic reinforcement learning scales to large models

    Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, et al. Offline actor-critic reinforcement learning scales to large models. arXiv preprint arXiv:2402.05546, 2024

    2024 arXiv

  28. [36]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021

    2021

  29. [37]

    Multi-game decision transformers

    Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921–27936, 2022

    2022

  30. [38]

    Contextual markov decision processes

    Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259, 2015

    2015 arXiv

  31. [39]

    Reinforcement learning, fast and slow

    Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):408–422, 2019

    2019

  32. [40]

    A survey of generalisa- tion in deep reinforcement learning, 2022

    Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisa- tion in deep reinforcement learning, 2022

    2022

  33. [41]

    Contextualize me–the case for context in reinforcement learning

    Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me–the case for context in reinforcement learning. arXiv preprint arXiv:2202.04500, 2022

    2022 arXiv

  34. [42]

    Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices

    Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International conference on machine learning, pages 6925–6935. PMLR, 2021

    2021

  35. [43]

    Efficient off-policy meta-reinforcement learning via probabilistic context variables

    Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331–5340. PMLR, 2019. 13

    2019

  36. [44]

    Splagger: Split aggregation for meta-reinforcement learning

    Jacob Beck, Matthew Jackson, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Splagger: Split aggregation for meta-reinforcement learning. arXiv preprint arXiv:2403.03020, 2024

    2024 arXiv

  37. [45]

    In-context reinforcement learning with algorithm distillation

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215, 2022

    arXiv

  38. [46]

    Supervised pretraining can learn in-context reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  39. [47]

    Generalization to new sequential decision making tasks with in-context learning.arXiv preprint arXiv:2312.03801, 2023

    Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Generalization to new sequential decision making tasks with in-context learning.arXiv preprint arXiv:2312.03801, 2023

    2023 arXiv

  40. [48]

    Cross-episodic curriculum for transformer agents

    Lucy Xiaoyang Shi, Yunfan Jiang, Jake Grigsby, Linxi Fan, and Yuke Zhu. Cross-episodic curriculum for transformer agents. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    A simple neural attentive meta-learner

    Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141, 2017

    2017 arXiv

  42. [50]

    Transformers are meta-reinforcement learners

    Luckeciano C Melo. Transformers are meta-reinforcement learners. In International Confer- ence on Machine Learning, pages 15340–15359. PMLR, 2022

    2022

  43. [51]

    Structured state space models for in-context reinforcement learning

    Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. arXiv preprint arXiv:2303.03982, 2023

    2023 arXiv

  44. [52]

    Meta-q-learning

    Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning. arXiv preprint arXiv:1910.00125, 2019

    1910 arXiv

  45. [53]

    Recurrent off-policy baselines for memory-based continuous control

    Zhihan Yang and Hai Nguyen. Recurrent off-policy baselines for memory-based continuous control. arXiv preprint arXiv:2110.12628, 2021

    2021 arXiv

  46. [54]

    Recurrent model-free rl can be a strong baseline for many pomdps, 2022

    Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps, 2022

    2022

  47. [55]

    Human-timescale adaptation in an open-ended task space

    Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023

    2023 arXiv

  48. [56]

    When do transformers shine in rl? decoupling memory from credit assignment

    Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in rl? decoupling memory from credit assignment. arXiv preprint arXiv:2307.03864, 2023

    2023 arXiv

  49. [57]

    AMAGO: Scalable in-context reinforcement learning for adaptive agents

    Jake Grigsby, Linxi Fan, and Yuke Zhu. AMAGO: Scalable in-context reinforcement learning for adaptive agents. In The Twelfth International Conference on Learning Representations,

  50. [58]

    Adapting auxiliary losses using gradient similarity

    Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224, 2018

    2018 arXiv

  51. [59]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020

    2020

  52. [60]

    Robust optimization for multilingual translation with imbalanced data

    Xian Li and Hongyu Gong. Robust optimization for multilingual translation with imbalanced data. Advances in Neural Information Processing Systems, 34:25086–25099, 2021

    2021

  53. [61]

    Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models

    Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representations, 2020. 14

    2020

  54. [62]

    Conflict-averse gradient descent for multi-task learning

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems , 34: 18878–18890, 2021

    2021

  55. [63]

    Scalarization for multi- task and multi-domain learning at scale

    Amelie Royer, Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Scalarization for multi- task and multi-domain learning at scale. Advances in Neural Information Processing Systems, 36, 2024

    2024

  56. [64]

    In defense of the unitary scalarization for deep multi-task learning

    Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35:12169–12183, 2022

    2022

  57. [65]

    Multi-task reinforcement learning with context-based representations

    Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning , pages 9767–9779. PMLR, 2021

    2021

  58. [66]

    Offline q-learning on diverse multi-task data both scales and generalizes

    Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  59. [67]

    Sharing knowledge in multi-task deep reinforcement learning

    Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. arXiv preprint arXiv:2401.09561, 2024

    2024 arXiv

  60. [68]

    Garage: A toolkit for reproducible reinforcement learning research

    The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https://github.com/rlworkgroup/garage, 2019

    2019

  61. [69]

    Some considerations on learning to explore via meta-reinforcement learning

    Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01118, 2018

    2018 arXiv

  62. [70]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–

  63. [71]

    Promp: Proximal meta-policy search

    Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018

    2018 arXiv

  64. [72]

    MAMBA: an effective world model approach for meta-reinforcement learning

    Zohar Rimon, Tom Jurgenson, Orr Krupnik, Gilad Adler, and Aviv Tamar. MAMBA: an effective world model approach for meta-reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=1RE0H6mU7M

    2024

  65. [73]

    Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents

    Jane X Wang, Michael King, Nicolas Pierre Mickael Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, H Francis Song, et al. Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents. In Thirty-fifth Conference on...

    2021

  66. [74]

    Procedural generalization by planning with self- supervised world models

    Ankesh Anand, Jacob C Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Theophane Weber, and Jessica B Hamrick. Procedural generalization by planning with self- supervised world models. In International Conference on Learning Representations, 2022. URL http...

    2022

  67. [75]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  68. [76]

    Soft actor-critic for discrete action settings

    Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019

    1910 arXiv

  69. [77]

    Randomized ensembled double q-learning: Learning fast without a model

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021. 15

    2021 arXiv

  70. [78]

    Hyperbolic discounting and learning over multiple horizons

    William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865, 2019

    1902 arXiv

  71. [79]

    Distributional reinforcement learning

    Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023

    2023

  72. [80]

    A distributional perspective on reinforce- ment learning

    Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International conference on machine learning , pages 449–458. PMLR, 2017

    2017

  73. [81]

    Master- ing atari, go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Master- ing atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  74. [82]

    Muesli: Combining improvements in policy optimization

    Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In International conference on machine learning, pages 4214–4226. PMLR, 2021

    2021

  75. [83]

    Recurrent experience replay in distributed reinforcement learning

    Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. InInternational conference on learning representations, 2018

    2018

  76. [84]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  77. [85]

    Maximum a posteriori policy optimisation

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018

    2018 arXiv

  78. [86]

    Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  79. [87]

    Exponentially weighted imitation learning for batched historical data

    Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018

    2018

  80. [88]

    Bail: Best-action imitation learning for batch deep reinforcement learning

    Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353–18363, 2020

    2020

  81. [89]

    Keep doing what worked: Behavioral modelling priors for offline reinforcement learning

    Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.0...

    2002 arXiv

  82. [90]

    A closer look at advantage-filtered behavioral cloning in high- noise datasets

    Jake Grigsby and Yanjun Qi. A closer look at advantage-filtered behavioral cloning in high- noise datasets. arXiv preprint arXiv:2110.04698, 2021

    2021 arXiv

  83. [91]

    Overcoming exploration in reinforcement learning with demonstrations

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE inter- national conference on robotics and automation (ICRA), pages 6292–6299. IEEE, 2018

    2018

  84. [92]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  85. [93]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  86. [94]

    Revisiting fundamentals of experience replay

    William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Interna- tional Conference on Machine Learning, pages 3061–3071. PMLR, 2020. 16

    2020

  87. [95]

    Reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021

    2021

  88. [96]

    You can’t count on luck: Why decision transformers fail in stochastic environments

    Keiran Paster, Sheila McIlraith, and Jimmy Ba. You can’t count on luck: Why decision transformers fail in stochastic environments. arXiv preprint arXiv:2205.15967, 2022

    2022 arXiv

  89. [97]

    Hierarchical transformers are efficient meta-reinforcement learners

    Gresa Shala, André Biedenkapp, and Josif Grabocka. Hierarchical transformers are efficient meta-reinforcement learners. arXiv preprint arXiv:2402.06402, 2024

    2024 arXiv

  90. [98]

    Learning phrase representations using rnn encoder– decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder– decoder for statistical machine translation. InProceedings of the 2014 Conference on Empirical Methods ...

    2014

  91. [99]

    Reinforcement learning with fast and forgetful memory

    Steven Morad, Ryan Kortvelesy, Stephan Liwicki, and Amanda Prorok. Reinforcement learning with fast and forgetful memory. Advances in Neural Information Processing Systems, 36, 2024

    2024

  92. [100]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence,...

    2018

  93. [101]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  94. [102]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neu...

    2023

  95. [103]

    Xland-minigrid: Scalable meta-reinforcement learning environments in jax

    Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforcement learning environments in jax. arXiv preprint arXiv:2312.12044, 2023

    2023 arXiv

  96. [104]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  97. [105]

    Normformer: Improved transformer pretraining with extra normalization

    Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456, 2021

    2021 arXiv

  98. [106]

    Stabilizing transformer training by preventing attention entropy collapse

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770– 40803. PMLR, 2023

    2023

  99. [107]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning,...

    2018

  100. [108]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018

    2018

  101. [109]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021

    2021 arXiv

  102. [110]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 17

    2016 arXiv

  103. [111]

    dependent

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasi...

    2023

  104. [114]

    Our results default to IMPALA with additional Group Normalization [108, 66]

    and the residual IMPALA CNN [107]. Our results default to IMPALA with additional Group Normalization [108, 66]. We apply the random pad and crop data augmentation from DrQV2 [109] to Procgen and Atari experiments. Image features extracted by the CNN are normalized before being...

  105. [115]

    Wide Limit

    by creating labels for B return bins b = [ b0, b1, . . . , bB]. Bins are typically spaced at fix intervals between pre-defined upper and lower bounds on the return (b0 = Rlow, bB = Rhigh). The critic network ( QB) outputs (softmax) probabilities over these bins, and its value ...

  106. [2019]

    URL https://openreview.net/forum?id=rJeXCo0cYX

  107. [2024]

    URL https://openreview.net/forum?id=M6XWoEdmwf

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.