Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change

Jonathan Clifford Balloch

REVIEW 5 major objections 5 minor 246 references

Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RL agents adapt to sudden change via broad exploration and preserved knowledge

desk verdict A substantial tool-building dissertation on online test-time adaptation in RL, with a benchmark and three methods worth engaging, but the universal 'must' thesis is stronger than the evidence, which is confined to transition-dynamics changes under fixed action/observation spaces. read the letter →

arxiv 2505.10330 v1 pith:XJGZMNSH submitted 2025-05-15 cs.LG cs.AI

Jonathan Clifford Balloch This is my paper · ORCID

classification cs.LGcs.AI

keywords reinforcementlearningonlinetest-timeadaptationnoveltyexplorationstrategiesmodel-basedworldmodelscatastrophicforgettingconceptbottleneck

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This dissertation claims that efficient online adaptation to sudden environmental change requires two capabilities: exploration and sampling strategies that prioritize task-agnostic interactions to reduce distribution shift, and selective preservation of reusable prior knowledge in symbolic or learned representations. The author grounds these claims in a novelty taxonomy and a grid-world benchmark, then tests them across eleven exploration algorithms, a new priority-sampling method for model-based RL, a neuro-symbolic world model, and a concept-bottleneck world model. A sympathetic reader would care because deployed RL systems must react to changes they were not trained on, and the dissertation offers a coherent explanation of why some methods adapt quickly while others fail.

What carries the argument

The argument is carried by three complementary mechanisms. First, the novelty ontology and the NovGrid benchmark define a shared vocabulary—object versus action novelties, unary versus relational changes, and barrier/delta/shortcut solution changes—so that adaptation experiments can be compared across environment types. Second, the exploration study decomposes methods into characteristics (stochasticity, explicit diversity, separate objective, temporal locality) and shows that stochastic and diversity-seeking methods generalize best to new tasks. Third, two world-model architectures embody the knowledge-preservation claim: WorldCloner, whose symbolic rule set uses axis-aligned bounding intervals (AABIs) with rule creation, relaxation, and collision resolution, so a single post-novelty observation can update a rule; and Concept Bottleneck World Models, which interpose a concept bottleneck in the Recurrent State Space Model so that gradients toward the task must pass through concept-grounded latents. DOPS supplies the sampling side by blending count-based and adversarial priorities for the world model while splitting the actor and critic batches by TD-error magnitude, balancing distribution overlap with objective-specific learning.

What would settle it

Run WorldCloner and Dreamer on a novelty that changes the reward or goal structure (for example, relocating the goal object to a new room) while keeping transition dynamics, the action space, and the observation space identical. The thesis holds that adaptation demands selective preservation of transition knowledge and task-agnostic sampling; if uniform-sampling Dreamer adapts as fast as WorldCloner under such a pure reward change, the claimed mechanism's scope would not generalize beyond transition novelties.

Watch

Extended reading notes

Core claim

The central claim is the thesis statement itself: to adapt online to novel changes efficiently, an RL agent must (1) explore and sample in a task-agnostic way so that the data it learns from covers the pre- and post-change worlds without being overfit to the old optimal trajectory, and (2) keep prior knowledge in structured representations—symbolic rules or concept-anchored latents—that can be updated in place without disturbing unaffected components. Across the dissertation, this claim is supported by controlled experiments: exploration methods based on stochasticity and explicit diversity adapt faster than curiosity-based or temporally local methods; DOPS, which samples world-model, actor, and critic data with different priorities, improves both tabula rasa learning and adaptation in Dreamer-style model-based RL; WorldCloner's interval-based symbolic rules update after a single post-novelty transition and drive imagination-based policy updates; and Concept Bottleneck World Models retain concept knowledge across adaptation better than unstructured world models. Read sympathetically, the dissertation establishes that careful management of data and representation, rather than more compute or larger models, is the lever for sample-efficient adaptation.

Load-bearing premise

The experiments assume that a novelty changes only the transition dynamics while observation and action dimensionality and the goal reward stay fixed; if real-world novelties change the action set, observation structure, or the mission, the proposed methods are not shown to transfer.

Editorial extensions

If this is right

If the thesis is correct, adaptation efficiency becomes a design target: agents should be built with exploration and sampling that are explicitly task-agnostic rather than optimized purely for fast convergence on a single task.
DOPS-type sampling implies that in any interleaved model-based RL architecture, the world model, actor, and critic should be trained on differently prioritized data, with low-TD-error samples for the actor to avoid gradient overshoot during novelty.
WorldCloner shows that symbolic representations of transition rules can be updated from a single observation, implying that hybrid neuro-symbolic architectures can dramatically reduce the number of environment interactions needed to re-adapt.
CBWM shows that grounding latent states in human-interpretable concepts preserves knowledge across domain shifts, so interpretability and adaptation are compatible rather than competing goals.
The barrier-novelty results indicate a boundary: when the post-novelty optimal solution is much longer than the source, exploration methods alone cannot transfer much prior knowledge, so adaptation gains are limited.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the dimensionality-restriction assumption in Chapter 3 is relaxed, the relative advantage of the proposed methods may shrink; an obvious extension is an ontology dimension for action- and observation-space changes, and testing DOPS and CBWM under those novelties.
The success of stochasticity and diversity in adaptation may partly reflect that these methods prevent overfitting to the source policy's state distribution; this suggests a testable recipe: combining DOPS sampling with WorldCloner-style symbolic rules should compound adaptation speed, though the dissertation does not test that combination.
The CBWM result that orthogonality loss also preserves concept similarity suggests an unsupervised route: concept-like latent factors can potentially be discovered without labels, which would relax the supervision requirement of concept bottlenecks.
Because the thesis treats robustness and adaptation as complementary, an interesting test is whether agents that are pre-trained with domain randomization (robustness) plus the proposed adaptation mechanisms adapt even faster; this is not examined in the dissertation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The headline: this is a solid, buildable dissertation that gives the OTTA-in-RL community a benchmark, a taxonomy, and three concrete methods. But the central 'must' claim is not earned. The experiments only cover novelties that change transition dynamics under fixed observation/action spaces and unchanged task rewards, so the thesis is more accurately a 'can help' than a 'must'.

What's genuinely new and useful: NovGrid and its novelty ontology give researchers a common vocabulary and testbed. The Chapter 4 comparison of eleven exploration algorithms across discrete and continuous novelties is a real contribution; the finding that stochasticity and explicit diversity beat task-specific intrinsic objectives is a useful empirical baseline. DOPS has a sensible idea (separate sampling for world model vs actor-critic) and shows gains over Dreamer and Curious Replay in the reported settings. WorldCloner and CBWM are creative attempts at knowledge preservation, and CBWM's concept-cosine-similarity results are the most interesting piece of evidence that bottleneck structure helps retain knowledge.

Now the soft spots. The thesis statement uses 'must,' but no experiment manipulates both capabilities factorially, so necessity is never tested. Section 3.1 narrows the setting explicitly, and the dissertation itself flags the limitation in 3.4 and 8.3.1. That honesty is good, but the abstract and summary still carry the universal phrasing. WorldCloner's advantage is inconsistent: Table 6.1 shows it needing 9.8E5 steps vs Dreamer's 5.3E5 on DoorKeyChange. The adaptive-efficiency metric filters out non-converged runs, a selection bias acknowledged in the text but still making the headline numbers flattering. WorldCloner's results are averaged over three seeds with no significance tests. DOPS's 'theoretical analysis' is heuristic; it identifies distribution-shift mechanisms but delivers no formal guarantees. And there is no shipped code, which limits the reusability of NovGrid and the methods.

Who this is for: any RL researcher working on non-stationarity, transfer, or world models will get value from the benchmark and the comparison studies. It deserves a serious referee. My recommendation: send it to peer review, but ask the authors to soften the 'must' to 'helps,' to report all runs or a principled subset, and to provide code or at least more seeds for WorldCloner. The package is worth engaging with; the claims need tightening.

Referee Report

5 major / 5 minor

Summary. This dissertation-style manuscript argues that efficient online adaptation of RL agents to sudden environmental novelty requires two capabilities: (1) exploration and sampling strategies that reduce distribution shift and prioritize task-agnostic data, and (2) selective preservation of reusable prior knowledge in symbolic or structured learned representations. The work introduces an ontology of novelties, the NovGrid benchmark, and four main technical contributions: a broad empirical comparison of eleven exploration algorithms for online test-time adaptation (OTTA), the DOPS prioritized-sampling method for Dreamer-style model-based RL, the neuro-symbolic WorldCloner system, and Concept Bottleneck World Models (CBWMs). Each chapter reports empirical results on NovGrid and/or continuous-control transfer tasks, and the manuscript closes with conclusions, limitations, and future-work directions covering continuous novelty, MDP-distance quantification, and multi-agent settings.

Significance. If the empirical findings hold, the manuscript makes a useful contribution to OTTA in RL: it provides a reusable benchmark (NovGrid) with an explicit novelty ontology and evaluation metrics, a systematic comparison of exploration methods, and two concrete mechanisms (DOPS and WorldCloner/CBWM) for improving adaptation efficiency. The dissertation is unusually candid about its limitations, and the authors deserve credit for building public-facing infrastructure (NovGrid, baselines) and for being explicit in Chapter 3 about the fixed observation/action-space assumption. However, the central normative claim—that efficient adaptation 'must' have both listed capabilities—is not established by the evidence, which demonstrates sufficiency of particular instantiations within a restricted novelty class rather than necessity. The filtering of non-converged runs, the small number of seeds, and the heuristic nature of the DOPS 'theoretical analysis' further weaken the load-bearing empirical support. The contribution is significant and publishable in principle, but the thesis statement and several key analyses need substantial tightening.

major comments (5)

[§1.1, §3.1, §8.3.1] The thesis statement's normative 'must' is stronger than the evidence. Section 3.1 explicitly restricts the study to novelties that preserve observation/action dimensionality and the mission, reducing all evaluated novelties to transition-dynamics changes, and the manuscript itself flags this in §3.4 and §8.3.1. No experiment combines or removes the two claimed capabilities in a factorial way, so the results cannot show that both are necessary; they show that particular exploration/sampling and knowledge-preservation methods can improve adaptation in specific environments. The claim should be weakened to a sufficiency or scoped claim, e.g., 'efficient adaptation can be achieved by...'.
[§4.4, Tables 4.2 and 4.3] The adaptive-efficiency metric in Table 4.2 filters out all runs that did not converge on both tasks, and the Tr-AUC metric in Table 4.3 filters out runs that did not converge on the first task. This is a selection bias: algorithms that fail often are evaluated only on their successful seeds, which can produce spuriously favorable efficiency numbers. The convergence frequencies in Appendix Table 2 should be reported prominently in the main text, and all-run analyses (or survival-style estimators) should be added before the chapter's conclusions about NoisyNets, RE3, and DIAYN can be accepted.
[§5.2, §5.3.1] The 'theoretical analysis' of DOPS is heuristic rather than formal: the four learning categories, the distribution-shift claims, and the proposed sampling remedies are motivated by prior work (CR, PER, LA3P) but no theorem or quantitative bound is derived. Moreover, the empirical support in §5.3.1 is limited to two Walker2d ThighLengthChange scenarios shown in Figure 5.1, with no significance tests reported, while the NovGrid experiments described in §5.3 are not given a results table or curves in the main text. The chapter should either present the NovGrid results with uncertainty estimates or explicitly scope the claim to continuous control.
[§6.2, Table 6.1] WorldCloner's central empirical claim rests on Table 6.1, which reports averages over only three runs with no significance tests or confidence intervals. Additionally, the novelty-detection threshold n=2 and the imagination-real mixing ratio η are chosen heuristically ('based on testing multiple values') without a reported ablation, so the sensitivity of the main result to these hyperparameters is unknown. The claim that knowledge preservation 'dramatically' improves adaptation efficiency is also contradicted by the DoorKeyChange row, where WorldCloner (9.8E5 steps) is slower than DreamerV2 (5.3E5 steps). The authors should add error bars/significance tests, an ablation of n and η, and a more nuanced interpretation of the inconsistent efficiency advantage.
[§7.4.2] The CBWM concept-retention result in §7.4.2 is partly by construction: the CBWM bottleneck is explicitly trained with concept-supervision labels, so its high concept cosine similarity across adaptation is expected. The comparison with BWM+O is interesting, but no statistical test or seed count is reported, and the orthogonality-loss baseline is noted to have 'high variance.' The chapter should acknowledge more directly that the concept-preservation advantage is a designed property of the architecture, not an emergent finding, and should provide variance statistics for the learning curves in Figure 7.6.

minor comments (5)

[§4.2, Table 4.1] Table 4.1 lists 'EVD' and 'RIS' as local exploration methods, but the text and elsewhere consistently use 'REVD' and 'RISE'; these abbreviations should be unified.
[§4.5] The sentence 'We built on these findings to do the work described in Chapter 1' appears to reference the wrong chapter; the follow-up work is described in Chapter 5.
[§5.2, Algorithm 2] Several equation references are placeholders (e.g., 'Equation 2.1.2' and 'Equation 2.2'), and Algorithm 2 contains an incomplete phrase, 'Compute the of the imagined trajectories'; these should be corrected.
[§1.1, Summary] The abstract and summary use 'catastrophically forgetting,' which is standard, but the introduction says 'catastrophic inference' instead of 'catastrophic forgetting'; the terminology should be made consistent throughout.
[§6.1.2] The rule-collision procedure is described as a 'min-cut' operation, but the text immediately explains it as a split along the largest feature axis; this terminology should be aligned with the actual operation to avoid confusing readers familiar with graph cuts.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor by-construction rule-update claim in WorldCloner; central adaptation claims remain empirical and independent.

fitted input called prediction [Section 6.1.3, Novelty Detection (rule update after novelty detection; Algorithm 3)]
"The rule model can thus be updated with as little as a single iteration of the rule learning algorithm, with guaranteed improved next-state prediction."

Algorithm 3's Rule Creation stores the observed prior state as the state precondition and the observed state difference as the effect (cs = st-1, e = st - st-1). The 'guaranteed improved next-state prediction' is therefore a prediction of the exact transition used to update the rule: for the stored precondition and action, the output is the stored effect by construction. This is fitting the training transition, not predicting unseen post-novelty states. The single-example update speed is a definitional property of the rule store, so any claim that it demonstrates generalization or faster adaptation from one example is not independent evidence.

full rationale

The dissertation is primarily an empirical study: exploration characteristics (Ch. 4), DOPS sampling (Ch. 5), WorldCloner (Ch. 6), and CBWM (Ch. 7) are evaluated against standard baselines (PPO, DreamerV2, Curious Replay, BWM) on task performance, sample efficiency, and adaptation curves. These comparisons are external to the fitted parameters, and the central claims do not reduce to the definitions of the proposed methods. NovGrid is an author-built benchmark, but it is specified in Chapter 3 and the methods are not tuned to its metrics in a way that makes the measured adaptation gains tautological; the results are still comparative against external algorithms. The one by-construction component is WorldCloner's rule-update 'guaranteed next-state prediction,' which is a restatement of the stored transition rather than a predictive result; this is a minor subclaim and is not the basis of the baseline comparisons. The CBWM concept-retention plots are influenced by the supervised concept loss, but the adaptation claims are also supported by task-return curves against BWM and BWM+O, so they are not purely circular. The thesis statement's normative 'must' is broader than the tested scope (Sec. 3.1 restricts to fixed observation/action dimensionalities and unchanged task reward), but this is a scope or evidence gap, not a circularity. Overall, no load-bearing circularity: score 2 for one minor non-load-bearing by-construction property.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several domain assumptions about what constitutes a novelty and on hyperparameters tuned in the reported experiments. No new physical entities are postulated.

free parameters (4)

Novelty detection threshold n = 2
In WorldCloner, novelty is declared after n consecutive rule violations; n=2 was chosen by the authors after testing multiple values (Section 6.1.3).
Imagination-real mixing ratio eta = not stated in truncated text
Controls ratio of imagined to real rollouts in WorldCloner policy updates (Section 6.1.4).
Masking overlap fraction W = 0.5
DOPS parameter controlling overlap between actor and critic data batches; set to 0.5 following Saglam et al. (Section 5.3).
World model data split = 20% uniform / 80% Curious Replay
DOPS sampling mix for world model training, selected for the reported experiments (Section 5.3).

assumptions (4)

domain assumption A novelty is a sudden, unanticipatable, previously unseen change transforming a source MDP to a target MDP (Definition 1).
Underpins the entire OTTA formulation; restricts non-stationarity to a single abrupt transition.
domain assumption Observation and action space dimensionality remain consistent across novelty, and the agent's mission (goal reward) is unchanged (Section 3.1).
Used to design all NovGrid novelties; if novelties change action/observation spaces or task rewards, the conclusions may not transfer.
ad hoc to paper Exploration methods can be characterized along exploration principle (stochasticity, explicit diversity, separate objective) and temporal locality (global, local, time-independent) (Section 4.2).
The taxonomy is introduced for this study and is not derived from prior theory; the results are interpreted through this lens.
ad hoc to paper The world model objective mismatch hypothesis extends to separate learning categories within Dreamer (prediction, representation, policy, critic) (Section 5.1.2).
Motivates DOPS; presented as analysis rather than a proven theorem.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change." pith.science (2026). https://pith.science/paper/XJGZMNSH

@misc{pith2026250510330,
  author       = {Pith},
  title        = {Pith review of: Efficient Adaptation of Reinforcement Learning Agents to Sudden Environmental Change},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJGZMNSH}},
  note         = {Machine review of arXiv:2505.10330}
}

read the original abstract

Real-world autonomous decision-making systems, from robots to recommendation engines, must operate in environments that change over time. While deep reinforcement learning (RL) has shown an impressive ability to learn optimal policies in stationary environments, most methods are data intensive and assume a world that does not change between training and test time. As a result, conventional RL methods struggle to adapt when conditions change. This poses a fundamental challenge: how can RL agents efficiently adapt their behavior when encountering novel environmental changes during deployment without catastrophically forgetting useful prior knowledge? This dissertation demonstrates that efficient online adaptation requires two key capabilities: (1) prioritized exploration and sampling strategies that help identify and learn from relevant experiences, and (2) selective preservation of prior knowledge through structured representations that can be updated without disruption to reusable components.

Figures

Figures reproduced from arXiv: 2505.10330 by the authors.

**Figure 2.1.** The agent-environment interaction that is fundamental to reinforcement learn [PITH_FULL_IMAGE:figures/full_fig_p028_2_1.png] view at source ↗

**Figure 3.1.** The NovGrid environments, where the agent (red triangle) must get to the [PITH_FULL_IMAGE:figures/full_fig_p049_3_1.png] view at source ↗

**Figure 3.2.** Evaluation metrics illustrated against a notional performance curve for an agent. [PITH_FULL_IMAGE:figures/full_fig_p056_3_2.png] view at source ↗

Figures from the paper (22 more)

**Figure 4.1.** Figure 4.1: Environments and novelties used to evaluate the exploration algorithms and [PITH_FULL_IMAGE:figures/full_fig_p061_4_1.png]

**Figure 4.2.** Figure 4.2: Full learning and adaptation process of eleven RL exploration algorithms on [PITH_FULL_IMAGE:figures/full_fig_p064_4_2.png]

**Figure 4.3.** Figure 4.3: The Adaptive Efficiency and Tr-AUC inter-quartile mean plots for DoorKey [PITH_FULL_IMAGE:figures/full_fig_p067_4_3.png]

**Figure 4.4.** Figure 4.4: The reward plot from dm control Walker-Walk ThighIncrease delta novelty transfer task. The vertical line at 1E7 steps indicates where novelty was injected. The shaded areas represent the variance over all seeds. NoisyNets and DIAYN are the highest performing and most…

**Figure 4.5.** Figure 4.5: Results from the LavaSafe shortcut novelty. The vertical line at 1E7 steps indicates where novelty was injected. The shaded areas represent the variance over all seeds. Some of the exploration algorithms are able to find the shortcut, rising above the pre-novelty per…

**Figure 5.1.** Figure 5.1: This graphic shows the learning curves of DOPS and the baselines solving [PITH_FULL_IMAGE:figures/full_fig_p093_5_1.png]

**Figure 6.1.** Figure 6.1: The WorldCloner architecture. The purple module and black arcs represent the [PITH_FULL_IMAGE:figures/full_fig_p098_6_1.png]

**Figure 6.2.** Figure 6.2: Top shows example environmental states passed to the rule learner (changes [PITH_FULL_IMAGE:figures/full_fig_p100_6_2.png]

**Figure 6.3.** Figure 6.3: Rule Relaxation example, where the blue underlined precondition AABI corre [PITH_FULL_IMAGE:figures/full_fig_p103_6_3.png]

**Figure 6.4.** Figure 6.4: Rule Collision, and the resulting rule split and creation. The blue-underlined preconditions in the newly split Rule 1’ and Rule 1” indicate the feature dimension along which the original Rule 1 is split. The newly created Rule 2 accounts for the state transition tha…

**Figure 6.5.** Figure 6.5: This plot shows the adaptive performance of agents post-novelty in the [PITH_FULL_IMAGE:figures/full_fig_p111_6_5.png]

**Figure 6.6.** Figure 6.6: This plot shows the adaptive performance of agents post-novelty in the [PITH_FULL_IMAGE:figures/full_fig_p112_6_6.png]

**Figure 6.7.** Figure 6.7: This plot shows the adaptive performance of agents post-novelty in the [PITH_FULL_IMAGE:figures/full_fig_p113_6_7.png]

**Figure 6.8.** Figure 6.8: This shows the WorldCloner 1-step prediction error vs environment steps dur [PITH_FULL_IMAGE:figures/full_fig_p116_6_8.png]

**Figure 7.1.** Figure 7.1: This shows our novel CBWM architecture as it interacts with the agent and [PITH_FULL_IMAGE:figures/full_fig_p124_7_1.png]

**Figure 7.2.** Figure 7.2: Figure illustrating the three-stage CBWM training procedure for balancing task [PITH_FULL_IMAGE:figures/full_fig_p127_7_2.png]

**Figure 7.3.** Figure 7.3: Concept classification accuracy across different object and state concepts in [PITH_FULL_IMAGE:figures/full_fig_p136_7_3.png]

**Figure 7.4.** Figure 7.4: This shows the observation predictions of the LIBERO space. While the image [PITH_FULL_IMAGE:figures/full_fig_p137_7_4.png]

**Figure 7.5.** Figure 7.5: Plotting concept cosine similarity for individual concepts for the BWM, [PITH_FULL_IMAGE:figures/full_fig_p138_7_5.png]

**Figure 7.6.** Figure 7.6: The OTTA learning curves averaged over all tasks for CBWM, BWM+O, and [PITH_FULL_IMAGE:figures/full_fig_p139_7_6.png]

**Figure 1.** Figure 1: In this figure, we see the ground truth observation in (a), followed by the unmod [PITH_FULL_IMAGE:figures/full_fig_p176_1.png]

**Figure 2.** Figure 2: CBWM tasks designed for testing the impact of concepts on adaptation. [PITH_FULL_IMAGE:figures/full_fig_p181_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

246 extracted references · 54 canonical work pages

[1]

Mastering the game of go without human knowledge,

D. Silver et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017

2017
[2]

Mastering atari, go, chess and shogi by planning with a learned model,

J. Schrittwieser et al., “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020

2020
[3]

Grandmaster level in starcraft ii using multi-agent reinforcement learning,

O. Vinyals et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, no. 7782, pp. 350–354, 2019

2019
[4]

Dota 2 with large scale deep reinforcement learning,

C. Berner et al. , “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680, 2019

arXiv 1912
[5]

Agent57: Outperforming the human atari benchmark,

A. Badia et al., “Agent57: Outperforming the human atari benchmark,” inProceed- ings of the 37th International Conference on Machine Learning, Online, PMLR , vol. 119, 2020, p. 2020

2020
[6]

Reinforcement learning based recommender systems: A survey,

M. M. Afsar, T. Crump, and B. Far, “Reinforcement learning based recommender systems: A survey,” ACM Computing Surveys, vol. 55, no. 7, pp. 1–38, 2022

2022
[7]

Deepmind ai reduces google data centre cooling bill by 40%,

R. Evans and J. Gao, “Deepmind ai reduces google data centre cooling bill by 40%,” DeepMind blog, vol. 20, p. 158, 2016

2016
[8]

Gnu-rl: A precocial reinforcement learning so- lution for building hvac control using a differentiable mpc policy,

B. Chen, Z. Cai, and M. Berg ´es, “Gnu-rl: A precocial reinforcement learning so- lution for building hvac control using a differentiable mpc policy,” in Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation, 2019, pp. 316–325

2019

Show all 246 references

[9]

Magnetic control of tokamak plasmas through deep reinforce- ment learning,

J. Degrave et al., “Magnetic control of tokamak plasmas through deep reinforce- ment learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022

2022
[10]

Adversarial policies beat superhuman go ais,

T. T. Wang et al., “Adversarial policies beat superhuman go ais,” in International Conference on Machine Learning, PMLR, 2023, pp. 35 655–35 739

2023
[11]

Adaptation in constant utility non-stationary environments.,

M. L. Littman and D. H. Ackley, “Adaptation in constant utility non-stationary environments.,” in ICGA, Citeseer, 1991, pp. 136–142

1991
[12]

R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018

2018
[13]

Impact of timing in post-warning prepositioning decisions on performance measures of disaster management: A 163 real-life application,

S. Rezapour, R. Z. Farahani, and N. Morshedlou, “Impact of timing in post-warning prepositioning decisions on performance measures of disaster management: A 163 real-life application,” European Journal of Operational Research, vol. 293, no. 1, pp. 312–335, 2021

2021
[14]

Domain randomization for transferring deep neural networks from simulation to the real world,

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and sys- tems (IROS), IEEE, 2017, pp. 23–30

2017
[15]

Learning optimal adap- tation strategies in unpredictable motor tasks,

D. A. Braun, A. Aertsen, D. M. Wolpert, and C. Mehring, “Learning optimal adap- tation strategies in unpredictable motor tasks,” Journal of Neuroscience, vol. 29, no. 20, pp. 6472–6478, 2009

2009
[16]

Reward learning: Reinforcement, incentives, and expectations,

K. C. Berridge, “Reward learning: Reinforcement, incentives, and expectations,” in Psychology of learning and motivation, vol. 40, Elsevier, 2000, pp. 223–278

2000
[17]

Inte- grating animal temperament within ecology and evolution,

D. R ´eale, S. M. Reader, D. Sol, P. T. McDougall, and N. J. Dingemanse, “Inte- grating animal temperament within ecology and evolution,” Biological Reviews, vol. 82, no. 2, pp. 291–318, 2007. eprint: https://onlinelibrary.wiley.com/doi/pdf/ 10.1111/j.1469-185X.2007.00010.x

2007 arXiv
[18]

Deep neural networks for youtube rec- ommendations,

P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube rec- ommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems, New York, NY , USA, 2016

2016
[19]

Scheduling on a budget: Avoiding stale recommendations with timely updates,

R. Verachtert, O. Jeunen, and B. Goethals, “Scheduling on a budget: Avoiding stale recommendations with timely updates,”Machine Learning with Applications, vol. 11, p. 100 455, 2023

2023
[20]

Catastrophic interference in connectionist net- works: The sequential learning problem,

M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist net- works: The sequential learning problem,” in Psychology of learning and motiva- tion, vol. 24, Elsevier, 1989, pp. 109–165

1989
[21]

Learning to predict by the methods of temporal differences,

R. S. Sutton, “Learning to predict by the methods of temporal differences,” Ma- chine learning, vol. 3, pp. 9–44, 1988

1988
[22]

Analysis of temporal-diffference learning with func- tion approximation,

J. Tsitsiklis and B. Van Roy, “Analysis of temporal-diffference learning with func- tion approximation,” Advances in neural information processing systems , vol. 9, 1996

1996
[23]

Human-level control through deep reinforcement learning,

V . Mnih et al., “Human-level control through deep reinforcement learning,”nature, vol. 518, no. 7540, pp. 529–533, 2015

2015
[24]

Deep reinforcement learning and the deadly triad,

H. Van Hasselt, Y . Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil, “Deep reinforcement learning and the deadly triad,” arXiv preprint arXiv:1812.02648 , 2018. 164

2018 arXiv
[25]

Simple statistical gradient-following algorithms for connectionist reinforcement learning,

R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, pp. 229–256, 1992

1992
[26]

Asynchronous methods for deep reinforcement learning,

V . Mnih et al., “Asynchronous methods for deep reinforcement learning,” in Inter- national conference on machine learning, PMLR, 2016, pp. 1928–1937

2016
[27]

Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,

L. Espeholt et al., “Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,” in International conference on machine learning , PMLR, 2018, pp. 1407–1416

2018
[28]

Continuous control with deep reinforcement learning,

T. P. Lillicrap et al. , “Continuous control with deep reinforcement learning,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y . Bengio and Y . Le- Cun, Eds., 2016

2016
[29]

Distributed distributional deterministic policy gradients,

G. Barth-Maron et al., “Distributed distributional deterministic policy gradients,” in International Conference on Learning Representations, 2018

2018
[30]

Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation,

Y . Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba, “Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation,” Advances in neural information processing systems, vol. 30, 2017

2017
[31]

Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor,

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning, PMLR, 2018, pp. 1861–1870

2018
[32]

Trust region pol- icy optimization,

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region pol- icy optimization,” in International conference on machine learning, PMLR, 2015, pp. 1889–1897

2015
[33]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[34]

Dyna, an integrated architecture for learning, planning, and reacting,

R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” SIGART Bull., vol. 2, no. 4, pp. 160–163, 1991

1991
[35]

First return, then explore,

A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “First return, then explore,” Nature, vol. 590, no. 7847, pp. 580–586, 2021

2021
[36]

Learning latent dynamics for planning from pixels,

D. Hafner et al., “Learning latent dynamics for planning from pixels,” in Proceed- ings of the 36th International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97, 2019, pp. 2555–2565

2019
[37]

Curious model-building control systems,

J. Schmidhuber, “Curious model-building control systems,” in Proc. international joint conference on neural networks, 1991, pp. 1458–1463. 165

1991
[38]

On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models,

J. Schmidhuber, “On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models,” arXiv preprint arXiv:1511.09249, 2015

2015 arXiv
[39]

Recurrent world models facilitate policy evolution,

D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” Advances in neural information processing systems, vol. 31, 2018

2018
[40]

Dream to control: Learning behav- iors by latent imagination,

D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behav- iors by latent imagination,” in International Conference on Learning Representa- tions, 2019

2019
[41]

Mastering atari with discrete world models,

D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in International Conference on Learning Representations, 2021

2021
[42]

Mastering diverse domains through world models,

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104, 2023

2023 arXiv
[43]

On the proper- ties of neural machine translation: Encoder-decoder approaches,

K. Cho, B. Van Merri ¨enboer, D. Bahdanau, and Y . Bengio, “On the proper- ties of neural machine translation: Encoder-decoder approaches,” arXiv preprint arXiv:1409.1259, 2014

2014 arXiv
[44]

Convolutional networks for images, speech, and time series,

Y . LeCun, Y . Bengio, et al., “Convolutional networks for images, speech, and time series,”
[45]

Estimating or propagating gradi- ents through stochastic neurons for conditional computation,

Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagating gradi- ents through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013

2013 arXiv
[46]

Dm control: Software and tasks for continuous con- trol,

S. Tunyasuvunakool et al., “Dm control: Software and tasks for continuous con- trol,” Software Impacts, vol. 6, p. 100 022, 2020

2020
[47]

The arcade learning environment: An evaluation platform for general agents,

M. G. Bellemare, Y . Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial In- telligence Research, vol. 47, pp. 253–279, 2013

2013
[48]

Policy invariance under reward transforma- tions: Theory and application to reward shaping,

A. Y . Ng, D. Harada, and S. Russell, “Policy invariance under reward transforma- tions: Theory and application to reward shaping,” in Icml, vol. 99, 1999, pp. 278– 287

1999
[49]

Self-improving reactive agents based on reinforcement learning, plan- ning and teaching,

L.-J. Lin, “Self-improving reactive agents based on reinforcement learning, plan- ning and teaching,” Machine learning, vol. 8, pp. 293–321, 1992

1992
[50]

Sample efficient actor-critic with experience replay,

Z. Wang et al., “Sample efficient actor-critic with experience replay,” in Interna- tional Conference on Learning Representations, 2016. 166

2016
[51]

Rainbow: Combining improvements in deep reinforcement learn- ing,

M. Hessel et al., “Rainbow: Combining improvements in deep reinforcement learn- ing,” in Thirty-second AAAI conference on artificial intelligence, 2018

2018
[52]

A deeper look at experience replay,

S. Zhang and R. S. Sutton, “A deeper look at experience replay,” arXiv preprint arXiv:1712.01275, 2017

2017 arXiv
[53]

Revisiting fundamentals of experience replay,

W. Fedus et al., “Revisiting fundamentals of experience replay,” in Proceedings of the 37th International Conference on Machine Learning , H. D. III and A. Singh, Eds., ser. Proceedings of Machine Learning Research, vol. 119, PMLR, Jul. 2020, pp. 3061–3071

2020
[54]

Prioritized experience replay,

T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in International Conference on Learning Representations, 2016

2016
[55]

Prioritized experience replay method based on experience reward,

J. Gao, X. Li, W. Liu, and J. Zhao, “Prioritized experience replay method based on experience reward,” in 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), 2021, pp. 214–219

2021
[56]

Model-augmented prioritized experience replay,

Y . Oh, J. Shin, E. Yang, and S. J. Hwang, “Model-augmented prioritized experience replay,” in International Conference on Learning Representations, 2022

2022
[57]

Prioritized experience replay based on dynamics priority,

H. Li, X. Qian, and W. Song, “Prioritized experience replay based on dynamics priority,” Scientific Reports, vol. 14, no. 1, p. 6014, 2024

2024
[58]

Image augmentation is all you need: Reg- ularizing deep reinforcement learning from pixels,

D. Yarats, I. Kostrikov, and R. Fergus, “Image augmentation is all you need: Reg- ularizing deep reinforcement learning from pixels,” inInternational Conference on Learning Representations, 2021

2021
[59]

Temporal difference learning for model pre- dictive control,

N. A. Hansen, H. Su, and X. Wang, “Temporal difference learning for model pre- dictive control,” in International Conference on Machine Learning, PMLR, 2022, pp. 8387–8406

2022
[60]

Z. Zhu, K. Lin, and J. Zhou, Transfer learning in deep reinforcement learning: A survey, 2021. arXiv: 2009.07888 [cs.LG]

2021 arXiv
[61]

Distilling the knowledge in a neural net- work,

G. Hinton, O. Vinyals, J. Dean, et al., “Distilling the knowledge in a neural net- work,” arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015

2015 arXiv
[62]

Knowledge distillation: A survey,

J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

2021
[63]

Teaching on a budget: Agents advising agents in rein- forcement learning,

L. Torrey and M. Taylor, “Teaching on a budget: Agents advising agents in rein- forcement learning,” in Proceedings of the 2013 international conference on Au- tonomous agents and multi-agent systems, 2013, pp. 1053–1060. 167

2013
[64]

Online transfer learning in reinforcement learning do- mains,

Y . Zhan and M. E. Taylor, “Online transfer learning in reinforcement learning do- mains,” in 2015 AAAI Fall Symposium Series, 2015

2015
[65]

A survey on transfer learning,

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2009

2009
[66]

Transfer learning for reinforcement learning domains: A survey.,

M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey.,”Journal of Machine Learning Research, vol. 10, no. 7, 2009

2009
[67]

A comprehensive survey on test-time adaptation under distribution shifts,

J. Liang, R. He, and T. Tan, “A comprehensive survey on test-time adaptation under distribution shifts,” arXiv preprint arXiv:2303.15361, 2023

2023 arXiv
[68]

A review of novelty detection,

M. A. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko, “A review of novelty detection,” Signal Processing, vol. 99, pp. 215–249, 2014

2014
[69]

Towards a unifying framework for formal theories of novelty,

T. Boult et al. , “Towards a unifying framework for formal theories of novelty,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, 2021, pp. 15 047–15 052

2021
[70]

Open-world learning for radically autonomous agents,

P. Langley, “Open-world learning for radically autonomous agents,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 13 539–13 543

2020
[71]

Mixtbn: A fully test-time adaptation method for visual reinforce- ment learning on robotic manipulation,

Z. Liu and W. Li, “Mixtbn: A fully test-time adaptation method for visual reinforce- ment learning on robotic manipulation,” in2023 IEEE 5th International Conference on Civil Aviation Safety and Information Technology (ICCASIT) , 2023, pp. 228– 234

2023
[72]

Active test-time adaptation: Theoretical analyses and an algorithm,

S. Gui, X. Li, and S. Ji, “Active test-time adaptation: Theoretical analyses and an algorithm,” in The Twelfth International Conference on Learning Representations, 2024

2024
[73]

Unknown sample discovery for source free open set domain adaptation,

C. S. Jahan and A. Savakis, “Unknown sample discovery for source free open set domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Jun. 2024, pp. 1067–1076

2024
[74]

Hidden-mode markov decision processes for nonstationary sequential decision making,

S. P. Choi, D.-Y . Yeung, and N. L. Zhang, “Hidden-mode markov decision processes for nonstationary sequential decision making,” Sequence learning: paradigms, algorithms, and applications, pp. 264–287, 2001

2001
[75]

Choosing search heuristics by non-stationary reinforcement learn- ing,

A. Nareyek, “Choosing search heuristics by non-stationary reinforcement learn- ing,” in Metaheuristics: Computer Decision-Making . Boston, MA: Springer US, 2004, pp. 523–544, ISBN : 978-1-4757-4137-7

2004
[76]

Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach,

C.-Y . Wei and H. Luo, “Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach,” inProceedings of Thirty Fourth Con- 168 ference on Learning Theory , M. Belkin and S. Kpotufe, Eds., ser. Proceedings of Machine Learning Research, vol. 134, ...

2021
[77]

Near-optimal model- free reinforcement learning in non-stationary episodic mdps,

W. Mao, K. Zhang, R. Zhu, D. Simchi-Levi, and T. Basar, “Near-optimal model- free reinforcement learning in non-stationary episodic mdps,” in Proceedings of the 38th International Conference on Machine Learning , M. Meila and T. Zhang, Eds., ser. Proceedings of Machine Learnin...

2021
[78]

Non-stationary reinforcement learning under general function approximation,

S. Feng, M. Yin, R. Huang, Y .-X. Wang, J. Yang, and Y . Liang, “Non-stationary reinforcement learning under general function approximation,” in Proceedings of the 40th International Conference on Machine Learning , A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, an...

2023
[79]

Addressing environment non-stationarity by repeat- ing q-learning updates,

S. Abdallah and M. Kaisers, “Addressing environment non-stationarity by repeat- ing q-learning updates,” Journal of Machine Learning Research , vol. 17, no. 46, pp. 1–31, 2016

2016
[80]

Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning,

E. Lecarpentier and E. Rachelson, “Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning,” in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelz- imer, F. d’Alch ´e-Buc, E. Fox, and R. Ga...

2019
[81]

Reinforcement learning algorithm for non-stationary environments,

S. Padakandla, P. KJ, and S. Bhatnagar, “Reinforcement learning algorithm for non-stationary environments,”Applied Intelligence, vol. 50, no. 11, pp. 3590–3606, 2020

2020
[82]

Reactive exploration to cope with non-stationarity in life- long reinforcement learning,

C. A. Steinparz et al., “Reactive exploration to cope with non-stationarity in life- long reinforcement learning,” in Conference on Lifelong Learning Agents, PMLR, 2022, pp. 441–469

2022
[83]

Transfer in reinforcement learning: A framework and a survey,

A. Lazaric, “Transfer in reinforcement learning: A framework and a survey,” in Reinforcement Learning: State-of-the-Art , M. Wiering and M. van Otterlo, Eds. Springer Berlin Heidelberg, 2012, pp. 143–173

2012
[84]

Cross-modal domain adaptation for cost-efficient visual reinforcement learning,

X.-H. Chen, S. Jiang, F. Xu, Z. Zhang, and Y . Yu, “Cross-modal domain adaptation for cost-efficient visual reinforcement learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 12 520–12 532, 2021

2021
[85]

Deep reinforcement learning amidst lifelong non- stationarity,

A. Xie, J. Harrison, and C. Finn, “Deep reinforcement learning amidst lifelong non- stationarity,” in 4th Lifelong Machine Learning Workshop at ICML 2020, 2020. 169

2020
[86]

Model-based nov- elty adaptation for open-world ai,

M. Klenk, W. Piotrowski, R. Stern, S. Mohan, and J. de Kleer, “Model-based nov- elty adaptation for open-world ai,” in International Workshop on Principles of Di- agnosis (DX), 2020

2020
[87]

Detecting and adapting to novelty in games,

X. Peng, J. C. Balloch, and M. O. Riedl, “Detecting and adapting to novelty in games,” in AAAI Workshop on Reinforcement Learning in Games, 2021

2021
[88]

Spotter: Extending symbolic planning operators through targeted reinforcement learning,

V . Sarathy, D. Kasenberg, S. Goel, J. Sinapov, and M. Scheutz, “Spotter: Extending symbolic planning operators through targeted reinforcement learning,” inProceed- ings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems, 2021, pp. 1118–1126

2021
[89]

An integrated architecture for online adaptation to novelty in open worlds using probabilistic programming and novelty-aware planning,

B. Loyall et al., “An integrated architecture for online adaptation to novelty in open worlds using probabilistic programming and novelty-aware planning,” in In Pro- ceedings of AAAI Symposium, Designing Artificial Intelligence for Open Worlds , 2022

2022
[90]

Lifelong machine learning systems: Beyond learning algorithms,

D. L. Silver, Q. Yang, and L. Li, “Lifelong machine learning systems: Beyond learning algorithms,” in 2013 AAAI spring symposium series, 2013

2013
[91]

Online learning and online convex optimization,

S. Shalev-Shwartz et al., “Online learning and online convex optimization,” Foun- dations and Trends® in Machine Learning, vol. 4, no. 2, pp. 107–194, 2012

2012
[92]

Introduction to online convex optimization,

E. Hazan, “Introduction to online convex optimization,” Foundations and Trends® in Optimization, vol. 2, no. 3-4, pp. 157–325, 2016

2016
[93]

Measuring catas- trophic forgetting in neural networks,

R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, “Measuring catas- trophic forgetting in neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018

2018
[94]

Memory efficient experience replay for streaming learning,

T. L. Hayes, N. D. Cahill, and C. Kanan, “Memory efficient experience replay for streaming learning,” in2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 9769–9776

2019
[95]

Memory-efficient semi-supervised continual learning: The world is its own replay buffer,

J. Smith, J. Balloch, Y .-C. Hsu, and Z. Kira, “Memory-efficient semi-supervised continual learning: The world is its own replay buffer,” arXiv preprint arXiv:2101.09536, 2021

2021 arXiv
[96]

Reinforcement learning with gaussian pro- cesses,

Y . Engel, S. Mannor, and R. Meir, “Reinforcement learning with gaussian pro- cesses,” in Proceedings of the 22nd international conference on Machine learning, 2005, pp. 201–208

2005
[97]

Chevalier-Boisvert, L

M. Chevalier-Boisvert, L. Willems, and S. Pal, Minimalistic gridworld environment for openai gym, https://github.com/maximecb/gym-minigrid, 2018. 170

2018
[98]

A multi-agent simulator for generating novelty in monopoly,

M. Kejriwal and S. Thomas, “A multi-agent simulator for generating novelty in monopoly,” Simulation Modelling Practice and Theory, p. 102 364, 2021

2021
[99]

Novelty gen- eration framework for ai agents in angry birds style physics games,

C. Gamage, V . Pinto, C. Xue, M. Stephenson, P. Zhang, and J. Renz, “Novelty gen- eration framework for ai agents in angry birds style physics games,” inConference on Games, 2021

2021
[100]

Schmidhuber, A possibility for implementing curiosity and boredom in model- building neural controllers, 1991

J. Schmidhuber, A possibility for implementing curiosity and boredom in model- building neural controllers, 1991

1991
[101]

Intrinsically motivated reinforcement learn- ing,

N. Chentanez, A. Barto, and S. Singh, “Intrinsically motivated reinforcement learn- ing,” Advances in neural information processing systems, vol. 17, 2004

2004
[102]

A survey on transfer learning for multiagent re- inforcement learning systems,

F. L. Da Silva and A. H. R. Costa, “A survey on transfer learning for multiagent re- inforcement learning systems,” Journal of Artificial Intelligence Research, vol. 64, pp. 645–703, 2019

2019
[103]

Sim-to-real transfer in deep reinforce- ment learning for robotics: A survey,

W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforce- ment learning for robotics: A survey,” in 2020 IEEE symposium series on compu- tational intelligence (SSCI), IEEE, 2020, pp. 737–744

2020
[104]

Transfer learning in deep reinforcement learning: A survey,

Z. Zhu, K. Lin, A. K. Jain, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023

2023
[105]

Exploration in deep reinforcement learn- ing: A survey,

P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in deep reinforcement learn- ing: A survey,”Information Fusion, vol. 85, pp. 1–22, 2022

2022
[106]

Yang et al., Exploration in deep reinforcement learning: A comprehensive sur- vey, 2021

T. Yang et al., Exploration in deep reinforcement learning: A comprehensive sur- vey, 2021

2021
[107]

Successor features for transfer in reinforcement learning,

A. Barreto et al., “Successor features for transfer in reinforcement learning,” Ad- vances in neural information processing systems, vol. 30, 2017

2017
[108]

Transfer in reinforcement learn- ing via shared features,

G. Konidaris, I. Scheidwasser, and A. Barto, “Transfer in reinforcement learn- ing via shared features,” Journal of Machine Learning Research , vol. 13, no. 45, pp. 1333–1371, 2012

2012
[109]

Large- scale study of curiosity-driven learning,

Y . Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large- scale study of curiosity-driven learning,” in International Conference on Learning Representations, 2018

2018
[110]

Novgrid: A flexible grid world for evaluating agent response to novelty,

J. C. Balloch et al., “Novgrid: A flexible grid world for evaluating agent response to novelty,” in In Proceedings of AAAI Symposium, Designing Artificial Intelligence for Open Worlds, 2022. 171

2022
[111]

T. P. Lillicrap et al., Continuous control with deep reinforcement learning , 2019. arXiv: 1509.02971 [cs.LG]

2019 arXiv
[112]

Exploration by random net- work distillation,

Y . Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random net- work distillation,” inInternational Conference on Learning Representations, 2018

2018
[114]

Never give up: Learning directed exploration strategies,

A. P. Badia et al. , “Never give up: Learning directed exploration strategies,” in International Conference on Learning Representations, 2020

2020
[115]

Ride: Rewarding impact-driven exploration for procedurally-generated environments,

R. Raileanu and T. Rockt ¨aschel, “Ride: Rewarding impact-driven exploration for procedurally-generated environments,” in International Conference on Learning Representations, 2019

2019
[116]

R ´enyi state entropy maximization for ex- ploration acceleration in reinforcement learning,

M. Yuan, M.-O. Pun, and D. Wang, “R ´enyi state entropy maximization for ex- ploration acceleration in reinforcement learning,” IEEE Transactions on Artificial Intelligence, 2022

2022
[117]

Rewarding episodic visitation discrepancy for exploration in reinforcement learning,

M. Yuan, B. Li, X. Jin, and W. Zeng, “Rewarding episodic visitation discrepancy for exploration in reinforcement learning,” in Deep Reinforcement Learning Work- shop NeurIPS 2022, 2022

2022
[118]

Intrinsic reward driven imitation learning via gen- erative model,

X. Yu, Y . Lyu, and I. Tsang, “Intrinsic reward driven imitation learning via gen- erative model,” in Proceedings of the 37th International Conference on Machine Learning, H. D. III and A. Singh, Eds., ser. Proceedings of Machine Learning Re- search, vol. 119, PMLR, Jul. 2020,...

2020
[119]

Parameter space noise for exploration,

M. Plappert et al., “Parameter space noise for exploration,” in International Con- ference on Learning Representations, 2018

2018
[120]

Diversity is all you need: Learn- ing skills without a reward function,

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learn- ing skills without a reward function,” inInternational Conference on Learning Rep- resentations, 2019

2019
[121]

Stable-baselines3: Reliable reinforcement learning implementations,

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021

2021
[122]

Challenges of real-world reinforcement learning: Defi- nitions, benchmarks and analysis,

G. Dulac-Arnold et al., “Challenges of real-world reinforcement learning: Defi- nitions, benchmarks and analysis,” Machine Learning, vol. 110, no. 9, pp. 2419– 2468, Sep. 2021. 172

2021
[123]

Minigrid & miniworld: Modular & customiz- able reinforcement learning environments for goal-oriented tasks,

M. Chevalier-Boisvert et al. , “Minigrid & miniworld: Modular & customiz- able reinforcement learning environments for goal-oriented tasks,” CoRR, vol. abs/2306.13831, 2023

2023 arXiv
[124]

Deep reinforcement learning at the edge of the statistical precipice,

R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” in Advances in Neural Information Processing Systems , vol. 34, Curran Associates, Inc., 2021, pp. 29 304–29 320

2021
[125]

Objective mismatch in model- based reinforcement learning,

N. Lambert, B. Amos, O. Yadan, and R. Calandra, “Objective mismatch in model- based reinforcement learning,” in Proceedings of the 2nd Conference on Learning for Dynamics and Control, A. M. Bayen et al., Eds., ser. Proceedings of Machine Learning Research, vol. 120, PMLR, Jun....

2020
[126]

Plan- ning to explore via self-supervised world models,

R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak, “Plan- ning to explore via self-supervised world models,” in International Conference on Machine Learning, PMLR, 2020, pp. 8583–8592

2020
[127]

Discovering and achieving goals via world models,

R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak, “Discovering and achieving goals via world models,”Advances in Neural Information Processing Systems, vol. 34, pp. 24 379–24 391, 2021

2021
[128]

Deep hierarchical planning from pixels,

D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel, “Deep hierarchical planning from pixels,” Advances in Neural Information Processing Systems, vol. 35, pp. 26 091– 26 104, 2022

2022
[129]

Curious replay for model-based adap- tation,

I. Kauvar, C. Doyle, L. Zhou, and N. Haber, “Curious replay for model-based adap- tation,” in Proceedings of the 40th International Conference on Machine Learning, 2023, pp. 16 018–16 048

2023
[130]

Objective mismatch in model- based reinforcement learning,

N. Lambert, B. Amos, O. Yadan, and R. Calandra, “Objective mismatch in model- based reinforcement learning,” in Learning for Dynamics and Control , PMLR, 2020, pp. 761–770

2020
[131]

Mismatched no more: Joint model-policy optimization for model-based rl,

B. Eysenbach, A. Khazatsky, S. Levine, and R. R. Salakhutdinov, “Mismatched no more: Joint model-policy optimization for model-based rl,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Bel- grave, K. Cho, and A. Oh, Eds., vol. 35, ...

2022
[132]

Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems,

S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems,”arXiv preprint arXiv:2005.01643, 2020. 173

2005 arXiv
[133]

Meta reinforcement learning with latent variable gaussian processes,

S. Sæmundsson, K. Hofmann, and M. P. Deisenroth, “Meta reinforcement learning with latent variable gaussian processes,” inConference on Uncertainty in Artificial Intelligence, 2018

2018
[134]

Dynamic update-to-data ratio: Min- imizing world model overfitting,

N. Dorka, T. Welschehold, and W. Burgard, “Dynamic update-to-data ratio: Min- imizing world model overfitting,” in The Eleventh International Conference on Learning Representations, 2023

2023
[135]

Maxent dreamer: Maximum entropy rein- forcement learning with world model,

H. Ma, W. Xue, R. Ying, and P. Liu, “Maxent dreamer: Maximum entropy rein- forcement learning with world model,” in 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 2022, pp. 1–9

2022
[136]

The effectiveness of world models for continual reinforcement learning,

S. Kessler et al., “The effectiveness of world models for continual reinforcement learning,” in Conference on Lifelong Learning Agents, PMLR, 2023, pp. 184–204

2023
[137]

Dr jekyll, mr hyde: The strange case of off- policy policy updates,

R. Laroche and R. Tachet des Combes, “Dr jekyll, mr hyde: The strange case of off- policy policy updates,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, Curran Associates, Inc., 2021, ...

2021
[138]

Actor prioritized experience replay,

B. Saglam, F. B. Mutlu, D. C. Cicek, and S. S. Kozat, “Actor prioritized experience replay,” Journal of Artificial Intelligence Research, vol. 78, pp. 639–672, 2023

2023
[139]

Transformers are sample-efficient world models,

V . Micheli, E. Alonso, and F. Fleuret, “Transformers are sample-efficient world models,” in The Eleventh International Conference on Learning Representations , 2023

2023
[140]

Unifying count-based exploration and intrinsic motivation,

M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,” inProceedings of the 30th International Conference on Neural Information Processing Systems, 2016

2016
[141]

Curiosity-driven exploration by self-supervised prediction,

D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” inProceedings of the 34th International Conference on Machine Learning - Volume 70, ser. ICML’17, Sydney, NSW, Australia: JMLR.org, 2017, pp. 2778–2787

2017
[142]

An equivalence between loss functions and non-uniform sampling in experience replay,

S. Fujimoto, D. Meger, and D. Precup, “An equivalence between loss functions and non-uniform sampling in experience replay,” Advances in neural information processing systems, vol. 33, pp. 14 219–14 230, 2020

2020
[143]

Actor-critic algorithms,

V . Konda and J. Tsitsiklis, “Actor-critic algorithms,” in Advances in Neural Infor- mation Processing Systems, S. Solla, T. Leen, and K. M ¨uller, Eds., vol. 12, MIT Press, 1999. 174

1999
[144]

The 1998 ai planning systems competition,

D. McDermott, “The 1998 ai planning systems competition,” AI Mag., vol. 21, pp. 35–55, 2000

1998
[145]

Game engine learning from video,

M. Guzdial and M. O. Riedl, “Game engine learning from video,” in Proceedings of the 2017 International Conference on Artificial Intelligence, 2017

2017
[146]

H. S. M. Coxeter, Regular polytopes. Courier Corporation, 1973

1973
[147]

Hastie, R

T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman, The elements of sta- tistical learning: data mining, inference, and prediction. Springer, 2009, vol. 2

2009
[148]

An elementary introduction to modern convex geometry,

K. Ball et al., “An elementary introduction to modern convex geometry,” Flavors of geometry, vol. 31, no. 1-58, p. 26, 1997

1997
[149]

Representation edit distance as a measure of novelty,

J. Alspector, “Representation edit distance as a measure of novelty,” arXiv preprint arXiv:2111.02770, 2021

2021 arXiv
[150]

Willems, Lcswillems/torch-ac: Recurrent and multi-process pytorch implemen- tation of deep reinforcement actor-critic algorithms a2c and ppo, 2020

L. Willems, Lcswillems/torch-ac: Recurrent and multi-process pytorch implemen- tation of deep reinforcement actor-critic algorithms a2c and ppo, 2020

2020
[151]

The role of exploration for task transfer in reinforcement learning,

J. C. Balloch, J. Kim, J. L. Inman, and M. O. Riedl, “The role of exploration for task transfer in reinforcement learning,” 2022

2022
[152]

Neuro-symbolic world models for adapting to open world nov- elty,

J. Balloch et al., “Neuro-symbolic world models for adapting to open world nov- elty,”arXiv preprint arXiv:2301.18536, 2023

2023
[153]

Concept whitening for interpretable image recog- nition,

Z. Chen, Y . Bei, and C. Rudin, “Concept whitening for interpretable image recog- nition,” Nature Machine Intelligence, vol. 2, no. 12, pp. 772–782, 2020

2020
[154]

State2explanation: Concept-based explanations to benefit agent learning and user understanding,

D. Das, S. Chernova, and B. Kim, “State2explanation: Concept-based explanations to benefit agent learning and user understanding,” Advances in Neural Information Processing Systems, vol. 36, pp. 67 156–67 182, 2023

2023
[155]

Zoom in: An introduction to circuits,

C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter, “Zoom in: An introduction to circuits,”Distill, 2020, https://distill.pub/2020/circuits/zoom-in

2020
[156]

Challenging common assumptions in the unsupervised learning of disentangled representations,

F. Locatello et al., “Challenging common assumptions in the unsupervised learning of disentangled representations,” in international conference on machine learning, PMLR, 2019, pp. 4114–4124

2019
[157]

Measuring disentangle- ment: A review of metrics,

M.-A. Carbonneau, J. Zaidi, J. Boilard, and G. Gagnon, “Measuring disentangle- ment: A review of metrics,” IEEE transactions on neural networks and learning systems, 2022. 175

2022
[158]

The interpretability of codebooks in model-based reinforcement learning is limited,

K. Eaton, J. C. Balloch, J. Kim, and M. Riedl, “The interpretability of codebooks in model-based reinforcement learning is limited,” inI Can’t Believe It’s Not Better Workshop: Failure Modes of Sequential Decision-Making in Practice (RLC 2024), 2024

2024
[159]

Nanda, A comprehensive mechanistic interpretability explainer and glossary , Dec

N. Nanda, A comprehensive mechanistic interpretability explainer and glossary , Dec. 2022

2022
[160]

Toward transparent ai: A survey on interpreting the inner structures of deep neural networks,

T. R ¨auker, A. Ho, S. Casper, and D. Hadfield-Menell, “Toward transparent ai: A survey on interpreting the inner structures of deep neural networks,” in 2023 ieee conference on secure and trustworthy machine learning (satml) , IEEE, 2023, pp. 464–483

2023
[161]

Beta-V AE: Learning basic visual concepts with a constrained variational framework,

I. Higgins et al., “Beta-V AE: Learning basic visual concepts with a constrained variational framework,” in International Conference on Learning Representations, 2017

2017
[162]

Right for the right reasons: Train- ing differentiable models by constraining their explanations,

A. S. Ross, M. C. Hughes, and F. Doshi-Velez, “Right for the right reasons: Train- ing differentiable models by constraining their explanations,” in Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017, pp. 2662– 2670

2017
[163]

Concept bottleneck models,

P. W. Koh et al., “Concept bottleneck models,” in International conference on ma- chine learning, PMLR, 2020, pp. 5338–5348

2020
[164]

Neural discrete representation learning,

A. Van Den Oord, O. Vinyals, et al. , “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017

2017
[165]

Object-centric learning with slot attention,

F. Locatello et al., “Object-centric learning with slot attention,”Advances in neural information processing systems, vol. 33, pp. 11 525–11 538, 2020

2020
[166]

Concept embedding models: Beyond the accuracy- explainability trade-off,

M. Espinosa Zarlenga et al., “Concept embedding models: Beyond the accuracy- explainability trade-off,” Advances in Neural Information Processing Systems , vol. 35, pp. 21 400–21 413, 2022

2022
[167]

Stulp and O

F. Stulp and O. Sigaud, Paladyn, Journal of Behavioral Robotics , vol. 4, no. 1, pp. 49–61, 2013

2013
[168]

Orthogonal projec- tion loss,

K. Ranasinghe, M. Naseer, M. Hayat, S. Khan, and F. S. Khan, “Orthogonal projec- tion loss,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 12 333–12 343

2021
[169]

Reinforcement learning with augmented data,

M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement learning with augmented data,”Advances in neural information processing systems, vol. 33, pp. 19 884–19 895, 2020. 176

2020
[170]

Curl: Contrastive unsupervised representa- tions for reinforcement learning,

M. Laskin, A. Srinivas, and P. Abbeel, “Curl: Contrastive unsupervised representa- tions for reinforcement learning,” inInternational conference on machine learning, PMLR, 2020, pp. 5639–5650

2020
[171]

Decoupling representation learning from reinforcement learning,

A. Stooke, K. Lee, P. Abbeel, and M. Laskin, “Decoupling representation learning from reinforcement learning,” in Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang, Eds., ser. Proceedings of Machine Learning Research, vol. 139, PMLR, J...

2021
[172]

Fine-tuning reinforcement learning models is secretly a for- getting mitigation problem,

M. Wolczyk et al., “Fine-tuning reinforcement learning models is secretly a for- getting mitigation problem,” in Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov et al., Eds., ser. Proceedings of Machine Learning Research, vol. 235, PMLR, ...

2024
[173]

Pretraining representations for data-efficient reinforcement learning,

M. Schwarzer et al., “Pretraining representations for data-efficient reinforcement learning,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021

2021
[174]

Robosuite: A modular simulation framework and benchmark for robot learning,

Y . Zhu et al., “Robosuite: A modular simulation framework and benchmark for robot learning,” in arXiv preprint arXiv:2009.12293, 2020

2009 arXiv
[175]

Mujoco: A physics engine for model-based control,

E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2012, pp. 5026–5033

2012
[176]

Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,

B. Liu et al., “Libero: Benchmarking knowledge transfer for lifelong robot learn- ing,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[177]

A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforce- ment learning with offline datasets, 2021. arXiv: 2006.09359 [cs.LG]

2021 arXiv
[178]

Angioni, F

EclecticSheep, D. Angioni, F. Belotti, R. Can Malli, and M. Milesi, SheepRL, ver- sion 0.5.7, May 2023

2023
[179]

Reinforcement learning with action-free pre-training from videos,

Y . Seo, K. Lee, S. James, and P. Abbeel, “Reinforcement learning with action-free pre-training from videos,” inInternational Conference on Machine Learning, 2022

2022
[180]

Do concept bottleneck models obey locality?

N. Raman, M. E. Zarlenga, J. Heo, and M. Jamnik, “Do concept bottleneck models obey locality?” In XAI in Action: Past, Present, and Future Applications, 2023

2023
[181]

Training language models to follow instructions with human feedback,

L. Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730– 27 744, 2022. 177

2022
[182]

Open problems and fundamental limitations of reinforcement learning from human feedback,

S. Casper et al. , “Open problems and fundamental limitations of reinforcement learning from human feedback,” Transactions on Machine Learning Research , 2023

2023
[183]

How to train your robot with deep reinforcement learning: Lessons we have learned,

J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine, “How to train your robot with deep reinforcement learning: Lessons we have learned,”The Inter- national Journal of Robotics Research, vol. 40, no. 4-5, pp. 698–721, 2021. eprint: https://doi.org/10.1177/0278...

2021 doi
[184]

Policy information capacity: Information-theoretic measure for task complexity in deep reinforcement learning,

H. Furuta et al., “Policy information capacity: Information-theoretic measure for task complexity in deep reinforcement learning,” in Proceedings of the 38th Inter- national Conference on Machine Learning, M. Meila and T. Zhang, Eds., ser. Pro- ceedings of Machine Learning Res...

2021
[185]

Lipschitz lifelong reinforcement learning,

E. Lecarpentier, D. Abel, K. Asadi, Y . Jinnai, E. Rachelson, and M. L. Littman, “Lipschitz lifelong reinforcement learning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 8270–8278

2021
[186]

The child as elicitor of parental sanctions for independent and de- pendent behavior: A simulation of parent-child interaction.,

R. F. Marcus, “The child as elicitor of parental sanctions for independent and de- pendent behavior: A simulation of parent-child interaction.,” Developmental Psy- chology, vol. 11, no. 4, p. 443, 1975

1975
[187]

Multimodal behaviors from children elicit parent responses in real-time social interaction,

J. R. Yurkovic, D. P. Kennedy, and C. Yu, “Multimodal behaviors from children elicit parent responses in real-time social interaction,” inProceedings of the Annual Meeting of the Cognitive Science Society, vol. 43, 2021

2021
[188]

Deep active learning: Unified and prin- cipled method for query and training,

C. Shui, F. Zhou, C. Gagn ´e, and B. Wang, “Deep active learning: Unified and prin- cipled method for query and training,” in Proceedings of the Twenty Third Inter- national Conference on Artificial Intelligence and Statistics , S. Chiappa and R. Calandra, Eds., ser. Proceedin...

2020
[189]

Y . Chen, H. Luo, T. Ma, and C. Zhang, Active online learning with hidden shifting domains, 2021. arXiv: 2006.14481 [cs.LG]

2021 arXiv
[190]

A survey of deep active learning,

P. Ren et al., “A survey of deep active learning,”ACM computing surveys (CSUR), vol. 54, no. 9, pp. 1–40, 2021

2021
[191]

Quantifying interpretability and trust in machine learning systems,

P. Schmidt and F. Biessmann, “Quantifying interpretability and trust in machine learning systems,” arXiv preprint arXiv:1901.08558, 2019

1901 arXiv
[192]

Expanding explain- ability: Towards social transparency in ai systems,

U. Ehsan, Q. V . Liao, M. Muller, M. O. Riedl, and J. D. Weisz, “Expanding explain- ability: Towards social transparency in ai systems,” inProceedings of the 2021 CHI 178 Conference on Human Factors in Computing Systems , ser. CHI ’21, Yokohama, Japan: Association for Computin...

2021
[193]

Interpretable deep learning: Interpretation, interpretability, trust- worthiness, and beyond,

X. Li et al. , “Interpretable deep learning: Interpretation, interpretability, trust- worthiness, and beyond,” Knowledge and Information Systems , vol. 64, no. 12, pp. 3197–3234, 2022

2022
[194]

Charting the sociotechni- cal gap in explainable ai: A framework to address the gap in xai,

U. Ehsan, K. Saha, M. De Choudhury, and M. O. Riedl, “Charting the sociotechni- cal gap in explainable ai: A framework to address the gap in xai,” Proceedings of the ACM on human-computer interaction, vol. 7, no. CSCW1, pp. 1–32, 2023

2023
[195]

Continual lifelong learning with neural networks: A review,

G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural networks, vol. 113, pp. 54–71, 2019

2019
[196]

Towards continual reinforcement learning: A review and perspectives,

K. Khetarpal, M. Riemer, I. Rish, and D. Precup, “Towards continual reinforcement learning: A review and perspectives,” Journal of Artificial Intelligence Research , vol. 75, pp. 1401–1476, 2022

2022
[197]

A definition of continual reinforcement learning,

D. Abel, A. Barreto, B. Van Roy, D. Precup, H. P. van Hasselt, and S. Singh, “A definition of continual reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[198]

Attention is all you need,

A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon et al., Eds., vol. 30, Curran Associates, Inc., 2017

2017
[199]

Diffusion models in vi- sion: A survey,

F.-A. Croitoru, V . Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vi- sion: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 10 850–10 869, 2023

2023
[200]

Efficiently modeling long sequences with structured state spaces,

A. Gu, K. Goel, and C. Re, “Efficiently modeling long sequences with structured state spaces,” in International Conference on Learning Representations, 2022

2022
[201]

Dynamic routing between capsules,

S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” Ad- vances in neural information processing systems, vol. 30, 2017

2017
[202]

Addressing leakage in concept bot- tleneck models,

M. Havasi, S. Parbhoo, and F. Doshi-Velez, “Addressing leakage in concept bot- tleneck models,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 386–23 397, 2022

2022
[203]

Multi-time models for temporally abstract planning,

D. Precup and R. S. Sutton, “Multi-time models for temporally abstract planning,” Advances in neural information processing systems, vol. 10, 1997

1997
[204]

Temporally abstract par- tial models,

K. Khetarpal, Z. Ahmed, G. Comanici, and D. Precup, “Temporally abstract par- tial models,” in Advances in Neural Information Processing Systems, M. Ranzato, 179 A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, Curran Associates, Inc., 2021, pp. 1979–1991

2021
[205]

Partial models for building adap- tive model-based reinforcement learning agents,

S. Alver, A. Rahimi-Kalahroudi, and D. Precup, “Partial models for building adap- tive model-based reinforcement learning agents,” inProceedings of the Conference on Lifelong Learning Agents (CoLLAs) 2024, 2024

2024
[206]

Apprenticeship learning via inverse reinforcement learn- ing,

P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse reinforcement learn- ing,” in Proceedings of the twenty-first international conference on Machine learn- ing, 2004, p. 1

2004
[207]

Safe policy improvement with baseline bootstrapping,

R. Laroche, P. Trichelair, and R. T. Des Combes, “Safe policy improvement with baseline bootstrapping,” in International conference on machine learning, PMLR, 2019, pp. 3652–3661

2019
[208]

Conservative q-learning for offline reinforcement learning,

A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” inAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., 2020, pp. 1179–1191

2020
[209]

Provably efficient maximum entropy exploration,

E. Hazan, S. Kakade, K. Singh, and A. Van Soest, “Provably efficient maximum entropy exploration,” in International Conference on Machine Learning , PMLR, 2019, pp. 2681–2691

2019
[210]

Hindsight experience replay,

M. Andrychowicz et al., “Hindsight experience replay,” in Advances in Neural In- formation Processing Systems , I. Guyon et al., Eds., vol. 30, Curran Associates, Inc., 2017

2017
[211]

On warm-starting neural network training,

J. Ash and R. P. Adams, “On warm-starting neural network training,” Advances in neural information processing systems, vol. 33, pp. 3884–3894, 2020

2020
[212]

Critical learning periods in deep neural networks,

A. Achille, M. Rovere, and S. Soatto, “Critical learning periods in deep neural networks,” arXiv preprint arXiv:1711.08856, 2017

2017 arXiv
[213]

Dohare, J

S. Dohare, J. F. Hernandez-Garcia, P. Rahman, R. S. Sutton, and A. R. Mahmood, Loss of plasticity in deep continual learning, 2023. arXiv: 2306.13812 [cs.LG]

2023 arXiv
[214]

The pri- macy bias in deep reinforcement learning,

E. Nikishin, M. Schwarzer, P. D’Oro, P.-L. Bacon, and A. Courville, “The pri- macy bias in deep reinforcement learning,” inInternational conference on machine learning, PMLR, 2022, pp. 16 828–16 847

2022
[215]

Generalized data distribution iteration,

J. Fan and C. Xiao, “Generalized data distribution iteration,” in Proceedings of the 39th International Conference on Machine Learning , K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., ser. Proceedings of Machine Learning Research, vol. 162, PMLR...

2022
[216]

Continual backprop: Stochastic gradient descent with persistent randomness,

S. Dohare, R. S. Sutton, and A. R. Mahmood, “Continual backprop: Stochastic gradient descent with persistent randomness,” arXiv preprint arXiv:2108.06325 , 2021

2021 arXiv
[217]

Sample-efficient reinforcement learning by breaking the replay ratio barrier,

P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville, “Sample-efficient reinforcement learning by breaking the replay ratio barrier,” in The Eleventh International Conference on Learning Representations , 2023

2023
[218]

Why does unsupervised pre- training help deep learning?

D. Erhan, A. Courville, Y . Bengio, and P. Vincent, “Why does unsupervised pre- training help deep learning?” In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics , JMLR Workshop and Conference Proceedings, 2010, pp. 201–208

2010
[219]

The unsurprising ef- fectiveness of pre-trained vision models for control,

S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta, “The unsurprising ef- fectiveness of pre-trained vision models for control,” inProceedings of the 39th In- ternational Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Saba...

2022
[220]

Rrl: Resnet as representation for reinforcement learn- ing,

R. M. Shah and V . Kumar, “Rrl: Resnet as representation for reinforcement learn- ing,” in International Conference on Machine Learning , PMLR, 2021, pp. 9465– 9476

2021
[221]

Scalable deep reinforcement learning for vision-based robotic manipulation,

D. Kalashnikov et al. , “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Proceedings of The 2nd Conference on Robot Learning , 2018

2018
[222]

Open x-embodiment: Robotic learning datasets and rt-x models : Open x-embodiment collaboration0,

A. O’Neill et al., “Open x-embodiment: Robotic learning datasets and rt-x models : Open x-embodiment collaboration0,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 6892–6903

2024
[223]

Video pretraining (VPT): Learning to act by watching unlabeled online videos,

B. Baker et al., “Video pretraining (VPT): Learning to act by watching unlabeled online videos,” in Advances in Neural Information Processing Systems, 2022

2022
[224]

Behavior from the void: Unsupervised active pre-training,

H. Liu and P. Abbeel, “Behavior from the void: Unsupervised active pre-training,” in Advances in Neural Information Processing Systems , M. Ranzato, A. Beygelz- imer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, Curran Associates, Inc., 2021, pp. 18 459–18 473

2021
[225]

Off-policy deep reinforcement learning without exploration,

S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in Proceedings of the 36th International Conference on Ma- chine Learning, K. Chaudhuri and R. Salakhutdinov, Eds., ser. Proceedings of Ma- chine Learning Research, vol. 97, PML...

2019
[226]

He, A survey on offline model-based reinforcement learning, 2023

H. He, A survey on offline model-based reinforcement learning, 2023. arXiv: 2305. 03360 [cs.LG]

2023
[227]

MOReL: Model-based offline reinforcement learning,

R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “MOReL: Model-based offline reinforcement learning,” inAdvances in Neural Information Processing Sys- tems, 2020

2020
[228]

Revisiting design choices in offline model based reinforcement learning,

C. Lu, P. Ball, J. Parker-Holder, M. Osborne, and S. J. Roberts, “Revisiting design choices in offline model based reinforcement learning,” inInternational Conference on Learning Representations, 2022

2022
[229]

Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble,

S. Lee, Y . Seo, K. Lee, P. Abbeel, and J. Shin, “Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble,” in Proceedings of the 5th Conference on Robot Learning, 2022

2022
[230]

Moto: Offline pre-training to online fine-tuning for model-based robot learning,

R. Rafailov, K. B. Hatch, V . Kolev, J. D. Martin, M. Phielipp, and C. Finn, “Moto: Offline pre-training to online fine-tuning for model-based robot learning,” in Pro- ceedings of The 7th Conference on Robot Learning , J. Tan, M. Toussaint, and K. Darvish, Eds., ser. Proceedin...

2023
[231]

MOORe: Model-based offline-to- online reinforcement learning,

Y . Mao, C. Wang, B. Wang, and C. Zhang, “MOORe: Model-based offline-to- online reinforcement learning,” arXiv preprint arXiv:2201.10070, 2022

2022 arXiv
[232]

Offline–online actor–critic,

X. Wang, D. Hou, L. Huang, and Y . Cheng, “Offline–online actor–critic,” IEEE Transactions on Artificial Intelligence, vol. 5, no. 1, pp. 61–69, 2024

2024
[233]

Challenges and opportunities in offline reinforcement learning from visual observations,

C. Lu, P. J. Ball, T. G. J. Rudner, J. Parker-Holder, M. A. Osborne, and Y . W. Teh, “Challenges and opportunities in offline reinforcement learning from visual observations,” Transactions on Machine Learning Research, 2023

2023
[234]

Making offline rl online: Collaborative world models for offline visual reinforcement learning,

Q. Wang, J. Yang, Y . Wang, X. Jin, W. Zeng, and X. Yang, “Making offline rl online: Collaborative world models for offline visual reinforcement learning,” in Advances in Neural Information Processing Systems, 2024

2024
[235]

Toy models of superposition,

N. Elhage et al. , “Toy models of superposition,” Transformer Circuits Thread , 2022

2022
[236]

Explainable ai methods-a brief overview,

A. Holzinger, A. Saranti, C. Molnar, P. Biecek, and W. Samek, “Explainable ai methods-a brief overview,” inInternational workshop on extending explainable AI beyond deep models and classifiers, Springer, 2022, pp. 13–38

2022
[237]

Sanity checks for saliency maps,

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” Advances in neural information processing systems , vol. 31, 2018. 182

2018
[238]

Attention is not explanation,

S. Jain and B. C. Wallace, “Attention is not explanation,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019, pp. 3543–3556

2019
[239]

Attention is not not explanation,

S. Wiegreffe and Y . Pinter, “Attention is not not explanation,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019, pp. 11–20

2019
[240]

Extensions of lipshitz mapping into a hilbert space,

W. B. Johnson and J. Lindenstrauss, “Extensions of lipshitz mapping into a hilbert space,” in Conference modern analysis and probability, 1984, 1984, pp. 189–206

1984
[241]

The human imagination: The cognitive neuroscience of visual mental imagery,

J. Pearson, “The human imagination: The cognitive neuroscience of visual mental imagery,” Nature reviews neuroscience, vol. 20, no. 10, pp. 624–634, 2019

2019
[242]

Planning in the brain,

M. G. Mattar and M. Lengyel, “Planning in the brain,” Neuron, vol. 110, no. 6, pp. 914–934, 2022

2022
[243]

Biological underpinnings for lifelong learning machines,

D. Kudithipudi et al., “Biological underpinnings for lifelong learning machines,” Nature Machine Intelligence, vol. 4, no. 3, pp. 196–210, 2022

2022
[244]

Learning to dream, dreaming to learn,

N. R. P. Deperrois, “Learning to dream, dreaming to learn,” Ph.D. dissertation, Universit¨at Bern, 2024

2024
[245]

G. B. Arfken, H. J. Weber, and F. E. Harris, Mathematical methods for physicists: a comprehensive guide. Academic press, 2011

2011
[246]

From maximal entropy random walk to quantum thermodynamics,

J. Duda, “From maximal entropy random walk to quantum thermodynamics,” in Journal of Physics: Conference Series, IOP Publishing, vol. 361, 2012, p. 012 039

2012
[247]

Simple local models for complex dynamical systems,

E. Talvitie and S. Singh, “Simple local models for complex dynamical systems,” in Advances in Neural Information Processing Systems, D. Koller, D. Schuurmans, Y . Bengio, and L. Bottou, Eds., vol. 21, Curran Associates, Inc., 2008. 183 VITA Jonathan Clifford Balloch was born i...

2008

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Mastering the game of go without human knowledge,

D. Silver et al., “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017

2017

[2] [2]

Mastering atari, go, chess and shogi by planning with a learned model,

J. Schrittwieser et al., “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020

2020

[3] [3]

Grandmaster level in starcraft ii using multi-agent reinforcement learning,

O. Vinyals et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, no. 7782, pp. 350–354, 2019

2019

[4] [4]

Dota 2 with large scale deep reinforcement learning,

C. Berner et al. , “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680, 2019

arXiv 1912

[5] [5]

Agent57: Outperforming the human atari benchmark,

A. Badia et al., “Agent57: Outperforming the human atari benchmark,” inProceed- ings of the 37th International Conference on Machine Learning, Online, PMLR , vol. 119, 2020, p. 2020

2020

[6] [6]

Reinforcement learning based recommender systems: A survey,

M. M. Afsar, T. Crump, and B. Far, “Reinforcement learning based recommender systems: A survey,” ACM Computing Surveys, vol. 55, no. 7, pp. 1–38, 2022

2022

[7] [7]

Deepmind ai reduces google data centre cooling bill by 40%,

R. Evans and J. Gao, “Deepmind ai reduces google data centre cooling bill by 40%,” DeepMind blog, vol. 20, p. 158, 2016

2016

[8] [8]

Gnu-rl: A precocial reinforcement learning so- lution for building hvac control using a differentiable mpc policy,

B. Chen, Z. Cai, and M. Berg ´es, “Gnu-rl: A precocial reinforcement learning so- lution for building hvac control using a differentiable mpc policy,” in Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation, 2019, pp. 316–325

2019