Pith. sign in

REVIEW 4 major objections 5 minor 20 references

This paper claims that offline model-based RL agents underperform online ones because they lack a self-correction feedback loop: their policies exploit world-model mistakes, drift into out-of-distribution states at evaluation, and then get

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Offline-trained world-model agents in DreamerV3 underperform online agents due to out-of-distribution states at test time; adding about 10% self-generated data or exploratory data largely recovers performance.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A careful, large-scale empirical study on offline MBRL with a load-bearing but imperfect OOD proxy; deserves review. the 4 major comments →

arxiv 2509.05735 v1 pith:HRX7JTX5 submitted 2025-09-06 cs.LG cs.AI

Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies

Jiaqi Chen , Ji Shi , Cansu Sancaktar , Jonas Frey , Georg Martius This is my paper
classification cs.LG cs.AI
keywords model-based reinforcement learningoffline RLworld modelsout-of-distribution statesdata collectionexplorationself-correctionDreamerV3
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a practical question: when training a model-based RL agent, should the data come from active online interaction or from a fixed offline dataset? Its answer, tested across 31 environments, is that online agents win because the feedback loop of collecting data with the current policy keeps the world model accurate where that policy actually goes. Offline agents—whether sampling uniformly from an online agent's final buffer or replaying its exact training batches—lack this self-correction loop, so their policies exploit small world-model errors, drift into states the model never saw, and suffer compounding prediction failures at evaluation. The paper turns this diagnosis into data-collection advice: mix exploration data into offline buffers, and allow a small amount of adaptive online interaction, roughly 6 percent of the offline dataset, to restore online-level performance.

Core claim

The central claim is that performance degradation in offline model-based RL is not primarily a value-function extrapolation problem but a world-model/policy coupling problem. An online agent's self-correction mechanism—periodically retraining the world model on transitions collected by its own latest policy—keeps the model's prediction error low on the states its policy visits. Offline agents trained on the same replay data, whether sampling uniformly (Passive) or replaying the exact training batches (Tandem), develop policies that exploit inaccuracies in imagination, visit novel states during evaluation, and enter a catastrophic cycle where bad actions lead to more novel states. Decoupling

What carries the argument

The experimental machinery is the three-way comparison of Active, Passive, and Tandem agents built on a world-model RL agent. Active trains online; Passive samples uniformly from the final replay buffer of an Active agent; Tandem replays the Active agent's exact training batches with a different initialization. The self-correction mechanism is the online feedback loop that retrains the world model on the agent's own rollouts, keeping the model accurate on states the policy actually visits. The measurable workhorse is the world-model loss computed on evaluation rollouts, used as a simple out-of-distribution proxy; the adaptive agent divides evaluation loss by training-buffer loss to get an OO

Load-bearing premise

The load-bearing assumption is that the world-model loss measured on evaluation rollouts faithfully reflects how out-of-distribution the visited states are, since the causal diagnosis and the adaptive 1.35 OOD-ratio trigger both rest on that correlation.

What would settle it

Train an offline agent on a dataset that provably covers every reachable state, such as exhaustive random rollouts, and check whether it still underperforms an online agent; any degradation without out-of-distribution states would falsify the explanation—or, more cheaply, replace the OOD-ratio trigger with random interaction at matched totals and see whether performance holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Offline model-based agents trained solely on task-oriented data will systematically visit out-of-distribution states at evaluation, even when the data comes from a strong online agent.
  • Adding roughly 10 percent self-generated online data to an offline buffer restores most of the online agent's performance; an adaptive out-of-distribution-ratio trigger restores it with about 5.67 percent additional data on average.
  • Exploration data, especially when mixed with task reward, broadens state-space coverage and mitigates degradation, while pure expert data exacerbates the out-of-distribution problem.
  • Both world-model deviation and policy deviation contribute to degradation; sharing the online agent's world model reduces but does not eliminate the gap.
  • The world-model loss on evaluation rollouts can act as a pessimistic early-warning signal that additional online interaction is needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the OOD-ratio diagnosis transfers beyond the tested architecture, the same trigger could be used during deployment to decide when a robot or autonomous system should request new data instead of continuing to act on a stale model.
  • The result suggests that large offline datasets for real-world robotics should budget a portion for exploratory and behavioral diversity, and that expert-only data may actively harm world-model learning despite being intuitively the most useful.
  • A fully offline proxy for out-of-distribution risk, without requiring evaluation rollouts, would make the adaptive method deployable in safety-critical settings—the paper explicitly leaves this open.
  • The Tandem-with-shared-world-model result hints that an accurate world model trained online can transfer some self-correction benefit to a different policy, which could be tested as a way to make offline agents inherit robustness from an online teacher.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies offline vs. online data collection for model-based RL using DreamerV3 across 31 tasks from DMC, MetaWorld, and MinAtar. It compares an Active agent (online), a Passive agent (offline, uniform sampling from the Active agent's final replay buffer), and a Tandem agent (offline, replaying the Active agent's training batches in the same order). The main empirical claim is that Active agents outperform both offline variants, and the paper attributes this degradation to out-of-distribution (OOD) state visitation during evaluation, caused by the lack of a self-correction mechanism and a resultant mismatch between imagined and real rollouts. The paper then studies two remedies: training on exploration data (pure or mixed reward) and adding self-generated data on fixed or adaptive schedules. The adaptive schedule uses a world-model-loss OOD ratio with threshold 1.35, reporting an average of 5.67% self-generated data. The paper concludes with a recommendation to collect exploration data alongside expert data.

Significance. If the claims hold, this is a valuable large-scale empirical contribution. The controlled Active/Passive/Tandem comparison, the same-world-model ablations (Sec. 3.5.1), the per-step analysis (App. B.2), and the breadth of 31 tasks with three seeds provide a solid descriptive foundation. The core finding that offline MBRL agents underperform online counterparts on task-oriented data is convincingly supported. The practical guidance about exploration data is also potentially useful. However, the mechanistic explanation and the adaptive remedy rest on world-model loss as an OOD proxy, and the paper itself documents cases where this proxy fails. The causal and prescriptive parts therefore need strengthening before the full scope of the claims is justified.

major comments (4)
  1. [Secs. 3.2, 3.5.2, App. B.5.6] World-model loss is the load-bearing OOD measure for both the causal diagnosis (Secs. 3.3-3.4) and the adaptive interaction trigger (Sec. 4.2). The manuscript acknowledges that this loss is a 'pessimistic indicator' (Sec. 3.5.2) and documents concrete counterexamples: Freeway Passive_same shows low model loss despite strong degradation, and Finger Turn Hard - Vision Tandem shows low loss with poor performance (App. B.5.6). Because the loss conflates aleatoric and epistemic uncertainty, and can be low in OOD states due to episode resets or broad exploration coverage, the current evidence does not establish that OOD visitation is the causal driver of degradation. I request an independent OOD metric (e.g., replay-buffer state density/count, policy-input reconstruction, or ensemble disagreement) applied across tasks, or a reframing of the explanation as correlation-only and a held-out evalua
  2. [Sec. 3.5.3, Fig. 5, App. B.3] The 'expert data' experiment treats the second half of the Active agent's replay buffer as expert data. This is not an expert demonstration dataset: the Active agent is still learning, and the second half is a policy-dependent mixture of learning trajectories, including potentially suboptimal or redundant transitions. Consequently, the broad conclusions that 'expert demonstrations alone are insufficient in MBRL' (Abstract, Sec. 6) and the recommendation to add exploration data to expert datasets go beyond the evidence. Please either use a proper expert dataset (e.g., from a fully trained policy or from human demonstrations) or restrict the wording to 'later-training, high-reward replay data' throughout the paper.
  3. [Sec. 4.2, App. A.7, App. B.5.1] The headline efficiency figure of 5.67% self-generated data is an in-sample result. The OOD-ratio threshold 1.35 is selected via ablation on the same benchmark tasks (App. A.7), and the exploration-bonus weight wexpl is tuned per task (App. B.5.1). Thus the adaptive schedule is partly a fitted procedure, and the claimed minimal interaction requirement is not validated on held-out tasks or with a threshold chosen without task-level tuning. Please report this explicitly and, if possible, validate on left-out environments or pre-registered thresholds; otherwise the prescriptive claim should be softened.
  4. [Secs. 3.3, 3.4, App. B.2] The causal direction 'OOD visitation causes degradation' is not fully established. The stepwise analysis (App. B.2) covers only two tasks and does not report seed-averaged statistics, and the Point Mass Maze evidence shows the policy deviating before entering novel states, which is consistent with OOD being a symptom of an already compromised policy. To support the causal claim, the authors would need an intervention that holds policy behavior fixed while varying whether states are familiar, or a systematic multi-seed comparison. Otherwise the mechanism should be presented as a plausible hypothesis consistent with the data, rather than as an established causal chain.
minor comments (5)
  1. [Sec. 4.2 / Abstract] The Abstract says '6 %' while Sec. 4.2 reports 5.67% average self-generated data; unify the numbers.
  2. [Alg. 3, Sec. 4.2] In the fixed-schedule condition i%N == 0 with N=4K, 20K, 200K, clarify whether i is an environment step or a training update; the stated mapping to 50%, 10%, and 1% self-generated data depends on this distinction.
  3. [App. B.3, Fig. S6] The Tandem weight-mixing experiment is reported for one seed with no error bars. It is useful as an illustrative sensitivity check, but please label it as such in the main text or appendix.
  4. [Sec. 3.2 / App. A.8] The world-model loss is computed from a FIFO buffer of up to 500 steps per episode. This truncation could affect comparability with MinAtar episodes that reset early on failure; a one-sentence caveat in Sec. 3.2 would help.
  5. [Sec. 4.1 / Eq. (5)] For pure exploration, specify how Eq. (5) reduces to the exploration-only case (presumably wtask=0, wexpl=1.0) to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical findings, not derivations, and the cited OOD proxy and tuned threshold are transparently presented rather than disguised as predictions.

full rationale

The paper's main claims—that offline model-based agents underperform online agents, that degradation coincides with visiting novel states, and that additional interaction or exploration data mitigates it—are empirical results obtained by running DreamerV3 across 31 tasks. There is no derivation chain in which an output quantity is constructed from the same quantity it is supposed to predict. The world-model loss is used as an OOD proxy, but the paper explicitly calls it a 'pessimistic indicator' (Sec. 3.5.2) and documents cases where loss and performance diverge (Appendix B.5.6); this is a measurement-validity limitation, not circular reasoning. The adaptive-interaction threshold of 1.35 is selected via an ablation (Appendix A.7) and the resulting 5.67% self-generated data is reported as a measured evaluation result, not as a parameter-free prediction; choosing a hyperparameter on the test tasks is a form of tuning and limits generalization claims, but it does not make the conclusion equivalent to its inputs by construction. The methods rely on external prior work (Ostrovski et al., 2021; Hafner et al., 2023; Sekar et al., 2020) rather than on load-bearing self-citations, and no uniqueness theorem or ansatz is smuggled in from the authors' own prior work. The paper's self-correction explanation is a causal interpretation of controlled experiments (Tandem/Passive/sameWM variants), not a definitional tautology. Accordingly, no circular step meeting the quoted-evidence standard is present.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central empirical claims do not depend on new free parameters beyond standard RL training. The two constants above are introduced by the remedies and are fitted to the evaluation set. No new physical or algorithmic entities are postulated.

free parameters (2)
  • wexpl (exploration bonus weight) = varies by task: 0.5 for Quadruped Run, 0.9 for Point Mass Maze and Pick-Place
    In Eq. (5), task reward and exploration reward are mixed with weight wexpl; the optimal value is task-dependent (Appendix B.5.1), so the remedy includes a tuned hyperparameter.
  • OOD ratio threshold for adaptive interaction = 1.35
    Selected via ablation over {2.0, 1.65, 1.35, 1.2} in three tasks (Appendix A.7); the paper notes low sensitivity, but the value is fitted to the benchmark set.
axioms (3)
  • domain assumption DreamerV3 is a representative model-based RL algorithm and conclusions transfer to other MBRL methods
    The paper's recommendations for data collection presume generality beyond DreamerV3; this is not validated.
  • ad hoc to paper World model loss is a valid OOD measure
    Sec. 3.2 uses it as an OOD measure, Sec. 3.5.2 admits it is a pessimistic indicator; the adaptive schedule relies on it.
  • ad hoc to paper The second half of the Active agent's replay buffer approximates an expert dataset
    Sec. 3.5.3 and Appendix B.3 treat the latter half of the buffer as expert data; this is a proxy and may differ from true expert demonstrations.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies." pith.science (2026). https://pith.science/paper/HRX7JTX5

@misc{pith2026250905735,
  author       = {Pith},
  title        = {Pith review of: Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRX7JTX5}},
  note         = {Machine review of arXiv:2509.05735}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Data collection is crucial for learning robust world models in model-based reinforcement learning. The most prevalent strategies are to actively collect trajectories by interacting with the environment during online training or training on offline datasets. At first glance, the nature of learning task-agnostic environment dynamics makes world models a good candidate for effective offline training. However, the effects of online vs. offline data on world models and thus on the resulting task performance have not been thoroughly studied in the literature. In this work, we investigate both paradigms in model-based settings, conducting experiments on 31 different environments. First, we showcase that online agents outperform their offline counterparts. We identify a key challenge behind performance degradation of offline agents: encountering Out-Of-Distribution states at test time. This issue arises because, without the self-correction mechanism in online agents, offline datasets with limited state space coverage induce a mismatch between the agent's imagination and real rollouts, compromising policy training. We demonstrate that this issue can be mitigated by allowing for additional online interactions in a fixed or adaptive schedule, restoring the performance of online training with limited interaction data. We also showcase that incorporating exploration data helps mitigate the performance degradation of offline agents. Based on our insights, we recommend adding exploration data when collecting large datasets, as current efforts predominantly focus on expert data alone.

Figures

Figures reproduced from arXiv: 2509.05735 by Cansu Sancaktar, Georg Martius, Jiaqi Chen, Ji Shi, Jonas Frey.

Figure 1
Figure 1. Figure 1: Investigation of the performance degradation in offline agents and potential remedies. a) Illustra￾tion of Active, Passive, and Tandem agents. The Active agent is trained using online RL and is allowed to interact with the environment. The Passive agent is trained from the full buffer of an Active agent, without performing any additional interactions. The Tandem agent, is also trained offline, but samples … view at source ↗
Figure 2
Figure 2. Figure 2: Example of the degraded performance during offline training in 2D point mass maze environment. The task is to move the yellow point mass from the top-left initial position to the red marker in the bottom-right of the maze, which is the goal position. The episode score of each agent is shown in (a). In (b-d), we show the point mass trajectory generated by the final model after 1M environment steps. The two … view at source ↗
Figure 3
Figure 3. Figure 3: Episode score and world model loss during evaluation rollouts of 4 selected tasks. The first two are from DMC and the last two are from the Metaworld domain. The performance degradation of offline agents, including Passive and Tandem, is common across domains and tasks, especially for Tandem agents. and/or model initializations (Passive and Tandem) lead to distinct policies during training. To effectively … view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison when keeping an equivalent world model in Passive or Tandem agents to the one of the Active agent throughout training. Despite utilizing the same world model during training, performance degradation still occurs, albeit to varying degrees. formance in Passive and Tandem agents is accompanied by a significantly larger world model loss on evaluation episodes than the Active agent. Give… view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of Active, Passive as well as Passive agents trained on expert, suboptimal, and mixed data, which is implemented by splitting the replay buffer of the Active agent in different ways. In particular, the latter factors can lead to high model loss without significant impacts on performance, depending on whether exact predictions are required for the task at hand. In addition, even when … view at source ↗
Figure 6
Figure 6. Figure 6: State visitation in the Point Mass Maze task. They are calculated using the discretized states from three different Active agents’ final replay buffers after 1M environment steps. (b) Agent in a pure task-oriented setting. (c) Agent with a mixed reward: task plus exploration rewards, see Eq. (5) with wexpl = 0.5. (d) Agent with pure exploration rewards based on ensemble disagreement (Sekar et al., 2020). T… view at source ↗
Figure 7
Figure 7. Figure 7: Performance comparison when training on pure exploration data. The dataset is generated by the Active-expl. agent with a behavioral policy based on ensemble disagreement (Sekar et al., 2020). We additionally show the baseline performance of a task-oriented Active agent. Active Passive Tandem Expl. bonus 0.0 Expl. bonus 0.1 Expl. bonus 0.5 Expl. bonus 0.9 Expl. bonus 1.0 [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 8
Figure 8. Figure 8: Training on pure exploration data is not optimal. Performance comparison when assigning different exploration bonuses wexpl in the reward function. The black dashed lines represent pure task-oriented policy without any exploration bonus. Specifically, for every N environment steps, the Passive agent is allowed to collect 2K-step transitions based on its learned policy. Then the interactive data will be add… view at source ↗
Figure 9
Figure 9. Figure 9: Performance comparison when allowing adding additional self-generated data for Passive agents. The Passive+Auto interact agent adds 6.5% self-generated data in Cheetah Run - Vision, 2.9% in Quadruped Run - Vision, 9.8% in Pick-Place - Proprio, and 0.5% in Spaceinvaders. The percentage is calculated w.r.t. to the size of the final replay buffer of Active agents. 5 Related Work Performance Degradation in Off… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages · 4 internal anchors

  1. [1]

    Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michael L. Littman. Combating the compounding-error problem with a multi-step model. CoRR, abs/1905.13320,

    Pith/arXiv arXiv 1905

  2. [3]

    Below, the training of the world model M includes training all components in Eq

    for better clarity. Below, the training of the world model M includes training all components in Eq. (2), while training π includes all components in Eq. (3). Algorithm 1 Learning agents Active Agent 1: Initialize: Replay buffer B = a few random episodes. 2: World model M + Policy π by seed SA. 3: for each step i do 4: Sample Di A ∼ B 5: Update M using Di...

    work page 2025

  3. [4]

    Regularized behavior value estimation

    Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. CoRR, abs/2103.09575,

    Pith/arXiv arXiv

  4. [7]

    A Survey on Offline Model-Based Reinforcement Learning

    Haoyang He. A survey on offline model-based reinforcement learning. CoRR, abs/2305.03360,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [10]

    Discor: Corrective feedback in reinforcement learning via distribution correction

    Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems , 33: 18560–18572, 2020a. Reinforcement Learning Journal 2025 Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement le...

    work page 2025

  6. [11]

    Andrew Bagnell

    Stéphane Ross and J. Andrew Bagnell. Agnostic system identification for model-based reinforce- ment learning. In Proceedings of the 29th International Conference on Machine Learning , pp. 1905–1912,

    work page 1905

  7. [12]

    The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

    Anya Sims, Cong Lu, and Yee Whye Teh. The edge-of-reach problem in offline model-based reinforcement learning. CoRR, abs/2402.12527,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [13]

    Understanding the performance gap between online and offline alignment algorithms

    Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. CoRR, abs/2405.08448,

    Pith/arXiv arXiv

  9. [14]

    Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning

    Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning. CoRR, abs/2201.13425,

    Pith/arXiv arXiv

  10. [15]

    Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments

    Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176,

    Pith/arXiv arXiv 1903

  11. [17]

    A Implementation Details A.1 Runtime Overview Our experiments comprised approximately 2000 runs, totaling 20000 GPU hours

    Reinforcement Learning Journal 2025 Supplementary Materials The following content was not necessarily subject to peer review. A Implementation Details A.1 Runtime Overview Our experiments comprised approximately 2000 runs, totaling 20000 GPU hours. Each run took between 8 and 15 hours, depending on the specific task. All experiments were conducted using N...

    work page 2025

  12. [19]

    -same”, while the different model initialization is marked with “-diff

    This is because Passive agents have access to high-quality trajectories from the beginning, while Active agents must wait until later in training to encounter those trajectories. We validate this hypothesis in Fig. S7, where Passive agents trained on suboptimal data generally perform worse than those trained on mixed data. It indicates that mixing expert ...

    work page 2022

  13. [20]

    Motivation: addressing performance degradation in model-based offline training Our work is motivated by the performance degradation often observed when training model-based agents offline. While we draw inspiration from the experimental setup of the original Tandem RL paper (Ostrovski et al., 2021), their focus is on model-free methods (specifically DQN),...

    work page 2021

  14. [2018]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104,

    Pith/arXiv arXiv

  15. [2019]

    Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft

    Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. CoRR, abs/2106.14876,

    Pith/arXiv arXiv

  16. [2020]

    Exploring generalization and adaptability of offline reinforcement learning for robot manipulation

    Hyun Kim, Injun Park, Ingook Jang, Seonghyun Kim, Samyeul Noh, and Joonmyon Cho. Exploring generalization and adaptability of offline reinforcement learning for robot manipulation. In 2023 23rd International Conference on Control, Automation and Systems (ICCAS) , pp. 1542–1547,

    work page 2023

  17. [2021]

    Boosting offline reinforcement learning via data rebalancing

    Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. CoRR, abs/2210.09241,

    Pith/arXiv arXiv

  18. [2022]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551,

    Pith/arXiv arXiv

  19. [2023]

    Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learning

    Patrick Emedom-Nnamdi, Abram L. Friesen, Bobak Shahriari, Nando de Freitas, and Matt W. Hoffman. Knowledge transfer from teachers to learners in growing-batch reinforcement learning. CoRR, abs/2305.03870,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [2024]

    Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL

    Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl.CoRR, abs/2106.09119,

    work page internal anchor Pith review Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.