Pith. sign in

REVIEW 6 major objections 6 minor 50 references

Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search

T0 review · 6 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a diversity term to safe model-based policy search makes iterative batch RL improve every round, not just repeat old behavior.

desk verdict A modest but legitimate extension combining behavior-regularized safety with MinLSED diversity in iterative batch RL; the empirical pattern is consistent, but the evaluation needs more baselines and cleanup. read the letter →

arxiv 2411.09722 v1 pith:J5TSXK7T submitted 2024-11-14 cs.LG cs.AIstat.ML

Amna Najib , Stefan Depeweg , Phillip Swazinna This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords iterativebatchreinforcementlearningofflinemodel-basedpolicysearchdiversitysafetyconstraintsindustrialcontrolMinLSEDbehaviorregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when offline reinforcement learning is applied repeatedly to an industrial system — train, deploy, collect new data, retrain — the loop can be made to improve every time, not just repeat the old behavior. The proposed method, iterative batch RL, adds two ingredients to an ensemble model-based policy search: a safety constraint that keeps actions within the support of the behavior data, and a diversity term that forces the ensemble's virtual trajectories apart. On a 2D navigation task and the Industrial Benchmark, the diversified safe policies lower cost faster over iterations and with less variance than the same method without diversity. If true, this gives a practical recipe for upgrading industrial controllers from their own deployment logs without risky exploration.

What carries the argument

The method combines three components: (1) an ensemble model-based policy search that learns a transition model and a reward model from all collected data and optimizes K policies through virtual rollouts; (2) a safety mechanism, instantiated either as a likelihood-based soft constraint that penalizes actions improbable under a learned Gaussian behavior policy, or as a policy architecture that directly bounds actions to keep states within known safety ranges; (3) a diversity objective, MinLSED, defined as the minimum over pairs of the average Euclidean distance between their virtual trajectories. The MinLSED term is designed to avoid the outlier problem of plain averaged lock-step distance, and one policy is excluded from the diversity term so that it remains purely reward- and safety-driven. The combined loss is trained end-to-end via automatic differentiation.

What would settle it

In an Industrial Benchmark run, record the actual state trajectories of the deployed diversified ensemble and compare their coverage to the non-diversified ensemble; if the diversified set does not cover a meaningfully larger or more distinct region of the (velocity, gain, shift) space, or if the improvement disappears when the learned model is made deliberately wrong in the regions the diversity term pushes toward, the central mechanism is refuted.

Watch

Extended reading notes

Core claim

The central claim is that incorporating diversity into safe model-based policy search yields targeted policy improvements at each iteration of an offline RL loop, while safety constraints prevent performance from dropping below the behavior policy. The paper introduces iterative batch reinforcement learning (IBRL), where each deployment produces a new batch appended to the dataset, and shows that an ensemble of policies trained with a MinLSED diversity term explores more informative regions of the state space. In experiments on the Industrial Benchmark, the diversified variant reduces cost from 216.5 to 182.7 over four iterations under a constrained policy, compared to 188.9 without diversity, and shows lower variance. The authors present this as evidence that diversity, not just iteration, drives improvement.

Load-bearing premise

The argument depends on the assumption that trajectories that are far apart inside the learned transition model are also far apart on the real system, so that simulated diversity translates into useful real-world data collection.

Editorial extensions

If this is right

  • A controller trained on a single static batch can be improved over successive deployments without ever falling below the behavior policy's cost.
  • The diversity term makes the learned ensemble visit more of the state space, which in turn gives the next iteration's transition model better data and reduces model bias.
  • Safety as a soft constraint does not have to be traded off against performance or diversity; the flat-in-the-safe-region loss keeps the policy within the support of the data.
  • The MinLSED choice prevents one outlier policy from dominating the diversity objective, so the ensemble remains genuinely behaviorally varied.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: the diversity gain should be largest when the learned model's uncertainty is spatially structured, and in settings where the model is uniformly accurate diversity may add little beyond safety.
  • The paper's observation that the soft-constraint safety metric implicitly encourages diversity suggests that some safety mechanisms may already provide exploratory drive, making an explicit diversity term redundant for those formulations.
  • The practice of excluding one policy from the diversity objective, keeping it purely reward-driven, is a pragmatic device; an interesting extension would be to let the role rotate or to tie the diversity weight to the estimated model error in the regions being explored.
  • Connecting to the growing-batch literature, the method suggests that the value of new data depends on its coverage difference from old data, so an information-theoretic data-collection criterion might be derived from the MinLSED objective.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes an iterative batch reinforcement learning (IBRL) framework in which, after each deployment, newly collected data is appended to the existing batch and the transition model and policy ensemble are retrained. The method augments model-based policy search with two components: safety mechanisms (safety-as-objective, safety as a soft constraint, and a directly constrained policy) and a diversity bonus based on the minimum pairwise lock-step Euclidean distance (MinLSED) between trajectories generated in the learned model. Experiments on a 2D grid environment and the Industrial Benchmark compare a diversity weight of α_d = 0.15 against α_d = 0.0, reporting lower costs, lower variance across repetitions, and, in the constrained-policy experiment, a smaller gap between predicted and true costs. The conclusion states that incorporating diversity provides targeted improvements in each iteration while safety prevents performance from dropping below the behavior policy.

Significance. The problem addressed is practically relevant: in industrial control, offline RL is often applied repeatedly as new deployment data becomes available, and an explicit diversity signal could improve data collection. The paper is among the first to combine safety and diversity in this iterative setting, and it has the virtue of reporting true deployment costs (straight lines in Figures 5 and 6) rather than only model-predicted costs. The internal comparison between α_d = 0 and α_d = 0.15 is consistent across two safety mechanisms. However, the evidence is thin: the method is compared only against itself, no alternative exploration incentive is tested, the diversity term is never validated against true-environment trajectory diversity, and several key hyperparameters and experimental details are missing. If the central claim is correct, the approach could be a useful heuristic for industrial deployment, but the manuscript as it stands does not establish that claim beyond an internal ablation.

major comments (6)
  1. [§3.2, Eqs. (8)-(9)] The diversity objective is not well-defined and is not reproducible. In Eq. (8), the sum runs over all ordered pairs (k, k') with k' ≠ k, which gives K(K-1) terms; dividing by K! is not the correct normalization for K > 2. In Eq. (9), the notation 'min_{k'≠k, k∈K}' does not specify whether the minimum is over ordered pairs or unordered pairs, and the factor 1/H is redundant because D(T_i, T_j) already averages over the horizon. Since MinLSED is the central new component of the method, the exact objective being optimized must be stated unambiguously.
  2. [§3.3, Eq. (10) and Algorithm 1] The description of the loss is internally inconsistent. The text states that 'by default we reduce the scope of the diversity term to exclude one (the first) policy,' but Eq. (10) as written applies the diversity term to all k and all k' ≠ k with no exclusion. Additionally, the reward term has a leading coefficient -1/(NKH) while the safety and diversity terms have 1/(KH), making the relative weighting of the reward term ambiguous. Equation (10) also contains a typo ('Lθ)'). The exact loss minimized in the experiments must be specified precisely.
  3. [§4.2, Table 1 and experimental setup] The number of repetitions is inconsistent: Section 4.2 states 'We repeat each experiment three times and report average results,' but Table 1's caption says 'standard error over 6 repetitions.' No seed values or a statement of how many random seeds were used are provided. Because the central empirical claim is a difference in mean cost and variance between α_d = 0 and α_d = 0.15, the reader cannot assess statistical significance without knowing the exact number of runs and the seeds.
  4. [§4.2, Eq. (10) and Experiments 1-2] Two of the three loss hyperparameters in Eq. (10), namely α_s and δ, are never reported. Only α_d = 0.15 is stated. Since the paper's theme is the interaction between safety and diversity, omitting α_s and δ makes the experiments irreproducible and prevents the reader from evaluating whether the safety constraint is active in the reported runs.
  5. [§4.2 and §5] The proposed mechanism is that maximizing MinLSED diversity in the learned model leads to more informative real-world data collection, which in turn improves the model and policy. However, the paper never measures diversity of the deployed policies in the true environment. The predicted-versus-true cost plots (Figures 5 and 6) are suggestive but do not isolate the diversity mechanism. To support the causal claim, the authors should either report true-environment diversity of the deployed ensemble or compare against an equally strong non-diversity exploration incentive, such as matched action noise or an uncertainty-based exploration bonus.
  6. [§4.2, all experiments] The experiments contain no external baselines. The only comparison is between α_d = 0 and α_d = 0.15 within the proposed method, and the improvement over iterations for α_d = 0 shows that data accumulation alone already reduces cost. Without comparing against, for example, repeated deployment of the behavior policy or a standard offline RL algorithm applied to the growing batch, the paper cannot support the broader claim that the proposed safe diversified IBRL procedure is a viable way to improve industrial controllers.
minor comments (6)
  1. [Eq. (7)] Equation (7) is self-referential: π_constr appears on both sides of the definition. Presumably the right-hand side should contain the unconstrained policy π(s_t; θ_k). Please correct the equation and clarify how the bounds B1, B2 are mapped to action bounds when only a subset of state variables is affected.
  2. [§3.2] The line 'Ld(η, ω, θ) = −D(T1, .., TK) = −D(θ, ω, η)' is confusing because D is not defined as a function of η, ω, and θ. Please define the diversity loss as an explicit function of the parameters and the model.
  3. [Algorithm 1] The algorithm step 'Train π(. ; θ1) . . . , π(. ; θk) on Eq. (10)' should state that the model parameters η and ω are held fixed while optimizing the policy parameters, and it should specify whether the safety and diversity terms are computed on the fly using the current policy or using a detached copy.
  4. [Figure 3] The panel labels read 'Policymap - = 0.0' and 'Policymap - = 0.4'; the 'λ' symbol is missing. Please correct the labels.
  5. [Appendix A.2] There is a typo: 'Wwe use a simple policy' should be 'We use a simple policy.'
  6. [§4.2] The data collection procedure is underspecified: the paper states that the initial random bounded batch is collected via five rollouts of horizon 200, but does not state how many rollouts are executed per policy in each new batch, whether starting states are sampled uniformly, or how the fifteen past observations are exactly concatenated into the state representation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the diversity-driven improvement is an empirical result measured on the external benchmark's true cost, not a quantity forced by the training objective; the self-citations are contextual and the self-referential Eq. (7) is a typographical slip.

full rationale

The paper derives no analytic prediction; its central claim is that MinLSED diversity (Eq. 9), optimized over virtual rollouts of the learned transition model f (Algorithm 1), accelerates cost reduction across batch iterations (Table 1). That claim is evaluated on the externally runnable Industrial Benchmark simulator and a 2D grid world, reporting true deployment costs in Table 1 and Figs. 5-6, which explicitly separate predicted model costs from true costs. No parameter is fitted to the true costs and then reported as a prediction: policies are trained against predicted reward (Eq. 1), the likelihood-based safety term (Eqs. 5-6), and model-computed trajectory separation (Eqs. 8-9), while the measured outcome is the simulator cost. The improvement is therefore not an equality-by-construction of the objective; the transfer of model-computed diversity to informative real-environment exploration is an empirical assumption, so any associated risk is a correctness or external-validity concern rather than circularity. Self-citations [1] and [45] support background regularization formulations and the benchmark's medium-policy setup, and the Industrial Benchmark [3] is a fixed simulator usable outside this paper; none of these is invoked to forbid alternative methods or to justify the diversity objective as forced. The only self-referential equation is Eq. (7), which defines the constrained policy in terms of itself; given the surrounding text ('let π(st; θ) be designed such that only actions in this bound can be computed'), this is a typographical slip rather than a load-bearing circular step, and the experiment using it reports empirical costs under the same external evaluation. Minor reporting inconsistencies (3 versus 6 repetitions in Table 1, omitted α_s and δ values) weaken causal attribution but do not constitute circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The algorithm introduces no invented entities. It depends on learned model fidelity, a Gaussian behavior-policy approximation for safety, a linear action-to-state relation for the constrained policy, and the choice of MinLSED as a diversity signal. Several hyperparameters are manually fixed (lambda=0.4, alpha_d=0.15) or unreported (alpha_s, delta).

free parameters (4)
  • lambda (safety weight in safety-as-objective) = 0.4
    Fixed after visually inspecting 2D policy maps in Section 4.1; no systematic sweep is reported, and the paper states that this validates the assumption of fixing lambda at 0.4.
  • alpha_d (diversity weight) = 0.15
    Used in all Industrial Benchmark runs; no sensitivity analysis or selection procedure is reported.
  • alpha_s (safety soft-constraint weight) = not reported
    Appears in loss Eq. 10 but its value is never stated, so the soft-constraint experiments cannot be reproduced from the manuscript.
  • delta (safety threshold in Eq. 6) = not reported
    Controls permissibility of the soft safety constraint; no value is given in Sections 3.1 or 4.2.
assumptions (4)
  • domain assumption A transition model f(s,a;eta) and reward model f(s,a;omega) learned from the batch are accurate enough for policy optimization and for computing trajectory diversity.
    Virtual rollouts in Algorithm 1 and the diversity term Eq. 9 are computed inside the learned model; if model errors differ across policies, diversity in the model may not transfer to the real system.
  • domain assumption The behavior policy is well approximated by a Gaussian with diagonal covariance, and its unnormalized likelihood is a valid safety measure.
    Eqs. 5 and 6 define the soft safety constraint; the approximation of a mixture of policies by one Gaussian is acknowledged as an assumption but not validated.
  • domain assumption Actions affect a subset of next-state variables in a known linear way for the constrained policy.
    Section 3.1, Constrained Policy, says this approach is only applicable if this linearity holds; the Industrial Benchmark uses this mechanism in Experiment 1.
  • ad hoc to paper Minimum pairwise trajectory distance (Eq. 9) is an effective exploration incentive for iterative batch RL.
    The paper motivates MinLSED as mitigating outlier behavior but provides no theoretical justification or systematic comparison to other diversity metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search." pith.science (2026). https://pith.science/paper/J5TSXK7T

@misc{pith2026241109722,
  author       = {Pith},
  title        = {Pith review of: Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5TSXK7T}},
  note         = {Machine review of arXiv:2411.09722}
}
read the original abstract

Batch reinforcement learning enables policy learning without direct interaction with the environment during training, relying exclusively on previously collected sets of interactions. This approach is, therefore, well-suited for high-risk and cost-intensive applications, such as industrial control. Learned policies are commonly restricted to act in a similar fashion as observed in the batch. In a real-world scenario, learned policies are deployed in the industrial system, inevitably leading to the collection of new data that can subsequently be added to the existing recording. The process of learning and deployment can thus take place multiple times throughout the lifespan of a system. In this work, we propose to exploit this iterative nature of applying offline reinforcement learning to guide learned policies towards efficient and informative data collection during deployment, leading to continuous improvement of learned policies while remaining within the support of collected data. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search, augmented with safety and, importantly, a diversity criterion.

Figures

Figures reproduced from arXiv: 2411.09722 by the authors.

Figure 1
Figure 1. Illustration of iterative model-based policy search. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. 2D grid environment: The behavior policy guides agent towards the nearest behavior goal. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Policy maps for different λ values, colors illustrate action directions in every cell of the grid. For λ = 0.0, the policy imitates behavior policy by navigating to the closest behavior goal. With increasing λ values, the policy moves slowly towards the reward goal. 4.1 2D Grid Environment: Single Iteration The 2D grid environment is a simplistic benchmark illustrating common navigation tasks. The state (x, y) repre… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Constrained Policy. 0 250 500 750 1000 1250 1500 1750 2000 Training Iterations 170 180 190 210 216 220 230 201 193 190 Cost Iteration 1 Iteration 2 Iteration 3 Initial Cost True Cost (a) αd = 0.0 0 250 500 750 1000 1250 1500 1750 2000 Training Iterations 170.0 180.0 19…
Figure 8
Figure 8. Figure 8: Bounded dataset for the industrial benchmark. Shown are trajectories of Velocity (left), [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Velocity, gain and shift rollouts generated by following the medium behavior policy starting [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 32 canonical work pages

  1. [1]

    Swazinna, S

    P. Swazinna, S. Udluft, and T. Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021

    work page 2021

  2. [2]

    T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954–28967, 2021

    work page 2021

  3. [3]

    D. Hein, S. Depeweg, M. Tokic, S. Udluft, A. Hentschel, T. A. Runkler, and V . Sterzing. A benchmark environment motivated by industrial control problems. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–8. IEEE, 2017

    work page 2017

  4. [4]

    Matsushima, H

    T. Matsushima, H. Furuta, Y . Matsuo, O. Nachum, and S. Gu. Deployment-efficient rein- forcement learning via model-based offline optimization. arXiv preprint arXiv:2006.03647, 2020

    arXiv 2006

  5. [5]

    X. Hu, Y . Ma, C. Xiao, Y . Zheng, and Z. Meng. In-sample policy iteration for offline reinforce- ment learning. arXiv preprint arXiv:2306.05726, 2023

    work page Pith review arXiv 2023

  6. [6]

    Zhang, L

    L. Zhang, L. Tedesco, P. Rajak, Y . Zemmouri, and H. Brunzell. Ac- tive learning for iterative offline reinforcement learning. In NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World , 2023. URL https://www.amazon.science/publications/ active-learning-for-iterative-offline-reinforcement-learning

    work page 2023

  7. [7]

    Ernst, M

    D. Ernst, M. Glavic, P. Geurts, and L. Wehenkel. Approximate value iteration in the reinforce- ment learning context. application to electrical power system control. International Journal of Emerging Electric Power Systems, 3(1), 2005

    work page 2005

  8. [8]

    Riedmiller

    M. Riedmiller. Neural fitted Q iteration–first experiences with a data efficient neural reinforce- ment learning method. In European Conference on Machine Learning, pages 317–328. Springer, 2005

    work page 2005

Show all 50 references
  1. [9]

    Riedmiller, T

    M. Riedmiller, T. Gabel, R. Hafner, and S. Lange. Reinforcement learning for robot soccer. Autonomous Robots, 27(1):55–73, 2009

    2009

  2. [10]

    Lange, T

    S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, pages 45–73. Springer, 2012

    2012

  3. [11]

    D. Hein, A. Hentschel, T. A. Runkler, and S. Udluft. Reinforcement learning with particle swarm optimization policy (PSO-P) in continuous state and action spaces. International Journal of Swarm Intelligence Research (IJSIR), 7(3):23–42, 2016

    2016

  4. [12]

    Depeweg, J

    S. Depeweg, J. M. Hernández-Lobato, F. Doshi-Velez, and S. Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. International Conference on Learning Representations, 2017

    2017

  5. [13]

    D. Hein, S. Udluft, and T. A. Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158–169, 2018

    2018

  6. [14]

    Depeweg, J.-M

    S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184–1193. PMLR, 2018

    2018

  7. [15]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. PMLR, 2014. 9

    2014

  8. [16]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  9. [17]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018

    2018 arXiv

  10. [18]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052–2062. PMLR, 2019

    2019

  11. [19]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761–11771, 2019

    2019

  12. [20]

    Y . Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning.arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  13. [21]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020

    2006 arXiv

  14. [22]

    Fujimoto and S

    S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  15. [23]

    Kostrikov, A

    I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  16. [24]

    N. Y . Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020

    2002 arXiv

  17. [25]

    Y . Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline rl? InInternational Conference on Machine Learning, pages 5084–5096. PMLR, 2021

    2021

  18. [26]

    Yin and Y .-X

    M. Yin and Y .-X. Wang. Towards instance-optimal offline reinforcement learning with pes- simism. Advances in neural information processing systems, 34:4065–4078, 2021

    2021

  19. [27]

    Deisenroth and C

    M. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In 28th International Conference on Machine Learning (ICML-11), pages 465–472, 2011

    2011

  20. [28]

    Nagabandi, G

    A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559–7566. IEEE, 2018

    2018

  21. [29]

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y . Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model- based offline policy optimization. In Advances in Neural Information Processing Systems , volume 33, pages 14129–14142, 2020

    2020

  22. [30]

    Kidambi, A

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. MOReL: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020

    2005 arXiv

  23. [31]

    J. Li, X. Hu, H. Xu, J. Liu, X. Zhan, and Y .-Q. Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305.15669, 2023

    2023 arXiv

  24. [32]

    M. S. Mark, A. Ghadirzadeh, X. Chen, and C. Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022 , 2022. URL https://openreview.net/forum?id=ELmiPlCOSw

    2022

  25. [33]

    A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. 10

    2006 arXiv

  26. [34]

    S. Lee, Y . Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning , pages 1702–1712. PMLR, 2022

    2022

  27. [35]

    X. Hu, Y . Ma, C. Xiao, Y . Zheng, and J. Hao. Iteratively refined behavior regularization for offline reinforcement learning. 2023

    2023

  28. [36]

    Schmidhuber

    J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222–227, 1991

    1991

  29. [37]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning , pages 2778–2787. PMLR, 2017

    2017

  30. [38]

    Eysenbach, A

    B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018

    2018 arXiv

  31. [40]

    Yarats, D

    D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13425, 2022

    2022 arXiv

  32. [41]

    Lambert, M

    N. Lambert, M. Wulfmeier, W. Whitney, A. Byravan, M. Bloesch, V . Dasagi, T. Hertweck, and M. Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201.11861, 2022

    2022 arXiv

  33. [42]

    Hong, T.-Y

    Z.-W. Hong, T.-Y . Shann, S.-Y . Su, Y .-H. Chang, T.-J. Fu, and C.-Y . Lee. Diversity-driven exploration strategy for deep reinforcement learning.Advances in neural information processing systems, 31, 2018

    2018

  34. [43]

    Parker-Holder, A

    J. Parker-Holder, A. Pacchiano, K. M. Choromanski, and S. J. Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050–18062, 2020

    2020

  35. [44]

    Kumar, A

    S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33: 8198–8210, 2020

    2020

  36. [45]

    Swazinna, S

    P. Swazinna, S. Udluft, and T. Runkler. User-interactive offline reinforcement learning. Interna- tional Conference on Learning Representations, 2023

    2023

  37. [46]

    Brandfonbrener, W

    D. Brandfonbrener, W. Whitney, R. Ranganath, and J. Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933–4946, 2021

    2021

  38. [47]

    N. K. Sinha and M. P. Griscik. A stochastic approximation method. IEEE Transactions on Systems, Man, and Cybernetics, SMC-1(4):338–344, 1971. doi:10.1109/TSMC.1971.4308316

    1971

  39. [48]

    Bottou, F

    L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223–311, 2018

    2018

  40. [49]

    Mangalam, H

    K. Mangalam, H. Girase, S. Agarwal, K.-H. Lee, E. Adeli, J. Malik, and A. Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part ...

    2020

  41. [50]

    C. Choi, J. H. Choi, J. Li, and S. Malla. Shared cross-modal trajectory prediction for au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 244–253, 2021

    2021

  42. [51]

    H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y . Shen, Y . Shen, Y . Chai, C. Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, pages 895–904. PMLR, 2021. 12 A Dataset visualization for industrial benchmark A.1 Random bounde...

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.