Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation

Amir Abolfazli; Avishek Anand; Wolfgang Nejdl; Zekun Song

REVIEW 5 major objections 4 minor 60 references

Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation

T0 review · 5 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that offline reinforcement learning on mixed datasets can be improved by scoring each transition for similarity to a target dataset and training on the highest-valued trajectories in curriculum order, with gains of up to…

desk verdict Under-specified trajectory valuation and a broken Pinsker argument, but the empirical recipe is interesting enough to warrant a careful revision, not a desk reject. read the letter →

arxiv 2502.00601 v2 pith:KPVR2GU5 submitted 2025-02-02 cs.LG

Amir Abolfazli , Zekun Song , Avishek Anand , Wolfgang Nejdl This is my paper · ORCID

classification cs.LG

keywords offlinereinforcementlearningcurriculumtrajectoryvaluationtransitionscoringdomainmismatchmixeddatasetsD4RLMuJoCo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that offline reinforcement learning agents can learn better policies when the training data is a mix of abundant low-quality source data and a small amount of high-quality target data. It proposes CLTV, a two-stage method: Transition Scoring (TS) gives each source transition a relevance score using domain classifiers and a REINFORCE-trained network, then a curriculum loop ranks whole trajectories by a similarity-and-return score and trains the base offline RL algorithm on the top trajectories merged with the target dataset. Across CQL and IQL on four MuJoCo domains, the paper reports that CLTV consistently outscores Vanilla, CUORL, and Harness, with total-score improvements of up to 95% over the second-best method. A theoretical analysis is offered to bound target-domain value in terms of source-domain value plus a KL-divergence term.

What carries the argument

The load-bearing object is the trajectory valuation formula in Eq. 13: $v_i = \exp\big(-D_{\mathrm{KL}}(\pi^T_{\theta_2}(u_t|x_t) \| \pi^S_{\theta_1}(u_t|x_t))\big) \cdot \sum_{t=1}^T \gamma^{t-1} r_t$. It combines a policy-similarity term, computed from the KL divergence between the target and source policies at the transition's state-action pair, with the discounted return, so that trajectories are ranked by a blend of 'looks like the target domain' and 'high reward.' The TS component supplies per-transition scores $w_j = v_\phi(t_j)$, trained with a REINFORCE-style objective using the dynamics factor $\Delta_\theta$ from domain classifiers $q_{xu}$ and $q_{xux'}$; these scores resize the source rewards before the curriculum loop runs.

What would settle it

Run Algorithm 2 with the target policy never updated and record the valuation scores from Eq. 13: if the KL term has no defined value at a single action, the formula cannot be computing what the paper claims and the reported gains must come from elsewhere. A concrete experiment is to set $\pi^T = \pi^S$ in the valuation formula and compare final normalized scores in Table 1; if the gains persist, the KL term is not load-bearing.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a trajectory-level valuation combining policy similarity and discounted return can fix the source-target mismatch in mixed offline RL datasets. Each trajectory is valued as $v_i = \exp\big(-D_{\mathrm{KL}}(\pi^T_{\theta_2}(u_t|x_t) \| \pi^S_{\theta_1}(u_t|x_t))\big) \cdot \sum_{t=1}^T \gamma^{t-1} r_t$, and the agent trains on the top $m$ such trajectories together with the target dataset. The paper argues that this curriculum ordering lets the agent learn from source transitions that resemble the target domain while discarding irrelevant ones, improving both performance and transferability of the learned policy. The authors support this with Theorem 1, bounding the target value of a policy by its source value minus a KL-dependent term, and Theorem 2, giving a policy-improvement lower bound in terms of KL divergence to the behavior policy.

Load-bearing premise

The paper assumes that the similarity between the target and learned policies can be measured at a single sampled action and that a target policy is available for this measurement, even though the algorithm never trains that target policy.

Editorial extensions

If this is right

If CLTV works as reported, practitioners can take an existing offline RL algorithm such as CQL or IQL and improve it on mixed datasets by adding transition scoring and curriculum ranking, without changing the base algorithm's internals.
The method reduces the need for large target datasets: with only 10% target data, the top source trajectories are selected to augment it.
The curriculum ordering yields faster convergence and lower variance in some environments, as shown by the learning curves in Figure 1.
The theoretical bounds, if valid, give a principled reason to minimize KL divergence between source and target dynamics when selecting data for transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

As written, eq. (13) uses the KL divergence between two policies evaluated at a single point $(u_t, x_t)$, which is not defined; a workable version would replace it with a log-density-ratio estimate like $\log(\pi^T(u_t|x_t)/\pi^S(u_t|x_t))$, connecting the method to off-policy importance weighting.
The reported gains come from a fixed 90/10 source-target mix; varying the mixing ratio would reveal whether the curriculum ranking or the TS reward resizing is the active ingredient.
Because Algorithm 2 never updates the target policy, the target side could be replaced by a fixed behavior-cloned policy; if results are insensitive to that choice, the method's success likely comes from TS reward shaping rather than the KL term.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things before anything else. First, the central valuation formula in Eq. 13, as written, is not a well-defined quantity: it plugs a single sampled action into the KL divergence between two continuous policies, and KL doesn't have a pointwise value. Second, Algorithm 2 lists pi^T_theta2 as an input but never initializes or updates it, so the trajectory values the method sorts on are computed with an unspecified policy. That makes the proposed procedure unimplementable from the description alone, and the code becomes the only real specification. The stress-test note is right on both counts.

What the paper does well is put together a sensible-looking recipe: use DARC-style domain classifiers to score transitions, use DVRL-style REINFORCE to learn the scoring network, then use the scores to reweight rewards and pick trajectories via a curriculum. The empirical results in Table 1 are strong, and the ablation against TS alone (Figure 1) does show the curriculum component contributes. There is also an available code repo, which is more than many submissions in this area.

The soft spots are proportional. The Pinsker error in Corollary 1 is real: they bound ||p_S - p_T||_infinity and then apply Pinsker, which controls half the L1 norm, not the sup norm. The theorem as stated is not proven. That said, the theorem is generic and doesn't do load-bearing work for the algorithm; the empirical claim doesn't rest on it. The reward modification in Algorithm 2 line 13 is also a bit mysterious, since w_j is a TS score, not a reward, and the justification for that linear interpolation is thin.

The deeper problem remains Eq. 13 and the missing target policy. If the authors fix that by defining the similarity term as, say, the negative log-density ratio at the sampled action (which is what the gradient of KL would involve) and actually train pi^T_theta2 on the target data, the method becomes coherent and the experiments could be reproduced. As submitted, the theory is not rigorous and the algorithm is incomplete.

Who gets value from this: researchers working on data selection and curriculum for offline RL will find the combination and the empirical gains interesting enough to read, but with the caveat that the description is aspirational. I would send it to peer review, but with the expectation of major revision: fix or remove the theoretical claims, specify the target policy, and re-run the baselines with the corrected algorithm.

My verdict: deserves a serious referee, not desk reject, but it needs real work before it can be trusted.

Referee Report

5 major / 4 minor

Summary. This manuscript proposes two components for offline RL with mixed datasets: Transition Scoring (TS), a REINFORCE-trained DNN that scores source transitions by similarity to a target dataset using domain classifiers, and Curriculum Learning-Based Trajectory Valuation (CLTV), which modifies source rewards with TS scores, ranks source trajectories using Eq. (13), and trains the source actor-critic on the top-m trajectories merged with target data. The authors report that CLTV outperforms Vanilla, CUORL, and Harness for CQL and IQL across Ant, HalfCheetah, Hopper, and Walker2d on mixed D4RL datasets, and they present two theorems bounding policy improvement.

Significance. The problem addressed, selecting valuable trajectories from mixed source/target offline datasets, is relevant, and the manuscript provides a broad D4RL comparison, ablations, and a code link. If the trajectory-valuation mechanism were well defined and the theoretical results connected to the algorithm, the empirical improvements would be interesting. As written, however, the core valuation formula is not computable, the target policy used in it is never trained or specified, the main theoretical bound relies on an invalid use of Pinsker's inequality, and the experimental protocol does not instantiate the paper's stated source-target dynamics mismatch. The central claim is therefore unsupported, and I do not see a local revision that would repair the paper.

major comments (5)

[Eq. (13); Algorithm 2] The trajectory value v_i, which is the sole criterion for curriculum selection, is undefined as written. D_KL(pi^T_theta2(u_t|x_t) || pi^S_theta1(u_t|x_t)) is evaluated at a single sampled action u_t, but the KL divergence is an integral over the action space; a pointwise log-density ratio is not a distributional divergence, and it can be negative, making exp(-D_KL) not a meaningful similarity weight. Moreover, Algorithm 2 lists pi^T_theta2 as an input but never initializes or updates it; the only update in the curriculum loop (line 20) changes omega_1 and theta_1. Consequently v_i cannot be computed, the top-m selection in lines 18-19 is not a concrete procedure, and the experimental results in Table 1 are not reproducible from the manuscript.
[Corollary 1, Eqs. (18)-(21)] Pinsker's inequality bounds the total variation distance (equivalently, half the L1 norm) by sqrt(1/2 KL), but Corollary 1 needs a bound on ||pS - pT||_infty. No such inequality follows from Pinsker: total variation does not control the sup norm. The replacement in Eq. (20) is therefore invalid, and the bounds in Theorem 1 (Eq. (22)) and Theorem 2 (Eq. (32)), both of which rely on Pinsker, are not proven by the provided argument.
[Section 5.1, Theorem 2] Even if the KL-TV inequalities were correct, Theorem 2 is a generic policy-improvement bound derived from the performance difference lemma and Pinsker's inequality; it involves a behavior policy pi_b and the next policy pi_{i+1}, but it does not mention TS scores, the trajectory value v_i, the target policy pi^T_theta2, the reward modification in Algorithm 2 line 13, or the top-m trajectory selection. Thus the claim that the theorems demonstrate the efficacy of CLTV is unsupported: no proof shows that CLTV controls the KL term in the bound or that the selected trajectories produce the stated improvement.
[Section 6, datasets; Section 3] The experimental setup does not instantiate the paper's stated problem. Section 3 defines source-target domain mismatch through different transition dynamics pS != pT, and the theory in Corollary 1 and Theorem 1 concerns differing transition probabilities. The D4RL mixed datasets used in Section 6 are random, medium, and expert trajectories from the same MuJoCo environment, which share the environment's transition dynamics; they differ in the behavior policies that generated them and in return levels. Consequently the experiments cannot validate the claimed improvement for source-target dynamics mismatch, and the classifier-based factor in Eq. (9) need not correspond to the domain shift analyzed in the theory.
[Algorithm 2, lines 12-13 and 17] Line 13 rewrites each source reward as r_j <- (1-lambda) r_j + lambda w_j, where w_j is the TS score, and Eq. (13) then multiplies the discounted sum of these modified rewards by an exponential policy-similarity factor. As a result, high TS scores enter v_i both through the reward term and indirectly through selection, so the valuation partially optimizes its own proxy. The paper should provide an analysis or ablation separating the reward-modification effect from the trajectory-selection effect; without this, the reported gains are not attributable to the proposed trajectory valuation.

minor comments (4)

[Eq. (13); Algorithm 2 line 17] The symbol written as P_T in Eq. (13) looks like a product, while the text describes a discounted sum; replace it with \sum and define the trajectory length, especially since trajectories in Algorithm 2 are written with a generic T although source trajectories earlier have variable lengths L_k.
[Algorithm 2 line 19; Table 4] The hyperparameter m is described as the number of episodes sampled from the source dataset but is set to 0.1 in Table 4, which suggests a fraction; please clarify whether m is a count or a proportion of the source dataset.
[Table 1] In the Ant/IQL row, the CLTV total score of 283.93 is essentially identical to the Vanilla total of 283.91, and the reported percentage increase is 0%; this is difficult to reconcile with the paper's claim that CLTV consistently outperforms all baselines, and no significance test is reported.
[References] References [41] and [42] appear to be the same Sutton 1988 paper, and the duplicate should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; Eq 13 is undefined/unimplementable but not a circular reduction.

full rationale

The paper's claimed derivation chain does not contain a step in which a prediction or first-principles result is equivalent to its input by construction. TS trains the scoring network v_phi with a REINFORCE objective whose reward includes the external classifier-based dynamics factor Delta_theta and a sum-of-scores regularizer; although the regularizer makes the training reward partly self-referential, the Delta_theta term is computed from independently trained source/target classifiers and the final experimental evaluation uses external D4RL normalized scores, so the empirical claims are not forced by the scoring mechanism. Theorems 1 and 2 are standard total-variation/KL policy-improvement bounds assembled from cited lemmas ([1], [4], [26], [47]); they are not derived from CLTV and are not used to fit any CLTV parameter, so they do not create a circular dependency. There are no load-bearing self-citations by the authors of this paper. The serious defect in Equation 13 is a correctness/reproducibility problem rather than a circularity problem: D_KL(pi^T_theta2(u_t|x_t) || pi^S_theta1(u_t|x_t)) is written as a pointwise quantity even though KL divergence is defined as an integral over actions, and Algorithm 2 lists pi^T_theta2 as an input but never initializes or updates it, leaving the valuation formula unimplementable as written. That missing definition is not an equivalence between an output and an input, so it does not raise the circularity score. Overall, no significant circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central derivation relies on standard RL inequalities, domain-classifier density ratios, availability of a target policy, and a reward-modification invariance that is never proven. The free parameters lambda, delta, m, and the TS/classifier hyperparameters are fitted to the same benchmark tasks on which the method is evaluated. No new physical or mathematical entities are postulated.

free parameters (5)

lambda (score-reward ratio) = 0.8
Balances original reward and TS score in the reward modification (Algorithm 2 line 13); selected by grid search on the same benchmark datasets.
delta (similarity-reward ratio) = 0.7
Balances dynamics factor versus selection probability in the TS unnormalized reward (Eq 4); selected by grid search.
m (trajectory episode ratio) = 0.1
Number of top-valued trajectories sampled per training step; selected by grid search.
TS batch size = 200
Batch size for the REINFORCE transition scorer; chosen by grid search and reported in Appendix C.2.
TS and classifier architecture hyperparameters = [256,256] hidden, classifier hidden 256, lr 3e-4, Gaussian std 0.1
Network sizes, learning rates, and noise scale chosen by grid search; these affect the learned scores and final performance.

assumptions (5)

standard math The performance difference lemma (Lemma 1) and value estimation error bounds (Lemma 2) hold in the stated form.
Used as the foundation for Corollaries 1-2 and Theorems 1-2 in Section 5.1; the paper does not prove them but cites standard references.
standard math Pinsker's inequality bounds the L-infinity norm of the difference of transition distributions.
Corollary 1's proof uses ||pS-pT||_inf <= sqrt(0.5 D_KL(pS||pT)), but Pinsker's inequality bounds total variation (L1), not L-infinity; this assumption is false in general.
domain assumption Domain classifiers q_xu and q_xux' trained with cross-entropy yield reliable density ratios via Bayes' rule (Eqs 5-9).
This is the standard DARC assumption; in the D4RL setting the implied dynamics ratio is near zero, so the TS signal is empty.
domain assumption A target policy pi^T_theta2 is available and informative for trajectory valuation.
Algorithm 2 takes pi^T as input and never updates it; no training procedure or verification of target policy quality is given.
ad hoc to paper Modifying rewards as r <- (1-lambda)*r + lambda*w preserves the optimal policy.
The modification is state-action dependent and is not shown to be potential-based shaping; arbitrary reward changes can alter the optimal policy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation." pith.science (2026). https://pith.science/paper/KPVR2GU5

@misc{pith2026250200601,
  author       = {Pith},
  title        = {Pith review of: Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPVR2GU5}},
  note         = {Machine review of arXiv:2502.00601}
}

read the original abstract

The success of deep reinforcement learning (DRL) relies on the availability and quality of training data, often requiring extensive interactions with specific environments. In many real-world scenarios, where data collection is costly and risky, offline reinforcement learning (RL) offers a solution by utilizing data collected by domain experts and searching for a batch-constrained optimal policy. This approach is further augmented by incorporating external data sources, expanding the range and diversity of data collection possibilities. However, existing offline RL methods often struggle with challenges posed by non-matching data from these external sources. In this work, we specifically address the problem of source-target domain mismatch in scenarios involving mixed datasets, characterized by a predominance of source data generated from random or suboptimal policies and a limited amount of target data generated from higher-quality policies. To tackle this problem, we introduce Transition Scoring (TS), a novel method that assigns scores to transitions based on their similarity to the target domain, and propose Curriculum Learning-Based Trajectory Valuation (CLTV), which effectively leverages these transition scores to identify and prioritize high-quality trajectories through a curriculum learning approach. Our extensive experiments across various offline RL methods and MuJoCo environments, complemented by rigorous theoretical analysis, demonstrate that CLTV enhances the overall performance and transferability of policies learned by offline RL algorithms.

Figures

Figures reproduced from arXiv: 2502.00601 by the authors.

**Figure 2.** Runtime analysis of offline RL algorithms. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Heatmaps illustrating the performance of CLTV (CQL) on mixed datasets with respect to different [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Heatmaps illustrating the performance of CLTV (IQL) on mixed datasets with respect to different [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 36 canonical work pages

[1]

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. 2019. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, W A, USA, Tech. Rep 32 (2019), 96

work page 2019
[2]

Cynthia Breazeal and Andrea L Thomaz. 2008. Learning from human teachers with socially guided exploration. In2008 IEEE International Conference on Robotics and Automation. IEEE, 3539–3544

work page 2008
[3]

Yuanying Cai, Chuheng Zhang, Hanye Zhao, Li Zhao, and Jiang Bian. 2023. Curriculum offline reinforcement learning. InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems . 1221–1229

work page 2023
[4]

Imre Csiszár and János Körner. 2011. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press

work page 2011
[5]

Rasheed El-Bouri, David Eyre, Peter Watkinson, Tingting Zhu, and David Clifton

work page
[6]

Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. 2020. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916 (2020)

arXiv 2020
[7]

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. 2020. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 (2020)

arXiv 2020
[8]

Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau

Show all 60 references

[9]

Scott Fujimoto and Shixiang Shane Gu. 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34 (2021), 20132–20145

2021
[10]

Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596

2018
[11]

Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep rein- forcement learning without exploration. In International conference on machine learning. PMLR, 2052–2062

2019
[12]

Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa
[13]

Amirata Ghorbani and James Zou. 2019. Data shapley: Equitable valuation of data for machine learning. InInternational conference on machine learning. PMLR, 2242–2251

2019
[14]

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . 1861–1870

2018
[15]

Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexan- der Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. 2017. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning . PML...

2017
[16]

Zhang-Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche
[17]

Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. 2018. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933 (2018)

2018 arXiv
[18]

Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen. 2017. A novel DDPG method with prioritized experience replay. In2017 IEEE international conference on systems, man, and cybernetics (SMC) . IEEE, 316–321

2017
[19]

David Isele, Alireza Nakhaei, and Kikuo Fujimura. 2018. Safe reinforcement learning on autonomous vehicles. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 1–6

2018
[20]

Chaohai Kang, Chuiting Rong, Weijian Ren, Fengcai Huo, and Pengyun Liu
[21]

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims
[22]

Pascal Klink, Carlo D’Eramo, Jan Peters, and Joni Pajarinen. 2021. Boosted curriculum reinforcement learning. In ICLR

2021
[23]

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 (2021)

2021 arXiv
[24]

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems 32 (2019)

2019
[25]

IEEE Access 9 (2021), 60296–60308

Deep deterministic policy gradient based on double network prioritized experience replay. IEEE Access 9 (2021), 60296–60308

2021
[26]

Sascha Lange, Thomas Gabel, and Martin Riedmiller. 2012. Batch reinforcement learning. In Reinforcement learning: State-of-the-art. Springer, 45–73

2012
[27]

Advances in neural information processing systems 33 (2020), 21810–21823

Morel: Model-based offline reinforcement learning. Advances in neural information processing systems 33 (2020), 21810–21823

2020
[28]

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

2015 arXiv
[29]

Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, and Tie-Yan Liu. 2021. Curriculum offline imitating learning. Advances in Neural Information Processing Systems 34 (2021), 6266–6277

2021
[30]

Amarsagar Reddy Ramapuram Matavalam, Kishan Prudhvi Guddanti, Yang Weng, and Venkataramana Ajjarapu. 2022. Curriculum based reinforcement learning of grid topology controllers to prevent thermal cascading. IEEE Transactions on Power Systems (2022)

2022
[31]

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems 33 (2020), 1179–1191

2020
[32]

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. 2020. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 (2020)

2020 arXiv
[33]

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

2020 arXiv
[34]

Ng, Daishi Harada, and Stuart J

Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML ’99). Morgan Kaufmann Publishers Inc., San Fra...

1999
[35]

Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini
[36]

Zhipeng Ren, Daoyi Dong, Huaxiong Li, and Chunlin Chen. 2018. Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning. IEEE transactions on neural networks and learning systems 29, 6 (2018), 2216–2226

2018
[37]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533

2015
[38]

Takuma Seno and Michita Imai. 2022. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research 23, 315 (2022), 1–20

2022
[39]

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research 21, 181 (2020), 1–50

2020
[40]

Yihao Sun. 2023. Offlinerl-kit: An elegant pytorch offline reinforcement learning library

2023
[41]

Richard S Sutton. 1988. Learning to predict by the methods of temporal differ- ences. Machine learning 3 (1988), 9–44

1988
[42]

IEEE Transactions on Neural Networks and Learning Systems (2023)

A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems (2023)

2023
[43]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An intro- duction. MIT press

2018
[44]

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)

2015 arXiv
[45]

Sebastian Thrun and Anton Schwartz. 1993. Issues in using function approxima- tion for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School. Hillsdale, NJ, 255–263

1993
[46]

Dylan Slack, Yinlam Chow, Bo Dai, and Nevan Wichers. 2022. Safer: Data- efficient and safe reinforcement learning via skill acquisition. arXiv preprint arXiv:2202.04849 (2022)

2022 arXiv
[47]

Tsybakov

Alexandre B. Tsybakov. 2008. Introduction to Nonparametric Estimation (1st ed.). Springer Publishing Company, Incorporated

2008
[48]

Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. 2020. Safe reinforcement learning via curriculum induction. Advances in Neural Information Processing Systems 33 (2020), 12151–12162

2020
[49]

Richard S. Sutton. 1988. Learning to predict by the methods of temporal dif- ferences. Machine Learning 3, 1 (01 Aug 1988), 9–44. https://doi.org/10.1007/ BF00115009

1988
[50]

Sijia Xu, Hongyu Kuang, Zhuang Zhi, Renjie Hu, Yang Liu, and Huyang Sun
[51]

Shengpu Tang and Jenna Wiens. 2021. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference. PMLR, 2–35

2021
[52]

Wenxuan Zhou, Sujay Bajracharya, and David Held. 2021. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning . PMLR, 1719–1735. APPENDIX A THEORETICAL ANALYSIS Derivation of Equation 11 The gradient of the objective function 𝐽 𝜋𝜙 (Equati...

2021
[53]

Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 5026–5033

2012
[56]

Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 (1992), 229–256

1992
[58]

In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol

Macro action selection with deep reinforcement learning in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 15. 94–99
[59]

Jinsung Yoon, Sercan Arik, and Tomas Pfister. 2020. Data valuation using re- inforcement learning. In International Conference on Machine Learning . PMLR, 10842–10851

2020
[2019]

arXiv preprint arXiv:1910.01708 (2019)

Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708 (2019)

2019 arXiv
[2020]

In International Conference on Machine Learning

Student-teacher curriculum learning via reinforcement learning: Predicting hospital inpatient admission location. In International Conference on Machine Learning. PMLR, 2848–2857
[2021]

Journal of Machine Learning Research 22, 77 (2021), 1–14

Chainerrl: A deep reinforcement learning library. Journal of Machine Learning Research 22, 77 (2021), 1–14

2021
[2023]

arXiv preprint arXiv:2306.13085 (2023)

Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.13085 (2023)

2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. 2019. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, W A, USA, Tech. Rep 32 (2019), 96

work page 2019

[2] [2]

Cynthia Breazeal and Andrea L Thomaz. 2008. Learning from human teachers with socially guided exploration. In2008 IEEE International Conference on Robotics and Automation. IEEE, 3539–3544

work page 2008

[3] [3]

Yuanying Cai, Chuheng Zhang, Hanye Zhao, Li Zhao, and Jiang Bian. 2023. Curriculum offline reinforcement learning. InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems . 1221–1229

work page 2023

[4] [4]

Imre Csiszár and János Körner. 2011. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press

work page 2011

[5] [5]

Rasheed El-Bouri, David Eyre, Peter Watkinson, Tingting Zhu, and David Clifton

work page

[6] [6]

Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. 2020. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916 (2020)

arXiv 2020

[7] [7]

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. 2020. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 (2020)

arXiv 2020

[8] [8]

Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau