Pith. sign in

REVIEW 2 major objections 4 minor 44 references

Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that an imitation objective combining KL-matching to good data and KL-repulsion from bad data stays convex when the good term is weighted at least as heavily as the bad term, and it turns this into a practical…

desk verdict Useful empirical paper with a strong method and a real hole: the paper never shows the Lagrangian dual step that turns Eq. (2) into Eq. (3), and the standard DICE dual looks different. read the letter →

arxiv 2505.21182 v1 pith:WJVZRSFM submitted 2025-05-27 cs.LG cs.AI

Huy Hoang , Tien Mai , Pradeep Varakantham , Tanvi Verma This is my paper
classification cs.LGcs.AI
keywords offlineimitationlearningcontrastivedemonstrationsdifferenceofKLdivergencesconvexoptimizationoccupancymeasureQ-learningbehaviorcloningsafe
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline imitation learning usually treats all demonstrations as positive examples, ignoring the signal in explicitly undesirable behavior. This paper proposes to learn from both expert and bad data by minimizing the KL divergence to the good policy while maximizing the KL divergence away from the bad policy, giving the objective $\min_{d_\pi} D_{\mathrm{KL}}(d_\pi \| d_G) - \alpha D_{\mathrm{KL}}(d_\pi \| d_B)$. The central claim is that despite being a difference of convex functions, this objective is convex in the occupancy measure whenever $\alpha \le 1$, so it can be recast as a tractable, non-adversarial Q-learning problem. The resulting algorithm, ContraDICE, is reported to consistently outperform prior baselines across standard offline imitation benchmarks.

What carries the argument

The load-bearing object is the occupancy-measure objective $\min_{d_\pi} D_{\mathrm{KL}}(d_\pi \| d_G) - \alpha D_{\mathrm{KL}}(d_\pi \| d_B)$, with $d_G$ and $d_B$ the visitation distributions of good and bad demonstrations. Its convexity for $\alpha \le 1$ allows the constrained problem to be dualized into a Q-learning form, and the identity $\exp(t) \ge t+1$ gives a linear lower bound that replaces the unstable exponential with a linear surrogate, keeping the final objective convex in $Q$ and concave in the policy $\pi$.

What would settle it

Write out the Lagrangian dual of Equation (2) explicitly; alternatively, on a small tabular MDP compute the true minimizer of $f(d_\pi)$ and compare it with the policy produced by minimizing the surrogate objective of Equation (6). If the two disagree on a case where $\alpha \le 1$, the claimed equivalence fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the natural 'imitate the good, avoid the bad' objective, written as a difference of KL divergences over state-action occupancy measures, remains convex as long as the good term is weighted at least as heavily as the bad term ($\alpha \le 1$). This convexity licenses a Lagrangian-duality reformulation into a soft Q-learning objective, and a linear lower bound on the exponential term turns that objective into a convex, non-adversarial surrogate in the Q-function space. The paper also introduces a Q-weighted behavior-cloning step, proven to recover the same optimal policy as advantage-weighted cloning, and reports that the full method outperforms existing offline imitation baselines on a wide range of benchmark dataset combinations.

Load-bearing premise

The Lagrangian-duality step that is supposed to convert the constrained convex program into the Q-learning objective is only referenced to the appendix, which does not actually derive it; if that step is invalid, the loss being optimized is not the stated difference-of-KL objective.

Editorial extensions

If this is right

  • With $\alpha \le 1$, the training objective is convex, so no adversarial discriminator is needed and optimization is stable.
  • The surrogate lower bound preserves the original objective's optima in the idealized case where the soft Bellman residual vanishes, and it generalizes the IQ-Learn objective, which it recovers exactly when $\alpha=0$ and $d_G=d_U$.
  • The proposed Q-weighted behavior cloning provably recovers the same optimal policy as advantage-weighted cloning while avoiding the bias introduced by a separately estimated value function.
  • Empirically, one good trajectory plus ten bad trajectories suffices for near-expert performance on the tested benchmarks, and the paper estimates that about 2–5 bad trajectories can substitute for one good trajectory.
  • The method is limited to $\alpha \le 1$; the authors explicitly note that relaxing this constraint remains future work and that noisy or weakly labeled demonstrations are not yet handled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The convexity result appears specific to KL divergence; the appendix shows the same difference-of-divergence objective fails to be convex for general $f$-divergences, so the framework may not transfer to other divergences without additional machinery.
  • If the duality derivation holds, the same recipe of pairing a convex KL-matching term with a weighted KL-avoidance term could be applied to other Q-learning-based imitation or offline reinforcement learning pipelines.
  • A natural testable extension is to replace the discriminator-based ratio estimates in the algorithm with more sample-efficient density-ratio estimators, which could matter more in low-data regimes.
  • The claimed 2–5 bad trajectories ≈ 1 good trajectory equivalence is empirical and task-dependent; checking it across more diverse environments would clarify when bad data is genuinely substitutable for expert data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes ContraDICE, an offline imitation learning method that learns from both expert (good) and undesirable (bad) demonstrations. The central idea is to optimize a difference of KL divergences, f(d_pi) = D_KL(d_pi || d_G) - alpha D_KL(d_pi || d_B), over state-action occupancy measures. The authors claim that for alpha <= 1 this objective is convex in d_pi (Proposition 4.1), which enables a Lagrangian-duality reformulation as a Q-learning objective (Equation 3). A lower-bound surrogate is then introduced (Equation 6) that is non-adversarial and convex in Q, and a Q-weighted behavior-cloning policy extraction is proposed (Proposition 5.1). The method is evaluated on 18 MuJoCo, Adroit, and FrankaKitchen dataset combinations, reporting strong performance against baselines including SMODICE, ILID, ReCOIL, SafeDICE, and DWBC-GB, together with ablations on the number of bad trajectories, the trade-off parameter alpha, and the entropy parameter beta.

Significance. If the theoretical claims are fully substantiated, this is a useful contribution: it provides a principled, non-adversarial objective for offline imitation learning that explicitly repels undesirable behavior, while reducing to IQ-Learn as a special case when alpha = 0. The empirical study is extensive, covers diverse domains, includes ablations, and the authors provide source code. The convexity observation for the difference-of-KL objective is plausible and the Q-weighted BC extraction is a practical simplification. However, the manuscript's central theoretical grounding is currently incomplete: the key Lagrangian-duality step from the constrained program (2) to the Q-learning objective (3) is asserted but never derived, and the proof of Proposition 4.1 in the appendix contains a typographical error that drops logarithmic terms. These gaps must be fixed before the 'provably balances' claim is supported.

major comments (2)
  1. [Section 4.1, Eq. (3)] The paper states that 'details of the derivation are given in the appendix' for converting the constrained difference-of-KL program (2) into the Q-learning objective (3) via Lagrangian duality, but Appendix A does not contain this derivation; it only proves Propositions 4.1, 4.2, 4.3, 4.4, and 5.1. This step is load-bearing because all downstream claims—convexity in Q, the lower-bound surrogate in Proposition 4.3, the min-max swap in Proposition 4.4, and the QW-BC extraction in Proposition 5.1—concern the derived objective (3)/(6) rather than the original occupancy-measure objective (1). A standard DICE-style dual of (2) would have a different max/min order and a different exponential form, so the claimed equivalence is not obvious. Please provide the full derivation, including the treatment of the pi-dependent entropy term, any normalization constants, and the exact max-min order, or provide a counterexample showing the equivalence fails.
  2. [Appendix A, Eq. (9) (proof of Proposition 4.1)] The displayed expansion in the proof of Proposition 4.1 is incorrect as written: it writes d_pi(s,a)(alpha d_B(s,a) - d_G(s,a)) in the second term, dropping the logarithms from d_B and d_G. The correct expansion is (1-alpha) d_pi log d_pi + d_pi(alpha log d_B - log d_G). Although the corrected expansion still supports the convexity conclusion, the proof as printed is invalid and should be fixed.
minor comments (4)
  1. [Section 3, Preliminaries] The definition of the occupancy measure d_pi(s,a) writes the sum over t starting at t=1 and omits the initial state distribution p_0; the standard definition is (1-gamma) sum_{t=0}^infinity gamma^t P(s_t=s) pi(a|s). The constraint in Eq. (2) uses the correct form, so this is a presentation error.
  2. [Section 4.1, Remark] Proposition 4.1 states convexity for alpha <= 1, but the objective in Eq. (3) is only valid for alpha < 1 because of the division by (1-alpha). The remark handles alpha = 1 separately, but the main text should state this restriction more prominently to avoid confusion.
  3. [Section 4.2 / Algorithm 2] The claim that maximizing the lower bound eL 'promotes' the original objective L is informal; a pointwise lower bound does not guarantee that the maximizer of eL is close to a maximizer of L. If the paper intends a theoretical justification for the surrogate, a formal statement with a bound would be helpful; otherwise the wording should be softened.
  4. [Algorithm 2, line 17] The Q-update in Algorithm 2 minimizes eL(Q|V) plus a quadratic term (Q - gamma V)^2/2, but this regularizer is only described in the text as a chi^2 divergence regularizer; the connection would be clearer if the equation number for the regularized objective were given.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the objective and its surrogate are independently derived, and the unproven duality step is a correctness gap rather than a circular reduction.

full rationale

The central derivation is not circular. The objective in Eq. (1) is a stated difference-of-KL program; Proposition 4.2 is an algebraic identity; Proposition 4.3 follows from e^t >= t + 1; Proposition 4.4 uses log-sum-exp convexity and a minimax swap; Proposition 5.1 solves both weighted-MLE objectives in closed form. The main load-bearing step, the duality passage from Eq. (2) to Eq. (3), is asserted with 'details of the derivation are given in the appendix', yet Appendix A contains only the auxiliary propositions and not that derivation; a standard DICE dual would have a different max/min order and exponential form. This is an unverified equivalence and a correctness risk, but it is not an identity by construction or a fitted parameter renamed as a prediction. Appendix A Eq. (9) also drops the log dG and log dB terms in the expansion, but the corrected expansion still gives convexity for alpha <= 1, so this is a typographical gap rather than a circular step. Self-citations ([10], [15], [16], [26]) are background literature statements, and the alpha = 0 reduction to IQ-Learn is explicitly acknowledged as a special case in the remarks. Alpha and beta are tuned per task in Table 3, which is ordinary hyperparameter selection, not a fitted input renamed as a prediction. Consequently, no specific reduction of a claimed prediction to its own inputs can be exhibited, and the paper is self-contained against external D4RL benchmarks.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central derivation uses standard convex optimization and density-ratio estimation tools; the only fitted numbers are the task-specific hyperparameters alpha and beta. No new physical or model entities are introduced.

free parameters (2)
  • alpha = 0.2-0.8 (per task, Table 3)
    Balances the KL divergence to the bad distribution; must satisfy alpha <= 1 for convexity, and is tuned per benchmark.
  • beta = 3.0-30.0 (per task, Table 3)
    Temperature in the MaxEnt soft value function and extreme-V update; tuned per benchmark.
assumptions (5)
  • domain assumption State-action occupancy measures are probability distributions over a finite support.
    The convexity and duality arguments work on the simplex of distributions; continuous state-action spaces are handled implicitly through function approximation.
  • domain assumption The density ratios d_G/d_U and d_B/d_U are consistently estimated by logistic regression discriminators.
    Algorithm 1 trains classifiers to compute the Psi(s,a) term; if the discriminators are inaccurate, the objective is biased.
  • standard math Strong duality holds for the Lagrangian of the constrained convex program in Eq. (2).
    Needed to convert the primal to the dual Q-learning form; the paper does not verify constraint qualification.
  • standard math The exponential lower bound e^t >= t+1 can be applied to the exponent of the dual objective.
    Used in Proposition 4.3 to form the surrogate objective; equality only when the Bellman residual is zero.
  • domain assumption alpha <= 1 is maintained.
    Convexity of the objective relies on this inequality; the paper notes alpha>1 violates convexity and gives only a naive adaptation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations." pith.science (2026). https://pith.science/paper/WJVZRSFM

@misc{pith2026250521182,
  author       = {Pith},
  title        = {Pith review of: Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJVZRSFM}},
  note         = {Machine review of arXiv:2505.21182}
}
read the original abstract

Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2505.21182 by the authors.

Figure 1
Figure 1. Effect of the size of the bad dataset B B on learning performance: The results are averaged over five different training seeds and reported using normalized scores. As the number of bad trajec￾tories increases, our method demonstrates a strong ability to leverage this data. In contrast, baseline methods such as SafeDICE and DWBC-GB struggle to make effective use of bad demonstrations. To answer question (Q2), we inv… view at source ↗
Figure 2
Figure 2. Sensitivity analysis on the trade-off parameter α. From our objective function (1), we introduce a hyper￾parameter 0 ≤ α < 1, which controls the weighting of the bad data objective—this relates to question (Q3). To evaluate the sensitivity of our method to α, we conduct experiments by varying its value and observing the ef￾fect on final performance, as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Full bad dataset size effect. SafeDICE and DWBC-GB do not have version that learn from [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Different of good dataset size without impact from bad dataset in MuJoCo Locomotion [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Different of good dataset size without impact from bad dataset in Adroit Manipulation [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]
Figure 6
Figure 6. Figure 6: AW-BC and QW-BC comparison [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: In the Mujoco locomotion tasks, increasing the quality of the unlabeled dataset has minimal effect on SafeDICE and ILID, and both methods continue to underperform on the Adroit hand manipulation tasks regardless of the number of expert trajectories included. In contras…
Figure 7
Figure 7. Figure 7: Effect of Unlabeled Dataset Quality on Performance: We evaluate the effect of increasing the number of expert trajectories in the unlabeled dataset B MIX. The results are calculated from 5 different training seeds, reported in normalized score. Our method outperforms S…
Figure 8
Figure 8. Figure 8: Performance of large α ≥ 1. D.7 Comparison Between L(Q, π) and the Surrogate Le(Q, π) As shown in Proposition 4.3, the original objective L(Q | V ) (Equation (3)) is transformed into a modified version Le(Q | V ) (Equation (6)). This experiment investigates the perform…
Figure 9
Figure 9. Figure 9: Exponetial ablation study. D.8 Sensitivity Analysis of β In this section, we explore how different values of the β parameter affect performance. The experiment results are provided in [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 24 canonical work pages

  1. [1]

    Learning from negative feedback, or positive feedback or both

    Abbas Abdolmaleki, Bilal Piot, Bobak Shahriari, Jost Tobias Springenberg, Tim Hertweck, Michael Bloesch, Rishabh Joshi, Thomas Lampe, Junhyuk Oh, Nicolas Heess, Jonas Buchli, and Martin Riedmiller. Learning from negative feedback, or positive feedback or both. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  2. [2]

    Ls-iq: Implicit reward regularization for inverse reinforcement learning

    Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. InEleventh International Conference on Learning Representations (ICLR), 2023

    work page 2023

  3. [3]

    Non-adversarial imitation learning and its connections to adversarial methods.arXiv preprint arXiv:2008.03525, 2020

    Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods.arXiv preprint arXiv:2008.03525, 2020

    arXiv 2008

  4. [4]

    Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations

    Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. InInternational conference on machine learning, pages 783–792. PMLR, 2019

    work page 2019

  5. [5]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  6. [6]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020

    2020

  7. [7]

    Learning robust rewards with adverserial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. InInternational Conference on Learning Representations, 2018

    2018

  8. [8]

    Iq-learn: Inverse soft-q learning for imitation.Advances in Neural Information Processing Systems, 34:4028–4039, 2021

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation.Advances in Neural Information Processing Systems, 34:4028–4039, 2021

    work page 2021

Show all 44 references
  1. [9]

    Extreme q-learning: Maxent rl without entropy

    Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  2. [10]

    Offline safe reinforcement learning using trajectory classification

    Ze Gong, Akshat Kumar, and Pradeep Varakantham. Offline safe reinforcement learning using trajectory classification. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 16880–16887, 2025

    2025

  3. [11]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    2014

  4. [12]

    Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  5. [13]

    Inverse preference learning: Preference-based rl without a reward function.Advances in Neural Information Processing Systems, 36, 2024

    Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function.Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [14]

    Generative adversarial imitation learning.Advances in neural information processing systems, 29, 2016

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning.Advances in neural information processing systems, 29, 2016

    2016

  7. [15]

    Imitate the good and avoid the bad: An incremental approach to safe reinforcement learning

    Huy Hoang, Tien Mai, and Pradeep Varakantham. Imitate the good and avoid the bad: An incremental approach to safe reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12439–12447, 2024. 10

    2024

  8. [16]

    SPRINQL: Sub-optimal demonstrations driven offline imitation learning

    Huy Hoang, Tien Anh Mai, and Pradeep Varakantham. SPRINQL: Sub-optimal demonstrations driven offline imitation learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  9. [17]

    Safedice: offline safe imitation learning with non-preferred demonstra- tions.Advances in Neural Information Processing Systems, 36, 2024

    Youngsoo Jang, Geon-Hyeong Kim, Jongmin Lee, Sungryull Sohn, Byoungjip Kim, Honglak Lee, and Moontae Lee. Safedice: offline safe imitation learning with non-preferred demonstra- tions.Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    Beyond reward: Offline preference-guided policy optimization

    Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. InInternational Conference on Machine Learning, pages 15753–15768. PMLR, 2023

    2023

  11. [19]

    Preference transformer: Modeling human preferences using transformers for rl

    Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  12. [20]

    Lobs- dice: Offline learning from observation via stationary distribution correction estimation.Ad- vances in Neural Information Processing Systems, 35:8252–8264, 2022

    Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobs- dice: Offline learning from observation via stationary distribution correction estimation.Ad- vances in Neural Information Processing Systems, 35:8252–8264, 2022

    2022

  13. [21]

    Demodice: Offline imitation learning with supplementary imperfect demonstrations

    Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. InInternational Conference on Learning Representations, 2021

    2021

  14. [22]

    Imitation learning via off-policy distribu- tion matching

    Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. InInternational Conference on Learning Representations, 2020

    2020

  15. [23]

    Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  16. [24]

    Optidice: Offline policy optimization via stationary distribution correction estimation

    Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. InInternational Conference on Machine Learning, pages 6120–6130. PMLR, 2021

    2021

  17. [25]

    Imitation learning from imperfection: Theoretical justifications and algorithms

    Ziniu Li, Tian Xu, Zeyu Qin, Yang Yu, and Zhi-Quan Luo. Imitation learning from imperfection: Theoretical justifications and algorithms. InAdvances in Neural Information Processing Systems 37, 2023

    2023

  18. [26]

    Semantic loss guided data efficient supervised fine tuning for safe responses in LLMs

    Yuxiao Lu, Arunesh Sinha, and Pradeep Varakantham. Semantic loss guided data efficient supervised fine tuning for safe responses in LLMs. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  19. [27]

    Versatile offline imitation from observations and examples via regularized state-occupancy matching

    Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. InInternational Conference on Machine Learning, pages 14639–14663. PMLR, 2022

    2022

  20. [28]

    ODICE: Revealing the mystery of distribution correction estimation via orthogonal-gradient update

    Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. ODICE: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  21. [29]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  22. [30]

    Learning multimodal rewards from rankings

    Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. InConference on robot learning, pages 342–352. PMLR, 2022

    2022

  23. [31]

    Al- gaedice: Policy gradient from arbitrary experience.arXiv preprint arXiv:1912.02074, 2019

    Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience.arXiv preprint arXiv:1912.02074, 2019

    1912 arXiv

  24. [32]

    John Wiley & Sons, 2014

    Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  25. [33]

    Sqil: Imitation learning via reinforcement learning with sparse rewards.arXiv preprint arXiv:1905.11108, 2019

    Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards.arXiv preprint arXiv:1905.11108, 2019. 11

    1905 arXiv

  26. [34]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Co...

    2011

  27. [35]

    Dual rl: Unification and new methods for reinforcement and imitation learning

    Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. InProceedings of the 12th International Conference on Learning Representations (ICLR), 2024

    2024

  28. [36]

    Value- decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296, 2017

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296, 2017

    2017 arXiv

  29. [37]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018

    2018

  30. [38]

    Behavioral cloning from observation.arXiv preprint arXiv:1805.01954, 2018

    Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation.arXiv preprint arXiv:1805.01954, 2018

    2018 arXiv

  31. [39]

    Imitation learning from imperfect demonstration

    Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, V oot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. InInternational Conference on Machine Learning, pages 6818–6827. PMLR, 2019

    2019

  32. [40]

    Discriminator-weighted offline imitation learning from suboptimal demonstrations

    Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. InProceedings of the 39th International Conference on Machine Learning, pages 24725–24742, 2022

    2022

  33. [41]

    How to leverage diverse demonstrations in offline imitation learning

    Sheng Yue, Jiani Liu, Xingyuan Hua, Ju Ren, Sen Lin, Junshan Zhang, and Yaoxue Zhang. How to leverage diverse demonstrations in offline imitation learning. InForty-first International Conference on Machine Learning, 2024

    2024

  34. [42]

    Confidence-aware imitation learning from demonstrations with varying optimality.Advances in Neural Information Pro- cessing Systems, 34:12340–12350, 2021

    Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality.Advances in Neural Information Pro- cessing Systems, 34:12340–12350, 2021

    2021

  35. [43]

    Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems XIX, 2023

    Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems XIX, 2023

    2023

  36. [44]

    The ingredients of real world robotic reinforcement learning

    Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations, 2020. 12 Appendix A Missing Proofs Proposi...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.