Pith. sign in

REVIEW 4 major objections 5 minor 56 references

SR-Reward: Taking The Path More Traveled

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The L2 norm of a successor representation learned from expert demonstrations can serve as the reward signal for offline RL, matching the performance of algorithms that use the true reward.

desk verdict Solid empirical offline IRL paper with a useful reward module, but the SR-norm proxy is oversold and the evaluation has a few selection/tuning soft spots. read the letter →

arxiv 2501.02330 v3 pith:EILILBS3 submitted 2025-01-04 cs.LG cs.AI

Seyed Mahdi B. Azad , Zahra Padar , Gabriel Kalweit , Joschka Boedecker This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearninginversesuccessorrepresentationrewardimitationnegativesamplingD4RLbenchmarkbehavioralcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SR-Reward is a way to manufacture a reward signal for offline reinforcement learning from expert demonstrations alone, with no true reward and no environment interaction. The method learns a successor representation of the state-action pairs in the demonstrations and uses its $\ell^2$-norm as the reward: frequently visited pairs get higher rewards. The paper argues this reward can be trained concurrently with ordinary TD-based offline RL algorithms and reports that on D4RL locomotion, Adroit hand, and ManiSkill2 tasks it matches offline RL with the true reward and often beats behavioral cloning. A negative-sampling term suppresses overestimated rewards for out-of-distribution states near the demonstrations, keeping the learned policy conservative. The stated goal is to make offline RL usable whenever demonstrations are easier to obtain than hand-engineered rewards.

What carries the argument

Successor representation (SR) — a vector whose elements estimate the expected discounted future occupancy of state-action features — is learned from demonstrations using a Bellman target $\phi(s,a) + \gamma M(s',a')$, with $\phi(s,a)$ the concatenation of a learned state encoder output and the action. The reward is the norm $\|M(s,a)\|_2$, which the paper treats as a visitation count of the expert. Two stabilizers carry the method: a magnitude-loss cap near 1, and negative sampling that perturbs demonstration pairs with Gaussian noise and regresses their reward down by an exponential distance kernel $\exp\left(-\|\phi(s,a)-\phi(\tilde{s},\tilde{a})\|_2/\sigma^2\right)$, injecting conservatism for out-of-distribution data.

What would settle it

Build an offline dataset in which expert trajectories contain a frequently visited but task-irrelevant region, such as a corridor the expert passes through many times before reaching the goal, and compare SR-Reward-trained policies against true-reward policies; if the SR-Reward agent lingers in the incidental region and its success rate drops, the visitation-maximization assumption fails. The paper's own online HalfCheetah experiment, where SR-Reward-trained TD3 underperforms true-reward TD3 after exploring out-of-distribution states, already points to this boundary.

Watch

Extended reading notes

Core claim

The central claim is that the $\ell^2$-norm of an action-augmented successor representation learned offline from expert demonstrations is itself a usable reward for offline RL: maximizing it over the learned policy amounts to maximizing the expert's state-action visitation, and this surrogate objective yields policies competitive with those trained on the environment's true reward. The paper estimates successor features via the Bellman equation, concatenating learned state features with the action, and converts the vector to a scalar by its norm. Because the reward is decoupled from the policy, it can be trained with TD-based RL algorithms without adversarial optimization, and it is bounded by a magnitude penalty and made conservative by negative sampling with a Gaussian decay kernel. The paper supports this with experiments on D4RL and ManiSkill2, including data-size and data-quality ablations, and shows that the policy trained from SR-Reward acts similarly to one trained from the true reward.

Load-bearing premise

The reward proxy assumes that maximizing how often the expert's states and actions are re-visited is the same as solving the task, even though the paper gives no proof of this equivalence.

Editorial extensions

If this is right

  • Offline RL can be applied to demonstration datasets that contain no reward annotations, as in the ManiSkill2 tasks, without changing the underlying RL algorithm.
  • Any TD-based offline RL algorithm can, in principle, substitute its reward stream with SR-Reward, since the reward module trains on the same replay buffer with only the next action added.
  • Because the reward carries a conservative bias from negative sampling, learned policies stay near the demonstrated behavior, which is beneficial when consequences far from the data are unknown.
  • The reward remains informative when demonstrations are few or mixed with suboptimal data, since performance degrades at the same rate as with the true reward in the ablation studies.
  • SR-Reward outperforms behavioral cloning on harder manipulation tasks, suggesting that bootstrapping over future states helps where copying actions does not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to learn SR-Reward from two datasets, expert and non-expert, and use the ratio of SR norms as a distribution-matching reward; the paper's appendix already establishes SR as a proxy for occupancy measure, so this would bypass adversarial discriminators.
  • The visitation-count interpretation predicts that demonstrations padded with redundant, task-irrelevant loops will distort the reward landscape; an ablation that deliberately adds such loops would test whether SR-Reward can ignore incidental high-frequency states.
  • The authors' online HalfCheetah result implies SR-Reward's utility is bounded by how well the demonstration coverage matches the region the learner must explore; a testable extension is to combine SR-Reward with an explicit uncertainty penalty for far out-of-distribution states.
  • Because the reward is a function of state and action only, it could be transferred across agents with different dynamics, but the SR must be relearned whenever the transition dynamics change; this transfer condition is not examined in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SR-Reward, a reward function learned offline from expert demonstrations using a successor representation (SR). The SR vector is trained via a Bellman equation with an auxiliary next-feature prediction task, and the L2 norm of the SR vector is used as the reward. A negative-sampling loss lowers rewards for states and actions perturbed away from the demonstrations. The method is evaluated on D4RL (MuJoCo and Adroit) and ManiSkill2 tasks, combined with f-DVL and SparseQL, and compared against behavioral cloning and offline RL with true rewards. The paper reports competitive or better performance for SR-Reward in most settings and includes ablation studies on data size, data quality, and negative sampling.

Significance. If the empirical findings are robust, the paper offers a simple, non-adversarial reward-learning module that can be plugged into existing offline RL algorithms without requiring environment rewards. The code is released and the approach is easy to reproduce, which strengthens its practical value. However, the central theoretical motivation for the reward function is not established, and the empirical evaluation leaves open important alternative explanations for the reported success.

major comments (4)
  1. [Section 3.2 and Appendix C] The claim that ||M(s,a)||_2 represents how often the expert has visited (s,a) is not supported by the derivation in Appendix C. Appendix C shows rho(s') = sum_s p(s) M(s,s'), which is an expectation of M over starting states, not a per-state equivalence. In an episodic task, M(s,a) is the expected discounted future feature stream from (s,a), which is typically largest at episode onset and decreases near success. The resulting reward is therefore anti-shaped: it incentivizes the agent to postpone termination rather than to reproduce the expert's behavior. This undermines the paper's core motivation, and the empirical results do not rule out the anti-shaping concern because the paper does not analyze reward values along trajectories in the episodic ManiSkill2 tasks. The authors should either provide a rigorous argument (or empirical evidence) that the reward is not anti-shaped in these tasks, or re-frame the contribution and add a control experiment with an intentionally anti-shaped reward to show that the reported gains are not simply an artifact of the offline RL algorithm's conservatism.
  2. [Section 4.1 and Appendix G] The negative-sampling hyperparameters (beta and sigma) were selected using a grid search on StackCube, as shown in Appendix G, and the selected configuration is used for the main StackCube result in Table 1. This means the reported StackCube performance is an in-sample optimum rather than an unbiased estimate of the method's performance under a fixed hyperparameter choice. The paper should report results with a separate validation split, or at least provide sensitivity analyses for all benchmark environments, to allow a fair assessment of the method's robustness.
  3. [Section 4.1] The evaluation protocol saves the checkpoint with the highest mean return over 25 evaluation rollouts during training and then reports that checkpoint's performance on 50 fresh rollouts. This is a best-of-training checkpoint selection that can inflate reported performance relative to a fixed training budget. The concern is especially relevant for the ManiSkill2 comparisons against behavioral cloning, which does not use such a selection protocol. The authors should also report the performance of the final checkpoint or an average over the last few checkpoints, or otherwise justify that the selection protocol does not alter the main conclusions.
  4. [Section 7] The Limitations section explicitly states that no convergence or optimality analysis is provided for using the L2 norm as a reward. Combined with the anti-shaping property discussed above, the empirical parity in Table 1 does not distinguish a well-designed reward proxy from a scenario where the offline dataset already contains enough signal for imitation. The paper would be substantially strengthened by a direct investigation of what the learned reward actually encodes, e.g., by plotting SR-Reward values along successful and unsuccessful trajectories in an episodic task.
minor comments (5)
  1. [Section 5] The text refers to "covariant shift" in the discussion of behavioral cloning; this should be "covariate shift."
  2. [Section 3.3] The notation N(0, beta) uses beta as the standard deviation of the Gaussian noise; the standard convention would be N(0, beta^2) if beta is the variance, so the authors should clarify the exact parameterization.
  3. [Algorithm 1] Line 22 builds batches by concatenating negative samples into the replay buffer for the RL update; this procedure is described in the text but the pseudocode is a little terse. A brief explanatory sentence in the caption would improve readability.
  4. [Appendix F] The heading contains a typo: "Ardoit Door" should be "Adroit Door."
  5. [Table 1] The table header groups f-DVL and SparseQL columns but the grouping is not visually obvious; it would be clearer to repeat the algorithm names above the True Reward and SR-Reward subcolumns.

Circularity Check

1 steps flagged · score 2.0 of 10

Core SR-Reward derivation is self-contained; minor circularity from tuning negative-sampling hyperparameters on the StackCube benchmark that is then reported in Table 1.

  1. fitted input called prediction [Appendix G (Figure 11) and Table 1]
    "To study the sensitivity of our method to these hyperparameters, we trained a series of SparseQL + SR-Reward models on the StackCube environment using a grid of different β and σ values. Figure 11 presents the normalized returns across these settings. The red circle marks the hyperparameter configuration used in our main experiments (Section 4)."

    Table 1 reports StackCube as a benchmark result for SparseQL+SR-Reward (70.71±12.55). Appendix G shows that the negative-sampling hyperparameters β and σ used in the main experiments were selected by running a grid on the StackCube environment and picking the configuration marked by the red circle. Therefore the StackCube row is not an out-of-sample prediction; it is the result of fitting two hyperparameters to that environment. The performance of SR-Reward on StackCube is thus partially forced by construction, and the comparison to BC on that task is not fully independent. The remainder of Table 1 (MuJoCo, Adroit, PickCube, TurnFaucet) and the external true-reward baselines are not subject to this particular tuning, so the central claim retains independent empirical content.

full rationale

The central derivation is not circular. SR-Reward is defined from demonstrations via the successor-representation Bellman equation (Section 2.2, 3.2), and its quality is evaluated against external task returns, BC, and true-reward offline RL baselines in Table 1. The relationship between occupancy measures and SR (Appendix C) provides the motivation for using the SR norm as a reward proxy; the paper does not claim that maximizing this norm is equivalent to task success, and the Limitations section explicitly concedes that no convergence or optimality analysis is provided, relying instead on empirical validation. No load-bearing self-citation is present: the key external premises are Machado et al. (2020), Luo et al. (2020), and the D4RL/ManiSkill2 benchmarks. The only concrete circular element is that the negative-sampling hyperparameters β and σ were selected using StackCube, which is also reported as a headline result in Table 1. This makes the StackCube row partially fitted rather than predicted, but the effect is localized to one environment; the MuJoCo, Adroit, and other ManiSkill2 results remain independent evidence. Overall circularity is therefore mild.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

SR-Reward introduces no new physical entities; it reuses successor representation. The load-bearing assumptions are that expert visitation is a valid reward proxy and that the negative sampling kernel is a reasonable prior. Both are stated in the paper and neither is derived from first principles.

free parameters (4)
  • Negative sampling noise beta = 1.0 (MuJoCo), 0.1 (Adroit), 0.03 (Maniskill2)
    Standard deviation of Gaussian perturbation for negative samples (Table 2). Tuned via grid search on StackCube (Appendix G).
  • Negative sampling decay sigma = 3.0 (MuJoCo), 0.3 (Adroit), 0.3 (Maniskill2)
    Width of exponential decay kernel in Equation 2. Tuned with beta on StackCube (Appendix G).
  • Reward magnitude cap = 1.0
    Soft upper bound for the SR-norm reward used in LMagnitude loss (Section 3.4).
  • SR pretraining steps = 10000
    Warm-start steps for SR networks before RL training (Section 3.4).
assumptions (5)
  • domain assumption The dataset is generated by an expert (or at least high-quality) policy.
    The paper states the goal is to imitate expert behavior; in practice D4RL 'expert' datasets are used. Suboptimal demonstrations degrade performance (Section 7, Figure 9).
  • domain assumption The transition dynamics are Markovian and the offline dataset provides sufficient coverage to learn successor representations via TD.
    The paper relies on standard RL assumptions and trains SR with Bellman error on the dataset (Section 3.4).
  • standard math The L2 norm of the successor representation is a valid proxy for state-action visitation count.
    Machado et al. (2020) show the norm of SR counts visitation, but the extension to using it as a reward for offline RL is this paper's assumption.
  • ad hoc to paper Maximizing the expert's expected visitation count (SR norm) produces a policy that solves the task.
    This is the central reward-design assumption, stated in Section 3.2 ('we set out to find a policy that maximizes the state-action visitation of the expert') and not proven; the Limitations section acknowledges no theoretical convergence or optimality results.
  • ad hoc to paper Negative samples, created by Gaussian perturbation of expert states and actions, should receive lower rewards according to an exponential distance kernel.
    Introduced in Section 3.3 to combat overestimation; the specific functional form (Equation 2) is a design choice, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SR-Reward: Taking The Path More Traveled." pith.science (2026). https://pith.science/paper/EILILBS3

@misc{pith2026250102330,
  author       = {Pith},
  title        = {Pith review of: SR-Reward: Taking The Path More Traveled},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EILILBS3}},
  note         = {Machine review of arXiv:2501.02330}
}
read the original abstract

In this paper, we propose a novel method for learning reward functions directly from offline demonstrations. Unlike traditional inverse reinforcement learning (IRL), our approach decouples the reward function from the learner's policy, eliminating the adversarial interaction typically required between the two. This results in a more stable and efficient training process. Our reward function, called \textit{SR-Reward}, leverages successor representation (SR) to encode a state based on expected future states' visitation under the demonstration policy and transition dynamics. By utilizing the Bellman equation, SR-Reward can be learned concurrently with most reinforcement learning (RL) algorithms without altering the existing training pipeline. We also introduce a negative sampling strategy to mitigate overestimation errors by reducing rewards for out-of-distribution data, thereby enhancing robustness. This strategy inherently introduces a conservative bias into RL algorithms that employ the learned reward. We evaluate our method on the D4RL benchmark, achieving competitive results compared to offline RL algorithms with access to true rewards and imitation learning (IL) techniques like behavioral cloning. Moreover, our ablation studies on data size and quality reveal the advantages and limitations of SR-Reward as a proxy for true rewards.

Figures

Figures reproduced from arXiv: 2501.02330 by the authors.

Figure 1
Figure 1. Standard offline RL training requires rewards to be present in the demonstrations (Left). SR [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The architecture of the SR networks. The output of the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Environments used for our experiments. From left to right: 2D Toy Maze, MuJoCo environments: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Mean return of corrupted expert trajectories for Relocate envi￾ronment. Negative sampling signifi￾cantly reduces the reward for states and actions further away from the ex￾pert demonstrations. Neural networks tend to overestimate the value of out-of-distribution data p…
Figure 5
Figure 5. Figure 5: The plots show the effect using negative sampling for a 2D Toy Maze environment (Figure 3)with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: PickCube (Left), StackCube (Middle) and TurnFaucet (Right) from Maniskill2 (Gu et al., 2023) For empirical validation, we utilize the widely-used MuJoCo-based (Todorov et al., 2012) environments for locomotion tasks, and the Adroit hand (Rajeswaran et al., 2018) enviro…
Figure 7
Figure 7. Figure 7: Performance on Maniskill2 environments. Turning the faucet requires a continuous connection [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Effect of data size on performance. RL agents (SparseQL) using SR-Reward show competitive [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Effect of data quality on performance. RL agents (SparseQL) using SR-Reward show similar [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Action trajectories for f-DVL model trained on true reward vs SR-Reward on the Ardoit Door [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Normalized return for SparseQL+SR-Reward for different hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: (TOP): Values of every dimension of observation (blue) and actions (orange) for 50 demonstrations [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Normalized reward of TD3 algorithm trained using true reward and SR-Reward. [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 36 canonical work pages

  1. [1]

    Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML '04: Proceedings of the twenty-first international conference on Machine learning. ACM, 2004

    work page 2004

  2. [2]

    Holo-dex: Teaching dexterity with immersive mixed reality

    Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, and Lerrel Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. arXiv preprint arXiv:2210.06463, 2022

    arXiv 2022

  3. [3]

    Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine

    Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Rese...

    work page 2023

  4. [4]

    Successor features for transfer in reinforcement learning

    Andr \'e Barreto, Will Dabney, R \'e mi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017

    work page 2017

  5. [5]

    Mankowitz, Hado van Hasselt, R \' e mi Munos, David Silver, and Tom Schaul

    Diana Borsa, Andr \' e Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, R \' e mi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

    work page 2019

  6. [6]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2334029

    arXiv 1952

  7. [7]

    Successor feature sets: Generalizing successor representations across policies

    Kiant \'e Brantley, Soroush Mehri, and Geoff J Gordon. Successor feature sets: Generalizing successor representations across policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 11774--11781, 2021

    work page 2021

  8. [8]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, February 2023. URL http://arxiv.org/abs/1706.03741

    arXiv 2023

Show all 56 references
  1. [9]

    Improving generalization for temporal difference learning: The successor representation

    Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Comput., 5 0 (4): 0 613--624, 1993

    1993

  2. [10]

    Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  3. [11]

    Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learning

    Angelos Filos, Clare Lyle, Yarin Gal, Sergey Levine, Natasha Jaques, and Gregory Farquhar. Psiphi-learning: Reinforcement learning with demonstrations using successor features and inverse temporal difference learning. In International Conference on Machine Learning, pp.\ 3305-...

    2021

  4. [12]

    Learning robust rewards with adverserial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, Conference Track Proceedings . OpenReview.net, 2018

    2018

  5. [13]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020

    2020

  6. [14]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Swe...

    2018

  7. [15]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA , volu...

    2019

  8. [16]

    For sale: State-action representation learning for deep reinforcement learning

    Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  9. [17]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Proce...

    2021

  10. [18]

    Extreme q-learning: Maxent RL without entropy

    Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, 2023 . OpenReview.net, 2023

    2023

  11. [19]

    A divergence minimization perspective on imitation learning methods, 2019

    Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods, 2019

    2019

  12. [20]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406.2661, 2014

    2014 arXiv

  13. [21]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Confere...

    2023

  14. [22]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. NIPS'16, pp.\ 4572–4580, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819

    2016

  15. [23]

    Revisiting successor features for inverse reinforcement learning

    Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, and Sanjiban Choudhury. Revisiting successor features for inverse reinforcement learning. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. URL https://openreview.net/forum?i...

    2024

  16. [24]

    Deep inverse q-learning with constraints

    Gabriel Kalweit, Maria H \" u gle, Moritz Werling, and Joschka Boedecker. Deep inverse q-learning with constraints. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virt...

    2020

  17. [25]

    Imitation learning as f -divergence minimization, 2020

    Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f -divergence minimization, 2020

    2020

  18. [26]

    Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning

    Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In 7th International Conference on Learning Representations, ICLR 2019, ...

    2019

  19. [27]

    Imitation learning via off-policy distribution matching

    Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum...

    2020

  20. [28]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=68n2s9ZJWF8

    2022

  21. [29]

    Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J

    Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J. Gershman. Deep successor reinforcement learning. CoRR, abs/1606.02396, 2016

    2016 arXiv

  22. [30]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN ...

    2020

  23. [31]

    Batch Reinforcement Learning, pp.\ 45--73

    Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pp.\ 45--73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27645-3. doi:10.1007/978-3-642-27645-3_2

    2012 doi

  24. [32]

    Energy-based imitation learning

    Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp.\ 809–817, Richland, SC, 2021. International Foundation for Autonomous Agents a...

    2021

  25. [33]

    Learning self-correctable policies and value functions from demonstrations with negative sampling

    Yuping Luo, Huazhe Xu, and Tengyu Ma. Learning self-correctable policies and value functions from demonstrations with negative sampling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020

    2020

  26. [34]

    Count-based exploration with the successor representation

    Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 5125--5133, 2020

    2020

  27. [35]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015 doi

  28. [36]

    A first-occupancy representation for reinforcement learning

    Ted Moskovitz, Spencer R Wilson, and Maneesh Sahani. A first-occupancy representation for reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JBAZe2yN6Ub

    2022

  29. [37]

    Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections, 2019

    Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections, 2019

    2019

  30. [38]

    Ng and Stuart Russell

    Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Pat Langley (ed.), Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000 , pp.\ 663--670. Morgan...

    2000

  31. [39]

    Bridging state and history representations: Understanding self-predictive rl, 2024

    Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history representations: Understanding self-predictive rl, 2024

    2024

  32. [40]

    Efficient training of artificial neural networks for autonomous navigation

    Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Comput., 3 0 (1): 0 88--97, 1991. doi:10.1162/NECO.1991.3.1.88

    1991 doi

  33. [41]

    Puterman

    Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779

    1994

  34. [42]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv...

    2023 arXiv

  35. [43]

    Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations . In Proceedings of Robotics: Science and Systems (RSS), 2018

    2018

  36. [44]

    A motion retargeting method for effective mimicry-based teleoperation of robot arms

    Daniel Rakita, Bilge Mutlu, and Michael Gleicher. A motion retargeting method for effective mimicry-based teleoperation of robot arms. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pp.\ 361--370, 2017

    2017

  37. [45]

    Dragan, and Sergey Levine

    Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: imitation learning via reinforcement learning with sparse rewards. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

    2020

  38. [46]

    Gordon, and J

    St \' e phane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011.0686, 2010

    2010 arXiv

  39. [47]

    Dual rl: Unification and new methods for reinforcement and imitation learning, 2023

    Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning, 2023

    2023

  40. [48]

    Lewis, and A

    Satinder Singh, R. Lewis, and A. Barto. Where do rewards come from? 01 2009

    2009

  41. [49]

    Issues in using function approximation for reinforcement learning

    Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. 1999. URL https://api.semanticscholar.org/CorpusID:1115058

    1999

  42. [50]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  43. [51]

    Munchausen reinforcement learning

    Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 4235--4246, 2020

    2020

  44. [52]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. Conference on Robot Learning, 2022

    2022

  45. [53]

    Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023

    Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023

    2023

  46. [54]

    Offline rl with no ood actions: In-sample learning via implicit value regularization

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023

    2023

  47. [55]

    Deep reinforcement learning with successor features for navigation across similar environments

    Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 2371--2378....

    2017

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.