Pith. sign in

REVIEW 3 major objections 6 minor 61 references

Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A video labeling model trained on three self-supervised tasks can clone policies from action-free videos using only 100k interactions.

desk verdict A solid empirical extension in video-based policy cloning, but the headline 12/16 superiority count is not backed by the paper's own Table I (strict count is 10/16). read the letter →

arxiv 2412.10778 v2 pith:AKCUZQYZ submitted 2024-12-14 cs.CV cs.AI

Xin Liu , Yaran Chen , Haoran Li This is my paper · ORCID
classification cs.CVcs.AI
keywords unsupervisedpolicylearningfromvideosbehaviorcloningobservationself-supervisedinversedynamicsmodelaction-freesample-efficientreinforcementprocedurallygeneratedenvironments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a policy can be learned from action-free expert videos, with no rewards and no action labels, if a video labeling model is trained jointly on three self-supervised tasks that use both the videos and a small number of reward-free interactions. The labeling model infers the expert action between neighboring video frames as a latent action, and a policy is obtained by behavior cloning those inferred actions. The authors report that this pipeline, called UPESV, beats five established baselines on 12 of 16 procedurally generated video-game tasks when every method is limited to 100,000 environment interactions. The broader point is that easily available video demonstrations could substitute for expensive expert supervision in sample-limited settings.

What carries the argument

Three self-supervised tasks carried by a shared inverse-dynamics labeling model: visual shift contrast (VSC), latent future reconstruction (LFR), and ground-truth action prediction (GAP). VSC applies a contrastive InfoNCE loss to positive pairs formed by two shifted crops of the same frame, teaching the encoder $f$ to ignore absolute positions and keep relative motion; LFR runs the predicted latent action through a discrete world model that must reconstruct the next latent observation, teaching $g$ to produce actions that explain observed changes; GAP minimizes cross-entropy between predicted actions and true actions on reward-free transitions, anchoring the latent action space to the real action space. The same encoder and action projector are reused by the cloned policy, so the labeling model and policy improve in alternation: the policy clones video-derived latent actions, interacts with the environment, and returns transitions that strengthen GAP.

What would settle it

Run UPESV with held-out true action labels for expert videos on all sixteen tasks and check whether the labeling model's action-prediction accuracy on those expert videos is above chance and correlates with the cloned policy's score; if accuracy is near chance on tasks where the policy fails, the cross-distribution transfer from non-expert transitions to expert videos is not holding.

Watch

Extended reading notes

Core claim

UPESV's central object is the video labeling model $V(a_i^v \mid o_i^v, o_{i+1}^v)$, an inverse dynamics model that maps neighboring video frames to a latent action via $a_i^v = h(g(f(o_i^v), f(o_{i+1}^v)))$, where $f$ is a feature encoder, $g$ a latent predictor, and $h$ an action projector. The model is trained with three objectives: a visual shift contrast task that aligns two randomly shifted versions of the same image in latent space, forcing the encoder to attend to relative rather than absolute visual differences; a latent future reconstruction task that trains a world model to reconstruct the next latent observation from the current one plus the predicted latent action, forcing the labeling model to understand environmental dynamics; and a ground-truth action prediction task on reward-free transitions, which aligns predicted actions to the true action space and acts as a cross-distribution generalization signal. The policy shares the encoder and action projector with the labeling model and is cloned by matching its latent action to the video-derived latent action; as the policy improves, it collects new reward-free transitions that enrich the ground-truth action prediction data. On sixteen procedurally generated environments with 100k interaction steps, the authors report state-of-the-art results, outperforming five baselines on 12/16 tasks and producing the only effective policy on the Fruitbot task.

Load-bearing premise

The load-bearing premise is that the labeling model, trained to predict actions on non-expert reward-free transitions, also predicts the true actions in expert videos well enough that behavior cloning the resulting labels yields a good policy.

Editorial extensions

If this is right

  • If the central claim holds, action-free videos become a usable supervision source for visually rich, procedurally varied environments, reducing reliance on expert action labels and hand-designed rewards.
  • The three-task combination should beat methods that use only one data source: using only non-expert transitions for labeling, or only expert videos for dynamics, leaves information on the table.
  • Because the policy shares the encoder and action projector with the labeling model, improvements to video labeling directly translate into policy improvements without extra parameters.
  • The reported held-out action-prediction accuracy on two unseen expert datasets suggests the labeling model transfers to new expert videos, implying that the pipeline does not merely memorize its training videos.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is measuring labeling accuracy on held-out expert videos for all sixteen tasks, not just two, to check whether per-task policy performance tracks per-task labeling accuracy.
  • The iterative labeling-and-cloning loop is a form of pseudo-labeling, so a natural concern the paper leaves implicit is confirmation bias: if early labeling errors are consistent, the policy may reinforce them when collecting new transitions.
  • Because the visual shift contrast task helped mainly by muting absolute-position differences, the same relative-difference prior could be injected into other inverse dynamics or world-model learners, not only this architecture.
  • The paper's own limitation section notes the gap to continuous control and real robots; a direct next step would be testing UPESV on continuous-action domains, where the discrete latent action and the shared action projector would be stressed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes UPESV, a framework for learning policies from action-free expert videos without rewards or expert action labels. A video labeling model is trained with three self-supervised objectives: visual shift contrast (VSC), latent future reconstruction (LFR), and ground-truth action prediction (GAP) on reward-free transitions collected by the policy itself. The labeling model's predicted latent actions on expert videos are used as targets for behavior cloning a policy, which in turn collects more transitions for GAP. Experiments on 16 Procgen environments with 100k interactions compare against BCO, ILPO, LAPO, PPO, and TVL, and the paper claims state-of-the-art performance on 12/16 tasks.

Significance. If substantiated, the paper would provide a practical recipe for leveraging large action-free video corpora with only a small number of environment interactions, combining representation learning, dynamics modeling, and inverse action prediction. The experimental study is broad (16 tasks, ablations, hyperparameter sensitivity, and a robustness check on two unseen expert datasets), and the method achieves the highest mean score (9.0 vs 6.8 for the runner-up LAPO). These are concrete strengths. However, the headline win count is not reproducible from the paper's own Table I, and the direct evidence for the labeling model's cross-distribution generalization is limited to two tasks. The core idea is sound and the results are positive, but the claims need to be calibrated and the evidence for the mechanism strengthened.

major comments (3)
  1. [Section IV.B, Table I, Abstract] The claim that UPESV 'outperforms five current advanced baselines on 12/16 tasks' is not supported by strict pairwise comparison of the mean scores reported in Table I. Under the standard interpretation of 'outperforms' as strictly higher mean score, UPESV beats every baseline on only 10 tasks (Bigfish, Maze, Coinrun, Dodgeball, Climber, Fruitbot, Ninja, Miner, Leaper, Bossfight). On Heist, UPESV ties LAPO (9.4±0.3 vs 9.4±0.3); on Chaser, it ties BCO and LAPO (0.8±0.1 vs 0.8±0.0 and 0.8±0.0); and on Plunder, Jumper, Starpilot, and Caveflyer, UPESV is strictly worse than at least one baseline. The same issue affects the Section IV.B statement about three video-based baselines: strict comparison gives 10/16, not 12/16. Please correct the count in the abstract, introduction, and Section IV.B, or explicitly define the metric (e.g., 'best or tied-best on 12/16 tasks').
  2. [Section III.D, Table II] The core mechanism of UPESV is that the video labeling model V, trained partly on ground-truth actions from non-expert reward-free transitions (GAP), generalizes to label expert videos correctly, and that the cloned policy (Eq. 6) inherits this accuracy. The only direct evidence for this cross-distribution transfer is Table II, which reports action prediction accuracy on just two unseen expert datasets (Starpilot and Miner). Because the policy's success on the remaining 14 tasks depends on V's labeling quality on expert videos for those tasks, the paper should either report labeling accuracy on expert videos for a broader set of tasks or provide a clear argument (e.g., with additional analysis) that the Task I policy results already validate the labeling model on all tasks.
  3. [Eqs. (3) and (5)] The written objectives for the two standard losses do not match their descriptions. Eq. (3) is supposed to be an InfoNCE loss, but as printed it lacks the exponential and temperature scaling and sums over all j including the positive pair, which does not correspond to any standard contrastive loss. Eq. (5) is described as a cross-entropy between predicted and ground-truth action vectors, but the arguments of the logarithm are reversed: the loss should be -Σ a^e_{i,(c)} log a^{e,V}_{i,(c)}. If these are typesetting errors, they should be corrected, because as written the equations would not train the model as intended.
minor comments (6)
  1. [Section IV.D, Fig. 4] The label 'UPESV v/o VSC' should read 'UPESV w/o VSC'.
  2. [Section IV.A] The description of the expert videos says they contain 8M steps; please clarify whether these are frames or transitions, and how many distinct video episodes are used.
  3. [Table II] Please report the number of transitions used for the accuracy evaluation and specify whether the 'unseen' expert datasets are disjoint from those used during training.
  4. [Section III.A] The EMA momentum is set to 0.05, which is unusually small for a momentum encoder; please confirm this value and cite a reference for the chosen range.
  5. [Section III.A, Eq. (2)] The cosine similarity Gc is used directly in the loss (Eq. 3) without a temperature parameter; if this is intentional, state so explicitly.
  6. [Section IV.C, Fig. 3] The ablation study is shown for only eight of the sixteen tasks; please justify the selection or provide results for all tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the policy-cloning target is anchored by external expert videos, ground-truth actions from non-expert transitions, and independent Procgen evaluation.

full rationale

UPESV's derivation chain is not circular. The video labeling model V (Eq. 1) is trained with three objectives, but only the ground-truth action prediction task (Eq. 5) uses real action labels, and those labels come from reward-free transitions collected in the environment, not from the expert videos whose actions are being inferred. The latent future reconstruction and visual shift contrast tasks (Eqs. 3-4) operate on expert videos but do not define the action labels; they provide representation and dynamics signals. The cloned policy (Eq. 6) is trained to match V's latent action predictions on expert videos, while the final evaluation is the external Procgen reward signal (Table I), not V's own predictions; Table II additionally checks V's action predictions against held-out expert action labels. The iterative policy-V loop is a self-training scheme, but it is anchored by fixed expert videos and by the GAP task's ground-truth actions, so no prediction reduces by construction to its inputs. Self-citations to the authors' prior work (refs [3], [16], [41]) appear only in related-work context and are not load-bearing for the method's claims. The limitations stated in Section V are acknowledged scope limitations, not circular steps. A separate reporting issue exists: the claim of outperforming baselines on 12/16 tasks is not strictly supported by Table I, since ties on Heist and Chaser are counted as wins (strictly 10/16); this is a factual-consistency concern, not circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the availability of expert videos, the learnability of inverse dynamics from observation pairs, and several hyperparameters (shift distance, learning rates, update counts, codebook size). The method introduces no new physical or ontological entities.

free parameters (6)
  • VSC shift distance s = 1
    Maximum random shift in pixels for the visual shift contrast task; the paper tests s=0,1,2,4,6 and reports s=1 as good but not uniquely optimal.
  • EMA momentum = 0.05
    Momentum for the target encoder update; standard in MoCo/CURL, but still a chosen hyperparameter.
  • Learning rates for four objectives = 3e-5 (VSC), 3e-4 (LFR), 1e-3 (GAP), 2e-4 (UPC)
    Separate learning rates for each self-supervised task; the authors acknowledge balancing them requires additional experimental trials (Section V).
  • Update counts = 3k for GAP, 50k for UPC, 60k for other objectives
    Number of update steps for the four objectives, chosen by the authors.
  • VQ codebook size
    The latent action space is discretized by vector quantization (Section III.B) but the codebook size is not specified, another tuning knob.
  • Batch sizes = 128 (videos), 512 (transitions)
    Chosen batch sizes; not swept.
assumptions (4)
  • domain assumption The expert videos are generated by well-trained RL policies and are representative of expert behavior in the target environments.
    The method inherits the 8M-step expert videos from [55] (LAPO) and assumes they contain expert demonstrations; Section IV.A says they are 'generated by well-trained RL-based policies, provided by [55]'.
  • domain assumption The observation pairs in expert videos contain enough information to infer the expert actions (inverse dynamics is learnable).
    The entire video labeling model V(av_i | ov_i, ov_{i+1}) rests on this; Section III states it.
  • ad hoc to paper Random shifts of up to s pixels preserve action-relevant semantics while removing absolute position information.
    The visual shift contrast task (Section III.A) is designed around this assumption; the paper shows s=1 works but does not prove it generally.
  • ad hoc to paper The learned latent action space z, after discretization by VQ, is a faithful and stable target for policy cloning.
    Section III.D clones the policy in latent space (Eq. 6); if z is not stable or not aligned with h, the cloned policy fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos." pith.science (2026). https://pith.science/paper/AKCUZQYZ

@misc{pith2026241210778,
  author       = {Pith},
  title        = {Pith review of: Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKCUZQYZ}},
  note         = {Machine review of arXiv:2412.10778}
}
read the original abstract

Current advanced policy learning methodologies have demonstrated the ability to develop expert-level strategies when provided enough information. However, their requirements, including task-specific rewards, action-labeled expert trajectories, and huge environmental interactions, can be expensive or even unavailable in many scenarios. In contrast, humans can efficiently acquire skills within a few trials and errors by imitating easily accessible internet videos, in the absence of any other supervision. In this paper, we try to let machines replicate this efficient watching-and-learning process through Unsupervised Policy from Ensemble Self-supervised labeled Videos (UPESV), a novel framework to efficiently learn policies from action-free videos without rewards and any other expert supervision. UPESV trains a video labeling model to infer the expert actions in expert videos through several organically combined self-supervised tasks. Each task performs its duties, and they together enable the model to make full use of both action-free videos and reward-free interactions for robust dynamics understanding and advanced action prediction. Simultaneously, UPESV clones a policy from the labeled expert videos, in turn collecting environmental interactions for self-supervised tasks. After a sample-efficient, unsupervised, and iterative training process, UPESV obtains an advanced policy based on a robust video labeling model. Extensive experiments in sixteen challenging procedurally generated environments demonstrate that the proposed UPESV achieves state-of-the-art interaction-limited policy learning performance (outperforming five current advanced baselines on 12/16 tasks) without exposure to any other supervision except for videos.

Figures

Figures reproduced from arXiv: 2412.10778 by the authors.

Figure 1
Figure 1. UPESV achieves sample-efficient policy learning with only [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. UPESV framework. UPESV learns a video labeling model and a policy network jointly through three organically combined self-supervised tasks, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation study of three self-supervised tasks on eight procgen environments. Each task is necessary in the proposed UPESV. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Hyper-parameter analysis of visual shift contrast. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 45 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  2. [2]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Sil- ver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in International Conference on Learning Representations , 2016

    work page 2016

  3. [3]

    Balancing state exploration and skill diversity in unsupervised skill discovery,

    X. Liu, Y . Chen, G. Chen, H. Li, and D. Zhao, “Balancing state exploration and skill diversity in unsupervised skill discovery,” IEEE Transactions on Cybernetics , 2025

    work page 2025

  4. [4]

    URLB: Unsupervised reinforcement learning benchmark,

    M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel, “URLB: Unsupervised reinforcement learning benchmark,” in Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2) , 2021

    work page 2021

  5. [5]

    Effective representation learning is more effective in reinforcement learning than you think,

    J. Zheng and Y . Song, “Effective representation learning is more effective in reinforcement learning than you think,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 9176–9182

    work page 2024

  6. [6]

    Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning,

    R. Zheng, X. Wang, Y . Sun, S. Ma, J. Zhao, H. Xu, H. Daum ´e III, and F. Huang, “Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 48 203–48 225, 2023

    work page 2023

  7. [7]

    Human-level control through directly trained deep spiking q-networks,

    G. Liu, W. Deng, X. Xie, L. Huang, and H. Tang, “Human-level control through directly trained deep spiking q-networks,”IEEE Trans- actions on Cybernetics , vol. 53, no. 11, pp. 7187–7198, 2023

    work page 2023

  8. [8]

    Deep reinforcement learning-based automatic exploration for navigation in unknown environment,

    H. Li, Q. Zhang, and D. Zhao, “Deep reinforcement learning-based automatic exploration for navigation in unknown environment,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 6, pp. 2064–2076, 2020

    work page 2020

Show all 61 references
  1. [9]

    Model based reinforcement learning for atari,

    Łukasz Kaiser, M. Babaeizadeh, P. Miłos, B. Osi ´nski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski, “Model based reinforcement learning for atari,” in International Conference on Learning R...

    2020

  2. [10]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=S1lOTC4tDS

    2020

  3. [11]

    Prototypical context-aware dynamics for gener- alization in visual control with model-based reinforcement learning,

    J. Wang, Q. Zhang, Y . Mu, D. Li, D. Zhao, Y . Zhuang, P. Luo, B. Wang, and J. Hao, “Prototypical context-aware dynamics for gener- alization in visual control with model-based reinforcement learning,” IEEE Transactions on Industrial Informatics , vol. 20, no. 9, pp. 10 717–10...

    2024

  4. [12]

    Mastering atari with discrete world models,

    D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in International Conference on Learning Representations , 2021. [Online]. Available: https: //openreview.net/forum?id=0oabwyZbOu

    2021

  5. [13]

    Data-efficient reinforcement learning with self- predictive representations,

    M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman, “Data-efficient reinforcement learning with self- predictive representations,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/ forum?id=uCQfPZwRaUu

    2021

  6. [14]

    Reinforcement learning with unsupervised auxiliary tasks,

    M. Jaderberg, V . Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” in International Conference on Learning Representations , 2017. [Online]. Available: https: //openreview.net/forum?id=SJ6yPD5xg

    2017

  7. [15]

    Masked and inverse dynamics modeling for data-efficient reinforcement learning,

    Y . J. Lee, J. Kim, Y . J. Park, M. Kwak, and S. B. Kim, “Masked and inverse dynamics modeling for data-efficient reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–14, 2024

    2024

  8. [16]

    Learning future representation with synthetic observations for sample-efficient reinforcement learning,

    X. Liu, Y . Chen, and D. Zhao, “Learning future representation with synthetic observations for sample-efficient reinforcement learning,” Science China Information Sciences , 2025

    2025

  9. [17]

    Design from policies: Conservative test-time adaptation for offline policy optimization,

    J. Liu, H. Zhang, Z. Zhuang, Y . Kang, D. Wang, and B. Wang, “Design from policies: Conservative test-time adaptation for offline policy optimization,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  10. [18]

    Hiql: Offline goal-conditioned rl with latent states as actions,

    S. Park, D. Ghosh, B. Eysenbach, and S. Levine, “Hiql: Offline goal-conditioned rl with latent states as actions,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  11. [19]

    A survey of imitation learning: Algorithms, recent developments, and challenges,

    M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi, “A survey of imitation learning: Algorithms, recent developments, and challenges,” IEEE Transactions on Cybernetics , 2024

    2024

  12. [20]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” Advances in Neural Information Processing Systems , vol. 29, 2016

    2016

  13. [21]

    Robotic offline rl from inter- net videos via value-function learning,

    C. Bhateja, D. Guo, D. Ghosh, A. Singh, M. Tomar, Q. Vuong, Y . Chebotar, S. Levine, and A. Kumar, “Robotic offline rl from inter- net videos via value-function learning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , 2024, pp. 16 977– 16 984

    2024

  14. [22]

    Reinforcement learning from passive data via latent intentions,

    D. Ghosh, C. A. Bhateja, and S. Levine, “Reinforcement learning from passive data via latent intentions,” in International Conference on Machine Learning . PMLR, 2023, pp. 11 321–11 339

    2023

  15. [23]

    Diffusion reward: Learning rewards via conditional video diffusion,

    T. Huang, G. Jiang, Y . Ze, and H. Xu, “Diffusion reward: Learning rewards via conditional video diffusion,” in European Conference on Computer Vision. Springer, 2024, pp. 478–495

    2024

  16. [24]

    Generative adversarial imitation from observation,

    F. Torabi, G. Warnell, and P. Stone, “Generative adversarial imitation from observation,” arXiv preprint arXiv:1807.06158 , 2018

    2018 arXiv

  17. [25]

    Learning from visual observation via offline pretrained state-to-go transformer,

    B. Zhou, K. Li, J. Jiang, and Z. Lu, “Learning from visual observation via offline pretrained state-to-go transformer,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  18. [26]

    Video prediction models as rewards for reinforcement learning,

    A. Escontrela, A. Adeniji, W. Yan, A. Jain, X. B. Peng, K. Goldberg, Y . Lee, D. Hafner, and P. Abbeel, “Video prediction models as rewards for reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  19. [27]

    Ilpo-mp: Mode priors prevent mode collapse when imitating latent policies from observations,

    O. Struckmeier and V . Kyrki, “Ilpo-mp: Mode priors prevent mode collapse when imitating latent policies from observations,” Transac- tions on Machine Learning Research , 2023

    2023

  20. [28]

    Behavioral cloning from obser- vation,

    F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from obser- vation,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018, pp. 4950–4957

    2018

  21. [29]

    Imitating latent policies from observation,

    A. Edwards, H. Sahni, Y . Schroecker, and C. Isbell, “Imitating latent policies from observation,” in International Conference on Machine Learning. PMLR, 2019, pp. 1755–1763

    2019

  22. [30]

    Steps: Joint self-supervised nighttime image enhancement and depth estimation,

    Y . Zheng, C. Zhong, P. Li, H.-a. Gao, Y . Zheng, B. Jin, L. Wang, H. Zhao, G. Zhou, Q. Zhang et al. , “Steps: Joint self-supervised nighttime image enhancement and depth estimation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 4916–4923

    2023

  23. [31]

    Reinforcement learn- ing with prototypical representations,

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Reinforcement learn- ing with prototypical representations,” in International Conference on Machine Learning. PMLR, 2021, pp. 11 920–11 931

    2021

  24. [32]

    Intrinsically motivated self- supervised learning in reinforcement learning,

    Y . Zhao, C. Du, H. Zhao, and T. Li, “Intrinsically motivated self- supervised learning in reinforcement learning,” in 2022 International Conference on Robotics and Automation (ICRA) , 2022, pp. 3605– 3615

    2022

  25. [33]

    Deep reinforcement learning for autonomous driving: A survey,

    B. R. Kiran, I. Sobh, V . Talpaert, P. Mannion, A. A. Al Sallab, S. Yo- gamani, and P. P ´erez, “Deep reinforcement learning for autonomous driving: A survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909–4926, 2021

    2021

  26. [34]

    Scalable deep reinforcement learning for vision-based robotic manipulation,

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhouckeet al., “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Conference on Robot Learning . PMLR, 2018, pp. 651–673

    2018

  27. [35]

    Mastering diverse domains through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104 , 2023

    2023 arXiv

  28. [36]

    Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representa- tions,

    F. Deng, I. Jang, and S. Ahn, “Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representa- tions,” in International Conference on Machine Learning . PMLR, 2022, pp. 4956–4975

    2022

  29. [37]

    A survey on model-based reinforcement learning,

    F.-M. Luo, T. Xu, H. Lai, X.-H. Chen, W. Zhang, and Y . Yu, “A survey on model-based reinforcement learning,” Science China Information Sciences, vol. 67, no. 2, p. 121101, 2024

    2024

  30. [38]

    Curl: Contrastive unsupervised representations for reinforcement learning,

    M. Laskin, A. Srinivas, and P. Abbeel, “Curl: Contrastive unsupervised representations for reinforcement learning,” in International Confer- ence on Machine Learning . PMLR, 2020, pp. 5639–5650

    2020

  31. [39]

    Masked visual pre- training for motor control,

    T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, “Masked visual pre- training for motor control,” arXiv preprint arXiv:2203.06173 , 2022

    2022 arXiv

  32. [40]

    Value-consistent representation learning for data-efficient reinforcement learning,

    Y . Yue, B. Kang, Z. Xu, G. Huang, and S. Yan, “Value-consistent representation learning for data-efficient reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, 2023, pp. 11 069–11 077

    2023

  33. [41]

    Cross-domain random pretraining with prototypes for reinforcement learning,

    X. Liu, Y . Chen, H. Li, B. Li, and D. Zhao, “Cross-domain random pretraining with prototypes for reinforcement learning,” IEEE Trans- actions on Systems, Man, and Cybernetics: Systems , 2025

    2025

  34. [42]

    Imitation learning: A survey of learning methods,

    A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,” ACM Computing Surveys (CSUR) , vol. 50, no. 2, pp. 1–35, 2017

    2017

  35. [43]

    Reinforcement learning with action-free pre-training from videos,

    Y . Seo, K. Lee, S. L. James, and P. Abbeel, “Reinforcement learning with action-free pre-training from videos,” in International Conference on Machine Learning . PMLR, 2022, pp. 19 561–19 579

    2022

  36. [44]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos,

    B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune, “Video pretraining (vpt): Learning to act by watching unlabeled online videos,” Advances in Neural Information Processing Systems , vol. 35, pp. 24 639–24 654, 2022

    2022

  37. [45]

    Masked world models for visual control,

    Y . Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and P. Abbeel, “Masked world models for visual control,” in Conference on Robot Learning. PMLR, 2023, pp. 1332–1344

    2023

  38. [46]

    Multi-view masked world models for visual robotic manipulation,

    Y . Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel, “Multi-view masked world models for visual robotic manipulation,” in Interna- tional Conference on Machine Learning . PMLR, 2023, pp. 30 613– 30 632

    2023

  39. [47]

    Visual imitation learning with patch rewards,

    M. Liu, T. He, W. Zhang, S. Y AN, and Z. Xu, “Visual imitation learning with patch rewards,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=OnM3R47KIiU

    2023

  40. [48]

    Adversarial imitation learning from visual observations using latent information,

    V . Giammarino, J. Queeney, and I. Paschalidis, “Adversarial imitation learning from visual observations using latent information,” Transactions on Machine Learning Research , 2024. [Online]. Available: https://openreview.net/forum?id=ydPHjgf6h0

    2024

  41. [49]

    Zero-shot visual imitation,

    D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y . Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, “Zero-shot visual imitation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 2050–2053

    2018

  42. [50]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9729–9738

    2020

  43. [51]

    Decoupling rep- resentation learning from reinforcement learning,

    A. Stooke, K. Lee, P. Abbeel, and M. Laskin, “Decoupling rep- resentation learning from reinforcement learning,” in International Conference on Machine Learning . PMLR, 2021, pp. 9870–9879

    2021

  44. [52]

    A simple frame- work for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple frame- work for contrastive learning of visual representations,” in Interna- tional Conference on Machine Learning . PMLR, 2020, pp. 1597– 1607

    2020

  45. [53]

    On mutual information maximization for representation learning,

    M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic, “On mutual information maximization for representation learning,” in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=rkxoh24FPH

    2020

  46. [54]

    Neural discrete representa- tion learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representa- tion learning,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  47. [55]

    Learning to act without actions,

    D. Schmidt and M. Jiang, “Learning to act without actions,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=rvUq3cxpDF

    2024

  48. [56]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  49. [57]

    Policy gradient without boostrapping via truncated value learning,

    M. Aitchison, P. Sweetser, G. Deletang, and M. Hutter, “Policy gradient without boostrapping via truncated value learning,” 2024. [Online]. Available: https://openreview.net/forum?id=nBYDP46s5N

    2024

  50. [58]

    Leveraging procedu- ral generation to benchmark reinforcement learning,

    K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, “Leveraging procedu- ral generation to benchmark reinforcement learning,” in International Conference on Machine Learning . PMLR, 2020, pp. 2048–2056

    2020

  51. [59]

    Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunninget al., “Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,” in International Conference on Machine Learning . PMLR, 2018, pp....

    2018

  52. [60]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Com- puting and Computer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Cham: Springer International P...

    2015

  53. [61]

    Adam: A method for stochastic optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.