Pith. sign in

REVIEW 4 major objections 6 minor 49 references

PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A self-supervised reward model trained only on unlabeled video can replace hand-designed rewards for robot reinforcement learning.

desk verdict Simulation results are credible, but the real-robot RWR weighting arithmetic doesn't support the reported success gap. read the letter →

arxiv 2411.17764 v1 pith:E3HPJIKS submitted 2024-11-26 cs.RO cs.AI

Tewodros Ayalew , Xiao Zhang , Kevin Yuanbo Wu , Tianchong Jiang , Michael Maire , Matthew R. Walter This is my paper · ORCID
classification cs.ROcs.AI
keywords rewardlearningself-supervisedgoal-conditionedreinforcementprogressestimationvideopretrainingdistributionshiftadversarialrefinementrobotmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PROGRESSOR claims that a robot can learn complex manipulation skills using only unlabeled video demonstrations, with no hand-designed reward function and no action labels. The reward is the predicted 'progress' of the current frame between the initial and goal frames of an expert video, learned self-supervised from frame triplets. During online RL, the reward model is adversarially refined by 'push-back' so that out-of-distribution observations get lower progress, countering distribution shift. The paper reports that this reward alone lets a DrQ-v2 agent solve six Meta-World tasks, and that an EPIC-KITCHENS-pretrained PROGRESSOR outperforms R3M and VIP reward models on real-robot offline RL with half of demonstrations failed.

What carries the argument

The progress estimator $E_\theta(o_i, o_j, o_g)$ — a shared visual encoder with MLP heads predicting $\mu$ and $\log\sigma^2$ — is trained by KL divergence against a Gaussian target centered at the frame-index ratio $\delta = |j-i|/|g-i|$, with variance bounded below by $1/(g-i)$. The reward $r_\theta = \mu - \alpha H(\mathcal{N}(\mu,\sigma^2))$ turns predicted progress into a dense scalar reward. The distinctive mechanism is adversarial online refinement ('push-back'): for frames sampled from online rollouts, the model is trained toward a shrunk prediction $\beta\mu$ ($\beta=0.9$) with a stop-gradient target, while expert triplets continue to be trained toward their true progress, so the reward explicitly penalizes states that look like non-expert behavior.

What would settle it

Run the identical DrQ-v2 setup on a task with cyclic observations (for example, stirring or turning a crank), using expert videos that revisit the same states; if PROGRESSOR's reward fails to separate progress from regress and the agent does not learn, the frame-index progress assumption is what breaks. A cheaper check is to take a single successful demonstration with temporary backtracking, compute PROGRESSOR's predicted reward at the backtracked frames, and show it rises when true progress falls.

Watch

Extended reading notes

Core claim

The central discovery is that a single task-agnostic reward model, trained only on unordered frame triplets from expert videos, can provide a dense reward signal strong enough to drive reinforcement learning from pixels. The model outputs a Gaussian distribution over progress, with mean given by the normalized frame-position ratio $\delta(o_i, o_j, o_g)=|j-i|/|g-i|$, and the reward is the predicted mean minus an entropy penalty. During online training, a push-back loss nudges predictions on non-expert rollouts toward a discounted progress value $\beta\mu$, while expert predictions are continually re-fitted, so the reward stays calibrated as the policy explores. The authors show this enables task-agnostic generalization: one model pretrained on human egocentric kitchen videos transfers zero-shot to real-robot tasks and, when used in reward-weighted regression over a mix of successful and failed demonstrations, yields higher success rates than rewards derived from R3M or VIP.

Load-bearing premise

The reward signal reduces to the assumption that every expert demonstration advances monotonically toward the goal: progress is defined by frame indices, so any successful trajectory that backtracks, pauses, or revisits states will be assigned the wrong progress regardless of network capacity.

Editorial extensions

If this is right

  • A policy can be trained end-to-end from pixels with no environment reward, because the predicted progress provides dense credit assignment at every step.
  • A single reward model pretrained on human video transfers across robotic tasks without task-specific fine-tuning, eliminating per-task reward engineering.
  • The push-back refinement keeps the reward calibrated during exploration by continuously shrinking rewards on non-expert states, which is what makes online RL viable.
  • In offline settings with noisy demonstrations, weighting behavior-cloning losses by progress rewards makes the policy resilient to failed trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the monotonic frame-index progress assumption is the binding constraint, PROGRESSOR should struggle on tasks where success requires revisiting states or cyclic motion; a natural extension would be a multimodal or latent progress model that can represent backtracking.
  • The same progress-reward recipe could be applied to any domain with ordered video of successful executions—surgical training, sports coaching, or household chores—where the monotonicity assumption approximately holds.
  • Because the reward compares the current frame to the initial and goal frames rather than to a fixed embedding, PROGRESSOR may show greater resilience to camera viewpoint changes than contrastive distance rewards; a viewpoint-transfer experiment would test this.
  • A controlled stress test—training on demonstrations that include temporary backtracking and checking whether the reward peaks at the wrong time—would directly isolate the cost of the monotonicity assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PROGRESSOR, a reward model that predicts a Gaussian distribution over task progress for an observation triplet (initial, current, goal), using only self-supervised frame-index labels from unlabeled expert videos. During online RL, a 'push-back' loss (Eq. 7) shrinks the model's own progress predictions on non-expert rollouts to mitigate distribution shift. The authors evaluate PROGRESSOR in Meta-World with DrQ-v2, and on a real UR5 robot using reward-weighted regression ACT (RWR-ACT) trained on a mix of success and failed demonstrations, comparing against R3M and VIP reward models. The central claim is that PROGRESSOR provides dense rewards that enable learning without environment rewards or action labels.

Significance. If correct, the paper demonstrates a promising direction: a single reward model pretrained on human videos (EPIC-KITCHENS) that can be applied zero-shot to real-robot tasks, even with noisy demonstrations. The simulation results across six Meta-World tasks are generally strong and the ablations show the online refinement helps. The paper also provides qualitative reward-prediction plots and is mostly reproducible in structure. However, the real-robot success-rate claim rests on a mechanism (RWR with omega=0.1) that appears too weak to explain the reported gap from 0% (ACT) to 70-90% (PROGRESSOR-RWR-ACT) unless some quantity is misreported. Until this is reconciled, the significance of the real-robot contribution is uncertain.

major comments (4)
  1. [Section 5.2.3, Eq. (8), Figure 6] The RWR weighting as specified cannot explain the reported success-rate gap. With omega=0.1 and the reward scale shown in Figure 6 (correct vs. incorrect demonstrations differ by roughly 0.4 in Drawer-Open), the weight ratio in Eq. (8) is exp(0.1*0.4) ≈ 1.04. Even taking the largest visible gaps (about 0.6-0.8), the ratio is at most exp(0.08) ≈ 1.08. Weighting this close to uniform would make RWR-ACT nearly equivalent to vanilla ACT, yet vanilla ACT achieves 0% while PROGRESSOR-RWR-ACT achieves 70-90% on the hard tasks. Please provide the exact reward values inserted into Eq. (8), clarify whether the Figure 6 'reward prediction' is the same quantity, and either show a sensitivity analysis over omega or identify a normalization/scale that is missing. This is load-bearing for the real-robot claim.
  2. [Section 4.2, Eq. (7), abstract, contributions] The 'adversarial online refinement' is not adversarial. The push-back loss minimizes the KL divergence between the model's own prediction scaled by beta and the model's current prediction; there is no adversary, discriminator, or min-max objective. This is more accurately a self-distillation shrinkage regularizer toward beta*mu. The terminology overstates the method and the comparison to domain-adversarial methods (Ganin et al., Goodfellow et al.) is misleading. Please rename this component or reformulate it as a true adversarial objective, and adjust the abstract and contribution claims accordingly.
  3. [Eq. (2) and text below it] There is a technical inconsistency: the sentence says 'Our epsilon upper-bound of sigma_tau_k downweights...', but the formula sigma_tau_k = max(1/(g-i), epsilon) makes epsilon a lower bound (sigma is always at least epsilon), not an upper bound. The formula and the verbal description must be aligned, as this affects the variance of the KL target and the claimed robustness behavior.
  4. [Section 4.2 and Algorithm 1] The online push-back update is underspecified: it is not stated how the triplet (o_i, o_j, o_g) is sampled from the replay buffer/current rollouts for Eq. (7), nor what g-i represents in the online setting (episode length? current step index?). Without this definition, the push-back loss is not reproducible, which is particularly important because it is a key contribution and the ablation shows it changes performance substantially.
minor comments (6)
  1. [Throughout] There are numerous typos, including 'pushing back pushing back predictions' (Section 1), 'demonstratin' (Section 2), 'lloning' (Section 5.2.3), 'camere' (Supplement S9.1), 'pretrianing' (S7.1), and 'the the' (Section 1). These should be corrected.
  2. [Section 4.1] The text says 'See S8 for an ablation of alpha,' but Section 8 of the supplement contains simulation details only; the beta ablation is in Section 10. Please update the cross-reference.
  3. [Equation (8)] In the paragraph following Eq. (8), the text refers to 'uθ' while Eq. (5) defines rθ. Please make the notation consistent.
  4. [Figure 4 caption] The caption says '(a) The evolution of the episodic reward (b) The evolution of the success rate,' but the figure contains two rows of six panels each. Please clarify which panels correspond to which metric.
  5. [Section 5.1 and Supplement S8] The paper does not specify how the goal image is chosen for the Meta-World online RL experiments (e.g., whether it is the final frame of a successful expert rollout). This detail is needed for reproducibility.
  6. [Figure 5] Success rates are averages over 20 rollouts with no confidence intervals or raw counts. Reporting standard errors or binomial confidence intervals would strengthen the comparison.

Circularity Check

1 steps flagged · score 2.0 of 10

Push-back loss is self-referential by construction but not load-bearing; central progress reward is anchored to expert frame indices and evaluated externally, so no significant circularity.

  1. self definitional [Section 4.2, Eq. (7)]
    "we update Eθ so that it learns to push-back the current estimation as βµτ′k with β ∈ [0, 1] as the decay factor. ... ppush-back = sg(N(βµτ′k, 1/(g−i)^2)), Lpush-back = DKL(ppush-back∥Eθ(oτ′ki, oτ′kj, oτ′kg))"

    The online refinement target is defined as β times the model's own current mean prediction µ. With stop-gradient, minimizing the KL loss contracts µ toward βµ (β=0.9), so the 'push-back' carries no information beyond the model's own output scaled by a constant. This makes the refinement self-referential by construction. It is not load-bearing for the paper's main external claims (Meta-World success, real-robot success) because those are evaluated against environment/teleoperation outcomes, and the pretraining reward (Eq. 4) is anchored to frame-index progress labels. Hence a minor self-referential step rather than full circularity.

full rationale

The paper's central derivation is self-contained: Eq. (1) defines progress labels directly from frame indices in expert trajectories, Eq. (4) learns a Gaussian progress predictor against those labels, and Eq. (5) converts the predicted mean into a dense reward. The resulting policy is evaluated with external environment rewards and success rates, so no fitted constant is disguised as a prediction. The only self-referential element is the online push-back loss of Eq. (7), whose target is a scaled version of the model's own output; however, this is an explicitly stated regularizer rather than a claimed external prediction, and the paper's headline results do not rest on it alone. There are no load-bearing self-citations: the cited works (Rank2Reward, VIP, R3M, ACT, etc.) are by different authors and are used for comparison or acknowledged assumptions. The monotonic-progress assumption is stated openly and its limitation for cyclic tasks is acknowledged in Section 6, so it is a modeling assumption rather than hidden circularity. The reported difficulty in explaining the real-robot success gap from the RWR weights (Eq. 8 with ω=0.1) is a quantitative/explanatory concern, not a circularity concern. Overall, the derivation chain is not circular in any load-bearing way.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. The free parameters are hyperparameters and hand-set training labels. The central load-bearing axiom is monotonic progress, inherited from Rank2Reward, which the paper itself acknowledges fails on cyclic tasks.

free parameters (5)
  • alpha = 0.4
    Entropy penalty in reward Eq (5), set by hand; the text references an ablation in S8 that does not exist.
  • beta = 0.9
    Push-back decay factor in Eq (7), selected via ablation on the hammer task (Figure 11).
  • epsilon = not reported
    Floor for sigma in Eq (2); affects target distribution width; formula and text description are inconsistent.
  • negative label for distractor triplets = -1
    Hand-set target mean for negative examples in EPIC-KITCHENS pretraining (Section 5.2.1).
  • max frame gap = 2000
    Sampling constraint for pretraining triplets (Sections 5.2.1 and 7.1).
assumptions (3)
  • domain assumption Expert demonstrations make monotonic progress toward the goal
    Section 4 states: 'Similar to Yang et al. [38], our method relies on the assumption that the values of states in optimal policies increase monotonically towards task completion.' The reward is a monotone function of this progress estimate.
  • domain assumption Environment is fully observable through images
    Section 3 assumes a fully observable MDP with high-dimensional image frames, so pixel observations are sufficient for progress estimation.
  • domain assumption A single reward model can represent progress across multiple tasks
    Section 4.1 claims that since triplets can be drawn from any trajectory in the dataset, a single reward model suffices for a variety of tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement." pith.science (2026). https://pith.science/paper/E3HPJIKS

@misc{pith2026241117764,
  author       = {Pith},
  title        = {Pith review of: PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3HPJIKS}},
  note         = {Machine review of arXiv:2411.17764}
}
read the original abstract

We present PROGRESSOR, a novel framework that learns a task-agnostic reward function from videos, enabling policy training through goal-conditioned reinforcement learning (RL) without manual supervision. Underlying this reward is an estimate of the distribution over task progress as a function of the current, initial, and goal observations that is learned in a self-supervised fashion. Crucially, PROGRESSOR refines rewards adversarially during online RL training by pushing back predictions for out-of-distribution observations, to mitigate distribution shift inherent in non-expert observations. Utilizing this progress prediction as a dense reward together with an adversarial push-back, we show that PROGRESSOR enables robots to learn complex behaviors without any external supervision. Pretrained on large-scale egocentric human video from EPIC-KITCHENS, PROGRESSOR requires no fine-tuning on in-domain task-specific data for generalization to real-robot offline RL under noisy demonstrations, outperforming contemporary methods that provide dense visual reward for robotic learning. Our findings highlight the potential of PROGRESSOR for scalable robotic applications where direct action labels and task-specific rewards are not readily available.

Figures

Figures reproduced from arXiv: 2411.17764 by the authors.

Figure 1
Figure 1. Trained in a self-supervised manner on expert videos, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Top Left: Initial phase of reward model pretraining on expert data, where the model learns to predict the parameters of a Gaussian distribution centered on normalized progress, reflecting expected progress as demonstrated by experts. Top Right: In online reinforcement learning (RL) training, an adversarial online refinement (i.e., push-back) is applied to counteract non-expert predictions made by the reward model, e… view at source ↗
Figure 3
Figure 3. Visualization of the robotic tasks: (a-d) Real world en [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Visualization of policy learning in the Meta-World [ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Success rates for four real-world tasks, where RWR [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Mean reward predictions, rˆ, for correct and incorrect demonstrations in the Drawer-Open task from PROGRESSOR, R3M, and VIP. PROGRESSOR provides more distinct reward weighting between correct and failed trajectories compared to the baseline models. achieve meaningful t…
Figure 7
Figure 7. Figure 7: Visualization of the predicted rewards from Epic [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The real-world experiments were conducted using a Uni [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Correct and incorrect demonstrations (every 80th frame) for each real-robot task from a third-person camera view used in our experiments. To see how PROGRESSOR and the baselines differentiate between correct and incorrect trajectories, see [PITH_FULL_IMAGE:figures/ful…
Figure 10
Figure 10. Figure 10: The goal images that we use for each task for reward [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 12
Figure 12. Figure 12: Visualization of the predicted reward by P [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Mean reward predictions rˆ of (left column) R3M, (middle column) VIP, and (right column) PROGRESSOR for correct and incorrect demonstrations for the Drawer-Close, Push-Block, and Pick-Place-Cup tasks. PROGRESSOR provides reward predictions (weights) that better differ…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 34 canonical work pages

  1. [1]

    Pieter Abbeel and Andrew Y . Ng. Apprenticeship learning via inverse reinforcement learning. In Pro- ceedings of the Twenty-First International Conference on Machine Learning , page 1, New York, NY , USA,

    work page

  2. [2]

    Learning reward functions for robotic manipulation by observ- ing humans, 2023

    Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, and Cordelia Schmid. Learning reward functions for robotic manipulation by observ- ing humans, 2023. 1

    work page 2023

  3. [3]

    Human-to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. arXiv preprint arXiv:2207.09450, 2022. 2

    arXiv 2022

  4. [4]

    Video pre- training (VPT): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pre- training (VPT): Learning to act by watching unlabeled online videos. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022. 2

    work page 2022

  5. [5]

    The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications

    Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2023. 1

    work page 2023

  6. [6]

    Scaling ego- centric vision: The EPIC-KITCHENS dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling ego- centric vision: The EPIC-KITCHENS dataset. In Pro- ceedings of the European Conference on Computer Vi- sion (ECCV), 2018. 2, 6, 1

    work page 2018

  7. [7]

    Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal on Computer Vision, 130:33—-55, 2022. 6

    work page 2022

  8. [8]

    Video predic- tion models as rewards for reinforcement learning

    Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video predic- tion models as rewards for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1, 2

    work page 2024

Show all 49 references
  1. [9]

    Contrastive learning as goal-conditioned reinforcement learning,

    Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning,

  2. [10]

    Guided cost learning: Deep inverse optimal con- trol via policy optimization

    Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal con- trol via policy optimization. arXiv preprint arXiv:1603.00448, 2016. 2

    2016 arXiv

  3. [11]

    Learning robust rewards with adversarial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2018. 2

    2018 arXiv

  4. [12]

    Domain- adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario March, and Victor Lempitsky. Domain- adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016. 4

    2016

  5. [13]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014. 4

    2014

  6. [14]

    Ego4D: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  7. [15]

    Inverse reward design

    Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 1

    2017

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , 2016. 6, 1

    2016

  9. [17]

    Generative adver- sarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adver- sarial imitation learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2016. 5

    2016

  10. [18]

    Generative adversarial imitation learning

    Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. arXiv preprint arXiv:1606.03476, 2016. 2

    2016 arXiv

  11. [19]

    Diffusion reward: Learning rewards via conditional video diffusion

    Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. arXiv preprint arXiv:2312.14134, 2023. 1, 2

    2023 arXiv

  12. [20]

    Auto-encoding variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 ,

    arXiv

  13. [21]

    In- foGAIL: Interpretable imitation learning from visual demonstrations

    Yunzhu Li, Jiaming Song, and Stefano Ermon. In- foGAIL: Interpretable imitation learning from visual demonstrations. arXiv preprint arXiv:1703.08840 ,

    arXiv

  14. [22]

    VIP: Towards universal visual reward and represen- tation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayara- man, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and represen- tation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022. 7

    arXiv

  15. [23]

    9 Vip: Towards universal visual reward and representa- tion via value-implicit pre-training, 2023

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayara- man, Osbert Bastani, Vikash Kumar, and Amy Zhang. 9 Vip: Towards universal visual reward and representa- tion via value-implicit pre-training, 2023. 1, 2

    2023

  16. [24]

    Towards theoretical understanding of inverse reinforcement learning

    Alberto Maria Metelli, Filippo Lazzati, and Mar- cello Restelli. Towards theoretical understanding of inverse reinforcement learning. arXiv preprint arXiv:2304.12966, 2023. 2

    2023 arXiv

  17. [26]

    R3M: A univer- sal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A univer- sal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022. 1, 2

    2022 arXiv

  18. [27]

    Advantage-weighted regression: Sim- ple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Sim- ple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. 7

    1910 arXiv

  19. [28]

    Reinforcement learning by reward-weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the International Confer- ence on Machine Learning (ICML) , pages 745–750,

  20. [29]

    DexMV: Imitation learning for dexterous manipula- tion from human videos

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipula- tion from human videos. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) , 2022. 2

    2022

  21. [30]

    Artificial Intelli- gence: A Modern Approach

    Stuart J Russell and Peter Norvig. Artificial Intelli- gence: A Modern Approach . Prentice Hall, Engle- wood Cliffs, NJ, USA, 1 edition, 1995. 1

    1995

  22. [31]

    Time-contrastive networks: Self-supervised learning from video

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jas- mine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In Proceedings of the IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 1134–1141, 2018. 5

    2018

  23. [32]

    Time-contrastive networks: Self-supervised learning from video, 2018

    Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jas- mine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video, 2018. 2

    2018

  24. [33]

    Lewis, and Andrew G

    Satinder Singh, Richard L. Lewis, and Andrew G. Barto. Where do rewards come from? In Proceedings of the International Symposium on AI Inspired Biol- ogy, pages 111–116, 2010. 1

    2010

  25. [34]

    Reinforcement Learning: An Introduction

    Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, Cambridge, MA, USA, 2 edition, 2018. 1

    2018

  26. [35]

    DeepMind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint arXiv:1801.00690, 2018. 8

    2018 arXiv

  27. [36]

    Wozniak, Andrea Gasparri, and Danica Kragic

    Michael C Welle, Nils Ingelhag, Martina Lippi, Ma- ciej K. Wozniak, Andrea Gasparri, and Danica Kragic. Quest2ROS: An app to facilitate teleoperating robots. In Proceedings of the International Workshop on Vir- tual, Augmented, and Mixed-Reality for Human-Robot Interactions, 2024. 1

    2024

  28. [37]

    Maximum entropy deep inverse reinforcement learning, 2016

    Markus Wulfmeier, Peter Ondruska, and Ingmar Pos- ner. Maximum entropy deep inverse reinforcement learning, 2016. 2

    2016

  29. [38]

    Rank2Reward: Learning shaped reward functions from passive video

    Daniel Yang, Davin Tjia, Jacob Berg, Dima Damen, Pulkit Agrawal, and Abhishek Gupta. Rank2Reward: Learning shaped reward functions from passive video. arXiv preprint arXiv:2404.14735, 2024. 1, 2, 3, 5

    2024 arXiv

  30. [39]

    Representation matters: Offline pretraining for sequential decision making

    Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815, 2021. 2

    2021 arXiv

  31. [40]

    Im- age augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Im- age augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. 5

    2021

  32. [41]

    Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference on Robot Learning (CoRL), 2020. 2, 5, 6, 1

    2020

  33. [42]

    Learning to drive by watching YouTube videos: Action-conditioned contrastive policy pretraining

    Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching YouTube videos: Action-conditioned contrastive policy pretraining. In Proceedings of the European Conference on Com- puter Vision (ECCV), 2022. 2

    2022

  34. [43]

    Learning fine-grained bimanual ma- nipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual ma- nipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. 7, 2

    2023 arXiv

  35. [44]

    Ziebart, Andrew Maas, J

    Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse rein- forcement learning. In Proceedings of the 23rd Na- tional Conference on Artificial Intelligence - Volume 3, page 1433–1438. AAAI Press, 2008. 2 10 PROGRESSOR : A Perceptually Guid...

    2008

  36. [46]

    PROGRESSOR Training Details 7.1. Architecture and Training PROGRESSOR can, in principle, be trained with any visual encoding architecture, requiring only minor modifications to the final layer to predict Gaussian parameters. In our ex- periments, we utilize the standard ResNet...

    2000

  37. [47]

    Simulation Experiment Details In this section, we describe the tasks and the data generation process employed using the MetaWorld environment [41] for our simulation experiments. 8.1. Meta-World Tasks We took six diverse tasks from the Meta-World environ- ment [41], described ...

  38. [48]

    Robotic Experiment Setup The real-robot experiments are performed using a Universal Robots UR5 robot arm equipped with a Robotiq 3-Finger Gripper (Figure 8)

    Real-World Robot Experiment Details 9.1. Robotic Experiment Setup The real-robot experiments are performed using a Universal Robots UR5 robot arm equipped with a Robotiq 3-Finger Gripper (Figure 8). The setup includes two RealSense cam- eras: one mounted on the robot’s wrist t...

  39. [49]

    The case of β = 0(PROGRESSOR with- out Push-back) is discussed in the main paper

    Ablation In this section, we present an ablation study evaluating dif- ferent values of the push-back decay factor (β) while train- ing a DrQ-v2 agent on Meta-World’shammer task, using a fixed seed of 121. The case of β = 0(PROGRESSOR with- out Push-back) is discussed in the m...

  40. [50]

    This figure serves as an extension to Figure 7 for complete- ness

    Qualitative Analysis Figure 12 presents zero-shot reward predictions from PRO- GRESSOR pretrained on the EPIC-KITCHENS dataset. This figure serves as an extension to Figure 7 for complete- ness. It includes zero-shot reward predictions for sample correct trajectories from our ...

  41. [2004]

    Association for Computing Machinery. 2

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.