{"id":"4e1c7d2c-1d7b-448f-a648-e3dd271470fb","arxiv_id":"2411.17764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A video-trained progress-estimation model, refined online with a push-back objective, provides dense rewards that enable goal-conditioned robot learning without manual reward design or action labels.","lead":"PROGRESSOR learns a reward function for robots from unlabeled videos by predicting how much progress a video frame shows toward a goal. It pretrains on human kitchen videos and then tunes itself during robot training, beating earlier visual reward methods in simulated and real manipulation tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-robot RWR weighting appears too weak to explain the reported success gap; the reward scale and weights used in Eq. 8 need verification.","rationale":"I agree with the reader that the temporal-progress assumption is a real limitation, but the authors explicitly acknowledge it in Section 6, and it does not threaten the reported evaluations. The more load-bearing issue is the quantitative inconsistency between the RWR weighting formula and the displayed reward scale. Since the real-robot experiments are the only evidence for zero-shot cross-domain generalization, this inconsistency directly affects the central claim that PROGRESSOR enables learning from noisy demonstrations and outperforms R3M/VIP. A single reproduction and a uniform-weight ablation would settle whether the reported success rates are attributable to the stated reward-weighting mechanism. Until then, the claim should remain conditional.","tokens_in":14937,"tokens_out":13294,"duration_ms":128377,"concrete_test":"Reproduce the Drawer-Open and Push-Block RWR-ACT runs, logging per-frame weights exp(ω·r̂). Verify whether the correct/failed weight ratio is ~1.04 as implied by Fig. 6 with ω=0.1. Then ablate by forcing all weights to 1.0 (same demonstrations, seed, and budget). If high success rates persist, the gains are not from the stated reward weighting; if they vanish, the numbers in Fig. 5 need re-checking.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.2.3 and Eq. 8 define RWR-ACT's loss as exp(ω·r̂) times the action error, with ω=0.1. Figure 6 shows PROGRESSOR's r̂ for Drawer-Open in the range ~0.1–0.6, i.e., a correct-vs-failed gap of about 0.4. This yields a relative weight exp(0.1×0.4)≈1.04 between successful and failed demonstrations. Even if the entropy penalty in Eq. 5 makes r̂ larger, the plausible gap is at most ~2, giving a ratio of about 1.22. Weights this close to uniform make RWR-ACT's objective nearly identical to vanilla ACT (ω=0). Yet PROGRESSOR-RWR-ACT is reported at 70–90% success while ACT is 0% on the same tasks. The stated mechanism cannot explain this gap. Either the plotted r̂ is not the quantity inserted into Eq. 8, the temperature ω or reward scale is misreported, or the comparison depends on unstated factors. This must be resolved before the real-robot claim that PROGRESSOR 'outperforms' R3M/VIP is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PROGRESSOR, a reward model that predicts a Gaussian distribution over task progress for an observation triplet (initial, current, goal), using only self-supervised frame-index labels from unlabeled expert videos. During online RL, a 'push-back' loss (Eq. 7) shrinks the model's own progress predictions on non-expert rollouts to mitigate distribution shift. The authors evaluate PROGRESSOR in Meta-World with DrQ-v2, and on a real UR5 robot using reward-weighted regression ACT (RWR-ACT) trained on a mix of success and failed demonstrations, comparing against R3M and VIP reward models. The central claim is that PROGRESSOR provides dense rewards that enable learning without environment rewards or action labels.","tokens_in":15185,"tokens_out":6720,"duration_ms":61385,"significance":"If correct, the paper demonstrates a promising direction: a single reward model pretrained on human videos (EPIC-KITCHENS) that can be applied zero-shot to real-robot tasks, even with noisy demonstrations. The simulation results across six Meta-World tasks are generally strong and the ablations show the online refinement helps. The paper also provides qualitative reward-prediction plots and is mostly reproducible in structure. However, the real-robot success-rate claim rests on a mechanism (RWR with omega=0.1) that appears too weak to explain the reported gap from 0% (ACT) to 70-90% (PROGRESSOR-RWR-ACT) unless some quantity is misreported. Until this is reconciled, the significance of the real-robot contribution is uncertain.","major_comments":[{"comment":"The RWR weighting as specified cannot explain the reported success-rate gap. With omega=0.1 and the reward scale shown in Figure 6 (correct vs. incorrect demonstrations differ by roughly 0.4 in Drawer-Open), the weight ratio in Eq. (8) is exp(0.1*0.4) ≈ 1.04. Even taking the largest visible gaps (about 0.6-0.8), the ratio is at most exp(0.08) ≈ 1.08. Weighting this close to uniform would make RWR-ACT nearly equivalent to vanilla ACT, yet vanilla ACT achieves 0% while PROGRESSOR-RWR-ACT achieves 70-90% on the hard tasks. Please provide the exact reward values inserted into Eq. (8), clarify whether the Figure 6 'reward prediction' is the same quantity, and either show a sensitivity analysis over omega or identify a normalization/scale that is missing. This is load-bearing for the real-robot claim.","section":"Section 5.2.3, Eq. (8), Figure 6"},{"comment":"The 'adversarial online refinement' is not adversarial. The push-back loss minimizes the KL divergence between the model's own prediction scaled by beta and the model's current prediction; there is no adversary, discriminator, or min-max objective. This is more accurately a self-distillation shrinkage regularizer toward beta*mu. The terminology overstates the method and the comparison to domain-adversarial methods (Ganin et al., Goodfellow et al.) is misleading. Please rename this component or reformulate it as a true adversarial objective, and adjust the abstract and contribution claims accordingly.","section":"Section 4.2, Eq. (7), abstract, contributions"},{"comment":"There is a technical inconsistency: the sentence says 'Our epsilon upper-bound of sigma_tau_k downweights...', but the formula sigma_tau_k = max(1/(g-i), epsilon) makes epsilon a lower bound (sigma is always at least epsilon), not an upper bound. The formula and the verbal description must be aligned, as this affects the variance of the KL target and the claimed robustness behavior.","section":"Eq. (2) and text below it"},{"comment":"The online push-back update is underspecified: it is not stated how the triplet (o_i, o_j, o_g) is sampled from the replay buffer/current rollouts for Eq. (7), nor what g-i represents in the online setting (episode length? current step index?). Without this definition, the push-back loss is not reproducible, which is particularly important because it is a key contribution and the ablation shows it changes performance substantially.","section":"Section 4.2 and Algorithm 1"}],"minor_comments":[{"comment":"There are numerous typos, including 'pushing back pushing back predictions' (Section 1), 'demonstratin' (Section 2), 'lloning' (Section 5.2.3), 'camere' (Supplement S9.1), 'pretrianing' (S7.1), and 'the the' (Section 1). These should be corrected.","section":"Throughout"},{"comment":"The text says 'See S8 for an ablation of alpha,' but Section 8 of the supplement contains simulation details only; the beta ablation is in Section 10. Please update the cross-reference.","section":"Section 4.1"},{"comment":"In the paragraph following Eq. (8), the text refers to 'uθ' while Eq. (5) defines rθ. Please make the notation consistent.","section":"Equation (8)"},{"comment":"The caption says '(a) The evolution of the episodic reward (b) The evolution of the success rate,' but the figure contains two rows of six panels each. Please clarify which panels correspond to which metric.","section":"Figure 4 caption"},{"comment":"The paper does not specify how the goal image is chosen for the Meta-World online RL experiments (e.g., whether it is the final frame of a successful expert rollout). This detail is needed for reproducibility.","section":"Section 5.1 and Supplement S8"},{"comment":"Success rates are averages over 20 rollouts with no confidence intervals or raw counts. Reporting standard errors or binomial confidence intervals would strengthen the comparison.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The primary concern is the real-robot RWR mechanism: the reported omega=0.1 and reward scales make the weighting essentially uniform, so the stark success-rate differences are inexplicable under the stated method. If this is a reporting error (e.g., omega should be larger, or the rewards are not on the scale shown), the paper can be revised; if the numbers are correct, the results are likely not reproducible. The 'adversarial' misuse of terminology is also a framing issue that should be corrected. I recommend major revision rather than rejection because the simulation component is sound and the real-robot issue may be fixable with transparency about the exact quantities used."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the Meta-World simulation results are credible, and the idea of a single task-agnostic progress-based reward, pretrained on EPIC-KITCHENS and refined online, is a reasonable extension of Rank2Reward. But the real-robot RWR experiment has an internal inconsistency that needs to be resolved before the headline claims are taken at face value.\n\nWhat's actually new: the Gaussian progress distribution gives a principled uncertainty estimate; the single model across tasks avoids Rank2Reward's per-task training; the push-back objective is a simple way to adapt to online rollouts; and the real-robot study with half failed demonstrations is a good stress test. The paper cites Rank2Reward properly and doesn't oversell the novelty. The simulation results are fairly convincing: PROGRESSOR beats TCN, GAIL, and Rank2Reward on six Meta-World tasks, and the push-back ablation supports its usefulness.\n\nThe soft spots: first, the real-robot RWR weighting. Equation (8) uses exp(ω·r̂) with ω=0.1. Figure 6 shows PROGRESSOR's r̂ for Drawer-Open sitting roughly in [0.1, 0.6], a correct-vs-failed gap of about 0.4. That gives a weight ratio exp(0.1×0.4)≈1.04 between successful and failed demonstrations. Even if the entropy term in Eq. (5) widens the gap, you'd be hard-pressed to get a ratio above 1.2 with that temperature. Yet PROGRESSOR-RWR-ACT reportedly gets 70–90% success while vanilla ACT gets 0% on the same data. That discrepancy suggests either the plotted r̂ is not what was fed into Eq. (8), the temperature or reward scale is misreported, or something else in the setup accounts for the gap. This needs to be verified—maybe the reward values are actually scaled up, or the weighting is done differently, but as written it doesn't add up.\n\nSecond, minor issues: Eq. (2)'s text says ε is an upper bound on σ, but the formula max(1/(g−i), ε) makes ε a lower bound. The cross-reference to an α ablation in Section 4.1 points to S8, which doesn't contain it; the actual ablation only covers β. And calling the push-back \"adversarial\" is a stretch—it's regression to the model's own scaled output, not a discriminator. These are not fatal, but they should be cleaned up.\n\nThe paper is worth a serious referee: the simulation evidence is meaningful, and the RWR issue is exactly the kind of thing peer review should catch. I'd bring it to a reading group as a case study in caught inconsistencies.","headline":"Simulation results are credible, but the real-robot RWR weighting arithmetic doesn't support the reported success gap.","tokens_in":15716,"tokens_out":5959,"would_cite":true,"duration_ms":50514,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-supervised reward model trained only on unlabeled video can replace hand-designed rewards for robot reinforcement learning.","keywords":["reward learning","self-supervised learning","goal-conditioned reinforcement learning","progress estimation","video pretraining","distribution shift","adversarial refinement","robot manipulation"],"falsifier":"Run the identical DrQ-v2 setup on a task with cyclic observations (for example, stirring or turning a crank), using expert videos that revisit the same states; if PROGRESSOR's reward fails to separate progress from regress and the agent does not learn, the frame-index progress assumption is what breaks. A cheaper check is to take a single successful demonstration with temporary backtracking, compute PROGRESSOR's predicted reward at the backtracked frames, and show it rises when true progress falls.","tokens_in":14761,"feed_emoji":"🤖","tokens_out":9706,"duration_ms":75554,"temperature":0.7,"pith_summary":"PROGRESSOR claims that a robot can learn complex manipulation skills using only unlabeled video demonstrations, with no hand-designed reward function and no action labels. The reward is the predicted 'progress' of the current frame between the initial and goal frames of an expert video, learned self-supervised from frame triplets. During online RL, the reward model is adversarially refined by 'push-back' so that out-of-distribution observations get lower progress, countering distribution shift. The paper reports that this reward alone lets a DrQ-v2 agent solve six Meta-World tasks, and that an EPIC-KITCHENS-pretrained PROGRESSOR outperforms R3M and VIP reward models on real-robot offline RL with half of demonstrations failed.","feed_headline":"Progress from unlabeled video replaces hand-designed robot rewards","feed_subtitle":"A frame-position ratio from expert video supplies dense rewards, beating R3M and VIP on real-robot offline RL.","key_machinery":"The progress estimator $E_\\theta(o_i, o_j, o_g)$ — a shared visual encoder with MLP heads predicting $\\mu$ and $\\log\\sigma^2$ — is trained by KL divergence against a Gaussian target centered at the frame-index ratio $\\delta = |j-i|/|g-i|$, with variance bounded below by $1/(g-i)$. The reward $r_\\theta = \\mu - \\alpha H(\\mathcal{N}(\\mu,\\sigma^2))$ turns predicted progress into a dense scalar reward. The distinctive mechanism is adversarial online refinement ('push-back'): for frames sampled from online rollouts, the model is trained toward a shrunk prediction $\\beta\\mu$ ($\\beta=0.9$) with a stop-gradient target, while expert triplets continue to be trained toward their true progress, so the reward explicitly penalizes states that look like non-expert behavior.","core_discovery":"The central discovery is that a single task-agnostic reward model, trained only on unordered frame triplets from expert videos, can provide a dense reward signal strong enough to drive reinforcement learning from pixels. The model outputs a Gaussian distribution over progress, with mean given by the normalized frame-position ratio $\\delta(o_i, o_j, o_g)=|j-i|/|g-i|$, and the reward is the predicted mean minus an entropy penalty. During online training, a push-back loss nudges predictions on non-expert rollouts toward a discounted progress value $\\beta\\mu$, while expert predictions are continually re-fitted, so the reward stays calibrated as the policy explores. The authors show this enables task-agnostic generalization: one model pretrained on human egocentric kitchen videos transfers zero-shot to real-robot tasks and, when used in reward-weighted regression over a mix of successful and failed demonstrations, yields higher success rates than rewards derived from R3M or VIP.","pith_inferences":["If the monotonic frame-index progress assumption is the binding constraint, PROGRESSOR should struggle on tasks where success requires revisiting states or cyclic motion; a natural extension would be a multimodal or latent progress model that can represent backtracking.","The same progress-reward recipe could be applied to any domain with ordered video of successful executions—surgical training, sports coaching, or household chores—where the monotonicity assumption approximately holds.","Because the reward compares the current frame to the initial and goal frames rather than to a fixed embedding, PROGRESSOR may show greater resilience to camera viewpoint changes than contrastive distance rewards; a viewpoint-transfer experiment would test this.","A controlled stress test—training on demonstrations that include temporary backtracking and checking whether the reward peaks at the wrong time—would directly isolate the cost of the monotonicity assumption."],"forward_implications":["A policy can be trained end-to-end from pixels with no environment reward, because the predicted progress provides dense credit assignment at every step.","A single reward model pretrained on human video transfers across robotic tasks without task-specific fine-tuning, eliminating per-task reward engineering.","The push-back refinement keeps the reward calibrated during exploration by continuously shrinking rewards on non-expert states, which is what makes online RL viable.","In offline settings with noisy demonstrations, weighting behavior-cloning losses by progress rewards makes the policy resilient to failed trajectories."],"supporting_citations":[{"why":"Supplies the monotonic-progress assumption and the ranking-based reward learning framework that PROGRESSOR extends.","marker":"[38]"},{"why":"The off-policy RL algorithm (DrQ-v2) used in all simulation experiments.","marker":"[40]"},{"why":"Provides the Meta-World benchmark tasks and oracle policies that generate expert demonstrations.","marker":"[41]"},{"why":"The EPIC-KITCHENS egocentric video dataset used to pretrain the reward model for real-world transfer.","marker":"[6]"},{"why":"R3M, a baseline visual-reward model that PROGRESSOR is compared against and outperforms.","marker":"[25]"},{"why":"VIP, a baseline visual-reward model and the source of the offline RL experimental setup that PROGRESSOR uses.","marker":"[22]"},{"why":"ACT, the behavior-cloning backbone that is modified into RWR-ACT via reward weighting.","marker":"[43]"},{"why":"The adversarial training formulation that motivates the push-back refinement.","marker":"[12, 13]"}],"fun_headline_variants":["Video frame order becomes a universal robot reward","Dense rewards from unlabeled video, no hand-tuning","One video-trained reward model beats R3M and VIP","Progress estimator from raw video powers offline RL","Robots learn complex tasks from plain video rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward signal reduces to the assumption that every expert demonstration advances monotonically toward the goal: progress is defined by frame indices, so any successful trajectory that backtracks, pauses, or revisits states will be assigned the wrong progress regardless of network capacity.","fun_headline_variants_meta":{"raw":{"variants":["Video frame order becomes a universal robot reward","Dense rewards from unlabeled video, no hand-tuning","One video-trained reward model beats R3M and VIP","Progress estimator from raw video powers offline RL","Robots learn complex tasks from plain video rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00054,"raw_usage":{"total_tokens":2587,"prompt_tokens":941,"completion_tokens":1646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1573}},"tokens_in":557,"tokens_out":1646,"duration_ms":11827,"temperature":1.0,"reasoning_tokens":1573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:30:37.863741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical DrQ-v2 setup on a task with cyclic observations (for example, stirring or turning a crank), using expert videos that revisit the same states; if PROGRESSOR's reward fails to separate progress from regress and the agent does not learn, the frame-index progress assumption is what breaks. A cheaper check is to take a single successful demonstration with temporary backtracking, compute PROGRESSOR's predicted reward at the backtracked frames, and show it rises when true progress falls.","supporting_citations":[{"cited_title":"Im- age augmentation is all you need: Regularizing deep reinforcement learning from pixels","cited_arxiv_id":null,"evidence_quote":"The off-policy RL algorithm (DrQ-v2) used in all simulation experiments."},{"cited_title":"Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the Meta-World benchmark tasks and oracle policies that generate expert demonstrations."},{"cited_title":"Scaling ego- centric vision: The EPIC-KITCHENS dataset","cited_arxiv_id":null,"evidence_quote":"The EPIC-KITCHENS egocentric video dataset used to pretrain the reward model for real-world transfer."}],"review_version":1}