{"id":"3c33f437-66c1-4627-b9c6-046254b246ec","arxiv_id":"2412.10778","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A video labeling model trained with visual shift contrast, latent future reconstruction, and ground-truth action prediction enables sample-efficient policy cloning from action-free videos on Procgen.","lead":"UPESV is a training method that learns to play video games by watching action-free expert videos and doing a small number of reward-free practice interactions, with no reward or action labels. It reports strong average scores on 16 Procgen games, though its headline claim of winning 12 of 16 tasks is not fully backed by the paper's own results table.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 12/16 superiority count is not supported by Table I: strict pairwise comparison gives 10/16, with the other two wins relying on ties.","rationale":"The central claim of the paper is that UPESV outperforms five baselines on 12/16 Procgen tasks. This claim is directly contradicted by Table I under the usual reading of 'outperforming' as strict superiority. A manual count from Table I gives 10 tasks where UPESV's mean is strictly higher than every baseline mean; Heist and Chaser are exact ties with LAPO (and BCO for Chaser). Thus the advertised '12/16' appears to count ties as wins, which is not 'outperforming'. This is not a minor wording issue; it is the paper's headline quantitative result. The reader's identified weakest assumption about the video labeling model's cross-distribution transfer is also important, but the empirical evidence in Table II and the policy scores partially mitigate that risk. The count discrepancy is a more immediate, checkable factual issue that can be settled from the provided table. The CONDITIONAL verdict remains appropriate: the paper should either correct the count or explicitly define the counting criterion, and release code/data to support reproducibility.","tokens_in":13491,"tokens_out":9421,"duration_ms":70793,"concrete_test":"Re-derive the count from Table I with an explicit criterion: for each task, determine whether UPESV's mean score is strictly greater than the mean score of each of the five baselines (BCO, ILPO, LAPO, PPO, TVL). If ties are excluded, the count is 10/16; if ties are included, it is 12/16. The correct count should be reported unambiguously in the Abstract and Section IV.B, with the definition of 'outperforming' stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that UPESV 'outperforms five current advanced baselines on 12/16 tasks' (Abstract, Section I, and Section IV.B). Table I does not support this count under the standard interpretation of 'outperforms' as strictly higher mean score. Counting from Table I, UPESV's mean strictly exceeds every baseline mean on only 10 tasks: Bigfish, Maze, Coinrun, Dodgeball, Climber, Fruitbot, Ninja, Miner, Leaper, and Bossfight. On Heist, UPESV ties LAPO (9.4 ± 0.3 vs 9.4 ± 0.3); on Chaser, UPESV ties BCO and LAPO (0.8 ± 0.1 vs 0.8 ± 0.0). On Plunder, Jumper, Starpilot, and Caveflyer, UPESV is strictly worse than at least one baseline. Therefore, the advertised '12/16' appears to include ties as wins, which conflicts with the wording 'outperforming'. This discrepancy is load-bearing because the quantitative superiority count is the paper's primary evidence for state-of-the-art performance. If the count is corrected to 10/16, the headline result remains positive but is meaningfully weaker. The reader's selected weakest assumption (cross-distribution generalization of the video labeling model) is a legitimate risk, but the count issue is a direct, internal inconsistency that can be resolved from the paper's own table.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UPESV, a framework for learning policies from action-free expert videos without rewards or expert action labels. A video labeling model is trained with three self-supervised objectives: visual shift contrast (VSC), latent future reconstruction (LFR), and ground-truth action prediction (GAP) on reward-free transitions collected by the policy itself. The labeling model's predicted latent actions on expert videos are used as targets for behavior cloning a policy, which in turn collects more transitions for GAP. Experiments on 16 Procgen environments with 100k interactions compare against BCO, ILPO, LAPO, PPO, and TVL, and the paper claims state-of-the-art performance on 12/16 tasks.","tokens_in":13880,"tokens_out":13117,"duration_ms":100633,"significance":"If substantiated, the paper would provide a practical recipe for leveraging large action-free video corpora with only a small number of environment interactions, combining representation learning, dynamics modeling, and inverse action prediction. The experimental study is broad (16 tasks, ablations, hyperparameter sensitivity, and a robustness check on two unseen expert datasets), and the method achieves the highest mean score (9.0 vs 6.8 for the runner-up LAPO). These are concrete strengths. However, the headline win count is not reproducible from the paper's own Table I, and the direct evidence for the labeling model's cross-distribution generalization is limited to two tasks. The core idea is sound and the results are positive, but the claims need to be calibrated and the evidence for the mechanism strengthened.","major_comments":[{"comment":"The claim that UPESV 'outperforms five current advanced baselines on 12/16 tasks' is not supported by strict pairwise comparison of the mean scores reported in Table I. Under the standard interpretation of 'outperforms' as strictly higher mean score, UPESV beats every baseline on only 10 tasks (Bigfish, Maze, Coinrun, Dodgeball, Climber, Fruitbot, Ninja, Miner, Leaper, Bossfight). On Heist, UPESV ties LAPO (9.4±0.3 vs 9.4±0.3); on Chaser, it ties BCO and LAPO (0.8±0.1 vs 0.8±0.0 and 0.8±0.0); and on Plunder, Jumper, Starpilot, and Caveflyer, UPESV is strictly worse than at least one baseline. The same issue affects the Section IV.B statement about three video-based baselines: strict comparison gives 10/16, not 12/16. Please correct the count in the abstract, introduction, and Section IV.B, or explicitly define the metric (e.g., 'best or tied-best on 12/16 tasks').","section":"Section IV.B, Table I, Abstract"},{"comment":"The core mechanism of UPESV is that the video labeling model V, trained partly on ground-truth actions from non-expert reward-free transitions (GAP), generalizes to label expert videos correctly, and that the cloned policy (Eq. 6) inherits this accuracy. The only direct evidence for this cross-distribution transfer is Table II, which reports action prediction accuracy on just two unseen expert datasets (Starpilot and Miner). Because the policy's success on the remaining 14 tasks depends on V's labeling quality on expert videos for those tasks, the paper should either report labeling accuracy on expert videos for a broader set of tasks or provide a clear argument (e.g., with additional analysis) that the Task I policy results already validate the labeling model on all tasks.","section":"Section III.D, Table II"},{"comment":"The written objectives for the two standard losses do not match their descriptions. Eq. (3) is supposed to be an InfoNCE loss, but as printed it lacks the exponential and temperature scaling and sums over all j including the positive pair, which does not correspond to any standard contrastive loss. Eq. (5) is described as a cross-entropy between predicted and ground-truth action vectors, but the arguments of the logarithm are reversed: the loss should be -Σ a^e_{i,(c)} log a^{e,V}_{i,(c)}. If these are typesetting errors, they should be corrected, because as written the equations would not train the model as intended.","section":"Eqs. (3) and (5)"}],"minor_comments":[{"comment":"The label 'UPESV v/o VSC' should read 'UPESV w/o VSC'.","section":"Section IV.D, Fig. 4"},{"comment":"The description of the expert videos says they contain 8M steps; please clarify whether these are frames or transitions, and how many distinct video episodes are used.","section":"Section IV.A"},{"comment":"Please report the number of transitions used for the accuracy evaluation and specify whether the 'unseen' expert datasets are disjoint from those used during training.","section":"Table II"},{"comment":"The EMA momentum is set to 0.05, which is unusually small for a momentum encoder; please confirm this value and cite a reference for the chosen range.","section":"Section III.A"},{"comment":"The cosine similarity Gc is used directly in the loss (Eq. 3) without a temperature parameter; if this is intentional, state so explicitly.","section":"Section III.A, Eq. (2)"},{"comment":"The ablation study is shown for only eight of the sixteen tasks; please justify the selection or provide results for all tasks.","section":"Section IV.C, Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The 12/16 count issue is the main blocker; the rest is fixable. The paper has a solid experimental foundation, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper has a load-bearing internal inconsistency in its headline claim. It says UPESV outperforms baselines on 12/16 tasks; Table I gives strict wins on only 10/16, with two ties and four losses. That's the main issue.\n\nWhat is actually new: the visual shift contrast task (VSC) is a genuinely new objective, and the combination of three self-supervised tasks—contrastive shift matching, latent future reconstruction, ground-truth action prediction—is a reasonable engineering contribution. The experiments are extensive: 16 Procgen tasks, ablations on 8, hyperparameter analysis, and an unseen-expert test on 2 tasks. The ablations show each task matters, and the method does beat the baselines on mean score overall (9.0 vs 6.8 best baseline). The paper is honestly written about its limitations: computational burden, learning-rate balancing, discrete-control gap.\n\nSoft spots: First, the 12/16 count. Counting strict wins from Table I, UPESV wins 10 tasks, ties two (Heist with LAPO, Chaser with BCO and LAPO), and loses four (Plunder, Jumper, Starpilot, Caveflyer) to at least one baseline. The abstract and Section IV.B repeat '12/16' without defining 'outperforming' to include ties. This is the paper's primary evidence for SOTA, so it needs fixing. Second, the weakest assumption is cross-distribution generalization: V is trained on non-expert transitions (GAP task) but used to label expert videos. Table II only checks two unseen expert datasets (Starpilot, Miner), not all 16, and on Starpilot the variance is huge (12.8 ± 13.9 for BCO), so the comparison is weak. Third, no code or data release, and several comparisons fall within error bars (e.g., Maze 9.7±0.2 vs 9.6±0.1; Leaper 2.9±0.3 vs 2.7±0.2). That said, the central mechanism is plausible and the average improvement is clear.\n\nWho this is for: people working on learning-from-observation, video-based RL, and sample-efficient imitation. It deserves serious peer review; the count must be corrected and the generalization question addressed, but the method and experiments are real. I'd send it to review, with a request for code or per-task seeds.","headline":"A solid empirical extension in video-based policy cloning, but the headline 12/16 superiority count is not backed by the paper's own Table I (strict count is 10/16).","tokens_in":14323,"tokens_out":1475,"would_cite":true,"duration_ms":12174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video labeling model trained on three self-supervised tasks can clone policies from action-free videos using only 100k interactions.","keywords":["unsupervised policy learning","learning from videos","behavior cloning from observation","self-supervised learning","inverse dynamics model","action-free videos","sample-efficient reinforcement learning","procedurally generated environments"],"falsifier":"Run UPESV with held-out true action labels for expert videos on all sixteen tasks and check whether the labeling model's action-prediction accuracy on those expert videos is above chance and correlates with the cloned policy's score; if accuracy is near chance on tasks where the policy fails, the cross-distribution transfer from non-expert transitions to expert videos is not holding.","tokens_in":13318,"feed_emoji":"🎮","tokens_out":8066,"duration_ms":67695,"temperature":0.7,"pith_summary":"This paper claims that a policy can be learned from action-free expert videos, with no rewards and no action labels, if a video labeling model is trained jointly on three self-supervised tasks that use both the videos and a small number of reward-free interactions. The labeling model infers the expert action between neighboring video frames as a latent action, and a policy is obtained by behavior cloning those inferred actions. The authors report that this pipeline, called UPESV, beats five established baselines on 12 of 16 procedurally generated video-game tasks when every method is limited to 100,000 environment interactions. The broader point is that easily available video demonstrations could substitute for expensive expert supervision in sample-limited settings.","feed_headline":"Video-only policy learning beats five baselines on 12 of 16 tasks","feed_subtitle":"Self-supervised labeling of expert videos clones a working policy with only 100k interactions and no rewards.","key_machinery":"Three self-supervised tasks carried by a shared inverse-dynamics labeling model: visual shift contrast (VSC), latent future reconstruction (LFR), and ground-truth action prediction (GAP). VSC applies a contrastive InfoNCE loss to positive pairs formed by two shifted crops of the same frame, teaching the encoder $f$ to ignore absolute positions and keep relative motion; LFR runs the predicted latent action through a discrete world model that must reconstruct the next latent observation, teaching $g$ to produce actions that explain observed changes; GAP minimizes cross-entropy between predicted actions and true actions on reward-free transitions, anchoring the latent action space to the real action space. The same encoder and action projector are reused by the cloned policy, so the labeling model and policy improve in alternation: the policy clones video-derived latent actions, interacts with the environment, and returns transitions that strengthen GAP.","core_discovery":"UPESV's central object is the video labeling model $V(a_i^v \\mid o_i^v, o_{i+1}^v)$, an inverse dynamics model that maps neighboring video frames to a latent action via $a_i^v = h(g(f(o_i^v), f(o_{i+1}^v)))$, where $f$ is a feature encoder, $g$ a latent predictor, and $h$ an action projector. The model is trained with three objectives: a visual shift contrast task that aligns two randomly shifted versions of the same image in latent space, forcing the encoder to attend to relative rather than absolute visual differences; a latent future reconstruction task that trains a world model to reconstruct the next latent observation from the current one plus the predicted latent action, forcing the labeling model to understand environmental dynamics; and a ground-truth action prediction task on reward-free transitions, which aligns predicted actions to the true action space and acts as a cross-distribution generalization signal. The policy shares the encoder and action projector with the labeling model and is cloned by matching its latent action to the video-derived latent action; as the policy improves, it collects new reward-free transitions that enrich the ground-truth action prediction data. On sixteen procedurally generated environments with 100k interaction steps, the authors report state-of-the-art results, outperforming five baselines on 12/16 tasks and producing the only effective policy on the Fruitbot task.","pith_inferences":["A testable extension the paper does not run is measuring labeling accuracy on held-out expert videos for all sixteen tasks, not just two, to check whether per-task policy performance tracks per-task labeling accuracy.","The iterative labeling-and-cloning loop is a form of pseudo-labeling, so a natural concern the paper leaves implicit is confirmation bias: if early labeling errors are consistent, the policy may reinforce them when collecting new transitions.","Because the visual shift contrast task helped mainly by muting absolute-position differences, the same relative-difference prior could be injected into other inverse dynamics or world-model learners, not only this architecture.","The paper's own limitation section notes the gap to continuous control and real robots; a direct next step would be testing UPESV on continuous-action domains, where the discrete latent action and the shared action projector would be stressed."],"forward_implications":["If the central claim holds, action-free videos become a usable supervision source for visually rich, procedurally varied environments, reducing reliance on expert action labels and hand-designed rewards.","The three-task combination should beat methods that use only one data source: using only non-expert transitions for labeling, or only expert videos for dynamics, leaves information on the table.","Because the policy shares the encoder and action projector with the labeling model, improvements to video labeling directly translate into policy improvements without extra parameters.","The reported held-out action-prediction accuracy on two unseen expert datasets suggests the labeling model transfers to new expert videos, implying that the pipeline does not merely memorize its training videos."],"supporting_citations":[{"why":"Baseline for learning from observation that labels videos using only non-expert transitions; UPESV is compared against it.","marker":"[28]"},{"why":"Baseline that extracts dynamics only from expert videos through a world model; provides the comparison for using both data sources.","marker":"[29]"},{"why":"Supplies the expert video dataset used by video-based methods and the LAPO pretraining baseline that UPESV outperforms under 100k steps.","marker":"[55]"},{"why":"Reward-based RL baseline on the same environments; its lower scores anchor the claim that video-only supervision can beat reward-based methods under limited interactions.","marker":"[56]"},{"why":"Second reward-based RL baseline used for comparison on the sixteen environments.","marker":"[57]"},{"why":"Provides the contrastive learning formulation used by the visual shift contrast task.","marker":"[52]"},{"why":"Provides the InfoNCE loss used as the objective of the visual shift contrast task.","marker":"[53]"},{"why":"Provides vector quantization, used to discretize the predicted latent action in the latent future reconstruction task.","marker":"[54]"}],"fun_headline_variants":["Video-only policy learning wins 12 of 16 tasks","Unsupervised policy from videos beats five baselines","No rewards, just videos: policy wins 12 of 16","Self-supervised video labeling clones policies efficiently","Sample-efficient unsupervised video policy outperforms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the labeling model, trained to predict actions on non-expert reward-free transitions, also predicts the true actions in expert videos well enough that behavior cloning the resulting labels yields a good policy.","fun_headline_variants_meta":{"raw":{"variants":["Video-only policy learning wins 12 of 16 tasks","Unsupervised policy from videos beats five baselines","No rewards, just videos: policy wins 12 of 16","Self-supervised video labeling clones policies efficiently","Sample-efficient unsupervised video policy outperforms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":3055,"prompt_tokens":1106,"completion_tokens":1949,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":1874}},"tokens_in":722,"tokens_out":1949,"duration_ms":13375,"temperature":1.0,"reasoning_tokens":1874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:24.156261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run UPESV with held-out true action labels for expert videos on all sixteen tasks and check whether the labeling model's action-prediction accuracy on those expert videos is above chance and correlates with the cloned policy's score; if accuracy is near chance on tasks where the policy fails, the cross-distribution transfer from non-expert transitions to expert videos is not holding.","supporting_citations":[{"cited_title":"Imitating latent policies from observation,","cited_arxiv_id":null,"evidence_quote":"Baseline that extracts dynamics only from expert videos through a world model; provides the comparison for using both data sources."},{"cited_title":"Learning to act without actions,","cited_arxiv_id":null,"evidence_quote":"Supplies the expert video dataset used by video-based methods and the LAPO pretraining baseline that UPESV outperforms under 100k steps."},{"cited_title":"Policy gradient without boostrapping via truncated value learning,","cited_arxiv_id":null,"evidence_quote":"Second reward-based RL baseline used for comparison on the sixteen environments."},{"cited_title":"A simple frame- work for contrastive learning of visual representations,","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive learning formulation used by the visual shift contrast task."},{"cited_title":"On mutual information maximization for representation learning,","cited_arxiv_id":null,"evidence_quote":"Provides the InfoNCE loss used as the objective of the visual shift contrast task."},{"cited_title":"Neural discrete representa- tion learning,","cited_arxiv_id":null,"evidence_quote":"Provides vector quantization, used to discretize the predicted latent action in the latent future reconstruction task."}],"review_version":1}