{"id":"b95cf6de-0054-42be-8381-dafed8d6534d","arxiv_id":"2505.06482","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"VeoRL, which aligns real-action rollouts with latent-behavior rollouts learned from unlabeled videos, reports substantial improvements over offline visual RL baselines across three benchmarks.","lead":"An offline RL method, VeoRL, trains a two-branch world model: one predicts next states from real actions, the other from discrete latent \"behavior\" codes learned from unlabeled internet videos. An intrinsic reward aligns the two branches, and the paper reports large performance gains over offline RL baselines in Meta-World, CARLA, and MineDojo tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Intrinsic reward in Eq. (5) takes a Euclidean distance between trunk and plan latent states that are never placed in a common metric space; MMD aligns only image encoder outputs, so the subgoal mechanism central to the video-transfer claim is not supported by the reported experiments.","rationale":"The reported headline numbers are internally consistent, and the ablations (no-video, MMD, latent-behavior inputs) point in the expected direction, so I do not object to the empirical claim as stated. The single most load-bearing gap is that the mechanism connecting unlabeled videos to policy improvement is Eq. (5), and its prerequisite, a common metric between trunk and plan latent states, is never checked or enforced. The reader's weakest assumption named MMD transfer; my concern is sharper: even with perfect MMD alignment of image embeddings, the full RSSM state spaces are not constrained to be comparable, so the Euclidean intrinsic reward may be an unvalidated regularizer rather than a subgoal objective. This is testable and does not by itself falsify the empirical results, so the verdict remains CONDITIONAL rather than moving to REJECT.","tokens_in":31883,"tokens_out":5934,"duration_ms":66200,"concrete_test":"On a trained VeoRL model for Meta-World Handle Press, collect paired (s_t, s_bar_t) from simultaneous trunk and plan rollouts at matched time steps, plus random permutation pairs, and compare mean ||s_t - s_bar_t||^2. If true pairs are not significantly closer than permuted pairs, the distance in Eq. (5) carries no subgoal information. Then rerun the main comparison with the intrinsic reward replaced by its empirical mean (a constant). If most of the gain over DreamerV2 survives, the video-guided subgoal mechanism is not the cause of the reported improvements, and the central claim loses its mechanism-specific support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. (5) in Section 3.3: r_bar_t = -||s_t - s_bar_t||^2, where s_t is a trunk-net RSSM state and s_bar_t is a plan-net RSSM state. These two branches have separate GRU/RSSM parameters and separate priors (Eq. 2); only the image encoder and decoder are shared. The training losses in Eqs. (3) and (4) regularize each branch to its own prior, and the MMD loss of Section 3.1 is applied to image embeddings e_t, not to the full state pairs (h_t,z_t) and (h_bar_t,z_bar_t). Nothing in the objectives establishes that the two latent spaces share a coordinate frame in which Euclidean distance is a meaningful subgoal metric. This matters because Eq. (6) injects omega * r_bar_t into every value target; if the distance mixes incomparable coordinates, the intrinsic reward is not a faithful 'video-guided behavior alignment' signal and could distort value estimates while still improving empirical scores for other reasons. The reported evidence for domain transfer, namely Figure 8's t-SNE of latent behavior/action clusters and Table 7's MMD ablation, does not measure whether plan-net states are useful subgoals in the trunk state space. Without such a check, the central mechanistic claim that unlabeled videos improve offline RL through model-based behavior guidance is only weakly supported, even if the reported empirical gains are real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VeoRL, a model-based offline visual RL method that augments a target offline dataset with unlabeled videos from a source domain. It trains a vector-quantized behavior abstraction network (BAN) on target-frame pairs to produce discrete latent behaviors, applies an MMD loss to align source and target image encoder features, and builds a two-stream world model: a trunk net driven by real actions and rewards, and a plan net driven by latent behaviors. Policy and value networks are conditioned on the inferred latent behavior, and an intrinsic reward (Eq. 5) is defined as the negative Euclidean distance between trunk and plan latent states and injected into the value targets (Eq. 6). Experiments on Meta-World, CARLA, and MineDojo report large gains over DreamerV2 and other baselines, with ablations isolating the contributions of the latent behavior input, the intrinsic reward, and the auxiliary videos, plus offline-to-online transfer experiments and qualitative analyses of latent behaviors.","tokens_in":32301,"tokens_out":6838,"duration_ms":72813,"significance":"If the results hold, VeoRL is a significant contribution: it targets a practically important problem (using action-free, unlabeled videos to improve offline visual RL), situates itself against relevant model-based, offline, and video-pretraining baselines, and reports large gains on external benchmarks. The evaluation design is largely non-circular: performance is measured on held-out task episodes, and the ablations show that each of the three proposed components matters. The main weakness is that the central mechanism—the intrinsic reward in Eq. (5)—relies on an unverified geometric assumption about the two latent state spaces, and some of the headline empirical claims lack tabulated numerical detail. With additional evidence for the mechanism and fuller reporting, this would be a strong paper.","major_comments":[{"comment":"The intrinsic reward is defined as the L2 distance between a trunk-net state s_t and a plan-net state \\bar{s}_t. These states are produced by two RSSMs with separate GRU, prior, and posterior parameters; only the image encoder and decoder are shared (Eq. 2). The MMD loss in Section 3.1 is applied to image embeddings e_t, not to the state pairs (h_t,z_t) and (\\bar{h}_t,\\bar{z}_t), and Objectives (3) and (4) separately regularize each branch toward its own prior. Nothing in the objectives establishes that Euclidean distance in this combined latent space is a meaningful subgoal error. Because Eq. (6) adds \\omega \\bar{r}_t to every value target, the paper's central claim that video-guided behavior alignment improves value learning rests on this unverified assumption. I request either (i) a theoretical argument that the shared decoder/encoder and KL structure align the two state spaces into a common metric frame, or (ii) an empirical validation, such as showing that minimizing \\|s_t - \\bar{s}_t\\| correlates with task-relevant progress, or comparing against a control where \\bar{s}_t is replaced by matched random plan states with the same reward scale. This experiment is necessary to support the mechanistic interpretation; without it, the reported gains could be driven by other aspects of the two-branch training.","section":"Section 3.3, Eq. (5)"},{"comment":"The CARLA result ('as much as 350%' over DreamerV2) and the MineDojo improvements (0.10 to 0.35, 0.36 to 0.83, 0.27 to 0.55) are stated as text percentages or point values, but the paper does not provide a table with the mean episode returns or success rates with standard deviations for all baselines. Given that the paper emphasizes 3 random seeds and 50 evaluation episodes, the error bars are important for assessing whether the gains overlap across methods. Please add tabulated numeric results for CARLA and MineDojo, including all baselines and standard deviations, or explicitly point to where such tables appear in the appendix.","section":"Section 4.1, Figures 4 and 5"},{"comment":"The intrinsic-reward weight \\omega differs by orders of magnitude across environments (0.05 for Meta-World, 0.1 for CARLA, 1e-5 for MineDojo), and no sensitivity analysis for \\omega is reported. Since this hyperparameter directly controls how much the proposed intrinsic reward influences the value target, the reader cannot tell whether the method is robust or whether the reported results depend on a narrow range of \\omega. Please include a sensitivity sweep for \\omega on at least one continuous-control task and one discrete-control task, or provide a principled scheme for setting it.","section":"Table 2 and Section 4.2"}],"minor_comments":[{"comment":"The behavior-cloning term in the plan-net objective is written as -\\ln p(\\bar{a}_t | s_t), but Eq. (2) and the surrounding text define the behavior cloning module as \\bar{a}_t = FBC(\\bar{s}_t); this should presumably be s_t corrected to \\bar{s}_t.","section":"Eq. (4)"},{"comment":"The baseline name 'LAMPO' appears twice in the second design consideration; it should be 'LOMPO' to match the rest of the paper.","section":"Section 4.3"},{"comment":"The claim that VeoRL 'consistently achieves the best performance' in success rate is contradicted by Table 3 for Handle Pull, where VIP reports 0.36 ± 0.15 and VeoRL reports 0.30 ± 0.10, and for Handle Press, where VeoRL ties with DrQ+CQL at 0.60. Please qualify the claim or provide a statistical test of differences.","section":"Section 4.1 and Table 3"},{"comment":"The t-SNE visualization is qualitative; it would be more convincing to report a quantitative correspondence measure between latent behaviors and real-action clusters, such as cluster purity or normalized mutual information computed on held-out target trajectories.","section":"Section 4.4, Figure 8"},{"comment":"Line 5 of Algorithm 1 trains BAN only on target observations, while source frames enter through the MMD loss and later plan-net training. This is clear in the text but should be stated explicitly in Algorithm 1 to avoid the impression that BAN is trained jointly on source videos.","section":"Algorithm 1 and Section 3.1"},{"comment":"There is a stray dot in the typeset equation 'V^λ_t ˙ ='; please fix the notation.","section":"Eq. (6)"},{"comment":"The limitations paragraph discusses computational overhead only. In light of the major comment above, the paper should also acknowledge the assumption that the trunk and plan latent spaces are comparable and discuss when this assumption could fail.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This is a promising paper with a strong empirical story and a clear set of ablations. My recommendation is driven by the unvalidated state-space metric in Eq. (5), which is load-bearing for the claimed mechanism, and by the absence of tabulated results for two of the three main benchmarks. Both issues are addressable within the scope of a revision. I see no scientific-ethics or novelty concern; the work is positioned appropriately against the literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VeoRL is a genuinely new method for offline visual RL that uses unlabeled video, with large reported gains on Meta-World, CARLA, and MineDojo. It deserves a read, but the core subgoal mechanism rests on a distance that the paper never shows is well-defined.\n\nThe architecture is clever: a VQ-based behavior abstraction network (BAN) learns a discrete latent action space from unlabeled video; then a two-stream world model has a \"trunk net\" driven by real actions and a \"plan net\" driven by latent behaviors. The policy and value function are conditioned on the latent behaviors, and an intrinsic reward is built from the distance between trunk and plan states. This is not a re-hash of APV, VIP, VPT, or LOMPO — the discrete latent-action construction is the paper's own.\n\nThe empirical work is solid in many respects: three diverse benchmarks, ablations that isolate each component (videos, latent behavior input, intrinsic reward, MMD), and large gains (e.g., Handle Press return 1201→2650; MineDojo Harvest water success 0.36→0.83). The ablations in Figure 7 and Table 7 show each piece contributes. They also report a limitations section (compute overhead) and an interesting robustness check on low-quality offline data.\n\nThe soft spots are in proportion. The stress-test note about Eq. (5) holds up: the trunk and plan nets have separate parameters, MMD aligns only the image embeddings, not the full latent states, and nothing puts s_t and ̄s_t in a common metric space. So the intrinsic reward is a distance between coordinates of unknown semantic alignment. The paper doesn't check whether plan states are actually good subgoals in the trunk state space (e.g., whether the distance correlates with task progress or rollouts). This weakens the mechanistic claim, though not the empirical result. It could be that the intrinsic reward acts as a regularizer or auxiliary signal, which is fine but should be stated honestly.\n\nOther concerns: no code or datasets released; the intrinsic reward weight ω is tuned per environment (0.05, 0.1, 1e-5) with no sensitivity analysis; CARLA and MineDojo numbers are only in figures without error bars in the text. Minor: the paper refers to the LOMPO baseline as \"LAMPO\" in Section 4.3.\n\nOverall, this is a paper worth engaging with. The empirical contributions are real, and the method is novel. The mechanism needs better evidence or a reframing. If I were handling it, I'd send it to review with a request for: (1) a direct check of the subgoal space (or rephrase the intrinsic reward as a regularization term), (2) code and data release, and (3) error bars for the non-MetaWorld results. Not a desk reject.","headline":"VeoRL is a novel method with strong empirical gains from unlabeled video, but the paper never shows the latent subgoal distance in its intrinsic reward is semantically meaningful.","tokens_in":32782,"tokens_out":3761,"would_cite":true,"duration_ms":37876,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that unlabeled videos, converted into discrete latent behavior codes, can more than double offline reinforcement learning performance across manipulation, driving, and Minecraft tasks.","keywords":["video-enhanced offline reinforcement learning","world models","latent behavior abstraction","vector quantization","intrinsic reward","domain adaptation","visual control","model-based RL"],"falsifier":"A direct test would be to run VeoRL exactly as specified with a source video collection deliberately chosen to contain no behavior related to the target task, such as random scenery or unrelated object footage. If the intrinsic reward $\\bar{r}_t = -\\|s_t - \\bar{s}_t\\|^2$ still produces large gains, then the improvement is not caused by transferring video behaviors and the paper's central mechanism is falsified.","tokens_in":31688,"feed_emoji":"📹","tokens_out":7203,"duration_ms":61810,"temperature":0.7,"pith_summary":"The paper is trying to establish that offline reinforcement learning, which normally learns only from a fixed static dataset, can be substantially improved by also training on unlabeled videos from a different domain. The proposed method, VeoRL, learns a discrete set of 'latent behaviors' from videos, builds a second world-model branch that predicts long-term state transitions from those behaviors, and adds an intrinsic reward that keeps the agent's real-action rollouts close to the video-informed plan. On Meta-World robotic manipulation, CARLA autonomous driving, and MineDojo games, VeoRL reports gains over 100% in several tasks, with specific jumps such as Handle Press return from 1201 to 2650 and Harvest water with bucket success rate from 0.36 to 0.83. If this holds, it matters because unlabeled video is cheap and abundant, while reward-labeled interaction data is expensive, so the method offers a practical path to injecting commonsense physical knowledge into agents without annotations.","feed_headline":"Watching unlabeled videos more than doubles offline RL scores","feed_subtitle":"A two-branch world model converts video behaviors into subgoals, lifting scores by 100% or more.","key_machinery":"The central object is the two-stream world model: a trunk net driven by real actions (trained only on the target offline dataset) and a plan net driven by learned latent behaviors (trained on both the target data and the source videos), sharing the image encoder and decoder but with separate dynamics parameters. The load-bearing mechanism is the intrinsic reward $\\bar{r}_t = -\\|s_t - \\bar{s}_t\\|^2$, the negative squared distance between the trunk net's short-term state and the plan net's long-term state; minimizing it aligns action-driven rollouts with video-informed subgoals. The latent behavior codebook from the behavior abstraction network is what makes the plan net trainable without action labels, and the maximum-mean-discrepancy loss is what lets those codes bridge the source-target domain gap.","core_discovery":"VeoRL is a model-based offline RL algorithm whose central claim is that task-agnostic, unlabeled video can be converted into high-level behavior guidance that improves a target policy. A behavior abstraction network (BAN) uses vector quantization to map pairs of consecutive observations to one of $K$ discrete latent behavior codes, trained jointly with a video-prediction plan net on both the target offline data and the source videos, while a maximum-mean-discrepancy loss aligns the source and target visual embeddings. The world model then has two state-transition branches: a trunk net that predicts next states and rewards from real actions using only the target dataset, and a plan net that predicts long-term states from latent behaviors. During imagination the policy and value networks are conditioned on the behavior codes, and an intrinsic reward $\\bar{r}_t = -\\|s_t - \\bar{s}_t\\|^2$ pushes the trunk net's short-term rollouts toward the plan net's long-term predictions. The paper argues that this alignment transfers the control policies and physical dynamics embedded in the videos, and that this is what produces the reported performance gains.","pith_inferences":["Beyond the paper, the mechanism suggests a research direction worth testing: performance should degrade smoothly as the source videos become less task-relevant, and the paper's own video-count ablation points in that direction.","A scale-invariant or normalized version of the intrinsic reward might reduce the need for the per-benchmark weight $\\omega$, since state-feature scales differ strongly across environments.","The discrete latent behavior space could be the key transferable currency; if so, other architectures that learn such codes from video should show similar benefits, independent of the specific world-model design.","A stronger domain-alignment objective than MMD, such as adversarial alignment, might close the source-target gap further, but the paper does not test this."],"forward_implications":["Offline visual RL agents can be improved with cheap unlabeled video instead of additional reward-labeled interaction data.","The same algorithm transfers behavioral knowledge across very different benchmarks: robot manipulation, driving, and Minecraft.","Value learning relies less on sparse environmental rewards, which should reduce the overestimation bias that hurts offline RL.","Latent behavior codes trained on one task can be reused on another task with minimal degradation, lowering the cost of adapting to new tasks.","Offline-to-online fine-tuning starts from a video-informed policy and converges faster when the agent is later allowed to interact."],"supporting_citations":[{"why":"Supplies the recurrent state-space model used as the dynamics backbone for both the behavior abstraction network and the world model.","marker":"Hafner et al., 2019"},{"why":"DreamerV2 is the model-based RL backbone and the primary baseline that VeoRL is compared against and built from.","marker":"Hafner et al., 2021"},{"why":"Supplies the VQ-VAE vector quantization objective that creates the discrete latent behavior codebook.","marker":"Van Den Oord et al., 2017"},{"why":"Supplies the maximum mean discrepancy loss used to align source-video and target-data visual embeddings.","marker":"Borgwardt et al., 2006"},{"why":"Supplies BridgeData-V2, the unlabeled real-robot video source for the Meta-World experiments.","marker":"Walke et al., 2023"},{"why":"Supplies NuScenes, the unlabeled driving video source for the CARLA experiments.","marker":"Caesar et al., 2019"},{"why":"Supplies VPT, a video-pretraining baseline that assigns pseudo-actions via an inverse dynamics model, contrasted with the unsupervised latent-behavior approach on Minecraft.","marker":"Baker et al., 2022"},{"why":"Supplies CQL, a model-free conservative Q-learning baseline for offline RL comparison.","marker":"Kumar et al., 2020"},{"why":"Supplies LOMPO, a model-based offline visual RL baseline that handles dynamics uncertainty with reward penalties.","marker":"Rafailov et al., 2021"}],"fun_headline_variants":["Watching videos boosts offline RL by 100% or more","Offline RL learns from unlabeled videos to double scores","Video-enhanced offline RL: model-based gains over 100%","VeoRL turns video data into offline RL performance boosts","Unlabeled videos provide 2x offline RL improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the behaviors visible in unrelated videos, after aligning their visual features with the target, correspond to actions that transfer, and that the long-term states predicted from those behaviors are useful goals rather than misleading states for the target task.","fun_headline_variants_meta":{"raw":{"variants":["Watching videos boosts offline RL by 100% or more","Offline RL learns from unlabeled videos to double scores","Video-enhanced offline RL: model-based gains over 100%","VeoRL turns video data into offline RL performance boosts","Unlabeled videos provide 2x offline RL improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1829,"prompt_tokens":884,"completion_tokens":945,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":863}},"tokens_in":500,"tokens_out":945,"duration_ms":8533,"temperature":1.0,"reasoning_tokens":863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:41:42.237395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run VeoRL exactly as specified with a source video collection deliberately chosen to contain no behavior related to the target task, such as random scenery or unrelated object footage. If the intrinsic reward $\\bar{r}_t = -\\|s_t - \\bar{s}_t\\|^2$ still produces large gains, then the improvement is not caused by transferring video behaviors and the paper's central mechanism is falsified.","supporting_citations":[],"review_version":1}