{"id":"08e6549e-6286-4ad6-9991-84e7c9fac330","arxiv_id":"2607.16602","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"PAVXploreRL post-trains action-conditioned world models with VJEPA-2 latent rewards and perturbed 'OOD' actions, reporting a 5.6% average gain and lowered policy-overestimation bias.","lead":"This paper adds a reinforcement-learning fine-tuning stage to action-conditioned video world models, using VJEPA-2 latent predictions as rewards and noise-perturbed robot actions to explore out-of-distribution behavior. The authors report consistent metric gains over pretrained baselines and reduced overestimation when the world model evaluates robot policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PAV-improvement claim is not independently validated: the same VJEPA-2 model supplies both the reward and the headline evaluation metrics, so reported gains may reflect proxy optimization rather than genuine physical plausibility or action adherence.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: VJEPA-2 is both the reward model and the principal judge of physical plausibility and action adherence. I agree with that identification. The paper's own evidence contains one genuinely independent signal — the real-world policy-evaluation comparison in Section 4.3 — but it is narrow and small-scale, so it does not resolve the circularity for the general PAV claim. Meanwhile, the non-VJEPA metrics (PSNR/SSIM/LPIPS/EPE/COS) support a weaker claim about visual fidelity and flow consistency, not the full PAV statement. The right verdict remains CONDITIONAL: the method is plausible and partly supported, but the central claim needs an external validation metric, release of code/checkpoints, and error bars before it can be accepted. My stress-test therefore does not change the reader's CONDITIONAL verdict; it sharpens why the condition should be imposed.","tokens_in":14856,"tokens_out":5792,"duration_ms":66581,"concrete_test":"Run a blinded evaluation on paired PT vs RL rollouts from the Agibot and Droid validation sets using an external, reward-independent metric: (1) a frozen off-the-shelf inverse dynamics model predicts actions from generated frames and action-error is computed; (2) an object tracker measures physical violations such as object disappearance or teleportation during occlusions. If RL does not significantly outperform PT on these external metrics, the PAV claim reduces to VJEPA-2 proxy optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RL post-training produces 'higher-quality PAV properties' rests on metrics that are partly circular. r_VJEPA-P (Eq. 8) is a negative L1 distance between the world model's encoded predictions and VJEPA-2's latent prediction, and the 'VJEPA Pred' row in Table 1 is exactly the quantity RL maximizes. Likewise, r_VJEPA-E (Eq. 7) uses the same encoder E, so 'Enc' improvements are also expected by construction. The non-circular Table 1 metrics (PSNR, SSIM, LPIPS, EPE, COS) do show visual/flow gains, which are genuine but do not validate Physical Plausibility or Action Adherence. The P and A components are assessed either with VJEPA-based losses or with qualitative examples; no independent physical metric (object permanence, contact consistency, inverse-dynamics action reconstruction) or human study is provided. Appendix A.5's offline diagnostic shows the reward function picks the GT video only 82% of the time in the best configuration, so VJEPA-2 is a noisy judge, and the coefficient selection is performed on the same validation distribution. The VLA evaluator experiment (Fig. 4) is independent and encouraging, but it is limited to policy overestimation on two policies, five tasks, and ten episodes per task, not a general validation of PAV properties. The conclusion only lists multi-view consistency and training time as limitations, not this reward/evaluation circularity. Without an external yardstick, the paper's strongest PAV claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PAVXploreRL, a reinforcement-learning post-training method for action-conditioned world models. It builds on a pretrained video-diffusion world model (Wan2.2-TI2V-5B with action conditioning) and fine-tunes VJEPA-2 on robot data to provide latent rewards for physical plausibility, action adherence, and visual fidelity. The reward function combines pixel-level metrics, VJEPA encoder/predictor losses, embodiment-focused VJEPA rewards, and a reference-free visual-fidelity reward. Training uses both in-distribution action-video pairs and out-of-distribution (OOD) actions generated by perturbing expert actions, without paired video supervision. Experiments on Agibot and DROID report consistent gains over the pretrained baseline (5.6% average improvement across benchmarks) and a policy-evaluation experiment showing reduced overestimation compared to Ctrl-World. The paper claims that RL post-training explicitly optimizes PAV objectives and yields higher-quality PAV properties.","tokens_in":15299,"tokens_out":2376,"duration_ms":24880,"significance":"If the central claim holds, the paper would provide a practical recipe for improving action-conditioned world models as policy evaluators, which is an important problem in embodied AI. The paper has several concrete strengths: it releases code, performs ablations over reward components and coefficients, reports validation curves showing that RL gains are not from additional SFT alone, and includes an independent (if small) real-robot/world-model correlation experiment. The policy-evaluation result (Fig. 4) is encouraging and less affected by the reward-evaluation overlap. However, the headline claim of 'higher-quality PAV properties' rests on metrics that are largely the same VJEPA-2 losses used as rewards, so the significance of that specific claim is currently not established.","major_comments":[{"comment":"The central claim of improved Physical Plausibility and Action Adherence is supported by metrics that are partly circular. r_VJEPA-E (Eq. 7) and r_VJEPA-P (Eq. 8) are negative L1 distances in the same VJEPA-2 embedding space used for the reward, and Table 1 reports 'VJEPA Loss Enc/Pred' as evaluation metrics. Improving those metrics during RL is expected by construction. The non-circular metrics in Table 1 (PSNR, SSIM, LPIPS, EPE, COS) do improve, but they measure low-level visual and optical-flow fidelity, not physical plausibility or action adherence. Without an external yardstick — e.g., human evaluation, object-permanence/contact-consistency checks, or inverse-dynamics action reconstruction — the paper's strongest claim is unsupported. Please provide independent evidence or soften the claim accordingly.","section":"§3.2, Eq. (7)–(10); Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any quantitative result. In Table 1, several differences are small (e.g., VJEPA Enc loss 0.524→0.521 on Agibot; SSIM 0.812 vs 0.793 on DROID), and without variance over validation clips or training seeds it is unclear whether these are robust. The VLA evaluation uses only 10 episodes per task across 5 tasks, and Fig. 4 reports no uncertainty. Given that the main claims rely on modest average gains, the absence of variance reporting is load-bearing.","section":"Table 1; §4.3, Fig. 4"},{"comment":"The reward-coefficient selection is performed on an offline diagnostic (assigning highest reward to GT video among 12 variants) rather than on the actual RL objective. In the best configuration the diagnostic accuracy is only 82%, meaning the reward function misranks GT video in 18% of cases. Moreover, the same validation distribution is used to select coefficients and to report main results, so there is a risk of selection bias. Since λ_static=0.8, λ_emb=0.5, λ_VF=0.1 are chosen from this diagnostic and then fixed for the full 5K-step RL, the paper should demonstrate that these choices also optimize the final RL evaluation metrics, or provide a held-out validation split for coefficient selection.","section":"Appendix A.5, Table 5; §4.2"},{"comment":"The paper calls the method 'model-agnostic' and 'general,' but experiments use a single base world model (Wan2.2-TI2V-5B) and a single reward evaluator (VJEPA-2 fine-tuned on the same robot datasets). The claim of model-agnosticism is not tested. Please either test on a second architecture or restrict the claim to the demonstrated setting.","section":"§1, Conclusion; §4 title"}],"minor_comments":[{"comment":"Several references are self-citations or unpublished preprints with incomplete author lists (e.g., 'rew, 2026', 'Wang et al., 2026a', 'min, 2026'). Please provide full citations and DOIs where available.","section":"References"},{"comment":"The list of five tasks duplicates numbering: '(4) stack-bowls (purple) and (4) stack-bowls (beige)' should be (4) and (5).","section":"§4.3"},{"comment":"The header 'VJPEA-' is a typo for 'VJEPA-'.","section":"Table 2 header"},{"comment":"The text refers to 'Figure 4' for the ablation table, but the table is Table 4.","section":"Appendix A.4"},{"comment":"The abbreviation 'PA V' appears with inconsistent spacing throughout (e.g., 'PA V' vs 'PAV'). Please standardize.","section":"General notation"},{"comment":"The Bernoulli-like sampling scheme with α_OOD = 0.05 is not described in terms of how the random draw is scheduled; later ablations vary α_OOD but not the schedule. Clarify whether the sampling is i.i.d. per rollout or per optimization step.","section":"§3.2, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main issue: the paper's most distinctive claim (PAV property improvement) is evaluated with the same model that provides the reward. This is fixable in principle by adding independent metrics or human evaluation, so I recommend major revision rather than rejection. I also note that the reference list contains a large fraction of self-citations and anonymous preprints; the authors should make sure all claims to prior work are verifiable. The paper is within scope for a CV/robotics venue, and the released code and thorough ablation study are positive signals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's my read.\n\nThe genuinely new thing is the GT-free OOD reward: using VJEPA-2's action-conditioned predictor to score rollouts under perturbed actions where no target video exists. That is a real extension of prior RL world-model work, which needed ground-truth video. The noise-driven action exploration and the embodiment-focused stream are sensible engineering. The paper also does something a lot of world-model papers don't: it checks the model as a policy evaluator against real rollouts. That experiment is small but points in the right direction.\n\nWhat the paper does well is showing consistent gains in the non-circular metrics. PSNR, SSIM, LPIPS, and the flow metrics all improve over the pretrained baseline on both datasets. So the scaled-down claim — RL post-training helps visual and motion fidelity — holds up.\n\nNow the soft spots. The main one is circularity. The VJEPA Enc and Pred losses in Table 1 are the same quantities the reward directly optimizes (Eq. 7 and Eq. 8). Reporting them as evidence of 'higher-quality PAV properties' is not independent evidence. The stress-test note is right on this. The offline diagnostic in Appendix A.5 actually shows the reward only ranks the GT video first 82% of the time in the best configuration, so VJEPA-2 is a noisy judge. That doesn't sink the paper, but it means the physical-plausibility and action-adherence claims are not yet established. There are no confidence intervals, so we can't tell how stable the 5.6% gain is. The code link is promised but nothing is released, and the reference list has some sloppy entries (e.g., 'rew, 2026' and 'Wang et al., 2026a' are not proper citations), which undercuts trust in the related-work scholarship.\n\nNet: the recipe is plausible and the non-circular improvements are real. The central PAV claim is overreaching relative to the evidence. This deserves a serious referee, but the authors should be asked to add an external physical-plausibility metric (object permanence, contact consistency, or inverse-dynamics action reconstruction), report error bars, release code, and clean up the references.\n\nI'd send it to peer review. It's not a paradigm shift, but it's a useful step for the world-model-as-policy-evaluator direction.","headline":"A useful RL post-training recipe for action-conditioned world models, but the headline PAV gains are partly an artifact of rewarding and evaluating with the same VJEPA-2 model.","tokens_in":15751,"tokens_out":2819,"would_cite":true,"duration_ms":29855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Action-conditioned world models get better as policy simulators when post-trained with reinforcement learning that explicitly rewards physical plausibility, action adherence, and visual fidelity.","keywords":["world models","reinforcement learning","action-conditioned video prediction","physical plausibility","action adherence","visual fidelity","out-of-distribution exploration","policy evaluation"],"falsifier":"A concrete check: gather human or real-robot assessments of world-model rollouts and compare them with VJEPA-2 reward rankings; if VJEPA-2 gives high scores to videos that humans or physical sensors judge as implausible or non-adherent, or if the world model's policy success estimates diverge from real-robot outcomes after RL, the reward is being gamed and the central claim fails.","tokens_in":14756,"feed_emoji":"🤖","tokens_out":5252,"duration_ms":51934,"temperature":0.7,"pith_summary":"Action-conditioned world models—video predictors that imagine future frames from a robot's actions—are only useful if their outputs are physically plausible, faithful to the commanded actions, and visually sharp, and if they generalize to actions outside the expert demonstrations they were trained on. This paper argues that the usual pixel-level reconstruction loss does not optimize these targets, and that reinforcement learning with a purpose-built reward can fix that. The proposed PAVXploreRL post-trains a pretrained latent-conditioned video diffusion model using a reward function that combines a latent video model's judgments (VJEPA-2) with no-reference visual-quality metrics, on both in-distribution expert action–video pairs and noise-permuted out-of-distribution actions for which no paired video exists. Across two manipulation datasets, the authors report a 5.6% average improvement over the pretrained baseline, higher-quality PAV properties, and more reliable policy success estimates with reduced overestimation compared with expert-only world models like Ctrl-World. If correct, the approach offers a scalable way to strengthen world models without collecting videos for every action a policy might take.","feed_headline":"Reward-driven training yields 5.6% better robot world models","feed_subtitle":"Post-trains video predictors on out-of-distribution actions to cut policy overestimation.","key_machinery":"The load-bearing mechanism is the reward function built on a fine-tuned, action-conditioned VJEPA-2 latent video model, which provides supervision without ground-truth frames: for in-distribution trajectories the encoder compares predicted and real latents, and for out-of-distribution actions the predictor forecasts the future latent from the start frame and actions, with a static-motion penalty. A parallel embodiment-focused branch, computed on robot-segmented frames, targets action adherence explicitly. The world model itself is a video diffusion model with action embeddings injected via adaptive layer normalization and history frames compressed by a Framepack strategy; RL updates only the","core_discovery":"The paper's central claim is that explicit, reward-driven optimization of Physical Plausibility, Action Adherence, and Visual Fidelity—rather than implicit pixel-level reconstruction—is what an action-conditioned world model needs to generalize beyond expert data. The authors build a world model from a video diffusion backbone with action conditioning and history frames, supervise-fine-tune it, then post-train with RL. The reward uses three signals: an encoder-based VJEPA-2 similarity for in-distribution pairs, a predictor-based VJEPA-2 latent forecast for out-of-distribution actions where ground-truth frames are unavailable, and a battery of no-reference visual fidelity metrics; a static-mo","pith_inferences":["If VJEPA-2's physical understanding is general, this approach could make learned simulators trustworthy enough to replace a fraction of real-robot rollouts in policy development; the reward-model dependence also raises a reward-hacking risk that the authors do not directly test.","The paper's own stated limitations—long training time and lack of multi-view consistency—suggest the next natural experiments are to pair this RL reward with multi-view or physics-constrained world models, and to test whether the reward can be optimized without the heavy RL phase.","The static-motion regularization findings hint that without an explicit anchor, latent-similarity rewards collapse toward static outputs; a similar collapse could occur for other object permanence violations not captured by VJEPA-2.","A direct testable extension would be to use the same reward function to post-train a world model on tasks with sparse rewards or long horizons, where overestimation bias is most damaging."],"forward_implications":["Post-training with the PAV reward improves visual quality, flow consistency, and VJEPA-based losses on both Agibot and DROID, with an average 5.6% gain over the pretrained model.","World models trained with OOD action exploration yield policy success estimates closer to real-world rollouts and reduce overestimation bias relative to expert-only models like Ctrl-World.","The method works from a single view and does not require paired videos for OOD actions, so it can be applied to unseen actions generated by perturbing expert trajectories.","Moderate OOD exploration (αOOD around 0.05) is optimal; larger proportions degrade performance.","Because RL updates only the action encoder and LoRA parameters on a frozen base model, the pipeline is model-agnostic and can be applied to other world model architectures."],"fun_headline_variants":["RL post-training lifts world model performance by 5.6%","Action-exploration RL trains world models that beat expert-only baselines","Explicit PAV rewards yield world models that generalize beyond experts","PAVXploreRL: RL-optimized world models cut policy overestimation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's load-bearing premise is that the fine-tuned VJEPA-2 latent model is a trustworthy judge of physical plausibility and action adherence, so that increasing its similarity scores constitutes genuine world-model improvement rather than optimization of a learned proxy.","fun_headline_variants_meta":{"raw":{"variants":["RL post-training lifts world model performance by 5.6%","Action-exploration RL trains world models that beat expert-only baselines","Explicit PAV rewards yield world models that generalize beyond experts","PAVXploreRL: RL-optimized world models cut policy overestimation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1418,"prompt_tokens":780,"completion_tokens":638,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":524,"tokens_out":638,"duration_ms":7628,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:26:48.120827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: gather human or real-robot assessments of world-model rollouts and compare them with VJEPA-2 reward rankings; if VJEPA-2 gives high scores to videos that humans or physical sensors judge as implausible or non-adherent, or if the world model's policy success estimates diverge from real-robot outcomes after RL, the reward is being gamed and the central claim fails.","supporting_citations":[],"review_version":1}