{"id":"272eec63-58c4-4299-b975-3547e2ac3a3b","arxiv_id":"2507.23391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reward-model-free offline RL method that uses VLM-generated pairwise preferences and contrastive preference learning to train manipulation policies.","lead":"PLARE trains robot policies directly from preference labels generated by a large vision-language model, skipping the reward-model step that prior methods need. If the results hold, it lowers the cost of applying offline reinforcement learning to real-world manipulation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PLARE's dependence on three-frame VLM labels is under-validated; the reported gains may be inflated by evaluation tuning.","rationale":"The reader identified VLM preference reliability as the weakest assumption, which I agree is central. I sharpen this by focusing on the specific three-frame subsampling choice in Section IV.A, which is the concrete mechanism behind the 70–80% label accuracy in Table III. The paper neither reports the preliminary experiments backing this choice nor provides a sensitivity analysis, so the method's robustness to temporal undersampling is unverified. I also flag the evaluation protocol (per-task dropout tuning and max-over-checkpoint) because it directly affects the strength of the empirical claim and interacts with the label-noise concern: if the reported gains are partly an artifact of selection, the evidence that VLM feedback alone is sufficient becomes weaker. These issues do not refute the method's plausibility on the tested tasks, but they do justify the conditional verdict: the central claim about bypassing reward modeling is not fully established without additional validation of the label signal and a more conservative evaluation. I therefore keep the reader's CONDITIONAL verdict unchanged, with the concrete test designed to settle whether the three-frame choice and evaluation protocol are indeed load-bearing.","tokens_in":12405,"tokens_out":7292,"duration_ms":75945,"concrete_test":"On the four MetaWorld tasks (or a randomly held-out subset), query Gemini on the same set of trajectory pairs with 1, 3, 5, 10, and all frames per segment; measure label accuracy against ground-truth preferences. Then train PLARE for each frame-count setting using a fixed dropout (e.g., 0.25) and evaluate using both the last checkpoint and the best-of-8 checkpoint rule. If label accuracy and success plateau at three frames and the last-checkpoint results still beat SoTA, the concern is mitigated; if accuracy improves significantly with more frames or the best-vs-last gap is large, the three-frame subsampling and evaluation tuning are load-bearing limitations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PLARE can train policies directly from VLM preferences without reward modeling rests on the assumption that Gemini's pairwise judgments over three uniformly spaced frames per segment (first, middle, last; Section IV.A) provide a sufficiently accurate training signal. Table III reports only 70–80% label accuracy against ground-truth preferences, meaning 20–30% of the labels are wrong. The paper does not report the 'preliminary experiments' that motivated the three-frame choice, nor does it analyze how label accuracy or downstream success varies with the number of frames or task complexity. On the four simple MetaWorld tasks, success is often visible in the last frame (e.g., drawer open, plate in goal), but on tasks requiring temporal or multi-stage reasoning, three frames can systematically mislead the VLM, producing biased preferences that a contrastive objective and dropout cannot correct. This concern is compounded by the evaluation protocol: the main results are selected via per-task dropout tuning (Table III) and max-over-checkpoint evaluation, so the reported 13.5% relative improvement over SoTA may not reflect a stable, replicable advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PLARE, a method for offline policy learning that replaces hand-designed rewards and learned reward models with preference labels generated by a large vision-language model (VLM). Given an unlabeled dataset and a natural-language task description, PLARE samples pairs of trajectory segments, represents each segment by three uniformly spaced frames (first, middle, last), and queries Gemini for a pairwise preference. The policy is then trained directly with the contrastive preference learning (CPL) objective, without an intermediate reward model. Experiments cover four MetaWorld manipulation tasks and two real-robot tasks with a 7-DOF Sawyer arm. The paper reports an average success rate of 70.0 on MetaWorld, a 13.5% relative improvement over the RL-VLM-F baseline, and ablations on equal-preference filtering and dropout as a regularizer for noisy labels.","tokens_in":12574,"tokens_out":8528,"duration_ms":92481,"significance":"If the claims are sustained, PLARE offers a practical approach to training robotic policies from language descriptions and unlabeled video, avoiding both manual reward engineering and learned reward models. The use of CPL is a sound and well-motivated choice, and the comparison against RL-VLM-F uses the same VLM-generated preference dataset, which isolates the effect of skipping reward modeling. The paper also includes informative ablations on label noise. The main limitations are the evaluation protocol (per-task dropout selection and max-over-checkpoint reporting), the lack of analysis of VLM label reliability as a function of frame count and task complexity, and the statistically weak real-robot results. The code is promised to be public, which supports reproducibility.","major_comments":[{"comment":"The main results in Table I are obtained by selecting, per task, the dropout probability that yields the best success rate in Table III (e.g., 0.4 for Drawer Open and Plate Slide, 0.25 for Sweep Into and Door Open), and success rates are the maximum average over an 8-checkpoint, 200-episode evaluation window. Because the same performance metric is used both for hyperparameter selection and for final evaluation, the reported 13.5% relative improvement over RL-VLM-F may be inflated by selection bias. The authors should use a validation split for dropout and checkpoint selection and report performance on a held-out split, or at least report results across all dropout values without selecting the best.","section":"Table III and Table I"},{"comment":"The choice of three uniformly spaced frames per segment is justified only by 'preliminary experiments' that are not reported, and Table III shows VLM label accuracy of only 69.4-80.7% against ground-truth preferences, meaning that 20-30% of training labels are incorrect. The paper does not analyze how label accuracy or downstream policy success varies with the number of frames or with task complexity, and the four MetaWorld tasks are short-horizon with success often visible in the final frame. The current evidence therefore does not establish that the pipeline works for temporally extended or multi-stage tasks. The authors should report the preliminary experiments and an analysis of label quality as a function of frame count, and ideally include a task that requires reasoning about intermediate states rather than just the outcome in the last frame.","section":"Section IV.A and Table III"},{"comment":"The real-robot comparison is based on only 10 trials per method, and no error bars, confidence intervals, or significance tests are reported. The statement that 'PLARE achieves the highest success rates' is therefore not statistically substantiated. The authors should report per-trial results, standard errors or confidence intervals, and details of the human assessment protocol (e.g., number of raters and whether they were blind to the method).","section":"Section V.E and Figure 4"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'large Vison-Language Models' should be 'large Vision-Language Models'.","section":"Section I"},{"comment":"The segment length L is an input to Algorithm 1, but its value is never specified for either the MetaWorld or the real-world experiments; please state the segment length used.","section":"Algorithm 1 and Section V.A"},{"comment":"The VLM prompt in Figure 2 instructs the model to reply '-1' when unsure or when there is no difference, but the preference dataset and the CPL objective use a label of 0.5 for equal preference; the mapping from the raw VLM response to the stored label should be described explicitly.","section":"Section IV.A and Figure 2"},{"comment":"The dropout used as a regularizer is not described in terms of where it is applied (e.g., in the policy MLP) or how it interacts with the CPL objective; this implementation detail is needed for reproducibility.","section":"Section IV.B"},{"comment":"The term γtα log πθ(at|st) is ambiguous: it is unclear whether α multiplies the log-probability only or also the discount factor, and the roles of α and λ are not defined in the text.","section":"Equation (2)"},{"comment":"The row labeled 'Accuracy' in Table III is not explained in the caption; please state that it measures the agreement between VLM-labeled preferences and the ground-truth preferences from the CPL dataset.","section":"Table III"},{"comment":"In MetaWorld, all policies are trained from 35-dimensional low-dimensional states, while VLM preferences are generated from rendered images; the paper does not discuss this modality mismatch, which limits the strength of the claim that the simulation results demonstrate vision-based policy learning.","section":"Section V.A"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a competent application of CPL to VLM-generated preferences and the core idea is interesting. The main risk is the evaluation protocol: per-task dropout selection on the test metric and max-over-checkpoint reporting can easily inflate the headline improvement. If the authors provide a validation-based selection procedure and a more thorough analysis of label quality, the claims would be substantially stronger. The limited real-robot evaluation is a secondary concern. The work fits the journal's scope, and the self-citations are not problematic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the paper does something simple and useful—use Gemini to label trajectory pairs, then train a policy with CPL, no reward model—and the evidence mostly backs it up, but the headline gains are inflated by per-task dropout tuning and max-over-checkpoint selection.\n\nWhat's actually new: not the ingredients. CPL is [35], VLM preference querying is [22]'s idea extended to multi-frame. The new part is the combination, and the authors are honest about that. I give them credit for three things. First, they share the same VLM preference dataset with RL-VLM-F, so the comparison isolates the effect of dropping the reward model. Second, the ablations on equal-preference labels and dropout are informative; they even report the VLM's label accuracy (70–80%), which is refreshing. Third, the compute table is useful: 0.28M parameters and 0.4h for PLARE vs 1.42M and 1.3h for RL-VLM-F. Code is promised, and the implementation builds on the public CPL repo.\n\nThe soft spots are in the evaluation protocol. The main results (Table I) use the best dropout rate per task from Table III, chosen on the same tasks used to report the headline numbers. That is test-set tuning, and it makes the 13.5% relative improvement over RL-VLM-F look less robust than the text suggests. The max-over-checkpoint evaluation follows the CPL benchmark, so it's a shared practice, but it is still an upper envelope. On the labeling side, 20–30% wrong labels is a real amount of noise; the paper's own argument is that dropout handles it, and the success on four simple tasks supports that. But the three-frame choice is under-validated—the preliminary experiments that motivated it are not reported, and there's no study of how label accuracy or downstream success varies with frame count or task complexity. The stress-test worry about multi-stage tasks is plausible but not demonstrated; I wouldn't call it a load-bearing failure. The real-robot results are 10 trials, no error bars, and the authors call the tasks simple themselves. That's a proof of concept, fine, but not strong evidence.\n\nThe central claim holds up as a conditional: if you accept these four tasks as representative, then yes, VLM preferences can drive offline policy learning without a reward model. The paper is not circular, the citation pattern is fine (the self-citations are to directly relevant prior work), and the thinking is clear. The soft spots are about the strength of the evidence, not the validity of the approach.\n\nWho this is for: people working on offline RL, preference-based RL, or VLM-based rewards. A serious referee should see it; it deserves peer review, with the expectation that the authors tighten the evaluation protocol (fixed dropout or nested selection), add error bars to the real-robot numbers, and report the frame-count analysis. I'd be happy to see it in a workshop or a main conference after moderate revision.","headline":"PLARE is a useful integration—VLM preference labels plus CPL, no reward model—and the evidence mostly backs it, but per-task dropout tuning and max-over-checkpoint selection inflate the headline gains.","tokens_in":13118,"tokens_out":3460,"would_cite":true,"duration_ms":36510,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that offline robot policies can be trained directly from a VLM's pairwise trajectory preferences, with no reward model, improving average success by 13.5 percent over the previous state of the art.","keywords":["offline reinforcement learning","preference-based RL","vision-language models","contrastive preference learning","reward-free policy learning","robotic manipulation","noisy labels","VLM feedback"],"falsifier":"Evaluate PLARE on a task with known ground-truth preferences; if VLM label accuracy drops below roughly 70 percent and final policy success falls to the level of behavior cloning, the claim that VLM preferences suffice for policy learning would fail.","tokens_in":12193,"feed_emoji":"🤖","tokens_out":5460,"duration_ms":61311,"temperature":0.7,"pith_summary":"This paper tries to show that a robotic policy can be learned from an unlabeled, suboptimal dataset using only a natural-language task description, by asking a large vision-language model (VLM) which of two short trajectory video segments makes more progress toward the goal. The resulting pairwise preferences are fed directly into a supervised contrastive preference learning objective, so no explicit reward model is trained. The claim matters because reward design is a major bottleneck in offline reinforcement learning: if a VLM can supply the signal, new tasks become trainable from language plus unlabeled video. On four simulated manipulation tasks the method reports an average success rate of 70.0 percent, a relative improvement of 13.5 percent over the previous state-of-the-art VLM-reward approach, and it also trains a real robot on two simple tabletop tasks.","feed_headline":"A vision-language model alone can train robot policies","feed_subtitle":"Pairwise video preferences drive a contrastive objective, beating reward-model baselines on manipulation tasks.","key_machinery":"The load-bearing object is the contrastive preference learning (CPL) objective, a supervised loss that treats each segment's discounted action log-likelihood as the segment's return and uses a softmax over the two compared segments to push probability mass toward the preferred one. The VLM labeling pipeline is the second piece: a two-stage prompt first asks the model to describe each three-frame sequence and then asks which sequence better achieves the language-specified goal. Dropout acts as the noise-handling mechanism, and dropping equal-preference labels removes ambiguous supervision.","core_discovery":"On its own terms, the paper establishes that pairwise preference labels from a large vision-language model, obtained by showing the model three frames (first, middle, and last) from each of two trajectory segments, are a sufficient supervisory signal to train a manipulation policy offline. The policy is optimized with a supervised contrastive preference learning objective that compares segment-level action log-likelihoods, so no reward function is ever fit. The paper reports an average success rate of 70.0 percent across four simulated manipulation tasks, a relative improvement of 13.5 percent over the previous state-of-the-art VLM-reward baseline, and successful real-robot policies on two simple tabletop tasks. The paper also finds that excluding labels where the VLM says the two segments are equal, and tuning dropout as a regularizer, are both important for handling the 20 to 30 percent label noise the VLM produces.","pith_inferences":["The paper's own comparison with an oracle trained on ground-truth labels shows the learning objective is not the bottleneck: the VLM's 70 to 80 percent label accuracy is. This suggests the next gains will come from better questions or better frame selection, not from a stronger loss.","Because the simulated policies use low-dimensional state vectors, the VLM preference signal may matter less when the state already encodes goal progress; a purely visual-control version would test whether the preference signal alone carries the information.","The two-stage prompting scheme is one design choice among many, and the reported gains could be sensitive to prompt wording; ablating the prompt across tasks would reveal how much of the improvement comes from the learning objective versus the query format.","The real-robot tasks are deliberately simple, so the method's ceiling on long-horizon, multi-stage manipulation remains untested; a natural extension is to evaluate whether three-frame segment comparisons can judge progress in tasks where early states look visually similar."],"forward_implications":["New manipulation tasks could be specified by a sentence and an unlabeled video collection, removing reward engineering entirely.","Without a reward model, training is cheaper and has fewer components to tune; the paper reports 0.28 million parameters and 0.4 hours of training versus 1.42 million parameters and 1.3 hours for the reward-model baseline.","Filtering out equal-preference labels and tuning dropout appear to be simple, transferable ways to cope with noisy VLM labels.","If the result holds, the same recipe could apply to other embodied domains where a vision-language model can judge trajectory progress from language instructions."],"supporting_citations":[{"why":"Supplies the contrastive preference learning objective, the offline datasets, and the evaluation protocol used for the MetaWorld experiments.","marker":"[35]"},{"why":"The state-of-the-art reward-model baseline whose VLM querying idea PLARE adapts and whose performance PLARE compares against using the same preference dataset.","marker":"[22]"},{"why":"A CLIP-based zero-shot reward baseline compared in the main simulation results.","marker":"[21]"},{"why":"A video-text similarity reward baseline compared in the main simulation results.","marker":"[20]"},{"why":"The large vision-language model that generates all preference labels.","marker":"[34]"},{"why":"The offline RL algorithm underlying the reward-based baseline methods.","marker":"[61]"},{"why":"The visual encoder used for the real-robot policy inputs.","marker":"[60]"},{"why":"Motivates training policies directly from preferences without a reward function, the line of work PLARE builds on.","marker":"[28]"}],"fun_headline_variants":["Robot policies trained from VLM preferences, no reward model","VLM preferences train robot policies without reward modeling","No reward model: VLM preferences directly train robot policies","VLM preferences beat reward models for robot policy training","Contrastive VLM preferences train robot policies offline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a VLM's pairwise comparisons of three frames per segment are reliable enough to guide a manipulation policy, even though the paper measures only 70 to 80 percent agreement with ground-truth preferences.","fun_headline_variants_meta":{"raw":{"variants":["Robot policies trained from VLM preferences, no reward model","VLM preferences train robot policies without reward modeling","No reward model: VLM preferences directly train robot policies","VLM preferences beat reward models for robot policy training","Contrastive VLM preferences train robot policies offline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00132,"raw_usage":{"total_tokens":5365,"prompt_tokens":928,"completion_tokens":4437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":4360}},"tokens_in":544,"tokens_out":4437,"duration_ms":26233,"temperature":1.0,"reasoning_tokens":4360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:47:52.826899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate PLARE on a task with known ground-truth preferences; if VLM label accuracy drops below roughly 70 percent and final policy success falls to the level of behavior cloning, the claim that VLM preferences suffice for policy learning would fail.","supporting_citations":[{"cited_title":"Contrastive preference learning: learning from human feedback without rl,","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive preference learning objective, the offline datasets, and the evaluation protocol used for the MetaWorld experiments."},{"cited_title":"Rl-vlm-f: Reinforcement learning from vision language foundation model feedback,","cited_arxiv_id":null,"evidence_quote":"The state-of-the-art reward-model baseline whose VLM querying idea PLARE adapts and whose performance PLARE compares against using the same preference dataset."},{"cited_title":"Vision-language models are zero-shot reward models for reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"A CLIP-based zero-shot reward baseline compared in the main simulation results."},{"cited_title":"Roboclip: One demonstration is enough to learn robot policies,","cited_arxiv_id":null,"evidence_quote":"A video-text similarity reward baseline compared in the main simulation results."},{"cited_title":"Offline reinforcement learning with implicit q-learning,","cited_arxiv_id":null,"evidence_quote":"The offline RL algorithm underlying the reward-based baseline methods."},{"cited_title":"R3m: A universal visual representation for robot manipulation,","cited_arxiv_id":null,"evidence_quote":"The visual encoder used for the real-robot policy inputs."},{"cited_title":"Inverse preference learning: Preference- based rl without a reward function,","cited_arxiv_id":null,"evidence_quote":"Motivates training policies directly from preferences without a reward function, the line of work PLARE builds on."}],"review_version":1}