{"id":"c152d498-9f1e-47e7-9af9-2ca27526ad65","arxiv_id":"2502.01616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PrefVLM combines VLM-generated trajectory preferences with selective human feedback and inverse-dynamics VLM adaptation, matching PEBBLE on five Meta-World tasks with up to 2x fewer human labels.","lead":"PrefVLM uses a vision-language model to produce rough preferences over pairs of robot trajectories, then asks humans to label only the cases the model is unsure about. On five Meta-World manipulation tasks, the authors report matching a standard preference-based RL method with up to two times less human feedback.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7) appears to reverse the bound's direction and the paper does not validate that the KL filter separates clean VLM-generated preferences from noisy ones; the 2x human-annotation claim rests on this unvalidated mechanism.","rationale":"The reader's weakest assumption correctly identifies the sample-selection mechanism as the load-bearing component of the 2x-annotation claim. I agree: the paper provides no direct evidence that the small-loss trick transfers from RIME's human-preference setting to the VLM-generated-preference setting, where label noise is systematic rather than random. The formulation of Eq. (7) is additionally suspect; because the KL divergence between a one-hot label and the model's distribution is exactly the per-sample cross-entropy, the inequality as written implies that lower loss leads to higher KL, which contradicts the intended use of selecting low-loss samples as clean. This suggests a misprint or a misapplication of RIME's theory, and without the correct bound the threshold scheduling in §4.3 is not justified. The paper does include useful ablations: PrefVLM w/o selection isolates the filtering benefit, Fig. 8 shows that filtering alone and inverse dynamics alone fail, and Fig. 7 provides qualitative evidence of VLM adaptation. These are positive elements. However, the only quantitative evidence for the filtering benefit is the gap in Fig. 4, which lacks error bars or significance tests on final performance. The task list inconsistency (drawer-open in §5.1 vs door-close in Fig. 4) and the absence of code further hinder verification. The central empirical claim could be true, but it is not yet supported at the level of confidence the paper claims. The verdict should remain conditional pending direct validation of the selection mechanism, matching the reader's assessment.","tokens_in":13167,"tokens_out":10523,"duration_ms":89489,"concrete_test":"Implement PrefVLM with oracle logging: at each feedback iteration, after computing D_KL for all VLM-labeled pairs, label a held-out subset of pairs with the ground-truth reward (oracle) and compare oracle agreement for (a) pairs in Dτl (clean), (b) pairs between τlower and τupper (uncertain), and (c) pairs in Dτu (relabeled). If the oracle agreement of Dτl is not substantially higher than that of the uncertain set, the filter is not separating noise from signal and the selective-annotation benefit is unsupported. A secondary check: re-derive Eq. (7) from RIME's assumptions to confirm the inequality direction; if the bound is reversed, the paper must correct the theoretical justification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—matching PEBBLE's success with half the human feedback—depends on the assumption that the sample-selection mechanism in §4.3 separates reliable VLM-generated preference labels from noisy ones and spends the human budget only on genuinely uncertain pairs. This assumption is not validated in the new setting. Eq. (7) is imported from RIME but is misstated: since D_KL(ỹ || P_θ) for a one-hot label equals the per-sample cross-entropy loss, clean samples (small loss ρ) have small KL divergence, not KL ≥ -ln ρ + ρ/2 (which diverges as ρ → 0). If the inequality direction is corrected, the threshold τlower = -ln ρ + αρ is not a tight lower bound and the filter's calibration is unclear. The only empirical evidence that filtering helps is the qualitative gap between PrefVLM and PrefVLM w/o selection (Fig. 4), with no significance testing, and the paper's own Fig. 8 shows that filtering without human feedback fails. Without a direct measure of whether the 'clean' set has higher label accuracy than the uncertain set, the 2x efficiency claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PrefVLM, a preference-based RL method in which a pretrained CLIP-style VLM (LIV) assigns coarse preference labels to trajectory pairs by comparing summed per-step cosine similarities (Eqs. 3-4); a KL-divergence filter (Eqs. 7-9) is used to split samples into clean, noisy, and uncertain sets, with a small human budget spent on the uncertain set; the VLM's two learned adapter layers are fine-tuned with human preferences and an inverse-dynamics loss (Eqs. 5-6). Experiments on five Meta-World tasks compare PrefVLM with 1000 human labels against PEBBLE with 1000 and 2000 labels, VLM-as-reward, VLM-pref-reward, and an ablation without selection, and report matching or better success rates with half the human feedback, plus a transfer experiment that cuts feedback by 4x.","tokens_in":13395,"tokens_out":6404,"duration_ms":56211,"significance":"If the claimed 2x feedback reduction holds, PrefVLM would be practically valuable for preference-based RL, combining scalable machine labels with targeted human oversight. The inverse-dynamics adaptation is a sensible, testable idea, and the paper includes pseudocode and hyperparameters. However, the current manuscript does not substantiate the filtering mechanism: the key KL bound in Eq. (7) is misstated, the VLM label orientation in Eq. (4) is inconsistent with the defined preference convention, and no quantitative or statistical validation of the central efficiency claim is provided. These issues must be resolved before the headline claim can be accepted.","major_comments":[{"comment":"The stated KL bound has the wrong direction. Since the preference label is one-hot, D_KL(ỹ || P_θ(σ0, σ1)) equals -log P_θ(chosen segment), which is exactly the per-sample cross-entropy loss in Eq. (2). If clean samples have loss at most ρ, then we get D_KL ≤ ρ, not D_KL ≥ -ln ρ + ρ/2 + O(ρ²). For small ρ, -ln ρ + ρ/2 is large, so the threshold τlower = -ln ρ + αρ is not a lower bound on clean-sample KL; the set Dτl in Eq. (8) can then contain essentially all samples and the selection mechanism becomes vacuous. The paper provides no measurement of label accuracy on the 'clean' versus 'uncertain' sets, so the 2x efficiency claim is not backed by a validated filtering step.","section":"Section 4.3, Eq. (7)"},{"comment":"The label assignment in Eq. (4) is reversed relative to the convention defined in Section 3.1. There, y = (1, 0) means σ0 is preferred over σ1. Eq. (4) assigns y = (0, 1) when R0 > R1, i.e., it labels the lower-return segment as preferred. If implemented as written, the VLM provides inverted preferences and the reward model is trained against the wrong ordering; if this is a typo, it must be fixed because the subsequent filtering in Eqs. (8-9) and the reward training in Eq. (2) all depend on label orientation.","section":"Section 4.1, Eq. (4)"},{"comment":"The central claim that PrefVLM 'matches PEBBLE's performance while requiring only half the human feedback' is supported only by learning curves with shaded standard error; no final success-rate table, per-task numerical values, or significance tests are given. With five seeds and visibly overlapping shaded regions, the 'across all tasks' claim cannot be verified. Please add a table of mean ± standard error at the end of training, a paired comparison, and a precise statement of what the 1000/2000 counts include (human labels only, or also VLM-generated labels under the 30,000-sample budget in Appendix C). The baseline set also omits feedback-efficient methods such as SURF and RIME, so the state-of-the-art claim is not demonstrated.","section":"Section 5.3, Figure 4"},{"comment":"The ablation results show that filtering alone (VLM-generated preferences with selection, but no human feedback) fails, and that PrefVLM without selection also performs reasonably. This leaves the contribution of the sample-selection mechanism itself unclear; the only evidence that selection helps is the qualitative gap between PrefVLM and PrefVLM w/o selection, which is not statistically tested. Please report a direct validation of the Eqs. (7-9) mechanism, for example by measuring agreement of the 'clean' machine-labeled set with held-out human labels versus agreement of the 'uncertain' set, so the reader can see that the filter is actually separating reliable from unreliable VLM labels.","section":"Section 5.4, Figure 8"}],"minor_comments":[{"comment":"The text refers to 'VLM-as-pref' but the baseline is named 'VLM-pref-reward' in Section 5.2; please unify the name.","section":"Section 5.3"},{"comment":"The expression |Dh| = min(|B| - |Dτl| - |Dτu|, 0.05 * N) uses N both for the number of newly sampled preference pairs and for the feedback batch size, and the argument may be negative if the clean and noisy sets exhaust the buffer; please clarify the units and clipping behavior.","section":"Algorithm 1, line 18"},{"comment":"The phrase 'reference feedback' should be 'preference feedback'.","section":"Section 5.1"},{"comment":"The reward curve lacks axis labels and units, making the claim of misalignment with task progress difficult to evaluate.","section":"Figure 2"},{"comment":"The hyperparameters α = 0.5, βmin = 1, βmax = 3, k = 1/300, and τupper = 3 ln 10 are said to follow Cheng et al. (2024), but Eq. (7) as stated does not match the bound derived for cross-entropy loss; please provide a derivation or a precise citation of the source bound.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable, but the technical inconsistencies in Eqs. (4) and (7) are substantial and affect the core method. I recommend sending the manuscript back with a request for corrected derivations, direct validation of the filtering mechanism, and quantitative per-task results, before any claim of reduced human feedback can be considered established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. PrefVLM is a sensible mash-up of things that already work: VLM trajectory preferences (from LIV-style similarity), inverse-dynamics adaptation, and RIME-style KL filtering with a human budget spent on the uncertain leftovers. The combination is new, and on five Meta-World tasks it consistently shows PrefVLM with 1000 human labels tracking PEBBLE at 2000. That's a real, useful result if it holds.\n\nWhat the paper does well: the empirical setup is honest. Five seeds, standard errors, learning curves for all methods. The ablation on the inverse dynamics loss (Fig. 8) is informative: without it, performance degrades as the policy shifts, and with it, training is stable. The transfer experiments are a nice extra, even though they only cover two settings. The authors also show that VLM-only preferences fail without human feedback, which sets a fair baseline for their method.\n\nNow the soft spots. The KL-thresholding in Sec. 4.3 has a real problem. For a one-hot label, D_KL(ỹ || P_θ) is exactly the per-sample cross-entropy loss ρ. Clean samples have small ρ and small KL. But Eq. (7) states D_KL ≥ -ln ρ + ρ/2, which for small ρ is a large positive number. That inequality is reversed; the right statement would be D_KL ≤ something like -ln ρ? Actually the correct bound for a one-hot label is D_KL = ρ, so the threshold τlower = -ln ρ + αρ makes no sense as a lower bound on clean-sample KL. The stress-test note is correct on this. The paper then uses τlower as a cutoff: samples with KL < τlower are 'clean.' Given the formula, if ρ is small, τlower is large, so almost everything would be clean, which undermines the whole selection mechanism. I suspect this is a typo or a mis-transcription from RIME, but it needs to be fixed and the filter's behavior needs empirical validation. There is no direct measurement of label accuracy on the 'clean' vs 'uncertain' sets, and the paper's own Fig. 8 shows filtering without human feedback fails. So the 2x efficiency claim rests on a mechanism that is currently unjustified.\n\nOther issues: the baseline set is thin—only PEBBLE, no SURF or newer preference-based methods. No numeric tables or significance tests, just learning curves. There is a task-list inconsistency (text lists drawer open, figure shows door close). No code is released. None of these are fatal, but they make the paper weaker than it could be.\n\nWho is this for? People working on preference-based RL and VLM reward models will want to read it. It deserves a serious referee; the core idea is sound and the results are suggestive. But the reviewer should push hard on the KL threshold derivation and demand either a corrected bound or an empirical demonstration that the filter selects higher-quality labels than random selection. If the authors fix that and add a couple more baselines, this becomes a solid paper. As is, it's a conditional accept at best.\n\nRecommendation: send to peer review, but with the expectation that the theoretical section needs rewriting.","headline":"A plausible efficiency gain from combining VLM preferences with selective human feedback, but the KL filter's theoretical justification is wrong and its effectiveness is unmeasured.","tokens_in":13938,"tokens_out":4044,"would_cite":true,"duration_ms":35549,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PrefVLM claims that pairing VLM-generated trajectory preferences with a KL-divergence filter for selective human feedback can match human-only preference-based RL with half the annotations.","keywords":["preference-based reinforcement learning","vision-language models","selective human feedback","noisy label filtering","reward learning","inverse dynamics","robotic manipulation","Meta-World"],"falsifier":"Collect a held-out set of trajectory pairs with human labels, run PrefVLM's KL filter over them, and compare the fraction of correct labels in the clean set ($D_{\\tau_l}$), the relabeled set ($D_{\\tau_u}$), and the uncertain set sent to humans. If the clean set's label accuracy is not substantially higher than the uncertain set's, or if the flipped labels in $D_{\\tau_u}$ are not more accurate than the originals, then the selection mechanism is not doing the work and the reported feedback savings would not transfer to new tasks.","tokens_in":12973,"feed_emoji":"🤖","tokens_out":6912,"duration_ms":57086,"temperature":0.7,"pith_summary":"PrefVLM is a framework for preference-based reinforcement learning that replaces most human preference labels with trajectory preferences generated by a vision-language model (a model trained to align images with text), reserving human annotations for the pairs the VLM and reward model are least certain about. The paper's central claim is that on five Meta-World manipulation tasks this selective combination reaches the same or better success rates as the human-only preference-based method PEBBLE while using about half the human feedback, and that adapting the VLM on one task transfers to a related task with up to four times fewer annotations. The mechanism has two parts: a CLIP-style VLM scores trajectory segments by similarity to the task description, and a KL-divergence thresholding rule splits VLM labels into clean, relabeled, and uncertain groups, with only the uncertain pairs sent to a human. A self-supervised inverse-dynamics loss keeps the adapted VLM aligned with the changing data distribution as the policy improves. The paper itself reports that VLM preferences plus filtering but no human feedback do not work, so the contribution is the synergy, not VLM-only reward learning.","feed_headline":"VLM preferences halve human feedback in robot RL","feed_subtitle":"The method replaces most human preference labels with VLM labels and spends human effort only on uncertain pairs.","key_machinery":"The machinery has three connected parts. The VLM is a CLIP-style model whose language and image encoders are topped with two small trainable layers; a trajectory segment's return is the summed cosine similarity between the task description and each frame, and the segment with higher return is preferred (Eqs. 3-5). A self-supervised inverse-dynamics loss (Eq. 6) trains a linear layer to predict the action between consecutive adapted image embeddings. The third part is a noise filter: after training the reward model, the KL divergence between the predicted preference distribution and the VLM's label is compared to two thresholds, $\\tau_{lower}$ and $\\tau_{upper}$; low-divergence pairs are treated as clean, high-divergence pairs are relabeled by flipping the label, and the uncertain middle band is where the human budget is spent. The load-bearing identity is the bound in Eq. (7), which claims that the clean-sample preference loss is small enough for a threshold on KL divergence to separate reliable from unreliable VLM labels.","core_discovery":"On the paper's own terms, the discovery is that a pretrained VLM can act as a coarse preference teacher for preference-based RL, and that a small amount of targeted human feedback is enough to turn those noisy labels into a reward model that trains policies as good as those trained entirely on human preferences. Concretely, PrefVLM uses an adapted language-image representation to rank trajectory segments, threshold-based selection sends only high-uncertainty pairs to a human, and the reward model is trained on the union of VLM and human labels. On door, drawer, and window tasks in Meta-World, the paper reports that PrefVLM with 1,000 human comparisons matches PEBBLE with 2,000, and that adapting the VLM on a source task lets it match PEBBLE with 2,000 comparisons using only 500 on the target task. The paper's reported caveat is equally explicit: without the initial human feedback to adapt the VLM, filtering by itself does not produce usable policies.","pith_inferences":["The claimed 2x gain is likely sensitive to the base VLM's zero-shot competence on the task family; on tasks where the VLM reward carries no signal, the KL filter has no reliable clean group to lean on and the method would degrade toward the filtering-alone failure mode the paper reports.","An immediate testable extension is using the VLM's own confidence or calibration to select human queries instead of the learned reward model's KL divergence; comparing the two selectors on the same human budget would isolate whether the gain comes from the robust-training trick or from the selection rule itself.","Since the inverse-dynamics objective is task-agnostic, the adaptation could in principle be pretrained on unlabeled interaction data from a target environment before any human feedback arrives, which the paper does not explore.","The paper reuses RIME's threshold schedule without validating the clean-loss bound for reward models trained on VLM labels, so the efficiency claim rests on that transfer being valid; a sensitivity study over $\\beta_{min}$, $k$, and $\\tau_{upper}$ would show how much the result depends on those hyperparameters."],"forward_implications":["A practitioner who currently pays for 2,000 human preference comparisons per Meta-World-style task could spend 1,000 and keep the same final success rate.","Starting from a VLM adapted on one source task cuts the annotation budget further: 500 comparisons on a related task match 2,000 comparisons of human-only PEBBLE.","The adapted VLM embeddings double as a dense reward signal that tracks task progress better after fine-tuning, so the method provides a useful reward-shaping signal during training.","Human annotators no longer label random pairs; they only see pairs the model cannot resolve, so annotation effort is spent where it changes the learned reward.","Because the pipeline runs on visual observations and task-language descriptions, it extends to any manipulation task with a describable goal, without hand-designed reward functions."],"supporting_citations":[{"why":"Supplies PEBBLE, the preference-based RL backbone that PrefVLM builds on and the main baseline it must match at half the feedback budget.","marker":"Lee et al., 2021"},{"why":"Supplies the KL-divergence thresholding rule and the bound in Eq. (7) that PrefVLM uses to separate clean from noisy VLM-generated labels.","marker":"Cheng et al., 2024"},{"why":"Provides the pretrained LIV vision-language model whose embeddings PrefVLM adapts and whose cosine-similarity reward formulation is used to score trajectory segments.","marker":"Ma et al., 2023a"},{"why":"Establishes the preference-based reward-learning framework and the Bradley-Terry preference loss that PrefVLM extends to machine-generated preferences.","marker":"Christiano et al., 2017"},{"why":"Defines the VLM-as-reward baseline that shows zero-shot VLM rewards without adaptation are too noisy for fine-grained manipulation.","marker":"Rocamonde et al., 2023"},{"why":"Provides the Meta-World benchmark and the five manipulation tasks and task descriptions used in all experiments.","marker":"Yu et al., 2020"}],"fun_headline_variants":["VLM preference labels halve human feedback for robot RL","PrefVLM uses VLM labels, keeps humans for uncertain cases","Robot RL: VLM does most labeling, humans only the hard pairs","VLM cuts preference annotation in half for robot training","Adaptive VLM reduces human preference feedback by 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the KL-divergence threshold derived from clean-sample loss bounds still separates correct VLM-generated preference labels from incorrect ones once the reward model is trained on those labels; if that separation fails, the human budget is spent on effectively random pairs and the claimed 2x efficiency gain is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["VLM preference labels halve human feedback for robot RL","PrefVLM uses VLM labels, keeps humans for uncertain cases","Robot RL: VLM does most labeling, humans only the hard pairs","VLM cuts preference annotation in half for robot training","Adaptive VLM reduces human preference feedback by 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2055,"prompt_tokens":908,"completion_tokens":1147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1061}},"tokens_in":524,"tokens_out":1147,"duration_ms":10422,"temperature":1.0,"reasoning_tokens":1061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:47:42.254424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of trajectory pairs with human labels, run PrefVLM's KL filter over them, and compare the fraction of correct labels in the clean set ($D_{\\tau_l}$), the relabeled set ($D_{\\tau_u}$), and the uncertain set sent to humans. If the clean set's label accuracy is not substantially higher than the uncertain set's, or if the flipped labels in $D_{\\tau_u}$ are not more accurate than the originals, then the selection mechanism is not doing the work and the reported feedback savings would not transfer to new tasks.","supporting_citations":[],"review_version":1}