{"id":"ad472f23-9210-4bf8-b513-2aa26d1a2a97","arxiv_id":"2607.08837","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Prompt-Driven Exploration refines language prompts from rollout videos via a VLM, enabling RL to escape zero-reward VLA and LLM policies where action-space noise fails.","lead":"Rewriting a robot policy's language prompt from video of its failures can unlock learning when action noise cannot. Prompt-Driven Exploration uses a VLM as an implicit posterior over prompts so RL can bootstrap VLAs and LLMs from near-zero reward.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Transfer of exploratory successes to the canonical prompt is the load-bearing hinge; if mixed backprop fails, PDE only improves under non-eval prompts.","rationale":"The reader correctly flags the VLM-as-posterior plus transfer premise as the weakest assumption. I sharpen it to the transfer half: discovery of nonzero-reward prompts is demonstrated (microwave categories, Fig. 10, Regime 2), but the claim that RL then produces a successful policy under the original pg depends on mixed backprop and the α schedule. The paper already contains partial evidence (mixed > original-only / curriculum-only; single-best fails under pg), yet does not ablate transfer while holding the discovered pool fixed on the hardest zero-init tasks. That is the single most load-bearing remaining check. No internal inconsistency appears; the concern is empirical isolation of the transfer mechanism. Verdict therefore stays CONDITIONAL with the same moderate confidence; no upgrade or downgrade is warranted until the proposed isolation is run. Agreement with the reader is full on the identity of the soft spot.","tokens_in":28520,"tokens_out":646,"duration_ms":6404,"concrete_test":"On the microwave task and the 10 Object/task-perturbation tasks of Table 2/9, re-run PDE with discovery frozen (same prompt pool) but replace mixed backprop (Eq. 3) by pure curriculum-prompt backprop (no pg term) while still evaluating only under pg; if final SR under pg falls by >15 points relative to the mixed baseline (or matches the curriculum-only 45% of Table 9), the transfer hinge fails and the zero-reward claim under canonical prompts weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that PDE yields successful policies under the canonical evaluation prompt pg even from zero-reward starts. That claim rests on two coupled pieces: (1) the VLM discovers prompts that unlock nonzero reward from a frozen weak VLA (§4.1, Alg. 1 lines 4–6), and (2) mixture sampling plus mixed backpropagation (Eq. 3 / §4.2) transfers those successes so that πθ(·|o,pg) improves. The paper’s own Regime 3 (Table 2) and the single-best-prompt ablation (Tables 8–9) show that (1) alone is insufficient: exploratory prompts can remain at 0% while later RL under the transfer machinery reaches high SR under pg. Conversely, training only on the best discovered prompt scores high under that prompt but collapses under pg (39.6% vs 63.8%). Thus the transfer step is load-bearing. If the geometric-mean log-prob average in Eq. 3 does not couple the two prompt-conditioned policies tightly enough—or if the adaptive α schedule keeps most mass on exploratory prompts—the reported gains under pg would be overstated relative to pure action-noise PPO. The microwave and hard-tier curves are consistent with successful transfer, but the mechanism is not isolated from the discovery stage on the hardest zero-reward tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Prompt-Driven Exploration (PDE): for language-conditioned VLAs/LLMs, a VLM refines a distribution over natural-language prompts from rollout videos and success feedback, inducing global behavior changes that action-space noise rarely produces. The authors cast this as posterior sampling over an induced prompt-conditioned policy class (Alg. 1, §4.1), then transfer exploratory successes to the canonical evaluation prompt via mixture sampling and mixed backpropagation (Eq. 3, §4.2). Empirically, PDE is evaluated on a microwave case study, difficulty-stratified LIBERO-PRO (Pi0.5/GR00T/Pi0), ManiSkill OOD axes, three real Franka tasks, LiveCodeBench, and AIME, with ablations on random paraphrases, single-best prompts, and mixed backprop.","tokens_in":28874,"tokens_out":1354,"duration_ms":23022,"significance":"Exploration under near-zero success is a genuine bottleneck for VLA post-training; using the existing language interface as a structured exploration axis is a clean and timely idea. Strengths include breadth of evidence (simulation, real robot, LLM coding/math), interpretable prompt categories in the microwave study, difficulty-stratified aggregation, and ablations showing that unoptimized paraphrases and single-best-prompt training are insufficient (Fig. 9, Tables 8–9). If the transfer and budget claims hold under fair accounting, PDE is a practical contribution that other VLA RL pipelines can adopt without new density models or reward densifiers.","major_comments":[{"comment":"Sample-budget accounting for the two-stage schedule (App. A.2) is load-bearing for the sample-efficiency claim. Prompt discovery freezes θ and spends T0×K×N rollouts per task before PPO; the microwave study alone uses ~85 discovery rollouts. Figures 4–6 and the abstract claim higher success with far fewer environment interactions, but it is unclear whether discovery rollouts are included on the Env Steps axis or only the RL stage. Please report total environment interactions (discovery + RL) for PDE and baselines on the same axes, or mark discovery cost separately, so the efficiency comparison is not understated for action-noise PPO.","section":"§4.2 / App. A.2 / Figs. 4–6"},{"comment":"Transfer to the canonical prompt pg is the hinge of the central claim (successful policies under evaluation prompts from zero-reward starts), not discovery alone. Table 2 Regime 3 and Tables 8–9 already show discovery can be 0% while later RL under mixture+mixed backprop succeeds, and that training only on the best exploratory prompt collapses under pg (39.6% vs 63.8%). On hard zero-init tasks, please isolate transfer more cleanly: e.g., discovery-only (no θ update), discovery+PPO without mixed backprop (Eq. 3), and full PDE, all evaluated only under pg, with discovery cost counted. Without this, gains on hard tiers could be partly attributed to extra successful data under non-eval prompts rather than improved πθ(·|o,pg).","section":"§4.2 Eq. (3) / Table 2 / Tables 8–9"},{"comment":"The PSRL framing (§4.1) is an analogy: ρ is an implicit VLM sampler with no explicit density, likelihood, or Bayesian update, and history is compressed to one-sentence summaries plus success rates (App. A.1). That is fine as motivation, but phrases such as “realizes posterior sampling” and “amortized posterior update” overstate formality relative to classical PSRL. Please reframe as “PSRL-inspired prompt-space exploration” unless a concrete correspondence (e.g., what constitutes the likelihood and prior) is stated, so the contribution is not judged as a theoretical PSRL result.","section":"§4.1 / Abstract"}],"minor_comments":[{"comment":"Table numbering in the appendix is inconsistent with in-text references (e.g., GR00T/Pi0 aggregate appears as Table 11 after Table 14 in the source order). Renumber tables in reading order.","section":"Appendix tables"},{"comment":"Hyperparameters for the VLM supervisor (which model, temperature, frames_per_video) should be stated once in the main experimental setup, not only implied by templates in App. A.5.","section":"§5.1 / App. A.5"},{"comment":"Figure 1 caption refers to improving from 10% to 50%, while the microwave narrative in §5.2 discusses ~0% to ~98%; align figure choice and caption with the section that cites it.","section":"Fig. 1 / §5.2"},{"comment":"Clarify whether language-perturbation tasks were excluded solely for confounding (§5.3) and whether that exclusion biases the hard-tier set toward tasks where prompt rephrasing is especially effective.","section":"§5.3"},{"comment":"In Eq. (3), the denominator uses π_old under p while the numerator averages log-probs under pg and p; a one-sentence justification that this remains a valid importance ratio (or is a heuristic) would help readers.","section":"Eq. (3)"},{"comment":"Real-world results (Fig. 8, Table 5) use only three tasks and 20 eval rollouts; report confidence intervals or binomial CIs so the “more than twice” claim is statistically readable.","section":"§5.6 / Table 5"}],"recommendation":"major_revision","confidential_remarks":"Solid empirical methods paper with real-robot and LLM extensions; the main risk is overstated sample-efficiency and PSRL rhetoric rather than a broken algorithm. If the authors fix budget accounting and add the transfer isolation on hard tasks, this is close to a clear accept at a strong ML venue. Fit is appropriate for a learning/robotics methods track."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: when a VLA starts near zero success, action noise rarely finds reward, but a VLM that rewrites the language prompt from rollout video can unlock globally different behaviors, and PPO with mixture sampling plus mixed backprop can transfer those successes back to the original evaluation prompt. That combination is the actual contribution.\n\nWhat is new is not prompt optimization or PSRL in isolation—they cite both—but treating the prompt distribution as a tractable policy posterior, updating it online from video feedback, and coupling exploratory prompts to the canonical one so evaluation gains are real. The microwave case study is clean: the SFT policy keeps grabbing the mug; prompts like “push the microwave door until it clicks” flip the motor program without weight updates, then RL drives canonical-prompt success near 100% while action-noise PPO stays low. Difficulty-stratified LIBERO-PRO on Pi0.5/GR00T/Pi0, ManiSkill OOD axes, three Franka tasks, plus LiveCodeBench/AIME early-efficiency gains make the empirical case broad. Ablations land where they should: random paraphrases fail; single-best-prompt scores high under itself but collapses under pg; mixed backprop beats original-only or curriculum-only.\n\nThe stress-test concern is fair but overstated as a hidden flaw. Transfer is load-bearing—Regime 3 and Tables 8–9 show discovery alone is not enough—but the paper isolates it rather than papering over it. Hard-tier variance is high, the two-stage freeze is a practical choice not a theory, and everything depends on a strong external VLM with no released code. The “posterior sampling” language is an analogy, not a derived Bayesian update. None of that breaks the central claim under the reported budgets.\n\nThis is for people doing RL post-training of VLAs or reasoning LLMs in the sparse-reward regime. It deserves a serious referee. I would engage with it and expect it to influence practice if the transfer recipe holds under other supervisors.","headline":"Solid practical method for bootstrapping weak VLAs via prompt-space exploration; transfer to the canonical prompt is the real hinge, and the paper mostly shows it works.","tokens_in":29520,"tokens_out":516,"would_cite":true,"duration_ms":5926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Rewriting the language prompt lets RL escape zero-success robot and coding policies that action noise cannot fix.","keywords":["prompt-driven exploration","reinforcement learning","vision-language-action models","posterior sampling","prompt optimization","sample efficiency","robot manipulation"],"falsifier":"On a held-out suite of hard LIBERO-PRO or real Franka tasks where the initial policy has zero success, run PDE versus action-noise PPO under identical rollout budgets: if the VLM-rewritten prompts never produce nonzero reward, or if success under the original prompt never rises while exploratory prompts succeed, the central claim fails.","tokens_in":29400,"feed_emoji":"🤖","tokens_out":624,"duration_ms":5717,"temperature":0.7,"pith_summary":"Reinforcement learning stalls when a policy almost never succeeds: random action jitter only produces near-identical failures, so there is no reward signal to reinforce. This paper argues that language-conditioned policies (robot VLAs and LLMs) open a different exploration axis. Because the whole rollout follows from the natural-language prompt, changing the prompt can redirect the entire strategy. The method, Prompt-Driven Exploration (PDE), has a vision-language model watch the failed rollout videos, diagnose what the policy did, and rewrite the prompt to try something more useful next time. Those rewritten prompts act like posterior samples over policies; once they produce successful trajectories, ordinary RL transfers the skill back to the original task wording. On hard robot-manipulation suites and real Franka tasks where the starting policy scores near zero, PDE reaches high success under the original prompt while action-noise baselines stay near zero; the same idea also speeds early learning on coding and math LLM tasks.","feed_headline":"Rewriting the prompt unlocks RL from zero-success starts","feed_subtitle":"A VLM diagnoses failed rollouts and rewrites instructions so policies finally succeed","key_machinery":"Prompt-Driven Exploration (PDE): a VLM maintains an implicit distribution over prompts, updates it from compressed rollout summaries and success rates, samples exploratory prompts, and transfers the resulting successes back to the canonical prompt via mixture sampling and mixed backpropagation inside PPO (or analogous LLM RL algorithms).","core_discovery":"Prompt-Driven Exploration turns a vision-language model into an implicit posterior sampler over natural-language prompts. By diagnosing rollout videos and rewriting the prompt, PDE discovers globally different behaviors that action-space noise cannot reach, supplies the first successful trajectories from a zero-reward start, and lets RL fine-tuning improve the policy under the original evaluation prompt.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["VLM diagnoses rollouts and rewrites prompts for RL exploration","Prompt rewriting supplies first successes from zero-reward starts","PDE turns VLMs into posterior samplers over language prompts","Global prompt changes escape weak policies action noise cannot","Rollout video diagnosis rewrites prompts so RL can improve"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The method assumes a frozen vision-language model can reliably diagnose short rollout videos and propose prompts that unlock nonzero reward from a fixed weak policy, and that those successes will transfer back to the original evaluation wording.","fun_headline_variants_meta":{"raw":{"variants":["VLM diagnoses rollouts and rewrites prompts for RL exploration","Prompt rewriting supplies first successes from zero-reward starts","PDE turns VLMs into posterior samplers over language prompts","Global prompt changes escape weak policies action noise cannot","Rollout video diagnosis rewrites prompts so RL can improve"]},"model":"grok-4.5","effort":"low","cost_usd":0.004098,"raw_usage":{"total_tokens":1263,"prompt_tokens":774,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":40980000,"prompt_tokens_details":{"text_tokens":774,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":408,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":774,"tokens_out":81,"duration_ms":3895,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T06:20:46.010336+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out suite of hard LIBERO-PRO or real Franka tasks where the initial policy has zero success, run PDE versus action-noise PPO under identical rollout budgets: if the VLM-rewritten prompts never produce nonzero reward, or if success under the original prompt never rises while exploratory prompts succeed, the central claim fails.","supporting_citations":[],"review_version":1}