{"id":"bbc4da8f-9bc4-465b-bd1f-ab0543eac47f","arxiv_id":"2505.11221","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Showing that an LVLM teacher's soft action probabilities, blended into the RL loss, speed up MiniGrid agents by about 2.5x in sample efficiency.","lead":"This paper distills action advice from a large vision-language model into a small reinforcement learning agent, cutting the number of environment interactions needed to solve simple grid tasks. The method removes the need for hand-written text descriptions of the environment and may make RL more practical for small, resource-limited systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control for teacher informativeness: the 2.5-2.9x speedup could come from the KL regularizer or from privileged coordinates, not from LVLM knowledge. A random/scripted teacher control is missing.","rationale":"The paper presents a clean empirical framework: a KL distillation term from a prompted LVLM to an on-policy student, with ablations on λ and hard/soft targets. The reported improvements are plausible and the code release helps reproducibility. However, the experiments lack a critical control that distinguishes 'knowledge from the LVLM' from two simpler confounds: the regularizing effect of any soft target, and the privileged-information effect of a teacher that sees a fully-observable map while the student sees only a partial view. The reader's weakest assumption focused on the partial-observability alignment between teacher and student; my concern is closely related but more directly internal-validity-oriented: even in the fully-observable interpretation, the teacher's advice may be valuable only because it encodes global coordinates, not because the LVLM performs visual-language reasoning. This is testable with a uniform-teacher baseline (a) and a scripted-teacher baseline (b). If either matches LVLM2P's sample efficiency, the central claim about LVLM distillation is not supported as stated, although the framework might still be a useful RL acceleration technique. I also flag the inconsistency between the 'no manual descriptors' claim and the hand-crafted prompt in Fig. 2, but that affects generalizability rather than the core speedup. Because the missing control is independently verifiable and the current evidence is insufficient to decide, the verdict remains CONDITIONAL: the paper should be accepted only if the authors add the proposed control or otherwise demonstrate that the gain is specific to the LVLM teacher.","tokens_in":8990,"tokens_out":6995,"duration_ms":72393,"concrete_test":"Re-run the LavaGap and GoToDoor experiments with the identical pipeline and λ=0.01, replacing the Gemini teacher's soft probabilities with (a) a uniform distribution over the action set and (b) a scripted shortest-path policy that reads the same fully-observable grid as the LVLM but performs no language reasoning. Compare the environment steps needed to reach the target mean returns used in Fig. 3. If (a) or (b) matches the LVLM2P sample efficiency within 20%, the reported speedup is not attributable to LVLM knowledge. If (b) matches but (a) does not, the gain is driven by privileged coordinates, not by the LVLM's visual-language reasoning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that distilling soft action probabilities from a prompted LVLM into an RL student reduces sample complexity by 2.56x (PPO) and 2.86x (A2C) on four MiniGrid tasks. The only ablations reported (Fig. 4) vary the distillation coefficient λ and hard vs. soft labels; there is no baseline in which the teacher is replaced by an uninformative but equally smooth signal (e.g., uniform distribution) or by a non-LVLM privileged planner that receives the same fully-observable map. This omission makes the causal attribution to LVLM knowledge insecure: the observed speedup could instead be explained by (i) the KL term acting as entropy regularization/label smoothing, which is known to stabilize early exploration, or (ii) the teacher's access to global object coordinates that any scripted navigation policy could also provide, without any visual-language reasoning. Either alternative would show the same sample-efficiency gain while undermining the paper's central mechanism. The claim that manual textual descriptors are eliminated (Abstract, Section III) is also overstated, because the prompt in Fig. 2 is hand-crafted with task-specific object names, colors, and coordinate questions; this is a generalizability concern rather than the primary threat to internal validity. The missing teacher-informativeness control should be the first fix because it directly determines whether 'LVLM distillation' is the cause of the reported improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes LVLM2P, a framework that distills soft action probabilities from a pretrained vision-language model into a compact reinforcement-learning student. During training, a Gemini-1.5-Flash teacher is prompted with the agent's visual observation and a task-specific analysis/action-inference prompt, and the student optimizes L_RL + lambda * D_KL(pi_T || pi_theta). Experiments on four MiniGrid tasks compare PPO and A2C with and without LVLM2P. The paper reports average sample-efficiency improvements of 2.56x for PPO and 2.86x for A2C, and ablations show that soft targets outperform hard targets and that lambda = 0.01 gives the best success rate. The claimed contribution is that the framework reduces sample complexity without requiring the large teacher at test time and without hand-crafted textual state descriptors.","tokens_in":9265,"tokens_out":4694,"duration_ms":47303,"significance":"If the attribution is supported, the framework would be a practical way to transfer internet-scale visual-linguistic knowledge into small deployable policies without test-time access to the teacher. The paper provides code, evaluates on environment reward rather than teacher reward, and includes ablations of the distillation coefficient and target encoding; these are genuine strengths. However, the central causal claim that LVLM knowledge, rather than the KL regularizer or the teacher's privileged full-observation coordinates, drives the speedup is not yet established, and the statistical basis for the headline ratios is thin. The contribution is promising but requires additional controls and experiments before the central mechanism can be accepted.","major_comments":[{"comment":"The ablation study in Fig. 4 varies only the distillation coefficient lambda and hard versus soft targets; there is no control in which the teacher is replaced by an uninformative smooth distribution (e.g., uniform over actions) or by a scripted non-LVLM planner given the same fully-observable image. Without such a control, the 2.56x/2.86x speedups in Eq. (1) could be attributed to the KL term acting as label smoothing or entropy regularization, which is known to help early exploration, or to the privileged global coordinates in the teacher's input, rather than to LVLM reasoning. This control is load-bearing for the paper's central mechanism and should be added before the claim that LVLM distillation reduces sample complexity can be evaluated.","section":"Section V, Fig. 4"},{"comment":"The headline sample-efficiency ratios are computed from the curves in the bottom row of Fig. 3, but the bottom row reports only mean values without error bars, confidence intervals, or a statistical test on the ratios. The caption states that three seeds were used, so the top-row standard-deviation ranges exist; the same error propagation should be applied to the sample-efficiency estimates. Reporting the per-seed number of samples needed to reach each threshold would make it possible to judge whether 2.56x and 2.86x are robust effects or artifacts of the chosen thresholds.","section":"Section V, Fig. 3"},{"comment":"The teacher receives a fully-observable view of the environment (Section V setup), while the student policy trains on its own state representation, and Dynamic Obstacles is described as partially observable. The paper does not test whether the teacher's privileged observations remain aligned with the student's state when observability differs. Moreover, the claim that manual textual descriptors are eliminated (Abstract, Section III) is overstated: the prompt in Fig. 2 is hand-crafted per task with object names, colors, coordinate questions, and few-shot examples. The authors should either quantify the prompt-engineering cost or soften the claim, and should discuss or experimentally test transfer to other observation spaces.","section":"Section V setup and Fig. 2"}],"minor_comments":[{"comment":"There are several typos, including 'proprosed' in the first paragraph and 'intergrated' in Section V; these should be corrected.","section":"Section I"},{"comment":"The action list in the prompt contains 'Go Forward' twice and does not include 'Turn left', while the example output includes 'Turn left'; this inconsistency should be fixed.","section":"Fig. 2"},{"comment":"The lambda ablation is shown only for PPO, not for A2C; the text should state whether the same lambda value was used for A2C in the main experiments.","section":"Section V, Fig. 4"},{"comment":"References [24] and [25] have malformed author fields ('J. at al' and 'V. et al') and should be completed.","section":"References"},{"comment":"The paper does not report the wall-clock time, API cost, or number of teacher queries required during training; since the introduction emphasizes resource-constrained deployment, the authors should at least clarify that the reported sample-efficiency metric counts environment steps only and not teacher-query cost.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is not circular, since the teacher is an external pretrained model and the student is evaluated on environment reward. The decisive issue is attribution: the missing teacher-informativeness control determines whether the LVLM specifically matters or whether any smooth privileged signal would produce the same speedup. If the authors add this control and the effect survives, I would support acceptance; as it stands, the central mechanism is plausible but not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it takes the old Kickstarting distillation recipe [48] and swaps in a frozen LVLM prompted to emit soft action probabilities from a visual observation. That specific combination appears new, and the experiments are honestly run: four MiniGrid tasks, two base RL algorithms, ablations over the distillation weight and hard-vs-soft labels, and code released. The observed speedups (2.56x PPO, 2.86x A2C) are consistent across tasks, and the student is indeed deployed without the teacher at test time. This is a real empirical contribution, more than a rehash.\n\nThe soft spots are proportionate but real. The big one is the missing teacher-informativeness control. The stress-test note asks for a uniform or scripted teacher baseline, and it is the right question: as it stands, the speedup could come from the KL regularizer alone (lambda ablation shows the term matters, not that the teacher's content matters) or from the teacher's privileged full-map coordinates, which any scripted navigation policy could also provide. Without a control that varies teacher informativeness while holding the distillation structure fixed, the claim that \"LVLM knowledge\" causes the improvement is not fully secured. That is the first thing I would ask a revised version to add. Second, the prompts in Fig. 2 are still hand-crafted per task, with object names, colors, and coordinate questions; saying the method \"eliminates manual textual descriptors\" is overstated—it moves the manual work from state descriptions to prompt templates. Third, the headline sample-efficiency ratios are computed from a single target return per task with three seeds and no interval on the ratio itself; the success-rate plots show standard deviation ranges, but the 2.56x/2.86x numbers need error bars. Fourth, the teacher API cost is never reported, and if every environment step triggers a Gemini query, that is a real practical cost that belongs in the comparison.\n\nNone of these are fatal. They are addressable, and the paper deserves a serious referee rather than a desk reject. My guess is the speedup will survive a scripted-teacher control but shrink, and the authors will need to claim something more precise: distillation from a smooth, privileged action source helps on MiniGrid, and an LVLM is a convenient way to get that source without per-task training. That is still a useful finding. This paper is for practitioners in RL and foundation-model-guided control, not for theorists. I would not build on the LVLM-specific claim until the control is added, but I would cite it as related work and would bring it to our reading group now to discuss exactly that control.\n\nRecommendation: send to peer review, with a request for the teacher-informativeness control, error bars on the headline ratios, and a short paragraph reporting query cost or steps-per-query.","headline":"A clean, modest empirical result—LVLM soft-action distillation cuts MiniGrid sample counts 2-3x—but the missing teacher-informativeness control leaves the causal role of the LVLM's knowledge unproven.","tokens_in":9770,"tokens_out":2515,"would_cite":true,"duration_ms":29800,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large vision-language model advice cuts reinforcement learning sample use by about 2.5x","keywords":["reinforcement learning","sample efficiency","knowledge distillation","large vision-language models","large language models","MiniGrid","soft action probabilities","policy distillation"],"falsifier":"Train LVLM2P and vanilla PPO/A2C on a held-out task where the teacher keeps a top-down full map but the student sees only an egocentric, partially observable view, and compare environment steps to a fixed success threshold; if the speedup vanishes, the reported gains depend on the teacher's privileged viewpoint rather than on distillation itself.","tokens_in":8820,"feed_emoji":"🤖","tokens_out":7861,"duration_ms":72448,"temperature":0.7,"pith_summary":"LVLM2P is a training framework that uses a large vision-language model as a teacher during reinforcement learning. As the student policy collects trajectories, the teacher looks at images of the environment and returns soft probabilities over actions, and the student is trained to imitate those suggestions while also optimizing reward. The paper claims this reduces the number of environment interactions needed to reach a given return by an average of 2.56x for PPO and 2.86x for A2C across four MiniGrid tasks. The practical point is that a small student policy can inherit visual understanding from a much larger model and then run without the teacher at deployment time, avoiding both wasteful exploration and the cost of online large-model inference.","feed_headline":"LVLM advice cuts RL sample needs by 2.5x","feed_subtitle":"A small student policy inherits the visual skills of a large model and needs far fewer environment steps.","key_machinery":"The load-bearing object is the combined training loss. The student updates by maximizing the RL objective plus a distillation term, $$ \\mathcal{L}(\\pi_\\$\\theta$) = \\mathcal{L}_{\\mathrm{RL}}(\\pi_\\$\\theta$) + \\$\\lambda$ \\, D_{\\mathrm{KL}}\\big(\\pi_T(\\cdot|o,\\ell_{\\mathrm{in}})\\,\\big\\|\\,\\pi_\\$\\theta$(\\cdot|s)\\big), $$ where $\\pi_T$ is the LVLM teacher policy, $\\ell_{\\mathrm{in}}$ is the textual prompt, $o$ is the visual observation, and $s$ is the student's state. The two-stage prompt converts an image into a soft distribution over actions, and the KL term transfers the teacher's relative confidence across all actions; the RL term keeps the student reward-driven. The teacher is queried online on batches sampled from the student's own trajectory buffer, so the process needs no pretrained task-specific teacher and no textual state description.","core_discovery":"The central claim is that prompting a pretrained LVLM with an image and a two-stage text prompt—first a scene-analysis question, then an action-inference question with few-shot examples—produces a teacher policy whose soft action probabilities accelerate RL when added as a KL-divergence term to PPO/A2C objectives. The paper reports a teacher success rate of about 0.96 and student sample-efficiency gains averaging 2.56x for PPO and 2.86x for A2C over vanilla baselines. Because the student keeps optimizing the RL objective, it can override teacher mistakes, and because the teacher reads the image directly, no handcrafted textual state descriptors are needed.","pith_inferences":["The distillation objective needs only a teacher distribution over the student's action space, so porting LVLM2P to off-policy algorithms (for example DQN) or continuous-action policies is a direct next step not tested in this paper.","A cost-saving variant would anneal or disable LVLM queries once the student's own return approaches the teacher's level, reducing the number of large-model calls during training; this paper keeps the teacher active for all training steps.","Environments with stronger partial observability or action vocabularies that differ from the teacher prompt would test whether the 2.5x speedup is a property of the distillation mechanism or of the four MiniGrid tasks."],"forward_implications":["PPO and A2C integrated with LVLM2P reach target returns with roughly 2.1–2.9 times fewer environment steps on the four MiniGrid tasks than the vanilla algorithms.","The distilled student policy operates without the LVLM at test time, so deployment cost stays that of a small policy network.","Soft teacher probabilities outperform hard one-hot labels in the LavaGap ablation, and a small distillation weight $\\lambda=0.01$ gives the best success rate; larger weights hurt performance.","Because the teacher consumes the raw image, the method removes the need for handcrafted textual descriptions and can be layered onto either PPO or A2C."],"supporting_citations":[{"why":"Supplies Gemini-1.5-Flash as the LVLM teacher, chosen for its inference-time and accuracy trade-off in the paper's experiments.","marker":"[15]"},{"why":"Establishes knowledge distillation with soft targets, the basis for the teacher's soft action probabilities.","marker":"[23]"},{"why":"PPO is one of the two base RL algorithms whose sample complexity LVLM2P improves.","marker":"[24]"},{"why":"A2C is the other base RL algorithm used in the experiments.","marker":"[25]"},{"why":"Kickstarting is the training paradigm LVLM2P extends by replacing pretrained teacher policies with a prompted LVLM.","marker":"[48]"},{"why":"Used to support soft over hard targets during distillation.","marker":"[49]"},{"why":"Provides the four MiniGrid benchmark tasks used for evaluation.","marker":"[50]"},{"why":"Motivates the few-shot prompting design for the LVLM teacher.","marker":"[13]"}],"fun_headline_variants":["LVLM teacher cuts RL sample needs by 2.5x","LVLM wisdom into RL: 2.5x sample savings","2.5x faster RL with LVLM distillation","LVLM2P: visual teacher speeds RL learning","Distilling LVLM to 2.5x sample-efficient RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speedup assumes that actions suggested by a large model looking at a full image of the scene remain sensible for a student that perceives the environment through its own state representation; the paper checks this on only four MiniGrid tasks, so the 2–3x gain may not transfer when that alignment breaks.","fun_headline_variants_meta":{"raw":{"variants":["LVLM teacher cuts RL sample needs by 2.5x","LVLM wisdom into RL: 2.5x sample savings","2.5x faster RL with LVLM distillation","LVLM2P: visual teacher speeds RL learning","Distilling LVLM to 2.5x sample-efficient RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1780,"prompt_tokens":840,"completion_tokens":940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":855}},"tokens_in":456,"tokens_out":940,"duration_ms":9020,"temperature":1.0,"reasoning_tokens":855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:23.784436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LVLM2P and vanilla PPO/A2C on a held-out task where the teacher keeps a top-down full map but the student sees only an egocentric, partially observable view, and compare environment steps to a fixed success threshold; if the speedup vanishes, the reported gains depend on the teacher's privileged viewpoint rather than on distillation itself.","supporting_citations":[{"cited_title":"Asynchronous methods for deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"A2C is the other base RL algorithm used in the experiments."},{"cited_title":"When does label smoothing help?","cited_arxiv_id":null,"evidence_quote":"Used to support soft over hard targets during distillation."},{"cited_title":"Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal- oriented tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the four MiniGrid benchmark tasks used for evaluation."}],"review_version":1}