{"id":"820fdb3b-5f0b-4995-a1c6-cb334cb914ad","arxiv_id":"2607.09042","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hindsight language-and-reward relabeling of failed VLA rollouts by a VLM yields roughly 5× sample-efficiency gains over standard GRPO on OOD LIBERO-PRO and real Franka tasks.","lead":"Learning from Hindsight (LfH) turns failed robot rollouts into training signal by having a vision-language model rewrite both the instruction and the reward for what the robot actually did. This yields about 5× better sample efficiency when fine-tuning vision-language-action models under sparse rewards, including on a real Franka arm.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The 5× sample-efficiency claim rests on VLM-generated hindsight advantages remaining informative after the importance-ratio correction of Eq. 7; that correction is unvalidated and can inject high-variance or biased gradients when g' diverges from g.","rationale":"The reader correctly flags VLM reliability as a fragility and assigns a CONDITIONAL verdict with moderate confidence; that assessment is sound given the multi-seed, multi-backbone, and real-robot evidence. The more precise soft spot, however, is not merely whether the VLM invents objects, but whether the subsequent importance-corrected GRPO update (Eq. 7) turns those labels into gradients that actually improve the commanded tasks. The paper never measures ratio statistics or abates the correction, so the quantitative 5× claim is one untested assumption away from being an artifact. Because the rest of the experimental design is careful and the real-robot transfer still holds under the same pipeline, the appropriate adjustment is not a rejection but a tighter CONDITIONAL: accept only after the ratio ablation (or equivalent diagnostics) is supplied. This is a refinement of the reader’s weakest-assumption rather than a contradiction of it.","tokens_in":17360,"tokens_out":748,"duration_ms":10463,"concrete_test":"On the same LIBERO-PRO OOD suites and seeds used for Fig. 3a, re-run LfH with three variants of the hindsight ratio: (i) the paper’s Eq. 7, (ii) the ordinary on-policy ratio under g' after a single re-rollout of each trajectory under g', and (iii) the ratio forced to 1 (no correction). If the 5× step-to-match-final-success disappears or shrinks by >40 % under (ii) or (iii), the importance correction is load-bearing and the headline efficiency claim is overstated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LfH recovers usable GRPO signal from all-zero groups and thereby reaches standard GRPO’s final success in ~1/5 the steps (Fig. 3a). That recovery depends on the hindsight objective (Eqs. 6–8). Because trajectories are sampled under the original instruction g but the policy is now evaluated under a VLM-proposed g', the paper replaces the ordinary importance ratio with the off-policy ratio of Eq. 7: π_\theta(a|o,g') / π_\theta old(a|o,g). When g' is semantically distant from g (as the microwave\to mug examples in Fig. 2b illustrate), the two conditionals can differ substantially; the ratio then becomes high-variance or systematically biased, and the clipped surrogate can still push the policy in directions that do not transfer back to the commanded task. The paper never reports the distribution of these ratios, the fraction of groups discarded by clipping, or an ablation that disables the correction (or replaces it with on-policy re-sampling under g'). Without that check, it remains possible that a non-negligible fraction of the reported 5× gain is an artifact of noisy off-policy updates rather than genuine language-mediated transfer. The VLM-quality assumption identified by the reader is real, but the importance-ratio step is the more immediate load-bearing link between the relabeled data and the claimed efficiency number.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Learning from Hindsight (LfH) for sample-efficient RL post-training of vision-language-action (VLA) policies under sparse rewards. Failed GRPO groups (near-zero mean reward) are converted into training signal by a single VLM that (i) proposes a shared hindsight instruction describing what an anchor failed rollout actually did and (ii) scores every trajectory in the group under that instruction with ternary rewards {0, 0.5, 1}. The policy is trained jointly on original and hindsight-relabeled groups via an importance-corrected GRPO objective (Eqs. 6–9, Algorithm 1). On out-of-distribution LIBERO-PRO task-perturbation suites, LfH is reported to match standard GRPO’s final success in roughly one-fifth the training steps (~5× sample efficiency), to outperform a dense progress-reward baseline (RoboMETER), and to transfer across Pi-0.5, GR00T, and OpenVLA-OFT as well as to a physical Franka task.","tokens_in":17715,"tokens_out":1467,"duration_ms":24927,"significance":"If the reported gains hold under scrutiny, LfH is a practical and timely contribution: robot rollouts are expensive, sparse-reward cold starts are common for VLA fine-tuning, and language-space hindsight is a natural fit for pretrained VLAs. Strengths include a clear algorithmic construction, careful same-budget comparisons with standard error over seeds, ablations that isolate rephrase-only, reward-only, and random-reward alternatives (all weaker than full LfH), group-selection and VLM-prompt ablations, multi-backbone transfer, and a real-robot result with human binary labels. The work is complementary to dense progress rewards and to offline instruction-augmentation methods, and it makes a concrete case that failed trajectories can be mined as multi-task supervision rather than discarded.","major_comments":[{"comment":"Section 4.2, Eq. (7): the hindsight importance ratio ˜r_{i,t}=π_θ(a|o,g′)/π_θold(a|o,g) is load-bearing for the claimed mechanism (language-mediated transfer from off-policy hindsight groups). When g′ is semantically distant from g (as in the microwave→mug examples of Fig. 2b), the two conditionals can differ substantially, producing high-variance or biased weights even after clipping. The manuscript never reports the distribution of these ratios, the fraction of mass affected by the hindsight-loss clip c (Appendix B.1), or an ablation that disables the correction / replaces it with on-policy re-rollout under g′. Without that check, it remains unclear how much of the ~5× gain in Fig. 3a is genuine language transfer versus noisy off-policy updates. A short diagnostic (ratio histograms, clip rates, and one ablation) would substantially strengthen the central efficiency claim’s interpretati","section":"Section 4.2, Eq. (7)"},{"comment":"Section 5.1 and Fig. 2b: the paper observes that hindsight instructions need not be semantically related to the commanded task, yet still improve target-task success, and offers a brief “contrastive grounding” hypothesis. This is central to the narrative that language generalization bridges hindsight successes back to Pg. The evidence is currently qualitative (one microwave example) plus the aggregate gain curves. A minimal quantitative check—e.g., success on the original instruction as a function of semantic distance between g and g′, or a controlled set of related vs. unrelated hindsight labels—would make the transfer story falsifiable rather than post-hoc. Absent that, the 5× claim remains empirically supported but mechanistically under-specified.","section":"Section 5.1, Fig. 2b"},{"comment":"Section 5.4 / Fig. 4b: real-world evaluation is limited to a single held-out task (“put the green container into the bowl”) with human binary rewards and a modest rollout budget. The sim results are multi-suite and multi-backbone; the physical claim in the abstract (“gains hold … on a physical Franka robot”) currently rests on one curve. At least one additional held-out task, or a clear statement that real-world evidence is preliminary and single-task, is needed so the abstract claim matches the evidence.","section":"Section 5.4, Fig. 4b"}],"minor_comments":[{"comment":"Eq. (1) vs. Algorithm 1 / Appendix B.1: the main text uses threshold η on mean group reward, while the appendix describes skipping groups with mean reward in [0.8, 1.0]. State η explicitly in the main method section and keep notation consistent.","section":"Section 4.1 / Algorithm 1"},{"comment":"Figure 3a caption and §5.2: “Gain(t)=SRt/SR0−1” is undefined or unstable when SR0≈0 (as in the microwave illustrative example). Clarify how Gain is computed when the initial success rate is zero, or report absolute success for those cases.","section":"Section 5.2, Fig. 3a"},{"comment":"Related Work: concurrent open-ended instruction relabeling [16] is acknowledged; a one-sentence contrast on online GRPO integration and robotics evaluation would help readers place the contribution more sharply.","section":"Section 2"},{"comment":"Appendix B.1: hyperparameters λ and clip c for the hindsight loss are described procedurally (“85th percentile”) but not given numerical values in the main text or a single table. A compact hyperparameter table would aid reproducibility.","section":"Appendix B.1"},{"comment":"Typos / polish: “Robometer” vs. “RoboMETER” inconsistency in Fig. 3/4 captions; “π 0.5” spacing in Fig. 3c caption; arXiv-style citation formatting is fine for preprint but should be cleaned for journal production.","section":"Figures 3–4"}],"recommendation":"minor_revision","confidential_remarks":"The empirical package is stronger than the mechanistic analysis. I would not block on the importance-ratio gap if the authors add the diagnostics requested; if they refuse any analysis of Eq. (7) and keep the strong language-transfer framing in the abstract, I would push toward major_revision. Scope fit for a methods-oriented ML/robotics journal is good. No circularity or novelty-disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: they turn all-zero GRPO groups into usable signal by having one VLM propose a shared hindsight instruction for a failed group and score every trajectory under it, then train jointly with an importance-corrected GRPO term. On OOD LIBERO-PRO (task perturbation) that is enough to match standard GRPO’s final success in roughly one-fifth the steps, beat a dense RoboMETER progress baseline, and transfer to GR00T, OpenVLA-OFT, and a Franka task that starts at zero success.\n\nWhat is actually new is the online joint instruction+reward procedure inside GRPO (shared g′, 0/0.5/1 scores, “Nothing”/unsure filters, threshold η, and the off-policy ratio of Eq. 7), not the high-level idea of hindsight or VLM relabeling. The experiments are careful: same rollout budget, multi-seed with SE, ablations that kill pure rephrase, reward-only, and random-reward explanations, group-selection and prompt ablations, and a real-robot curve that roughly doubles success at 128–160 rollouts. That package is stronger than most concurrent language-hindsight notes.\n\nSoft spots, in proportion. The load-bearing assumption is still VLM quality from RGB alone; they mitigate it but do not remove it, and they use a 235B model. The stress-test point on Eq. 7 is fairer than I first thought: when g′ is distant (mug vs microwave), the ratio can be high-variance, and they never show ratio histograms, clip rates, or an ablation that drops the correction or re-samples under g′. That does not collapse the result—the kept-group fraction and multi-backbone transfer still look real—but it means part of the 5× number is less tightly attributed than the abstract implies. Real-robot evidence is one held-out task; free parameters (η, λ, clip) are tuned but not wildly free. Citations are standard and honest.\n\nThis is for people doing online RL post-training of VLAs who care about sparse-reward cold starts. It deserves a serious referee. I would engage with it, cite the empirical package if I am working in the same regime, and ask for the ratio ablation in revision.","headline":"Solid online hindsight for VLA RL with real multi-backbone and Franka gains; the 5\times claim is empirically supported but the off-policy ratio is under-checked.","tokens_in":18385,"tokens_out":585,"would_cite":true,"duration_ms":7296,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Failed robot rollouts can train VLAs if a VLM relabels them as successes at the tasks they actually did.","keywords":["hindsight relabeling","vision-language-action models","reinforcement learning","sample efficiency","sparse rewards","robot manipulation","GRPO","VLM reward models"],"falsifier":"Run the same OOD LIBERO-PRO protocol with a deliberately degraded or blinded VLM relabeler (or with random hindsight instructions) and check whether the 5\times sample-efficiency gap over standard GRPO and the dense-progress baseline disappears; if the gap remains, the VLM-quality premise is not load-bearing.","tokens_in":18227,"feed_emoji":"🤖","tokens_out":963,"duration_ms":9906,"temperature":0.7,"pith_summary":"Robot fine-tuning with reinforcement learning is starved for data: each update needs slow physical rollouts, and sparse success rewards make early failures look useless even when the robot executed coherent skills. This paper argues that those failures are successes under different language instructions, and that a vision-language model can recover that signal. Learning from Hindsight (LfH) takes groups of failed rollouts, has one VLM invent a shared hindsight instruction describing what the robot actually did, scores every trajectory against that instruction, and jointly trains the policy on original and relabeled data with an importance-corrected group-relative update. Because VLAs already generalize across language, credit for nearby skills transfers back to the commanded tasks. On hard out-of-distribution LIBERO-PRO suites the method reaches standard RL's final success in roughly one-fifth the steps (about 5\times sample efficiency), beats a dense progress-reward baseline, and the gains transfer across VLA backbones and to a real Franka arm.","feed_headline":"Failed robot trials train VLAs 5\times faster via hindsight","feed_subtitle":"A VLM turns sparse-reward failures into successes at the tasks the robot actually did","key_machinery":"Learning from Hindsight (LfH): for low-mean-reward GRPO groups, a single VLM proposes one shared hindsight instruction from an anchor failed trajectory and assigns 0/0.5/1 rewards to the whole group under that instruction; the policy is then updated with both the original GRPO loss and an importance-corrected hindsight GRPO loss that evaluates actions under the new instruction while correcting for having been sampled under the old one.","core_discovery":"On out-of-distribution LIBERO-PRO tasks where standard sparse-reward GRPO improves only slowly from a weak initial policy, Learning from Hindsight recovers usable training signal from most failed rollouts by language-and-reward relabeling, reaches the final success rate of standard RL in roughly one-fifth the training steps (approximately 5\times sample-efficiency gain), and outperforms a dense progress-reward baseline; the same gains hold across multiple VLA backbones and on a physical Franka robot.","pith_inferences":["If VLM relabel quality continues to improve, the same recipe could turn large offline failure corpora into free multi-task post-training data without new robot collection.","Hindsight language credit may also help counteract behavioral collapse during RL, because alternative skills keep receiving positive updates instead of being uniformly treated as failures.","The method suggests a practical division of labor: let the VLA explore physical contact while a frozen VLM decides which language goals those contacts support."],"forward_implications":["Each costly robot trajectory can supervise multiple language goals consistent with what actually happened, raising bits extracted per physical sample.","In low-success sparse-reward regimes, changing which task a failure is credited for can be more valuable than densifying progress rewards for the original command alone.","The same relabeling loop works across different VLA architectures and transfers from simulation to a physical Franka without changing the human binary reward.","Failed attempts cease to be pure negative examples and become reusable experience for nearby object interactions and grounding skills."],"fun_headline_variants":["Hindsight relabeling gives VLAs 5× sample efficiency from failures","VLM turns sparse failed rollouts into 5× faster VLA post-training","Language-and-reward hindsight yields 5× gains on LIBERO-PRO VLAs","Failed trials train VLAs 5× faster when relabeled by hindsight VLM"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The whole method rests on one off-the-shelf vision-language model being able to invent a shared, accurate language description of what a group of failed robot videos actually achieved and to score them consistently enough that the resulting rewards give useful advantages rather than noise.","fun_headline_variants_meta":{"raw":{"variants":["Hindsight relabeling gives VLAs 5× sample efficiency from failures","VLM turns sparse failed rollouts into 5× faster VLA post-training","Language-and-reward hindsight yields 5× gains on LIBERO-PRO VLAs","Failed trials train VLAs 5× faster when relabeled by hindsight VLM"]},"model":"grok-4.5","effort":"low","cost_usd":0.005112,"raw_usage":{"total_tokens":1390,"prompt_tokens":807,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":51120000,"prompt_tokens_details":{"text_tokens":807,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":807,"tokens_out":76,"duration_ms":5212,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T00:45:19.036202+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same OOD LIBERO-PRO protocol with a deliberately degraded or blinded VLM relabeler (or with random hindsight instructions) and check whether the 5\times sample-efficiency gap over standard GRPO and the dense-progress baseline disappears; if the gap remains, the VLM-quality premise is not load-bearing.","supporting_citations":[],"review_version":1}