{"id":"80378a93-0c37-4d3f-8b70-fb5bb4c69e1a","arxiv_id":"2506.20061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OIR relabels failed trajectories via an LLM into open-ended instructions and trains a unified instruction-following policy, outperforming PQN and ELLM on Craftax.","lead":"Using a large language model to rewrite failed agent episodes into new task instructions, this paper trains a single instruction-following RL policy without human-labeled instruction data. The approach improves sample efficiency and instruction coverage on the Craftax benchmark, pointing toward cheaper, more open-ended RL training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 8's cosine-similarity reward is the only training signal (all environment rewards removed), yet its correlation with true task completion is never measured; if misaligned, OIR's reported gains could come from reward hacking rather than instruction following.","rationale":"The reader's weakest assumption is also the most load-bearing. The central claim is empirical: failed trajectories can be converted into useful instruction-following data via LLM relabeling. That conversion is operationalized entirely through Eq. 8: rewards are assigned by embedding similarity and success is declared by crossing δ. In Section 5, all environment rewards are removed during training, so the cosine-similarity proxy is not an auxiliary shaping term; it is the sole credit-assignment signal. If this proxy does not track actual task completion, then the relabeled dataset contains false-positive successes, the policy is optimized toward spurious state-instruction alignment, and the reported ground-truth evaluation gains (Figs. 2-3) are open to alternative explanations (e.g., dense reward shaping or exploitation of embedding quirks). The paper provides no test of proxy validity; the Fig. 4 threshold sweep only demonstrates sensitivity to δ, not that r_t > δ matches true achievement. Section 4.1's acknowledgment that LLM candidates can be inaccurate, and its mention of rule-based instructions that are never specified, makes the missing validation more conspicuous. I therefore recommend keeping the reader's CONDITIONAL stance: the concern is addressable with a straightforward correlation/precision-recall check, and the verdict should be conditional on that check. Secondary issues (missing HIR baseline, inconsistent numeric text vs. figure captions, buffer size 10) reinforce the need for care but are not the single load-bearing point.","tokens_in":12400,"tokens_out":7341,"duration_ms":81673,"concrete_test":"During or after OIR training, collect all (transition, instruction) pairs that received r_t > δ (for δ ∈ {0.3, 0.5, 0.7, 0.9}). For the 22 environment-grounded instructions, use Craftax's built-in achievement flags as the ground-truth label of whether the instruction was actually completed at that transition; for LLM-generated instructions, have a human or a second independent LLM annotate completion. Then compute precision, recall, and AUC of the binary classifier 'r_t > δ' against ground-truth completion. If precision/recall are substantially below 0.9 (or AUC below ~0.95), the training reward is not a faithful proxy and OIR's ground-truth evaluation gains cannot be attributed to genuine instruction following; in that case, rerun the comparison with relabeled ground-truth rewards to see whether the headline improvement persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Algorithm 1 and Section 4.2, every training reward is r_t = cosim(f_state(o_t,a_t,o_{t+1}), f_instr(i)), success is declared when r_t > δ, and Section 5 states that all built-in rewards and achievement signals are removed during training. The paper never validates that this semantic similarity actually coincides with the environment's ground-truth achievement for the relabeled instruction. Because evaluation (Figs. 2-3) uses the true achievement signals, a policy that merely exploits embedding similarity—e.g., high cosim between 'collect wood' and a state with trees nearby without an inventory change—can receive dense training reward without completing the task. The consequences are direct: (i) the claimed transformation of failed trajectories into useful instruction-following data depends on the proxy being a reliable success detector; (ii) the advantage over PQN/ELLM could be an artifact of reward density/shaping rather than of open-ended relabeling. Section 4.1 even concedes LLM candidates can be inaccurate and says 'rule-based instructions' are used, but no rule-based instructions or filtering steps are specified, so the proxy is the only quality control. The threshold ablation (Fig. 4) shows sensitivity but not alignment: it never compares predicted success to actual success on the same trajectories.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Open-Ended Instruction Relabeling (OIR), an off-policy instruction-conditioned RL algorithm that uses an LLM to generate open-ended natural-language instructions from collected trajectories, including unsuccessful ones. These instructions are embedded and used to compute dense cosine-similarity rewards (Eq. 8); success is declared when the similarity exceeds a threshold δ, and a prioritized replay buffer manages the growing instruction set. Experiments on Craftax-Classic compare OIR against PQN with a cosine-similarity reward, PQN with the ground-truth reward, and ELLM, and the authors report improvements in aggregate score, completed instructions, average success rate, generalization to simple and complex instruction variants, and semantic coverage, with an ablation over δ.","tokens_in":12664,"tokens_out":7167,"duration_ms":70291,"significance":"If the results hold, the paper makes a useful contribution by reducing reliance on human-annotated instruction datasets and by turning failed trajectories into instruction-following data. The idea of using LLMs to reinterpret trajectories retrospectively is timely, the algorithm is clearly presented, and evaluation on a public benchmark with external ground-truth success signals is a strength. However, the evidence as presented is not yet convincing: the most directly relevant baseline, HIR, is omitted; the cosine-similarity training reward is never validated against true task completion; and several numerical claims in the text contradict the figures. The contribution is potentially significant, but the current manuscript requires additional verification before the claims can be accepted.","major_comments":[{"comment":"The paper does not compare against Hindsight Instruction Relabeling (HIR) [Zhang et al., 2023], which is the most relevant prior method and is cited in Section 2. Because OIR is explicitly a hindsight instruction relabeling method, the claimed advantage over state-of-the-art methods cannot be assessed without an HIR baseline, for example HIR using template-based or non-LLM relabeling. Please add such a comparison or temper the claim.","section":"§5, Baselines"},{"comment":"The training reward is defined entirely by the cosine similarity between transition and instruction embeddings, and Section 5 states that all environment rewards and achievement signals are removed during training. The paper never measures the correlation between this proxy and ground-truth completion on the same trajectories. If high cosine similarity can occur without task completion (e.g., proximity to trees without an inventory change), the dense reward may reward spurious behavior, and the reported gains over PQN and ELLM could be a reward-density artifact. Please validate the proxy, for example by reporting precision and recall of the threshold rule against ground-truth success on relabeled trajectories, or include an oracle-relabeling ablation that uses true success signals.","section":"§4.2, Eq. (8)"},{"comment":"The numerical claims in the text are not consistent with the figure. The text reports 'Defeat Zombie' (87.78%), 'Place Table' (95.83%), and 'Make Wood Pickaxe' (76.59%), but Figure 2(d) displays values 3.40, 29.93, and 1.82 for these tasks. Likewise, the text reports approximately 10 completed instructions and an average success rate of 33.10%, while Figure 2(b) and (c) show final values of 7.28 and 18.20. These discrepancies must be reconciled before the empirical claims can be evaluated.","section":"§5.1, Figure 2(d)"},{"comment":"Section 4.1 states that rule-based instructions are incorporated as an additional source of supervision and that candidate instructions are subject to further filtering or selection, but no rule-based instructions, filtering criteria, or selection step appear in Algorithm 1 or in the implementation details. Moreover, the prompt template in Section 4.1 ('What instruction is this trajectory following?') differs from the prompt in Appendix B.5, which asks for 'Completed Instructions' and prohibits movement instructions. The method as described is therefore not fully specified; please align the description and the implementation.","section":"§4.1, Algorithm 1"},{"comment":"Several hyperparameters required by Algorithm 1 are missing from Table 1, including K (number of LLM candidates), the buffer thresholds τ_low and τ_high, and the exact δ values used for the PQN baselines (Table 1 lists '0.5&0.9' for PQN, while Figures 2 and 3 report separate PQN (δ=0.5) and PQN (δ=0.9) curves). In addition, ELLM uses 1024 parallel environments while OIR and PQN use 64; because total timesteps are matched, this may be acceptable, but the potential effect of parallelism on exploration and LLM query batching should be justified. Please provide complete hyperparameters and discuss the fairness of the ELLM setup.","section":"Table 1, Appendix B.2"}],"minor_comments":[{"comment":"The equation writes fstate(ot) but the expression uses fstate(ot, at, ot+1); please fix the notation.","section":"§4.2, Eq. (8)"},{"comment":"The sentence 'an episode is deemed successful the first time the reward exceeds a predefined threshold δ' should specify that the reward is computed on transitions and that termination is applied only once per episode.","section":"§4.2"},{"comment":"The reward rcand is computed from the full trajectory τe and instruction i′, but Eq. (8) requires a single transition; please clarify how the trajectory-level reward is aggregated over timesteps.","section":"Algorithm 1, line 9"},{"comment":"The claim that the method 'completely eliminates' dependence on manual annotations is too strong, since the prompt template, instruction-generation principles, and hyperparameters are manually designed.","section":"Abstract and Introduction"},{"comment":"The diversity claim is based on a visual t-SNE plot; please include a quantitative coverage metric, such as the number of distinct semantic clusters or an embedding-volume estimate.","section":"§5.3, Figure 5"},{"comment":"Minor typos include 'acessible' in Section 4, 'extual' in the Figure 1 caption, and '1S-2046640' in Section 7; please proofread the manuscript.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The numerical inconsistencies in Section 5.1 are serious enough that a simple textual revision may not suffice; the authors should re-run or carefully re-report the experiments. I would also encourage the editor to insist on the HIR baseline and the proxy-validation experiment, as those are central to the paper's claim of being an advance over existing hindsight relabeling methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: OIR (open-ended instruction relabeling) is a real, useful extension of HER/HIR. Instead of replaying template goals, it asks an LLM to read a trajectory and propose instructions the agent plausibly accomplished, then trains a single instruction-conditioned policy with dense cosine-similarity rewards and an eviction-based instruction buffer. The combination is new relative to HIR (template-based) and ELLM (forward goal generation), and the Craftax results—better sample efficiency, more completed instructions, and generalization to simple/complex variants than the PQN variants and ELLM—are consistent with the central claim. This paper deserves a serious referee.\n\nWhat it does well: the idea is clearly motivated and simple enough to build on; the eviction buffer is a sensible way to keep the growing instruction set bounded; and the evaluation is decently designed, with ground-truth success used at test time and ablations over the threshold.\n\nThe soft spots are real but mostly addressable. First, the training reward (Eq. 8) is a cosine-similarity proxy, and the paper never validates that it actually tracks task completion. Since all environment rewards are removed during training, this proxy is the only learning signal. The stress-test concern about reward hacking is not speculative. The threshold ablation shows sensitivity, but it does not compare predicted success against true success on the same trajectories. This needs a direct correlation analysis or a version with ground-truth relabeling. Second, the most relevant baseline, HIR, is cited but not compared. That weakens the empirical novelty claim. Third, the method says rule-based instructions are added as supervision, but no rule-based instructions or filtering steps are described. Finally, the reported average success rate (33.10% in Section 5.1) does not match Figure 2(c) (18.20), which is the kind of inconsistency a referee will catch immediately.\n\nNone of these sinks the paper. The core mechanism is plausible and the experiments support the qualitative story. I'd send it to review with a request for proxy validation, the HIR baseline, and a cleanup pass. It will be useful for anyone working on LLM-guided RL or instruction-conditioned policies.","headline":"A useful new combination that repurposes failed trajectories with LLM-generated instructions, but the unvalidated reward proxy and missing HIR baseline keep it from being fully convincing.","tokens_in":13180,"tokens_out":2371,"would_cite":true,"duration_ms":26266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that reads failed trajectories and relabels them with accomplished subtasks turns sparse failures into training signal, and the resulting policy beats baselines on Craftax and generalizes to unseen instruction variants.","keywords":["instruction-following reinforcement learning","hindsight instruction relabeling","large language models","sparse reward","open-ended environments","Craftax","semantic reward","prioritized replay buffer"],"falsifier":"On a held-out set of rollouts, compare the success labels produced by the training reward (cosine similarity above $\\delta$, Eq. 8) against the environment's own ground-truth achievement flags for the relabeled instructions: if a large fraction of cosine-declared successes are not real completions, the reward is rewarding spurious behavior and OIR's measured gains are not evidence of genuine instruction following. A complementary check is to relabel trajectories with deliberately irrelevant instructions while keeping the same cosine-similarity machinery, and to see whether the gap over baselines persists; if it does, the reward proxy, not the LLM's semantic relabeling, is carrying the result.","tokens_in":12209,"feed_emoji":"🤖","tokens_out":15182,"duration_ms":142593,"temperature":0.7,"pith_summary":"This paper argues that a reinforcement-learning agent's failed rollouts are not wasted data: a large language model can read a text rendering of any collected trajectory, including ones that missed their original goal, and generate open-ended instructions for subtasks the agent did complete. These relabeled instructions supply reward signals through embedding-based cosine similarity (Eq. 8), converting sparse, unsuccessful episodes into dense training batches for a single instruction-following policy. On the Craftax benchmark, with all built-in rewards removed, the proposed method, open-ended instruction relabeling (OIR), outperforms parallelised Q-network (PQN) variants and the LLM-exploration baseline ELLM in sample efficiency, number of instructions completed, success rate, and generalization to simple and complex linguistic variants of the training instructions. If the claim holds, LLM-guided hindsight relabeling removes the human-annotation bottleneck in instruction-following RL and turns exploration waste into curriculum.","feed_headline":"Mining failed runs with an LLM beats standard RL baselines","feed_subtitle":"A hindsight relabeler turns sparse failed trajectories into dense instruction-following signals, with no human annotation.","key_machinery":"The central mechanism is LLM-guided hindsight instruction relabeling: each rollout is converted by a function $h$ into a temporally structured text prompt, from which a pretrained LLM samples $K$ candidate instructions for subtasks the agent fully accomplished (Eq. 4). Each candidate is scored by the embedding-based reward $r_t = \\mathrm{cosim}\\big(f_{state}(o_t,a_t,o_{t+1}),\\, f_{instr}(i)\\big)$ (Eq. 8), where $f_{state}$ embeds the transition and $f_{instr}$ is a pretrained sentence-embedding encoder such as SBERT, and an episode is declared successful and terminated the first time the reward exceeds threshold $\\delta$. The acceptance criterion is formalized as the oracle value comparison of Eq. (7), $V^{\\pi_{oracle}}_{i'}(s_{t+1}) > V^{\\pi_{oracle}}_{i_{orig}}(s_{t+1})$, and the prioritized instruction replay buffer (Section 4.3), inspired by Prioritized Level Replay, sorts instructions into failing, learning-boundary, and mastered categories, evicting by priority so that the uniform sampling of instructions at reset concentrates training on tasks near the agent's competence frontier.","core_discovery":"On its own terms, the paper's discovery is that hindsight can operate at the level of natural-language instructions rather than states or goals. Given a trajectory rendered as a temporally ordered text prompt, the LLM returns candidate instructions describing what the agent accomplished, including in episodes that failed their originally assigned instruction, and each candidate is scored by the cosine similarity between the transition embedding and the instruction embedding (Eq. 8), with a threshold $\\delta$ marking success and episode termination. The acceptance ideal is that a relabeled instruction yields higher oracle value than the original instruction (Eq. 7), and a prioritized instruction buffer, modeled on Prioritized Level Replay, keeps the policy training on tasks at its learning frontier. Empirically, OIR completes roughly ten of the Craftax instructions versus fewer than four for other baselines, reaches an average success rate of 33.10%, masters hard multi-step tasks such as 'Defeat Zombie' (87.78%), 'Place Table' (95.83%), and 'Make Wood Pickaxe' (76.59%), and retains its edge on unseen simple and complex instruction variants; the paper also records a trade-off in which infrequently sampled instructions such as 'Wake Up' lose proficiency to the ground-truth-reward baseline.","pith_inferences":["The paper does not measure how often a cosine-declared success (reward above $\\delta$ in Eq. 8) matches a ground-truth accomplishment, so a fair share of the relabeled signal may reward trajectories that merely resemble the instruction in embedding space; one testable consequence is that OIR's advantage should shrink in environments where textual observations are less informative about actual stat","There is a closed loop worth probing: the instruction encoder that scores the reward is also the encoder conditioning the policy, and the LLM generates instructions from the same textual observations that get embedded; this coupling could make measured instruction diversity (the t-SNE coverage) grow faster than genuine behavioral diversity.","Because the discoverable instruction space is bounded by what the LLM can recognize from text, OIR's ceiling is set by the model's world knowledge rather than by the environment; a human-in-the-loop verification of relabeled instructions, which the paper itself lists as future work, would directly test how much of the relabeled signal is true task completion."],"forward_implications":["Failed and sparse trajectories become reusable training data: without any human annotation, a single policy learns to follow many open-ended instructions.","The policy generalizes to unseen instruction variants, both simple rewrites and semantically enriched compositions, because the relabeled instruction space is wider than the environment's predefined achievements.","Hard multi-step tasks that reward-free baselines never master ('Defeat Zombie', 'Make Wood Pickaxe', 'Place Table') become learnable, since hindsight relabeling supplies intermediate semantic goals.","Sample efficiency improves on every reported metric: higher aggregate score, more completed instructions, and higher average success rate at the same environment-step budget.","Because OIR needs only a textual rendering of observations rather than environment-specific reward logic, the same recipe applies in principle to any environment that can describe its states in text."],"supporting_citations":[{"why":"Hindsight Experience Replay is the relabeling foundation that OIR extends from state-based goals to open-ended natural-language instructions.","marker":"[Andrychowicz et al., 2017]"},{"why":"Craftax is the sparse-reward benchmark on which every experiment and the ground-truth evaluation signals are defined.","marker":"[Matthews et al., 2024]"},{"why":"PQN supplies the parallel Q-network backbone used by all methods, and its citation is attached to the cosine-similarity reward definition of Eq. 8.","marker":"[Gallici et al., 2024]"},{"why":"ELLM is the LLM-guided exploration baseline that OIR is compared against and must beat.","marker":"[Du et al., 2023]"},{"why":"Sentence-BERT provides the pretrained instruction encoder whose embeddings are compared in the reward of Eq. 8.","marker":"[Reimers and Gurevych, 2019]"},{"why":"Prioritized Level Replay is the eviction logic behind the prioritized instruction buffer that keeps training focused on learning-boundary tasks.","marker":"[Jiang et al., 2021]"},{"why":"Prior hindsight instruction relabeling (HIR) is the template-based approach that OIR contrasts with and generalizes.","marker":"[Zhang et al., 2023]"}],"fun_headline_variants":["LLM hindsight turns failed runs into new instruction data","Failed runs become new tasks via LLM instruction relabeling","Hindsight relabeling with LLMs boosts instruction-following RL","From failures to instructions: LLM relabeling beats baselines","LLM relabeling of failed runs lifts RL instruction coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the reward proxy of Eq. 8: an episode is treated as successfully completing an LLM-generated instruction when the cosine similarity between the trajectory's transition embeddings and the instruction's embedding exceeds threshold $\\delta$, and the paper never verifies that this proxy corresponds to the agent actually doing what the instruction describes.","fun_headline_variants_meta":{"raw":{"variants":["LLM hindsight turns failed runs into new instruction data","Failed runs become new tasks via LLM instruction relabeling","Hindsight relabeling with LLMs boosts instruction-following RL","From failures to instructions: LLM relabeling beats baselines","LLM relabeling of failed runs lifts RL instruction coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1437,"prompt_tokens":972,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":588,"tokens_out":465,"duration_ms":4778,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:57:06.812292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of rollouts, compare the success labels produced by the training reward (cosine similarity above $\\delta$, Eq. 8) against the environment's own ground-truth achievement flags for the relabeled instructions: if a large fraction of cosine-declared successes are not real completions, the reward is rewarding spurious behavior and OIR's measured gains are not evidence of genuine instruction following. A complementary check is to relabel trajectories with deliberately irrelevant instructions while keeping the same cosine-similarity machinery, and to see whether the gap over baselines persists; if it does, the reward proxy, not the LLM's semantic relabeling, is carrying the result.","supporting_citations":[{"cited_title":"Prioritized level replay","cited_arxiv_id":null,"evidence_quote":"Prioritized Level Replay is the eviction logic behind the prioritized instruction buffer that keeps training focused on learning-boundary tasks."}],"review_version":1}