{"id":"8f54c993-a4ef-4a23-98fc-9909c109ce97","arxiv_id":"2507.06628","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GO-Skill learns a discrete library of goal-oriented skills from offline multi-task data and selects them with a hierarchical policy, improving average episode returns on MetaWorld MT30 and MT50.","lead":"An offline multi-task reinforcement learning method called GO-Skill extracts reusable skills from pre-collected robot data, then uses a high-level policy to combine them. It reports higher average returns than several decision-transformer and diffusion baselines on the MetaWorld robotic manipulation benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward-free state-difference representation of Eq. 4 conflates behaviors that share the same final state delta but require different actions, so the skill decoder may average incompatible action distributions and the claimed transfer advantage is not yet demonstrated.","rationale":"I read the paper as an empirical claim that reward-free state-difference skills improve offline multi-task RL on MetaWorld. For that claim to hold, the learned skill abstraction must preserve task-relevant behavioral distinctions, not just endpoint state differences. The reader's weakest assumption identifies exactly this point, and the paper's own design choices make it acute: the goal encoder uses only s_{t+H} - s_t, the skill decoder is trained by MSE action reconstruction, and the dataset contains heterogeneous-quality trajectories. The concern is not an external disagreement with the field; it is internal to the method's representation. It is also testable, which is why a conditional verdict is appropriate rather than an accept or reject. I considered two other potential concerns. First, hyperparameters are selected by sweeps on MT50 Near-Optimal, the same setup used for the headline numbers; however, the paper shows that a wide range of hyperparameter values still outperforms baselines in Figure 6, so this is less decisive. Second, the claim that more tasks help is made by comparing MT30 and MT50, which contain different task sets; this is a real methodological weakness, but it affects a supporting narrative rather than the core per-setup superiority result. The state-difference ambiguity, by contrast, attacks the mechanism that is supposed to produce the superiority. The proposed check would settle it: if high-variance, mixed-return clusters exist inside single codes, the skill library is demonstrably conflating distinct behaviors, and a simple representation change would reveal whether that conflation is responsible for the performance gap. Until that check is run, the central claim is plausible but not fully secured, which supports keeping the reader's conditional verdict.","tokens_in":16603,"tokens_out":15483,"duration_ms":182196,"concrete_test":"Using the frozen MT50 Near-Optimal skill model, collect all H-step segments assigned to each codebook index. After binning by initial state s_t, compute (a) the variance of the first action a_t within each (code, state) bin and (b) the variance of per-segment returns within each bin. If bins contain high action variance with mixed returns, train a variant whose goal encoder input is [s_t, s_{t+H}] instead of the difference alone (or a variant that adds a reward-conditioned term) and re-evaluate MT50 Near-Optimal; if the alternative's average return rises by more than the reported GO-Skill margin over the best baseline, the state-difference abstraction is the load-bearing weakness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core premise of GO-Skill is that s_{t+H} - s_t, after vector quantization, is a sufficient task-agnostic description of a reusable skill. The paper deliberately excludes reward signals and action sequences from this representation (Section 3.1). But the offline dataset (Appendix A) deliberately mixes random, suboptimal, and expert SAC-Replay transitions. Two H-step segments can therefore have the same final state difference while requiring very different actions and having very different returns: for example, pushing an object to a position versus picking it up and placing it at the same position, or making contact versus avoiding contact while producing the same endpoint displacement. Because the skill decoder is trained with an MSE action-reconstruction loss (Algorithm 1, L_MSE) and a single code is assigned to each state-delta cluster, the decoder must fit one action distribution to all such conflated segments. The optimal fit is an average that matches neither behavior. The reached-goal history of Eq. 6 only partially disambiguates the trajectory, and at the first step of a skill the history is empty, so the ambiguity is sharpest exactly when the skill must begin. This misspecification is plausible and would directly hurt the transfer and versatility claims, and it is consistent with the large per-task failures in Table 6 (e.g., coffee-push-v2 at 569±119 near-optimal and peg-unplug-side-v2 at 170±44 suboptimal). The central claim would still hold if the learned codebook separates task-relevant behaviors or if the decoder can disambiguate them from state; the paper does not provide evidence for either.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GO-Skill, a hierarchical offline multi-task reinforcement learning method. A skill model encodes H-step sub-trajectories through the state difference s_{t+H} - s_t, maps them to a latent goal embedding, and vector-quantizes these embeddings into a discrete skill codebook. A Transformer-based skill decoder reconstructs actions conditioned on the skill embedding, the state history, and the reached-goal history, and a skill enhancement phase resamples skill classes to address class imbalance. A high-level skill-based Decision Transformer, trained with focal loss, selects skill indices at intervals of H steps. The method is evaluated on MetaWorld MT30/MT50 with near-optimal and sub-optimal offline datasets, compared against six baselines, and ablated on its components and hyperparameters; a fine-tuning experiment on ML45 is also reported.","tokens_in":16896,"tokens_out":7460,"duration_ms":85771,"significance":"If the performance claims hold, the paper makes a useful contribution by showing that state-difference-based skill abstraction with vector quantization can serve as an effective hierarchical inductive bias for offline multi-task RL, with strong aggregate results on MT30/MT50. The evaluation is broad and includes five seeds, component ablations, hyperparameter sweeps, skill visualizations, and per-task results for the proposed method. The method is empirical rather than a theoretical derivation, so the main claims are falsifiable benchmark comparisons rather than parameter-free predictions. The paper also provides a clear and actionable design: reward-free goal encoding, discrete codebook, class-imbalance mitigation, and a Prompt-DT-style high-level policy.","major_comments":[{"comment":"The headline claim that GO-Skill achieves superior performance compared to existing state-of-the-art approaches is not supported by any significance testing. The reported error bars overlap for some of the comparisons, particularly in the MT30 Sub-Optimal setting, and only five seeds are used; Table 6 shows very large per-task standard deviations (for example, assembly-v2 at 2366.7±1082.3 and pick-place-v2 at 358.6±409.1 in the Near-Optimal setting). The aggregate mean differences may therefore not be statistically reliable. Please add paired significance tests across tasks (for example, a Wilcoxon signed-rank test of GO-Skill against each baseline per task) or bootstrap confidence intervals over tasks and seeds, and provide per-task results for all baselines rather than only for GO-Skill.","section":"Section 5.2, Figure 5"},{"comment":"The skill representation is a potential misspecification risk that is load-bearing for the claimed transfer mechanism. The goal encoder maps the H-horizon sub-trajectory to only s_{t+H} - s_t, and at the first step of a skill the decoder conditions on the skill code, the current state, and an empty reached-goal and action history. Consequently, two H-step segments with the same start state and the same endpoint difference but different intermediate states or actions receive the same skill code, and the MSE action-reconstruction objective in Algorithm 1 must fit a single action distribution to potentially incompatible behaviors. The paper's motivation that different action sequences can lead to the same transfer outcome is only one side of this conflation. The AE ablation in Table 1, row 4, provides a relevant check, but it is confounded because that row also removes reached-goal history. Please add a controlled comparison in which both the goal-based and action-based encoders are evaluated with reached-goal history, and report codebook purity or per-skill action-reconstruction error to demonstrate that the learned skill codes are behaviorally coherent.","section":"Section 3.1, Equations (4) and (7)"},{"comment":"The hyperparameters H, M, |Z|, and t_e are selected via ablations on the same MT50 Near-Optimal setup that is subsequently used for the main results, meaning the reported margins over baselines include selection on the evaluation tasks. This is a particular concern because the differences over the best baselines in Figure 5 are not large relative to the seed noise. Please either fix the hyperparameters before evaluating on the final task sets, report results on a held-out validation split, or show that the relative ranking of GO-Skill against the baselines is stable across the swept hyperparameter range.","section":"Section 5.3, Figure 6"}],"minor_comments":[{"comment":"The checkmark and cross symbols in the ablation tables are not defined in the captions; please add an explicit legend.","section":"Tables 1 and 2"},{"comment":"The symbol T' is used in the focal loss computation before it is defined; clarify that it denotes the decision point at which the skill index is predicted.","section":"Algorithm 2"},{"comment":"Because the error bars overlap and the plotted values are difficult to read, please also provide a numeric table of mean and standard deviation for every method and setting.","section":"Figure 5"},{"comment":"The multi-head variant of GO-Skill is described only briefly; please specify the number of heads, where they are inserted in the skill model, and how they are trained and combined.","section":"Section 5.2, MH-GO-Skill"},{"comment":"The fine-tuning result is presented as single learning curves without error bars or per-task breakdowns; please report mean and standard deviation over seeds and a table for the five new tasks.","section":"Section 5.5, Figure 8"},{"comment":"The Limitations paragraph identifies the reliance on a predefined skill horizon and skill set size, but it does not discuss the sufficiency or ambiguity of the state-difference representation in Section 3.1; acknowledging this would help readers calibrate the scope of the claims.","section":"Conclusion and Discussion"},{"comment":"The paper does not include a code availability statement; releasing the implementation and random seeds would substantially aid reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution and I see no grounds for rejection. The revision should focus on three issues: statistical support for the headline comparison, a controlled analysis of whether the state-difference skill codes are behaviorally coherent, and clarity about hyperparameter selection on the evaluation benchmark. If the authors can add paired significance tests and the controlled representation analysis, the paper would be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GO-Skill is a genuinely useful empirical contribution to offline multi-task RL. What's new is the specific assembly: a VQ codebook of skills defined by state-differences over a fixed horizon, a two-phase extraction/enhancement scheme to handle skill-class imbalance, and a Prompt-DT high-level policy that selects skill indices. The reached-goal history in the decoder is a nice touch. The experiments are broad and mostly well done—MT30/MT50 with Near/Sub-Optimal data, five seeds, component ablations, hyperparameter sweeps, and an ML45 fine-tuning study. The main claim, that GO-Skill improves average return over strong baselines, is credible, and the improvement growing with task count is a point in its favor.\n\nSoft spots: no code released, so the numbers aren't independently checkable; hyperparameters are selected on the same MT50 benchmark used for the headline result; and there's no comparison with prior skill-based offline RL methods (SPiRL-style or skill priors), which would better locate the novelty. The per-task results in Table 6 show a few large failures (coffee-push, pick-place, peg-unplug-side under sub-optimal), which tempers the versatility claim.\n\nOn the representation: the stress-test concern that s_{t+H}-s_t conflates trajectories with the same endpoint displacement is real, but it's not fatal. The decoder sees the reached-goal history and current states, so only the first step is genuinely ambiguous, and the empirical average suggests the codebook still finds useful clusters. The ablations support this: reached-goal history helps, and goal-based skills beat action-based skills. Still, the paper should explicitly acknowledge this ambiguity; it doesn't.\n\nThe transfer claim is not undemonstrated—the fine-tuning curves in Figure 8 show faster adaptation—so the stress-test overstates that point.\n\nBottom line: this is a solid, citable advance for people working on offline MTRL or skill-based RL. It deserves a serious referee; I'd send it out. My own verdict would be conditional accept, pending code release and a transparent statement about hyperparameter selection.","headline":"A solid empirical skill-abstraction method with a real ambiguity in the state-difference representation and some evaluation gaps, but worth a serious referee.","tokens_in":17492,"tokens_out":3505,"would_cite":true,"duration_ms":37149,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GO-Skill claims that distilling offline trajectories into a discrete library of goal-oriented skills and selecting among them with a hierarchical policy improves average episode return on MetaWorld relative to action-level…","keywords":["offline reinforcement learning","multi-task reinforcement learning","skill abstraction","goal-oriented representation","vector quantization","hierarchical policy","robotic manipulation","MetaWorld"],"falsifier":"Train GO-Skill on a pair of MetaWorld tasks where the same state displacement corresponds to different required contacts (for example, a task that needs a grasp and a task that needs a push from the same start and end positions), and check whether the learned codebook assigns the same skill index to both sub-trajectories and whether evaluation return drops relative to a model that includes object identity in the goal. If performance does not degrade, the state-difference assumption is sufficient for that setting; if it does, the claim that state differences are task-agnostic skills is falsified.","tokens_in":16391,"feed_emoji":"🤖","tokens_out":5207,"duration_ms":49999,"temperature":0.7,"pith_summary":"GO-Skill is an offline multi-task reinforcement learning method that claims to improve average episode return on the MetaWorld robotic manipulation benchmark by first distilling a discrete library of reusable skills from task-mixed offline data, then learning a high-level policy that selects among those skills. The paper argues that representing a skill as the state difference $s_{t+H} - s_t$ over a fixed horizon $H$ gives a task-agnostic goal description, and that vector quantization turns these goal embeddings into a discrete skill codebook. A skill-enhancement phase resamples under-represented skills, and a Prompt-DT-style policy with focal loss orchestrates skill selection. On MT30 and MT50 under both Near-Optimal and Sub-Optimal datasets, GO-Skill reports higher average episode return than six baselines, with the advantage growing as the number of tasks increases. The central claim is that explicit goal-oriented skill abstraction is an effective knowledge-sharing mechanism for offline multi-task RL.","feed_headline":"GO-Skill outruns six offline multi-task RL baselines","feed_subtitle":"State-difference skills plus a discrete codebook raise episode return on MetaWorld and scale with task count.","key_machinery":"The load-bearing object is the goal-oriented skill, defined as a fixed-horizon state transition $s_{t+H} - s_t$ encoded by a goal encoder $G$ into an embedding $z_{t,H} = G(s_{t+H} - s_t)$, then quantized against a codebook $\\mathcal{Z} = \\{\\hat{z}^{(1)}, \\ldots, \\hat{z}^{(M)}\\}$ via vector quantization. This codebook is the discrete skill library; a skill decoder $P$ reconstructs actions from the skill embedding plus the reached-goal history, and a skill-based decision transformer selects skill indices at $H$-step intervals. The goal representation excludes rewards and action sequences, which the paper argues makes skills transferable across tasks whose reward functions differ despite identical trajectories.","core_discovery":"The paper's central discovery is that goal-oriented skill abstraction—encoding an $H$-horizon behavior by the change in state it produces rather than by its action sequence—lets an offline agent extract reusable, task-agnostic skills from mixed-quality multi-task data, and that a hierarchical policy acting in the resulting discrete skill space solves more tasks better than action-level sequence models. The authors show that a Transformer-based skill decoder conditioned on a quantized goal embedding can reproduce the actions of an $H$-step sub-trajectory, and that a high-level skill-based decision transformer can learn to chain these skills. On MetaWorld MT30 and MT50, GO-Skill achieves the highest average episode return among the compared offline multi-task methods in both Near-Optimal and Sub-Optimal data settings, and its margin over baselines widens from 30 to 50 tasks. The evidence used to support this includes ablations showing that removing the reached-goal history, replacing discrete vector quantization with a continuous skill space, or replacing goal-based encoding with action-based encoding each lowers performance.","pith_inferences":["Beyond the paper, the state-difference goal representation could be tested against goal representations that include object identity or contact features; the paper itself notes that more robust goal-oriented representations remain an open direction.","A testable extension is whether the same learned skill codebook transfers across different benchmarks or embodiments without fine-tuning, since the paper only demonstrates re-use through fine-tuning within MetaWorld.","Because the paper identifies the predefined skill set size and skill horizon as limitations, an adaptive or growing codebook is a natural follow-up that could reduce sensitivity to these two hyper-parameters."],"forward_implications":["GO-Skill reports higher average episode return than six offline multi-task baselines on MetaWorld MT30 and MT50 under both Near-Optimal and Sub-Optimal data, and the advantage grows as the task count increases.","The method extracts useful skill fragments from sub-optimal trajectories, so lower-quality offline data still yields transferable skills for other tasks.","Ablations show that each component contributes: reached-goal history, discrete vector quantization, goal-based encoding, resampling during skill enhancement, and focal loss during policy learning all matter for the reported performance.","In the ML45 fine-tuning setup, GO-Skill reaches higher episode return after only 3,000 fine-tuning iterations on five new tasks than MTDT and PromptDT.","Because the skill-based policy makes decisions at $H$-step intervals instead of every time step, the decision horizon shortens, which the paper argues simplifies policy learning and improves sample efficiency."],"supporting_citations":[{"why":"Supplies the vector quantization module that produces the discrete skill codebook from goal embeddings.","marker":"Van Den Oord et al., 2017"},{"why":"Prompt-DT is the sequence-modeling backbone on which GO-Skill's skill-based policy and prompt mechanism are built.","marker":"Xu et al., 2022"},{"why":"Decision Transformer frames offline RL as sequence modeling and serves as the MTDT baseline.","marker":"Chen et al., 2021"},{"why":"MTDIFF-P is a diffusion-based baseline and the source of the SAC-Replay dataset construction protocol used for Near-Optimal and Sub-Optimal data.","marker":"He et al., 2023"},{"why":"HarmoDT is a parameter-masking baseline that GO-Skill compares against, representing the task-specific-network-sharing line of work.","marker":"Hu et al., 2024"},{"why":"MetaWorld supplies the 50-task robotic manipulation benchmark and the MT30, MT50, and ML45 evaluation setups.","marker":"Yu et al., 2020b"},{"why":"Focal loss is used in skill-based policy training to counter the imbalance between broadly applicable and task-specific skills.","marker":"Lin et al., 2017"},{"why":"Soft Actor-Critic replay buffers generate the offline datasets whose transitions are segmented into skills.","marker":"Haarnoja et al., 2018"}],"fun_headline_variants":["GO-Skill: goal states unlock reusable offline skills","Offline RL skills from state goals beat sequence baselines","Hierarchical skill library tops offline multi-task on MetaWorld","Goal-driven skill abstraction wins offline MT50 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the state difference over a fixed horizon is a sufficient, task-agnostic description of a reusable skill; if two behaviors with the same state difference require different actions or different task context, the skill codebook and the policy built on it can be misspecified.","fun_headline_variants_meta":{"raw":{"variants":["GO-Skill: goal states unlock reusable offline skills","Offline RL skills from state goals beat sequence baselines","Hierarchical skill library tops offline multi-task on MetaWorld","Goal-driven skill abstraction wins offline MT50 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1274,"prompt_tokens":923,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":287}},"tokens_in":539,"tokens_out":351,"duration_ms":4674,"temperature":1.0,"reasoning_tokens":287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:58:25.892423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train GO-Skill on a pair of MetaWorld tasks where the same state displacement corresponds to different required contacts (for example, a task that needs a grasp and a task that needs a push from the same start and end positions), and check whether the learned codebook assigns the same skill index to both sub-trajectories and whether evaluation return drops relative to a model that includes object identity in the goal. If performance does not degrade, the state-difference assumption is sufficient for that setting; if it does, the claim that state differences are task-agnostic skills is falsified.","supporting_citations":[{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the vector quantization module that produces the discrete skill codebook from goal embeddings."},{"cited_title":"Decision transformer: Reinforcement learning via sequence modeling","cited_arxiv_id":null,"evidence_quote":"Decision Transformer frames offline RL as sequence modeling and serves as the MTDT baseline."},{"cited_title":"Focal loss for dense object detection","cited_arxiv_id":null,"evidence_quote":"Focal loss is used in skill-based policy training to counter the imbalance between broadly applicable and task-specific skills."},{"cited_title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","cited_arxiv_id":null,"evidence_quote":"Soft Actor-Critic replay buffers generate the offline datasets whose transitions are segmented into skills."}],"review_version":1}