{"id":"d750a0ec-8bcb-409c-be01-f5c8b439c9e4","arxiv_id":"2504.14805","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DCSL learns skills from state transitions and contrastively clusters similar behaviors, then relabels each skill with a dynamically chosen length.","lead":"This paper introduces a new way for robots to learn reusable skills from unlabeled logged data by grouping similar behaviors even when the exact movements differ. It also lets each skill last a different number of steps, which helps in noisy datasets where fixed-length skills fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Theorem 1 proof equating binary NCE with conditional mutual information is algebraically wrong, leaving the semantic-similarity mechanism unsubstantiated.","rationale":"The paper's central claim has two legs: empirical downstream performance and the proposed mechanism of semantic similarity plus dynamic length adjustment. The experiments and ablations provide real, if incomplete, evidence for the first leg, so I would not reject the paper on theory alone. However, the stated mechanism is exactly what differentiates DCSL from fixed-length baselines, and it fails at the point where it is most load-bearing: the proof that contrastive learning yields semantic skill clustering is algebraically incorrect, the negative sampling distribution is ill-defined and does not match the algorithm, and Eq. (8) disagrees with Algorithm 2 on how skill lengths are relabeled. These issues mean the abstract's claim that DCSL 'effectively captures the semantic context of behaviors' is not currently substantiated, even though the method may still work for reasons other than the stated mechanism. The reader's CONDITIONAL verdict already flags missing baseline tables, informal theory, and absent code; this concern sharpens the condition: correct or remove the A.1.1 derivation, specify and justify the negative sampling distribution, and reconcile Eq. (8) with Algorithm 2. Since the reader's conditional verdict already anticipates that such gaps must be resolved, my read does not change the verdict.","tokens_in":18476,"tokens_out":5080,"duration_ms":50239,"concrete_test":"Re-derive the algebra in Appendix A.1.1 and run a controlled synthetic experiment with known skill boundaries: train only the contrastive module, then measure precision/recall of the relabeled segments produced by Algorithm 2 against ground-truth behavior segments under threshold ε=0. If the corrected loss is not -I(Z;S'|S) or the recovered boundaries do not match the known segments, the semantic-clustering claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.1.1 claims that with the optimal discriminator σ(f)=p(s'|s,z)/(p(s'|s,z)+p(s')), the contrastive loss equals -I(Z;S'|S). Substituting this σ into the stated binary NCE loss yields E[log(1+p(s')/p(s'|s,z)) + log(1+p(s'|s,z)/p(s'))], not E[log(p(s'|s,z)/p(s'))]; the denominator terms do not cancel. Binary NCE with a single negative is a logistic discrimination objective, not an exact mutual information estimator, so the theoretical grounding for fθf as a semantic skill-similarity measure is invalid as written. The operational definition is also inconsistent: Eq. (4) defines negatives as states with z'≠z, but z is a continuous latent code, so almost every pair satisfies z'≠z, while Algorithm 1 samples negatives from a different trajectory, which is a different distribution. Consequently the learned similarity function and the Eq. (8) relabeling are not shown to align with semantic behavior boundaries; the central mechanism rests on an untested assumption. A further internal discrepancy exists: Eq. (8) defines H'_t via max α with f>ε, while Algorithm 2 stops at the first α with f≤ε, so the specification and the implemented procedure disagree.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DCSL, an offline skill-learning framework that represents skills through four key states rather than full action sequences, learns a contrastive skill-similarity function f_theta_f, and dynamically relabels skill lengths by thresholding that similarity function on future states. The method is evaluated on AntMaze, Kitchen, Pick-and-Place, and Peg-Insert-Side datasets, with downstream learning performed via SAC (model-free) and CEM (model-based). The central claims are that state-transition-based skills cluster semantically similar behaviors and that dynamic length adjustment improves success and efficiency, especially in noisy datasets.","tokens_in":18778,"tokens_out":3654,"duration_ms":35495,"significance":"If validated, the state-transition skill representation and length relabeling would be a useful contribution to offline long-horizon RL, since existing fixed-length action-sequence methods are known to be brittle to action variation and noisy data. The paper also includes useful ablations on the relabeling threshold, the number of key states, and an additional environment (Peg-Insert-Side). However, the current manuscript has serious gaps in its theoretical justification, algorithmic consistency, and experimental reporting, so the central claims are not yet established.","major_comments":[{"comment":"Theorem 1 is algebraically incorrect as stated. Substituting the claimed optimal discriminator sigma(f)=p(s'|s,z)/(p(s'|s,z)+p(s')) into the binary NCE loss does not yield -E[log(p(s'|s,z)/p(s'))] = -I(Z;S'|S); instead the two log terms produce log(1+p(s')/p(s'|s,z)) + log(1+p(s'|s,z)/p(s')), whose expectation is not the conditional mutual information. Binary NCE with a single negative is a logistic discrimination objective, not an exact mutual-information estimator. This invalidates the theoretical grounding for f_theta_f as a semantic skill-similarity measure, and Theorem 2 is similarly asserted without a derivation. The paper should either provide a correct derivation, or explicitly reframe the loss as a heuristic discrimination objective and remove the claim of an information-theoretic foundation.","section":"Appendix A.1.1, Eq. (9)"},{"comment":"There is a direct inconsistency between the definition and the implementation of skill length relabeling. Eq. (8) defines H'_t = 1 + max{alpha : f > epsilon}, i.e., the longest prefix on which similarity stays above threshold, whereas Algorithm 2 breaks at the first alpha with f <= epsilon and sets H'_t = alpha + 1. These differ whenever the similarity falls below epsilon and later rises above it. The authors should specify which procedure is actually used, align the text and pseudocode, and discuss whether the choice affects the reported results.","section":"Section 4.2, Eq. (8) vs. Algorithm 2"},{"comment":"The negative-sampling distribution is ambiguous. Eq. (4) defines negatives as states s- for which there exists z' != z with s- in tau_skill(z'), but z is a continuous latent code, so the condition z' != z holds for almost every pair and does not characterize a different skill trajectory. Algorithm 1 instead samples negatives from a different skill trajectory tau_skill_j != tau_skill_i, which is a different distribution. The paper should precisely define the negative distribution used in training and justify why it approximates unreachability under skill z.","section":"Section 4.1, Eq. (4) and Algorithm 1, line 13"},{"comment":"The main success-rate table omits the two key skill-learning baselines, SPiRL and SkiMo, which appear only as learning curves in Fig. 3 and as timesteps in Table 2. Because the paper's headline claim is competitive or better task-completion rates than SPiRL and SkiMo, the success rates of these baselines should be included in Table 1 so the comparison can be checked. As it stands, the success-rate comparison is not verifiable.","section":"Table 1 and Section 5.2"},{"comment":"The core assumption that semantically similar behaviors produce similar state-change patterns is not directly tested. Fig. 5 is qualitative, and the ablation on epsilon in Appendix D.4.2 does not measure whether the learned similarity function aligns with human-meaningful semantic skill boundaries. The paper would be substantially strengthened by a quantitative evaluation, for example comparing DCSL's relabeled skill boundaries against ground-truth behavior segments in a controlled setting, or by reporting a clustering-quality metric on data with known skill labels.","section":"Section 4.1 and Section 5.4"}],"minor_comments":[{"comment":"The text in Section 4.2 promises a 'detailed convergence analysis' in Appendix A.2, but that appendix contains only intuitive arguments and explicit caveats that exact bounds are not provided. Please reword the promise to match the content.","section":"Appendix A.2"},{"comment":"There are typos such as 'pblueicted' instead of 'predicted' and 'desiblue' instead of 'desired'. Please proofread the appendix.","section":"Appendix C.1"},{"comment":"In Algorithm 2, if no state with similarity <= epsilon is encountered, the loop exits with alpha = len(episode) - t + 1 after the final iteration, so the assigned H'_t can be much larger than intended; the pseudocode should specify the behavior when the threshold is never crossed.","section":"Algorithm 2"},{"comment":"The notation for the skill prior and the KL weight alpha is introduced, but the relationship between the skill prior p_a(z|s) and the later prior p_theta_p(z|s) in Eq. (3) should be clarified, since both are used without an explicit connection.","section":"Section 3.2, Eq. (1)"},{"comment":"The footnote in Table 1 says boxes with '-' denote results that could not be obtained due to unavailability of public implementation, but the table does not show any '-' boxes; either add the missing entries or remove the footnote.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a conference-style submission that would benefit from a round of major revision before journal publication. The core idea is plausible, but the current version contains an incorrect theoretical claim, an ambiguity between the formal definition and the pseudocode of the main relabeling mechanism, and an incomplete comparison table. All of these are fixable within the scope of the manuscript, so I am not recommending rejection, but they must be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read DCSL. The one thing to know: the dynamic length relabeling idea is the real contribution—it is simple, plausible, and the ablations suggest it matters in noisy data—but the theory that supposedly grounds it does not hold, and the tables do not let you check the headline claims against the main baselines. I would not desk-reject it, but it needs substantial revision.\n\nWhat is actually new: the four-state, state-transition skill encoding and the similarity-based length relabeling rule (Eq. 8). Those are not in SPiRL, SkiMo, or C-learning. The contrastive loss itself is standard NCE, and the paper properly cites the earlier work, so the novelty is in the pipeline, not the objective. The empirical pattern—dynamic lengths helping most in the noisiest pick-and-place data—is coherent and worth taking seriously.\n\nNow the soft spots, in order of severity. First, Appendix A.1.1's Theorem 1 proof is algebraically wrong. Substituting the optimal discriminator into the binary NCE loss gives E[log(1+p'/p)+log(1+p/p')], not E[log(p/p')]. So the claimed equivalence to conditional mutual information is false as written. The method may still work empirically, but the theory needs to be corrected or downgraded. Second, Eq. (8) and Algorithm 2 disagree: the formula uses the maximum alpha with f>epsilon, while the algorithm stops at the first alpha with f≤epsilon and sets H'=alpha+1. Those are different relabeling rules, and the paper needs to pick one. Third, Table 1, the success-rate table, omits SPiRL and SkiMo, the primary baselines; the reader is asked to accept learning curves and a timestep table instead. That is a serious reporting gap. Fourth, the negative sampling in Eq. 4 is ill-defined for continuous z (almost every pair satisfies z'≠z), though Algorithm 1's practical choice of sampling from another trajectory is a reasonable workaround. Fifth, Appendix A.2 calls itself a convergence analysis but is heuristic, and Theorem 2 is stated without proof.\n\nCitation pattern is fine: the contrastive core is correctly attributed to Eysenbach et al. 2022 and NCE. The limitations section is honest about CEM instability, data diversity, and interpretability.\n\nWho this is for: people working on hierarchical RL or unsupervised skill discovery from offline data. The dynamic-length idea deserves attention. But as it stands, do not rely on the paper's theory or its untabulated baseline comparisons; the empirical claim is plausible but under-supported. Send it to peer review with explicit requests to fix the proof, align the algorithm with the equation, and provide the missing baseline numbers. Without those changes, the evidence does not support acceptance.","headline":"DCSL's dynamic skill-length relabeling is a genuinely novel and plausible idea, but the theory appendix is algebraically wrong and the main baseline comparisons are missing from the success-rate table; worth refereeing, not worth accepting as is.","tokens_in":19342,"tokens_out":4192,"would_cite":false,"duration_ms":39824,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DCSL defines skills by state transitions and lets a learned similarity function set each skill's length, reporting competitive or better success than fixed-length skill methods on long-horizon offline RL benchmarks.","keywords":["offline reinforcement learning","skill learning","contrastive learning","state transitions","temporal abstraction","dynamic skill length","hierarchical reinforcement learning","long-horizon tasks"],"falsifier":"Hold out a subset of trajectories with hand-labeled behavior boundaries; train DCSL, then measure agreement between the relabeled skill boundaries and the human boundaries, and also compare states that $f_{\\theta_f}$ rates similar with the empirical distribution of states actually reachable from the same skill. Low agreement, or similar ratings for states with divergent future distributions, would refute the core assumption; separately, disabling relabeling while keeping the similarity function should clearly degrade the noisy Pick-and-Place success rate if dynamic length is the operative mechanism.","tokens_in":18222,"feed_emoji":"🤖","tokens_out":9642,"duration_ms":79642,"temperature":0.7,"pith_summary":"This paper proposes Dynamic Contrastive Skill Learning (DCSL), a method for extracting reusable skills from unlabeled offline reinforcement learning data. Its central claim is that a skill should be represented by the state transitions it produces rather than by the raw action sequence executed, and that the skill should end when a learned similarity function no longer judges future states reachable. DCSL clusters semantically similar behaviors into one skill via a noise-contrastive loss, periodically relabels skill lengths, and then plugs the variable-length skills into established skill-based downstream learners. A sympathetic reader would care because fixed-length, action-sequence skills fragment one behavior into many skills and handle noisy data poorly; DCSL reports competitive or better success rates with fewer task-completion timesteps, with its largest gains on the noisiest Pick-and-Place dataset.","feed_headline":"Adaptive-length skills beat fixed-length skills in offline RL","feed_subtitle":"DCSL clusters behaviors by state change, then relabels skill lengths to match, helping noisy long-horizon tasks.","key_machinery":"The load-bearing object is the skill similarity function $f_{\\theta_f}(s,z,s') = \\langle \\phi_{\\theta_\\phi}(s,z), \\psi_{\\theta_\\psi}(s') \\rangle$, a dot product between a skill-conditioned state encoder and a state encoder trained with a noise-contrastive binary loss. It does two jobs: it clusters semantically similar state transitions into the same skill embedding, and its thresholded evaluation defines the dynamic skill length $H'_t = 1 + \\max\\{\\alpha : f_{\\theta_f}(s_t,z_t,s_{t+\\alpha}) > \\epsilon\\}$ during periodic relabeling. The skill embedding itself is an LSTM over four key states (initial, terminal, and two intermediate states), anchoring the representation to state-change pattern rather than to specific actions. A skill target-state predictor is added so that downstream execution can recognize when a variable-length skill has finished.","core_discovery":"On the paper's own terms, the discovery is that a skill can be embedded from four states along its execution — initial, terminal, and two intermediate samples — and compared through a learned skill similarity function $f_{\\theta_f}(s,z,s') = \\langle \\phi_{\\theta_\\phi}(s,z), \\psi_{\\theta_\\psi}(s') \\rangle$. Training this function with a noise-contrastive binary loss on positive pairs (initial state, skill, intermediate state) versus negative states sampled from other skill trajectories is shown to maximize conditional mutual information $I(Z;S'|S)$ and to lower-bound true reachability discrimination. The same function then fixes the skill's duration by $H'_t = 1 + \\max\\{\\alpha : f_{\\theta_f}(s_t,z_t,s_{t+\\alpha}) > \\epsilon\\}$, so skill length tracks the temporal extent of the behavior instead of a fixed horizon. Applied to offline RL benchmarks, the paper reports that this state-transition representation clusters varied action sequences into common skills, avoids dimensional collapse, and yields success rates competitive with or better than fixed-length baselines, with the largest advantage in noisy Pick-and-Place data.","pith_inferences":["The mutual-information reading of the contrastive loss suggests $f_{\\theta_f}$ could be treated as an implicit reachability model; a natural extension is to test whether thresholding it predicts actual future-state reachability better than a separately learned dynamics model.","Because relabeling uses a fixed threshold $\\epsilon=0$ and initial length $H=10$, the method's benefit should shrink as data become cleaner and behavior lengths approach the fixed horizon; injecting controlled random actions into a clean dataset and measuring the success-rate gap would test this.","The paper's qualitative evidence of dimensional collapse in baselines is not quantified; a testable extension is to measure the effective rank of each method's skill embedding space and correlate it with downstream success."],"forward_implications":["If state-transition similarity is the right notion of skill identity, then the same behavior performed with different action sequences collapses into one skill, shrinking the skill space needed for downstream search.","Dynamic length relabeling lets short behaviors embedded inside longer action sequences become independent skills, which the paper identifies as the reason noisy datasets with irrelevant actions are handled better.","DCSL reports competitive or better success rates than SPiRL and SkiMo on AntMaze-Medium, AntMaze-Large, Kitchen, and Pick-and-Place, with the largest margins on the noisiest Pick-and-Place data.","Because skill durations vary during execution, the skill target-state predictor must decide when a skill ends; the paper reports this variable horizon works with SAC-based downstream learning but destabilizes CEM-based planning."],"supporting_citations":[{"why":"Provides the SPiRL skill-prior framework and the fixed-length model-free downstream baseline that DCSL extends.","marker":"Pertsch et al. (2021)"},{"why":"Provides the SkiMo model-based planning baseline and the dynamics-module design that DCSL repurposes as a skill target-state predictor.","marker":"Shi et al. (2022)"},{"why":"Supplies the contrastive similarity form $f(s,z,s')=\\langle\\phi(s,z),\\psi(s')\\rangle$ and the NCE-style training objective for reachability discrimination.","marker":"Eysenbach et al. (2022)"},{"why":"Basis for the contrastive objective and the paper's mutual-information interpretation of the learned similarity.","marker":"Oord et al. (2018)"},{"why":"Gives the noise-contrastive binary loss used to train the skill similarity function.","marker":"Ma & Collins (2018)"},{"why":"Provides the D4RL AntMaze and Kitchen datasets used for evaluation and comparison.","marker":"Fu et al. (2020)"},{"why":"Provides the noisy Meta-World Pick-and-Place datasets that make the dynamic length adjustment visible in results.","marker":"Yoo et al. (2022)"}],"fun_headline_variants":["Skills that adapt their length beat fixed ones in offline RL","State-transition skills with dynamic lengths improve offline RL","Dynamic skill lengths from state transitions outperform fixed","Contrastive skill clustering adapts skill duration to behavior","Skill similarity from state transitions sets adaptive lengths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise that semantically similar behaviors display similar state-change patterns carries the entire method; if that mapping fails, the learned similarity function will not align with true skill boundaries, and thresholding it will either fragment one behavior into many skills or merge unrelated behaviors.","fun_headline_variants_meta":{"raw":{"variants":["Skills that adapt their length beat fixed ones in offline RL","State-transition skills with dynamic lengths improve offline RL","Dynamic skill lengths from state transitions outperform fixed","Contrastive skill clustering adapts skill duration to behavior","Skill similarity from state transitions sets adaptive lengths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3062,"prompt_tokens":953,"completion_tokens":2109,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2035}},"tokens_in":569,"tokens_out":2109,"duration_ms":15163,"temperature":1.0,"reasoning_tokens":2035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:40:40.680351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a subset of trajectories with hand-labeled behavior boundaries; train DCSL, then measure agreement between the relabeled skill boundaries and the human boundaries, and also compare states that $f_{\\theta_f}$ rates similar with the empirical distribution of states actually reachable from the same skill. Low agreement, or similar ratings for states with divergent future distributions, would refute the core assumption; separately, disabling relabeling while keeping the similarity function should clearly degrade the noisy Pick-and-Place success rate if dynamic length is the operative mechanism.","supporting_citations":[{"cited_title":"Skills regularized task decomposition for multi-task offline reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the noisy Meta-World Pick-and-Place datasets that make the dynamic length adjustment visible in results."}],"review_version":1}