Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment

Jinwoo Choi; Seung-Woo Seo

REVIEW 5 major objections 5 minor 1 cited by

Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DCSL defines skills by state transitions and lets a learned similarity function set each skill's length, reporting competitive or better success than fixed-length skill methods on long-horizon offline RL benchmarks.

desk verdict DCSL's dynamic skill-length relabeling is a genuinely novel and plausible idea, but the theory appendix is algebraically wrong and the main baseline comparisons are missing from the success-rate table; worth refereeing, not worth accepting as is. read the letter →

arxiv 2504.14805 v1 pith:TMQ775VC submitted 2025-04-21 cs.LG cs.AIcs.RO

Jinwoo Choi , Seung-Woo Seo This is my paper · ORCID

classification cs.LGcs.AIcs.RO

keywords offlinereinforcementlearningskillcontrastivestatetransitionstemporalabstractiondynamiclengthhierarchicallong-horizontasks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Dynamic Contrastive Skill Learning (DCSL), a method for extracting reusable skills from unlabeled offline reinforcement learning data. Its central claim is that a skill should be represented by the state transitions it produces rather than by the raw action sequence executed, and that the skill should end when a learned similarity function no longer judges future states reachable. DCSL clusters semantically similar behaviors into one skill via a noise-contrastive loss, periodically relabels skill lengths, and then plugs the variable-length skills into established skill-based downstream learners. A sympathetic reader would care because fixed-length, action-sequence skills fragment one behavior into many skills and handle noisy data poorly; DCSL reports competitive or better success rates with fewer task-completion timesteps, with its largest gains on the noisiest Pick-and-Place dataset.

What carries the argument

The load-bearing object is the skill similarity function $f_{\theta_f}(s,z,s') = \langle \phi_{\theta_\phi}(s,z), \psi_{\theta_\psi}(s') \rangle$, a dot product between a skill-conditioned state encoder and a state encoder trained with a noise-contrastive binary loss. It does two jobs: it clusters semantically similar state transitions into the same skill embedding, and its thresholded evaluation defines the dynamic skill length $H'_t = 1 + \max\{\alpha : f_{\theta_f}(s_t,z_t,s_{t+\alpha}) > \epsilon\}$ during periodic relabeling. The skill embedding itself is an LSTM over four key states (initial, terminal, and two intermediate states), anchoring the representation to state-change pattern rather than to specific actions. A skill target-state predictor is added so that downstream execution can recognize when a variable-length skill has finished.

What would settle it

Hold out a subset of trajectories with hand-labeled behavior boundaries; train DCSL, then measure agreement between the relabeled skill boundaries and the human boundaries, and also compare states that $f_{\theta_f}$ rates similar with the empirical distribution of states actually reachable from the same skill. Low agreement, or similar ratings for states with divergent future distributions, would refute the core assumption; separately, disabling relabeling while keeping the similarity function should clearly degrade the noisy Pick-and-Place success rate if dynamic length is the operative mechanism.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a skill can be embedded from four states along its execution — initial, terminal, and two intermediate samples — and compared through a learned skill similarity function $f_{\theta_f}(s,z,s') = \langle \phi_{\theta_\phi}(s,z), \psi_{\theta_\psi}(s') \rangle$. Training this function with a noise-contrastive binary loss on positive pairs (initial state, skill, intermediate state) versus negative states sampled from other skill trajectories is shown to maximize conditional mutual information $I(Z;S'|S)$ and to lower-bound true reachability discrimination. The same function then fixes the skill's duration by $H'_t = 1 + \max\{\alpha : f_{\theta_f}(s_t,z_t,s_{t+\alpha}) > \epsilon\}$, so skill length tracks the temporal extent of the behavior instead of a fixed horizon. Applied to offline RL benchmarks, the paper reports that this state-transition representation clusters varied action sequences into common skills, avoids dimensional collapse, and yields success rates competitive with or better than fixed-length baselines, with the largest advantage in noisy Pick-and-Place data.

Load-bearing premise

The premise that semantically similar behaviors display similar state-change patterns carries the entire method; if that mapping fails, the learned similarity function will not align with true skill boundaries, and thresholding it will either fragment one behavior into many skills or merge unrelated behaviors.

Editorial extensions

If this is right

If state-transition similarity is the right notion of skill identity, then the same behavior performed with different action sequences collapses into one skill, shrinking the skill space needed for downstream search.
Dynamic length relabeling lets short behaviors embedded inside longer action sequences become independent skills, which the paper identifies as the reason noisy datasets with irrelevant actions are handled better.
DCSL reports competitive or better success rates than SPiRL and SkiMo on AntMaze-Medium, AntMaze-Large, Kitchen, and Pick-and-Place, with the largest margins on the noisiest Pick-and-Place data.
Because skill durations vary during execution, the skill target-state predictor must decide when a skill ends; the paper reports this variable horizon works with SAC-based downstream learning but destabilizes CEM-based planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The mutual-information reading of the contrastive loss suggests $f_{\theta_f}$ could be treated as an implicit reachability model; a natural extension is to test whether thresholding it predicts actual future-state reachability better than a separately learned dynamics model.
Because relabeling uses a fixed threshold $\epsilon=0$ and initial length $H=10$, the method's benefit should shrink as data become cleaner and behavior lengths approach the fixed horizon; injecting controlled random actions into a clean dataset and measuring the success-rate gap would test this.
The paper's qualitative evidence of dimensional collapse in baselines is not quantified; a testable extension is to measure the effective rank of each method's skill embedding space and correlate it with downstream success.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read DCSL. The one thing to know: the dynamic length relabeling idea is the real contribution—it is simple, plausible, and the ablations suggest it matters in noisy data—but the theory that supposedly grounds it does not hold, and the tables do not let you check the headline claims against the main baselines. I would not desk-reject it, but it needs substantial revision.

What is actually new: the four-state, state-transition skill encoding and the similarity-based length relabeling rule (Eq. 8). Those are not in SPiRL, SkiMo, or C-learning. The contrastive loss itself is standard NCE, and the paper properly cites the earlier work, so the novelty is in the pipeline, not the objective. The empirical pattern—dynamic lengths helping most in the noisiest pick-and-place data—is coherent and worth taking seriously.

Now the soft spots, in order of severity. First, Appendix A.1.1's Theorem 1 proof is algebraically wrong. Substituting the optimal discriminator into the binary NCE loss gives E[log(1+p'/p)+log(1+p/p')], not E[log(p/p')]. So the claimed equivalence to conditional mutual information is false as written. The method may still work empirically, but the theory needs to be corrected or downgraded. Second, Eq. (8) and Algorithm 2 disagree: the formula uses the maximum alpha with f>epsilon, while the algorithm stops at the first alpha with f≤epsilon and sets H'=alpha+1. Those are different relabeling rules, and the paper needs to pick one. Third, Table 1, the success-rate table, omits SPiRL and SkiMo, the primary baselines; the reader is asked to accept learning curves and a timestep table instead. That is a serious reporting gap. Fourth, the negative sampling in Eq. 4 is ill-defined for continuous z (almost every pair satisfies z'≠z), though Algorithm 1's practical choice of sampling from another trajectory is a reasonable workaround. Fifth, Appendix A.2 calls itself a convergence analysis but is heuristic, and Theorem 2 is stated without proof.

Citation pattern is fine: the contrastive core is correctly attributed to Eysenbach et al. 2022 and NCE. The limitations section is honest about CEM instability, data diversity, and interpretability.

Who this is for: people working on hierarchical RL or unsupervised skill discovery from offline data. The dynamic-length idea deserves attention. But as it stands, do not rely on the paper's theory or its untabulated baseline comparisons; the empirical claim is plausible but under-supported. Send it to peer review with explicit requests to fix the proof, align the algorithm with the equation, and provide the missing baseline numbers. Without those changes, the evidence does not support acceptance.

Referee Report

5 major / 5 minor

Summary. The paper proposes DCSL, an offline skill-learning framework that represents skills through four key states rather than full action sequences, learns a contrastive skill-similarity function f_theta_f, and dynamically relabels skill lengths by thresholding that similarity function on future states. The method is evaluated on AntMaze, Kitchen, Pick-and-Place, and Peg-Insert-Side datasets, with downstream learning performed via SAC (model-free) and CEM (model-based). The central claims are that state-transition-based skills cluster semantically similar behaviors and that dynamic length adjustment improves success and efficiency, especially in noisy datasets.

Significance. If validated, the state-transition skill representation and length relabeling would be a useful contribution to offline long-horizon RL, since existing fixed-length action-sequence methods are known to be brittle to action variation and noisy data. The paper also includes useful ablations on the relabeling threshold, the number of key states, and an additional environment (Peg-Insert-Side). However, the current manuscript has serious gaps in its theoretical justification, algorithmic consistency, and experimental reporting, so the central claims are not yet established.

major comments (5)

[Appendix A.1.1, Eq. (9)] Theorem 1 is algebraically incorrect as stated. Substituting the claimed optimal discriminator sigma(f)=p(s'|s,z)/(p(s'|s,z)+p(s')) into the binary NCE loss does not yield -E[log(p(s'|s,z)/p(s'))] = -I(Z;S'|S); instead the two log terms produce log(1+p(s')/p(s'|s,z)) + log(1+p(s'|s,z)/p(s')), whose expectation is not the conditional mutual information. Binary NCE with a single negative is a logistic discrimination objective, not an exact mutual-information estimator. This invalidates the theoretical grounding for f_theta_f as a semantic skill-similarity measure, and Theorem 2 is similarly asserted without a derivation. The paper should either provide a correct derivation, or explicitly reframe the loss as a heuristic discrimination objective and remove the claim of an information-theoretic foundation.
[Section 4.2, Eq. (8) vs. Algorithm 2] There is a direct inconsistency between the definition and the implementation of skill length relabeling. Eq. (8) defines H'_t = 1 + max{alpha : f > epsilon}, i.e., the longest prefix on which similarity stays above threshold, whereas Algorithm 2 breaks at the first alpha with f <= epsilon and sets H'_t = alpha + 1. These differ whenever the similarity falls below epsilon and later rises above it. The authors should specify which procedure is actually used, align the text and pseudocode, and discuss whether the choice affects the reported results.
[Section 4.1, Eq. (4) and Algorithm 1, line 13] The negative-sampling distribution is ambiguous. Eq. (4) defines negatives as states s- for which there exists z' != z with s- in tau_skill(z'), but z is a continuous latent code, so the condition z' != z holds for almost every pair and does not characterize a different skill trajectory. Algorithm 1 instead samples negatives from a different skill trajectory tau_skill_j != tau_skill_i, which is a different distribution. The paper should precisely define the negative distribution used in training and justify why it approximates unreachability under skill z.
[Table 1 and Section 5.2] The main success-rate table omits the two key skill-learning baselines, SPiRL and SkiMo, which appear only as learning curves in Fig. 3 and as timesteps in Table 2. Because the paper's headline claim is competitive or better task-completion rates than SPiRL and SkiMo, the success rates of these baselines should be included in Table 1 so the comparison can be checked. As it stands, the success-rate comparison is not verifiable.
[Section 4.1 and Section 5.4] The core assumption that semantically similar behaviors produce similar state-change patterns is not directly tested. Fig. 5 is qualitative, and the ablation on epsilon in Appendix D.4.2 does not measure whether the learned similarity function aligns with human-meaningful semantic skill boundaries. The paper would be substantially strengthened by a quantitative evaluation, for example comparing DCSL's relabeled skill boundaries against ground-truth behavior segments in a controlled setting, or by reporting a clustering-quality metric on data with known skill labels.

minor comments (5)

[Appendix A.2] The text in Section 4.2 promises a 'detailed convergence analysis' in Appendix A.2, but that appendix contains only intuitive arguments and explicit caveats that exact bounds are not provided. Please reword the promise to match the content.
[Appendix C.1] There are typos such as 'pblueicted' instead of 'predicted' and 'desiblue' instead of 'desired'. Please proofread the appendix.
[Algorithm 2] In Algorithm 2, if no state with similarity <= epsilon is encountered, the loop exits with alpha = len(episode) - t + 1 after the final iteration, so the assigned H'_t can be much larger than intended; the pseudocode should specify the behavior when the threshold is never crossed.
[Section 3.2, Eq. (1)] The notation for the skill prior and the KL weight alpha is introduced, but the relationship between the skill prior p_a(z|s) and the later prior p_theta_p(z|s) in Eq. (3) should be clarified, since both are used without an explicit connection.
[Table 1] The footnote in Table 1 says boxes with '-' denote results that could not be obtained due to unavailability of public implementation, but the table does not show any '-' boxes; either add the missing entries or remove the footnote.

Circularity Check

1 steps flagged · score 4.0 of 10

The dynamic skill-length relabeling uses the same learned similarity function that was trained on the current skill segmentation, making the 'semantic' boundaries a self-referential fixed point rather than an independent prediction.

self definitional [Section 4.1 (Eq. 5) and Section 4.2 (Eq. 8); Algorithm 2]
"Positive samples pair the initial state st with the skill zt and an intermediate state st+b, chosen to capture long-term skill effects. Negative samples are states s− that are unreachable from st given the skill zt, sampled from different skill trajectories. ... We then compute similarity values for subsequent states st+α (α > 0) and define the new skill length as the maximum number of consecutive time steps where the similarity remains above a threshold ε: H′t = 1 + max{α | fθf (st, zt, st+α) > ε}."

The function fθf in Eq. (8) is the same classifier optimized in Eq. (5), where positives are states inside the current fixed-length segment [t, t+Ht−1] and negatives are states from other skill segments. Thus f is trained to reproduce the segmentation defined by the current Ht. Eq. (8) then uses f's own threshold crossing to define the new length H′t, so the relabeled 'semantic' boundary is not an externally measured property of the behavior; it is the model's learned boundary applied to the same trajectories that generated its supervision. The skill embeddings zt used in Eq. (8) are also inferred from the pre-relabeling segments, closing the loop: segmentation determines z and f, and f determines the new segmentation.

full rationale

The main circularity risk is the self-referential loop between skill segmentation and the learned similarity function: Eq. (5) trains f with positive/negative pairs defined by the current fixed-length segments, and Eq. (8) uses that same f to relabel segment lengths. This makes the discovered 'semantic' boundaries a fixed point of the model's own predictions rather than an independent measurement. The paper's Limitations section itself concedes that 'the semantic meaning of these clustered skills is not always clear or interpretable,' which further weakens the claim that the clusters are externally validated semantic categories. I did not count the algebraic error in Appendix A.1.1 (binary NCE is not exactly conditional mutual information) as circularity; that is a correctness issue, not a reduction of the result to its inputs. The discrepancy between Eq. (8) (max α) and Algorithm 2 (break at the first sim≤ε) is also a consistency/correctness concern rather than a circularity. There is no load-bearing self-citation or imported uniqueness theorem: the cited prior work (Eysenbach et al. 2022, Ma & Collins 2018, SkiMo) is external. Because the central mechanism is partially self-definitional but the paper is also benchmarked against external methods, a score of 4 is appropriate.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central method rests on several hand-set thresholds and weights. The relabeling rule (Eq. 8) depends directly on epsilon, delta_min, delta_max, and the initial length H; the losses depend on five lambda coefficients. These are not fitted from data, but they are chosen by the authors and directly shape the claimed behavior. No new physical or mathematical entities are introduced. The theory uses standard NCE results plus two assumptions that are not proved (Theorem 2 and the convergence claim), so those appear as axioms.

free parameters (7)

Relabeling similarity threshold epsilon = 0.0
Eq. (8) defines new skill length by thresholding f_theta_f at epsilon; global value 0.0 is hand-set in Table 3. D.4.2 ablates five values and reports robustness, but the main experiments use one chosen value.
Initial skill length H = 10
All skills start at length 10 (Section 5.1), matching baseline fixed length; dynamic relabeling then adjusts between delta_min and delta_max.
Skill length bounds delta_min, delta_max = 4 and 30
Constraints in Eq. (8) and Algorithm 2 line 12 prevent abrupt length changes; hand-set in Table 3.
Loss weighting coefficients lambda_BC, lambda_SP, lambda_CL, lambda_RE, lambda_ST = 2, 1, 1, 1, 2
Weights of behavior cloning, skill prior KL, contrastive loss, reconstruction, and target-state prediction in Eqs. (3), (5), (6), (7); hand-set in Table 3.
Target-state distance thresholds = 0.5 (AntMaze), 0.1 (Kitchen), 0.02 (PickPlace)
Used in Appendix C.1 to decide when the predicted skill target state has been reached; environment-specific and hand-set.
Relabeling interval T_relabel = 20,000
Skill lengths are recomputed every 20,000 training steps (Algorithm 1 line 23, Table 3); chosen to allow model adaptation between relabelings.
Number of key states = 4
Skill embedding uses initial, terminal, and two random intermediate states (Section 4.1); D.4.1 ablates 4 versus 10 and shows 4 works well.

assumptions (6)

standard math Optimal discriminator for the logistic NCE loss takes the form sigma(f(s,z,s')) = p(s'|s,z)/(p(s'|s,z)+p(s')).
Used in Appendix A.1.1 to show the contrastive loss reduces to -I(Z;S'|S). A standard result for NCE with equal positive and negative samples.
domain assumption Semantically similar behaviors display similar state change patterns after applying a skill.
Stated in Section 4.1 as the core assumption motivating state-transition based skill representation and the skill similarity function. If false, the contrastive objective does not cluster semantic behaviors.
domain assumption Negative samples drawn from states of other skills (Eq. 4) approximate the distribution of unreachable states p(s|z' != z).
The paper approximates the negative distribution by sampling states from different skill trajectories in a batch. This is an approximation to true unreachability, and is ill-defined for continuous latent codes.
ad hoc to paper Skill length relabeling with threshold epsilon and periodic updates converges to stable, meaningful skill boundaries.
Appendix A.2 provides only informal stability arguments ('expected to be bounded', 'help smooth the optimization landscape') and no quantitative convergence proof. The paper claims a 'detailed convergence analysis' but none is given.
domain assumption The skill target state predictor T_theta_T can accurately predict the state after a variable-length skill, and downstream termination by distance threshold works.
Used in downstream learning (Section 4.3 and Appendix B.2). If the target-state prediction is inaccurate, dynamic skill termination breaks down.
ad hoc to paper Theorem 2: the contrastive loss lower-bounds the true skill discrimination task.
Stated in Appendix A.1.2 with 'It can be shown' but no proof is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment." pith.science (2026). https://pith.science/paper/TMQ775VC

@misc{pith2026250414805,
  author       = {Pith},
  title        = {Pith review of: Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TMQ775VC}},
  note         = {Machine review of arXiv:2504.14805}
}

read the original abstract

Reinforcement learning (RL) has made significant progress in various domains, but scaling it to long-horizon tasks with complex decision-making remains challenging. Skill learning attempts to address this by abstracting actions into higher-level behaviors. However, current approaches often fail to recognize semantically similar behaviors as the same skill and use fixed skill lengths, limiting flexibility and generalization. To address this, we propose Dynamic Contrastive Skill Learning (DCSL), a novel framework that redefines skill representation and learning. DCSL introduces three key ideas: state-transition based skill representation, skill similarity function learning, and dynamic skill length adjustment. By focusing on state transitions and leveraging contrastive learning, DCSL effectively captures the semantic context of behaviors and adapts skill lengths to match the appropriate temporal extent of behaviors. Our approach enables more flexible and adaptive skill extraction, particularly in complex or noisy datasets, and demonstrates competitive performance compared to existing methods in task completion and efficiency.

Figures

Figures reproduced from arXiv: 2504.14805 by the authors.

**Figure 2.** The overall architecture of our framework, including skill extraction, skill length relabel [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Performance comparison across Antmaze-Medium, Antmaze-Large, Kitchen, and Pick [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Ablation study results in the Pick-and-Place environment across the ME, MR, and RP [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Comparison of execution results for 50 randomly sampled skills in the Antmaze environ [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Environments from left-to-right: Antmaze, a long-horizon navigation task with two vari [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

**Figure 8.** Figure 8: Performance comparison of various methods in the Peg-Insert-Side task across different [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 7.** Figure 7: Peg-Insert-Side task: The robot must grasp a long object and insert it into a hole in the box. First, there’s a difference in the size of the object the robot needs to grasp. In Pick-and-Place, the object is very small, so even a slight misalignment of the robot’s gri…

**Figure 9.** Figure 9: Visualization of skill usage in the pick-and-place task. The x-axis represents the skills [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

**Figure 10.** Figure 10: Distribution of relabeled skill lengths across various environments using our framework. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: The ablation study on the number of key states for the Kitchen task. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

**Figure 12.** Figure 12: The ablation study on the similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
cs.RO 2025-12 conditional novelty 6.0 of 10

An imitation-learning system that segments demonstrations into VLM-labeled atomic skills, aligns them with contrastive learning, and uses keypose prediction to chain skills, outperforming prior baselines in multi-task...

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Opal: Offline primitive discovery for accelerating offline reinforcement learning

Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020

arXiv 2010
[2]

The option-critic architecture

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

2017
[3]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

2020
[4]

Challenges of real-world reinforcement learning

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901, 2019

arXiv 1904
[5]

C-learning: Learning to achieve goals via recursive classification

Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020

arXiv 2011
[6]

Contrastive learning as goal-conditioned reinforcement learning

Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 35603--35620, 2022

2022
[7]

Learning temporally abstractworld models without online experimentation

Benjamin Freed, Siddarth Venkatraman, Guillaume Adrien Sartoretti, Jeff Schneider, and Howie Choset. Learning temporally abstractworld models without online experimentation. In International Conference on Machine Learning, pp.\ 10338--10356. PMLR, 2023

work page 2023
[8]

D4rl: Datasets for deep data-driven reinforcement learning

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

arXiv 2004

Show all 39 references

[9]

Soft actor-critic algorithms and applications

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018

2018 arXiv
[10]

Skill-critic: Refining learned skills for hierarchical reinforcement learning

Ce Hao, Catherine Weaver, Chen Tang, Kenta Kawamoto, Masayoshi Tomizuka, and Wei Zhan. Skill-critic: Refining learned skills for hierarchical reinforcement learning. IEEE Robotics and Automation Letters, 2024

2024
[11]

The termination critic

Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critic. arXiv preprint arXiv:1902.09996, 2019

1902 arXiv
[12]

Momentum contrast for unsupervised visual representation learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9729--9738, 2020

2020
[13]

Learning options via compression

Yiding Jiang, Evan Liu, Benjamin Eysenbach, J Zico Kolter, and Chelsea Finn. Learning options via compression. Advances in Neural Information Processing Systems, 35: 0 21184--21199, 2022

2022
[14]

Understanding dimensional collapse in contrastive self-supervised learning

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021

2021 arXiv
[15]

Conservative q-learning for offline reinforcement learning

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

2020
[16]

Curl: Contrastive unsupervised representations for reinforcement learning

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp.\ 5639--5650. PMLR, 2020

2020
[17]

Learning latent plans from play

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp.\ 1113--1132. PMLR, 2020

2020
[18]

Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency

Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812, 2018

2018 arXiv
[19]

Representation learning with contrastive predictive coding

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[20]

Accelerating reinforcement learning with learned skill priors

Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp.\ 188--204. PMLR, 2021

2021
[21]

Cross-domain transfer via semantic skill imitation

Karl Pertsch, Ruta Desai, Vikash Kumar, Franziska Meier, Joseph J Lim, Dhruv Batra, and Akshara Rai. Cross-domain transfer via semantic skill imitation. arXiv preprint arXiv:2212.07407, 2022

2022 arXiv
[22]

Alvinn: An autonomous land vehicle in a neural network

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988

1988
[23]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

2021
[24]

Optimization of computer simulation models with rare events

Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99 0 (1): 0 89--112, 1997

1997
[25]

Learning robot skills with temporal variational inference

Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pp.\ 8624--8633. PMLR, 2020

2020
[26]

Dynamics-aware unsupervised discovery of skills

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019

1907 arXiv
[27]

Emergent real-world robotic skills via unsupervised off-policy reinforcement learning

Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off-policy reinforcement learning. arXiv preprint arXiv:2004.12974, 2020

2004 arXiv
[28]

Skill-based model-based reinforcement learning

Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560, 2022

2022 arXiv
[29]

Mastering the game of go with deep neural networks and tree search

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484-...

2016
[30]

Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning

Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112 0 (1-2): 0 181--211, 1999

1999
[31]

Grandmaster level in starcraft ii using multi-agent reinforcement learning

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha \"e l Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575 0 (7782): 0 350--...

2019
[32]

Behavior contrastive learning for unsupervised skill discovery

Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In International Conference on Machine Learning, pp.\ 39183--39204. PMLR, 2023

2023
[33]

Skills regularized task decomposition for multi-task offline reinforcement learning

Minjong Yoo, Sangwoo Cho, and Honguk Woo. Skills regularized task decomposition for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 37432--37444, 2022

2022
[34]

Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.\ 1094--1100. PMLR, 2020

2020
[35]

Stabilizing contrastive rl: Techniques for robotic goal reaching from offline data

Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing contrastive rl: Techniques for robotic goal reaching from offline data. arXiv preprint arXiv:2306.03346, 2023

2023 arXiv
[36]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[37]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[38]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[39]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Opal: Offline primitive discovery for accelerating offline reinforcement learning

Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020

arXiv 2010

[2] [2]

The option-critic architecture

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

2017

[3] [3]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

2020

[4] [4]

Challenges of real-world reinforcement learning

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901, 2019

arXiv 1904

[5] [5]

C-learning: Learning to achieve goals via recursive classification

Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020

arXiv 2011

[6] [6]

Contrastive learning as goal-conditioned reinforcement learning

Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 35603--35620, 2022

2022

[7] [7]

Learning temporally abstractworld models without online experimentation

Benjamin Freed, Siddarth Venkatraman, Guillaume Adrien Sartoretti, Jeff Schneider, and Howie Choset. Learning temporally abstractworld models without online experimentation. In International Conference on Machine Learning, pp.\ 10338--10356. PMLR, 2023

work page 2023

[8] [8]

D4rl: Datasets for deep data-driven reinforcement learning

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

arXiv 2004