Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints

Georg Martius; Marin Vlastelica; Pavel Kolev

REVIEW 3 major objections 6 minor 1 cited by

Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dual-Force maximizes offline diversity in imitation-constrained skill learning with a discriminator-free Van der Waals objective estimated by DICE, and recalls every skill via pretrained Functional Reward Encoding.

desk verdict A plausible recombination of VdW diversity, DICE, and FRE with a real reproducibility gap: the feature map defining diversity is never specified, and the experiments lack baselines and error bars; worth a serious referee but needs substantial revision. read the letter →

arxiv 2501.04426 v2 pith:EWGPDOUI submitted 2025-01-08 cs.LG cs.AIcs.RO

Pavel Kolev , Marin Vlastelica , Georg Martius This is my paper · ORCID

classification cs.LGcs.AIcs.RO

keywords offlinereinforcementlearningdiversitymaximizationimitationconstraintssuccessorfeaturesVanderWaalsforceDICEfunctionalrewardencodingskilldiscovery

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that offline diversity maximization under imitation constraints can be made stable and practical by replacing the mutual-information objective and its skill discriminator with a Van der Waals force objective whose gradient has closed form in successor features. The entire objective is estimated off-policy through DICE importance ratios, and the value function and policy are conditioned on a pretrained Functional Reward Encoding to handle non-stationary rewards and to recall each encountered skill by its latent code. On two Solo12 simulation benchmarks, the recovered skill sets are claimed to match a target expert state occupancy, recover all expert modes and several offline modes, and include solutions that outperform the expert under adversarial obstacle variations. The motivation is that a demonstration set can then yield a portfolio of distinct, robust behaviors without any environment interaction.

What carries the argument

The VdW force objective over successor features: each skill $i$ is represented by mean feature $\psi_i = E_{d_i}[\phi(s)]$, and its diversity reward at state $s$ is $\beta_i(s,a) = (1 - (\ell_i/\ell_0)^3)\langle\phi(s), \psi_i - \psi_{j^*}\rangle$, where $j^*$ is the nearest other skill and $\ell_i$ is the distance to it; close skills repel and far skills attract, with $\ell_0$ controlling the target spacing. The off-policy engine is the DICE importance ratio $\eta_i = \exp\{\delta_i\}/E_{d_O}\exp\{\delta_i\}$, with $\delta_i$ the TD error of the KL-regularized dual value function, which turns offline expectations over $d_O$ into expectations under the skill's occupancy. FRE latent codes $z_r$ condition the value function and policy and act as a look-up table for non-stationary rewards.

What would settle it

Train Dual-Force twice on the same offline datasets with two different feature maps, one encoding behaviorally relevant variables such as base height and route side and one encoding irrelevant variables; if both yield separated successor features but only the first corresponds to visibly distinct rollouts, the diversity claim is an artifact of the chosen $\phi$. A second check is to remove expert states from the offline mixture so $d_E$ has support where $d_O$ is zero, where Assumption A.1 fails and the constraint estimator should degrade or become undefined.

Watch

Extended reading notes

Core claim

The central claim is that all quantities needed for VdW-force diversity — successor features, the dual-conjugate gradient variable, and KL-divergence constraint estimates — admit off-policy estimators, so no skill discriminator or online rollouts are required. Dual-Force relaxes the imitation constraint to a state-only KL bound, computes importance ratios $\eta_i$ from the DICE dual solution, and uses them to estimate successor features, the VdW reward, and constraint violations. Conditioning the value function and policy on FRE latent codes of the current reward makes the non-stationary reward updates stable and gives zero-shot recall of every skill encountered during training, removing the fixed-number-of-skills requirement. The experimental claim is that on locomotion and obstacle navigation with the Solo12 quadruped, this yields diverse, high-performing skills that imitate the expert and are more robust to adversarial obstacles.

Load-bearing premise

Diversity is measured entirely through a fixed feature map of the state, so if that map misses the behaviors that matter, the skills can be well-separated in the objective while behaving alike; the estimator also assumes the offline mixture covers every expert state.

Editorial extensions

If this is right

Offline skill discovery no longer needs a separately trained skill discriminator, removing a phase that was unstable under non-stationary rewards.
The number of skills is not a fixed input; every reward encountered during training leaves a latent code that can be recalled at evaluation time.
On Solo12 locomotion and navigation datasets, the recovered skill sets are claimed to match expert state occupancy while spanning different base heights, velocities, and obstacle routes.
In adversarial obstacle configurations, some learned skills outperform the SMODICE-expert, so the portfolio affords robustness to distribution shift.
The estimator generalizes to arbitrary f-divergence imitation constraints, not only KL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct ablation that fixes the FRE conditioning and swaps only the diversity objective (VdW versus the mutual-information bound) would isolate whether the stability gain comes from the discriminator-free objective or from conditioning; the paper reports the combined method only.
Because the FRE code is a hash of the reward, the same mechanism could be used after training to compose skills sequentially or to interpolate between latent codes, though the paper does not test composition.
If the feature map is learned or expanded during training, the VdW force would act on behaviorally meaningful dimensions and the method would generalize beyond hand-chosen features; this is an extension, not a claim of the paper.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper combines three known pieces — the VdW diversity objective from Zahavy et al., DICE-style off-policy estimation, and FRE conditioning — into an offline skill discovery method that avoids a learned discriminator. That combination is new relative to the cited literature, and the DICE machinery is applied in a way that is mostly self-consistent. The FRE conditioning for non-stationary rewards is sensible, and the authors are honest that it serves as a hash map rather than a diversity driver. The behavioral trajectory plots do show genuinely different gaits and obstacle strategies, which is evidence that something is working.

The soft spots are real. The most load-bearing is the feature map phi. It appears in the VdW reward, in Algorithm 1, and in the Figure 3 diversity plots, but the paper never says what phi is for the experiments. That is not a small omission: the entire diversity measure is defined along phi, so without a specified phi the method is not reproducible and the diversity claim is not falsifiable. The successor-feature clustering in Figure 3 is also partly circular, because the algorithm optimizes exactly those successor features; the trajectory plots are more convincing, but they also depend on the unspecified phi. Second, the move from the Lagrangian in Eq. (11) to the practical objective in Eq. (12) is heuristic: the KL penalty coefficient is 1 rather than sigma, and the diversity reward is weighted by (1-sigma) without formal support. Third, there are no baseline comparisons, no seeds, and no error bars anywhere in the experiments. The main empirical claim — stability and efficiency over prior offline diversity methods — is asserted, not demonstrated.

None of these are fatal in principle. The derivation is not fitted to the target results, the FRE is pretrained on independent random rewards, and the core idea could work with a specified phi and a proper experimental comparison. But as submitted, the paper is under-specified where it matters most. I would send it to reviewers, because the combination is interesting and the math mostly checks out, but I would expect major revision, not acceptance.

For a reading group, this is a useful case study in how an unspecified design choice can undermine an otherwise clever method. I would not cite it in its current form.

Referee Report

3 major / 6 minor

Summary. The paper proposes Dual-Force, an offline algorithm for diversity maximization under imitation constraints. The method replaces the skill discriminator of prior work with an off-policy DICE-based estimator of a Van der Waals (VdW) force objective defined on successor features, and it conditions the value function and policy on a pre-trained Functional Reward Encoding (FRE) to handle non-stationary intrinsic rewards. The authors evaluate on two Solo12 simulation benchmarks (locomotion and obstacle navigation), reporting that the learned skill sets are diverse, match an expert state occupancy, and improve robustness to blocked obstacle paths.

Significance. If the derivation and experiments hold, the paper would make a useful contribution to offline skill discovery: it removes the skill-discriminator training phase, provides a DICE-based off-policy estimator of a physically inspired diversity objective, and leverages FRE for zero-shot skill recall without pre-specifying the number of skills. The appendices contain clean derivations of the state-only KL relaxation and its finite-sample estimator, and the authors are transparent about the role of the FRE as a reward encoder. However, the load-bearing feature map phi is never specified, the Lagrangian-to-implemented-objective step is not justified as written, and the experiments lack quantitative comparisons with existing offline diversity methods; these issues currently prevent the central claims from being fully substantiated.

major comments (3)

[Sec. 3.1, Sec. 4.1, Algorithm 1] The feature mapping phi: S -> R^n that defines successor features psi_i = E_{d_i(s)}[phi(s)] is never specified anywhere in the paper. Every downstream quantity depends on it: the VdW reward beta_i^k(s,a) in Sec. 4.1, the successor-feature distance matrix and UMAP plots in Fig. 3, and the weights w_i through the DICE ratios. Because phi is free, the optimized objective is underspecified, and the diversity evidence in Fig. 3 is computed from the same unspecified quantity that the algorithm optimizes. The authors should specify phi precisely (e.g., the full proprioceptive state vector, a learned encoding, or a defined set of features), state where it enters Algorithm 1 as an input, and report sensitivity to the choice of phi.
[Sec. 4.1, Eqs. (11)-(13)] The transition from the Lagrangian in Eq. (11) to the practical objective in Eq. (12) is not mathematically justified. In Eq. (11), the constraint term is multiplied by lambda_i and the KL term appears as -lambda_i D_KL(d_i||d_O), whereas Eq. (12) has an unconditional -D_KL(d_i||d_O) term with coefficient 1 and a reward R_mu^i(s,a) = (1-sigma(mu_i)) beta_i^k(s,a) + sigma(mu_i) log(c^*(s)/(1-c^*(s))). If sigma(mu_i) is the bounded Lagrange multiplier, the Lagrangian would instead yield a reward beta + sigma log(c/(1-c)) with a KL coefficient sigma, or, after dividing by (1+sigma), weights (1-sigma)beta and sigma log(c/(1-c)) but with KL coefficient (1-sigma). As written, Eqs. (12)-(13) are a heuristic convex combination, not a relaxation of Eq. (11). The authors should either provide a correct derivation, state explicitly that this is a heuristic approximation, or revise the equations so that the KL coefficient and the Lagrange multiplier are consistent.
[Sec. 5, Experiments] The experimental section does not include quantitative comparisons with the most relevant prior offline diversity method, Vlastelica et al. (2024), nor with standard DICE-based imitation baselines such as SMODICE/DemoDICE beyond the SMODICE-expert. The claims in the abstract and introduction that Dual-Force is 'enhanced', 'more stable', and 'more efficient' than prior work are therefore not directly supported. The authors should add tables reporting, for each task, the number of recovered skills, imitation constraint violation (e.g., estimated KL to the expert state occupancy), diversity metrics, and mean returns, together with at least one baseline that also performs offline diversity maximization under imitation constraints. The trajectory plots in Figs. 2, 4, and 5 are informative but do not by themselves substantiate the quantitative superiority claims.

minor comments (6)

[Algorithm 1, Inputs] Algorithm 1 lists the number of occupancies n, number of state subsets m, number of state-reward pairs t, and Polyak scale alpha, but it does not list the feature map phi, despite phi(s) being used inside the loop. This omission should be fixed.
[Appendix A, Theorem numbering] The main text refers to 'Thm. A.2' and 'Theorem A.3', but the appendix labels the corresponding results as 'Lemma A.2' and 'Corollary A.3', and 'Assumption A.1' is sometimes called 'Theorem A.1' inside the appendix proofs. The numbering should be made consistent.
[Sec. 5.1, Data Collection] The sentence describing offline behaviors says the dataset contains 'base-heights (low, middle, orange)', where 'orange' appears to be a typo for 'high' or another height category. This should be corrected.
[Fig. 3, captions] The axes of the distance matrices and UMAP plots in Fig. 3 are unlabeled, and it is not stated which skill corresponds to which point in the UMAP panels. Adding labels and a legend would improve interpretability.
[Sec. 5, Experimental setup] The paper does not report hyperparameters for the value function and policy networks, the discriminator, or the batch sizes and learning rates, nor does it state whether code or trained models will be released. These details are needed for reproducibility.
[Sec. 4.3, FRE claim] The statement that the FRE latent representation 'cannot affect the diversity of skills learned' is too strong: the FRE-conditioned value function and policy are trained on rewards R_i^k that depend on the VdW term, so the representation can influence training dynamics and thus indirectly the skill set. The authors should soften this claim or qualify it as referring only to the diversity objective itself.

Circularity Check

1 steps flagged · score 6.0 of 10

Figure 3's successor-feature diversity evidence restates the VdW objective that Algorithm 1 optimizes; the rest of the derivation is self-contained.

self definitional [Algorithm 1 (VdW Force block) and Section 5.1 / Figure 3 caption]
"compute closest distance ℓ_i^k := ∥ψ_i^k−ψ_{j*}^k∥_2 ... compute VdW reward β_i^k(s,a) := (1− (ℓ_i^k/ℓ_0)^3)⟨ϕ(s),ψ_i^k−ψ_{j*}^k⟩ ... Figure 3 (a,b) shows that the successor features of the learned skills are clustered into three groups (according to the base-height). While the ℓ2 pairwise distance between the successor features within a cluster is small, the distance between clusters is large."

The VdW reward in Algorithm 1 is a decreasing function of the pairwise ℓ2 distance between successor features ψ_i^k and their nearest neighbor ψ_{j*}^k, with ψ_i^k defined as Σ_{(s,a)∈D_O} w_i^k(s,a) ϕ(s). The quantitative diversity evidence in Figure 3 is exactly the same object: pairwise ℓ2 distances and UMAP projections of successor features. Since the policy and value updates are driven by maximizing this precise separation (Eq. 9), observing separated successor features is a check that the optimization succeeded, not an independent confirmation that diverse behaviors were discovered. The trajectory plots in Figures 2 and 4 are independent behavioral evidence, but the successor-feature clustering claim is tied by construction to the optimized objective.

full rationale

The formal off-policy derivation is mostly self-contained: Eq. (14) is the standard DICE Fenchel dual, Eq. (15) gives the importance ratio, and Eq. (17) estimates the constraint; none of these steps imports a load-bearing result from the authors' prior work. FRE conditioning is taken from Frans et al. (2024) (external) and Supp. D explicitly states that the FRE latent cannot affect skill diversity. Self-citations to Vlastelica et al. (2024) and Cheng et al. (2024) are used for motivation and dataset collection, not as proof of the central claim. The one genuine circular element is that the paper's quantitative diversity evidence (Figure 3) uses the same successor-feature separation that the VdW objective in Eq. (9) and Algorithm 1 are designed to maximize, so the clustering result is partially self-definitional. The missing specification of ϕ and the heuristic move from Eq. (11) to Eq. (12) are real reproducibility and rigor concerns, but they are not themselves circularity. Overall, the central algorithmic derivation is not forced by definition, yet one of the main experimental predictions reduces to the optimized objective.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on the DICE dual, the coverage assumption, the suitability of the VdW diversity measure, and an unstated feature map phi. The experimental claims add further dependence on FRE generalization and on hand-set hyperparameters epsilon, l0, n, alpha, m, and t.

free parameters (7)

VdW length scale l0 = not reported
Controls the scale at which the repulsive/attractive VdW force balances; central to the diversity objective in eq. (9).
Imitation tolerance epsilon = not reported
Constraint threshold in eqs. (6) and (10); chosen by hand and affects how much diversity is allowed.
Number of occupancies n = 3 in all experiments
The algorithm optimizes n state-action occupancies; results may depend on this choice.
Feature map phi(s) = unspecified
Defines successor features and hence what counts as diverse; never stated in the paper.
Polyak scale alpha = not reported
Controls time-averaging of occupancy weights in Algorithm 1.
FRE subset count m and size t = not reported
Parameters for averaging FRE latent representations; details in Supp. D.
Bounded Lagrange multipliers sigma(mu_i) = learned during training
Adaptive multipliers balancing diversity and imitation; the sigmoid bounds them to [0,1].

assumptions (5)

domain assumption Expert coverage: dE(s)>0 implies dO(s)>0 (Assumption A.1).
Required for the state-only KL estimator and the discriminator c* to be well-defined.
standard math Fenchel duality / DICE dual for KL-regularized RL (Nachum and Dai, 2020).
Eq. (14) uses the DICE dual to solve the inner maximization in eq. (11).
domain assumption The FRE latent code generalizes to all rewards encountered during training.
The value function and policy conditioning only help if the pretrained FRE encodes novel rewards accurately.
domain assumption The Van der Waals force in eq. (9) is an appropriate diversity measure with the stated gradient.
Taken from Zahavy et al. 2023; whether it induces the desired skill distribution is assumed.
ad hoc to paper The practical objective in eqs. (12)-(13) is a valid Lagrange relaxation of eq. (11).
The transition to the practical reward does not match the Lagrangian as written; this is an assumed heuristic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints." pith.science (2026). https://pith.science/paper/EWGPDOUI

@misc{pith2026250104426,
  author       = {Pith},
  title        = {Pith review of: Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWGPDOUI}},
  note         = {Machine review of arXiv:2501.04426}
}

read the original abstract

Offline diversity maximization under imitation constraints can transform demonstration data into a set of distinct behavioral policies, improving robustness to distribution shift without additional environment interaction. In practice, however, existing offline approaches often rely on mutual-information objectives that require training a skill discriminator and can become unstable under the non-stationary rewards induced by alternating Lagrangian optimization. We introduce Dual-Force, an offline algorithm that (i) maximizes diversity using an off-policy estimator of a Van der Waals (VdW) force objective computed from successor features, eliminating the skill discriminator, and (ii) stabilizes training under non-stationary intrinsic rewards by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE). The FRE code also enables zero-shot recall of every encountered skill via its associated latent representation, removing the need to pre-specify a fixed number of skills. On two Solo12 simulation benchmarks (locomotion and obstacle navigation), Dual-Force recovers diverse high-performing behaviors while matching a target expert state occupancy and improves robustness in adversarial obstacle variations.

Figures

Figures reproduced from arXiv: 2501.04426 by the authors.

**Figure 2.** A performance benchmark of skills learned in the locomotion task. The [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Tasks: (a,b) Locomotion and (c,d) Navigation. The [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: A performance benchmark of skills learned in the obstacle navigation task, where the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: A performance benchmark with additional fence obstacles, of height 0.6m, partially blocking [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Geometry of Nonlinear Reinforcement Learning
cs.LG 2025-09 conditional novelty 5.0 of 10

Actor-critic reinforcement learning methods are reformulated as mirror descent on the occupancy manifold, and a Hessian-based update is proposed for nonlinear and constrained objectives.

Reference graph

Works this paper leans on

36 extracted references · 27 canonical work pages · cited by 1 Pith paper

[1]

Abbeel and A

P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004

work page 2004
[2]

Achiam, H

J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. CoRR, abs/1807.10299, 2018. URL http://arxiv.org/abs/1807.10299

arXiv 2018
[3]

A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=HyxQzBceg

work page 2017
[4]

Barreto, W

A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. Van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv:1606.05312, 2016

arXiv 2016
[5]

V. S. Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54 0 (3): 0 207--213, 2005

work page 2005
[6]

Campos, A

V. Campos, A. Trott, C. Xiong, R. Socher, X. Gir \' o - i - Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pages 1317--1327. PMLR ,...

work page 2020
[7]

Cheng, M

J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius. Learning diverse skills for local navigation under multi-constraint optimality. In IEEE International Conference on Robotics and Automation, ICRA 2024, PACIFICO, Yokohama, May 13th to 17th, 2024 . IEEE , 2024

work page 2024
[8]

B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesv \'a ri, and D. Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33: 0 9398--9411, 2020

work page 2020

Show all 36 references

[9]

P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5 0 (4): 0 613--624, 1993. doi:10.1162/neco.1993.5.4.613

1993 doi
[10]

Eysenbach, A

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net...

2019
[11]

Frans, S

K. Frans, S. Park, P. Abbeel, and S. Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview....

2024
[12]

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

2004 arXiv
[13]

Gregor, D

K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. In International Conference on Learning Representations, 2017

2017
[14]

Grimminger , A

F. Grimminger , A. Meduri , M. Khadiv , J. Viereck , M. Wüthrich , M. Naveau , V. Berenz , S. Heim , F. Widmaier , T. Flayols , J. Fiene , A. Badri-Spröwitz , and L. Righetti . An open torque-controlled modular robot architecture for legged locomotion research. IEEE Robotics a...

2020
[15]

Gulrajani, F

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pag...

2017
[16]

Hansen, W

S. Hansen, W. Dabney, A. Barreto, D. Warde - Farley, T. V. de Wiele, and V. Mnih. Fast task inference with variational intrinsic successor features. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.ne...

2020
[17]

G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demo DICE : Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=BrPdX1bDZkQ

2022
[18]

J. Kim, S. Park, and G. Kim. Unsupervised skill discovery with bottleneck option learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research, pages 5572--5...

2021
[19]

J. Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120--6130. PMLR, 2021

2021
[20]

J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022

2022
[21]

C. Li, M. Vlastelica, S. Blaes, J. Frey, F. Grimminger, and G. Martius. Learning agile skills via adversarial imitation of rough partial demonstrations. In Conference on Robot Learning, pages 342--352. PMLR, 2023

2023
[22]

Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedin...

2022
[23]

Y. J. Ma, J. Yan, D. Jayaraman, and O. Bastani. Offline goal-conditioned reinforcement learning via f -advantage regression. In NeurIPS, 2022 b . URL http://papers.nips.cc/paper\_files/paper/2022/hash/022a39052abf9ca467e268923057dfc0-Abstract-Conference.html

2022
[24]

McInnes, J

L. McInnes, J. Healy, N. Saul, and L. Gro berger. UMAP: uniform manifold approximation and projection. J. Open Source Softw., 3 0 (29): 0 861, 2018. doi:10.21105/JOSS.00861. URL https://doi.org/10.21105/joss.00861

2018 doi
[25]

Nachum and B

O. Nachum and B. Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866, 2020

2001 arXiv
[26]

Nachum, B

O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience, 2019

2019
[27]

M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

2014
[28]

Sharma, S

A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=...

2020
[29]

Stooke, J

A. Stooke, J. Achiam, and P. Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Resear...

2020
[30]

Strouse, K

D. Strouse, K. Baumli, D. Warde - Farley, V. Mnih, and S. S. Hansen. Learning more skills through optimistic exploration. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openrevi...

2022
[31]

Tishby, F

N. Tishby, F. C. N. Pereira, and W. Bialek. The information bottleneck method. CoRR, physics/0004057, 2000. URL http://arxiv.org/abs/physics/0004057

2000 arXiv
[32]

Vlastelica, S

M. Vlastelica, S. Blaes, C. Pinneri, and G. Martius. Risk-averse zero-order trajectory optimization. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 444--454. PMLR,...

2022
[33]

Vlastelica, J

M. Vlastelica, J. Cheng, G. Martius, and P. Kolev. Diverse offline imitation learning. In Reinforcement Learning Conference, 2024

2024
[34]

Zahavy, B

T. Zahavy, B. O'Donoghue, G. Desjardins, and S. Singh. Reward is enough for convex mdps. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 25746--25759, 20...

2021
[35]

Zahavy, Y

T. Zahavy, Y. Schroecker, F. M. P. Behbahani, K. Baumli, S. Flennerhag, S. Hou, and S. Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, ...

2023
[36]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Abbeel and A

P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004

work page 2004

[2] [2]

Achiam, H

J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. CoRR, abs/1807.10299, 2018. URL http://arxiv.org/abs/1807.10299

arXiv 2018

[3] [3]

A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=HyxQzBceg

work page 2017

[4] [4]

Barreto, W

A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. Van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. arXiv preprint arXiv:1606.05312, 2016

arXiv 2016

[5] [5]

V. S. Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54 0 (3): 0 207--213, 2005

work page 2005

[6] [6]

Campos, A

V. Campos, A. Trott, C. Xiong, R. Socher, X. Gir \' o - i - Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pages 1317--1327. PMLR ,...

work page 2020

[7] [7]

Cheng, M

J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius. Learning diverse skills for local navigation under multi-constraint optimality. In IEEE International Conference on Robotics and Automation, ICRA 2024, PACIFICO, Yokohama, May 13th to 17th, 2024 . IEEE , 2024

work page 2024

[8] [8]

B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesv \'a ri, and D. Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems, 33: 0 9398--9411, 2020

work page 2020