Pith. sign in

REVIEW 5 major objections 5 minor 60 references

Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a bisimulation-style metric on latent task beliefs bounds value and transfer differences between tasks, and that using it speeds adaptation in sparse-reward meta-RL.

desk verdict SimBelief's empirical recipe is plausible, but the theoretical transfer claims do not hold up as written; still worth refereeing for the method. read the letter →

arxiv 2506.19785 v1 pith:LF6QH5KY submitted 2025-06-24 cs.AI

Menglong Zhang , Fuyuan Qian This is my paper · ORCID
classification cs.AI
keywords meta-reinforcementlearningtaskbeliefsimilaritybisimulationmetricBayes-AdaptiveMDPsparserewardslatentdynamicsinferenceout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that meta-reinforcement learning can adapt quickly to unknown tasks in sparse-reward environments if the agent measures task similarity in a latent space instead of trying to reconstruct rewards. The proposed method, SimBelief, defines a latent task belief metric $d^\pi(z_i, z_j)$ inspired by bisimulation metrics: the distance between two tasks' latent beliefs is the combined difference of their learned reward, transition, and inverse-dynamics models. The paper proves that tasks whose latent beliefs are close have close value functions under the same policy (Theorem 1), and that a policy trained on one task transfers to another with bounded error (Theorem 2). If correct, an agent that succeeds on one task can propagate that success to similar unseen tasks quickly, without dense reward signals, which is why the authors report state-of-the-art results on sparse-reward MuJoCo and panda-gym benchmarks, including out-of-distribution tasks.

What carries the argument

The load-bearing object is the latent task belief metric of Definition 2, $d^\pi(z_i, z_j) = |R_i^\pi(s_i^+, a_i) - R_j^\pi(s_j^+, a_j)| + W_2(d^\pi)(T_i^\pi(s_i^+, a_i), T_j^\pi(s_j^+, a_j)) + \|I_i^\pi(s_i^+, s_i'^+) - I_j^\pi(s_j^+, s_j'^+)\|_1$, evaluated on augmented states $s^+ = (g(s), z)$ in a latent space shared by all tasks. It is a bisimulation-style metric that measures behavioral similarity of task beliefs through one-step predictions of reward, transition, and inverse dynamics, with the 2-Wasserstein distance chosen for its closed-form solution. This metric serves as both the training objective for the belief similarity learner and the quantity the theorems claim bounds value and transfer differences between tasks. The companion mechanism is the Gaussian mixture $b = w_r \mathcal{N}(z_r | \mu_r, \sigma_r^2) + w_l \mathcal{N}(z_l | \mu_l, \sigma_l^2)$, which injects the latent shared-structure belief into the specific task belief so the policy sees global task structure without losing task-specific detail.

What would settle it

Take two tasks that share reward and inverse dynamics but differ only in a transition feature that affects rewards after many steps, with discount factor $\gamma = 0.99$; learn SimBelief's belief metric on them and compare the measured latent distance $d^\pi(z_i, z_j)$ with the actual long-horizon value difference $|V^\pi(s_i^+) - V^\pi(s_j^+)|$. If the value difference systematically exceeds the stated bound, the sufficiency of $z$ or the dropped discount factor is the failure point; a concrete instance would be two sparse-reach tasks whose goals are reached by routes of very different lengths with identical final rewards.

Watch

Extended reading notes

Core claim

The central claim is that task similarity in meta-RL is measurable in a shared latent space and that this measurement, not reward reconstruction, should drive task identification. Formally, the paper defines the latent task belief metric $d^\pi(z_i, z_j)$ over augmented states $s^+ = (g(s), z)$ as the sum of an absolute reward difference, a 2-Wasserstein transition difference, and an $\ell^1$ inverse-dynamics difference, and proves $|V^\pi(s_i^+) - V^\pi(s_j^+)| \leq d^\pi(z_i, z_j)$: similar latent beliefs imply similar value functions under the same policy. A second theorem bounds the suboptimality of applying a policy learned on task $M_i$ to task $M_j$ by $\epsilon_R + \gamma(\epsilon_T + \epsilon_I + \|z_i - z_j\|_1) R_{\max}/(2(1-\gamma))$. The agent learns the latent belief from a shared latent dynamics model and blends it with the specific task belief via a Gaussian mixture, so global task structure guides exploration while task-specific detail is preserved; the authors report that this yields superior online adaptation and out-of-distribution generalization compared to belief-based baselines.

Load-bearing premise

The load-bearing premise is that the latent task belief $z$ captures all task-relevant differences and that the discount factor can be set aside, so the measured similarity genuinely bounds value and transfer differences; if the belief misses task information, or if discounted long-horizon rewards matter, the bounds and the claimed adaptation advantage collapse.

Editorial extensions

If this is right

  • Task identification in meta-RL can be driven by learned dynamics similarity instead of reward reconstruction, so belief-based adaptation should keep working when rewards are sparse or absent.
  • Tasks with close latent beliefs share a policy with bounded value loss, so a success on one task propagates to similar unseen tasks within one or two adaptation episodes.
  • Out-of-distribution tasks that land near the training distribution in latent belief space inherit the learned similarity structure, which supports the reported OOD generalization.
  • Because the latent belief metric is trained jointly with the policy through SAC, adaptation speed and exploration efficiency improve together rather than being optimized separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dropped discount factor is the hidden hinge: reinserting $\gamma$ into the metric, or testing long-horizon tasks, would likely require discount-weighting the transition and inverse-dynamics terms, and a natural testable extension is to check whether the value bound survives when distant-future rewards dominate.
  • Because the transfer bound is linear in $\|z_i - z_j\|_1$, deliberately shaping the belief space with contrastive or triplet losses could make transfer bounds provably smaller for desired groups of tasks, a use the paper does not explore.
  • The latent dynamics $p_\theta((s'^+, r, a) | z_l)$ is effectively a task-conditioned world model, so the belief distance could double as an exploration bonus: an agent could be rewarded for visiting states that reduce uncertainty about which latent belief matches the current task.
  • If the metric generalizes as claimed, the practical consequence is that similarity of latent dynamics, not reward density, is the right currency for transferring skills between tasks in robotics, matching the paper's opening intuition that pulling open a drawer and pulling open a window should transfer through shared structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SimBelief, a context-based meta-RL method within a Bayes-Adaptive MDP framework. It learns a latent task belief metric d^pi that compares reward, transition, and inverse-dynamics models in a learned latent space, then combines the resulting latent belief with a VAE-based specific-task belief via a Gaussian mixture and trains the policy with SAC. The authors claim state-of-the-art performance on sparse-reward MuJoCo and panda-gym benchmarks, improved out-of-distribution adaptation, and theoretical guarantees given by Theorem 1 (value difference bound) and Theorem 2 (latent transfer bound).

Significance. If the empirical results are reproducible, the paper addresses a relevant problem—sample-efficient adaptation in sparse-reward meta-RL—and the latent-dynamics formulation is a reasonable and potentially useful design. The paper includes ablations of the latent belief and inverse dynamics, OOD evaluations, t-SNE visualizations, and a public code link, all of which are strengths. However, the theoretical claims are not currently supported by the proofs, and the empirical evidence is too sparse (three seeds, no error bars) to substantiate the 'outperforms state-of-the-art' claim. The contribution is therefore conditional on both a repaired theoretical argument and stronger statistical reporting.

major comments (5)
  1. [Appendix B, Theorem 1 proof (Eq. 16)] The induction step bounds the transition contribution by gamma*W_2(...) plus an inverse-dynamics term and then asserts this equals d^pi(z_i,z_j), relying on 'we disregard the effect of the discount factor gamma'. In a discounted BAMDP this is not a harmless assumption: the fixed-point operator in Lemma 1 includes gamma on the transition term (as in Definition 1), and dropping it changes the metric. The proof also invokes 'the Lipschitz property of the value function' without stating the Lipschitz constant or proving that V^pi is Lipschitz with respect to d^pi. As written, Theorem 1 is unproven.
  2. [Definition 2 and Eq. (5)] The latent task belief metric d^pi is defined with an inverse-dynamics term, but the proof of Theorem 1 introduces that term only inside the transition bound, without any derivation. More importantly, Eq. (5) optimizes psi_l so that ||psi_l(h_i) - psi_l(h_j)||_1 matches exactly the reward/transition/inverse-dynamics differences appearing in d^pi. Theorem 2 is then expressed in terms of ||z_i - z_j||_1, the same quantity that was trained to equal those dynamics differences. This makes the transfer bound tautological rather than an independent generalization guarantee.
  3. [Theorem 2 proof, Eq. (21)] Eq. (21) asserts that the combined reward, transition, and inverse-dynamics difference is bounded by epsilon_R + epsilon_T + epsilon_I + ||z_i - z_j||_1. This is precisely the kind of transfer statement the theorem is supposed to prove, and the proof does not derive it from the metric properties established earlier. The final bound also contains an unexplained factor R_max/(2(1-gamma)), and the intermediate step involving ||V*_Mj - (R_max/(2(1-gamma)))1||_infty is not justified. Consequently, Theorem 2 does not provide a formal transfer guarantee as stated.
  4. [Appendix B, Lemma 1] The fixed-point argument only shows monotonicity and continuity of F; it does not address why W_2(d^pi) is a well-defined operator on the intended space of metrics (e.g., compactness or measurability conditions), nor does it show that the metric realized by the learned psi_l corresponds to the least fixed point. This matters because Eq. (3) uses d^pi inside W_2(d^pi), so the metric's existence and the fixed-point property are load-bearing for Theorem 1.
  5. [Section 4, Figures 3-4] The main performance claim is based on three random seeds, and no error bars, standard deviations, or per-seed results are reported anywhere in the main text or appendices. With six environments and five baselines, this is insufficient to establish that SimBelief 'consistently performed well' or 'outperforms' the baselines. Please report mean plus/minus standard deviation (or individual seeds) and, where possible, statistical significance tests.
minor comments (5)
  1. [Appendix A and Appendix G] There are several typos: 'Defination' should be 'Definition', 'WIEGHTS' should be 'WEIGHTS', and the figure captions in G.1/G.2 read 'SimBlief' instead of 'SimBelief'.
  2. [Main text and Appendix B] Equation numbering is inconsistent: Theorem 1 is Eq. (10) in the main text but Eq. (16) in Appendix B, and Theorem 2 is Eq. (11) in the main text but Eq. (17) in Appendix B. Please unify the numbering.
  3. [Footnote 2 and Section 3.2] The notation z_i, z_j, and z_l is used interchangeably without a precise definition; this makes Eq. (5) and the policy input in Eq. (9) difficult to parse. Please define each symbol at first use and use one consistent convention throughout.
  4. [Definition 2 and Appendix B] The transition model is written as T^pi_i(s^+_i, a_i) in Definition 2 but as T^pi_i(s'^+_i | s^+_i, a_i) in the proof of Theorem 1. Please use a consistent conditional-notation convention for transitions and inverse dynamics.
  5. [Appendix E] The code link (https://github.com/mlzhang-pr/SimBelief) could not be verified from the manuscript alone; please confirm the repository is public and include a reproducibility statement with software versions and random-seed handling.

Circularity Check

2 steps flagged · score 8.0 of 10

The value-difference and transfer bounds in SimBelief reduce to the definition of the latent task belief metric and to the belief-similarity training loss, respectively.

  1. self definitional [Definition 2 (Eq. 3) and Theorem 1 proof, Appendix B (Eq. 16)]
    "By the definition of the latent task belief metric d π (z i ,z j ), we know: d π (z i ,z j )=|R π i (s + i ,a i )−R π j (s + j ,a j )|+W 2 (T π i (s + i ,a i ),T π j (s + j ,a j ))+||I π i (s + i ,s′ + i )−I π j (s + j ,s′ + j )|| 1 . We assume that z contains sufficient information about the task differences at the current time step, and we disregard the effect of the discount factor γ during the experiments."

    The theorem's RHS is exactly Definition 2 with γ removed. The proof's upper bound is |R_i−R_j|+γW_2(T_i,T_j)+||I_i−I_j||_1; the inverse-dynamics term appears in the 'Lipschitz' step without derivation, and γ is then dropped to make the bound equal d^π. Thus Eq. 16 does not derive a new value-difference result: it states that the V-difference is bounded by the quantity that Definition 2 named as the metric, under an explicit ignore-γ assumption. The central claim that similar latent task beliefs z_i,z_j imply similar value functions is therefore built into the definition and the gamma-dropping assumption rather than established independently.

  2. fitted input called prediction [Eq. 5 (belief similarity objective) and Theorem 2 proof, Eq. 21]
    "L bisim (ψ l )=∥ψ l (h i )−ψ l (h j )∥ 1 −|R̂(s + i ,a i )−R̂(s + j ,a j )|−W 2 (T̂(·|s + i ,a i ),T̂(·|s + j ,a j ))−||Î(·|s + i ,s′ + i )−Î(·|s + j ,s′ + j )|| 1 . ... ||T M i (s + i ,a i )−T M j (s + j ,a j )|| 1 +|R M i (s + i ,a i )−R M j (s + j ,a j )|+||I M i (s + i ,s′ + i )−I M j (s + j ,s′ + j )|| 1 ≤ε R +ε T +ε I +∥z i −z j ∥ 1 ."

    Eq. 5 is the training objective for ψ_l: it drives the belief distance ∥ψ_l(h_i)−ψ_l(h_j)∥_1 to match exactly the reward, transition-W2, and inverse-dynamics differences, with dynamics gradients stopped. In Theorem 2's proof, Eq. 21 asserts that those same three differences are bounded by ε_R+ε_T+ε_I+∥z_i−z_j∥_1, where z_i,z_j are the learned latent beliefs. This is the zero-loss/converged form of Eq. 5 with approximation errors renamed. The transfer bound is therefore a restatement of what the network was fitted to do, not an independent theorem about transfer: the 'prediction' is forced by the training loss.

full rationale

The empirical comparisons against PEARL, MetaCURE, VariBAD, HyperX, and RL2 are external benchmarks and are not circular; I found no load-bearing self-citation chain. However, the theoretical validation claimed as a contribution in Section 3.4 is circular. Theorem 1's proof equates its upper bound with Definition 2 after adding an inverse-dynamics term and discarding the discount factor, so the value-difference bound is the metric definition under an assumption. Theorem 2's bound is obtained from Eq. 21, which is the converged form of the Eq. 5 fitting loss; the theorem thus 'predicts' a bound that was enforced during training. These reductions affect the paper's central claim that latent task belief similarity theoretically guarantees similar value and transfer behavior. Score 8: the theoretical result is forced by definition and by the fitted objective, although the experimental results retain independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 3 invented entities

The central claim rests on unproven structural assumptions: the latent belief z is a sufficient statistic for task-relevant dynamics, the discount-free metric is a valid bisimulation metric, and the learned dynamics models approximate the truth well enough. It also depends on hand-tuned mixture and KL weights, a per-environment embedding size, and a Q-fit offset. No independently falsifiable entity is introduced.

free parameters (4)
  • Mixture weights w_r and w_l (Eq. 7) = [0.5, 0.5] chosen after ablation
    Hand-selected in Appendix G.3 from an ablation on Cheetah-Vel-Sparse; these weights control how much the latent task belief contributes to the policy input, so the central method depends on this choice.
  • KL weight in belief similarity objective (Eq. 8) = 1.0 (Cheetah), 0.1 (others)
    Hand-tuned per environment in Table 2; balances alignment between latent and specific beliefs against the similarity loss.
  • Task embedding size (dimension of z) = 10 (MuJoCo), 5 (panda)
    Chosen per environment in Table 2; sets the capacity of the belief space Z that carries task similarity information.
  • Distribution offset (Delta_mu, Delta_sigma) = Learned, no fixed value
    Trained with the Q-function loss (Eq. 9, Appendix G.4) to align the latent belief with the real environment; it is a fitted part of the policy input.
assumptions (4)
  • ad hoc to paper The latent task belief z_l is a sufficient statistic for task-relevant dynamics and value differences.
    Invoked in the proof of Theorem 1 ('We assume that z contains sufficient information about the task differences at the current time step') and needed for Theorems 1-2 to connect belief distance to value and transfer error.
  • ad hoc to paper The discount factor gamma can be omitted from the latent task belief metric without invalidating the value difference bound.
    The metric in Eq. 3 has no discount factor, while standard bisimulation metrics require it for contraction; the proof of Theorem 1 explicitly disregards gamma 'during the experiments'.
  • domain assumption The learned reward, transition, and inverse dynamics models approximate the true task dynamics well enough that the metric d_pi is a meaningful bisimulation metric.
    Lemma 1 and Theorems 1-2 assume the learned models and the latent space abstraction satisfy the bisimulation conditions; the approximation errors epsilon_R, epsilon_T, epsilon_I are never measured or bounded empirically.
  • ad hoc to paper The inverse dynamics model I is well-defined and its inclusion in the metric preserves the Lipschitz property needed in Theorem 1.
    The inverse dynamics term appears in Definition 2 and Eq. 5, but the proof of Theorem 1 introduces it without derivation.
invented entities (3)
  • latent task belief metric d_pi
    purpose: Measures similarity between two task beliefs in latent space; used as the target for the belief similarity loss (Eq. 5) and in Theorems 1-2.
    An internal construct defined from the paper's learned dynamics models; it makes no falsifiable prediction outside the paper.
  • latent task belief b_l in latent space Z
    purpose: Captures shared structure across tasks and is combined with the specific task belief b_r as a Gaussian mixture to condition the policy.
    No independent handle; validated only through downstream task performance and t-SNE visualizations.
  • specific task belief b_r (VAE posterior)
    purpose: Represents the current task from trajectory history; standard VariBAD-style component kept for fine-grained task identification.
    Not a new physical entity; included for completeness because it is part of the combined belief input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning." pith.science (2026). https://pith.science/paper/LF6QH5KY

@misc{pith2026250619785,
  author       = {Pith},
  title        = {Pith review of: Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LF6QH5KY}},
  note         = {Machine review of arXiv:2506.19785}
}
read the original abstract

Meta-reinforcement learning requires utilizing prior task distribution information obtained during exploration to rapidly adapt to unknown tasks. The efficiency of an agent's exploration hinges on accurately identifying the current task. Recent Bayes-Adaptive Deep RL approaches often rely on reconstructing the environment's reward signal, which is challenging in sparse reward settings, leading to suboptimal exploitation. Inspired by bisimulation metrics, which robustly extracts behavioral similarity in continuous MDPs, we propose SimBelief-a novel meta-RL framework via measuring similarity of task belief in Bayes-Adaptive MDP (BAMDP). SimBelief effectively extracts common features of similar task distributions, enabling efficient task identification and exploration in sparse reward environments. We introduce latent task belief metric to learn the common structure of similar tasks and incorporate it into the specific task belief. By learning the latent dynamics across task distributions, we connect shared latent task belief features with specific task features, facilitating rapid task identification and adaptation. Our method outperforms state-of-the-art baselines on sparse reward MuJoCo and panda-gym tasks.

Figures

Figures reproduced from arXiv: 2506.19785 by the authors.

Figure 1
Figure 1. Learning shared structures among tasks in the latent space as task beliefs enables the agent to rapidly adapt to new tasks. In meta-reinforcement learning, an agent is required to efficiently explore the environment to gather information relevant to the current task and use that information to adapt to new, unseen tasks (Duan et al., 2016; Finn et al., 2017; Gupta et al., 2018b; Humplik et al., 2019). However, in re… view at source ↗
Figure 2
Figure 2. SimBelief architecture. Our framework consists of three components: learning latent task [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Meta-testing performance on sparse reward MuJoCo and panda-gym tasks over 3 random [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Average test performance for the first 5 rollouts on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Exploration and adaptation performance in the 5 random [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: t-SNE visualization of task beliefs learned by the algorithm on 10 randomly sampled [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Panda-gym tasks. The Panda-Reach task involves controlling the Panda robotic arm to [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Visualization of different task beliefs during the training phase. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: The correlation matrix of task beliefs across different tasks. Specific task belief focuses [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Ablation study on SimBlief’s latent task belief. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Ablation study on SimBlief’s inverse dynamic module in latent space. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: The performance of different Gaussian mixture weights on the Cheetah-Vel-Sparse task. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Ablation study on offset (∆µ, ∆σ). these, [wr, wl ] = [0.5, 0.5] achieved higher exploration efficiency during training and demonstrated stronger adaptability to OOD tasks ( [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: The task belief representation of all experimental algorithms for 10 random in [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Average test performance for the first 5 rollouts on randomly generated tasks of Panda [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Exploration and adaptation performance in the 5 random in-distribution (radius = 1.0) [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 26 canonical work pages

  1. [1]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    arXiv 2018

  3. [3]

    Acting optimally in partially observable stochastic domains

    Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. Acting optimally in partially observable stochastic domains. In Aaai, volume 94, pp.\ 1023--1028, 1994

    work page 1994

  4. [4]

    Scalable methods for computing state similarity in deterministic markov decision processes

    Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.\ 10069--10076, 2020

    work page 2020

  5. [5]

    Contrabar: Contrastive bayes-adaptive deep rl

    Era Choshen and Aviv Tamar. Contrabar: Contrastive bayes-adaptive deep rl. In International Conference on Machine Learning, pp.\ 6005--6027. PMLR, 2023

    work page 2023

  6. [6]

    Offline meta reinforcement learning--identifiability challenges and effective data collection strategies

    Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning--identifiability challenges and effective data collection strategies. Advances in Neural Information Processing Systems, 34: 0 4607--4618, 2021

    work page 2021

  7. [7]

    Provably efficient rl with rich observations via latent state decoding

    Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pp.\ 1665--1674. PMLR, 2019

    2019

  8. [8]

    Rl2: Fast reinforcement learning via slow reinforcement learning

    Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016

    arXiv 2016

Show all 60 references
  1. [9]

    Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes

    Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. University of Massachusetts Amherst, 2002

    2002

  2. [10]

    Challenges of real-world reinforcement learning

    Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901, 2019

    1904 arXiv

  3. [11]

    Metrics for finite markov decision processes

    Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume 4, pp.\ 162--169, 2004

    2004

  4. [12]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.\ 1126--1135. PMLR, 2017

    2017

  5. [13]

    Meta learning shared hierarchies

    Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018

    2018

  6. [14]

    panda-gym: Open-source goal-conditioned environments for robotic learning

    Quentin Gallou \'e dec, Nicolas Cazin, Emmanuel Dellandr \'e a, and Liming Chen. panda-gym: Open-source goal-conditioned environments for robotic learning. In 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning@ NeurIPS 2021, 2021

    2021

  7. [15]

    Deepmdp: Learning continuous latent space models for representation learning

    Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International conference on machine learning, pp.\ 2170--2179. PMLR, 2019

    2019

  8. [16]

    Bayesian reinforcement learning: A survey

    Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning , 8 0 (5-6): 0 359--483, 2015

    2015

  9. [17]

    Equivalence notions and model minimization in markov decision processes

    Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial intelligence, 147 0 (1-2): 0 163--223, 2003

    2003

  10. [18]

    Learning action translator for meta reinforcement learning on sparse-reward tasks

    Yijie Guo, Qiucheng Wu, and Honglak Lee. Learning action translator for meta reinforcement learning on sparse-reward tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.\ 6792--6800, 2022

    2022

  11. [19]

    Unsupervised meta-learning for reinforcement learning

    Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv:1806.04640, 2018 a

    2018 arXiv

  12. [20]

    Meta-reinforcement learning of structured exploration strategies

    Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018 b

    2018

  13. [21]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  14. [22]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019

    1912 arXiv

  15. [23]

    Bisimulation makes analogies in goal-conditioned reinforcement learning

    Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in goal-conditioned reinforcement learning. In International Conference on Machine Learning, pp.\ 8407--8426. PMLR, 2022

    2022

  16. [24]

    Learning an embedding space for transferable robot skills

    Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018

    2018

  17. [25]

    Meta reinforcement learning as task inference

    Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06424, 2019

    1905 arXiv

  18. [26]

    Notes on state abstractions, 2018

    Nan Jiang. Notes on state abstractions, 2018

    2018

  19. [27]

    Towards robust bisimulation metric learning

    Mete Kemertas and Tristan Aumentado-Armstrong. Towards robust bisimulation metric learning. Advances in Neural Information Processing Systems, 34: 0 4764--4777, 2021

    2021

  20. [28]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  21. [29]

    Meta reinforcement learning with task embedding and shared policy

    Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta reinforcement learning with task embedding and shared policy. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp.\ 2794--2800, 2019

    2019

  22. [30]

    Parameterizing non-parametric meta-reinforcement learning tasks via subtask decomposition

    Suyoung Lee, Myungsik Cho, and Youngchul Sung. Parameterizing non-parametric meta-reinforcement learning tasks via subtask decomposition. Advances in Neural Information Processing Systems, 36: 0 43356--43383, 2023

    2023

  23. [31]

    Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices

    Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International conference on machine learning, pp.\ 6925--6935. PMLR, 2021

    2021

  24. [32]

    Behavior from the void: Unsupervised active pre-training

    Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 0 18459--18473, 2021

    2021

  25. [33]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp.\ 278--287, 1999

    1999

  26. [34]

    (more) efficient reinforcement learning via posterior sampling

    Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013

    2013

  27. [35]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  28. [36]

    Ride: Rewarding impact-driven exploration for procedurally-generated environments

    Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-generated environments. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkg-TJBFPB

    2020

  29. [37]

    Efficient off-policy meta-reinforcement learning via probabilistic context variables

    Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pp.\ 5331--5340. PMLR, 2019

    2019

  30. [38]

    Residual skill policies: Learning an adaptable skill-based action space for reinforcement learning for robotics

    Krishan Rana, Ming Xu, Brendan Tidd, Michael Milford, and Niko S \"u nderhauf. Residual skill policies: Learning an adaptable skill-based action space for reinforcement learning for robotics. In Conference on Robot Learning, pp.\ 2095--2104. PMLR, 2023

    2023

  31. [39]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pp.\ 8583--8592. PMLR, 2020

    2020

  32. [40]

    Multi-task reinforcement learning with context-based representations

    Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pp.\ 9767--9779. PMLR, 2021

    2021

  33. [41]

    Reinforcement learning: An introduction

    Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018

    2018

  34. [42]

    Distral: Robust multitask reinforcement learning

    Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  35. [43]

    On the likelihood that one unknown probability exceeds another in view of the evidence of two samples

    William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25 0 (3-4): 0 285--294, 1933

    1933

  36. [44]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 0 (11), 2008

    2008

  37. [45]

    Optimal transport: old and new, volume 338

    C \'e dric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009

    2009

  38. [46]

    Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards

    Shanchuan Wan, Yujin Tang, Yingtao Tian, and Tomoyuki Kaneko. Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp.\ 4289--4298, 2023

    2023

  39. [47]

    Latent skill planning for exploration and transfer

    Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. arXiv preprint arXiv:2011.13897, 2020

    2011 arXiv

  40. [48]

    Behavior contrastive learning for unsupervised skill discovery

    Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In International Conference on Machine Learning, pp.\ 39183--39204. PMLR, 2023

    2023

  41. [49]

    Improving sample efficiency in model-free reinforcement learning from images

    Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the aaai conference on artificial intelligence, pp.\ 10674--10681, 2021

    2021

  42. [50]

    Robust task representations for offline meta-reinforcement learning via contrastive learning

    Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning, pp.\ 25747--25759. PMLR, 2022

    2022

  43. [51]

    Learning invariant representations for reinforcement learning without reconstruction

    Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021 a

    2021

  44. [52]

    Learning robust state abstractions for hidden-parameter block mdps

    Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-parameter block mdps. In International Conference on Learning Representations, 2021 b

    2021

  45. [53]

    Metacure: Meta reinforcement learning with empowerment-driven exploration

    Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with empowerment-driven exploration. In International Conference on Machine Learning, pp.\ 12600--12610. PMLR, 2021 c

  46. [54]

    What can learned intrinsic rewards capture? In International Conference on Machine Learning, pp.\ 11436--11446

    Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pp.\ 11436--11446. PMLR, 2020

    2020

  47. [55]

    Varibad: A very good method for bayes-adaptive deep rl via meta-learning

    Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019

    1910 arXiv

  48. [56]

    Exploration in approximate hyper-state space for meta reinforcement learning

    Luisa M Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pp.\ 12991--13001. PMLR, 2021

    2021

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  50. [58]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  51. [59]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  52. [60]

    iӆVb ߛuѪZ̮' No` ԰r tVXo+Ra 4ON i]auד/Ut a l B j*f o v1 ZM1 P' i .B e L w`CN` t 翧FS&yDĖ ]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.