Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Epistemically-guided forward-backward exploration

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a zero-shot RL agent can drive its own exploration with the same forward-backward representation it is learning, by executing the policy whose reward embedding maximizes the ensemble variance of the projected value…

desk verdict A solid, worthwhile paper whose headline claim is undercut by a missing ensemble-size-controlled baseline; fix the ablation and it's a clean contribution. read the letter →

arxiv 2507.05477 v1 pith:JKVZYLLE submitted 2025-07-07 cs.LG cs.AI

Núria Armengol Urpí , Marin Vlastelica , Georg Martius , Stelian Coros This is my paper
classification cs.LGcs.AI
keywords zero-shotreinforcementlearningforward-backwardrepresentationsexplorationepistemicuncertaintydeepensemblessamplingsuccessormeasureunsupervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a zero-shot reinforcement learning agent should collect its own data using the very representation it is trying to learn, rather than relying on a separate exploration policy. Concretely, it targets forward-backward (FB) representations, which factorize the occupancy measure of all near-optimal policies into a forward map and a backward map. The authors maintain an ensemble of forward networks, hold the backward network fixed to avoid a rotational ambiguity, and use the projection of ensemble disagreement onto Q-values as a measure of epistemic uncertainty. They then explore by executing the policy whose reward embedding maximizes this projected variance. On five continuous-control domains, this uncertainty-guided collection improves sample complexity considerably relative to random reward-embedding sampling, random actions, and data from a decoupled random-network-distillation explorer.

What carries the argument

The load-bearing object is the projected predictive variance $\mathrm{Var}[Q^{\pi_z}(s,a) \mid \mathcal{D}] = \frac{1}{K} \sum_k \langle F_k(s,a,z) - \bar{F}(s,a,z), z\rangle^2$, an ensemble-disagreement estimate of the posterior uncertainty of the value function for policy $\pi_z$. To make this quantity meaningful, the backward network is held fixed and orthonormalized, eliminating the rotation non-uniqueness of the FB factorization that would otherwise contaminate ensemble disagreement. The exploration policy then selects the reward embedding $z$ maximizing this variance and rolls out the greedy policy for that $z$, a variant of uncertainty sampling.

What would settle it

On a fixed FB checkpoint, compute the correlation between the ensemble disagreement at a query state-action-embedding triple and the actual squared error of the predicted Q-value against a high-accuracy Monte Carlo estimate. If the disagreement does not positively correlate with the true error, or if collecting data at the highest-disagreement embedding does not reduce posterior variance more than collecting at a random embedding, the exploration signal is uninformative and the stated sample-complexity gains would not replicate.

Watch

Extended reading notes

Core claim

The central claim is that the predictive variance of the factorized Q-function, computed as ensemble disagreement in the forward representation projected onto a reward embedding, is a useful and sufficient signal for driving exploration in zero-shot RL. Executing policies with the highest such variance, an instance of uncertainty sampling over the reward-embedding space, reduces the number of environment steps needed to learn FB representations that generalize zero-shot. Because the FB representation encodes all optimal policies, reducing the posterior uncertainty of the projected value function is argued to reduce uncertainty of the successor measure across embeddings, and the paper demonstrates empirically that this holds across walker, cheetah, hopper, quadruped, and maze control tasks.

Load-bearing premise

Ensemble disagreement among the forward networks, with a fixed and orthonormalized backward network, is a faithful proxy for the epistemic uncertainty that matters: the reward embedding with the highest projected Q-variance is assumed to be the one whose data would most reduce the approximation error of the successor measure across all embeddings.

Editorial extensions

If this is right

  • Zero-shot RL agents can replace decoupled exploration with self-guided collection driven by the FB representation's own uncertainty signal.
  • The uncertainty-guided collector reaches the same zero-shot scores as standard FB with fewer environment steps across all five evaluated control domains.
  • Exploring over reward embeddings outperforms exploring over raw actions, indicating that the FB policy space carries more useful exploration structure than action-level noise.
  • More frequent re-optimization of the uncertain reward embedding within an episode improves sample efficiency, supporting the value of non-myopic uncertainty tracking.
  • The method matches or exceeds the asymptotic performance of standard FB, so the uncertainty signal does not sacrifice final policy quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same 'fix the non-identifiable factor, disagree on the rest' recipe could transfer to successor features or other factorized occupancy-measure methods, whose exploration also suffers from representation non-uniqueness.
  • Inference: the greedy per-embedding uncertainty sampling ignores correlation in uncertainty reduction across the embedding space; an information-theoretic criterion maximizing expected reduction in the full posterior could yield further gains.
  • Inference: because the paper finds F-uncertainty and Q-uncertainty are only weakly correlated, the projection choice matters; a testable extension is to weight disagreement by the expected information gain of the trajectory actually collected, rather than the instantaneous variance.
  • Inference: the same posterior variance could support UCB or Thompson-sampling style exploration within the embedding space, directions the paper names but does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes FBEE Q, an exploration strategy for forward-backward (FB) representations in zero-shot RL. It maintains an ensemble of K forward functions with a shared backward representation, defines the predictive variance of the Q-function Qπz = ⟨F(·,z), z⟩ as in Eq. (7), selects the reward embedding z maximizing that variance, and executes the corresponding greedy policy for data collection. The authors frame this as epistemic-uncertainty minimization in the spirit of uncertainty sampling, and report experiments on five DeepMind Control Suite domains showing improved zero-shot sample complexity relative to FB with uniformly sampled z, a random-action policy, and an RND-based offline baseline.

Significance. If the reported gains are attributable to the uncertainty-guided selection rule, the paper makes a useful contribution: it connects Bayesian experimental design to zero-shot RL, introduces a simple method with public code, and runs 10-seed evaluations across 15 tasks. The ablation set (FBEE F, FBEE Q-EPISODE, FBEE Q-POLICY) is a strength. However, the headline comparison is confounded by ensemble size, and no statistical testing is provided; the contribution is therefore promising but not yet established.

major comments (3)
  1. [Section 6 (Baselines) and Appendix C (Table 1)] The headline comparison between FBEE Q and the FB baseline varies two factors simultaneously: the ensemble size (K=5 in Table 1 for FBEE Q, versus the single-F 'original FB algorithm' described in Section 6) and the z-selection rule (Eq. (8) versus uniform random sampling). All reported ablations—FBEE Q-EPISODE, FBEE F, and FBEE Q-POLICY—retain K=5 and change only the selection objective, its update frequency, or the learned policy. None is a K=5 ensemble that samples z uniformly at random, so the observed sample-complexity gains cannot be attributed to the epistemic-uncertainty signal in Eq. (8) rather than to ensemble averaging, extra capacity, or stabilization of the shared B target. Please state the FB baseline's ensemble size explicitly and add at least a K=5 uniform-z control; a 2x2 (K, selection rule) design would cleanly separate the factors.
  2. [Section 5, Eq. (8); Section 7] The abstract claims that the exploration policy minimizes posterior variance, hence epistemic uncertainty. The paper does not verify that executing πE actually reduces Eq. (7) (or Eq. (6)) for the selected z or across z. The 'information never hurts' argument concerns conditional entropy and does not justify the greedy one-step variance proxy, and the Conclusion acknowledges that correlated uncertainty across z is not handled. Please include a diagnostic (e.g., ensemble disagreement as a function of environment steps for FBEE Q versus K=5 uniform-z), or soften the causal/variance-reduction wording to a heuristic claim. Without such evidence, the 'epistemically-guided' framing is not empirically supported.
  3. [Section 6 (Results); Abstract] The text repeatedly uses 'significant' to describe sample-efficiency gains (e.g., 'significant sample efficiency gains' in Section 6 and 'improve sample complexity ... considerably' in the Abstract), but no statistical significance tests or confidence intervals are reported; the shaded regions in Fig. 4 overlap substantially for several domains. Please report per-seed curves or bootstrap CIs/paired tests across the 10 seeds, and qualify the claims accordingly.
minor comments (7)
  1. [Eq. (6)] The covariance formula should read (F_k(x) − \bar{F}(x))(F_k(x) − \bar{F}(x))^T; the current typesetting is missing a closing parenthesis and is ambiguous.
  2. [Eq. (7)] The index in Eq. (7) runs from i=0 to K; it should run over the K ensemble members, e.g., k=0,…,K−1.
  3. [Eq. (3)] The sum bounds 0≤i,j<b−1 appear off by one; with b sampled transitions the sums should include j=b−1 (or use ≤ b−1).
  4. [Algorithm 1] Algorithm 1 does not specify how often πE is recomputed; the text says every 100 interaction steps (Section 6), so this frequency should appear in the pseudocode and in captions of Fig. 4.
  5. [Eq. (8)] Eq. (8) is written as an argmax over πz but returns an embedding z; please write it as selecting z_E and then setting πE=π_{z_E}.
  6. [Section 6, Baselines] The RANDOM baseline is not described beyond 'random exploration over the action space'; specify the action sampling distribution and whether it uses the same replay buffer/updates as other methods.
  7. [Appendix B] Cheetah is misspelled 'Chetah' in the appendix feature-map list.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the exploration policy is an uncertainty-sampling objective evaluated against external baselines, with no load-bearing self-citation or by-construction equivalence.

full rationale

The derivation chain is explicit and non-circular: Eq. (6) defines the ensemble covariance of the F representations, Eq. (7) projects that covariance onto z to obtain the predictive variance of Q^{pi_z}, and Eq. (8) selects the policy whose Q-variance is maximal. Each step is either an algebraic identity (Q = <F,z> makes Eq. (7) a direct projection of Eq. (6)) or a stated active-learning heuristic (uncertainty sampling, credited to Lewis & Gale, 1994). The paper does not fit a parameter and then rename that fit as a prediction; the exploration objective is used online to collect data, and the reported sample-complexity gains are measured against external baselines: FB with uniformly sampled z, a RANDOM action policy, and FB-RND with pre-collected data. These baselines are not constructed from Eq. (8), so the outcome is not forced by the method's definition. The self-citations in the paper (Vlastelica et al. 2021, Sancaktar et al. 2022, Vlastelica et al. 2024, Bagatella et al. 2024, Kolev et al. 2025, Sukhija et al. 2024) appear only as related-work context and are not load-bearing; no core theorem, uniqueness claim, or central premise is imported from the authors' own prior work. The skeptic's K=5-versus-K=1 concern is a potentially serious experimental-design confound if the FB baseline truly used a single ensemble member while FBEE used five, but that is a benchmark-ablation issue, not a circular reduction: the equations do not make the reported improvement equal to the algorithm's inputs. The paper also openly acknowledges its main gap, namely that it does not account for correlated information across z, and it does not claim a rigorous minimization of posterior variance, only an uncertainty-sampling approximation. No circular step meets the required quote-and-reduce standard; the central claim has independent empirical content.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The empirical results depend on hand-chosen hyperparameters (representation dimension, ensemble size, z-mix ratio, z-update frequency, B-regularization strength), a per-environment hand-specified feature map, and several assumptions about the reliability of ensemble disagreement as a proxy for posterior uncertainty. No new entities are introduced.

free parameters (6)
  • Z-dimension d = 50 (100 for maze)
    Representation dimension chosen by hand per environment; the factorized approximation's capacity depends on it (Table 1).
  • Ensemble size K = 5
    Number of forward-function ensemble members; the uncertainty estimate in Eq (7) is an average over K members. Chosen by hand, no sensitivity analysis reported.
  • Mix ratio for z sampling = 0.3
    Ratio governing how often exploration z's versus random z's are used in training the FB loss; affects what the ensemble is trained on (Table 1).
  • Frequency of z updates = 0.01 (every 100 steps)
    How often the exploration embedding zE is re-optimized; ablated in Section 6 and shown to affect results.
  • Orthonormality regularization strength on B = not specified
    Regularization on B added 'as per Touati & Ollivier' but its coefficient is not listed in the hyperparameter table; the uncertainty estimate depends on B staying normalized.
  • Environment-specific feature map phi = hand-specified per domain
    B is trained on a task-relevant subset of the state (Appendix B); this injects prior knowledge about which rewards matter and likely influences the observed gains.
assumptions (6)
  • domain assumption The FB factorization Mπz(s,a,s',a') ≈ <F(s,a,z), B(s',a')> holds with learned F and B.
    Invoked in Eq (2) and throughout Section 3; the zero-shot guarantee (Touati & Ollivier 2021, Theorem 2) is inherited from this assumption.
  • ad hoc to paper The ensemble of K forward functions with fixed B approximates the posterior distribution p(Θ|D).
    Section 4, Eqs (5)-(6): the ensemble is a naive posterior approximation, not a Bayesian posterior; its validity is assumed.
  • ad hoc to paper Ensemble disagreement (predictive variance in Eq (7)) is a monotonically informative measure of epistemic uncertainty for exploration.
    The paper states uncertainty sampling 'provably minimizes posterior uncertainty under a homoscedastic, independent Gaussian noise model' but gives no proof or reference for the FB setting.
  • ad hoc to paper Reducing uncertainty at the greedily selected z transfers to reduced uncertainty for other z (information never hurts).
    Section 5, paragraph after Eq (8): the transfer is motivated by entropy monotonicity but not proven; the paper acknowledges the objective is greedy.
  • domain assumption The orthonormality regularization on B is sufficient to make the F-posterior well-defined despite non-uniqueness.
    Section 4: fixing B and regularizing its covariance is assumed to remove the rotation ambiguity that would make ensemble disagreement meaningless.
  • domain assumption The Monte Carlo estimate zR = E_{(s,a)∼D}[r(s,a)B(s,a)] over the replay buffer is a sufficient approximation of the optimal z for the test reward.
    Used at evaluation in Section 6; the accuracy of this estimate is not analyzed and could affect zero-shot scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Epistemically-guided forward-backward exploration." pith.science (2026). https://pith.science/paper/JKVZYLLE

@misc{pith2026250705477,
  author       = {Pith},
  title        = {Pith review of: Epistemically-guided forward-backward exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKVZYLLE}},
  note         = {Machine review of arXiv:2507.05477}
}
read the original abstract

Zero-shot reinforcement learning is necessary for extracting optimal policies in absence of concrete rewards for fast adaptation to future problem settings. Forward-backward representations (FB) have emerged as a promising method for learning optimal policies in absence of rewards via a factorization of the policy occupancy measure. However, up until now, FB and many similar zero-shot reinforcement learning algorithms have been decoupled from the exploration problem, generally relying on other exploration algorithms for data collection. We argue that FB representations should fundamentally be used for exploration in order to learn more efficiently. With this goal in mind, we design exploration policies that arise naturally from the FB representation that minimize the posterior variance of the FB representation, hence minimizing its epistemic uncertainty. We empirically demonstrate that such principled exploration strategies improve sample complexity of the FB algorithm considerably in comparison to other exploration methods. Code is publicly available at https://sites.google.com/view/fbee-url.

Figures

Figures reproduced from arXiv: 2507.05477 by the authors.

Figure 1
Figure 1. We condition an exploration policy on a reward embedding [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Epistemically guided FB exploration (FBEEQ). During exploration we uniformly sample reward embeddings from a hypersphere (left), and take samples over our posterior distribution F as represented by the K ensemble members F1:K (K = 4 in the figure) (middle-left). Then we project our F-posterior to a Q-posterior via Qπz = ⟨F(s, πz(s), z), z⟩ (middle-right) and compute the Q-predictive uncertainty for all sampled z’s (… view at source ↗
Figure 3
Figure 3. Environments used in our experiments. (Left to right): [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Zero-shot scores averaged over different downstream task as number of environment [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Regression scatter plot of the trace of CoVar[F πz | s, a, z] and Var[Qπz | s, a, z] for a FB checkpoint in Maze experiment. A second hypothesis would be that our exploration strategy π E is not guaranteed to pick the z E that maximizes the cumulative posterior uncerta…
Figure 6
Figure 6. Figure 6: Scores comparison when using F-uncertainty versus Q-uncertainty exploration. Zero-shot scores averaged over different downstream task as number of environment samples increases. Metrics are averaged over 30 evaluation episodes and 10 independent random seeds. Shaded ar…
Figure 7
Figure 7. Figure 7: Zero-shot scores for different downstream task as number of environment samples increases. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Zero-shot scores comparison when using F-uncertainty versus Q-uncertainty exploration for different downstream task as number of environment samples increases. Metrics are averaged over 30 evaluation episodes and 10 independent random seeds. Shaded area is 1-standard d…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation

    cs.RO 2026-02 conditional novelty 7.0 of 10

    Guiding goal-conditioned reinforcement learning with samples from a constrained feasible-state manifold lets a simulated double-sphere and a Panda-arm policy succeed far more often than RL with random resets.

Reference graph

Works this paper leans on

26 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [7]

    In general we observe that it performs in par with FBEE Q-SAMPLING , and we attribute the mismatches in performance to not extensive hyperparameter finetuning. Reinforcement Learning Journal 2025 0 2 4 6 8 10 Environment steps×105 0 200 400 600 800 1000Task reward walker stand 0 2 4 6 8 10 Environment steps×105 0 200 400 600 800 1000Task reward walker wal...

    work page 2025

  2. [9]

    Urlb: Unsupervised reinforcement learning benchmark

    Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2110.15191,

    arXiv

  3. [10]

    Cic: Contrastive intrinsic control for unsupervised skill discovery

    Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161,

    arXiv

  4. [11]

    Efficient exploration via state marginal matching

    Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274,

    arXiv 1906

  5. [14]

    Radford M Neal

    URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/f83630579d055dc5843ae693e7cdafe0-Paper.pdf. Radford M Neal. Bayesian learning for neural networks, volume

    work page 2019

  6. [17]

    arXiv:2310.08887

    URL http://arxiv.org/abs/2310.08887. arXiv:2310.08887. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778–2787. PMLR,

    arXiv

  7. [18]

    arXiv:2406.06615 [cs]

    URL http://arxiv.org/abs/2406.06615. arXiv:2406.06615 [cs]. Cansu Sancaktar, Sebastian Blaes, and Georg Martius. Curious exploration via structured world models yields zero-shot object manipulation. Advances in Neural Information Processing Systems, 35:24170–24183,

    arXiv

  8. [20]

    arXiv:2107.14226 [cs]

    URL http://arxiv.org/abs/2107.14226. arXiv:2107.14226 [cs]. Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause. Optimistic Active Exploration of Dynamical Systems, October

    arXiv

Show all 26 references
  1. [21]

    arXiv:2306.12371 [cs, eess]

    URL http: //arxiv.org/abs/2306.12371. arXiv:2306.12371 [cs, eess]. Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Maxinforl: Boosting exploration in reinforcement learning through information gain maximization. arXiv preprint arXiv:2412.12098,

    arXiv

  2. [22]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,

    arXiv

  3. [24]

    Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209.14935,

    Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? arXiv preprint arXiv:2209.14935,

    arXiv

  4. [25]

    The laplacian in rl: Learning representations with efficient approximations

    Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv:1810.04586,

    arXiv

  5. [1933]

    Zero-shot whole-body humanoid control via behavioral foundation models

    Reinforcement Learning Journal 2025 Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behavioral foundation models. In The Thirteenth International ...

    2025

  6. [1992]

    DOI: 10.1162/neco.1992.4.4.590

    ISSN 0899-7667. DOI: 10.1162/neco.1992.4.4.590. URL https://doi.org/10.1162/neco.1992.4.4.590. David John Cameron Mackay. Bayesian methods for adaptive models . California Institute of Technology,

    1992 doi

  7. [1993]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018a. Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Le...

    arXiv

  8. [1994]

    ISBN 978-1-4471-2099-5

    Springer London. ISBN 978-1-4471-2099-5. Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736–6747. PMLR,

  9. [1995]

    URL https://doi.org/10

    DOI: 10.1214/ss/1177009939. URL https://doi.org/10. 1214/ss/1177009939. Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502,

  10. [2014]

    Fast task inference with variational intrinsic successor features.arXiv preprint arXiv:1906.05030,

    Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and V olodymyr Mnih. Fast task inference with variational intrinsic successor features.arXiv preprint arXiv:1906.05030,

    1906 arXiv

  11. [2015]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894,

    arXiv

  12. [2018]

    • Point-mass Maze: a 2-dimensional continuous maze with four rooms

    and some adapted by (Touati et al., 2022). • Point-mass Maze: a 2-dimensional continuous maze with four rooms. The states are 4-dimensional vectors encoding for positions and velocities of the point mass, and the actions are 2-dimensional vectors. Importantly, the initial posi...

    2022

  13. [2019]

    Active fine-tuning of generalist policies

    Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of generalist policies. arXiv preprint arXiv:2410.05026,

    arXiv

  14. [2020]

    URL http://arxiv.org/abs/1907. 01657. arXiv:1907.01657. D. J. Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration, May

    1907 arXiv

  15. [2021]

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra

    URL https://arxiv.org/ abs/2101.07123. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613–1622. PMLR,

    arXiv

  16. [2022]

    Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel

    URL https://arxiv.org/abs/2202.00914. Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsupervised skill discovery,

    arXiv

  17. [2023]

    Seohong Park, Oleh Rybkin, and Sergey Levine

    URL https://arxiv.org/abs/2302.05103. Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction, March

    arXiv

  18. [2024]

    Zero-shot reinforcement learning from low quality data

    Reinforcement Learning Journal 2025 Scott Jeen, Tom Bewley, and Jonathan M Cullen. Zero-shot reinforcement learning from low quality data. arXiv preprint arXiv:2309.15178,

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.