Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

Jiajun Hu; Jin Cheng; Nuria Armengol Urpi; Stelian Coros

REVIEW 5 major objections 4 minor 2 references

FB-MEBE claims that an online zero-shot RL algorithm for quadrupeds can explore by seeking out rare, low-density behaviors, learn a diverse base of policies without task rewards, and transfer them to a real robot with zero fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:26 UTC pith:QGNBRZQM

load-bearing objection Promising exploration scheme for online zero-shot RL with a real quadruped demo, but the reported large-angle orientation results conflict with the paper's own state filter. the 5 major comments →

arxiv 2603.25464 v2 pith:QGNBRZQM submitted 2026-03-26 cs.LG cs.AI

Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

Jiajun Hu , Nuria Armengol Urpi , Jin Cheng , Stelian Coros This is my paper

classification cs.LG cs.AI

keywords zero-shot reinforcement learningforward-backward representationsbehavior entropyinverse-density explorationnormalizing flowsquadruped locomotionsim-to-real transferonline RL

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tackles a bottleneck in zero-shot reinforcement learning: pretraining data for legged robots must be diverse, but no large reward-free datasets exist for quadrupeds. The authors propose to collect that data online while training, using a forward-backward value representation and an exploration strategy that steers the robot toward behaviors that have been visited rarely. A normalizing flow estimates the density of achieved states on a task-relevant projection of the state space, and exploration policies are sampled from the inverse density. A regularization critic penalizes unnatural motion such as foot slippage. The authors show in simulation that this raises behavior entropy and downstream task performance, and they deploy the resulting policies on a real quadruped robot without additional fine-tuning.

Core claim

The paper's central claim is that the exploration problem in online zero-shot RL can be solved by maximizing the entropy of the achieved behavior distribution, implemented as inverse-density sampling over states, rather than by uniformly sampling task embeddings or relying on external motion-capture priors. Combined with a regularizer critic, this yields forward-backward representations that cover high-velocity and large-angle behaviors, and recovered policies that look natural and transfer to hardware. The authors state that this is the first fully online zero-shot RL algorithm deployed on a real robot without external datasets as motion priors.

What carries the argument

The core mechanism is the inverse-density exploration rule: at each episode the agent fits a normalizing flow q_psi(s) to states in the replay buffer, then samples exploration behaviors z_E^j = B(s_E^j) from states s_E^j drawn with probability proportional to (q_psi(s)+epsilon)^(-beta). This biases the agent toward rarely visited parts of a hand-picked state projection (planar velocity for locomotion, body-frame gravity for orientation). A critic Q_reg trained on a smoothness and slippage reward is added to the actor objective to keep gaits natural.

Load-bearing premise

The method assumes that low density in a hand-picked projection of the state space (e.g., planar velocity or base orientation) is a sufficient proxy for maximizing the entropy of behaviors that matter for downstream tasks; if a future reward depends on a state dimension outside that projection, the exploration and the learned representation will not cover it.

What would settle it

Run FB-MEBE on a task family whose downstream rewards depend on a state coordinate excluded from the exploration projection (e.g., a specific joint angle or terrain height). If zero-shot performance collapses while a baseline that explores uniformly over the full state succeeds, the inverse-density proxy is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If correct, zero-shot RL for legged robots no longer requires large pre-collected motion-capture datasets; the data can be collected online during the same run.
The inverse-density exploration rule directly counteracts replay-buffer collapse, maintaining coverage of extreme behaviors such as high-speed and large-tilt commands.
The regularization critic bounds the solution manifold, suppressing foot drag and making recovered policies deployable without a sim-to-real finetuning step.
The method's success implies that a single forward-backward representation trained without task rewards can support a family of velocity- and orientation-tracking tasks at test time by linear regression of the reward.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The choice of projection phi(s) is the strongest hidden prior; extending the density model to the full state space, or learning phi, would be a natural next step and would test whether the entropy proxy generalizes beyond velocity and orientation tasks.
Inverse-density sampling and epistemic-uncertainty exploration are complementary: a hybrid that samples rare states and high-uncertainty embeddings might yield faster coverage.
The results suggest a recipe for other robots: choose a task-relevant low-dimensional behavior space, fit a density model, and use its tail as the exploration curriculum; this could apply to manipulation, where the projection would be end-effector pose or object pose.
Because the method is fully online, it could be adapted to continual or non-stationary settings, re-fitting the density model as the environment changes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Promising exploration scheme for online zero-shot RL with a real quadruped demo, but the reported large-angle orientation results conflict with the paper's own state filter.

read the letter

The headline is: a sensible combination — FB with inverse-density maximum-entropy exploration plus a regularizer critic — demonstrated on a real quadruped with zero-shot transfer. That's a worthwhile step. But there is a load-bearing internal inconsistency that needs resolving before I trust the empirical core.

What's actually new: online FB with exploration guided by inverse density over a state projection, no external motion-capture or expert data, and a regularizer critic that shapes natural gaits. The claim that this beats uniform exploration is plausible, and the ablation on β is a good check. The hardware demo is a practical contribution if it holds up.

Soft spots, in order of severity:

1. Appendix B discards states with pitch or roll > 21.5°, yet Figure 6 reports strong performance on orientation tasks targeting 45° and 60°. If those states never enter the replay buffer, the density model, the B representation, and the linear regression for the task embedding all have zero support there. The reported returns on those large-angle tasks cannot be explained by the described mechanism. This is not a minor quibble; it's an internal contradiction. Either the filter is applied differently than written, the evaluation protocol differs, or something is missing.

2. The closest baseline — Urpí et al. 2025, which the paper itself identifies as most similar — is never compared in the experiments. That weakens the central exploration claim.

3. The feet-slippage term appears in the regularizer reward and again as the evaluation metric for "natural" behaviors, so that result is partly circular.

4. Hardware deployment is supported only by qualitative video, not quantitative tracking metrics.

5. The paper says code is available but doesn't link it in the manuscript.

These are not equal in weight. The first is the big one; if it cannot be resolved, the large-angle orientation results are uninterpretable. The others are standard but should be addressed.

Who this is for: researchers working on behavior foundation models, unsupervised exploration, or legged locomotion. It deserves a serious referee, primarily to sort out the filter contradiction and to get a real comparison with the epistemic baseline.

Recommendation: engage with the work, but require the authors to clarify the filter implementation and reconcile it with their reported results before accepting the empirical claims.

Referee Report

5 major / 4 minor

Summary. The paper proposes FB-MEBE, an online zero-shot reinforcement learning method for quadrupedal locomotion built on the Forward-Backward (FB) representation. It replaces uniform random exploration with inverse-density sampling of states under a normalizing flow (Eq. 11), and adds a regularization critic that penalizes joint accelerations, action rates, and foot slippage (Eq. 27). The authors claim that FB-MEBE improves downstream zero-shot performance over uniform exploration and regularized variants in simulation, produces more natural gaits, and can be deployed zero-shot to a real Unitree Go2. The paper includes simulations on velocity-tracking and base-orientation tasks, ablations on the inverse-temperature β and training sampling strategy, and qualitative hardware results.

Significance. If fully substantiated, this work would be a useful practical advance: it addresses the data-diversity bottleneck of online FB training without relying on external motion-prior datasets, and it provides a concrete instantiation of zero-shot RL on a legged robot. The paper is clearly structured and includes useful experimental details: pseudocode, hyperparameters (Tables 5–8), density-model specifications, domain randomization, and code/video release. However, several load-bearing issues currently weaken the empirical claims: an apparent inconsistency between the state filter in Appendix B and the reported large-angle orientation results, a circular gait-quality metric, the absence of the closest exploration baseline, and only qualitative hardware evidence. These need to be resolved before the main claims can be accepted.

major comments (5)

[Appendix B and Figure 6] Appendix B states: 'we discard states where the robot pitch or roll exceeds 21.5°.' Figure 6 reports successful performance on orientation tasks with target pitch/roll magnitudes of 45° and 60°. Under the gravity-vector projection [gx, gy, gz] used for orientation exploration, a 45° roll corresponds to |gx| ≈ 0.707, far above the 21.5° threshold (sin 21.5° ≈ 0.366). If this filter is applied to transitions stored in the replay buffer, the FB representation and policies are never trained on states beyond the threshold, so the reported large-angle orientation returns in Figure 6 are unexplained by the described mechanism. Please clarify exactly where the filter is applied (replay buffer, density model, episode termination) and reconcile it with the orientation results.
[Section F, Eq. (27) and Figure 4] The regularizer reward is defined as r_reg = -2.5e-7 r_joint_acc - 0.1 r_action_rate - 0.1 r_feet_slide (Eq. 27), and Section 6.1 reports 'Feet Slippage' (Figure 4, right) as evidence that FB-MEBE produces natural gaits. Since foot slippage is directly penalized in the training reward, this metric is not an independent measure of gait quality; lower foot slippage is expected by construction. To support the natural-behavior claim, the authors should report an external metric (e.g., foot contact patterns, ground reaction forces, energy expenditure, or human evaluation) or explicitly state that foot slippage is a training objective rather than an evaluation result.
[Sections 2 and 6] The paper identifies Urpí et al. (2025) as the closest work on exploration in online FB, and derives a related objective in Section 5.1, but it is never compared experimentally. The abstract's claim of 'improved performance compared to other exploration strategies' is under-supported without this baseline. At minimum, include Urpí et al. (2025) in the comparisons in Figures 4–6; ideally also include Sun et al. (2025) (RND-based exploration).
[Section 6.1 (Hardware Tests)] The hardware section is only qualitative: it reports that the robot 'tracks commanded translational velocities, adjusts pitch and roll orientations, and regulates its height' and refers to Figure 7, but no quantitative data are given. Given the central claim of seamless zero-shot sim2real transfer, please provide quantitative hardware results such as tracking errors, success rates, or a comparison with a baseline on the real robot.
[Figures 4–6] Results are averaged over 5 random seeds with shaded regions / error bars, but no significance tests or effect sizes are reported. For the claim that 'FB-MEBE achieves the highest average performance' on velocity-tracking tasks, report per-task paired differences, confidence intervals, or a statistical test; otherwise the improvements may be within seed noise.

minor comments (4)

[General] Typos and grammar: 'Pratical' should be 'Practical' (Section 5.2), 'an improved' should be 'an improved' (Abstract), 'succesor' should be 'successor' (Section K), and 'the later is analogous...uses' is ungrammatical (Appendix C.2).
[Section 6] The text says both velocity-tracking and orientation settings 'consists of 17 downstream tasks,' but Figure 6 reports 'Top-12 at 300K Steps' and states that tasks with all algorithms below reward 1 were removed. Clarify whether the 17-task averages in Figure 4 include those discarded tasks and why the subset is shown.
[Section 5.2, Eq. (10)–(11)] Notation is inconsistent: q_Ψ and q_ψ are used interchangeably. Also, the text says the density model is learned on achieved states q_Ψ(s) 'or on a projection φ(s),' but Section 6 always uses projections; specify which object is used in each experiment.
[Section 6, projection choice] The exploration projection φ(s) is task-specific ([vx, vy] for velocity tasks, [gx, gy, gz] for orientation tasks), which encodes prior knowledge of the downstream task family. The paper acknowledges this and leaves full-state exploration to future work, but the 'no prior knowledge' phrasing in the abstract should be tempered.

Circularity Check

2 steps flagged

Downstream performance comparison is independent, but two secondary evaluation metrics (feet slippage and behavior entropy) are the same quantities used by the method's own regularizer and exploration objective.

specific steps

fitted input called prediction [Section F Eq. (27); Section 6.1 Overall Performance / Figure 4]
"rreg = −2.5×10−7 rjoint_acc −0.1 raction_rate −0.1 rfeet_slide (27) ... FB-MEBE substantially reduces the feet slide penalty compared to the original FB, indicating that the regularization critic successfully suppresses unnatural dragging behaviors."

The paper presents reduced feet slippage as evidence that FB-MEBE yields 'natural behaviors'. But r_feet_slide is one of the three terms explicitly minimized by the regularization critic Q_reg in Eq. (27), and this critic is added to the actor loss in Eq. (29) with weight λ_reg=20. The observed reduction in feet slippage is therefore an optimization target, not an independent outcome. Using this metric as evidence of naturalness validates the method on its own training signal.
self definitional [Section 5.2 Eq. (11); Section 6 / Figure 4 middle panel]
"we learn the density model on the space of achieved states qΨ(s) instead (or on a projection φ(s) of it) ... {zE j =B(sE j)}Kj=1, sE j ∼(q t ψ(s))−β (11) ... For base-orientation tasks exploration is performed in the space of the gravity vector ... [gx, gy, gz] ... FB-MEBE yields the highest entropy among all baselines. This indicates that FB-MEBE expands behavioral coverage during online training."

The exploration rule in Eq. (11) explicitly selects states on the projection φ(s) that have low density under q_ψ, where q_ψ is fitted on the replay buffer's achieved states on that same projection. Reporting high 'entropy on [gx,gy,gz]' as evidence of expanded behavioral coverage therefore measures the algorithm's own optimization objective rather than an independent property. For orientation tasks the reported entropy is computed on exactly the projection φ(s) used for inverse-density exploration, so the entropy result is partly by construction. The independent evidence for coverage is downstream task return, which is not circular.

full rationale

The central quantitative claim—improved downstream returns on velocity- and orientation-tracking tasks—is not circular: the task rewards in Eqs. (16)-(21) are defined independently of the exploration objective and regularizer, and FB-MEBE is compared against baselines using those returns. I found no load-bearing self-citation: Urpí et al. (2025) is mentioned only as related work, and the FB optimality guarantee is imported from the external Touati & Ollivier line rather than from the authors' own unverified results. The circularity is confined to two secondary evaluation claims. First, the 'natural behaviors' claim is substantiated by feet slippage, yet r_feet_slide is explicitly one of the terms minimized by the regularization critic (Eqs. 27 and 29), so the reduction is built into the training objective. Second, the 'expanded behavioral coverage' claim is substantiated by entropy on the same state projection φ(s) used for inverse-density exploration (Eq. 11); for orientation tasks the reported entropy on [gx,gy,gz] is exactly the projection on which the density model is fitted and low-density states are sampled. These metrics validate the method on its own objectives rather than as independent predictions. A separate, non-circularity concern: Appendix B says states with pitch/roll exceeding 21.5° are discarded, while Figure 6 reports orientation tasks at 45° and 60°; if that filter is applied to the replay buffer, the large-angle results are not explained by the described mechanism. This is an internal consistency issue, not a circularity, and does not change the circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on the FB factorization being approximately valid in a real quadruped environment, on a heuristic mapping from entropy maximization to inverse-density sampling, on the density model estimating replay coverage, on linear task embeddings capturing downstream rewards, and on sim-to-real transfer. No new physical entities are introduced.

free parameters (5)

β (inverse-density sampling strength) = 2.0
Controls the inverse reweighting in Eq. 10; ablation shows larger β improves entropy and reward, so 2.0 is chosen performance-wise. Central exploration/performance claim depends on it.
λ_reg (regularizer coefficient) = 20
Scales Q_reg in actor loss Eq. 6/29; chosen to balance behavior regularization and FB objective; not tuned per task, but value is a design choice.
80/20 exploration-training embedding split = 80% MEBE + 20% uniform
Hyperparameter in Table 5; the method's performance is sensitive to the ratio (Figure 9 and text mention capacity balancing).
task-specific projection φ(s) = [vx, vy] or [gx, gy, gz]
Exploration and density estimation are run on a hand-picked state projection chosen to match the known downstream task family (velocity vs orientation); this is a free modeling choice that encodes task prior.
state filtering threshold = pitch/roll > 21.5° discarded
Appendix B; excludes fall states from training data. This is a data-exclusion parameter that shapes the buffer distribution.

axioms (6)

domain assumption The successor measure of any policy admits a low-rank factorization M^{π_z} ≈ F(s,a,z)^T B(s') for rank d=50 (Eq. 3).
Inherited from FB (Touati & Ollivier 2021) and needed for the theoretical optimality guarantee; on a real quadruped this is approximate.
ad hoc to paper Maximizing the entropy of achieved behavior distribution is the right target when the desired behavior distribution is unknown (Eq. 8).
Intractable oracle objective Eq. 7 is replaced by entropy maximization, which is justified only by analogy to Pitis et al. (2020).
ad hoc to paper Low density in the replay buffer is a proxy for the unknown q(z'_r | z_r) in Eq. 9, so sampling states with minimum density approximately maximizes entropy.
The paper states this as an observation; no proof connects inverse density sampling to the entropy objective.
domain assumption A normalizing flow trained on replay-buffer states gives a reliable density estimate q_ψ(s) for inverse sampling.
Assumes the flow model accurately estimates visitation density on a changing data distribution; normalizing flows can be brittle.
standard math Downstream rewards of interest lie in the linear span of the learned B features, so the z_r = E[B(s) r(s)] embedding recovers near-optimal policies.
Part of the FB guarantee; used for all zero-shot task inference.
domain assumption Simulation with domain randomization transfers to the real Unitree Go2 without finetuning.
The hardware deployment claim depends on sim2real transfer; the paper's evidence is qualitative (videos).

pith-pipeline@v1.3.0-alltime-deepseek · 15782 in / 14476 out tokens · 135997 ms · 2026-08-03T02:26:27.196334+00:00 · methodology

0 comments

read the original abstract

Zero-shot reinforcement learning (RL) algorithms aim to learn a family of policies from a reward-free dataset, and recover optimal policies for any reward function directly at test time. Naturally, the quality of the pretraining dataset determines the performance of the recovered policies across tasks. However, pre-collecting a relevant, diverse dataset without prior knowledge of the downstream tasks of interest remains a challenge. In this work, we study $\textit{online}$ zero-shot RL for quadrupedal control on real robotic systems, building upon the Forward-Backward (FB) algorithm. We observe that undirected exploration yields low-diversity data, leading to poor downstream performance and rendering policies impractical for direct hardware deployment. Therefore, we introduce FB-MEBE, an online zero-shot RL algorithm that combines an unsupervised behavior exploration strategy with a regularization critic. FB-MEBE promotes exploration by maximizing the entropy of the achieved behavior distribution. Additionally, a regularization critic shapes the recovered policies toward more natural and physically plausible behaviors. We empirically demonstrate that FB-MEBE achieves and improved performance compared to other exploration strategies in a range of simulated downstream tasks, and that it renders natural policies that can be seamlessly deployed to hardware without further finetuning. Videos and code available on our website.

Figures

Figures reproduced from arXiv: 2603.25464 by Jiajun Hu, Jin Cheng, Nuria Armengol Urpi, Stelian Coros.

**Figure 3.** Figure 3: Maximum Entropy Behavior Forward-Backward Exploration: FB-MEBE is an online [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 2.** Figure 2: Limitations of undirected exploration: Standard FB algorithm with undirected exploration [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 4.** Figure 4: Left: Zero-shot performance averaged over 17 downstream velocity tracking tasks (top) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Comparison of different methods under detailed locomotion command settings. Each [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

**Figure 6.** Figure 6: Comparison of different methods under detailed orientation command settings. Each [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

**Figure 7.** Figure 7: FB-MEBE allows for zero-shot sim2real transfer to Unitree Go2, a commercial [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗

**Figure 8.** Figure 8: Comparison between different betas on FB-MEBE on locomotion tasks [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: Comparison on velocity-tracking tasks between FB-MEBE and an ablation FB-MEBE [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

**Figure 10.** Figure 10: Velocity [vx, vy] distribution of FB replay buffer at different training step. The color represents the density of data points. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

[2023]

for an extensive list), however, they requires a separate methods for learning featuresϕand one for learning its successor features, usually via a TD-learning approach. In contrast, the forward backward representation (Touati & Ollivier, 2021) provides an alternative framework for zero-shot RL that jointly learns the features and its successor features, v...

2021
[2025]

Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu

URLhttps://arxiv.org/abs/2505.22642. Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off-policy reinforcement learning.CoRR, abs/2004.12974, 2020. URLhttps://arxiv.org/abs/2004.12974. Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, S...

Pith/arXiv arXiv 2004

[1] [2023]

for an extensive list), however, they requires a separate methods for learning featuresϕand one for learning its successor features, usually via a TD-learning approach. In contrast, the forward backward representation (Touati & Ollivier, 2021) provides an alternative framework for zero-shot RL that jointly learns the features and its successor features, v...

2021

[2] [2025]

Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu

URLhttps://arxiv.org/abs/2505.22642. Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off-policy reinforcement learning.CoRR, abs/2004.12974, 2020. URLhttps://arxiv.org/abs/2004.12974. Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, S...

Pith/arXiv arXiv 2004