Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Behavioral Exploration: Learning to Explore via In-Context Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single offline-trained diffusion policy, conditioned on its own history and a coverage score, can explore like an expert and adapt online without gradient updates.

desk verdict BE is a clean and mostly convincing offline exploration recipe; the random-feature coverage proxy is the open question that decides how deep the result runs. read the letter →

arxiv 2507.09041 v1 pith:RRN5YP3J submitted 2025-07-11 cs.LG cs.ROcs.SYeess.SY

Andrew Wagenmaker , Zhiyuan Zhou , Sergey Levine This is my paper
classification cs.LGcs.ROcs.SYeess.SY
keywords behavioralexplorationin-contextadaptationcoveragemaximizationcloningdiffusionpolicyfromdemonstrationsD4RLAntmazeroboticmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces behavioral exploration, a training scheme that turns an expert demonstration dataset into a policy that explores by adapting in-context. Instead of random exploration or slow online reinforcement-learning updates, the method trains a long-context generative policy to predict expert actions given the current state, a history of past states, and a coverage score measuring how much new state coverage a future trajectory would add. The paper argues that this conditional distribution reweights the expert policy toward actions not yet tried, so at deployment the agent can feed its own past into the context and choose different expert behaviors to solve ambiguous tasks. The claim is that this supervised, offline-only objective produces fast online adaptation that competes with or beats RL exploration methods, and the paper supports it with results on D4RL Antmaze and Kitchen, the Libero manipulation benchmark, and a real WidowX robot.

What carries the argument

The object that carries the argument is the coverage functional $$\mathrm{cov}(h) := \frac{1}{\operatorname{tr}\left((\Lambda(h)+\$\lambda$ I)^{-1}\right)}, \qquad \Lambda(h) = \sum_{s \in h} \$\varphi$(s)\$\varphi$(s)^\top,$$ which quantifies how well a set of states spans a feature space. For the paper, its role is to turn "exploratory" into a conditioning variable: the model is trained on expert trajectories labeled with the coverage of history union future, so at deployment a high user-set coverage value makes the policy select expert behaviors that add new feature directions rather than repeat the past. The architecture is a transformer-based diffusion policy, with one token for each state in the history, the current state, and the coverage value, and the formal justification is Proposition A.1, which assumes one-hot terminal features and shows that the history-conditioned, coverage-maximizing policy is optimal for the behavior-coverage objective.

What would settle it

Run BE on a demonstration dataset with two goal regions, where the expert strongly favors one region and the other is reachable but rare, then vary the coverage conditioning value and measure rollout density in each region. If raising the coverage value does not shift behavior toward the under-covered region while preserving expert-like trajectories, the central mechanism—that history plus coverage reweights the expert distribution toward novel useful behaviors—is not doing the work.

Watch

Extended reading notes

Core claim

The central claim is that conditioning a behavioral-cloning policy on the history of visited states and on the coverage of the trajectory that follows a candidate action makes the policy imitate the expert's conditional distribution $P^{\pi_\beta}(\cdot \mid s, h, \mathrm{cov}(h \cup \tau) = \max)$, which is optimal for the goal of maximizing coverage over the behavior policy's feature span. Concretely, the paper proposes maximizing $\mathbb{E}[\log \pi_{\mathrm{BE}}(a_t^k \mid s_t^k, h, \mathrm{cov}(h \cup \tau_t^k))]$ over demonstrations, and shows in Proposition A.1 that under deterministic dynamics, one-hot terminal-state features, finite actions, a fixed start, and exactly $n_\beta$ episodes, the policy that conditions on maximal future coverage visits every reachable terminal feature direction. With this conditional distribution learned by a transformer-based diffusion model, BE reaches 0.556 of Antmaze goals versus 0.285 for BC and at most 0.195 for the strongest skill-based RL baseline, solves hidden Libero tasks with 0.824 success versus 0.590 for BC, and on a real WidowX robot succeeds roughly 40 percent more often than BC or OpenVLA by interacting with both objects.

Load-bearing premise

The load-bearing premise is that coverage computed in a fixed random feature space is a faithful proxy for task-relevant exploration; if random-feature novelty does not align with what the task actually requires, conditioning on high coverage just reweights toward behaviors that are novel in an irrelevant space.

Editorial extensions

If this is right

  • BE-trained policies can be deployed online with no gradient-based updates; exploration happens entirely through in-context conditioning on the agent's own past, so the approach scales to settings where online RL adaptation is too slow.
  • On D4RL Antmaze medium and large, BE reaches 55.6% of goals across 20k steps, compared with 28.5% for BC and at most 19.5% for SUPE, HILP, ExPLORe, or RND.
  • On the Libero hidden-task evaluation, BE solves 82.4% of tasks across 15 attempts versus 59.0% for BC, needing 2-3 times fewer attempts, and it remains strong when the task is provided (99.3% vs 97.7%).
  • On a real WidowX robot, BE interacts with both objects in roughly 40% more trials than BC or OpenVLA and succeeds in about two fewer attempts.
  • The learned policy is calibrated: increasing the coverage conditioning value increases the number of maze regions visited, giving the user a dial to trade exploration against sticking close to demonstrated behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether its fixed random feature space (a random MLP encoder or random Fourier features) is the right measure of task-relevant novelty; if that proxy fails, BE would degrade toward behavioral cloning with a history-dependent reweighting rather than true exploration.
  • A natural testable extension is to compare BE trained with different feature spaces, such as learned encoders or task-informed features, and check whether online goal discovery tracks coverage in those spaces; the Maze2D visualizations are the only direct evidence of history-conditioned adaptation.
  • The same coverage-conditioned maximum-likelihood objective could in principle be applied to other generative policy classes, such as flow matching or vision-language-action models, and its in-context structure suggests it could be combined with language prompting for ambiguous instruction-following tasks.
  • Because BE restricts exploration to the expert's behavior space, it will not find solutions outside that space; the paper acknowledges this in its discussion, so the method is best understood as an efficient way to search a known behavior prior rather than to expand it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes "behavioral exploration" (BE), an offline training method that learns a long-context diffusion policy to predict expert actions conditioned on the current state, a history of previously visited states, and a scalar coverage measure indicating how much the future trajectory adds to the feature-space coverage of the history. At deployment, the policy receives its own past states in context and a user-specified coverage value that controls how explorative the behavior should be, enabling in-context adaptation without online gradient updates. The authors prove an optimality result (Proposition 4.2 / A.1) in an idealized setting with one-hot terminal-state features, deterministic dynamics, finite actions, a fixed start state, and a fixed number of episodes. They then evaluate BE on D4RL Antmaze and Kitchen, the Libero benchmark, and real WidowX manipulation tasks, reporting gains over BC and several RL-based exploration baselines.

Significance. If the empirical results hold, BE provides a conceptually simple, fully offline, supervised method for fast in-context exploration that avoids the computational cost of online RL fine-tuning. The experimental evaluation is broad and generally careful: error bars are reported, hyperparameter sweeps are applied to both BE and BC, multiple seeds are used, and the evaluation metrics (goals reached, regions visited, task success, object interactions) are external to the training signal, so the headline comparisons are not self-referential. The paper is also honest about the scope of its formal result, stating the restrictive assumptions of Proposition A.1. The main weakness is that the proof's idealized features are replaced in all continuous experiments by random features, and the paper does not provide a formal or empirical bridge showing that coverage in the random feature space is a faithful proxy for task-relevant exploration. This gap is load-bearing but addressable.

major comments (3)
  1. [4.1-4.2, Eq. (4), Prop. A.1] The optimality guarantee is proved only for one-hot features that are nonzero at terminal states (Prop. A.1), while every continuous experiment uses a random MLP encoder (D4RL) or random Fourier features (Libero and WidowX; Section B). The paper asserts that the conditioned policy "explores over U_beta,epsilon by construction" because its action distribution is a reweighting of pi_beta, but this does not establish that conditioning on coverage in a random feature space selects behaviors that increase task-relevant coverage. If the random features do not align with goal-relevant variation, the deployment-time coverage value exp could reweight BC toward behaviors that are merely diverse in an arbitrary space. Please either provide an analysis of the random-feature setting or an empirical test of the proxy, for example by comparing coverage computed with random features against coverage computed with features learned from the demonstration data, or by measuring the correlation between cov and goal-reaching progress in Antmaze and Libero. Without such support, the continuous-state results are not explained by the theory.
  2. [5.1, Fig. 6, Table 2] The deployment coverage value exp is a free parameter, and Figure 6 demonstrates that the number of regions visited in Antmaze medium changes substantially with this value. The paper does not report the chosen exp for each environment, nor the sensitivity of the reported results to this choice. Since the D4RL results in Table 2 and the Libero and WidowX results all depend on this setting, please report the selected exp values (or the normalization behind the percentage axis in Figure 6) and provide a sensitivity analysis for at least one environment per domain. This is necessary to determine whether the gains over BC and the RL baselines are robust or rely on per-environment tuning.
  3. [4.2, Eq. (4), Prop. 4.2] Proposition 4.2 concerns the policy that conditions on the event that the future trajectory attains the maximum coverage, while the training objective (4) conditions on the scalar value cov(h union tau_t^k). The paper states without derivation that maximizing (4) yields the conditional distribution in (3). If the demonstration data do not contain a trajectory attaining the maximum coverage for a given history, or if that maximum is attained only by very low-probability behaviors, then the learned policy evaluated at a high deployment exp will extrapolate beyond the training support. Please clarify the correspondence between the two conditionals, or explicitly state that the deployment procedure is a heuristic approximation of the idealized policy.
minor comments (5)
  1. [Table 2 caption] The caption reads "Final success rate from Figure 7 and 8," but Table 2 reports D4RL results from Figures 3-5; the cross-reference should be corrected.
  2. [Appendix B.1] The text says "randomly exploring over the maze is itself an affective strategy"; "affective" should be "effective."
  3. [Appendix B.3] The sentence "We note theBC fails to succeed a single time on 2/3 tasks" is missing a space between "the" and "BC."
  4. [Figure 6] The x-axis is labeled "Coverage Conditioning Value (%)" but Section 4.2 defines exp as a real value; please specify how the percentage is computed from the raw coverage value.
  5. [Section 4.3] The phrase "state-of-the-art transformer-based diffusion policy proposed in Dasari et al. (2024)" should be softened or substantiated, since the paper does not compare against alternative diffusion policy backbones in this work.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: BE's empirical comparisons are external, and the only definitional step is Proposition 4.2, which restates the coverage objective as its own optimizer.

  1. self definitional [Section 4.2, Proposition 4.2 (informal); Appendix A, Proposition A.1]
    "use D to find a policy π which maximizes covβ(Λπ) ... the policy which takes actions at trajectory t with probability P πβ [ · |s, ht, cov(ht ∪ τ) = maxτ′ cov(ht ∪ τ′)] is an optimal solution to Objective 4.1."

    The policy in Proposition 4.2 is defined by conditioning on the event that the continuation trajectory achieves the maximum of the very coverage function that Objective 4.1 asks a policy to maximize. In the one-hot terminal-state setting of Proposition A.1, this event is equivalent to reaching a not-yet-visited behavior feature, so the proposition's optimality conclusion is already contained in the definition of the conditioning event: it restates the argmax of the objective rather than independently deriving why coverage-conditioned cloning explores. The subsequent step to Eq. (4) replaces the future-trajectory conditioning with a scalar exp, a leap not covered by the proposition.

full rationale

The central empirical claims of the paper are not self-referential. The coverage value cov(·) is computed from random feature maps (a random MLP for D4RL, random Fourier features for Libero and WidowX), not from the task reward or from the evaluation metric. The evaluation metrics are external to the training signal: goals reached, regions visited, tasks accomplished, and real-robot object interactions. The training objective (4) is a conditioned behavioral cloning objective, and the deployment procedure conditions on a user-set coverage value exp plus the agent's own history; neither the random features nor the evaluation quantities are fitted to the reported results. Proposition A.1 is a self-contained proof with stated assumptions, but as noted above it is close to a definitional restatement of Objective 4.1 in the special one-hot terminal setting; this does not compromise the empirical comparisons, which are the paper's main evidence. Self-citations (Dasari et al. 2024 for the diffusion architecture, Wilcoxson et al. 2024 for baseline checkpoints) are used for implementation and baselines, not to justify the exploration mechanism. The concern raised about random-feature coverage being an unverified proxy for task-relevant novelty is a correctness and generalization risk, not a circularity: the paper does not define success in terms of its own coverage metric. Overall, the derivation chain is mostly independent, with one minor definitional step that does not force the reported results.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method imports the A-optimal experiment-design coverage machinery (Eqs. 1 to 2), the transformer diffusion policy (Dasari et al. 2024), and maximum-likelihood conditioning from the return-conditioned policy literature, and adds history conditioning. The items the paper itself costs are the random feature map defining coverage, the deployment coverage value exp, the uniform-over-trajectories history distribution, and the proxy assumption that random-feature coverage tracks task-relevant novelty. No new entities are postulated.

free parameters (4)
  • Deployment coverage conditioning value exp = Not reported
    Section 4.2 samples actions from πBE(·|s_k, h_k, exp) and states that larger exp increases exploration; Figure 6 shows total regions visited varies with exp (16 to 22 on Antmaze medium). Appendix B's tables never list the per-environment value used for the headline results.
  • Coverage regularization λ in Eq. (1) = 0.01 (D4RL)
    The coverage metric is cov(h) = 1/tr((Λ(h) + λI)^{-1}); the ridge λ is chosen by hand and changes the coverage geometry, especially for small histories.
  • Random feature map φ = 32-dim random MLP (D4RL); 16-dim random Fourier features (Libero, WidowX)
    Both training conditioning (Eq. 4) and deployment conditioning use coverage in a fixed random feature space chosen by hand (Section 4.1, Appendix B). The method's behavior is defined only up to this choice.
  • Context history length and coverage future length = 50 to 100 and 20 to 200 across environments (Tables 3, 7, 9)
    These hyperparameters define what history and future coverage mean in Eq. (4); they are swept and fixed per environment without a sensitivity study.
assumptions (4)
  • domain assumption The demonstration policy πβ spans the space of useful behaviors
    Section 3 assumes the behaviors exhibited by πβ are a reasonable prior over useful behaviors. If the demonstrations are unhelpful or omit the behavior needed in the test task, BE explores within an unhelpful space; the Discussion concedes exploration may need to go beyond the demo space.
  • ad hoc to paper Random-feature coverage is a faithful proxy for task-relevant exploration
    Proposition A.1 is proven only for one-hot terminal-state features with deterministic dynamics, a fixed start state, finite actions, and exactly nβ episodes. The empirical method uses random features over raw states, and transfer of the idealized result to that setting is assumed rather than derived. Figures 12 to 13 give indirect support only in Maze2D.
  • domain assumption The online history distribution is a reweighting of the πβ-induced distribution
    Section 4.3 chooses H(D) as uniform over trajectories instead of the induced online distribution and asserts it is 'simply a reweighting' with no argument. The in-context adaptation mechanism depends on training histories resembling deployment histories.
  • ad hoc to paper A scalar coverage value at deployment aligns with training-time future-trajectory coverage
    Training (Eq. 4) conditions on cov(h ∪ τ_t^k) of the actual expert future; deployment feeds a user-chosen scalar exp. Figure 6 shows the policy responds to the scalar, but the paper gives no argument that the learned conditional distribution transfers across this train/test conditioning gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behavioral Exploration: Learning to Explore via In-Context Adaptation." pith.science (2026). https://pith.science/paper/RRN5YP3J

@misc{pith2026250709041,
  author       = {Pith},
  title        = {Pith review of: Behavioral Exploration: Learning to Explore via In-Context Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRN5YP3J}},
  note         = {Machine review of arXiv:2507.09041}
}
read the original abstract

Developing autonomous agents that quickly explore an environment and adapt their behavior online is a canonical challenge in robotics and machine learning. While humans are able to achieve such fast online exploration and adaptation, often acquiring new information and skills in only a handful of interactions, existing algorithmic approaches tend to rely on random exploration and slow, gradient-based behavior updates. How can we endow autonomous agents with such capabilities on par with humans? Taking inspiration from recent progress on both in-context learning and large-scale behavioral cloning, in this work we propose behavioral exploration: training agents to internalize what it means to explore and adapt in-context over the space of ``expert'' behaviors. To achieve this, given access to a dataset of expert demonstrations, we train a long-context generative model to predict expert actions conditioned on a context of past observations and a measure of how ``exploratory'' the expert's behaviors are relative to this context. This enables the model to not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected, thereby allowing for fast online adaptation and targeted, ``expert-like'' exploration. We demonstrate the effectiveness of our method in both simulated locomotion and manipulation settings, as well as on real-world robotic manipulation tasks, illustrating its ability to learn adaptive, exploratory behavior.

Figures

Figures reproduced from arXiv: 2507.09041 by the authors.

Figure 1
Figure 1. Overview of our proposed approach, behavioral exploration. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrative example. A natural choice for a policy maximizing cov￾erage over the space spanned by πβ would be πβ itself—by defi￾nition, πβ covers the space spanned by πβ— which suggests simply training a BC policy on D to solve Objective 4.1. To illustrate the shortcomings of BC, how￾4 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Performance on D4RL Kitchen, measuring percentage of tasks accomplished. ploratory behavior by labeling observations with a pseudo￾reward intended to reflect uncertainty. To achieve this, a neural network is randomly initialized and, online, the agent trains another network to match the predictions of this net￾work over the observed states. The pseudo-reward is the mis￾match between the predictions of the networks; … view at source ↗
Figures from the paper (17 more)
Figure 6
Figure 6. Figure 6: Calibration of BE policy on Antmaze medium. In [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 10
Figure 10. Figure 10: Example Libero tasks. Environment details. The Libero benchmark contains a variety of scenes, simulating different robotic manipulation and pick-and-place tasks [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 7
Figure 7. Figure 7: Performance on Libero in evaluation with task hidden. 1 2 3 4 5 6 Number of Attempts 50 60 70 80 90 100 Task Success (%) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 11
Figure 11. Figure 11: Example WidowX setup. Environment details. We eval￾uate our policy on the WidowX 250 6-DoF robot arm. We con￾sider 3 different reaching tasks, where for each task, we place 2 objects in front of the robot, and the goal is to interact with each ob￾ject in some way (see…
Figure 12
Figure 12. Figure 12: BE trajectory rollouts (blue) when the BE policy is conditioned on a particular history segment (green). Red “x” denotes the starting point for the BE rollouts. In all cases BE adapts its behavior to explore the regions not covered by the conditioning trajectory. (a) …
Figure 13
Figure 13. Figure 13: Comparison of BE conditioned with history of past online observations, BE conditioned on only the first state, and BC. We see that conditioning on the history of past states visited is critical to achieving high coverage, and that this induces coverage over low-densit…
Figure 14
Figure 14. Figure 14: Visualization of D4RL Antmaze medium and large environments. Red squares denote a single “region”, used to calculate the region count in [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 16
Figure 16. Figure 16: Performance on Antmaze medium, measuring number of goals reached. 0.0 0.5 1.0 1.5 2.0 Steps ×10 4 0 5 10 15 20 Number of Regions Visited [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 18
Figure 18. Figure 18: Performance on individual goals on Antmaze [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Performance on Antmaze large, measuring number of goals reached. 0.0 0.5 1.0 1.5 2.0 Steps ×10 4 0 5 10 15 20 25 Number of Regions Visited [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 21
Figure 21. Figure 21: Performance on individual goals on Antmaze [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: Performance on individual Libero tasks (1-45) in evaluation with task hidden. [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: Performance on individual Libero tasks (46-90) in evaluation with task hidden. [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 24
Figure 24. Figure 24: Performance on individual Libero tasks (1-45) in evaluation with task provided. [PITH_FULL_IMAGE:figures/full_fig_p025_24.png]
Figure 25
Figure 25. Figure 25: Performance on individual Libero tasks (46-90) in evaluation with task provided. [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]
Figure 26
Figure 26. Figure 26: WidowX Task 1 [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 29
Figure 29. Figure 29: Results on WidowX Task 1. Cloth Pot [PITH_FULL_IMAGE:figures/full_fig_p027_29.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 11 canonical work pages

  1. [1]

    Opal: Offline primitive discovery for acceler- ating offline reinforcement learning

    Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for acceler- ating offline reinforcement learning. arXiv preprint arXiv:2010.13611,

    arXiv 2010

  2. [3]

    pick up the cloth

    Individual results on each WidowX task are given in Appendix B.3. We note theBC fails to succeed a single time on 2/3 tasks, and that our approach achieves the highest success rate and lowest average number of attempts on all tasks. We use delta end-effector control with a frequency of 5 Hz. We use an RGB camera to capture the top-down third-person view o...

    work page

  3. [4]

    (2024) for goal locations

    Please see Figure 2(a) of Wilcoxson et al. (2024) for goal locations. Figure 15: Visualization of D4RL Kitchen environment We utilize the default D4RL environments and datasets in all respects except for the episode length of Antmaze: for Antmaze medium we shorten the episode length from 1000 to 500, and for Antmaze Large we shorten from 1000 to

    work page 2024

  4. [6]

    π0: A vision-language-action flow model for general robot control

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164,

    arXiv

  5. [8]

    Rt-1: Robotics transformer for real-world control at scale

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817,

    arXiv

  6. [10]

    Ex- ploration by random network distillation

    Burda, Y ., Edwards, H., Storkey, A., and Klimov, O. Ex- ploration by random network distillation. arXiv preprint arXiv:1810.12894,

    arXiv

  7. [11]

    Contingency-aware exploration in re- inforcement learning

    Choi, J., Guo, Y ., Moczulski, M., Oh, J., Wu, N., Norouzi, M., and Lee, H. Contingency-aware exploration in re- inforcement learning. arXiv preprint arXiv:1811.01483,

    arXiv

  8. [14]

    L., Sutskever, I., and Abbeel, P

    Duan, Y ., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. Rl 2: Fast reinforcement learn- ing via slow reinforcement learning. arXiv preprint arXiv:1611.02779,

    arXiv

Show all 57 references
  1. [17]

    J., Block, A., and Misra, D

    Foster, D. J., Block, A., and Misra, D. Is behavior cloning all you need? understanding horizon in imitation learning. arXiv preprint arXiv:2407.15007,

    arXiv

  2. [18]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    2004 arXiv

  3. [19]

    Y ., Panitch, W

    Fu, L., Huang, H., Datta, G., Chen, L. Y ., Panitch, W. C.- H., Liu, F., Li, H., and Goldberg, K. In-context imita- tion learning via next-token prediction. arXiv preprint arXiv:2408.15980,

    arXiv

  4. [20]

    Furuta, H., Matsuo, Y ., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364,

    arXiv

  5. [21]

    Learning to reach goals via iterated supervised learning

    Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088,

    1912 arXiv

  6. [22]

    G., Rao, K., Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z., et al

    Gu, J., Kirmani, S., Wohlhart, P., Lu, Y ., Arenas, M. G., Rao, K., Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z., et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977 ,

    arXiv

  7. [23]

    In-context decision transformer: Reinforcement learn- ing via hierarchical chain-of-thought

    Huang, S., Hu, J., Chen, H., Sun, L., and Yang, B. In-context decision transformer: Reinforcement learn- ing via hierarchical chain-of-thought. arXiv preprint arXiv:2405.20692,

    arXiv

  8. [24]

    Learning adap- tive exploration strategies in dynamic environments through informed policy regularization

    Kamienny, P.-A., Pirotta, M., Lazaric, A., Lavril, T., Usunier, N., and Denoyer, L. Learning adap- tive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.02934,

    2005 arXiv

  9. [25]

    J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al

    Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et al. Openvla: An open-source vision-language- action model. arXiv preprint arXiv:2406.09246,

    arXiv

  10. [26]

    J., Zhang, C., and Slivkins, A

    Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore in-context? arXiv preprint arXiv:2403.15371,

    arXiv

  11. [27]

    B., and Levine, S

    Kumar, A., Peng, X. B., and Levine, S. Reward-conditioned policies. arXiv preprint arXiv:1912.13465,

    1912 arXiv

  12. [28]

    In-context reinforcement learning with algorithm distillation

    Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215,

    arXiv

  13. [29]

    S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al

    Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 27921–27936, 2022a. Lee, S., Seo, Y ., Lee, K., Abbeel, ...

    2010 arXiv

  14. [30]

    What matters in learning from offline human demonstrations for robot manipulation

    Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y ., and Mart´ın-Mart´ın, R. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298,

    arXiv

  15. [31]

    A simple neural attentive meta-learner

    Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141,

    arXiv

  16. [32]

    N., Chi, E

    Nie, A., Su, Y ., Chang, B., Lee, J. N., Chi, E. H., Le, Q. V ., and Chen, M. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238,

    arXiv

  17. [33]

    and Clune, J

    Norman, B. and Clune, J. First-explore, then exploit: Meta-learning intelligent exploration. arXiv preprint arXiv:2307.02276,

    arXiv

  18. [34]

    Foundation policies with hilbert representations

    Park, S., Kreiman, T., and Levine, S. Foundation policies with hilbert representations. arXiv preprint arXiv:2402.15567,

    arXiv

  19. [36]

    Vision-based multi-task manipulation for inexpen- sive robots using end-to-end learning from demonstration

    Rahmatizadeh, R., Abolghasemi, P., B¨ol¨oni, L., and Levine, S. Vision-based multi-task manipulation for inexpen- sive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3758–3765. IEEE,

    2018

  20. [37]

    C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R

    13 Behavioral Exploration: Learning to Explore via In-Context Adaptation Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R. Generalization to new sequential decision making tasks with in-context learning. arXiv preprint arXiv:2312.03801,

    arXiv

  21. [39]

    Retrieval-augmented de- cision transformer: External memory for in-context rl

    Schmied, T., Paischer, F., Patil, V ., Hofmarcher, M., Pas- canu, R., and Hochreiter, S. Retrieval-augmented de- cision transformer: External memory for in-context rl. arXiv preprint arXiv:2410.07071,

    arXiv

  22. [40]

    Parrot: Data-driven behavioral priors for re- inforcement learning

    Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for re- inforcement learning. arXiv preprint arXiv:2011.10024,

    2011 arXiv

  23. [41]

    K., Shyam, P., Mutz, F., Ja´skowski, W., and Schmidhuber, J

    Srivastava, R. K., Shyam, P., Mutz, F., Ja´skowski, W., and Schmidhuber, J. Training agents using upside-down re- inforcement learning. arXiv preprint arXiv:1912.02877,

    1912 arXiv

  24. [42]

    C., Levine, S., and Abbeel, P

    Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing ex- ploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814,

    arXiv

  25. [43]

    M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al

    Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213,

    arXiv

  26. [44]

    X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J

    Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763,

    arXiv

  27. [45]

    Leverag- ing skills from unlabeled prior data for efficient online exploration

    Wilcoxson, M., Li, Q., Frans, K., and Levine, S. Leverag- ing skills from unlabeled prior data for efficient online exploration. arXiv preprint arXiv:2410.18076,

    arXiv

  28. [46]

    G., and Wang, Y .-X

    Yan, K., Schwing, A. G., and Wang, Y .-X. Reinforcement learning gradients as vitamin for online finetuning de- cision transformers. arXiv preprint arXiv:2410.24108,

    arXiv

  29. [47]

    Policy expansion for bridging offline-to-online reinforcement learning

    Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935,

    arXiv

  30. [48]

    Learn to effectively explore in context-based meta-rl

    Zhang, J., Wang, J., Hu, H., Chen, Y ., Fan, C., and Zhang, C. Learn to effectively explore in context-based meta-rl. arXiv preprint arXiv:2006.08170,

    2006 arXiv

  31. [49]

    Deep imitation learning for complex manipulation tasks from virtual reality teleoper- ation

    Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Gold- berg, K., and Abbeel, P. Deep imitation learning for complex manipulation tasks from virtual reality teleoper- ation. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 5628–5635. IEEE,

    2018

  32. [50]

    Z., Kumar, V ., Levine, S., and Finn, C

    Zhao, T. Z., Kumar, V ., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705,

    arXiv

  33. [51]

    Autonomous improvement of instruction 14 Behavioral Exploration: Learning to Explore via In-Context Adaptation following skills via foundation models

    Zhou, Z., Atreya, P., Lee, A., Walke, H., Mees, O., and Levine, S. Autonomous improvement of instruction 14 Behavioral Exploration: Learning to Explore via In-Context Adaptation following skills via foundation models. arXiv preprint arXiv:2407.20635,

    arXiv

  34. [52]

    Varibad: A very good method for bayes-adaptive deep rl via meta-learning

    Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y ., Hof- mann, K., and Whiteson, S. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348,

    1910 arXiv

  35. [53]

    The inner induction then immediately implies the outer induction step, that we reach a terminal state at episode k in ¯C t β, so that | ¯C t+1 β | = nβ − t

    Together this proves that (5) holds, which proves the inner inductive step. The inner induction then immediately implies the outer induction step, that we reach a terminal state at episode k in ¯C t β, so that | ¯C t+1 β | = nβ − t. Repeating this for each k ≤ nβ, we have that...

    2024

  36. [56]

    For each task, we run with a horizon of 300 steps, and utilize the environment’s built-in success detector

    Scene Task Commands Upper Left pick up the alphabet soup and put it in the tray pick up the butter and put it in the tray pick up the cream cheese and put it in the tray pick up the ketchup and put it in the tray pick up the tomato sauce and put it in the tray Upper Right clos...

    2024

  37. [750]

    As stated in the text, we evaluate based on the number of goals reached (for Antmaze) or tasks completed (for Kitchen)

    The shorter horizon better tests the ability of each approach to realize directed exploration; with a longer horizon randomly exploring over the maze is itself an affective strategy. As stated in the text, we evaluate based on the number of goals reached (for Antmaze) or tasks...

    2024

  38. [2006]

    K., Yu, T., Singh, A., Phielipp, M., and Finn, C

    Rafailov, R., Vijay, V . K., Yu, T., Singh, A., Phielipp, M., and Finn, C. The reflective explorer: Online meta- exploration from offline data in realistic robotic tasks. In Deep RL Workshop NeurIPS 2021,

    2021

  39. [2011]

    Reinforcement learning upside down: Don’t predict rewards–just map them to actions

    Schmidhuber, J. Reinforcement learning upside down: Don’t predict rewards–just map them to actions. arXiv preprint arXiv:1912.02875,

    1912 arXiv

  40. [2016]

    Gr00t n1: An open foundation model for generalist humanoid robots

    Bjorck, J., Casta˜neda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y ., Fox, D., Hu, F., Huang, S., et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734,

    arXiv

  41. [2017]

    O., and Clune, J

    Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for hard- exploration problems. arXiv preprint arXiv:1901.10995,

    1901 arXiv

  42. [2018]

    Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for dynamics and control conference, pp

    Cui, Y ., Niekum, S., Gupta, A., Kumar, V ., and Rajeswaran, A. Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for dynamics and control conference, pp. 893–905. PMLR, 2022a. Cui, Z. J., Wang, Y ., Shafiullah, N. M. M., and Pint...

    arXiv

  43. [2019]

    Rvs: What is essential for offline rl via supervised learn- ing? arXiv preprint arXiv:2112.10751,

    Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline rl via supervised learn- ing? arXiv preprint arXiv:2112.10751,

    arXiv

  44. [2020]

    Is conditional generative model- ing all you need for decision-making? arXiv preprint arXiv:2211.15657,

    Ajay, A., Du, Y ., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative model- ing all you need for decision-making? arXiv preprint arXiv:2211.15657,

    arXiv

  45. [2021]

    K., and Levine, S

    Dasari, S., Mees, O., Zhao, S., Srirama, M. K., and Levine, S. The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410.10088,

    arXiv

  46. [2022]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  47. [2023]

    Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S

    Beck, J., Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028,

    arXiv

  48. [2024]

    End to end learning for self-driving cars

    Bojarski, M. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316,

    arXiv

  49. [2025]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models

    Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.