Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Habitizing Diffusion Planning for Efficient and Effective Decision Making

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight 'habitized' policy can replace a slow diffusion planner, keeping its D4RL performance while deciding at 800+ Hz on a laptop CPU.

desk verdict Habi's speedup is real and well-measured, but the critic-based selection has a distribution-shift problem that shows up in the N-sweep. read the letter →

arxiv 2502.06401 v1 pith:77J2752I submitted 2025-02-10 cs.LG

Haofei Lu , Yifei Shen , Dongsheng Li , Junliang Xing , Dongqi Han This is my paper · ORCID
classification cs.LG
keywords offlinereinforcementlearningdiffusionplanninghabitizationvariationalBayesevidencelowerbounddecisionfrequencyD4RLcritic-basedactionselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a slow diffusion planner — a model that generates a plan of future states before each action and can take more than a second per decision — can be converted into a fast, lightweight policy without sacrificing decision quality. The conversion, called Habi, mirrors the brain's habitization process: a deliberate, goal-directed planner is compressed into an automatic, habitual policy that maps the current state directly to an action. On the D4RL benchmark, the resulting policy decides at an average of over 800 Hz on a laptop CPU while matching or exceeding the planner it came from. If this holds, the inference cost that has kept diffusion planners out of real-time applications is no longer a blocker, because the expensive planner is needed only during training.

What carries the argument

The carrying object is the ELBO-style habitization objective of Eq. 7, $$L = \|\mathrm{Decoder}(z_q) - a^*\|^2 + \beta_{KL}\,D_{KL}\big(q(z|s,a^*)\,\|\,p(z|s)\big),$$ which binds the habitual prior to the goal-directed posterior in a shared latent space. Around it sit three supporting mechanisms: the adaptive $\beta_{KL}$ schedule of Eq. 8, which sets the KL weight from a target divergence and removes per-task tuning; the critic trained on posterior latents against the planner's pretrained Q-function (Eq. 9); and best-of-$N$ selection at inference (Eq. 12), where $N=5$ prior samples are decoded and the critic picks the winner. The KL term is the channel through which the fast prior learns the slow planner's decision space.

What would settle it

On a task where the prior and posterior latents visibly diverge, record whether the action the critic selects among five prior-sampled candidates actually achieves higher true return than a randomly chosen prior-sampled candidate; if ranking accuracy on prior samples is near chance, the best-of-N selection is not what drives the reported performance.

Watch

Extended reading notes

Core claim

Habi treats the diffusion planner's action choice as the output of a goal-directed posterior and trains a habitual prior to imitate it through a variational objective. A posterior encoder maps the state and the planner's chosen action $a^*$ to a latent $z_q \sim \mathcal{N}(\mu_q, \sigma_q)$; a prior encoder maps only the state to $z_p \sim \mathcal{N}(\mu_p, \sigma_p)$. The habitization loss $$L = \|\mathrm{Decoder}(z_q) - a^*\|^2 + \beta_{KL}\,D_{KL}\big[q(z|s,a^*)\,\|\,p(z|s)\big],$$ an ELBO with adaptively weighted KL (Eqs. 7 and 8), forces the decoder to reproduce $a^*$ from the posterior latent while aligning the prior with the posterior. At deployment only the prior encoder and decoder run: five latents are sampled, decoded into candidate actions, and the best one is picked by a critic trained on the planner's own Q-values (Eqs. 10 to 12). The paper reports that the resulting habitual inference policy matches or beats the corresponding diffusion planner on MuJoCo, FrankaKitchen, AntMaze, and Maze2D from D4RL, at 385 to 1533 Hz on a laptop CPU versus 0.03 to 215 Hz for the diffusion baselines.

Load-bearing premise

The critic that scores candidate actions is trained on latent codes produced with knowledge of the planner's chosen action, but at deployment it scores candidates drawn from a different, state-only distribution of latent codes; the paper assumes, without checking, that the critic's quality judgments stay valid across that gap.

Editorial extensions

If this is right

  • Any pretrained diffusion planner or diffusion policy can be habitized with the same recipe, since the framework needs only the planner's chosen actions and its Q-values.
  • The measured 800+ Hz laptop-CPU frequency means the deployed policy needs no GPU, clearing the latency bar for real-time control on commodity hardware.
  • Habitual inference without critic selection already beats direct distillation of the planner on every task reported, so the ELBO-style objective itself contributes beyond imitation.
  • The adaptive KL target removes per-task hyperparameter search, so one training configuration transfers across locomotion, manipulation, and navigation tasks.
  • Critic-based selection helps most on long-horizon tasks such as AntMaze and Maze2D, indicating that the value of the critic grows with planning depth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the critic is trained on posterior latents but scores prior-sampled candidates at deployment, and the paper does not analyze whether its rankings stay calibrated across that distribution shift; a ranking-accuracy comparison across the two latent sources on each task family would settle it.
  • Editorial extension: because the framework needs only planner actions and planner Q-values, it could be applied recursively, habitizing the already-habitized policy against fresh planning labels, which the paper does not explore.
  • Editorial extension: the biological analogy predicts habitual policies are context-bound, so performance under distribution shift — new maze layouts, sensor noise, or domain randomization — is the untested regime most relevant to real deployment.
  • Editorial extension: the same prior/posterior habitization loss applies verbatim to diffusion policies with horizon one, which is exactly the visuomotor and 3D-policy settings the conclusion names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript introduces Habi, a framework that converts a pretrained diffusion planner into a lightweight habitual policy using variational Bayesian principles. The method trains a prior encoder (from state) and a shared decoder via an ELBO-style loss (Eq. 7), with the planner's selected action as the reconstruction target, and trains a critic (Eq. 9) to evaluate candidate actions. At inference, the prior encoder and decoder sample N candidate actions and the critic selects the best (Eq. 10-12). Experiments on D4RL (MuJoCo, FrankaKitchen, AntMaze, Maze2D) report decision frequencies of 385-1532 Hz on a laptop CPU with performance competitive with or better than diffusion planners (Table 1), and ablations over candidate count (Table 5) and against distillation baselines (Table 2).

Significance. If the results hold, this is a practically valuable contribution: it shows that a pretrained diffusion planner can be distilled into a small MLP policy with orders-of-magnitude faster inference while retaining competitive decision quality. The empirical methodology is a strength: 500 episode seeds per task, 5 training seeds for HI, baselines reproduced via CleanDiffuser for fair comparison, and frequency measurements on multiple devices with different parallelism levels. The variational habitization framing is a nice bridge between cognitive science and practical RL, though the biological analogy is not the main contribution. The central risk is the unexamined distribution shift of the critic between posterior latents (training) and prior latents (inference), which manifests as non-monotonic best-of-N performance in Table 5 and creates uncertainty about when the selection mechanism helps.

major comments (4)
  1. [§3.4-3.5, Eq. (9), Eq. (25), Eq. (10)-(12)] The critic is trained exclusively on posterior latents zq = PosteriorEncoder(s, a*_i) with planner-provided Q-values as targets (Eq. 9 and Eq. 25), but at inference it is evaluated on prior latents zp ~ N(µp, σp) and their decoded actions (Eq. 10-12). The KL term (Eq. 6) drives the marginal prior and posterior toward Dtar_KL = 1.0, but it does not guarantee that the critic remains calibrated on the prior-conditioned input distribution. Since the best-of-N mechanism (Eq. 12) is central to the method's performance, this distribution shift is load-bearing. The paper should provide a direct calibration check, e.g., comparing Critic(zp, Decoder(zp)) against the true discounted return of that action, or an ablation that trains the critic on prior latents, to rule out selection-driven overestimation.
  2. [§4.6, Table 5] The claim in §4.6 that 'selecting from multiple candidates consistently improves decision quality across various tasks' is not supported by Table 5. With the default N = 5, HI is worse than N = 1 on several tasks: Diverse Antmaze-Medium (88.8 vs 92.1), Play Antmaze-Medium (85.3 vs 88.0), and Large Maze2D (199.2 vs 201.8). The sweep is also non-monotonic over N: Hopper Medium-Expert rises from 85.9 at N=1 to 92.7 at N=2 then falls to 74.0 at N=100, and Diverse Antmaze-Medium falls from 92.4 at N=2 to 64.7 at N=100. This internal evidence is consistent with the critic overestimating low-quality prior-sampled actions as the candidate pool grows. The paper should either restrict N to a range where selection is reliably helpful and justify that choice, or substantially qualify the 'consistently improves' statement.
  3. [§4.3, Table 2] The statement in §4.3 that 'even without the critic, Habi consistently outperforms standard distillation across all tasks' is contradicted by Table 2. HI without critic scores lower than Direct Distill on HalfCheetah Medium-Expert (96.9 vs 97.1), Hopper Medium (98.5 vs 102.2), Walker2d Medium (91.4 vs 91.9), Mixed Kitchen (66.6 vs 69.1), and Diverse Antmaze-Large (3.8 vs 72.0). The aggregate result (87.5 vs 85.0) still favors HI without critic, but the 'across all tasks' claim is factually wrong and should be corrected to a claim about average performance.
  4. [§1, §2.2, Table 3] The paper repeatedly describes Habi as a 'general framework' applicable to 'any diffusion planning and diffusion policy models' (§1, §2.2), but the experiments habitize only two teachers: DQL for MuJoCo and AntMaze, and DV for Kitchen and Maze2D (Table 3). The method also assumes the teacher provides a pre-trained Q or value function, which is not available for all planners in the comparison set. To substantiate the generality claim, the authors should either habitize at least one additional planner family (e.g., Diffuser or Decision Diffuser) or soften the claim to indicate that the framework is applicable in principle to planners that provide a value function.
minor comments (6)
  1. [§4.4] The word 'visualing' should be 'visualizing'.
  2. [App. C.1] The word 'implemneted' should be 'implemented'.
  3. [Table 4 note] The phrase 'decison-making' should be 'decision-making'.
  4. [App. B] Eq. (13) uses 'log Ez∼p(z)P(x|z)' while Eq. (14) writes 'log Ez∼p(z)P(x)'; the notation is inconsistent and should be unified.
  5. [§3.3] The text reads 'we provide implementation details in Appendix Appendix C.3'; the duplicate 'Appendix' should be removed.
  6. [§3.4] The output of the critic is described as similar to a Q-function, but for some teachers (e.g., DV) the target is a value function; please clarify whether the regressed quantity is a state-action value or a state value.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central speed and performance claims are measured against external D4RL benchmarks, and the ELBO-style derivation is standard variational Bayes rather than an imported or fitted result.

full rationale

The paper's central claims are empirical and externally evaluated: the 800+ Hz decision frequency is measured on laptop CPU/GPU against independent baselines, and the 'comparable or even higher performance' is measured by environment return on D4RL, not derived from the training objective by construction. The habitized policy is trained to imitate the frozen diffusion planner's selected actions via Eq. 4 and Eq. 24, but the evaluation is a separate online rollout, so the performance claim is not statistically forced by the fit. The ELBO-style objective (Eq. 7) is justified by the standard variational argument in Appendix B, which uses importance sampling and Jensen's inequality; it does not depend on any special theorem from the authors' prior work. The self-citations to Han et al. [27] and Diffusion Veteran [45] supply conceptual framing and a teacher model, respectively, but neither is load-bearing for the derivation: [27]'s habit/goal framework is an interpretation, and [45] is an external, measurable teacher whose outputs are compared rather than assumed correct. The paper also discloses its scope limitation (state-based tasks only) in Section 6, which is an honest scope statement rather than a circular step. The main technical risk noted by the manuscript itself is the critic's potential distribution shift from posterior latents in training (Eq. 9/25) to prior latents in inference (Eq. 12), and Table 5 shows non-monotonic best-of-N performance; however, this is a calibration and correctness concern, not a circular reduction of the result to its inputs. Overall, no equation or claim in the derivation chain reduces to its own inputs or to an unverified self-citation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on standard variational inference plus several domain assumptions about teacher quality, latent capacity, and critic calibration. The hand-set hyperparameters (Dtar_KL, N, latent dim) are free parameters that the central result depends on but that are not derived from data.

free parameters (5)
  • Target KL divergence (Dtar_KL) = 1.0
    Global target for adaptive beta_KL (Eq. 8, Table 3). Chosen by hand; controls the balance between reconstruction and prior alignment across all tasks.
  • Number of inference candidates (N) = 5
    Number of prior latents sampled and scored by the critic during Habitual Inference (Eq. 12, Table 3). Selected based on the ablation in Figure 7.
  • Number of training candidates = 50
    Candidate actions from the planner used to train the critic (Eq. 25, Table 3).
  • Latent dimension (Dim(z)) = 256
    Dimensionality of the Gaussian latent variable in both prior and posterior encoders (Table 3). Architecture choice that affects capacity.
  • KL stability epsilon = 0.01
    Additive constant in softplus for sigma (Eq. 20) to ensure numerical stability.
assumptions (4)
  • domain assumption The pretrained diffusion planners (DQL for locomotion/AntMaze, DV for Kitchen/Maze2D) and their value functions provide reliable action targets and Q-values for training the student and critic.
    The entire method inherits the teacher's quality; no independent verification of teacher optimality is provided (Sections 3.3-3.4, Table 3).
  • domain assumption A diagonal Gaussian latent of dimension 256 with MLP encoder/decoder is expressive enough to represent the action distributions of the diffusion planners on the tested tasks.
    No capacity analysis is given; the reconstruction loss (Eq. 4) is assumed optimizable to sufficient accuracy (Sections 3.3, Appendix C.1).
  • domain assumption The critic trained on posterior latents generalizes to prior latents at inference.
    Section 3.4 trains Critic(zq, a*) while Section 3.5 applies Critic(zp, a); the shift is not analyzed or corrected.
  • standard math The ELBO inequality (Eq. 13-19) is a valid lower bound on log P(x), justifying the objective.
    Standard variational inference result, proven in Appendix B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Habitizing Diffusion Planning for Efficient and Effective Decision Making." pith.science (2026). https://pith.science/paper/77J2752I

@misc{pith2026250206401,
  author       = {Pith},
  title        = {Pith review of: Habitizing Diffusion Planning for Efficient and Effective Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77J2752I}},
  note         = {Machine review of arXiv:2502.06401}
}
read the original abstract

Diffusion models have shown great promise in decision-making, also known as diffusion planning. However, the slow inference speeds limit their potential for broader real-world applications. Here, we introduce Habi, a general framework that transforms powerful but slow diffusion planning models into fast decision-making models, which mimics the cognitive process in the brain that costly goal-directed behavior gradually transitions to efficient habitual behavior with repetitive practice. Even using a laptop CPU, the habitized model can achieve an average 800+ Hz decision-making frequency (faster than previous diffusion planners by orders of magnitude) on standard offline reinforcement learning benchmarks D4RL, while maintaining comparable or even higher performance compared to its corresponding diffusion planner. Our work proposes a fresh perspective of leveraging powerful diffusion models for real-world decision-making tasks. We also provide robust evaluations and analysis, offering insights from both biological and engineering perspectives for efficient and effective decision-making.

Figures

Figures reproduced from arXiv: 2502.06401 by the authors.

Figure 1
Figure 1. Performance vs. Frequency. Performance is normal￾ized across MuJoCo, AntMaze, and Kitchen tasks from D4RL. Decision frequency (Hz) is measured on a laptop CPU (Apple M2, MacBook). Habitual Inference (HI), a lightweight model generated by our Habi, achieves an optimal balance between performance and speed. See Tab. 1 for more results. models remains a significant challenge – models like dif￾fuser and its variants usu… view at source ↗
Figure 2
Figure 2. An illustrative example of the process of habitization in playing the Minesweeper game. With practice, one’s decision￾making relies less on deliberate goal-directed planning and more on context-dependent habitual behavior. slow, deliberate goal-directed behavior into fast, habitual behavior when repetitively doing a task (see [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The diagram of Habi. (a) During the Habitization (Training) stage, Habi learns to reconstruct actions from plans generated by a diffusion planner, with the decision spaces of habits (prior) and planning (posterior) aligned via KL divergence in the latent space. Trainable parts include Prior Encoder, Posterior Encoder, Decoder, and Critic. (b) During the Habitual Inference (HI) stage, only the lightweight prior encod… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Visualized results of Table [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Action distributions of Diffusion Planner (DV) and Habitual Inference (HI). Visualization of the action distributions from a state-of-the-art diffusion planner (DV, 2.8Hz, top) [45] and its corresponding Habitual Inference policy (HI, 1532.6Hz, middle) generated by our…
Figure 6
Figure 6. Figure 6: Performance Comparison of Searching βKL and our Adaptive βKL Mechanism. Fixed βKL requires grid search for each task to achieve peak performance, while our adaptive βKL achieves comparable performance without task-specific tuning, as shown by the consistent red dashed …
Figure 7
Figure 7. Figure 7: Effect of the number of sampling candidates on performance. Increasing the number of candidates N improves performance initially, but excessive candidates (N ≥ 50) lead to diminishing returns or slight degradation. A moderate choice (e.g., N = 5) provides a good balanc…
Figure 8
Figure 8. Figure 8: Rendering of the benchmarking tasks considered in this study. The tasks encompass a diverse set of evaluation metrics, including locomotion tasks that emphasize short-term planning, robotic arm tasks requiring long-term strategic planning, and navigation tasks focused …
Figure 9
Figure 9. Figure 9: Action distribution visualization on Kitchen, plotted the same way as Figure 5. The actions are dimension-reduced by PCA. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Another example of action distribution visualization on Kitchen, plotted the same way as Figure 5. The actions are dimension [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Action distribution visualization on AntMaze, plotted the same way as Figure 5. The actions are dimension-reduced by PCA. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Action distribution visualization on MuJoCo, plotted the same way as Figure 5. The actions are dimension-reduced by PCA. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Another example of action distribution visualization on Maze2D, plotted the same way as Figure 5. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 50 canonical work pages

  1. [1]

    Is con- ditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2022

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenen- baum, Tommi S Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2022. 1, 2, 5

    work page 2022

  2. [2]

    Deep variational information bottle- neck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottle- neck. In International Conference on Learning Repre- sentations, 2017. 3, 4

    work page 2017

  3. [3]

    Uncertainty-based offline reinforce- ment learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforce- ment learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436– 7447, 2021. 8

    work page 2021

  4. [4]

    A Markovian decision process.Jour- nal of Mathematics and Mechanics, pages 679–684,

    Richard Bellman. A Markovian decision process.Jour- nal of Mathematics and Mechanics, pages 679–684,

    work page

  5. [5]

    Offline reinforcement learning via high- fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high- fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representa- tions, 2023. 2

    work page 2023

  6. [6]

    Score regularized policy optimization through diffusion behavior

    Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 5, 6, 8

    work page 2024

  7. [7]

    Diffusion policies creating a trust region for offline reinforcement learning

    Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. arXiv preprint arXiv:2405.19690, 2024. 5, 8

    arXiv 2024

  8. [8]

    Diffusion policy: Visuomotor policy learn- ing via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shu- ran Song. Diffusion policy: Visuomotor policy learn- ing via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 2, 3, 8

    work page 2023

Show all 66 references
  1. [9]

    A recurrent latent variable model for sequential data

    Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, pages 2980–2988, 2015. 8

    2015

  2. [10]

    The computational cost of active information sampling before decision-making under uncertainty

    Luke Clark, Trevor W Robbins, Karen D Ersche, and Barbara J Sahakian. The computational cost of active information sampling before decision-making under uncertainty. Nature Human Behaviour, 5(6):789–800,

  3. [11]

    Goals and habits in the brain

    Ray J Dolan and Peter Dayan. Goals and habits in the brain. Neuron, 80(2):312–325, 2013. 1, 3

    2013

  4. [12]

    Diffuserlite: Towards real-time diffusion planning

    Zibin Dong, Jianye HAO, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and YAN ZHENG. Diffuserlite: Towards real-time diffusion planning. In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. 5, 8

    2024

  5. [13]

    Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making

    Zibin Dong, Yifu Yuan, Jianye HAO, Fei Ni, Yi Ma, Pengyi Li, and YAN ZHENG. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 5, 6, 14

    2024

  6. [14]

    Learning universal policies via text- guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- guided video generation. Advances in Neural Informa- tion Processing Systems, 36, 2024. 1, 8

    2024

  7. [15]

    Virel: A variational inference framework for reinforcement learning

    Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019. 8

    2019

  8. [16]

    A free energy principle for the brain

    Karl Friston, James Kilner, and Lee Harrison. A free energy principle for the brain. Journal of Physiology- Paris, 100(1-3):70–87, 2006. 3

    2006

  9. [17]

    D4rl: Datasets for deep data-driven reinforcement learning, 2020

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020. 1, 2, 5

    2020

  10. [18]

    Off- policy deep reinforcement learning without explo- ration

    Scott Fujimoto, David Meger, and Doina Precup. Off- policy deep reinforcement learning without explo- ration. In International Conference on Machine Learn- ing, pages 2052–2062. PMLR, 2019. 2

    2019

  11. [19]

    V oce: Variational optimization with conservative estimation for offline safe reinforcement learning

    Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhi- jun Li, et al. V oce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 8

    2024

  12. [20]

    Recurrent world models facilitate policy evolution

    David Ha and J¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neu- ral Information Processing Systems 31, pages 2450–

  13. [21]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018. 2, 4

    2018 arXiv

  14. [22]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018. 8

    2018 arXiv

  15. [23]

    Dream to control: Learning 9 behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning 9 behaviors by latent imagination. In International Con- ference on Learning Representations, 2019. 1

    2019

  16. [24]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Tim- othy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 1, 8

    2023 arXiv

  17. [25]

    Variational recurrent models for solving partially observable con- trol tasks

    Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable con- trol tasks. In International Conference on Learning Representations, 2020. 8

    2020

  18. [26]

    Variational oracle guiding for reinforcement learning

    Dongqi Han, Tadashi Kozuno, Xufang Luo, Zhao-Yun Chen, Kenji Doya, Yuqing Yang, and Dongsheng Li. Variational oracle guiding for reinforcement learning. In International Conference on Learning Representa- tions, 2022. 4, 8

    2022

  19. [27]

    Synergizing habits and goals with variational Bayes

    Dongqi Han, Kenji Doya, Dongsheng Li, and Jun Tani. Synergizing habits and goals with variational Bayes. Nature Communications, 15(1):4461, 2024. 1, 2, 3, 8

    2024

  20. [28]

    Idql: Implicit q-learning as an actor-critic method with diffu- sion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Jan- ner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffu- sion policies. arXiv preprint arXiv:2304.10573, 2023. 2, 4, 5

    2023 arXiv

  21. [29]

    β-V AE: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. β-V AE: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017. 4

    2017

  22. [30]

    Deep variational re- inforcement learning for POMDPs

    Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational re- inforcement learning for POMDPs. arXiv preprint arXiv:1806.02426, 2018. 8

    2018 arXiv

  23. [31]

    Offline reinforcement learning as one big sequence modeling problem

    Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Process- ing Systems, 2021. 1

    2021

  24. [32]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902–9915. PMLR, 2022. 2, 4, 5

    2022

  25. [33]

    Thinking, fast and slow

    Daniel Kahneman. Thinking, fast and slow. macmillan,

  26. [34]

    Pri- mate ventral striatum maintains neural representations of the value of previously rewarded objects for habitual seeking

    Joonyoung Kang, Hyeji Kim, Seong Hwan Hwang, Minjun Han, Sue-Hyun Lee, and Hyoung F Kim. Pri- mate ventral striatum maintains neural representations of the value of previously rewarded objects for habitual seeking. Nature communications, 12(1):2100, 2021. 4

    2021

  27. [35]

    Auto-encoding variational Bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014. 2, 3

    2014

  28. [36]

    Of- fline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Of- fline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021. 2

    2021 arXiv

  29. [37]

    On infor- mation and sufficiency

    Solomon Kullback and Richard A Leibler. On infor- mation and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951. 3, 4

    1951

  30. [38]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Informa- tion Processing Systems, 33:1179–1191, 2020. 2

    2020

  31. [39]

    Neural computations underlying arbitra- tion between model-based and model-free learning

    Sang Wan Lee, Shinsuke Shimojo, and John P O’Doherty. Neural computations underlying arbitra- tion between model-based and model-free learning. Neuron, 81(3):687–699, 2014. 2

    2014

  32. [40]

    Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018. 8

    2018

  33. [41]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. 1, 2

    2005 arXiv

  34. [42]

    Hierarchical diffusion for offline decision making

    Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning , pages 20035–20064. PMLR, 2023. 2

    2023

  35. [43]

    Adaptdiffuser: Diffusion models as adaptive self-evolving planners

    Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. ICML, 2023. 1, 2, 4, 5

    2023

  36. [44]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022. 7

    2022

  37. [45]

    What makes a good diffusion planner for decision making? In The Thirteenth International Conference on Learning Representations, 2025

    Haofei Lu, Dongqi Han, Yifei Shen, and Dongsheng Li. What makes a good diffusion planner for decision making? In The Thirteenth International Conference on Learning Representations, 2025. 1, 3, 4, 5, 6, 7, 14, 15

    2025

  38. [46]

    Bridging state and history repre- sentations: Understanding self-predictive RL

    Tianwei Ni, Benjamin Eysenbach, Erfan SeyedSalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging state and history repre- sentations: Understanding self-predictive RL. In The Twelfth International Conference on Learning Repre- sentations, 2024. 8

    2024

  39. [47]

    The hip- pocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat

    John O’Keefe and Jonathan Dostrovsky. The hip- pocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain research,

  40. [48]

    Mazes, maps, and memory

    David S Olton. Mazes, maps, and memory. American psychologist, 34(7):583, 1979. 3

    1979

  41. [49]

    Barron, and Ben Mildenhall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d dif- fusion. In The Eleventh International Conference on Learning Representations, 2023. 7 10

    2023

  42. [50]

    Learning a diffusion model policy from rewards via Q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via Q-score matching. arXiv preprint arXiv:2312.11752, 2023. 2

    2023 arXiv

  43. [51]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Ben- jamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024. 2

    2024 arXiv

  44. [52]

    Proximal policy optimiza- tion algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.06347 ,

    arXiv

  45. [53]

    Cost-benefit trade-offs in decision-making and learning

    Nura Sidarus, Stefano Palminteri, and Val´erie Cham- bon. Cost-benefit trade-offs in decision-making and learning. PLoS Computational Biology , 15(9): e1007326, 2019. 1

    2019

  46. [54]

    Learn- ing structured output representation using deep condi- tional generative models

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learn- ing structured output representation using deep condi- tional generative models. Advances in neural informa- tion processing systems, 28, 2015. 3

    2015

  47. [55]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 7

    2010 arXiv

  48. [56]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press Cambridge,

  49. [57]

    Diffu- sion actor-critic with entropy regulator

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Lim- ing Xiao, Jiang Wu, Jingliang Duan, et al. Diffu- sion actor-critic with entropy regulator. arXiv preprint arXiv:2405.15177, 2024. 2

    2024 arXiv

  50. [58]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh In- ternational Conference on Learning Representations,

  51. [59]

    Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural In- formation Processing Systems, 36, 2024. 7

    2024

  52. [60]

    Psychology of habit

    Wendy Wood and Dennis R¨unger. Psychology of habit. Annual review of psychology, 67:289–314, 2016. 1, 2, 3

    2016

  53. [61]

    Policy representation via diffusion probability model for reinforcement learning

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122, 2023. 2

    2023 arXiv

  54. [62]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems , 35:23851–23866, 2022. 8

    2022

  55. [63]

    Sequential generative exploration model for partially observable reinforcement learning

    Haiyan Yin, Jianda Chen, Sinno Jialin Pan, and Sebas- tian Tschiatschek. Sequential generative exploration model for partially observable reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10700–10708, 2021. 8

    2021

  56. [64]

    Generalizable humanoid manipulation with improved 3D diffusion policies

    Yanjie Ze, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Jiajun Wu. Generalizable humanoid manipulation with improved 3D diffusion policies. arXiv preprint arXiv:2410.10803, 2024. 2, 3

    2024 arXiv

  57. [65]

    3D diffusion policy

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3D diffusion policy. arXiv preprint arXiv:2403.03954, 2024. 2, 3, 8 11 A. Benchmarking Tasks As shown in Figure 8, we consider a diverse set of benchmarking tasks to evaluate the performance of Habi. ...

    2024 arXiv

  58. [2462]

    Curran Associates, Inc., 2018. 1, 8

    2018

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.