Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Trajectory Entropy Reinforcement Learning for Predictable and Robust Control

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Minimizing trajectory entropy makes reinforcement-learned policies more predictable, higher-scoring, and more robust to perturbation.

desk verdict A useful incremental regularizer with strong locomotion results, but the trajectory-entropy mechanism is asserted more strongly than the evidence supports. read the letter →

arxiv 2505.04193 v1 pith:NBCX4MEM submitted 2025-05-07 cs.LG cs.ROstat.ML

Bang You , Chenxu Wang , Huaping Liu This is my paper · ORCID
classification cs.LGcs.ROstat.ML
keywords trajectoryentropysimplicityinductivebiasreinforcementlearningrobustcontrolactionpredictionvariationalinferencelocomotionpredictablepolicies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that adding a trajectory-entropy regularizer to a deep reinforcement learning objective biases policies toward simple, repetitive action sequences, and that this improves both reward and robustness on high-dimensional locomotion tasks. It estimates the trajectory entropy $H(a_{1:T-1} | z_{1:T})$ with a learned variational action prediction model $q_\psi(a_t | z_t, z_{t+1}, a_{t-1})$, then rewards actions that this model can anticipate. The resulting method, Trajectory Entropy Reinforcement Learning (TERL), is built as an extension of soft actor-critic in which the environment reward is augmented with an information-based bonus. On six simulated locomotion tasks, the learned policies produce more cyclical and consistent gaits, achieve higher average scores than the baselines on most tasks, and show smaller performance drops under mass changes, action noise, and observation noise.

What carries the argument

The central object is the trajectory entropy $H(a_{1:T-1} | z_{1:T})$, the number of bits needed to describe an action sequence after seeing the corresponding state-representation sequence. Since this quantity is intractable, the paper introduces an autoregressive variational distribution $q_\psi(a_t | z_t, z_{t+1}, a_{t-1})$ that upper-bounds it, and turns the bound into an information-regularized reward $r^*(s_t, a_t) = r(s_t, a_t) + \alpha \log q_\psi(a_t | z_t, z_{t+1}, a_{t-1})$. This reward bonus pushes the policy toward actions that are easy to predict from the previous action and two consecutive state representations, which in practice produces periodic, compressible behaviors.

What would settle it

Estimate the true trajectory entropy of a trained TERL policy by Monte Carlo sampling with a high-capacity conditional density estimator and compare it to the variational upper bound from Eq. (5); if the bound is loose or the true entropy does not decrease during training, the method is not actually optimizing the stated objective. Alternatively, replace $q_\psi$ with a fixed or random predictor during training: if the reported performance and robustness gains persist, the entropy-minimization objective is not the cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that minimizing the conditional entropy of entire action trajectories given state-representation trajectories is a tractable and effective simplicity inductive bias for reinforcement learning. Because the true conditional distribution of actions is unknown, the entropy is replaced by a variational upper bound built from an autoregressive action prediction model, and this upper bound is folded into the reward. The discovery, as the authors state it, is that policies optimized for this information-regularized reward are more predictable, achieve higher scores on the majority of the evaluated locomotion tasks, and degrade more gracefully under environmental perturbations than policies trained by prior baseline methods.

Load-bearing premise

The learned prediction model $q_\psi$ is assumed to estimate the true conditional distribution of actions accurately enough that $\alpha \log q_\psi$ is a faithful surrogate for reducing true trajectory entropy; if this assumption fails, the reward bonus rewards actions that are predictable to the learned model rather than actions that actually lower trajectory entropy.

Editorial extensions

If this is right

  • Policies trained with TERL produce action trajectories that are more cyclical and consistent, and that compress to smaller sizes under a lossless compression algorithm such as bzip2.
  • TERL achieves higher average scores than LZ-SAC, RPC, and SAC on the majority of the six locomotion tasks, with the largest gains on the high-dimensional H1 Walk and Humanoid Walk tasks.
  • TERL shows smaller performance drops than the baselines under mass scaling, action noise, and observation noise, especially at small noise strengths.
  • Because SAC is a special case of TERL with $\alpha = 0$, the reported improvements are attributed specifically to the trajectory entropy regularizer rather than to the underlying actor-critic machinery.
  • The simplicity bias can harm performance on tasks that inherently demand complex and unpredictable behavior, a limitation the paper itself acknowledges.
  • The hyperparameter $\alpha$ trades off reward against trajectory entropy, and increasing it monotonically increases the compressibility of learned behavior on the Walker Walk task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the policy is Markov and the variational model factorizes autoregressively, the trajectory-level entropy reduces to a sum of per-step conditional entropies, so the trajectory formulation may not be essential; a per-step action-prediction bonus could produce equivalent behavior.
  • Editorial inference: if the benefit comes from smoothness and compressibility, simpler regularizers such as action-rate penalties or Lipschitz constraints on the policy might match TERL's results, and a direct comparison would clarify whether the information-theoretic form adds anything beyond these.
  • Editorial inference: the robustness gains are reported zero-shot in simulation, which suggests a concrete testable extension of transferring TERL-trained policies to physical legged robots with added mass and actuator noise.
  • Editorial inference: the paper relies on grid search to set $\alpha$; a testable improvement would be to tie $\alpha$ to task properties such as action dimension or state dimension, which the authors list as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Trajectory Entropy Reinforcement Learning (TERL), a regularizer for SAC that penalizes an estimate of the conditional entropy of action trajectories given a learned state-representation trajectory. The claimed motivation is that low trajectory entropy induces simple, predictable, and robust policies. The authors derive a variational upper bound on this entropy using an autoregressive action-prediction model q_psi, add alpha times log q_psi to the reward, and jointly train the policy, encoder, and prediction model. Experiments on six continuous-control locomotion tasks compare TERL with SAC, LZ-SAC, RPC, and the original RPC implementation, reporting higher average scores on most tasks, smaller performance drops under mass/gravity changes, action noise, and observation noise, and higher bzip2 compressibility of action trajectories. The derivation in Eq. (3) is correct as stated, and the empirical study is substantial (20 seeds per task, 10^6 training steps).

Significance. If the mechanism were established, the paper would offer a simple, task-independent inductive bias for robustness in high-dimensional control, with potential value for legged locomotion and human-interactive systems. The strengths are the correct variational inequality in Eq. (3), the use of a public SAC codebase, the breadth of tasks, and the falsifiable compression proxy. However, the central causal claim is currently supported only indirectly: the paper never measures the gap between the true trajectory entropy and the variational upper bound, and the reward model is trained on the agent's own replay buffer while being used to shape the policy. These issues do not invalidate the empirical results, but they leave the interpretation of the gains as 'trajectory entropy minimization' unverified. The paper's own limitation paragraph in Section VI acknowledges that the simplicity bias can hurt performance on tasks requiring complex behavior and that alpha is chosen by grid search, but it does not acknowledge the bound-mismatch or co-adaptation concerns.

major comments (3)
  1. [Section IV-A/B, Eqs. 3-6] The implemented objective is not the stated objective, and the direction of the approximation is not flagged. Eq. (3) gives H <= H_u, so the objective in Eq. (1), E[sum r] - alpha H, satisfies E[sum r] - alpha H >= E[sum r] - alpha H_u. Eq. (6) maximizes E[sum(r + alpha log q_psi)] = E[sum r] - alpha H_u, which is a lower bound of Eq. (1), not an equivalent form. The slack is the expected KL divergence D_KL(p(a_{1:T-1}|z_{1:T}) || q_psi(a_{1:T-1}|z_{1:T})), and this gap is never measured. Because q_psi is trained on the agent's own replay buffer and simultaneously used as the reward model, the policy and the estimator can co-adapt, so reducing H_u need not reduce the true conditional entropy H. Please report an empirical estimate of the gap (for example, held-out log-likelihood of recorded trajectories, or a comparison between H_u and a direct Monte Carlo entropy estimate), and revise the wording so the paper does not claim to minimize H itself. I note that the trajectory-level entropy is not identical to the sum of per-step entropies, because future states z_{t+1:T} can carry information about a_t under the Markov policy; the load-bearing issue is the unmeasured bound tightness, not the factorization.
  2. [Section IV-C, Eq. 11, and Figure 3] The encoder e_phi is optimized jointly with the policy and the prediction model. Since z_{t+1} is a function of s_{t+1}, and s_{t+1} depends on a_t, the encoder can reduce the upper bound by encoding information about the current action into the next representation, rather than by making the policy's action distribution more predictable. In the extreme, if z_{t+1} encodes a_t, then q_psi can predict a_t almost perfectly regardless of how erratic the policy is. This means the reported low conditional entropy may reflect representation leakage rather than simplicity of the policy. The paper should add a control experiment with a fixed or information-bottlenecked encoder, or separately report the marginal entropy of action sequences and the conditional entropy given z, to show that the regularizer is not being gamed by the learned representation.
  3. [Section V-C, Tables IV-VI, and Figure 5] The robustness comparison is presented primarily as percentage performance drops relative to each method's unperturbed score. Because TERL's unperturbed scores in Table III are much higher than those of the baselines, the percentage metric can look favorable even when absolute losses are comparable or larger. The aggregated normalized curves in Figure 5 are more informative, but the paper reports no confidence intervals or significance tests for the drop percentages. Please report raw perturbed rewards with intervals (the 20 seeds would support paired bootstrap or Wilcoxon tests), and state whether the differences in Tables IV-VI are statistically significant.
minor comments (5)
  1. [Section III, Eq. (2)] The text after Eq. (2) says the expectation is over the joint distribution of action sequences and 'representation sequences a_{1:T-1}'; the second occurrence should be z_{1:T}.
  2. [Algorithm 1, Section IV-B] In Algorithm 1, the line 'Compute upper bound: u <- E[log q_psi]' is inconsistent with Eq. (7), where the reward bonus is the per-transition log q_psi(a_t|z_t,z_{t+1},a_{t-1}), not an expectation over a batch. Please clarify whether u is a per-sample value or a batch mean.
  3. [Section V-C, first paragraph] The text says the robustness test 'scales the gravity of the robot body,' while the tables and captions describe 'mass changes'; these are different perturbations and the paper should use consistent terminology.
  4. [Section V-A and Table VII] The hyperparameter search is described as a grid over alpha in [10^-5, 10^-4], but Table VII sweeps alpha = 0.001 and 0.01. Please state whether the ablation extends outside the tuned range and whether those larger values were considered for the other tasks.
  5. [Figure 3 caption] The caption says the joint optimization uses the 'lower bound of the trajectory entropy,' while Eq. (5) derives an upper bound; the figure caption should match the equation.

Circularity Check

0 steps flagged · score 0.0 of 10
Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method's central claim rests on q_psi being a faithful entropy estimator and on per-task tuning of alpha. No new physical entities are introduced.

free parameters (1)
  • alpha (trajectory entropy coefficient) = 10^-5 for most tasks, 10^-4 for Walker Walk
    Selected by grid search over two values to maximize task performance; this fitted constant directly controls the strength of the simplicity bias.
assumptions (4)
  • domain assumption The learned encoder e_phi preserves the information needed for optimal control, stated as 'without loss of generality' in Section III.
    If the 30-dimensional representation z_t is insufficient, the policy is suboptimal and comparisons to baselines are not apples-to-apples.
  • domain assumption The variational distribution q_psi closely approximates the true conditional action distribution p(a1:T-1|z1:T).
    Eq. 3 is an upper bound that is tight only when q_psi equals p; the paper never measures this gap.
  • ad hoc to paper The autoregressive factorization q_psi(a_t|z_t,z_{t+1},a_{t-1}) is a sufficiently expressive model for the action trajectory.
    This specific conditioning choice is introduced by the authors; no evidence shows it is the right inductive bias compared to simpler predictors.
  • standard math Non-negativity of KL divergence used in Eq. 3.
    Standard result; no concern.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trajectory Entropy Reinforcement Learning for Predictable and Robust Control." pith.science (2026). https://pith.science/paper/NBCX4MEM

@misc{pith2026250504193,
  author       = {Pith},
  title        = {Pith review of: Trajectory Entropy Reinforcement Learning for Predictable and Robust Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBCX4MEM}},
  note         = {Machine review of arXiv:2505.04193}
}
read the original abstract

Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive results of deep reinforcement learning in complex control tasks, it is prone to capturing intricate and spurious correlations between observations and actions, leading to failure under slight perturbations to the environment. To tackle this problem, in this work we introduce a novel inductive bias towards simple policies in reinforcement learning. The simplicity inductive bias is introduced by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories. Our reinforcement learning agent, Trajectory Entropy Reinforcement Learning, is optimized to minimize the trajectory entropy while maximizing rewards. We show that the trajectory entropy can be effectively estimated by learning a variational parameterized action prediction model, and use the prediction model to construct an information-regularized reward function. Furthermore, we construct a practical algorithm that enables the joint optimization of models, including the policy and the prediction model. Experimental evaluations on several high-dimensional locomotion tasks show that our learned policies produce more cyclical and consistent action trajectories, and achieve superior performance, and robustness to noise and dynamic changes than the state-of-the-art.

Figures

Figures reproduced from arXiv: 2505.04193 by the authors.

Figure 1
Figure 1. Performance comparison, and action trajectory visualizations of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our TERL agent minimizes the entropy of action trajectories [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We use one objective to jointly optimize our policy, the encoder, and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: We evaluate our method and previous methods on six high-dimensional [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Zero-shot robustness to changes in body gravity (left), action noise (middle), and observation noise (right) on 6 locomotion control tasks. This plot [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: We visualize action sequences generated by our method and baselines on the Walker Walk task. Behaviors produced by our agent show more periodic [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualizations of the gaits produced by our method and baselines on the Walker Walk task. Gaits produced by our method show more cyclical and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: We compress trajectories produced by TERL and baselines on all DMC [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 45 canonical work pages

  1. [1]

    Neural manifold modulated continual reinforcement learning for musculoskeletal robots,

    J. Chen, Z. Chen, C. Yao, and H. Qiao, “Neural manifold modulated continual reinforcement learning for musculoskeletal robots,” IEEE Trans. Cogn. Devel. Syst. , vol. 16, no. 1, pp. 86–99, 2022

    work page 2022

  2. [2]

    Supervised meta- reinforcement learning with trajectory optimization for manipulation tasks,

    L. Wang, Y . Zhang, D. Zhu, S. Coleman, and D. Kerr, “Supervised meta- reinforcement learning with trajectory optimization for manipulation tasks,” IEEE Trans. Cogn. Devel. Syst. , vol. 16, no. 2, pp. 681–691, 2023

    work page 2023

  3. [3]

    Visual spatial attention and proprioceptive data-driven reinforcement learning for robust peg-in-hole task under variable conditions,

    A. Y . Yasutomi, H. Ichiwara, H. Ito, H. Mori, and T. Ogata, “Visual spatial attention and proprioceptive data-driven reinforcement learning for robust peg-in-hole task under variable conditions,” IEEE Robot. Autom. Lett., vol. 8, no. 3, pp. 1834–1841, 2023

    work page 2023

  4. [4]

    Deep reinforcement learning for real-time assembly planning in robot-based prefabricated construction,

    A. Zhu, T. Dai, G. Xu, P. Pauwels, B. De Vries, and M. Fang, “Deep reinforcement learning for real-time assembly planning in robot-based prefabricated construction,” IEEE Trans. Autom. Sci. Eng. , vol. 20, no. 3, pp. 1515–1526, 2023

    work page 2023

  5. [5]

    Integrating contrastive learning with dynamic models for reinforcement learning from images,

    B. You, O. Arenz, Y . Chen, and J. Peters, “Integrating contrastive learning with dynamic models for reinforcement learning from images,” Neurocomputing, 2022

    work page 2022

  6. [6]

    Rapid locomotion via reinforcement learning,

    G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, “Rapid locomotion via reinforcement learning,” Int. J. Robot. Res. , vol. 43, no. 4, pp. 572–587, 2024

    work page 2024

  7. [7]

    Biorobotics: Using robots to emulate and investigate agile locomotion,

    A. J. Ijspeert, “Biorobotics: Using robots to emulate and investigate agile locomotion,” Science, vol. 346, no. 6206, pp. 196–203, 2014

    work page 2014

  8. [8]

    Predictive modeling of periodic behavior for human-robot symbiotic walking,

    G. Clark, J. Campbell, S. M. R. Sorkhabadi, W. Zhang, and H. B. Amor, “Predictive modeling of periodic behavior for human-robot symbiotic walking,” in Int. Conf. Robot. Autom. (ICRA) . IEEE, 2020, pp. 7599– 7605

    work page 2020

Show all 48 references
  1. [9]

    Mutual-information regularization in markov decision processes and actor-critic learning,

    F. Leibfried and J. Grau-Moya, “Mutual-information regularization in markov decision processes and actor-critic learning,” in Conf. Robot. Learn. (CoRL) . PMLR, 2020, pp. 360–373

    2020

  2. [10]

    Infobot: Transfer and exploration via the information bottleneck,

    A. Goyal, R. Islam, D. Strouse, Z. Ahmed, H. Larochelle, M. Botvinick, Y . Bengio, and S. Levine, “Infobot: Transfer and exploration via the information bottleneck,” in Int. Conf. Learn. Represent. (ICLR) , 2018

    2018

  3. [11]

    Robust predictable control,

    B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Robust predictable control,” Conf. Neural Inf. Process. Syst. (NeurIPS) , vol. 34, 2021

    2021

  4. [12]

    Reinforcement learning with simple sequence priors,

    T. Saanum, N. Elteto, P. Dayan, M. Binz, and E. Schulz, “Reinforcement learning with simple sequence priors,” in Conf. Neural Inf. Process. Syst. (NeurIPS), 2023

    2023

  5. [13]

    Minimum entropy deconvolution and simplicity: A noniterative algorithm,

    C. A. Cabrelli, “Minimum entropy deconvolution and simplicity: A noniterative algorithm,” Geophysics, vol. 50, no. 3, pp. 394–413, 1985

    1985

  6. [14]

    Information theoretical analysis of multivariate correlation,

    S. Watanabe, “Information theoretical analysis of multivariate correlation,” IBM J. Res. Dev. , vol. 4, no. 1, pp. 66–82, 1960

    1960

  7. [15]

    Relative entropy inverse reinforce- ment learning,

    A. Boularias, J. Kober, and J. Peters, “Relative entropy inverse reinforce- ment learning,” J. Mach. Learn. Res. , vol. 15, pp. 182–189, 2011

    2011

  8. [16]

    Relative entropy policy search,

    J. Peters, K. Mulling, and Y . Altun, “Relative entropy policy search,” in Conf. Artif. Intell. (AAAI) , 2010

    2010

  9. [17]

    Entropy enhanced multi-agent coordination based on hierarchical graph learning for continuous action space,

    Y . Chen, K. Wang, G. Song, and X. Jiang, “Entropy enhanced multi-agent coordination based on hierarchical graph learning for continuous action space,” IEEE Trans. Cogn. Devel. Syst. , 2023

    2023

  10. [18]

    Historical decision-making regularized maximum entropy reinforcement learning,

    B. Dong, L. Huang, N. Pang, H. Chen, and W. Zhang, “Historical decision-making regularized maximum entropy reinforcement learning,” IEEE Trans. on Neural Netw. Learn. Syst. , 2024

    2024

  11. [19]

    Soft actor-critic algorithms and applications,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel et al. , “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905 , 2018

    2018 arXiv

  12. [20]

    Maximum entropy inverse reinforcement learning,

    B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” in Conf. Artif. Intell. (AAAI) , 2008, pp. 1433–1438

    2008

  13. [21]

    Multi-alpha soft actor-critic: Overcoming stochastic biases in maximum entropy reinforcement learning,

    C. Igoe, S. Pande, S. Venkatraman, and J. Schneider, “Multi-alpha soft actor-critic: Overcoming stochastic biases in maximum entropy reinforcement learning,” in Int. Conf. Robot. Autom. (ICRA) . IEEE, 2023, pp. 7162–7168

    2023

  14. [22]

    Accelerating reinforcement learning with value-conditional state entropy exploration,

    D. Kim, J. Shin, P. Abbeel, and Y . Seo, “Accelerating reinforcement learning with value-conditional state entropy exploration,” Conf. Neural Inf. Process. Syst. (NeurIPS) , vol. 36, 2024

    2024

  15. [23]

    Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation,

    C. Sferrazza, D.-M. Huang, X. Lin, Y . Lee, and P. Abbeel, “Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation,” arXiv preprint arXiv:2403.10506 , 2024

    2024 arXiv

  16. [24]

    Deepmind control suite,

    Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq et al. , “Deepmind control suite,” arXiv preprint arXiv:1801.00690 , 2018

    2018 arXiv

  17. [25]

    Autoencoders, minimum description length and helmholtz free energy,

    G. E. Hinton and R. Zemel, “Autoencoders, minimum description length and helmholtz free energy,” Conf. Neural Inf. Process. Syst. (NeurIPS) , vol. 6, 1993. 10

    1993

  18. [26]

    Network simplification with minimal loss of connectivity,

    F. Zhou, S. Malher, and H. Toivonen, “Network simplification with minimal loss of connectivity,” in IEEE Int. Conf. Data Min. IEEE, 2010, pp. 659–668

    2010

  19. [27]

    Convolutional neural network pruning with structural redundancy reduction,

    Z. Wang, C. Li, and X. Wang, “Convolutional neural network pruning with structural redundancy reduction,” in IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2021, pp. 14 913–14 922

    2021

  20. [28]

    Deep learning and the information bottleneck principle,

    N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in IEEE Inf. Theory Workshop (ITW) . IEEE, 2015, pp. 1–5

    2015

  21. [29]

    A survey on information bottleneck,

    S. Hu, Z. Lou, X. Yan, and Y . Ye, “A survey on information bottleneck,” IEEE Trans. Pattern Anal. Mach. Intell. , 2024

    2024

  22. [30]

    Keeping the neural networks simple by minimizing the description length of the weights,

    G. E. Hinton and D. Van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in Annu. Conf. Comput. Learn. Theory. , 1993, pp. 5–13

    1993

  23. [31]

    Soft q-learning with mutual- information regularization,

    J. Grau-Moya, F. Leibfried, and P. Vrancx, “Soft q-learning with mutual- information regularization,” in Int. Conf. Learn. Represent. (ICLR) , 2018

    2018

  24. [32]

    Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion,

    J. Urain, N. Funk, J. Peters, and G. Chalvatzaki, “Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion,” in Int. Conf. Robot. Autom. (ICRA) . IEEE, 2023, pp. 5923–5930

    2023

  25. [33]

    Rvt: Robotic view transformer for 3d object manipulation,

    A. Goyal, J. Xu, Y . Guo, V . Blukis, Y .-W. Chao, and D. Fox, “Rvt: Robotic view transformer for 3d object manipulation,” in Conf. Robot. Learn. (CoRL) . PMLR, 2023, pp. 694–710

    2023

  26. [34]

    Composable energy policies for reactive motion generation and reinforcement learning,

    J. Urain, A. Li, P. Liu, C. D’Eramo, and J. Peters, “Composable energy policies for reactive motion generation and reinforcement learning,” Int. J. Robot. Res. , vol. 42, no. 10, pp. 827–858, 2023

    2023

  27. [35]

    Robot model identification and learning: A modern perspective,

    T. Lee, J. Kwon, P. M. Wensing, and F. C. Park, “Robot model identification and learning: A modern perspective,” Annu. Rev. Contr . Robot. Autonom. Syst. , vol. 7, 2023

    2023

  28. [36]

    Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation,

    J. Yang, C. Deng, J. Wu, R. Antonova, L. Guibas, and J. Bohg, “Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation,” in Int. Conf. Robot. Autom. (ICRA) . IEEE, 2024, pp. 9249–9255

    2024

  29. [37]

    Multimodal information bottleneck for deep reinforcement learning with multiple sensors,

    B. You and H. Liu, “Multimodal information bottleneck for deep reinforcement learning with multiple sensors,” Neural Netw., vol. 176, p. 106347, 2024

    2024

  30. [38]

    Leveraging symmetries in pick and place,

    H. Huang, D. Wang, A. Tangri, R. Walters, and R. Platt, “Leveraging symmetries in pick and place,” Int. J. Robot. Res. , vol. 43, no. 4, pp. 550–571, 2024

    2024

  31. [39]

    Pid-inspired inductive biases for deep reinforcement learning in partially observable control tasks,

    I. Char and J. Schneider, “Pid-inspired inductive biases for deep reinforcement learning in partially observable control tasks,” Conf. Neural Inf. Process. Syst. (NeurIPS) , vol. 36, 2024

    2024

  32. [40]

    R ´enyi state entropy maximization for exploration acceleration in reinforcement learning,

    M. Yuan, M.-O. Pun, and D. Wang, “R ´enyi state entropy maximization for exploration acceleration in reinforcement learning,” IEEE Trans. Artif. Intell., vol. 4, no. 5, pp. 1154–1164, 2022

    2022

  33. [41]

    Efficient sampling- based maximum entropy inverse reinforcement learning with application to autonomous driving,

    Z. Wu, L. Sun, W. Zhan, C. Yang, and M. Tomizuka, “Efficient sampling- based maximum entropy inverse reinforcement learning with application to autonomous driving,” IEEE Robot. Autom. Lett. , vol. 5, no. 4, pp. 5355–5362, 2020

    2020

  34. [42]

    Entropy maximization for partially observable markov decision processes,

    Y . Savas, M. Hibbard, B. Wu, T. Tanaka, and U. Topcu, “Entropy maximization for partially observable markov decision processes,” IEEE Trans. Autom. Control , vol. 67, no. 12, pp. 6948–6955, 2022

    2022

  35. [43]

    Maximum entropy rl (provably) solves some robust rl problems,

    B. Eysenbach and S. Levine, “Maximum entropy rl (provably) solves some robust rl problems,” in Int. Conf. Learn. Represent. (ICLR) , 2023

    2023

  36. [44]

    Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control,

    Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control,” Int. J. Robot. Res. , pp. 1–24, 2024

    2024

  37. [45]

    Distributionally robust model-based offline reinforce- ment learning with near-optimal sample complexity,

    L. Shi and Y . Chi, “Distributionally robust model-based offline reinforce- ment learning with near-optimal sample complexity,” J. Mach. Learn. Res., vol. 25, no. 200, pp. 1–91, 2024

    2024

  38. [46]

    Robust quadrupedal locomotion via risk-averse policy learning,

    J. Shi, C. Bai, H. He, L. Han, D. Wang, B. Zhao, M. Zhao, X. Li, and X. Li, “Robust quadrupedal locomotion via risk-averse policy learning,” in Int. Conf. Robot. Autom. (ICRA) . IEEE, 2024, pp. 11 459–11 466

    2024

  39. [47]

    Natural actor-critic for robust reinforcement learning with function approximation,

    R. Zhou, T. Liu, M. Cheng, D. Kalathil, P. Kumar, and C. Tian, “Natural actor-critic for robust reinforcement learning with function approximation,” Conf. Neural Inf. Process. Syst. (NeurIPS) , vol. 36, 2024

    2024

  40. [48]

    Improving sample efficiency in model-free reinforcement learning from images,

    D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus, “Improving sample efficiency in model-free reinforcement learning from images,” in Conf. Artif. Intell. (AAAI) , no. 12, 2021, pp. 10 674–10 681

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.