Pith. sign in

REVIEW 3 major objections 5 minor 42 references

How Well Do Latent World Models Understand Partially Observable Safety Constraints?

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Latent safety filters become myopic when safety-relevant features are unobservable in the training data, and multimodal supervision during training can restore their reliability.

desk verdict Plausible and useful core findings, but the hardware headline is undercut by checkpoint selection and the abstract promises a mitigation the body never delivers. read the letter →

arxiv 2510.06492 v2 pith:VMPSLIBM submitted 2025-10-07 cs.RO

Matthew Kim , Kensuke Nakamura , Andrea Bajcsy This is my paper · ORCID
classification cs.RO
keywords latentworldmodelssafetyfilterspartialobservabilitymutualinformationmultimodalsupervisionHamilton-Jacobireachabilityrobotmanipulationtemperatureconstraints
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies robot safety controllers that are computed inside a learned latent world model, which compresses camera images into a hidden state and imagines the future. The authors argue that if the training observations do not reveal the quantity that actually determines safety—here, temperature—the resulting 'latent safety filter' becomes myopic: it learns to avoid seeing failures rather than preventing them, e.g., staying over a hotplate instead of lifting the pan. They propose a mutual-information lower bound between an observation embedding and the safety label as a diagnostic that separates sufficient from insufficient representations better than classification accuracy. As a fix, they train the world model to reconstruct both RGB and infrared images during training while deploying with RGB only, which reshapes the latent state so the filter learns the correct recovery action; on hardware the lift rate rises from 15% to 100% over 20 trials. The paper's broader message is that reconstruction-based world models need observability checks before their latent safety filters are trusted.

What carries the argument

The argument runs through the latent safety filter: a Dreamer-style recurrent state-space world model (encoder, recurrent dynamics, decoder trained by reconstruction) with a hinge-loss safety classifier ℓ(z) and a safety filter obtained by solving a latent Hamilton-Jacobi-Bellman equation with DDPG; the value function's sign selects between the task policy and a safe fallback. Two tools carry the analysis: (1) a Barber-Agakov lower bound on the mutual information I(Y;X) between observation embeddings and the binary safety label, used as a diagnostic for estimation gaps, and (2) a rollout-based F1 measure over 16-step open-loop imagined trajectories for prediction gaps. The mitigation is a mu

What would settle it

Run the hardware wax-melting experiment again with the same multimodal-supervised training, but deploy the final training checkpoint instead of the best one for all 20 trials; if the lift rate collapses toward the RGB-only level, the reported recovery is an artifact of checkpoint selection. A second check: vary the initial wax temperature or hotplate power; the 100% lift rate should persist across operating conditions if the latent truly encodes action-outcome semantics.

Watch

Extended reading notes

Core claim

The paper's central claim is that partial observability, not model capacity or classifier error, is what breaks latent safety filters. When the safety-relevant variable (heat) is invisible in RGB input, the filter's value function learns to keep the system in states where failure is not yet visible instead of steering away from failure—an estimation gap—and its imagined rollouts mispredict the safety consequences of actions, e.g., classifying a pan-lift as unsafe—a prediction gap. The paper demonstrates both gaps in a simulation (thermal unicycle) and on a Franka manipulator preventing wax overheating, where the RGB-only filter fails to lift the pan in 17 of 20 trials. It further claims that

Load-bearing premise

The whole chain depends on open-loop rollouts in the learned latent dynamics (at most 16 imagination steps from a 5-step history) being faithful enough for the learned value function's sign to transfer to the real robot; the appendix concedes 'minor instabilities' in the RL training and reports the best checkpoint per model rather than the final one, so a fortuitous checkpoint could contribute to the 100% lift rate.

Editorial extensions

If this is right

  • Before deployment, computing the MI lower bound on the world model's input modality can flag whether a latent safety filter will be myopic, giving a cheap screening test.
  • Multimodal supervision during training can replace the need for privileged safety states at runtime, at least when a complementary sensor (IR, tactile) reveals the missing quantity.
  • Safety classification accuracy on current observations is not enough; the filter must also correctly imagine how actions change future safety outcomes.
  • A fixed safety filter threshold applied to the learned value function produces conservative intervention when absolute quantities are unobservable, which may be an acceptable trade for reliability.
  • The failure mode identified here extends beyond temperature constraints to any safety-critical variable that is latent in the observation space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The MI diagnostic could be computed over candidate sensor channels during data collection to decide which modalities to add, e.g., estimating I(Y; RGB) vs I(Y; IR) before training any world model.
  • The masked-decoder recipe should transfer to force, torque, or auditory supervision, where the supplementary modality correlates with collision or contact constraints invisible in RGB; a concrete test would replace IR with tactile input on a contact-rich manipulation task.
  • Because the paper reports best-checkpoint results, a testable robustness prediction is that the 20/20 lift rate survives across random seeds and the final checkpoint; if not, part of the reported recovery is a training-stability artifact rather than observability shaping alone.
  • The 'relative judgment' result implies the latent state learned under multimodal supervision encodes the direction of action effects on safety rather than the absolute level; one could verify this by probing whether the multimodal-supervised world model's imagined rollouts show higher action-to-safety-label mutual information than the RGB-only model's.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether latent space safety filters computed from Dreamer world models remain reliable when safety-relevant features are only partially observable. It reports that RGB-only world models produce myopic safety behavior in a thermal unicycle simulation and in a Franka hardware wax-melting task; introduces a mutual-information lower bound as a diagnostic for safety observability in the latent state; and proposes a multimodal supervision training recipe in which the decoder reconstructs RGB+IR during training while deployment uses RGB only. The body reports simulation and hardware results supporting the main qualitative claims, and the abstract also promises a conformal risk-calibration mitigation for prediction gaps. The paper includes appendices with hyperparameters and a statement that safety-filter results come from the best checkpoint per model.

Significance. If the main claims hold, the work is a valuable contribution to latent-space safe control: it identifies a concrete failure mode (observability-induced myopia), proposes an independent information-theoretic diagnostic, and demonstrates a training-time mitigation that requires no extra deployment sensors. The closed-loop hardware evaluation, the use of an independent MI measure, and the explicit limitation statements are notable strengths. However, the missing conformal mitigation promised in the abstract and the lack of a transparent checkpoint-selection rule for the central hardware comparison currently prevent the paper from being accepted as-is.

major comments (3)
  1. [Abstract; Section VI] The abstract states: 'Finally, we present mitigation strategies for each failure mode: privileged multimodal supervision for estimation gaps and conformal risk calibration for prediction gaps.' The body, however, only describes and evaluates multimodal supervision (Section VI). No conformal risk calibration procedure appears anywhere in Sections IV–VII or the appendix. This is not a presentation typo: the paper claims a mitigation for prediction gaps that is never defined, implemented, or tested. Either include the conformal calibration method and its results, or revise the abstract and contributions to match the actual content.
  2. [Appendix VIII, Safety Filter Synthesis] The appendix states: 'All reported results are from the best performing checkpoint for each model (not necessarily the final checkpoint, due to minor instabilities of the RL training procedure).' Table IV reports the central result: WM_RGB has lift rate 0.15 [0.04,0.39] while WM_RGB-MM and WM_MM achieve 1.0 [0.80,1.0]. If 'best performing' was selected by the same lift rate used for evaluation, the comparison is circular. No held-out selection criterion, multi-seed variance, or final-checkpoint comparison is reported. Given that each model has 15 checkpoints and RL training is described as unstable, this threatens the main hardware claim. I ask for a fixed, deployment-independent checkpoint-selection rule (e.g., a validation-set safety metric) and ideally multiple seeds or at least final-checkpoint results.
  3. [Equation (3); Appendix VIII, Table VIII] The safety filter is synthesized in imagination by solving Eq. (3) with DDPG, using open-loop rollouts of at most 16 imagination steps initialized from N=5 history segments. Hardware data are downsampled to 3 Hz, so 16 steps correspond to roughly 5.3 seconds, which may be short relative to the wax-heating dynamics. The paper does not test sensitivity to the imagination horizon or history length, and does not analyze whether sign errors in the learned value function could flip the least-restrictive intervention in Eq. (4). This is a load-bearing modeling premise: if imagined rollouts misrepresent how actions change safety-relevant state, the 100% lift rate could be an artifact of favorable horizon choice. Please add an ablation or at least report value-function sign accuracy against the ground-truth safety outcome in the hardware task.
minor comments (5)
  1. [Section IV.A] Typo: 'by by sampling initial states' should read 'by sampling initial states'.
  2. [Table I] The table caption says 'Normalized mutual information estimates (I(Y;X)/H(Y))', but the row label appears as 'I(Y;X) H(Y)' with values bounded by 0 and 1. Please clarify whether the first row reports the normalized quantity or the unnormalized MI. Also, the phrase in the text 'eRGB omitted for space, we notice no significant devations' has a typo and should be rewritten.
  3. [Section VI] The statement 'We will release open source code with all hyperparameters upon acceptance' is appropriate but the hyperparameters for the hardware safety filter, including epsilon and the hinge loss margin, are not all listed in the main text or appendix. Please include a single table with all safety-filter parameters used on hardware.
  4. [Section IV.B] The sentence 'We observe alarge gap between the safety filter that has a sufficient latent representation and the one that does not' is vague; the specific numbers are only in Table IV. Consider referencing the table here.
  5. [Conclusion and Limitations] The limitations paragraph correctly notes that formal assurances are future work. However, given the paper's safety-critical framing, please also state explicitly which reported results are from single runs versus multiple seeds, and whether the hardware deployments were the only trials or the result of a selection process.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the observability diagnostic and multimodal-supervision result are independently measured; only minor self-citation of prior latent-safety-filter machinery appears.

full rationale

The paper's central derivation chain is not circular. The mutual-information diagnostic (Eq. 5, Sec. V) is a standard Barber-Agakov lower bound; the linear probes are trained on separate held-out splits and temperature-scaled, and the resulting Table I values are measurements of how much safety-label information each observation modality carries, not parameters fitted to the downstream filter outcome. The behavioral failure of RGB-only latent safety filters is established independently in Sec. IV (Thermal Unicycle and hardware wax-melting) and Table IV via closed-loop deployment, not read off the MI estimate. The multimodal-supervised WM_RGB-MM is a distinct training intervention that is evaluated by hardware lift rate, so its 100% lift rate is not a re-labeling of the diagnostic. The paper does reuse the authors' prior latent Hamilton-Jacobi machinery [8,9] for Eq. (3) and the DDPG synthesis in the appendix, and it follows [8] for the tanh margin function; this is load-bearing as a building block but not an imported uniqueness theorem or an ansatz used to forbid alternatives, so it is at most minor self-citation. Two flagged robustness concerns are correctness risks rather than circularity: (i) the appendix states 'All reported results are from the best performing checkpoint for each model (not necessarily the final checkpoint, due to minor instabilities of the RL training procedure)' without a stated, deployment-independent selection rule, so the Table IV lift-rate gap could be inflated by checkpoint selection; (ii) the abstract promises 'conformal risk calibration for prediction gaps' but the body does not present a conformal-calibration experiment. Neither concern exhibits an equation or fitted parameter that is equivalent to its own input by construction, so the circularity score remains low.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central results rest on the Dreamer latent-space safety-filtering pipeline from prior work, a learned MI lower bound whose validity depends on probe quality, and IR-intensity-based failure labels. The hand-fitted safety parameters are listed above; no new physical entities are introduced.

free parameters (7)
  • Safety filter activation threshold epsilon = 0.3
    Fixed threshold in the least-restrictive switching law (Eq. 4, Sec. VI) for all deployments; directly controls when the fallback policy activates.
  • Failure label threshold (IR pixel intensity) = 0.75
    Labels wax as unsafe when average IR pixel intensity of the wax region exceeds 0.75, corresponding to ~77C; defines ground-truth labels used to train classifiers and evaluate safety.
  • Hinge margin delta = 0.75
    Margin in the failure-classifier hinge loss (Eq. 2, Sec. III and Table VI); chosen by hand.
  • tanh margin scale lambda = 0.1
    Used in safety filter synthesis as the margin function tanh(lambda * ell(z)) to smooth the loss landscape (Appendix).
  • Discount factor gamma = 0.9999
    Latent HJB value learning (Eq. 3, Table VIII).
  • Imagination horizon / history length = 16 / 5
    DDPG rollouts max 16 imagination steps, latent initialized from N=5 history segments; used for filter synthesis and open-loop prediction metrics.
  • Simulation heat dynamics constants = alpha=0.03, beta=0.06, R=0.5, h_fail=0.8
    Hand-designed thermal unicycle environment; controls the observability manipulation (Sec. IV-A).
assumptions (6)
  • standard math Barber-Agakov lower bound: I(Y;X) >= H(Y) - H(P,Q)
    Used in Eq. 5 to estimate mutual information between observations and safety labels from a learned variational predictor Q.
  • domain assumption Dreamer/RSSM reconstruction and KL objectives yield a latent state and dynamics that can support downstream HJB safety filter synthesis
    The whole latent safety filter pipeline is built on this premise from [27], [8]; Sec. III.
  • domain assumption DDPG with the latent HJB backup (Eq. 3) approximately converges to a value function whose sign has the claimed safety semantics
    Used to synthesize fallback policy and value function; Appendix notes instabilities and best-checkpoint selection.
  • domain assumption IR average pixel intensity is a reliable ground-truth measure of wax overheating
    Failure labels, training labels, and evaluation metrics all derive from the IR threshold 0.75 (Sec. IV-B).
  • domain assumption Linear probe + temperature scaling provides a valid variational Q(Y|X) for the MI lower bound
    Table I estimates normalized MI from probe logits; if the probe family is too weak, low MI may reflect probe capacity rather than observability.
  • domain assumption Open-loop imagination rollouts from offline N=5 segments are representative of real closed-loop dynamics for synthesizing the filter
    Safety filter is optimized entirely in latent imagination (Sec. VI, Appendix); mismatch would invalidate deployment behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Well Do Latent World Models Understand Partially Observable Safety Constraints?." pith.science (2026). https://pith.science/paper/VMPSLIBM

@misc{pith2026251006492,
  author       = {Pith},
  title        = {Pith review of: How Well Do Latent World Models Understand Partially Observable Safety Constraints?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMPSLIBM}},
  note         = {Machine review of arXiv:2510.06492}
}
read the original abstract

Latent world models are a promising approach for learning state representations and dynamics directly from high-dimensional observations, enabling robot control in hard-to-model settings. However, control performance ultimately depends on the latent representation encoding the required information for the task. In this work, we study latent-space safe control problems and show how partial observability can induce control failures when safety-relevant information is not preserved in the latent state. Specifically, we identify two world model failure modes: estimation gaps, where current observations do not reveal safety-critical quantities (e.g., temperature in a cooking task), and prediction gaps, where failures are observable once they occur but cannot be reliably anticipated from available observations. We introduce two diagnostics for these gaps: a mutual-information-based measure of safety observability and a rollout-based measure of future safety predictability. Finally, we present mitigation strategies for each failure mode: privileged multimodal supervision for estimation gaps and conformal risk calibration for prediction gaps. Across two hardware case studies -- using unimodal RGB world models and multimodal RGB+Tactile and RGB+Thermal variants -- we show that these mitigation strategies improve the safety of a Franka Research 3 manipulator on challenging cooking tasks under partial observability, albeit with increased conservativeness. More broadly, our work raises the question of when world model state representations are sufficient for reliable robot control

Figures

Figures reproduced from arXiv: 2510.06492 by the authors.

Figure 1
Figure 1. We design a series of controlled experiments to test how latent safety filters behave under partially observable [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Experiment Testbeds. In simulation (left) and hard￾ware (right) we control data from two sensors: RGB and infrared (IR) camera. In our controlled experiments, the ground-truth safety-relevant state variable is heat, which is more observable from the IR data than the RGB. A. Illustrative Example: Thermal Unicycle For a low-dimensional running example, we simulate a planar vehicle which can navigate into a “hot plate”… view at source ↗
Figure 3
Figure 3. Safety Controllers without & with Observability. In the Thermal Unicycle example, the robot only appears burnt in the RGB camera after it leaves the hot plate. Left: The safety policy is myopic when trained within a world model with an incomplete state representation: it stays in the hot region, avoiding seeing failure. Right: When the world model is trained with sufficiently rich observations (RGB + IR), the result… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Real vs. Imagined Observations & Safety Labels. Open-loop prediction of an evaluation subtrajectory reveals that WMRGB incorrectly predicts the safety outcomes of actions, such as predicting that lifting the wax plate leads to failure. In contrast, observing temperatur…
Figure 5
Figure 5. Figure 5: Distribution of ground-truth pixel intensities when the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: WM Supervised by Multimodal Data: Open-Loop Predictions. Despite only taking in RGB input, multimodal training objectives enables the world model to estimate unobserved quantities (visualized via decoded IR images) and better predict safety outcomes. should intervene d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages

  1. [1]

    Hamilton-jacobi reachability: A brief overview and recent advances,

    S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin, “Hamilton-jacobi reachability: A brief overview and recent advances,” in2017 IEEE 56th Annual Conference on Decision and Control (CDC). IEEE, 2017

    2017

  2. [2]

    Control barrier functions: Theory and applications,

    A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in2019 18th European control conference (ECC), 2019

    2019

  3. [3]

    How to train your neural control barrier function: Learning safety filters for complex input-constrained systems,

    O. So, Z. Serlin, M. Mann, J. Gonzales, K. Rutledge, N. Roy, and C. Fan, “How to train your neural control barrier function: Learning safety filters for complex input-constrained systems,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 532–11 539

    2024

  4. [4]

    Reachability barrier networks: Learning hamilton-jacobi solutions for smooth and flexible control barrier functions,

    M. Kim, W. Sharpless, H. J. Jeong, S. Tonkens, S. Bansal, and S. Herbert, “Reachability barrier networks: Learning hamilton-jacobi solutions for smooth and flexible control barrier functions,”arXiv preprint arXiv:2505.11755, 2025

    arXiv 2025

  5. [5]

    Safe reinforcement learning with nonlinear dynamics via model predictive shielding,

    O. Bastani, “Safe reinforcement learning with nonlinear dynamics via model predictive shielding,” in2021 American control conference. IEEE, 2021

    2021

  6. [6]

    Robust model predictive shielding for safe reinforcement learning with stochastic dynamics,

    S. Li and O. Bastani, “Robust model predictive shielding for safe reinforcement learning with stochastic dynamics,” in2020 IEEE International Conference on Robotics and Automation. IEEE, 2020

    2020

  7. [7]

    Sharp: Shielding-aware ro- bust planning for safe and efficient human-robot interaction,

    H. Hu, K. Nakamura, and J. F. Fisac, “Sharp: Shielding-aware ro- bust planning for safe and efficient human-robot interaction,”IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 5591–5598, 2022

    2022

  8. [8]

    Generalizing safety beyond collision-avoidance via latent-space reachability analysis,

    K. Nakamura, L. Peters, and A. Bajcsy, “Generalizing safety beyond collision-avoidance via latent-space reachability analysis,”Robotics: Science and Systems (RSS), 2025

    2025

Show all 42 references
  1. [9]

    Uncertainty-aware latent safety filters for avoiding out-of-distribution failures,

    J. Seo, K. Nakamura, and A. Bajcsy, “Uncertainty-aware latent safety filters for avoiding out-of-distribution failures,”Conference on Robot Learning (CoRL), 2025

    2025

  2. [10]

    World models,

    D. Ha and J. Schmidhuber, “World models,”arXiv preprint arXiv:1803.10122, vol. 2, no. 3, 2018

    2018 arXiv

  3. [11]

    Learning latent dynamics for planning from pixels,

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in International Conference on machine learning (ICML), 2019

    2019

  4. [12]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning,

    G. Zhou, H. Pan, Y . LeCun, and L. Pinto, “Dino-wm: World models on pre-trained visual features enable zero-shot planning,”International Conference on machine learning (ICML), 2025

    2025

  5. [13]

    A new approach to linear filtering and prediction problems,

    R. E. Kalman, “A new approach to linear filtering and prediction problems,”Journal of Basic Engineering, vol. 82, no. 1, 1960

    1960

  6. [14]

    Provably filtering exogenous distractors using multistep inverse dynamics,

    Y . Efroni, D. Misra, A. Krishnamurthy, A. Agarwal, and J. Langford, “Provably filtering exogenous distractors using multistep inverse dynamics,” inInternational Conference on Learning Representations, 2022. [Online]. Available: https: //openreview.net/forum?id=RQLLzMCefQu

    2022

  7. [15]

    Guaranteed discovery of control-endogenous latent states with multi-step inverse models,

    A. Lamb, R. Islam, Y . Efroni, A. Didolkar, D. Misra, D. Foster, L. Molu, R. Chari, A. Krishnamurthy, and J. Langford, “Guaranteed discovery of control-endogenous latent states with multi-step inverse models,”arXiv preprint arXiv:2207.08229, 2022

    2022 arXiv

  8. [16]

    Toward understanding state representation learning in muzero: A case study in linear quadratic gaussian control,

    Y . Tian, K. Zhang, R. Tedrake, and S. Sra, “Toward understanding state representation learning in muzero: A case study in linear quadratic gaussian control,” in2023 62nd IEEE Conference on Decision and Control (CDC). IEEE, 2023, pp. 6166–6171

    2023

  9. [17]

    Td-mpc2: Scalable, robust world models for continuous control,

    N. Hansen, H. Su, and X. Wang, “Td-mpc2: Scalable, robust world models for continuous control,” inInternational Conference on Learn- ing Representations (ICLR), 2024

    2024

  10. [18]

    Action-sufficient state representation learning for control with structural constraints,

    B. Huang, C. Lu, L. Leqi, J. M. Hern ´andez-Lobato, C. Glymour, B. Sch ¨olkopf, and K. Zhang, “Action-sufficient state representation learning for control with structural constraints,” inInternational Con- ference on Machine Learning. PMLR, 2022, pp. 9260–9279

    2022

  11. [19]

    Multimodal sensors and ml-based data fusion for advanced robots,

    S. Duan, Q. Shi, and J. Wu, “Multimodal sensors and ml-based data fusion for advanced robots,”Advanced Intelligent Systems, vol. 4, no. 12, p. 2200213, 2022

    2022

  12. [20]

    Mul- timodal sensor fusion with differentiable filters,

    M. A. Lee, B. Yi, R. Mart ´ın-Mart´ın, S. Savarese, and J. Bohg, “Mul- timodal sensor fusion with differentiable filters,” in2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 10 444–10 451

    2020

  13. [21]

    A review of multi-sensor fusion slam systems based on 3d lidar,

    X. Xu, L. Zhang, J. Yang, C. Cao, W. Wang, Y . Ran, Z. Tan, and M. Luo, “A review of multi-sensor fusion slam systems based on 3d lidar,”Remote Sensing, vol. 14, no. 12, p. 2835, 2022

    2022

  14. [22]

    Multisensor fusion and integra- tion: approaches, applications, and future research directions,

    R. C. Luo, C.-C. Yih, and K. L. Su, “Multisensor fusion and integra- tion: approaches, applications, and future research directions,”IEEE Sensors journal, vol. 2, no. 2, pp. 107–119, 2002

    2002

  15. [23]

    Deep learning with rgb and thermal images onboard a drone for monitoring operations,

    S. Speth, A. Gonc ¸alves, B. Rigault, S. Suzuki, M. Bouazizi, Y . Matsuo, and H. Prendinger, “Deep learning with rgb and thermal images onboard a drone for monitoring operations,”J. Field Robotics, vol. 39, no. 6, pp. 840–868, 2022. [Online]. Available: https://doi.org/10.1002...

    2022 doi

  16. [24]

    Vitacformer: Learning cross-modal representation for visuo-tactile dexterous manipulation,

    L. Heng, H. Geng, K. Zhang, P. Abbeel, and J. Malik, “Vitacformer: Learning cross-modal representation for visuo-tactile dexterous manipulation,” 2025. [Online]. Available: https://arxiv.org/abs/2506. 15953

    2025

  17. [25]

    That sounds right: Auditory self- supervision for dynamic robot manipulation,

    A. Thankaraj and L. Pinto, “That sounds right: Auditory self- supervision for dynamic robot manipulation,” in7th Annual Conference on Robot Learning, 2023. [Online]. Available: https: //openreview.net/forum?id=sLhk0keeiseH

    2023

  18. [26]

    Hearing touch: Audio-visual pretraining for contact-rich manipulation,

    J. Mejia, V . Dean, T. Hellebrekers, and A. Gupta, “Hearing touch: Audio-visual pretraining for contact-rich manipulation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6912–6919

    2024

  19. [27]

    Mastering diverse control tasks through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse control tasks through world models,”Nature, vol. 640, no. 8059, 2025

    2025

  20. [28]

    dreamerv3-torch: Implementation of dreamer v3 in pytorch,

    N. Morihira, “dreamerv3-torch: Implementation of dreamer v3 in pytorch,” https://github.com/NM512/dreamerv3-torch, 2025, accessed: 2025-01-25

    2025

  21. [29]

    Bridging hamilton-jacobi safety analysis and reinforcement learning,

    J. F. Fisac, N. F. Lugovoy, V . Rubies-Royo, S. Ghosh, and C. J. Tomlin, “Bridging hamilton-jacobi safety analysis and reinforcement learning,” in2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8550–8556

    2019

  22. [30]

    Continuous control with deep reinforcement learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2019. [Online]. Available: https: //arxiv.org/abs/1509.02971

    2019 arXiv

  23. [31]

    Certifiable reachability learning using a new lipschitz continuous value function,

    J. Li, D. Lee, J. Lee, K. S. Dong, S. Sojoudi, and C. Tomlin, “Certifiable reachability learning using a new lipschitz continuous value function,”IEEE Robotics and Automation Letters, vol. 10, no. 4, pp. 3582–3589, 2025

    2025

  24. [32]

    The safety filter: A unified view of safety-critical control in autonomous systems,

    K.-C. Hsu, H. Hu, and J. F. Fisac, “The safety filter: A unified view of safety-critical control in autonomous systems,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 7, 2023

    2023

  25. [33]

    On infusing reachability-based safety assurance within planning frameworks for human–robot vehicle interactions,

    K. Leung, E. Schmerling, M. Zhang, M. Chen, J. Talbot, J. C. Gerdes, and M. Pavone, “On infusing reachability-based safety assurance within planning frameworks for human–robot vehicle interactions,” The International Journal of Robotics Research, vol. 39, no. 10-11, pp. 1326–1...

    2020

  26. [34]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, pp. 379–423,

  27. [35]

    Estimating mutual information,

    A. Kraskov, H. St ¨ogbauer, and P. Grassberger, “Estimating mutual information,”Phys. Rev. E, vol. 69, p. 066138, Jun 2004. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevE.69.066138

    2004 doi

  28. [36]

    The im algorithm: a variational approach to information maximization,

    D. Barber and F. Agakov, “The im algorithm: a variational approach to information maximization,” inProceedings of the 17th International Conference on Neural Information Processing Systems, ser. NIPS’03. Cambridge, MA, USA: MIT Press, 2003, p. 201–208

    2003

  29. [37]

    Understanding intermediate layers using linear classifier probes,

    G. Alain and Y . Bengio, “Understanding intermediate layers using linear classifier probes,”International Conference on Learning Repre- sentations, 2017

    2017

  30. [38]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inProceedings of the 34th Interna- tional Conference on Machine Learning - Volume 70, ser. ICML’17. JMLR.org, 2017, p. 1321–1330

    2017

  31. [39]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep...

    2015 doi

  32. [40]

    Rma: Rapid motor adaptation for legged robots,

    A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” 2021. [Online]. Available: https://arxiv.org/abs/2107.04034

    2021 arXiv

  33. [41]

    Two-sided confidence intervals for the single proportion: comparison of seven methods,

    R. G. Newcombe, “Two-sided confidence intervals for the single proportion: comparison of seven methods,”Statistics in Medicine, vol. 17, no. 8, pp. 857–872, 1998. VIII. APPENDIX A. Hyperparameters For all experiments, we use an open-source implementa- tion of the Dreamerv3 Rec...

    1998

  34. [1948]

    Available: http://plan9.bell-labs.com/cm/ms/what/ shannonday/shannon1948.pdf

    [Online]. Available: http://plan9.bell-labs.com/cm/ms/what/ shannonday/shannon1948.pdf

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.