pith. machine review for the scientific record. sign in

arxiv: 2604.24894 · v1 · submitted 2026-04-27 · 💻 cs.RO · cs.CV· cs.LG· cs.SY· eess.SY· math.OC

Recognition: unknown

VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis

Antoine P. Leeman , Shuyu Zhan , Melanie N. Zeilinger , Glen Chou
Authors on Pith no claims yet

Pith reviewed 2026-05-08 02:38 UTC · model grok-4.3

classification 💻 cs.RO cs.CVcs.LGcs.SYeess.SYmath.OC
keywords safe controlvisual perceptionsystem level synthesisoutput feedbackuncertainty boundsnonlinear dynamicsroboticsperception-based control
0
0 comments X

The pith

Learned visual observation maps with state-dependent error bounds plus system-level synthesis enable safe nonlinear output-feedback control from high-resolution RGB images.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops VISION-SLS to perform safe perception-based control using high-resolution images. It learns a low-dimensional observation map from visual features that includes state-dependent error bounds. An efficient solver optimizes a causal affine output-feedback policy via system level synthesis to guarantee constraint satisfaction under uncertainty. This approach is demonstrated on simulated tasks including a car, quadrotor, and humanoid, as well as hardware experiments on a ground vehicle. The key is that it achieves information-gathering behavior while maintaining safety guarantees despite partial observability and nonlinear dynamics.

Core claim

The central claim is that coupling a learned low-dimensional observation map from pretrained visual features, equipped with state-dependent error bounds, with a causal affine time-varying output-feedback policy optimized via System Level Synthesis provides robust constraint satisfaction guarantees for nonlinear systems under partial observability, sensor noise, and nonlinear dynamics. This is achieved through a scalable solver based on sequential convex programming and Riccati recursions, enabling safe information-gathering behavior that reduces uncertainty while respecting constraints, as validated on high-dimensional simulated and hardware tasks.

What carries the argument

The learned low-dimensional observation map with state-dependent error bounds combined with System Level Synthesis (SLS) for optimizing causal affine time-varying output-feedback policies, solved using sequential convex programming and efficient Riccati recursions.

If this is right

  • Enables safe information-gathering behavior that reduces uncertainty while satisfying constraints with calibrated bounds.
  • Scalable computation for high-resolution images and high-dimensional systems like 10D quadrotor and 59D humanoid.
  • Outperformance over baselines in safety rates and computation times on hardware validation.
  • Makes SLS-based safe visuomotor output-feedback practical at scale for nonlinear systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could extend to other sensor modalities if similar bounded observation maps can be learned from pretrained features.
  • Online recalibration of error bounds might be needed when real-world dynamics deviate from the training distribution.
  • The SLS formulation may integrate with receding-horizon variants to handle longer prediction horizons without losing guarantees.

Load-bearing premise

The state-dependent error bounds produced by the learned observation map remain valid under the closed-loop nonlinear dynamics and sensor noise encountered at runtime.

What would settle it

A closed-loop trajectory on the car, quadrotor, or hardware vehicle where the actual observation errors exceed the state-dependent bounds and cause constraint violations despite the optimized policy.

Figures

Figures reproduced from arXiv: 2604.24894 by Antoine P. Leeman, Glen Chou, Melanie N. Zeilinger, Shuyu Zhan.

Figure 1
Figure 1. Figure 1: We stabilize a Unitree G1 humanoid (59 states) around a backflip trajectory that is jointly optimized, enabling robust constraint satisfaction. (a) view at source ↗
Figure 2
Figure 2. Figure 2: ), which we call VISION-SLS. To make solving (3) tractable, we introduce two approximations. First, instead of planning directly from high-dimensional images, we map each observation y ∈ R ny to a low-dimensional reduced measurement y r ∈ R nr . Concretely, we compose a pre-trained DINO feature extractor [28] with a learned projection to obtain p dino(·) : R ny → R nr together with a corresponding state-ba… view at source ↗
Figure 3
Figure 3. Figure 3: On the classic light-dark domain, our method optimally visits the view at source ↗
Figure 4
Figure 4. Figure 4: (a) Car in an occluded parking lot with a top-down camera avoiding red obstacles. (b) Grayscale shows the learned uncertainty for the reduced view at source ↗
Figure 5
Figure 5. Figure 5: Controlling a quadcopter with an onboard camera, showcasing view at source ↗
Figure 6
Figure 6. Figure 6: For quadrotor visuomotor control, we plot the reachable tubes (shaded view at source ↗
Figure 7
Figure 7. Figure 7: Humanoid (all tubes, closed-loop only) view at source ↗
Figure 8
Figure 8. Figure 8: Humanoid (all tubes, open-loop and closed-loop). view at source ↗
read the original abstract

We propose VISION-SLS, a method for nonlinear output-feedback control from high-resolution RGB images which provides robust constraint satisfaction guarantees under calibrated uncertainty bounds despite partial observability, sensor noise, and nonlinear dynamics. To enable scalability while retaining guarantees, we propose: (i) a learned low-dimensional observation map from pretrained visual features with state-dependent error bounds, and (ii) a causal affine time-varying output-feedback policy optimized via System Level Synthesis (SLS). We develop a scalable, novel solver for the resulting nonconvex program that leverages sequential convex programming coupled with efficient Riccati recursions. On two simulated visuomotor tasks (a 4D car and a 10D quadrotor) with >= 512 x 512 pixels and a 59D humanoid task with partial observability, our method enables safe, information-gathering behavior that reduces uncertainty while guaranteeing constraint satisfaction with empirically-calibrated error bounds. We also validate our method on hardware, safely controlling a ground vehicle from onboard images, outperforming baselines in safety rate and solve times. Together, these results show that learned visual abstractions coupled with an efficient solver make SLS-based safe visuomotor output-feedback practical at scale. The code implementation of our method is available at https://github.com/trustworthyrobotics/VISION-SLS.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper proposes VISION-SLS, a framework for safe nonlinear output-feedback control from high-resolution RGB images. It learns a low-dimensional observation map from pretrained visual features equipped with state-dependent error bounds and optimizes a causal affine time-varying output-feedback policy via System Level Synthesis (SLS). A scalable solver using sequential convex programming and Riccati recursions is introduced. The method is evaluated on simulated visuomotor tasks (4D car, 10D quadrotor, 59D humanoid with partial observability) and hardware validation on a ground vehicle, claiming to enable safe information-gathering behavior that reduces uncertainty while guaranteeing constraint satisfaction under empirically calibrated bounds. Open-source code is provided.

Significance. If the state-dependent error bounds remain valid under closed-loop nonlinear dynamics, this work would be a meaningful contribution to scalable safe visuomotor control by integrating learned visual representations with SLS-based robust synthesis. The demonstrations across increasing state dimensions and real hardware, combined with the open code release, support practical applicability. The approach could influence research on perception-based control with formal guarantees in robotics.

major comments (2)
  1. [§3 and §4.1] §3 (SLS Formulation) and §4.1 (Observation Map): The robust constraint satisfaction guarantees are conditional on the empirically calibrated state-dependent error bounds containing the true state with the stated coverage probability. However, the calibration uses separate offline data, and no formal transfer argument or empirical verification of bound coverage is provided for trajectories under the closed-loop SLS policy, which is explicitly designed to gather information and alter the state-observation distribution relative to calibration data. This is load-bearing for the central claim of guaranteed safety under nonlinear dynamics and sensor noise.
  2. [§5] §5 (Experiments): While safety rates and solve times are reported for the 4D car, 10D quadrotor, 59D humanoid, and hardware ground vehicle, the manuscript does not quantify the actual coverage probability of the learned error bounds on the closed-loop trajectories generated by the optimized policy. This omission weakens support for the claim that guarantees hold with the calibrated bounds.
minor comments (3)
  1. [Abstract and §4.1] The abstract states 'empirically-calibrated error bounds' but the exact calibration procedure (e.g., dataset size, coverage target, validation split) should be stated more explicitly in §4.1 for reproducibility.
  2. [Figures in §5] Figure captions and axis labels in the results section could more clearly indicate whether plotted uncertainty reductions correspond to the state-dependent bounds or to empirical variance.
  3. [§2] Minor notation inconsistency: the observation map is referred to interchangeably as 'learned low-dimensional' and 'from pretrained visual features'; a single consistent definition early in §2 would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review, which highlights an important aspect of our claims regarding bound validity. We agree that direct empirical verification of coverage under the closed-loop policy strengthens the manuscript and have revised accordingly. Below we respond point by point.

read point-by-point responses

  1. Referee: [§3 and §4.1] §3 (SLS Formulation) and §4.1 (Observation Map): The robust constraint satisfaction guarantees are conditional on the empirically calibrated state-dependent error bounds containing the true state with the stated coverage probability. However, the calibration uses separate offline data, and no formal transfer argument or empirical verification of bound coverage is provided for trajectories under the closed-loop SLS policy, which is explicitly designed to gather information and alter the state-observation distribution relative to calibration data. This is load-bearing for the central claim of guaranteed safety under nonlinear dynamics and sensor noise.

    Authors: We agree that the guarantees are conditional on the bounds holding and that the information-gathering policy alters the state distribution relative to offline calibration data. The SLS synthesis provides robust constraint satisfaction for any realization within the stated bounds; the state-dependent nature of the bounds is intended to adapt to visited states. While a general formal transfer theorem would require strong assumptions on the visual feature map that are difficult to verify, we have added in the revision an empirical coverage analysis on the closed-loop trajectories (new subsection 5.4). This analysis shows that the observed coverage probability remains at or above the calibrated level across all tasks, supporting that the bounds continue to contain the true state with the required probability under the optimized policies. revision: yes

  2. Referee: [§5] §5 (Experiments): While safety rates and solve times are reported for the 4D car, 10D quadrotor, 59D humanoid, and hardware ground vehicle, the manuscript does not quantify the actual coverage probability of the learned error bounds on the closed-loop trajectories generated by the optimized policy. This omission weakens support for the claim that guarantees hold with the calibrated bounds.

    Authors: We thank the referee for identifying this gap. In the revised manuscript we now explicitly quantify the empirical coverage probability of the learned error bounds on the closed-loop trajectories for every task and the hardware experiment. The reported coverage rates meet or exceed the target probability used during calibration, even as the policy reduces uncertainty. This addition directly addresses the concern and provides stronger empirical support for the safety claims. revision: yes

Circularity Check

0 steps flagged

No significant circularity; established SLS and separate-data calibration

full rationale

The paper's derivation chain invokes established System Level Synthesis (SLS) theory to optimize the causal affine time-varying output-feedback policy and calibrates state-dependent error bounds for the learned observation map on separate data. The resulting nonconvex program is solved via sequential convex programming and Riccati recursions, with guarantees conditional on the calibrated bounds. No load-bearing step reduces a prediction or guarantee to a quantity defined in terms of itself, nor does any self-citation chain or ansatz smuggling substitute for independent content. The central claim of safe information-gathering behavior therefore remains self-contained against external SLS results and empirical calibration.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the accuracy of empirically calibrated state-dependent error bounds and on the correctness of the SLS-based robust synthesis under those bounds; both are taken from prior literature or data rather than derived inside the paper.

free parameters (1)
  • state-dependent error bounds = empirically calibrated
    Bounds are calibrated empirically from data to quantify uncertainty in the learned observation map.
axioms (1)
  • standard math System Level Synthesis yields robust constraint satisfaction for linear systems under bounded disturbances
    The paper invokes established SLS theory to obtain the output-feedback policy and its guarantees.

pith-pipeline@v0.9.0 · 5561 in / 1250 out tokens · 28035 ms · 2026-05-08T02:38:51.498435+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Over-Approximating Minimizer Sets of Constrained Convex Programs with Parametric Uncertainty via Reachability Analysis

    math.OC 2026-04 unverdicted novelty 6.0

    A reachability-analysis method on projected gradient descent dynamics produces certified outer approximations to the minimizer sets of strongly convex programs whose costs depend on bounded uncertain parameters.

Reference graph

Works this paper leans on

57 extracted references · 11 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Deep vit features as dense visual descriptors.arXiv preprint arXiv:2112.05814, 2(3):4, 2021

    Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. CoRR, abs/2112.05814, 2021

    work page arXiv 2021

  2. [2]

    Doyle, Steven H

    James Anderson, John C. Doyle, Steven H. Low, and Nikolai Matni. System level synthesis.Annu. Rev. Control., 47:364–393, 2019

    2019

  3. [3]

    Casadi: a software frame- work for nonlinear optimization and optimal control

    Joel AE Andersson, Joris Gillis, Greg Horn, James B Rawlings, and Moritz Diehl. Casadi: a software frame- work for nonlinear optimization and optimal control. Mathematical Programming Computation, 11(1):1–36, 2019

    2019

  4. [4]

    Version 11.1., 2019

    MOSEK ApS.The MOSEK optimization toolbox for Python manual. Version 11.1., 2019. URL http://docs. mosek.com/9.0/toolbox/index.html

    2019

  5. [5]

    Autoencoders, unsupervised learning, and deep architectures

    Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham W. Taylor, and Daniel L. Silver, editors,Unsupervised and Transfer Learning - Workshop held at ICML 2011, Bellevue, Washington, USA, July 2, 2011, volume 27 ofJMLR Proceedings, pages 37–50. JMLR.org, 2012

    2011

  6. [6]

    Robust locally-linear con- trollable embedding

    Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Bui, and Ali Ghodsi. Robust locally-linear con- trollable embedding. InAISTATS, 2018

    2018

  7. [7]

    Dynamic programming and sub- optimal control: A survey from adp to mpc.European journal of control, 11(4-5):310–334, 2005

    Dimitri P Bertsekas. Dynamic programming and sub- optimal control: A survey from adp to mpc.European journal of control, 11(4-5):310–334, 2005

    2005

  8. [8]

    The pinocchio c++ library: A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives

    Justin Carpentier, Guilhem Saurel, Gabriele Buondonno, Joseph Mirabel, Florent Lamiraux, Olivier Stasse, and Nicolas Mansard. The pinocchio c++ library: A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. In2019 IEEE/SICE International Symposium on System Integra- tion (SII), pages 614–619. IEEE, 2019

    2019

  9. [9]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  10. [10]

    Synthesizing stable reduced-order visuomotor policies for nonlinear systems via sums-of-squares optimization

    Glen Chou and Russ Tedrake. Synthesizing stable reduced-order visuomotor policies for nonlinear systems via sums-of-squares optimization. In2023 62nd IEEE Conference on Decision and Control (CDC), pages 624–

  11. [11]

    Model error propagation via learned contraction metrics for safe feedback motion planning of unknown systems

    Glen Chou, Necmiye Ozay, and Dmitry Berenson. Model error propagation via learned contraction metrics for safe feedback motion planning of unknown systems. In2021 60th IEEE Conference on Decision and Control (CDC), pages 3576–3583. IEEE, 2021

    2021

  12. [12]

    Safe output feedback motion planning from images via learned perception modules and contraction theory

    Glen Chou, Necmiye Ozay, and Dmitry Berenson. Safe output feedback motion planning from images via learned perception modules and contraction theory. In International Workshop on the Algorithmic Foundations of Robotics, pages 349–367. Springer, 2022

    2022

  13. [13]

    Certainty equivalent perception-based control

    Sarah Dean and Benjamin Recht. Certainty equivalent perception-based control. InLearning for Dynamics and Control, pages 399–411. PMLR, 2021

    2021

  14. [14]

    Guaranteeing safety of learned perception modules via measurement-robust control bar- rier functions

    Sarah Dean, Andrew Taylor, Ryan Cosner, Benjamin Recht, and Aaron Ames. Guaranteeing safety of learned perception modules via measurement-robust control bar- rier functions. InConference on Robot Learning, pages 654–670. PMLR, 2021

    2021

  15. [15]

    Lim, Johnathan Tucker, Zachary N

    Sampada Deglurkar, Michael H. Lim, Johnathan Tucker, Zachary N. Sunberg, Aleksandra Faust, and Claire J. Tomlin. Compositional learning-based planning for vi- sion pomdps. InLearning for Dynamics and Control Conference, L4DC 2023, volume 211 ofProceedings of Machine Learning Research, pages 469–482. PMLR, 2023

    2023

  16. [16]

    Safe Large-Scale Robust Nonlinear MPC in Milliseconds via Reachability-Constrained System Level Synthesis on the GPU

    Jeffrey Fang and Glen Chou. Safe large-scale robust non- linear mpc in milliseconds via reachability-constrained system level synthesis on the gpu.arXiv preprint arXiv:2604.07644, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [17]

    Grossman

    Walter D. Grossman. Observers for discrete-time non- linear systems. InPhD thesis, New Jersey Institute of Technology, 1999

    1999

  18. [18]

    Gurobi Optimizer Reference Manual, 2026

    Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2026. URL https://www.gurobi.com

    2026

  19. [19]

    Belief space planning assuming maximum likelihood observations

    Robert Platt Jr., Russ Tedrake, Leslie Kaelbling, and Tom´as Lozano-P ´erez. Belief space planning assuming maximum likelihood observations. InR:SS, 2010

    2010

  20. [20]

    Planning with learned dynamics: Probabilis- tic guarantees on safety and reachability via lipschitz constants.IEEE Robotics and Automation Letters, 6(3): 5129–5136, 2021

    Craig Knuth, Glen Chou, Necmiye Ozay, and Dmitry Berenson. Planning with learned dynamics: Probabilis- tic guarantees on safety and reachability via lipschitz constants.IEEE Robotics and Automation Letters, 6(3): 5129–5136, 2021

    2021

  21. [21]

    Statistical safety and robustness guarantees for feedback motion planning of unknown underactuated stochastic systems

    Craig Knuth, Glen Chou, Jamie Reese, and Joseph Moore. Statistical safety and robustness guarantees for feedback motion planning of unknown underactuated stochastic systems. In2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 12700– 12706, 2023

    2023

  22. [22]

    Optimization- based locomotion planning, estimation, and control de- sign for the atlas humanoid robot.Autonomous robots, 40(3):429–455, 2016

    Scott Kuindersma, Robin Deits, Maurice Fallon, Andr ´es Valenzuela, Hongkai Dai, Frank Permenter, Twan Koolen, Pat Marion, and Russ Tedrake. Optimization- based locomotion planning, estimation, and control de- sign for the atlas humanoid robot.Autonomous robots, 40(3):429–455, 2016

    2016

  23. [23]

    IFAC-PapersOnLine48(28), 1082–1087 (2015) https://doi.org/10.1016/j.ifacol

    Antoine P. Leeman, Johannes K ¨ohler, Florian Messerer, Amon Lahr, Moritz Diehl, and Melanie N. Zeilinger. Fast system level synthesis: Robust model predictive control using Riccati recursions.IFAC-PapersOnLine (Conference on Nonlinear MPC), 58(18):173–180, 2024. ISSN 2405-8963. doi: https://doi.org/10.1016/j.ifacol. 2024.09.027

    work page doi:10.1016/j.ifacol 2024

  24. [24]

    Guaranteed robust nonlinear mpc via distur- bance feedback.arXiv preprint arXiv:2509.18760, 2025

    Antoine P Leeman, Johannes K ¨ohler, and Melanie N Zeilinger. Guaranteed robust nonlinear mpc via distur- bance feedback.arXiv preprint arXiv:2509.18760, 2025

    work page arXiv 2025

  25. [25]

    Leeman, Johannes K ¨ohler, Andrea Zanelli, Samir Bennani, and Melanie N

    Antoine P. Leeman, Johannes K ¨ohler, Andrea Zanelli, Samir Bennani, and Melanie N. Zeilinger. Robust non- linear optimal control via system level synthesis.IEEE Transactions on Automatic Control, 70(7):4780–4787,

  26. [26]

    doi: 10.1109/TAC.2025.3552482

    work page doi:10.1109/tac.2025.3552482 2025

  27. [27]

    End-to-end training of deep visuomotor policies

    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016

    2016

  28. [28]

    A dual-control effect preserving formulation for nonlin- ear output-feedback stochastic model predictive control with constraints.IEEE Control Systems Letters, 7:1171– 1176, 2022

    Florian Messerer, Katrin Baumg ¨artner, and Moritz Diehl. A dual-control effect preserving formulation for nonlin- ear output-feedback stochastic model predictive control with constraints.IEEE Control Systems Letters, 7:1171– 1176, 2022

    2022

  29. [29]

    Zhang, Sebas- tian Baltes, and Christoph Treude

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv ´e J ´egou, Julien Mair...

    work page internal anchor Pith review doi:10.48550/arxiv 2023

  30. [30]

    Robust output feedback control with guaranteed constraint satisfaction

    Sadra Sadraddini and Russ Tedrake. Robust output feedback control with guaranteed constraint satisfaction. InProceedings of the 23rd International Conference on Hybrid Systems: Computation and Control, pages 1–10, 2020

    2020

  31. [31]

    Gpu acceleration of admm for large-scale quadratic program- ming.Journal of Parallel and Distributed Computing, 144:55–67, 2020

    Michel Schubiger, Goran Banjac, and John Lygeros. Gpu acceleration of admm for large-scale quadratic program- ming.Journal of Parallel and Distributed Computing, 144:55–67, 2020

    2020

  32. [32]

    Ving: Learn- ing open-world navigation with visual goals

    Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Ving: Learn- ing open-world navigation with visual goals. InIEEE International Conference on Robotics and Automation, ICRA 2021, Xi’an, China, May 30 - June 5, 2021, pages 13215–13222. IEEE, 2021

    2021

  33. [33]

    A system level approach to tube-based model predictive control.IEEE Control Systems Letters, 6:776–781, 2021

    Jerome Sieber, Samir Bennani, and Melanie N Zeilinger. A system level approach to tube-based model predictive control.IEEE Control Systems Letters, 6:776–781, 2021

    2021

  34. [34]

    Robust tracking with model mismatch for fast and safe planning: an sos optimization approach

    Sumeet Singh, Mo Chen, Sylvia L Herbert, Claire J Tomlin, and Marco Pavone. Robust tracking with model mismatch for fast and safe planning: an sos optimization approach. InInternational Workshop on the Algorithmic Foundations of Robotics, pages 545–564. Springer, 2018

    2018

  35. [35]

    Safety beyond the training data: Robust out-of- distribution mpc via conformalized system level synthe- sis.arXiv preprint arXiv:2602.12047, 2026

    Anutam Srinivasan, Antoine Leeman, and Glen Chou. Safety beyond the training data: Robust out-of- distribution mpc via conformalized system level synthe- sis.arXiv preprint arXiv:2602.12047, 2026

    work page arXiv 2026

  36. [36]

    Osqp: An operator splitting solver for quadratic programs.Mathematical Programming Computation, 12(4):637–672, 2020

    Bartolomeo Stellato, Goran Banjac, Paul Goulart, Al- berto Bemporad, and Stephen Boyd. Osqp: An operator splitting solver for quadratic programs.Mathematical Programming Computation, 12(4):637–672, 2020

    2020

  37. [37]

    A generalized itera- tive lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems

    Emanuel Todorov and Weiwei Li. A generalized itera- tive lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. InProceedings of the 2005, American Control Conference, 2005., pages 300–306. IEEE, 2005

    2005

  38. [38]

    En- forcing safety for vision-based controllers via control barrier functions and neural radiance fields, 2023

    Mukun Tong, Charles Dawson, and Chuchu Fan. En- forcing safety for vision-based controllers via control barrier functions and neural radiance fields, 2023. URL https://arxiv.org/abs/2209.12266

    work page arXiv 2023

  39. [39]

    acados—a modular open-source framework for fast embedded optimal control.Mathematical Programming Computation, 14 (1):147–183, 2022

    Robin Verschueren, Gianluca Frison, Dimitris Kouzoupis, Jonathan Frey, Niels van Duijkeren, Andrea Zanelli, Branimir Novoselnik, Thivaharan Albin, Rien Quirynen, and Moritz Diehl. acados—a modular open-source framework for fast embedded optimal control.Mathematical Programming Computation, 14 (1):147–183, 2022

    2022

  40. [40]

    On the imple- mentation of an interior-point filter line-search algorithm for large-scale nonlinear programming.Mathematical programming, 106(1):25–57, 2006

    Andreas W ¨achter and Lorenz T Biegler. On the imple- mentation of an interior-point filter line-search algorithm for large-scale nonlinear programming.Mathematical programming, 106(1):25–57, 2006

    2006

  41. [41]

    Localized lqg optimal control for large-scale systems

    Yuh-Shyang Wang and Nikolai Matni. Localized lqg optimal control for large-scale systems. In2016 Ameri- can Control Conference (ACC), pages 1954–1961. IEEE, 2016

    1954

  42. [42]

    Riedmiller

    Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. InNeural Information Processing Systems, pages 2746–2754, 2015

    2015

  43. [43]

    Pappas, Rahul Mangharam, and Lars Lindemann

    Shuo Yang, George J. Pappas, Rahul Mangharam, and Lars Lindemann. Safe perception-based control under stochastic sensor uncertainty using conformal prediction,

  44. [44]

    URL https://arxiv.org/abs/2304.00194

    work page arXiv

  45. [45]

    Robustly Constrained Dynamic Games for Uncertain Nonlinear Dynamics

    Shuyu Zhan, Chih-Yuan Chiu, Antoine P Leeman, and Glen Chou. Robustly constrained dynamic games for uncertain nonlinear dynamics.arXiv preprint arXiv:2509.16826, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  46. [46]

    Learning invariant representations for reinforcement learning without recon- struction

    Amy Zhang, Rowan Thomas McAllister, Roberto Calan- dra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without recon- struction. In9th International Conference on Learning Representations, ICLR, 2021

    2021

  47. [47]

    Efficient neural network ro- bustness certification with general activation functions

    Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network ro- bustness certification with general activation functions. Neural Information Processing Systems, 31, 2018

    2018

  48. [48]

    Smith, Pieter Abbeel, Matthew J

    Marvin Zhang, Sharad Vikram, Laura M. Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SO- LAR: deep structured representations for model-based reinforcement learning. InICML, 2019

    2019

  49. [49]

    2411.04983 , archiveprefix =

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained vi- sual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024

    work page arXiv 2024

  50. [50]

    T−1X k=0 ∆ˆxk ∆uk ⊤ Q0 0R ∆ˆxk ∆uk + ∆ˆx⊤ T P∆ˆxT # , (46) Jobs :=E

    Hongyu Zhou and Vasileios Tzoumas. Safe control of partially-observed linear time-varying systems with minimal worst-case dynamic regret, 2023. APPENDIX A. Proof of Proposition 2 Proof:As per (8), the state and input trajectories are affine images ofℓ ∞ balls, hence can be written as Minkowski sums x∈ ¯x⊕Φ xwEBnxT ⊕Φ xeFBnxT , u∈ ¯u⊕Φ uwEBnxT ⊕Φ ueFBnxT ,...

    2023

  51. [51]

    In the humanoid experiment, we have a reference trajectory instead of a single goal state

    Cost Function:The cost function (27a) is the sum of a trajectory cost (54) and a tube cost (55) wherex g is the goal state. In the humanoid experiment, we have a reference trajectory instead of a single goal state. J(z,v,τ,Φ) =J traj +J tube (53) Jtraj(z,v) = T−1X k=0 ((zk −x g)⊤ ¯Q(zk −x g) +v ⊤ k ¯Rvk) + (zT −x g)⊤ ¯P(z T −x g)(54) Jtube(z,v,τ,Φ) = Q1/2...

  52. [52]

    The constraints include state constraints (x k ∈[−2,5]× [−2,5]) and control constraints (u k ∈[−1,1]×[−1,1]), as well as terminal state constraintsx T ∈[−0.15,0.15]× [−0.15,0.15]

    Light-Dark:We consider 2D single integrator dynamics: ˙px ˙py = vx vy (56) with statex= (p x, py)∈R 2 and controlu= (v x, vy)∈R 2. The constraints include state constraints (x k ∈[−2,5]× [−2,5]) and control constraints (u k ∈[−1,1]×[−1,1]), as well as terminal state constraintsx T ∈[−0.15,0.15]× [−0.15,0.15]. The initial nominal state is¯x 0 = (0,2)and go...

  53. [53]

    The state constraints are[−3.5,0.5]× [−2,1.9]×[−2π,2π]×[−1,2]and input constraints are [−π, π]×[−4,4]

    Parking-Lot:We consider Dubins’ car dynamics:   ˙px ˙py ˙θ ˙v   =   vcosθ vsinθ u1 u2   (57) with x-positionp x, y-positionp y, orientationθ, and linear velocityvas states. The state constraints are[−3.5,0.5]× [−2,1.9]×[−2π,2π]×[−1,2]and input constraints are [−π, π]×[−4,4]. The terminal state constraints are[0,0.45]× [0.35,0.65]×[−2π,2π]×...

  54. [54]

    Quadrotor:We consider 3D quadrotor dynamics with ten states [33]:   ˙px ˙py ˙pz ˙vx ˙vy ˙vz ˙θx ˙θy ˙ωx ˙ωy   =   vx vy vz gtan(θ x)−3v x gtan(θ y)−3v y uz −g−v z −10θx +ω x −10θy +ω y −10θx + 50ux −10θy + 50uy   (58) The state constraints arex T ∈[−10,10]×[−10,10]×[0,10]× [−5,5]×[−5,5]×[−5,5...

  55. [55]

    Parking-Lot Perception Error Calibration:To quantify how well the learned measurement-error boundˆe(x)captures the true perception residuals, we evaluate its empirical cov- erage on a held-out dataset. For each sample{x i, yi}, we compute the residualr i =∥p(ϕ(y i))−h r(xi)∥∞ and report the fraction of points whose residual lies below the predicted bound,...

  56. [56]

    Unitree G1 Humanoid:Refer to the URDF of G1 with 23 DOF for joint angle bounds. The pose ([p x, py, pz, qx, qy, qz, qw]) bounds are [−10,−10,−10,−1,−1,−1,−1]×[10,10,10,1,1,1,1], linear velocity bounds are all−[30,30], angular velocity bounds are all[−40π,40π], and joint velocity bounds are all[−20π,20π]. We constrain the wrist states to be within [−0.5,0....

  57. [57]

    For this experiment, the robot plans over a horizon of T= 60from¯x 0 = (0,0,0,0,0)to a target state xg = (3,1.5,0,0,0)

    Hardware:We consider a modified Dubin’s car dynam- ics:   ˙px ˙py ˙θ ˙v ˙ω   =   vcosθ vsinθ ω u1 u2   (60) The Turtlebot robot should satisfy state constraints [px, py, pθ, v, ω]∈[−5,5]×[−5,5]×[−2π,2π]×[−0.2,0.4]× [−1,1](position, heading, linear velocity, angular velocity), input constraints[u 1, u2]∈[−0.1,0.1]×[−0.2,0.2](linear ...