Pith. sign in

REVIEW 3 major objections 6 minor 45 references

Fast Non-Episodic Adaptive Tuning of Robot Controllers with Online Policy Optimization

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a single continuous trajectory with no resets, M-GAPS tunes a nonlinear quadrotor controller to near-expert performance in about 15 seconds, beats episodic baselines, and adapts to unmodeled wind and payloads.

desk verdict A credible hardware study of non-episodic online policy optimization with honest limitations, a reproducibility gap, and an unverified but acknowledged contractiveness assumption; worth a serious referee. read the letter →

arxiv 2507.10914 v1 pith:VKRL2T22 submitted 2025-07-15 cs.RO cs.SYeess.SY

James A. Preiss , Fengze Xie , Yiheng Lin , Adam Wierman , Yisong Yue This is my paper · ORCID
classification cs.ROcs.SYeess.SY
keywords non-episodiconlinepolicyoptimizationM-GAPSquadrotorgeometriccontrolcontrollerparametertuningmodel-basedgradienthardwareexperimentsdescentadaptive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tuning a nonlinear robot controller normally means either hand-tuning or episodic optimization that resets the problem, but a deployed robot faces a single continuous trajectory with time-varying dynamics, costs, and goals. This paper argues that model-based online policy optimization can work non-episodically on real hardware: it implements M-GAPS, an algorithm that keeps one sensitivity matrix per timestep and updates the policy parameters with an approximate gradient that costs $O(1)$ per step. On a quadrotor with a nonlinear geometric controller, M-GAPS recovers near-expert tracking after roughly 15 seconds from a detuned start, beats episodic model-based and model-free baselines, and keeps improving under a periodic fan array and a heavy payload. The paper also shows the same method tunes a 1:6-scale Ackermann car, cutting tracking error about fivefold within one lap. The significance is a practical middle path between adaptive control, which is too restrictive in policy class, and model-free reinforcement learning, which is too data-hungry for fast adaptivity.

What carries the argument

The load-bearing object is the $O(1)$ sensitivity recursion $y_{t+1} = (\partial g_t/\partial x_t + \partial g_t/\partial u_t\, \partial \pi_t/\partial x_t) y_t + \partial g_t/\partial u_t\, \partial \pi_t/\partial \theta_t$, together with the resulting approximate gradient $G_t = (\partial f_t/\partial x_t + \partial f_t/\partial u_t\, \partial \pi_t/\partial x_t) y_t + \partial f_t/\partial u_t\, \partial \pi_t/\partial \theta_t$. This is an online, non-episodic approximation of the ideal surrogate-cost gradient $\nabla F_t(\theta_t)$, where $F_t(\theta)$ is the cost at time $t$ of having used the same parameter $\theta$ at every earlier timestep; computing the true gradient would require re-simulating the whole trajectory from scratch at every step. M-GAPS instead treats the deployed parameter as if it had been constant and differentiates along the actually visited trajectory. The validity of this approximation rests on closed-loop contractiveness: each fixed parameter value steers the state toward its own preferred trajectory at an exponential rate, so slow parameter changes keep the sensitivity matrix accurate. Two reparameterizations improve the landscape: the rotation state lives in the Lie algebra $\mathfrak{so}(3)$ so the state space is Euclidean, and the positive feedback gains are optimized through $\theta = \log \vartheta$, matching the empirically observed scaling of gradient sensitivity.

What would settle it

Simulate or run the same figure-8 experiment while comparing the approximate gradient $G_t$ from the paper's equation (6) with the true surrogate gradient $\nabla F_t(\theta_t)$ computed by full re-simulation at each timestep; if their inner product is often negative or the sensitivity error grows without bound, the contractiveness assumption fails and the theory no longer covers the method.

Watch

Extended reading notes

Core claim

The paper's central claim is that M-GAPS, a model-based non-episodic online policy optimizer, is practical on real quadrotor hardware and finds near-optimal controller parameters more quickly than episodic baselines, even when the dynamics model is imperfect. The algorithm maintains a sensitivity state and updates $\theta_{t+1} = \theta_t - \eta G_t$, using a gradient estimate taken along the actually visited trajectory while ignoring the fact that the policy parameter has been changing. With a logarithmic reparameterization of the feedback gains and an $\mathfrak{so}(3)$-valued rotation state, the paper reports that M-GAPS closes most of the gap to an expert-tuned parameter in about 15 seconds, is less sensitive than the model-based episodic baseline to the choice of episode length, and substantially outperforms the expert parameter under periodic fan wind and a 60 percent payload increase; the same method cuts tracking error by about five times within one lap on a 1:6-scale car. The theory carries local-regret guarantees if the closed-loop dynamics are contractive, which the paper conjectures rather than proves for this controller.

Load-bearing premise

Everything theoretical rests on the closed-loop quadrotor under the logarithmic policy class being contractive, so the one-step sensitivity recursion tracks the true surrogate-cost gradient; the paper conjectures this rather than proving it.

Editorial extensions

If this is right

  • M-GAPS removes episode length as a hyperparameter and avoids the large parameter steps that episodic updates cause.
  • Because it uses analytic derivatives of an approximate model rather than model-free exploration, it runs at 500 hertz on a 168 MHz microcontroller with 192 kB of memory.
  • The wind and payload experiments show that a fixed nominal dynamics model can support continued improvement under unmodeled disturbances, so model error does not by itself block online policy optimization.
  • The car experiment indicates the method transfers across robot platforms and can serve as automatic controller initialization on new hardware even when online adaptivity is not the main goal.
  • If the contractiveness conjecture is verified, M-GAPS inherits regret guarantees that the episodic baselines do not have.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, not tested in the paper, is to feed an online dynamics estimate into the same sensitivity recursion, letting the optimizer track both policy parameters and model parameters.
  • The differentiability requirement will likely bind on contact-rich systems such as walking robots, and a testable next step is to benchmark M-GAPS on such systems and measure how often gradient steps worsen the cost near discontinuities.
  • The parameter trajectories differ sharply between wind and payload scenarios, which suggests the adapted gains could serve as a diagnostic for what disturbance is present, a use the paper does not explore.
  • Because the local regret guarantee does not rule out bad local minima, a deployed implementation should monitor gradient norm or cost and re-initialize, as with any nonconvex optimizer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies non-episodic online policy optimization for tuning robot controllers, with a focus on nonlinear geometric quadrotor controllers. It implements the M-GAPS algorithm, compares it against two episodic baselines (DiffTune and OPRF), and introduces a logarithmic reparameterization of the policy gains intended to improve the optimization landscape. The experimental evaluation is conducted on real hardware: a Crazyflie 2.0 quadrotor in three scenarios (detuned initialization, time-varying wind, heavy payload) and a 1:6-scale Ackermann-steered car. The central claims are that M-GAPS finds near-optimal parameters more quickly than the episodic baselines, especially when their episode length is not well chosen, and that it substantially outperforms an expert-tuned baseline under unmodeled disturbances.

Significance. If the empirical results hold, this is a valuable demonstration that a non-episodic, model-based online policy optimization algorithm can run in real time on embedded hardware and adapt to unmodeled disturbances without modifying the dynamics model. The detuned-initialization comparison is carefully executed with five trials and error bars (Fig. 2), and the episode-length sensitivity study (Fig. 4) provides a useful practical insight. The logarithmic reparameterization is a simple and potentially transferable contribution. The paper is also honest in its Limitations section about the unverified contractiveness assumption, which is a point in its favor. The main weaknesses are the absence of trial counts and error bars for the large-disturbance experiments and the gap between the theoretical framing and the empirically verified conditions.

major comments (3)
  1. [Section V-C, Figs. 5, 7, 9] The large-disturbance experiments that support the central adaptivity claim (wind, payload, and car) report no trial counts or error bars. Only Fig. 2 explicitly states '±1 standard deviation over 5 trials.' If Figs. 5, 7, and 9 are single traces, the claim that M-GAPS 'substantially outperforms' the expert baseline under wind and payload is not statistically supported, and this is load-bearing for the paper's main conclusion. Please report the number of trials and per-trial variability (e.g., mean ± std over at least three to five runs for each condition), and state whether the disturbance phase and initial conditions were held fixed across trials.
  2. [Section IV-C and Section III-A1] The paper explicitly leaves contractiveness as a conjecture in Section IV-C, yet Section III-A1 states that 'this, in turn, leads to the regret guarantees of M-GAPS [20].' Because the approximation G_t in Eq. (6) is only justified when the closed-loop dynamics are contractive, the theoretical grounding of the quadrotor experiments is not established. The Limitations section (VIII) acknowledges this, but the main-text phrasing still overstates the connection between the theory and the empirical results. The authors should either (a) provide empirical evidence of contractiveness (e.g., measure the decay of the sensitivity y_t or closed-loop contraction during the experiments), or (b) explicitly decouple the empirical claims from the regret framework and avoid implying that the regret bounds apply to the systems tested.
  3. [Section V-B, Fig. 4] The 'hindsight best' and 'hindsight worst' episode lengths for DiffTune appear to be selected using the same experimental data shown in Fig. 4. This is a post-hoc selection and makes the comparison in Fig. 2 somewhat circular: DiffTune⋆ is the best-performing variant selected from the same runs. The sensitivity analysis itself is valuable, but the claim that 'DiffTune is close when the episode length is optimal' would be stronger if the selection protocol were specified in advance or if the full episode-length sweep were presented as a hyperparameter study with appropriate caveats about multiple comparisons.
minor comments (6)
  1. [Section I, first paragraph] There is a typo: 'data-inefficient learning that that impedes adaptivity' should be 'that impedes adaptivity.'
  2. [Section IV-A] The statement 'The logarithmic map log : SO(n) 7→ so(n) is a continuous function' is imprecise for n=3 because the logarithm is multi-valued; please specify the chosen branch (e.g., the principal branch with rotation angle in [0, π]) or clarify that a branch is fixed.
  3. [Section IV-B] In the definition softclamp(x, y) = y tanh(x/y), the behavior for y = 0 is undefined; since Bxy and Bz are positive in context, this is not a logical error, but a brief note would improve clarity.
  4. [Fig. 3 caption] The caption says 'Row: Optimizer. Left: Gains for horizontal motion axes. Right: Gains for vertical motion axis,' but each optimizer row contains two panels that are not explicitly labeled in the caption; consider referring to panel letters or adding explicit subfigure labels.
  5. [Appendix A.1] The line 'where δ >0 is the discretization interval, 0 < Cξ, Cτ ≪ 1/δ are time delay constants' appears to have a typo: it should likely read '0 < Cξ, Cτ δ ≪ 1' or similar; please check the intended inequality.
  6. [Section V-A] The term 'quasi-regret' is used informally; consider giving a formal definition (e.g., cumulative cost difference from the expert parameter θm) at first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical claims rest on external hardware baselines, and the theory assumption is an explicitly labeled conjecture, not a derived result.

full rationale

The paper's load-bearing claims are empirical hardware comparisons, not derivations. M-GAPS is imported from the authors' prior work [20], but it is evaluated against external baselines (DiffTune [7], OPRF [42]) and an external symbolic differentiation tool (SymForce [24]), and the expert baseline θm is manually tuned in separate prior work [30]. The only theoretical input that could be questioned is the contractiveness assumption behind the gradient approximation G_t; however, the paper does not derive its results from that assumption. Instead, Section IV-C explicitly states: 'we conjecture that with suitable assumptions on the initial error and disturbances, our controller and state parameterization is contractive,' and Section VIII lists contractiveness verification as a limitation. An unproven conjecture weakens the theoretical guarantees but does not make the empirical results circular. No fitted parameter is renamed as a prediction: the quasi-regret is explicitly labeled as a surrogate because the true optimal is unknown, and the detuned initialization is defined relative to the manual baseline rather than fit to the outcome. The logarithmic reparameterization is motivated by an empirical scaling observation, but it is a preprocessing choice, not the predicted output. The self-citations to [19,20] are normal use of previously published theory, and the theory's assumptions do not include the hardware outcomes being reported. Therefore no circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on a handful of unverified or hand-tuned ingredients: contractiveness is conjectured rather than checked, the nominal model and differentiable simulation are trusted, cost weights and hyperparameters are selected by hand, and the expert parameter theta_m is used as the optimality benchmark. None of these are new physical entities; they are modeling and evaluation choices that the paper mostly discloses.

free parameters (7)
  • learning rate eta (M-GAPS, DiffTune, OPRF) = not stated in text
    The learning rate is the main tuning hyperparameter for each optimizer, and the paper does not report the values used, so the experimental results depend on unstated hand tuning.
  • episode length H (DiffTune and OPRF) = hindsight best and worst values not stated
    Episode length is chosen after the fact as hindsight best and hindsight worst in Fig. 4 and Table I; this selection strongly affects DiffTune's performance and the resulting comparison.
  • OPRF perturbation radius epsilon = not stated
    The one-point residual feedback baseline samples perturbed parameters with radius epsilon; the chosen value is not reported, so the baseline performance depends on an undisclosed hyperparameter.
  • quadrotor cost regularization weights = 1e-4, 1e-3, 1e-7, 1e-8
    These weights shape what counts as near-optimal; the text says they were chosen empirically to be as small as possible while preventing attitude oscillations (Section IV-D), so they are hand-fitted to the test scenario.
  • car cost regularization weights = 1/30, 1/15
    Chosen empirically to suppress oscillations (Appendix A), so they are hand-fitted to make the car experiment behave as reported.
  • softclamp bounds Bxy, Bz = not stated
    User-defined upper bounds for the desired angular acceleration, described as critical for the cascaded attitude controller, but exact values are not given.
  • expert baseline parameters theta_m = not reported (10 gains)
    Used as the reference for quasi-regret and the near-optimal claim; it is manually tuned over several days and not proven optimal, so the speed and near-optimality conclusions are relative to this particular vector.
assumptions (4)
  • domain assumption Closed-loop dynamics under the policy class are contractive.
    Section IV-C states a conjecture that the controller and state parameterization are contractive; M-GAPS's theoretical regret guarantees and the validity of the O(1) sensitivity recursion (5) rely on this condition, which is not verified.
  • domain assumption The nominal differentiable dynamics model is accurate enough for gradient computation.
    Section IV-A uses a unit-mass, identity-inertia model with Lie-group rotation discretization; M-GAPS and DiffTune take derivatives of this model about real trajectories, so the gradient directions depend on model fidelity. The wind and payload tests keep the model unchanged to test robustness, but success still assumes the model is not badly wrong.
  • domain assumption The surrogate cost landscape is benign enough that approximate online gradient descent avoids bad local minima.
    Section III-A notes that even with the local regret bound, the algorithm can get stuck in bad local minima; the empirical claims require that the chosen initialization and reparameterization fall in a favorable basin.
  • domain assumption Manually tuned theta_m is near-optimal in the disturbance-free setting.
    Section V-A uses theta_m as the reference for quasi-regret and calls the result near-optimal; if theta_m is not close to the true optimum of the cost, the speed and near-optimality claims are weakened.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fast Non-Episodic Adaptive Tuning of Robot Controllers with Online Policy Optimization." pith.science (2026). https://pith.science/paper/VKRL2T22

@misc{pith2026250710914,
  author       = {Pith},
  title        = {Pith review of: Fast Non-Episodic Adaptive Tuning of Robot Controllers with Online Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKRL2T22}},
  note         = {Machine review of arXiv:2507.10914}
}
read the original abstract

We study online algorithms to tune the parameters of a robot controller in a setting where the dynamics, policy class, and optimality objective are all time-varying. The system follows a single trajectory without episodes or state resets, and the time-varying information is not known in advance. Focusing on nonlinear geometric quadrotor controllers as a test case, we propose a practical implementation of a single-trajectory model-based online policy optimization algorithm, M-GAPS,along with reparameterizations of the quadrotor state space and policy class to improve the optimization landscape. In hardware experiments,we compare to model-based and model-free baselines that impose artificial episodes. We show that M-GAPS finds near-optimal parameters more quickly, especially when the episode length is not favorable. We also show that M-GAPS rapidly adapts to heavy unmodeled wind and payload disturbances, and achieves similar strong improvement on a 1:6-scale Ackermann-steered car. Our results demonstrate the hardware practicality of this emerging class of online policy optimization that offers significantly more flexibility than classic adaptive control, while being more stable and data-efficient than model-free reinforcement learning.

Figures

Figures reproduced from arXiv: 2507.10914 by the authors.

Figure 1
Figure 1. Trajectories of a quadrotor tracking an aggressive figure-8 trajectory under online policy optimization algorithms and [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. “Regret” (cumulative cost difference) versus expert [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Parameter evolution in figure-8 tracking experiment [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Total cost of M-GAPS vs. DiffTune with var￾ious episode lengths in detuned initialization experiment. DiffTune with optimal episode length performs nearly as well as M-GAPS, but degrades with other episode lengths. lengths. The figure-8 trajectory lasts 4 sec and is sy…
Figure 5
Figure 5. Figure 5: Position tracking error under M-GAPS for periodic fan disturbance (§V-C1). Error is averaged per “lap” due to within￾lap variance from the airflow pattern. M-GAPS substantially outperforms the expert θ m tuned for smaller disturbances. 0.9 1.0 value / init param: ki, a…
Figure 6
Figure 6. Figure 6: Parameter evolution under M-GAPS for periodic fan disturbance (§V-C1). Shaded bands indicate when the fans are energized. Some parameters react strongly to fan phase, showing rapid adaptation. C. Large-Disturbance Experiments 1) Time-varying wind: To study short-term a…
Figure 8
Figure 8. Figure 8: , we compare the parameter evolution for the time￾varying-wind and heavy-payload scenarios. We observe large differences. In particular, k z p has the largest change in the heavy-payload scenario, but changes minimally in the wind scenario. The weight acts in the z-axi…
Figure 9
Figure 9. Figure 9: M-GAPS tuning parameters of a nonlinear controller for an Ackermann-steered 1:6-scale car traveling in a circle with period of 12.5 sec and radius of 2.2 m. Tracking error reduced by over 5× within one period. y-axis. This confirms that M-GAPS is adapting to the specif…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 42 canonical work pages

  1. [20]

    Preiss, Fengze Xie, Emile Anand, Soon-Jo Chung, Yisong Yue, and Adam Wierman

    Yiheng Lin, James A. Preiss, Fengze Xie, Emile Anand, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online policy optimization in unknown nonlinear systems. In Conference on Learning Theory (COLT) , 2024

    work page 2024

  2. [1]

    Optimal Algorithms for Online Convex Optimization with Multi- Point Bandit Feedback

    Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal Algorithms for Online Convex Optimization with Multi- Point Bandit Feedback. In Conference on Learning Theory (COLT), 2010

    work page 2010

  3. [2]

    Kakade, and Karan Singh

    Naman Agarwal, Brian Bullins, Elad Hazan, Sham M. Kakade, and Karan Singh. Online Control with Ad- versarial Disturbances. In International Conference on Machine Learning (ICML) , 2019

    work page 2019

  4. [3]

    Zico Kolter

    Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, and J. Zico Kolter. Differentiable MPC for End-to-end Planning and Control. In Advances in Neural Information Processing Systems (NeurIPS) , 2018

    work page 2018

  5. [4]

    On the model-based stochastic value gradient for continuous reinforcement learning

    Brandon Amos, Samuel Stanton, Denis Yarats, and An- drew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Conference on Learning for Dynamics and Control (L4DC), 2021

    work page 2021

  6. [5]

    Infinite-horizon policy-gradient estimation

    Jonathan Baxter and Peter L Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelli- gence Research, 15:319–350, 2001

    work page 2001

  7. [6]

    A survey of iterative learning control

    Douglas A Bristow, Marina Tharayil, and Andrew G Alleyne. A survey of iterative learning control. IEEE Control Systems Magazine , 26(3):96–114, 2006

    work page 2006

  8. [7]

    Difftune: Autotuning through autodifferentiation

    Sheng Cheng, Minkyung Kim, Lin Song, Chengyu Yang, Yiquan Jin, Shenlong Wang, and Naira Hovakimyan. Difftune: Autotuning through autodifferentiation. IEEE Transactions on Robotics , 40:4085–4101, 2024

    work page 2024

Show all 45 references
  1. [8]

    Differentiable simulation

    Stelian Coros, Miles Macklin, Bernhard Thomaszewski, and Nils Th ¨urey. Differentiable simulation. In SIG- GRAPH Asia 2021 Courses , pages 1–142. 2021

    2021

  2. [9]

    Zico Kolter

    Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. End-to- End Differentiable Physics for Learning and Control. In Advances in Neural Information Processing Systems (NeurIPS), 2018

    2018

  3. [10]

    Adaptive Regret for Control of Time-Varying Dynamics

    Paula Gradu, Elad Hazan, and Edgar Minasyan. Adaptive Regret for Control of Time-Varying Dynamics. In Con- ference on Learning for Dynamics and Control (L4DC) , 2023

    2023

  4. [11]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018

    2018

  5. [12]

    Introduction to Online Convex Optimization

    Elad Hazan. Introduction to Online Convex Optimization. MIT Press, 2022

    2022

  6. [13]

    Introduction to Online Nonstochastic Control

    Elad Hazan and Karan Singh. Introduction to Online Nonstochastic Control. CoRR, abs/2211.09619, 2022

    2022 arXiv

  7. [14]

    The Nonstochastic Control Problem

    Elad Hazan, Sham Kakade, and Karan Singh. The Nonstochastic Control Problem. In Conference on Al- gorithmic Learning Theory (ALT) , 2020

    2020

  8. [15]

    Ioannou and Jing Sun

    Petros A. Ioannou and Jing Sun. Robust Adaptive Control. Prentice-Hall, 1995

    1995

  9. [16]

    Scalable deep reinforcement learning for vision- based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning (C...

    2018

  10. [17]

    Kokotovic, and Ioannis Kanel- lakopoulos

    Miroslav Krstic, Petar V . Kokotovic, and Ioannis Kanel- lakopoulos. Nonlinear and Adaptive Control Design . John Wiley & Sons, 1995

    1995

  11. [18]

    Harris McClamroch

    Taeyoung Lee, Melvin Leok, and N. Harris McClamroch. Geometric tracking control of a quadrotor UA V on SE(3). In Conference on Decision and Control (CDC) , 2010

    2010

  12. [19]

    Preiss, Emile Anand, Yingying Li, Yisong Yue, and Adam Wierman

    Yiheng Lin, James A. Preiss, Emile Anand, Yingying Li, Yisong Yue, and Adam Wierman. Online adaptive policy selection in time-varying systems: No-regret via contrac- tive perturbations. In Advances in Neural Information Processing Systems (NeurIPS) , 2023

    2023

  13. [21]

    Universal adaptive control of nonlinear systems

    Brett T Lopez and Jean-Jacques E Slotine. Universal adaptive control of nonlinear systems. IEEE Control Systems Letters, 6:1826–1830, 2021

    2021

  14. [22]

    Preiss, Jedidiah Alindogan, Matthew Anderson, and Soon-Jo Chung

    Elena Sorina Lupu, Fengze Xie, James A. Preiss, Jedidiah Alindogan, Matthew Anderson, and Soon-Jo Chung. MAGIC-VFM: Meta-learning adaptation for ground interaction control with visual foundation models. IEEE Transactions on Robotics , pages 1–20, 2024

    2024

  15. [23]

    Simple random search of static linear policies is competitive for reinforcement learning

    Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  16. [24]

    SymForce: Symbolic Com- putation and Code Generation for Robotics

    Hayk Martiros, Aaron Miller, Nathan Bucki, Bradley Solliday, Ryan Kennedy, Jack Zhu, Tung Dang, Do- minic Pattison, Harrison Zheng, Teo Tomic, Peter Henry, Gareth Cross, Josiah VanderMey, Alvin Sun, Samuel Wang, and Kristen Holtz. SymForce: Symbolic Com- putation and Code Gene...

    2022

  17. [25]

    Minimum snap trajectory generation and control for quadrotors

    Daniel Mellinger and Vijay Kumar. Minimum snap trajectory generation and control for quadrotors. In International Conference on Robotics and Automation (ICRA), 2011

    2011

  18. [26]

    Nonlinear and adap- tive intelligent control techniques for quadrotor UA V–a survey

    Hongwei Mo and Ghulam Farid. Nonlinear and adap- tive intelligent control techniques for quadrotor UA V–a survey. Asian Journal of Control, 21(2):989–1008, 2019

    2019

  19. [27]

    Pods: Policy op- timization via differentiable simulation

    Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy op- timization via differentiable simulation. In International Conference on Machine Learning (ICML) , 2021

    2021

  20. [28]

    Neural-fly enables rapid learning for agile flight in strong winds

    Michael O’Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics , 7(66), 2022

    2022

  21. [29]

    Policy gradient for continuing tasks in dis- counted Markov decision processes

    Santiago Paternain, Juan Andr ´es Bazerque, and Alejan- dro Ribeiro. Policy gradient for continuing tasks in dis- counted Markov decision processes. IEEE Transactions on Automatic Control , 67(9):4467–4482, 2022

    2022

  22. [30]

    Preiss, Wolfgang H ¨onig, Gaurav S

    James A. Preiss, Wolfgang H ¨onig, Gaurav S. Sukhatme, and Nora Ayanian. Crazyswarm: A large nano- quadcopter swarm. In International Conference on Robotics and Automation (ICRA) , 2017

    2017

  23. [31]

    SPNets: Differentiable Fluid Dynamics for Deep Neural Networks

    Connor Schenck and Dieter Fox. SPNets: Differentiable Fluid Dynamics for Deep Neural Networks. In Confer- ence on Robot Learning (CoRL) , 2018

    2018

  24. [32]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

    2017 arXiv

  25. [33]

    Parameter-exploring policy gradients

    Frank Sehnke, Christian Osendorfer, Thomas R ¨uckstieß, Alex Graves, Jan Peters, and J ¨urgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551–559, 2010

    2010

  26. [34]

    Deterministic policy gradient algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML) , 2014

    2014

  27. [35]

    Im- proper Learning for Non-Stochastic Control

    Max Simchowitz, Karan Singh, and Elad Hazan. Im- proper Learning for Non-Stochastic Control. In Confer- ence on Learning Theory (COLT) , 2020

    2020

  28. [36]

    Slotine and W

    J.J.E. Slotine and W. Li. Applied Nonlinear Control . Prentice Hall, 1991

    1991

  29. [37]

    Do differentiable simulators give better policy gradients? In International Conference on Ma- chine Learning (ICML) , 2022

    Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Ma- chine Learning (ICML) , 2022

    2022

  30. [38]

    Paul J. Werbos. A Menu of Designs for Reinforcement Learning Over Time. In Neural Networks for Control . MIT Press, 1991

    1991

  31. [39]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Ma- chine Learning, 8:229–256, 1992

    1992

  32. [40]

    JAX- FEM: A differentiable GPU-accelerated 3D finite ele- ment solver for automatic inverse design and mechanistic data science

    Tianju Xue, Shuheng Liao, Zhengtao Gan, Chanwook Park, Xiaoyu Xie, Wing Kam Liu, and Jian Cao. JAX- FEM: A differentiable GPU-accelerated 3D finite ele- ment solver for automatic inverse design and mechanistic data science. Comput. Phys. Commun. , 291:108802, 2023

    2023

  33. [41]

    Zavlanos

    Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M. Zavlanos. A new one-point residual-feedback oracle for black-box learning and control. Automatica, 136:110006, 2022

    2022

  34. [42]

    Zavlanos

    Yan Zhang, Yi Zhou, Kaiyi Ji, Yi Shen, and Michael M. Zavlanos. Boosting one-point derivative-free online op- timization via residual feedback. IEEE Transactions on Automatic Control, pages 1–8, 2024. APPENDIX A. Details of Ackermann-Steered Car Experiment In this appendix we ...

    2024

  35. [43]

    The system state comprises the car’s position p ∈ R2, body-frame velocity v ∈ R2, heading angle r ∈ so(2), angular velocity ω ∈ so(2), and steering angle ψ ∈ R

    Dynamics and Representation: We consider a planar dynamics model. The system state comprises the car’s position p ∈ R2, body-frame velocity v ∈ R2, heading angle r ∈ so(2), angular velocity ω ∈ so(2), and steering angle ψ ∈ R. (Note that we overload notations for analogous sta...

  36. [44]

    Note that [vd t ]y = 0 for all desired trajectories

    Policy class: The policy class is: τt = − K1 exp −rd t (pt − pd t ) y − K2(vy t + re t vx t ) − K3re t − K4(ωt − ωd t ), ξt = − Kp exp(−rt)(pt − pd t ) x + vd t x , where re t = log exp(rt) exp −rd t denotes the rotation error (acting as an element of R instead of so(2)), and ...

  37. [45]

    The regularization weights were chosen empirically to be as small as possible while suppressing oscillations

    Cost function: The cost function penalizes squared position tracking error with regularization terms on angular velocity and steering: ct = pt − pd t 2 2 + 1 30 (ωt − ωd t )2 + 1 15 τ 2 t . The regularization weights were chosen empirically to be as small as possible while sup...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.