Predictive Lagrangian Optimization for Constrained Reinforcement Learning

Guojian Zhan; Jingliang Duan; Liping Zhang; Puzhen Yuan; Shengbo Eben Li; Tianqi Zhang; Yao Lyu; Zhenzhi Qin; Ziyu Lin

REVIEW 5 major objections 3 minor 15 references

Predictive Lagrangian Optimization for Constrained Reinforcement Learning

T0 review · 5 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For smooth bowl-shaped objectives, choosing the Lagrange multiplier by driving constraint violations to zero solves the constrained RL problem; the paper builds the PLO algorithm on that equivalence and reports a feasible region up to…

desk verdict A genuinely new MPC-based multiplier update and a correct-but-narrow equivalence theorem, undercut by the fact that the theorem does not actually cover PLO as deployed and the experiments are too thin to carry the empirical claims. read the letter →

arxiv 2501.15217 v1 pith:GLRR6RTO submitted 2025-01-25 cs.LG cs.SYeess.SY

Tianqi Zhang , Puzhen Yuan , Guojian Zhan , Ziyu Lin , Yao Lyu , Zhenzhi Qin , Jingliang Duan , Liping Zhang

show 1 more author

Shengbo Eben Li

This is my paper · ORCID

classification cs.LGcs.SYeess.SY

keywords constrainedreinforcementlearningLagrangemultiplierfeedbackcontrolmodelpredictivePIDLagrangianpolicygradientsafetyconstraintsdualoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Constrained reinforcement learning asks an agent to maximize reward while keeping some cost below a limit, and the standard mechanism is a Lagrange multiplier that grows when the constraint is violated. This paper reinterprets that multiplier update as a feedback-control loop: the multiplier is the control input, the policy parameters are the state, and policy gradient descent is the dynamics. The central theorem says that, when the reward and cost objectives are differentiable and strongly convex, the multiplier that maximizes the dual objective is exactly the multiplier that minimizes the remaining constraint violation, so alternating a multiplier update that drives violations toward zero with a policy-gradient step reaches the constrained optimum. This places the existing PID Lagrangian method as one special case of the framework and licenses a new algorithm, predictive Lagrangian optimization (PLO), which uses model predictive control to anticipate future violations. In two control experiments, PLO attains a feasible region up to 7.2% larger than PID Lagrangian with comparable average reward.

What carries the argument

The load-bearing identity is $\Gamma'(\lambda)=J_c(\theta(\lambda))$: the derivative of the dual function $\Gamma(\lambda)=\min_\theta[-J(\theta)+\lambda J_c(\theta)]$ equals the constraint value at the inner-loop minimizer. Under strong convexity of $-J$ and $J_c$, Proposition 1 gives strict monotonicity of $J_c(\theta(\lambda))$ in $\lambda$, and Lemma 1 then yields the equality between the dual optimizer and the minimizer of $|J_c(\theta(\lambda))|$. That equality is what lets the outer-loop multiplier update be replaced by any feedback controller that zeroes the violation; PLO uses an MPC controller whose receding-horizon objective is the sum of predicted squared violations plus a small penalty on the multiplier magnitude.

What would settle it

Evaluate the Hessian of $J_c(\theta)$ for the three-layer tanh policy on the double-integrator or cartpole task at several training checkpoints; if its minimum eigenvalue is negative, the strong-convexity hypothesis of Theorem 1 fails, and the theorem alone cannot support PLO's behavior on that task.

Watch

Extended reading notes

Core claim

The paper's central discovery is an equivalence theorem: under differentiability and strong convexity of $-J(\theta)$ and $J_c(\theta)$, the solution of the primal constrained RL problem (2) coincides with the solution of the multiplier feedback optimal control problem (5), whose objective is to minimize the absolute constraint violation $|J_c(\theta(\lambda))|$ at the policy $\theta(\lambda)$ that minimizes $-J(\theta)+\lambda J_c(\theta)$. The proof runs through the dual function $\Gamma(\lambda)=\min_\theta[-J(\theta)+\lambda J_c(\theta)]$, whose derivative is $J_c(\theta(\lambda))$; Proposition 1 shows $J_c(\theta(\lambda))$ is strictly decreasing in $\lambda$, and Lemma 1 converts this into $\arg\max_{\lambda\ge 0}\Gamma(\lambda)=\arg\min_{\lambda\ge 0}|J_c(\theta(\lambda))|$. Consequently any feedback controller that drives constraint violations to zero is a theoretically sound multiplier update, and the PID Lagrangian method is the special case where that controller is a PID. The authors instantiate the framework with model predictive control in PLO, whose MPC objective accumulates predicted violations over a finite horizon, and report that PLO reaches a larger feasible region than PID Lagrangian at comparable reward.

Load-bearing premise

The load-bearing premise is that $-J(\theta)$ and $J_c(\theta)$ are smooth and strongly convex (bowl-shaped) functions of the policy parameters; the experiments use neural-network policies for which this property has not been verified, so the theorem may not cover the algorithm as actually deployed.

Editorial extensions

If this is right

Any feedback controller that drives $|J_c(\theta(\lambda))|$ to zero becomes a principled multiplier update, enlarging the design space for constrained RL to the full toolbox of feedback control.
PID Lagrangian is recovered as the special case where the controller is a PID, which explains rather than merely stipulates its multiplier rule.
PLO's lookahead lets the multiplier react to predicted violations, and the experiments show a feasible region up to 7.2% larger than PID Lagrangian with comparable mean reward.
Because the equivalence holds at the level of the multiplier loop, new constrained RL algorithms can be assembled by swapping in other controllers without redoing the optimality argument.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves open whether the strong-convexity assumption holds for the three-layer tanh policies used in the experiments; if the theorem could be extended to non-convex settings, it would turn controller design into a safety-tuning recipe for deep RL.
A systematic sweep of the MPC horizon $N$, the regularization $R$, and the safety threshold would show whether the reported 7.2% feasible-region gain persists or is specific to the two tested hyperparameter settings.
Because Theorem 1 treats the outer control problem as exact while PLO solves a finite-horizon approximation with $N=20$ and $R=10^{-4}$, comparing PLO's multiplier sequence against the exact MFOCP solution on a linear-quadratic instance would quantify the approximation gap the proof does not cover.
The theorem assumes noiseless policy-gradient dynamics, whereas PLO runs on estimated gradients; quantifying how gradient noise propagates through the MPC multiplier update is a natural extension the paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is worth a look: recast the Lagrange multiplier update as a feedback control problem, show that an exact solution of that problem matches the primal optimum under strong convexity, and then instantiate the controller with MPC to get PLO. That is a real step beyond the PID Lagrangian of Stooke et al. The proof of Theorem 1 is correct as far as it goes: Eq (8) is a standard implicit-function step, Eq (13) is the envelope theorem, and Lemma 1 is elementary. I think the framework genuinely unifies penalty and Lagrangian methods as P/I controllers, and the suggestion to import other controllers is productive.

Where it goes soft is exactly where your stress-test puts it. Theorem 1 analyzes the exact MFOCP (5). PLO's Algorithm 1 solves (16), a finite-horizon MPC with N=20, R=1e-4, and a receding-horizon step that discards everything but the first multiplier. Nothing in the paper shows that this approximation inherits the property of (5). So the theorem cannot be invoked to justify PLO's multiplier update, even on problems satisfying strong convexity. That is the load-bearing gap. On top of that, the strong convexity assumption is silently dropped when the experiments use a tanh MLP; the paper never acknowledges that the theory does not apply to the deployed policy class.

The empirical section is too thin. One baseline (PID Lagrangian, which is a special case of the framework), no multiple seeds, no error bars. There is also a small internal inconsistency: the conclusion claims PLO outperforms PID in average reward, while Section IV.C says "comparable average reward." That is a wording slip, but it should be cleaned up.

All of this is fixable. Add seeded runs and error bars, include a standard constrained-RL baseline such as CPO or FOCOPS, and either prove an approximation bound for the finite-horizon MPC or explicitly downgrade the theorem to apply only to the idealized update. Even without a full bound, a careful statement of what the theorem does and does not cover would help.

Bottom line: the paper deserves a serious referee. The idea is promising and the formal core is sound under its assumptions, but the theory-algorithm gap is real and must be addressed before publication. I would not cite it as it stands, but I would point students to it as a useful example of a control-theoretic view on Lagrangian methods.

Referee Report

5 major / 3 minor

Summary. The paper proposes a feedback-control perspective on constrained reinforcement learning, treating the Lagrange multiplier as a control input, policy parameters as the system state, and constraint violation as the feedback error. It defines the multiplier feedback optimal control problem (MFOCP) and proves Theorem 1, which claims that solving the MFOCP is equivalent to solving the primal constrained RL problem under differentiability and strong convexity. The paper then instantiates the framework with a model predictive control (MPC) multiplier update, called predictive Lagrangian optimization (PLO), and reports experiments on double integrator and cartpole tasks showing a larger feasible region and comparable reward relative to PID Lagrangian.

Significance. The derivation in Section III.B is mathematically coherent: Eq. (8) is a valid implicit-function step, Eq. (13) follows from the envelope theorem, and Lemma 1 is correct under its stated regularity, though the attainment of the argmax is not addressed. The framework offers a useful unifying viewpoint and correctly identifies PID Lagrangian as a proportional-integral controller. However, the main advertised asset of the paper—the theoretical grounding of PLO—depends on identifying the finite-horizon MPC update of Eq. (16) with the exact MFOCP of Eq. (5), and no such identification is proved. The empirical results are suggestive but not statistically substantiated. If the approximation gap can be closed, the paper would make a solid contribution; as it stands, the central claim is stronger than what is supported.

major comments (5)

[Section III.C, Eq. (16)] The PLO algorithm solves a finite-horizon MPC problem with N=20 and R=1e-4, whose objective is a sum of squared predicted violations plus a multiplier regularization term, subject to gradient-descent dynamics. This is not the exact MFOCP of Eq. (5), which requires theta(lambda) to be a global minimizer of -J(theta)+lambda*Jc(theta) for the current lambda. No error bound is provided for the finite horizon, the nonzero regularization R, or the receding-horizon execution of only the first multiplier. Therefore, Theorem 1 does not justify the PLO multiplier update, even under the paper's own assumptions.
[Section IV.B] The experiments use three-layer tanh MLP policies; for such parameterizations, J(theta) and Jc(theta) are generally nonconvex and are certainly not strongly convex in theta. Since Proposition 1 and Theorem 1 both require strong convexity, the theoretical equivalence does not cover the deployment setting used in the experiments. The abstract and introduction state the equivalence without this qualification, which overstates the scope of the result.
[Theorem 1 proof, Section III.B] The step from the dual problem (4) to the primal problem (2) requires strong duality. For the convex problem under consideration, a constraint qualification such as Slater's condition, namely existence of theta with Jc(theta)<0, is needed; the paper does not state or verify it. Without such a qualification, the dual optimum may not coincide with the primal optimum, and the claimed equivalence between (2) and (5) is not fully established.
[Lemma 1, Section III.B] The lemma asserts that arg max f = arg min |f'| for any C^1 function with strictly decreasing derivative on R+. This statement is not true without an attainment assumption: if f'(x)>0 for all x, both argmax and argmin may be empty unless an infinity boundary convention is introduced, and the proof's case analysis assumes that x* and x' exist. The lemma needs an explicit existence/attainment condition, such as coercivity of -f or the requirement that f' crosses zero at a finite point.
[Section IV.C] The experimental section reports only single learning curves, with no random seeds, error bars, or significance tests. Given that the theoretical support for PLO is currently missing, the empirical evidence alone is insufficient to support the claim that PLO achieves a larger feasible region than PID Lagrangian.

minor comments (3)

[Section II.A, Eq. (1)] The constraint is written as h(s_i)<=0, but the state is denoted x_i in the dynamics; please unify the notation for states and cost signals.
[Fig. 3 caption] Sub-figure (b) states that lighter colors indicate higher reward, but the color scale is not defined and the two rows appear to use different scales, which makes the 'comparable reward' claim difficult to read from the figure.
[Table I] The hyperparameter table lists PID gains, the prediction horizon N, and the regularization weight R, but omits the policy update step size eta and other FHADP parameters needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 1 is a genuine Lagrangian-duality derivation; PLO's MPC approximation is an applicability gap, not a self-referential reduction.

full rationale

The paper's central equivalence claim is that the primal constrained problem (2), the dual problem (4), and the multiplier feedback optimal control problem (5) share the same solution under differentiability and strong convexity assumptions. This is derived, not assumed: Proposition 1 uses the implicit function theorem to show monotonicity of Jc(θ(λ)), and Theorem 1 applies Lemma 1 to the dual objective Γ(λ), whose derivative equals Jc(θ(λ)). The target statement—that the primal optimizer corresponds to the minimizer of |Jc(θ(λ))|—is not baked into the definition of MFOCP; it follows from the stated convexity assumptions. The framework is a repackaging of classical Lagrangian duality into control-theoretic language, which may limit novelty but is not circular. The PLO algorithm (Eq. 16) solves a finite-horizon, R-regularized MPC rather than the exact MFOCP, so Theorem 1 does not by itself certify PLO's multiplier update; this is an unsupported-approximation gap, not a circular derivation. Self-citations [14], [15] provide the implementation backbone and are not load-bearing in the proof. No circular step is exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The free parameters are algorithm hyperparameters (N = 20, R = 1e-4, the unreported step size η, and the baseline PID gains); none is fitted to force the claimed result, but R > 0 means the deployed PLO objective is only a regularized, finite-horizon approximation of the exact MFOCP in the theorem. The central axiom is strong convexity and differentiability of -J and Jc in θ, which guarantees uniqueness of θ(λ), a positive-definite Hessian, and monotonicity of Jc(θ(λ)); it is stated explicitly in Proposition 1 and Theorem 1 but does not hold for the MLP policies in the experiments. Additional implicit axioms are strong duality between (2) and (4), existence of the arg maxima used in Lemma 1, and the accuracy of the deterministic MPC rollout model in Eq. (16). The invented entities are the MFOCP formulation and the PLO controller; both are validated only inside the paper, with no falsifiable external handle.

free parameters (4)

MPC prediction horizon N = 20
Algorithm hyperparameter (Table I); trades computation against how well the finite-horizon cost approximates the infinite-horizon MFOCP objective of Eq. (5).
Multiplier regularization weight R = 1e-4
Algorithm hyperparameter (Table I); with R > 0 the MPC cost in Eq. (16) is a regularized version of min |Jc(θ(λ))|, so the deployed objective differs from the proven MFOCP.
Policy update step size eta = not tabulated
Step size used in Eq. (6) and in the MPC rollout dynamics of Eq. (16); its value is not reported in Table I.
PID gains KP, KI, KD (baseline) = 1e-2, 1e-4, 1e-4
Hyperparameters of the PID Lagrangian baseline (Table I); the fairness of the comparison depends on these being competitively tuned.

assumptions (5)

domain assumption -J(θ) and Jc(θ) are differentiable and strongly convex in θ
Central premise of Proposition 1 and Theorem 1 (Section III.B); yields uniqueness of θ(λ), a positive-definite Hessian in Eq. (10), and monotonicity of Jc(θ(λ)). It is not satisfied for the MLP policies in the experiments.
domain assumption Strong duality holds between the primal problem (2) and the dual (4)
Invoked at the end of the proof of Theorem 1 (Section III.B) when equating the dual solution with the primal solution; standard under convexity/Slater-type conditions, but not stated.
standard math The dual function Γ(λ) attains a maximum over λ ≥ 0
Needed for Lemma 1 and for Theorem 1 to speak of arg max over λ ≥ 0; existence is assumed without proof.
domain assumption The rollout model θ_{i+1} = θ_i + η(∇θJ - λ ∇θJc) in Eq. (16) describes the true policy-gradient learning dynamics
Underlies the PLO prediction (Section III.C); in practice ∇θJ and ∇θJc are estimated by FHADP value networks, and the deterministic rollout ignores gradient noise.
domain assumption J(θ) and Jc(θ) and their gradients can be estimated from sampled trajectories
Standard RL assumption used by Algorithm 1 and the experiments, not discussed in the paper.

invented entities (2)

MFOCP (multiplier feedback optimal control problem)
purpose: Reformulates the multiplier update at each iteration as an optimal control problem: multiplier as input, policy parameters as state, constraint violation as objective (Eq. 5).
A new mathematical formulation introduced in Section III.A. Its validity is internal to the paper (Theorem 1) and rests on strong convexity that fails in the experimental regime; it makes no falsifiable prediction outside the paper.
PLO predictive Lagrangian controller
purpose: New multiplier update rule that uses receding-horizon MPC over predicted constraint violations instead of the current violation only (Eq. 16, Algorithm 1).
The proposed algorithm of Section III.C. Its only evidence is two low-dimensional experiments with no error bars, and no release of code or data, so no independent replication handle exists yet.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predictive Lagrangian Optimization for Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/GLRR6RTO

@misc{pith2026250115217,
  author       = {Pith},
  title        = {Pith review of: Predictive Lagrangian Optimization for Constrained Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLRR6RTO}},
  note         = {Machine review of arXiv:2501.15217}
}

read the original abstract

Constrained optimization is popularly seen in reinforcement learning for addressing complex control tasks. From the perspective of dynamic system, iteratively solving a constrained optimization problem can be framed as the temporal evolution of a feedback control system. Classical constrained optimization methods, such as penalty and Lagrangian approaches, inherently use proportional and integral feedback controllers. In this paper, we propose a more generic equivalence framework to build the connection between constrained optimization and feedback control system, for the purpose of developing more effective constrained RL algorithms. Firstly, we define that each step of the system evolution determines the Lagrange multiplier by solving a multiplier feedback optimal control problem (MFOCP). In this problem, the control input is multiplier, the state is policy parameters, the dynamics is described by policy gradient descent, and the objective is to minimize constraint violations. Then, we introduce a multiplier guided policy learning (MGPL) module to perform policy parameters updating. And we prove that the resulting optimal policy, achieved through alternating MFOCP and MGPL, aligns with the solution of the primal constrained RL problem, thereby establishing our equivalence framework. Furthermore, we point out that the existing PID Lagrangian is merely one special case within our framework that utilizes a PID controller. We also accommodate the integration of other various feedback controllers, thereby facilitating the development of new algorithms. As a representative, we employ model predictive control (MPC) as the feedback controller and consequently propose a new algorithm called predictive Lagrangian optimization (PLO). Numerical experiments demonstrate its superiority over the PID Lagrangian method, achieving a larger feasible region up to 7.2% and a comparable average reward.

Figures

Figures reproduced from arXiv: 2501.15217 by the authors.

**Figure 2.** Feasible region comparison. During the training process, we periodically save a policy per 200 iterations, identify and quantify its feasible region [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Feasible region and mean reward visualization during training process in the double integrator experiment. Sub-figure (a) illustrates the feasible [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[1]

Human-level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

2015
[2]

Mastering the game of go with deep neural networks and tree search,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol....

work page 2016
[3]

Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,

J. Duan, Y . Guan, S. E. Li, Y . Ren, Q. Sun, and B. Cheng, “Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6584–6598, 2021

work page 2021
[4]

A transformation-aggregation framework for state representation of au- tonomous driving systems,

G. Zhan, Y . Jiang, S. E. Li, Y . Lyu, X. Zhang, and Y . Yin, “A transformation-aggregation framework for state representation of au- tonomous driving systems,” IEEE Transactions on Intelligent Trans- portation Systems, 2024

work page 2024
[5]

S. E. Li, Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023

work page 2023
[6]

Direct and indirect reinforcement learning,

Y . Guan, S. E. Li, J. Duan, J. Li, Y . Ren, Q. Sun, and B. Cheng, “Direct and indirect reinforcement learning,” International Journal of Intelligent Systems, vol. 36, no. 8, pp. 4439–4467, 2021

work page 2021
[7]

A dynamic penalty function approach for constraint-handling in reinforcement learning,

H. Yoo, V . M. Zavala, and J. H. Lee, “A dynamic penalty function approach for constraint-handling in reinforcement learning,” IFAC- PapersOnLine, vol. 54, no. 3, pp. 487–491, 2021

work page 2021
[8]

Self- learned intelligence for integrated decision and control of automated vehicles at signalized intersections,

Y . Ren, J. Jiang, G. Zhan, S. E. Li, C. Chen, K. Li, and J. Duan, “Self- learned intelligence for integrated decision and control of automated vehicles at signalized intersections,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 24 145–24 156, 2022

work page 2022

Show all 15 references

[9]

Learning safe policies via primal-dual methods,

S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro, “Learning safe policies via primal-dual methods,” in Conference on Decision and Control . IEEE, 2019, pp. 6491–6497. (a) Double integrator (b) Cartpole Fig. 2. Feasible region comparison. During the training process...

2019
[10]

Dynamical, symplectic and stochastic perspectives on gradient-based optimization,

M. I. Jordan, “Dynamical, symplectic and stochastic perspectives on gradient-based optimization,” in Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018 . World Scientific, 2018, pp. 523–549

2018
[11]

Responsive safety in re- inforcement learning by PID lagrangian methods,

A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in re- inforcement learning by PID lagrangian methods,” in International Conference on Machine Learning, vol. 119. PMLR, 2020, pp. 9133– 9143

2020
[12]

Separated proportional-integral lagrangian for chance constrained reinforcement learning,

B. Peng, Y . Mu, J. Duan, Y . Guan, S. E. Li, and J. Chen, “Separated proportional-integral lagrangian for chance constrained reinforcement learning,” in Intelligent Vehicles Symposium. IEEE, 2021, pp. 193– 199

2021
[13]

Model- based actor-critic with chance constraint for stochastic system,

B. Peng, Y . Mu, Y . Guan, S. E. Li, Y . Yin, and J. Chen, “Model- based actor-critic with chance constraint for stochastic system,” in Conference on Decision and Control . IEEE, 2021, pp. 4694–4700

2021
[14]

Gops: A general optimal control problem solver for autonomous driving and industrial control applications,

W. Wang, Y . Zhang, J. Gao, Y . Jiang, Y . Yang, Z. Zheng, W. Zou, J. Li, C. Zhang, W. Cao, G. Xie, J. Duan, and S. E. Li, “Gops: A general optimal control problem solver for autonomous driving and industrial control applications,” Communications in Transportation Research , v...

2023
[15]

Enhance generality by model-based reinforcement learning and domain ran- domization,

G. Zhan, Y . Lyu, S. E. Li, Y . Jiang, X. Zhang, and L. Tao, “Enhance generality by model-based reinforcement learning and domain ran- domization,” in CAA International Conference on Vehicular Control and Intelligence. IEEE, 2023, pp. 1–6

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Human-level control through deep reinforcement learning,

V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

2015

[2] [2]

Mastering the game of go with deep neural networks and tree search,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vol....

work page 2016

[3] [3]

Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,

J. Duan, Y . Guan, S. E. Li, Y . Ren, Q. Sun, and B. Cheng, “Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6584–6598, 2021

work page 2021

[4] [4]

A transformation-aggregation framework for state representation of au- tonomous driving systems,

G. Zhan, Y . Jiang, S. E. Li, Y . Lyu, X. Zhang, and Y . Yin, “A transformation-aggregation framework for state representation of au- tonomous driving systems,” IEEE Transactions on Intelligent Trans- portation Systems, 2024

work page 2024

[5] [5]

S. E. Li, Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023

work page 2023

[6] [6]

Direct and indirect reinforcement learning,

Y . Guan, S. E. Li, J. Duan, J. Li, Y . Ren, Q. Sun, and B. Cheng, “Direct and indirect reinforcement learning,” International Journal of Intelligent Systems, vol. 36, no. 8, pp. 4439–4467, 2021

work page 2021

[7] [7]

A dynamic penalty function approach for constraint-handling in reinforcement learning,

H. Yoo, V . M. Zavala, and J. H. Lee, “A dynamic penalty function approach for constraint-handling in reinforcement learning,” IFAC- PapersOnLine, vol. 54, no. 3, pp. 487–491, 2021

work page 2021

[8] [8]

Self- learned intelligence for integrated decision and control of automated vehicles at signalized intersections,

Y . Ren, J. Jiang, G. Zhan, S. E. Li, C. Chen, K. Li, and J. Duan, “Self- learned intelligence for integrated decision and control of automated vehicles at signalized intersections,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 24 145–24 156, 2022

work page 2022