REVIEW 4 major objections 6 minor 9 references
Further extensions on the successive approximation method for hierarchical optimal control problems and its application to learning
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Two modifications to a nested optimal-control algorithm aim to make leader–follower learning converge faster and run in parallel.
desk verdict The second extension's key equivalence is false, so the paper's main promise fails; the first extension is unproven and the paper as a whole lacks the analysis to support its claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two objects carry the argument. First, the augmented Hamiltonian $\tilde H_i = H_i + \frac{\gamma_i}{2}\left\|\frac{\partial H_i}{\partial p_i}-\frac{\partial H_i}{\partial \bar p_i}\right\|^2 + \frac{\gamma_i}{2}\left\|\frac{\partial H_i}{\partial \theta_{u_i}}-\frac{\partial H_i}{\partial \theta_{\bar u_i}}\right\|^2$ turns each admissible-control update into a sub-problem $\arg\min_{u_i} \tilde H_i$; the added terms vanish at the nominal trajectory, so stationary points coincide with the original Pontryagin conditions while the iteration is better conditioned. Second, the intermediate state $m_i(t_k)=\frac{T-t_k}{T}\theta_{u_i}(t_k)+\frac{t_k}{T}p_i(t_k)$ is a convex blending of the forward state and the backward adjoint at grid points; it decouples the time interval so each subinterval $[t_k,t_{k+1}]$ can be solved independently, and the identities $\bar J_2[u_2|m]=J_2[u_2]$ and $\nabla J_2|_{[t_k,t_{k+1}]}=\frac{T}{\delta}\nabla J_2^k$ certify that the decomposed problems represent the original one.
What would settle it
Run Algorithm 1 and the original Algorithm O on a linear-quadratic leader–follower problem with a known optimal $\theta^*$ and unique $F$; if the augmented updates with $\gamma_i>0$ do not converge faster to a point with $\partial H_1/\partial u_1=0$, the better-convergence claim is refuted. Separately, on a two-subinterval example compute $\bar J_2[u_2|m]$ and $\nabla J_2$; Algorithm 2's time-parallelization is only valid if the identities $\bar J_2=J_2$ and $\nabla J_2|_{[t_k,t_{k+1}]}=(T/\delta)\nabla J_2^k$ hold exactly at the concatenated control.
Extended reading notes
Core claim
The paper claims that the nested algorithm for a hierarchical leader–follower optimal control problem can be extended in two ways that preserve the overall objective—an optimal parameter estimate $\theta^* = \theta_{u^{(n)}}(T)$—while improving convergence and enabling parallel computation. The first extension replaces each agent's plain Hamiltonian $H_i$ by an augmented Hamiltonian $\tilde H_i$ that adds squared deviations of the state and adjoint dynamics from their nominal trajectories, penalized by $\gamma_i \in [0,1)$; the control update at each iteration becomes the minimizer of $\tilde H_i$, and when both $\gamma_i=0$ the method reduces to the original. The second extension introduces an intermediate state $m_i(t_k) = \frac{T-t_k}{T}\theta_{u_i}(t_k) + \frac{t_k}{T}p_i(t_k)$ on a time grid and splits each agent's update into independent subinterval problems whose costs satisfy $\bar J_2 = J_2$ and whose gradients scale as $\nabla J_2|_{[t_k,t_{k+1}]} = (T/\delta)\nabla J_2^k$, which is what makes the updates fully time-parallelizable. The paper presents these as computational frameworks; it states that numerical work with them has been done and will be reported elsewhere.
Load-bearing premise
The construction assumes the follower's optimal response is a unique, smooth functional map $u_2(t)=F[u_1(t)]$ for every admissible leader strategy, as stated without proof in Remark 1; if a single leader strategy admits multiple equally optimal follower responses, the leader's Hamiltonian and both modified algorithms are undefined.
Editorial extensions
If this is right
- If Algorithm 1 converges as claimed, it returns the same parameter estimate $\theta^*$ as the original nested algorithm, since the augmented Hamiltonians reduce to the original Hamiltonians when $\gamma_1=\gamma_2=0$ and the added penalties vanish at fixed points.
- Algorithm 2 makes the follower and leader control updates independently solvable on every time subinterval $[t_k,t_{k+1}]$, so the computations can be spread across multiple cores with only the concatenation step synchronized.
- Convergence of Algorithm 2 is monitored by the average subinterval gradient $(1/T)\sum_{k=0}^{N-1}\int_{t_k}^{t_{k+1}}\|\nabla J_1^k\|\,dt$, which the paper expects to approach zero as $(u_1^{(n)},u_2^{(n)})\to(u_1^*,F[u_1^*])$.
- In the learning application, the extensions preserve the existing pipeline: training on dataset $Z^{(1)}$, validation via the target set $\Gamma$ from $Z^{(2)}$, and output of the final parameter estimate; only the inner update mechanics change.
- The paper's benefit claim about better convergence and efficiency is asserted rather than demonstrated numerically here; numerical results are promised elsewhere.
Reading between the lines
- One testable extension beyond the paper: treat the $\gamma_i$ penalties as proximal regularization around the previous control and measure the contraction rate on a linear-quadratic Stackelberg problem with a known solution; this would separate the claimed convergence gain from any shift in the fixed point.
- The intermediate-state construction is a form of multiple shooting in time; combining it with coarse-grid correction would let the subinterval solves themselves be accelerated, an option the paper does not discuss.
- If the follower's response map $F$ is not unique, the same update rules could be reinterpreted with subgradient selections after replacing the leader's state equation by a differential inclusion; the paper stops short of this extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two extensions to the nested successive-approximation algorithm for a hierarchical (Stackelberg) optimal control problem introduced in the author's prior work arXiv:2410.20617, with applications to learning/parameter estimation. Extension 1 (Section 3.1) augments the Hamiltonians of the follower and leader with quadratic penalty terms and redefines each agent's update as an auxiliary minimization problem, claiming better convergence. Extension 2 (Section 3.2) introduces intermediate state variables m^{u2}_2 and m^{u1}_1, splits each agent's problem into subproblems on uniform time subintervals, and claims full time-parallelization while preserving the original problem's solution. The paper restates the optimality conditions from [1], defines the augmented Hamiltonians and intermediate states, and lists Algorithms 1 and 2. It contains no theorems, proofs, or numerical experiments; the central claims of equivalence and convergence are asserted in the text.
Significance. If the claims were correct, the paper would offer meaningful computational improvements to the nested algorithm, namely faster convergence (via augmented Hamiltonians) and time-parallel execution (via intermediate states). The writing is clear and the notation is mostly consistent. However, the central mathematical claims are either unsupported or demonstrably false: the equivalence identity in Eq. (3.8) does not hold, and no convergence analysis is provided for either algorithm. The paper also contains no numerical evidence, no machine-checked proofs, and no falsifiable predictions beyond the assertions. Because the main advertised results (better convergence, preserved optimal parameter estimate, full time-parallelization) rest on unproved or incorrect equivalences, the contribution does not meet the bar for publication in its current form.
major comments (4)
- [Section 3.2, Eq. (3.8)] The claimed identity \bar J2[u2|m^{u2}_2] = J2[u2] is false. Each subproblem J^k_2 contains the terminal matching term (1/2)||m^{u2}_2(t_{k+1}) - θ^{u2}(t_{k+1})||^2, and \bar J2 is defined as (T/δ) times the sum of these subproblems. By Eq. (3.7), m^{u2}_2(t_{k+1}) = ((T-t_{k+1})/T)θ(t_{k+1}) + (t_{k+1}/T)p2(t_{k+1}). For k < N-1 this differs from θ(t_{k+1}) whenever p2(t_{k+1}) ≠ θ(t_{k+1}), which is the generic case; for k = N-1, m^{u2}_2(T) = p2(T) = 0, so the last matching term is (1/2)||θ(T)||^2, generically nonzero. Hence \bar J2 = J2 + (T/δ) Σ_{k=0}^{N-1} (1/2)||m^{u2}_2(t_{k+1}) - θ(t_{k+1})||^2, a functional different from J2. The gradient identity following Eq. (3.8) is therefore unjustified, and Algorithm 2's convergence to the same θ* as the original nested algorithm is unsupported. The same defect applies to the leader subproblem in Eq. (3.10) and J^k_1, where p1(T) is generically nonzero.
- [Section 3.1, Eqs. (3.1)-(3.6)] The claim of 'better convergence property' is asserted without any supporting analysis. The augmented Hamiltonian update is defined only as an argmin of \tilde H_i at each time t, and the text says the improvement δu_i 'will involve solving the first and second order variations', but no explicit update formula, descent property, or convergence theorem is given. Under the stated definitions, the method could be non-implementable as written because the subproblem (3.2) or (3.5) is not solved explicitly, and no conditions on γ1, γ2 ensure a decrease of the original objectives J1 and J2. In the absence of a proof or numerical demonstration, the advertised convergence improvement over Algorithm O is not established.
- [Section 2, Remark 1 and Eq. (2.13)] The paper assumes, without proof or conditions, that the follower's optimal response is a unique functional map u2(t) = F[u1(t)χ_{U1}]. This assumption is load-bearing: the leader's dynamics in Eq. (2.16) and Hamiltonian in Eq. (2.20) require F to be a single, differentiable response for every admissible leader strategy. If the follower's optimal control problem has multiple minimizers, or if F is not regular, then the leader's problem is not well-defined and both algorithms are undefined in the stated generality. The manuscript gives no sufficient conditions (convexity, strict convexity of J2 in u2, uniqueness of the minimizer of H2, etc.) for the existence or regularity of F, so this gap affects the validity of the entire framework.
- [Section 3.2, Eq. (3.9) and Algorithm 2] The assertion 'm^{u2}_2(t) ∈ arg min_{m2(t)} \bar J2[u2|m^{u2}_2]' is unclear and appears to be circular: the intermediate state m^{u2}_2 is defined by Eq. (3.7), not obtained by minimizing \bar J2. If the intent is to claim a variational characterization of m^{u2}_2, that claim is not proved. Moreover, the convergence criterion in Algorithm 2, Step 10, checks only the leader gradients (1/T)Σ∫||∇J^k_1|| dt, with no analogous check for the follower's subproblems; thus the algorithm's stopping rule does not ensure that the follower's optimality conditions are satisfied. These issues further undermine the claim that Algorithm 2 solves the original hierarchical problem.
minor comments (6)
- [Abstract and Section 3.1] The phrase 'as as sub-problems' contains a duplicated 'as'; please correct it.
- [Algorithm 1, Step 7 and Algorithm O, Step 10] The output line says u(n)(t) = (u(n)_2(t), u(n)_2(t)); this should presumably be (u(n)_1(t), u(n)_2(t)).
- [Section 3.2.2, step i] The text says 'solve the forward and backward-equations w.r.t. the system dynamics of the "follower"' in the leader's intermediate-state procedure; it should say 'leader' (compare with Algorithm 2, Step 6).
- [Eq. (3.9)] The notation 'm^{u2}_2(t) ∈ arg min_{m2(t)} \bar J2[u2|m^{u2}_2]' is confusing because m^{u2}_2 is already defined by Eq. (3.7); please clarify what the minimization is over and how it relates to the definition.
- [Section 3.2] The rescaling constants \bar α = δ/T α and \bar β = δ/T β are introduced but not motivated; a sentence explaining why these specific factors are needed would improve readability.
- [References] The paper cites [8] for multiple shooting, but does not connect the proposed time-parallel method to the extensive literature on parareal or other time-parallel algorithms; adding a brief comparison would help situate the contribution.
Circularity Check
The paper's modifications inherit their optimality conditions and the uniqueness of the follower's response from the same author's prior preprint, and the time-parallelized objective's equivalence to the original cost is asserted rather than derived; the claimed optimal-parameter output is therefore not supported by an independent derivation.
-
self citation load bearing
[Section 2, optimality statements for follower and leader (Eqs. (2.9)-(2.11) and (2.17)-(2.19)); see also Algorithm-O in Appendix.]
"In what follows, we provide the optimality conditions for both agents, i.e., the “follower” and the “leader,” where such conditions are the direct consequence of the Pontryagin’s maximum principle (see [1] for detailed proofs)."
Both Section 3 algorithms are stated as updates within the nested algorithm of reference [1]; they take over the Euler-Lagrange equations, the adjoint terminal conditions, and the convergence target θ*=θ(T) from that paper without proof. Reference [1] is the same author's earlier arXiv preprint, not an independent or machine-checked source, so the paper's claim of 'better convergence ... ultimately leading to an optimal parameter estimate' inherits its theoretical foundation from a self-citation chain. Pontryagin's principle itself is external, but the specific application to this hierarchical learning formulation is not independently established here.
-
uniqueness imported from authors
[Section 2, Remark 1, Eq. (2.13).]
"Hence, such a correspondence problem between the “leader” and that of the “follower” admits a unique functional mapping F [ ], i.e., u2(t) = F [u1(t)χU1 ] ∈ U2 (i.e., F : U1 ↦→ U2 on [0, T ]) ."
The unique response map F is asserted in a remark and inherited from the authors' prior framework, not proven here. It is then used as the defining ingredient of the leader's dynamics (Eq. 2.16) and Hamiltonian (Eq. 2.20). If F is not unique or not differentiable, the leader's optimality conditions and both algorithms are not well-defined. Presenting this uniqueness as a necessity of the formulation makes it a load-bearing assumption imported from the same authors' work.
1 more flagged steps
-
other
[Section 3.2.1, after Eq. (3.8).]
"Moreover, the following conditions hold true ¯J2 [ u2|mu2 2 ] = J2 [ u2 ] and ∇J2 [ u2 ] | [tk,t k+1]= T δ ∇J n 2 [ u2 |[tk,t k+1]|mu2 2 ] ."
The equality is asserted, not derived. In Eq. (3.8), ¯J2=(T/δ)Σ J^k2, and each J^k2 contains the terminal matching term (1/2)||mu2_2(t_{k+1})−θ^{u2}(t_{k+1})||^2. The scalings ¯α=δ/T α and ¯β=δ/T β make the running part telescope to J2's running cost by construction, but the matching terms do not vanish: for k=N−1, mu2_2(T)=p_2(T)=0, so the term is (1/2)||θ(T)||^2. Thus ¯J2=J2 only if an extra condition is silently imposed. The gradient identity and the claim that Algorithm 2 minimizes the original J2 and returns the original θ* therefore depend on an unsupported identification of the constructed objective with the original one.
full rationale
The paper is not circular in the narrow sense of fitting data and then predicting the same data: it contains no numerical fits and no external benchmarks. Its circularity risk lies in reliance on the same author's prior paper [1] for all load-bearing optimality conditions, the nested algorithm, and the uniqueness of the follower's response map F; the present paper adds modified update rules but does not re-derive these foundations. The most serious issue is the time-parallelized extension: the identity ¯J2 = J2 (and the corresponding gradient identity) is asserted without proof, and the definitions themselves show that the terminal matching terms do not cancel, so the claimed preservation of the optimal solution is not established. This is arguably a correctness defect rather than a tautology, but it means the paper's headline claim of an 'optimal parameter estimate' is not supported by an independent derivation. The score reflects partial circularity through the self-citation chain and the effectively assumed equivalence in Algorithm 2, while acknowledging that the augmented-Hamiltonian modification itself has independent content.
Assumptions & free parameters
free parameters (4)
- gamma_1, gamma_2 =
gamma1, gamma2 in [0,1), unspecified
- alpha, beta =
alpha > 0, beta > 0, unspecified
- final time T =
fixed, unspecified
- time step delta =
delta = T/N, unspecified
assumptions (5)
- standard math Pontryagin's maximum principle gives the stated necessary optimality conditions for the follower and leader problems (Eqs. 2.9-2.11 and 2.17-2.19).
- domain assumption The set Theta~ = {-nabla J0(theta, Z(1)) + u | u in U} is convex and the controlled-gradient system has a global solution on [0,T] for every admissible control (Section 2(iv)).
- domain assumption The follower's optimal response defines a unique functional mapping u2 = F[u1] from leader strategies to follower strategies (Eq. 2.13).
- ad hoc to paper The augmented-Hamiltonian update decreases the original objectives for gamma in [0,1).
- ad hoc to paper The intermediate-state decomposed cost equals the original cost and the gradients match (Eqs. 3.8-3.9).
invented entities (2)
-
Augmented Hamiltonians Htilde_1, Htilde_2
-
Intermediate state variables mu_1, mu_2
Cite this review
Pith. "Pith review of Further extensions on the successive approximation method for hierarchical optimal control problems and its application to learning." pith.science (2026). https://pith.science/paper/5M5BUISR
@misc{pith2026241115889,
author = {Pith},
title = {Pith review of: Further extensions on the successive approximation method for hierarchical optimal control problems and its application to learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/5M5BUISR}},
note = {Machine review of arXiv:2411.15889}
}
read the original abstract
In this paper, further extensions of the result of the paper "A successive approximation method in functional spaces for hierarchical optimal control problems and its application to learning, arXiv:2410.20617 [math.OC], 2024" concerning a class of learning problem of point estimations for modeling of high-dimensional nonlinear functions are given. In particular, we present two viable extensions within the nested algorithm of the successive approximation method for the hierarchical optimal control problem, that provide better convergence property and computationally efficiency, which ultimately leading to an optimal parameter estimate. The first extension is mainly concerned with the convergence property of the steps involving how the two agents, i.e., the "leader" and the "follower," update their admissible control strategies, where we introduce augmented Hamiltonians for both agents and we further reformulate the admissible control updating steps as as sub-problems within the nested algorithm of the hierarchical optimal control problem that essentially provide better convergence property. Whereas the second extension is concerned with the computationally efficiency of the steps involving how the agents update their admissible control strategies, where we introduce intermediate state variable for each agent and we further embed the intermediate states within the optimal control problems of the "leader" and the "follower," respectively, that further lend the admissible control updating steps to be fully efficient time-parallelized within the nested algorithm of the hierarchical optimal control problem.
Reference graph
Works this paper leans on
-
[7]
S.K. Mitter . Successive approximation methods for the solution of optimal control problems. Automatica , 3(3-4), 135--149, 1966
work page 1966
-
[8]
W.H. Press, S.A. Teukolsky, W.T. Vetterling & B.P. Flannery . Numerical recipes: The art of scientific computing. 3rd Edition, Cambridge University Press, 2007
work page 2007
-
[1]
G.K. Befekadu . A successive approximation method in functional spaces for hierarchical optimal control problems and its application to learning. arXiv:2410.20617 [math.OC] , 2024. Available at https://doi.org/10.48550/arXiv.2410.20617
-
[2]
Liberzon
D. Liberzon . Calculus of variations and optimal control theory. Princeton University Press, Princeton, NJ, 2012
2012
-
[3]
H. von Stackelberg . Marktform und Gleichgewicht . Springer, Berlin, Germany, 1934
work page 1934
-
[4]
J.L. Lions . Some remarks on Stackelberg’s optimization. Math. Models Methods Appl. Sci. , 4, 477--487, 1994
work page 1994
-
[5]
G.K. Befekadu & E.L. Pasiliao . On the hierarchical optimal control of a chain of distributed systems. J. Dynamics and Games , 2(2), 187--199, 2015
work page 2015
-
[6]
Pontryagin, V
L.S. Pontryagin, V. Boltianski, R. Gamkrelidze, & E. Mitchtchenko . The mathematical theory of optimal processes . John Wiley & Sons, New York, 1962
1962
Show all 9 references
-
[9]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTION or pop #1 'skip if FUNCTION new.block.check...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.