{"id":"94dca766-9743-40bd-9db7-cbe4cbecb601","arxiv_id":"2411.15889","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A successor to arXiv:2410.20617 proposes augmented-Hamiltonian and intermediate-state updates for its nested optimal control loop, claiming faster convergence and time-parallelizability without derivation or numerical support.","lead":"This paper adds two tweaks to a hierarchical optimal control algorithm for learning: extra penalty terms in the Hamiltonians and time-sliced intermediate states. It claims faster convergence and parallel computation, but gives no proof, no data, and no numerical experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's equivalence identity (Eq. 3.8) is false: the terminal matching terms do not vanish, so the time-parallelized objective differs from the original J2 and the claimed convergence to theta* is unsupported.","rationale":"The reader's verdict of REJECT is well supported: neither convergence theorems nor numerical experiments are provided for either algorithm. My stress-test identifies a more specific, internal obstruction in Algorithm 2 than the reader's weakest assumption about uniqueness of F. The false identity in Eq. (3.8) is load-bearing because the entire time-parallelization argument claims equivalence to the original problem; if the decomposed objective is different, the output theta* is not the optimal parameter of the original hierarchical problem. The uniqueness of F remains a separate concern, but Algorithm 2 would fail even if F were unique and smooth. I therefore agree partially with the reader: the rejection stands, but the decisive technical flaw is the incorrect equivalence identity rather than the unproved functional-map regularity. The concrete one-dimensional test above settles the matter directly from the paper's own definitions, so no numerical code is needed to confirm the objection.","tokens_in":11621,"tokens_out":4482,"duration_ms":41912,"concrete_test":"Set T=1, delta=1 (so N=1), take a scalar system theta_dot = u with theta(0)=0, p_2(1)=0, alpha=beta=1, and any admissible control u. Directly compute both sides of Eq. (3.8) from the definitions: J2 = (1/2) integral_0^1 (||theta||^2 + ||u||^2) dt, while \\bar J2 = (1/2)||theta(1)||^2 + (1/2) integral_0^1 (||theta||^2 + ||u||^2) dt. If the discrepancy is nonzero, Eq. (3.8) is false and Algorithm 2 does not reproduce the original optimal control problem.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Algorithm 2 preserves the original nested algorithm's solution relies on the asserted identity in Eq. (3.8): \\bar J2[u2|m^{u2}_2] = J2[u2]. This identity does not follow from the definitions. Each subproblem J^k_2 contains a terminal matching term (1/2)||m^{u2}_2(t_{k+1}) - theta(t_{k+1})||^2, and \\bar J2 is defined as (T/delta) times the sum of these terms. By Eq. (3.7), m^{u2}_2(t_k) = ((T-t_k)/T) theta(t_k) + (t_k/T) p_2(t_k). For k < N-1 this does not equal theta(t_{k+1}) in general, and for k = N-1 we have m^{u2}_2(T) = p_2(T) = 0, so the last matching term is (1/2)||theta(T)||^2. Hence \\bar J2 = J2 + positive terms, except in trivial cases. The same defect affects the leader subproblem in Eq. (3.10), where p_1(T) is generically nonzero. Consequently Algorithm 2 minimizes a different functional than the original J2 (and J1), and the gradient identity stated after Eq. (3.8) is not justified. The abstract's promise of 'an optimal parameter estimate' and Section 3.2's claim of full time-parallelization while preserving the optimal solution therefore fail at a definitional level, independent of whether the follower's response map F[u1] is unique.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two extensions to the nested successive-approximation algorithm for a hierarchical (Stackelberg) optimal control problem introduced in the author's prior work arXiv:2410.20617, with applications to learning/parameter estimation. Extension 1 (Section 3.1) augments the Hamiltonians of the follower and leader with quadratic penalty terms and redefines each agent's update as an auxiliary minimization problem, claiming better convergence. Extension 2 (Section 3.2) introduces intermediate state variables m^{u2}_2 and m^{u1}_1, splits each agent's problem into subproblems on uniform time subintervals, and claims full time-parallelization while preserving the original problem's solution. The paper restates the optimality conditions from [1], defines the augmented Hamiltonians and intermediate states, and lists Algorithms 1 and 2. It contains no theorems, proofs, or numerical experiments; the central claims of equivalence and convergence are asserted in the text.","tokens_in":12029,"tokens_out":4299,"duration_ms":36942,"significance":"If the claims were correct, the paper would offer meaningful computational improvements to the nested algorithm, namely faster convergence (via augmented Hamiltonians) and time-parallel execution (via intermediate states). The writing is clear and the notation is mostly consistent. However, the central mathematical claims are either unsupported or demonstrably false: the equivalence identity in Eq. (3.8) does not hold, and no convergence analysis is provided for either algorithm. The paper also contains no numerical evidence, no machine-checked proofs, and no falsifiable predictions beyond the assertions. Because the main advertised results (better convergence, preserved optimal parameter estimate, full time-parallelization) rest on unproved or incorrect equivalences, the contribution does not meet the bar for publication in its current form.","major_comments":[{"comment":"The claimed identity \\bar J2[u2|m^{u2}_2] = J2[u2] is false. Each subproblem J^k_2 contains the terminal matching term (1/2)||m^{u2}_2(t_{k+1}) - θ^{u2}(t_{k+1})||^2, and \\bar J2 is defined as (T/δ) times the sum of these subproblems. By Eq. (3.7), m^{u2}_2(t_{k+1}) = ((T-t_{k+1})/T)θ(t_{k+1}) + (t_{k+1}/T)p2(t_{k+1}). For k < N-1 this differs from θ(t_{k+1}) whenever p2(t_{k+1}) ≠ θ(t_{k+1}), which is the generic case; for k = N-1, m^{u2}_2(T) = p2(T) = 0, so the last matching term is (1/2)||θ(T)||^2, generically nonzero. Hence \\bar J2 = J2 + (T/δ) Σ_{k=0}^{N-1} (1/2)||m^{u2}_2(t_{k+1}) - θ(t_{k+1})||^2, a functional different from J2. The gradient identity following Eq. (3.8) is therefore unjustified, and Algorithm 2's convergence to the same θ* as the original nested algorithm is unsupported. The same defect applies to the leader subproblem in Eq. (3.10) and J^k_1, where p1(T) is generically nonzero.","section":"Section 3.2, Eq. (3.8)"},{"comment":"The claim of 'better convergence property' is asserted without any supporting analysis. The augmented Hamiltonian update is defined only as an argmin of \\tilde H_i at each time t, and the text says the improvement δu_i 'will involve solving the first and second order variations', but no explicit update formula, descent property, or convergence theorem is given. Under the stated definitions, the method could be non-implementable as written because the subproblem (3.2) or (3.5) is not solved explicitly, and no conditions on γ1, γ2 ensure a decrease of the original objectives J1 and J2. In the absence of a proof or numerical demonstration, the advertised convergence improvement over Algorithm O is not established.","section":"Section 3.1, Eqs. (3.1)-(3.6)"},{"comment":"The paper assumes, without proof or conditions, that the follower's optimal response is a unique functional map u2(t) = F[u1(t)χ_{U1}]. This assumption is load-bearing: the leader's dynamics in Eq. (2.16) and Hamiltonian in Eq. (2.20) require F to be a single, differentiable response for every admissible leader strategy. If the follower's optimal control problem has multiple minimizers, or if F is not regular, then the leader's problem is not well-defined and both algorithms are undefined in the stated generality. The manuscript gives no sufficient conditions (convexity, strict convexity of J2 in u2, uniqueness of the minimizer of H2, etc.) for the existence or regularity of F, so this gap affects the validity of the entire framework.","section":"Section 2, Remark 1 and Eq. (2.13)"},{"comment":"The assertion 'm^{u2}_2(t) ∈ arg min_{m2(t)} \\bar J2[u2|m^{u2}_2]' is unclear and appears to be circular: the intermediate state m^{u2}_2 is defined by Eq. (3.7), not obtained by minimizing \\bar J2. If the intent is to claim a variational characterization of m^{u2}_2, that claim is not proved. Moreover, the convergence criterion in Algorithm 2, Step 10, checks only the leader gradients (1/T)Σ∫||∇J^k_1|| dt, with no analogous check for the follower's subproblems; thus the algorithm's stopping rule does not ensure that the follower's optimality conditions are satisfied. These issues further undermine the claim that Algorithm 2 solves the original hierarchical problem.","section":"Section 3.2, Eq. (3.9) and Algorithm 2"}],"minor_comments":[{"comment":"The phrase 'as as sub-problems' contains a duplicated 'as'; please correct it.","section":"Abstract and Section 3.1"},{"comment":"The output line says u(n)(t) = (u(n)_2(t), u(n)_2(t)); this should presumably be (u(n)_1(t), u(n)_2(t)).","section":"Algorithm 1, Step 7 and Algorithm O, Step 10"},{"comment":"The text says 'solve the forward and backward-equations w.r.t. the system dynamics of the \"follower\"' in the leader's intermediate-state procedure; it should say 'leader' (compare with Algorithm 2, Step 6).","section":"Section 3.2.2, step i"},{"comment":"The notation 'm^{u2}_2(t) ∈ arg min_{m2(t)} \\bar J2[u2|m^{u2}_2]' is confusing because m^{u2}_2 is already defined by Eq. (3.7); please clarify what the minimization is over and how it relates to the definition.","section":"Eq. (3.9)"},{"comment":"The rescaling constants \\bar α = δ/T α and \\bar β = δ/T β are introduced but not motivated; a sentence explaining why these specific factors are needed would improve readability.","section":"Section 3.2"},{"comment":"The paper cites [8] for multiple shooting, but does not connect the proposed time-parallel method to the extensive literature on parareal or other time-parallel algorithms; adding a brief comparison would help situate the contribution.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript's central equivalence claim in Eq. (3.8) is incorrect at a definitional level, and the convergence claims are entirely unproved. The paper is essentially a set of definitions and algorithm descriptions built on the author's own prior work [1], with no theorem, proof, or experiment. Even as an 'extension' paper, the absence of any validation of the central claims makes it unsuitable for publication. I would suggest the author verify the equivalence identities with concrete numerical examples before resubmitting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the second extension, the time-parallelized intermediate-state method, has a definitional error in its central equivalence, and the first extension has no supporting analysis. I do not think this is publishable in current form.\n\nTo be fair, the paper is a clean write-up of two algorithmic ideas: augmented Hamiltonians of the Mitter type for the leader/follower updates, and a multiple-shooting style decomposition with intermediate states. It cites Mitter [7] and Numerical Recipes [8] rather than presenting them as new, and it is transparent that it builds on the author's own prior work. The algorithms are explicit. That is honest.\n\nThe problem is that the central claims are assertions. There is no theorem establishing better convergence for Algorithm 1; the update rules in (3.3) and (3.6) are just definitions. More seriously, the equivalence in Eq. (3.8) does not hold. Each subproblem J^k_2 includes a terminal matching term (1/2)||m_2(t_{k+1}) - θ(t_{k+1})||^2, and \\bar J2 is T/δ times the sum of those terms. These terms are nonnegative and generally positive. For k=N-1, m_2(T)=p_2(T)=0, so the last term is (1/2)||θ(T)||^2. Thus \\bar J2 = J2 plus positive terms, not J2. The gradient identity after (3.8) inherits the same problem. Algorithm 2 therefore minimizes a different functional, and the claimed convergence to the same θ* is unsupported. The leader subproblem has the same issue, since p_1(T) is generically nonzero.\n\nThe paper also assumes without proof that the follower's optimal response is a unique functional map F[u1] (Remark 1, Eq. 2.13). That may be carried over from [1], but here it is a standing assumption.\n\nWho gets value? A reader interested in a concrete template for extending a successive approximation scheme with parallelization might look at it, but they should know the equivalence is wrong. The first extension could become viable with a genuine convergence analysis; the second could be repaired by adjusting the matching terms or using a different scaling. As it stands, the abstract's promises are not supported.\n\nI would not send this to peer review. The false identity would be caught by any serious referee within minutes. Recommend desk reject pending a rewritten version with proofs and corrected identities.","headline":"The second extension's key equivalence is false, so the paper's main promise fails; the first extension is unproven and the paper as a whole lacks the analysis to support its claims.","tokens_in":12499,"tokens_out":4026,"would_cite":false,"duration_ms":34926,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["49K15","49M37","91A65"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two modifications to a nested optimal-control algorithm aim to make leader–follower learning converge faster and run in parallel.","keywords":["hierarchical optimal control","Stackelberg optimization","successive approximation method","augmented Hamiltonian","intermediate state method","time-parallelized computation","Pontryagin maximum principle","learning as optimal control"],"falsifier":"Run Algorithm 1 and the original Algorithm O on a linear-quadratic leader–follower problem with a known optimal $\\theta^*$ and unique $F$; if the augmented updates with $\\gamma_i>0$ do not converge faster to a point with $\\partial H_1/\\partial u_1=0$, the better-convergence claim is refuted. Separately, on a two-subinterval example compute $\\bar J_2[u_2|m]$ and $\\nabla J_2$; Algorithm 2's time-parallelization is only valid if the identities $\\bar J_2=J_2$ and $\\nabla J_2|_{[t_k,t_{k+1}]}=(T/\\delta)\\nabla J_2^k$ hold exactly at the concatenated control.","tokens_in":3,"feed_emoji":"⚙️","tokens_out":10896,"duration_ms":145016,"temperature":0.7,"pith_summary":"This paper tries to establish that a nested leader–follower algorithm for optimal control—originally designed for learning the parameters of high-dimensional nonlinear functions—can be upgraded in two ways. The first upgrade adds augmented-Hamiltonian penalty terms to each agent's update, which the author argues gives better convergence. The second upgrade introduces intermediate state variables on a time grid, which splits each control update into independent subinterval problems that can be solved in parallel. If both work as claimed, the same learning procedure would produce the same final parameter estimate $\\theta^*$, but reach it faster and on multi-core hardware. The paper is a framework paper rather than a numerical study, and it explicitly defers numerical results.","feed_headline":"Hierarchical optimal control gains faster, parallel updates","feed_subtitle":"Augmented Hamiltonians and intermediate states split leader–follower control updates into independent subintervals","key_machinery":"Two objects carry the argument. First, the augmented Hamiltonian $\\tilde H_i = H_i + \\frac{\\gamma_i}{2}\\left\\|\\frac{\\partial H_i}{\\partial p_i}-\\frac{\\partial H_i}{\\partial \\bar p_i}\\right\\|^2 + \\frac{\\gamma_i}{2}\\left\\|\\frac{\\partial H_i}{\\partial \\theta_{u_i}}-\\frac{\\partial H_i}{\\partial \\theta_{\\bar u_i}}\\right\\|^2$ turns each admissible-control update into a sub-problem $\\arg\\min_{u_i} \\tilde H_i$; the added terms vanish at the nominal trajectory, so stationary points coincide with the original Pontryagin conditions while the iteration is better conditioned. Second, the intermediate state $m_i(t_k)=\\frac{T-t_k}{T}\\theta_{u_i}(t_k)+\\frac{t_k}{T}p_i(t_k)$ is a convex blending of the forward state and the backward adjoint at grid points; it decouples the time interval so each subinterval $[t_k,t_{k+1}]$ can be solved independently, and the identities $\\bar J_2[u_2|m]=J_2[u_2]$ and $\\nabla J_2|_{[t_k,t_{k+1}]}=\\frac{T}{\\delta}\\nabla J_2^k$ certify that the decomposed problems represent the original one.","core_discovery":"The paper claims that the nested algorithm for a hierarchical leader–follower optimal control problem can be extended in two ways that preserve the overall objective—an optimal parameter estimate $\\theta^* = \\theta_{u^{(n)}}(T)$—while improving convergence and enabling parallel computation. The first extension replaces each agent's plain Hamiltonian $H_i$ by an augmented Hamiltonian $\\tilde H_i$ that adds squared deviations of the state and adjoint dynamics from their nominal trajectories, penalized by $\\gamma_i \\in [0,1)$; the control update at each iteration becomes the minimizer of $\\tilde H_i$, and when both $\\gamma_i=0$ the method reduces to the original. The second extension introduces an intermediate state $m_i(t_k) = \\frac{T-t_k}{T}\\theta_{u_i}(t_k) + \\frac{t_k}{T}p_i(t_k)$ on a time grid and splits each agent's update into independent subinterval problems whose costs satisfy $\\bar J_2 = J_2$ and whose gradients scale as $\\nabla J_2|_{[t_k,t_{k+1}]} = (T/\\delta)\\nabla J_2^k$, which is what makes the updates fully time-parallelizable. The paper presents these as computational frameworks; it states that numerical work with them has been done and will be reported elsewhere.","pith_inferences":["One testable extension beyond the paper: treat the $\\gamma_i$ penalties as proximal regularization around the previous control and measure the contraction rate on a linear-quadratic Stackelberg problem with a known solution; this would separate the claimed convergence gain from any shift in the fixed point.","The intermediate-state construction is a form of multiple shooting in time; combining it with coarse-grid correction would let the subinterval solves themselves be accelerated, an option the paper does not discuss.","If the follower's response map $F$ is not unique, the same update rules could be reinterpreted with subgradient selections after replacing the leader's state equation by a differential inclusion; the paper stops short of this extension."],"forward_implications":["If Algorithm 1 converges as claimed, it returns the same parameter estimate $\\theta^*$ as the original nested algorithm, since the augmented Hamiltonians reduce to the original Hamiltonians when $\\gamma_1=\\gamma_2=0$ and the added penalties vanish at fixed points.","Algorithm 2 makes the follower and leader control updates independently solvable on every time subinterval $[t_k,t_{k+1}]$, so the computations can be spread across multiple cores with only the concatenation step synchronized.","Convergence of Algorithm 2 is monitored by the average subinterval gradient $(1/T)\\sum_{k=0}^{N-1}\\int_{t_k}^{t_{k+1}}\\|\\nabla J_1^k\\|\\,dt$, which the paper expects to approach zero as $(u_1^{(n)},u_2^{(n)})\\to(u_1^*,F[u_1^*])$.","In the learning application, the extensions preserve the existing pipeline: training on dataset $Z^{(1)}$, validation via the target set $\\Gamma$ from $Z^{(2)}$, and output of the final parameter estimate; only the inner update mechanics change.","The paper's benefit claim about better convergence and efficiency is asserted rather than demonstrated numerically here; numerical results are promised elsewhere."],"supporting_citations":[{"why":"Supplies the original nested algorithm, problem formulation, and optimality conditions that this paper extends.","marker":"[1]"},{"why":"Provides the Filippov reachable-set theorem underlying the general assumptions and solvability of the controllability-type objective.","marker":"[2]"},{"why":"Defines the Stackelberg leader–follower equilibrium concept that structures the hierarchical problem.","marker":"[3]"},{"why":"Supplies the Pontryagin maximum principle used to derive the agents' forward, backward, and extremum equations.","marker":"[6]"},{"why":"Supplies the successive approximation method whose augmented-Hamiltonian reformulation is the basis of Algorithm 1.","marker":"[7]"},{"why":"Supplies the multiple-shooting numerical framework that motivates the intermediate-state time-parallelization of Algorithm 2.","marker":"[8]"}],"fun_headline_variants":["Leader-follower control learns faster with augmented Hamiltonians","Parallelizing hierarchical control: two extensions speed learning","Optimal control gets a parallel boost for learning problems","Augmented Hamiltonians and states cut control update time","Hierarchical control: faster convergence via parallel subintervals"],"cache_read_input_tokens":14592,"weakest_assumption_plain":"The construction assumes the follower's optimal response is a unique, smooth functional map $u_2(t)=F[u_1(t)]$ for every admissible leader strategy, as stated without proof in Remark 1; if a single leader strategy admits multiple equally optimal follower responses, the leader's Hamiltonian and both modified algorithms are undefined.","fun_headline_variants_meta":{"raw":{"variants":["Leader-follower control learns faster with augmented Hamiltonians","Parallelizing hierarchical control: two extensions speed learning","Optimal control gets a parallel boost for learning problems","Augmented Hamiltonians and states cut control update time","Hierarchical control: faster convergence via parallel subintervals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1356,"prompt_tokens":1060,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":220}},"tokens_in":676,"tokens_out":296,"duration_ms":3025,"temperature":1.0,"reasoning_tokens":220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:46:49.080641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 and the original Algorithm O on a linear-quadratic leader–follower problem with a known optimal $\\theta^*$ and unique $F$; if the augmented updates with $\\gamma_i>0$ do not converge faster to a point with $\\partial H_1/\\partial u_1=0$, the better-convergence claim is refuted. Separately, on a two-subinterval example compute $\\bar J_2[u_2|m]$ and $\\nabla J_2$; Algorithm 2's time-parallelization is only valid if the identities $\\bar J_2=J_2$ and $\\nabla J_2|_{[t_k,t_{k+1}]}=(T/\\delta)\\nabla J_2^k$ hold exactly at the concatenated control.","supporting_citations":[{"cited_title":"von Stackelberg","cited_arxiv_id":null,"evidence_quote":"Defines the Stackelberg leader–follower equilibrium concept that structures the hierarchical problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the successive approximation method whose augmented-Hamiltonian reformulation is the basis of Algorithm 1."},{"cited_title":"Press, S.A","cited_arxiv_id":null,"evidence_quote":"Supplies the multiple-shooting numerical framework that motivates the intermediate-state time-parallelization of Algorithm 2."}],"review_version":1}