{"id":"0f69d6ad-fb05-4e9d-a361-353a1259c109","arxiv_id":"2505.02688","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Batch sample-wise stochastic maximum principle with a higher-order forward scheme achieves O(sqrt(1/K + 1/N^2)) convergence for strongly convex stochastic optimal control problems.","lead":"This paper modifies a stochastic maximum principle method for stochastic optimal control by averaging gradients over batches of paths and using a higher-order scheme for the forward process, and proves an improved convergence rate. The claimed rate, O(sqrt(1/K + 1/N^2)) instead of O(sqrt(N/K + 1/N)), matters because it removes the growth of gradient noise with time-grid refinement in sample-wise control algorithms.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 4.7's unproved moment condition is the key premise behind the N^{-2} term; if it fails, Theorem 4.10's rate degrades to 1/N and the improvement over [1] disappears.","rationale":"The reader's weakest_assumption is exactly the moment condition in Lemma 4.7, and I agree it is the most load-bearing premise for the central rate claim. The N^{-2} term in Theorem 4.10 enters only through ε_N^2 ≤ C h^2, whose proof relies on Lemma 4.7; the lemma's additional moment condition is not derived from Assumption 1 and is not implied by L2 strong order alone. Other issues noted by the reader — the loose variance bound in (4.52), the unexplained wall-clock speedup in Table 1, and the contraction-analysis gaps — are either non-central to the rate theorem or readily fixable. The central claim is plausible and the theorem structure is standard, so I do not recommend rejecting the paper; the appropriate disposition is the same conditional acceptance the reader gave, pending either a proof or a verified numerical check of the moment condition.","tokens_in":26505,"tokens_out":18422,"duration_ms":223954,"concrete_test":"Estimate the two quantities used by Lemma 4.7 for the scheme (7.30) on the SDE of Example 5.1 (drift u(t)−r(t), diffusion σu(t), analytic solution available from the known control). Fix u = u* and simulate M = 10^5 paths at N = 40, 80, 160, 320. Compute R_N = sqrt(E|X^N_T−X_T|^4)/E|X^N_T−X_T|^2 and the empirical strong order Θ from a least-squares fit of log E|X^N_T−X_T|^2 versus log h. If R_N remains bounded and Θ ≥ 1.5, Lemma 4.7's hypothesis is satisfied and the N^{-2} rate stands; if R_N grows with N or Θ < 1.5, the proof's ε_N^2 bound is invalid and the advertised rate should be weakened to O(sqrt(1/K + 1/N)).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claimed rate E||u*−u^{K+1}||^2 ≤ C(1/K + 1/N^2) in Theorem 4.10 inherits its N^{-2} term from ε_N^2 ≤ C h^2 in Lemma 4.8, which is proved via Lemma 4.7. Lemma 4.7 assumes (i) the forward scheme has strong order Θ with 2Θ ≥ 3 and (ii) sqrt(E|X^N_{t_n}−X_{t_n}|^4) ≤ C E|X^N_{t_n}−X_{t_n}|^2. Assumption 1 guarantees neither. The usual L2 strong-order bound E|X^N−X|^2 ≤ C h^{2Θ} says nothing about the ratio of the fourth moment to the second moment; rare large excursions can make that ratio unbounded. In the proof of Lemma 4.7, the step after (4.30) uses the condition to replace sqrt(E|Y|^4 E|X^N−X|^4) + E|X^N−X|^2 by C h^{2Θ}. Without it, the second term is controlled but the first is not, so the bound E|Z^N−Z|^2 ≤ h^{2Θ−1} and hence the h^2 total in (4.27) can fail. If (4.27) drops to O(h), then ε_N^2 in (4.35) becomes O(h), and Theorem 4.10 yields O(sqrt(1/K + 1/N)) — the same discretization error as in [1]. This is not an internal inconsistency, but an unsupported hypothesis at the exact point where the claimed improvement over prior work is produced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies finite-horizon stochastic optimal control problems through the stochastic maximum principle, with deterministic controls restricted to piecewise-constant functions in time. It proposes two batch-sampling algorithms: a projected batch stochastic gradient descent update for the control, using a higher-order scheme for the forward SDE, and a damped contraction update for the control that minimizes the Hamiltonian. The main theoretical claim (Theorems 4.9--4.10) is that, under strong convexity of the cost functional, the projected batch SGD satisfies E||u* - u^{K+1}||^2 <= C(1/K + 1/N^2), improving the earlier rate O(sqrt(N/K + 1/N)) of Archibald--Bao--Cao--Sun [1]. A secondary result (Theorems 4.11--4.12) gives a contraction-type convergence bound for the damped algorithm under a linear-quadratic structure and an explicit contraction assumption. Numerical experiments for controlled and uncontrolled diffusions, a high-dimensional HJB equation, and a randomized neural network extension illustrate the behavior of the algorithms and their time efficiency relative to the original SGD method.","tokens_in":26746,"tokens_out":10805,"duration_ms":121996,"significance":"If the central rate bound were fully justified, the paper would provide a meaningful incremental advance over the sample-wise backpropagation analysis in [1]: it shows how batch sampling and a higher-order forward scheme turn the iteration complexity N/K into 1/K and the discretization complexity 1/N into 1/N^2 in the squared-error sense. The manuscript is transparent about the special assumptions needed for the damped contraction result, and it ships reproducible code and careful numerical comparisons against the prior SGD method. The high-dimensional HJB example with a randomized neural network is a useful practical demonstration, even though its optimizer study is heuristic rather than covered by the theorems. The main barrier to acceptance is the unsupported moment hypothesis in Lemma 4.7 on which the advertised N^{-2} rate rests.","major_comments":[{"comment":"The advertised N^{-2} discretization rate is obtained only under the fourth-moment ratio condition sqrt(E|X^N_{t_n}-X_{t_n}|^4) <= C E|X^N_{t_n}-X_{t_n}|^2, which is stated as an assumption inside Lemma 4.7 and is not part of Assumption 1 or of the hypotheses of Theorems 4.9/4.10. In the proof of Lemma 4.7, this condition is used at the last inequality of (4.30) to control the cross term sqrt(E|\\hat Y^N_{t_n}|^4 E|X^N_{t_n}-X_{t_n}|^4) by C h^{2\\Theta}. Without it, (4.27) does not follow from the standard L2 strong-order estimate E|X^N-X|^2 <= C h^{2\\Theta}, because the ratio (E|Y|^4)^{1/2}/E|Y|^2 can be unbounded when the discretization error has rare large excursions. Consequently, the step epsilon_N^2 <= C h^2 in Lemma 4.8, and hence the 1/N^2 term in Theorem 4.10, is unsupported as the theorems are stated. This is load-bearing: if the condition cannot be proved from Assumption 1, the main theorem degrades to O(1/K + 1/N), which removes the claimed improvement over [1].","section":"Section 4.1, Lemma 4.7 and Theorems 4.9-4.10"},{"comment":"The batch variance estimate is written as * <= C(1 + N/B). As stated, the right-hand side does not tend to zero as B grows, which is inconsistent with the fact that the batch estimator converges to the conditional expectation; the proof then chooses B = N to obtain * <= C. However, Theorem 4.9 states no assumption linking the batch size M in Algorithm 1 to N, and the proof introduces B without relating it to M. The claimed passage from the N/K term in [1] to 1/K relies on this variance estimate. The theorem statement should be amended to include the needed batch-size assumption, for example M = N, and the estimate in (4.52) should be repaired so that it does not assert a non-vanishing variance for the sample mean.","section":"Theorem 4.9, Eq. (4.52)"},{"comment":"The contraction recursion is not proved as written. In (4.76), the first line contains a term proportional to ||u^k - u^{N,*}|| on the right-hand side while the left-hand side is a squared norm, and the second line converts this into ||u^k - u^{N,*}||^2 without displaying the Young's inequality step that would justify the conversion. Such a step would introduce additional constants and would affect the condition on rho under which eta < 1 holds. The induction leading to (4.77) is therefore not a valid proof in the present form. This is fixable by rewriting the estimate, but the displayed argument needs to be corrected.","section":"Theorem 4.11, Eqs. (4.73)-(4.76)"}],"minor_comments":[{"comment":"In the display after (4.28), the first term on the right should be E[|Delta Y^N_{t_{n+1}}|^2] rather than E[|Y^N_{t_{n+1}}|^2]; otherwise the Gronwall argument does not control the quantity of interest.","section":"Section 4.1, Eq. (4.29)"},{"comment":"The batch size is called M in Algorithm 1 and in Theorem 4.9, but Eq. (4.52) uses B for the same quantity. Please use one symbol consistently and state the relation between M and N in the theorem hypotheses.","section":"Algorithm 1 and Section 4.1"},{"comment":"The caption of Table 2 says 'Example 1 Controlled diffusion', but the table reports results for Example 2, which has uncontrolled diffusion. The caption should be corrected.","section":"Table 2 caption"},{"comment":"The HJB numerical experiments compare SGD, Adam, and AdaGrad, but the convergence theorems in Section 4 cover only the SGD update. The text should state explicitly that the optimizer comparison is heuristic and outside the theoretical results, especially since the randomized neural network parametrization is not covered by the strong-convexity framework of Theorem 4.9.","section":"Section 5.3"},{"comment":"The text says 'we pick eta ~ O(1-rho) to be a large constant, in this case 0.995', but eta in Theorem 4.11 is defined as C(1-rho)^2 + rho^2. The authors should clarify how the numerical choice of eta corresponds to an admissible value of rho under that definition.","section":"Section 5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central new claim, the O(1/K + 1/N^2) rate, currently rests on the unproved fourth-moment ratio condition in Lemma 4.7 and on a variance bound in Theorem 4.9 that is not stated with the required batch-size assumption. If the authors can supply a proof of that condition from Assumption 1, or alternatively state it explicitly as a hypothesis of the main theorem, the paper would be a solid incremental contribution. If the condition cannot be justified, the contribution reduces to a batch-variance reduction with the same O(1/N) discretization error as the prior work [1], and the abstract and conclusion would need to be scaled back accordingly. The manuscript also relies heavily on the authors' own prior work [1], though it uses it as an independent published result rather than in a circular way."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper is a straightforward but useful extension of the authors' sample-wise stochastic maximum principle method: it replaces single-sample gradient estimates with batch averages and uses a strong-order-2 forward scheme, and that buys a formal rate O(sqrt(1/K+1/N^2)) instead of O(sqrt(N/K+1/N)) — but only when the batch size is of order N and only under an extra moment hypothesis that is not part of Assumption 1. The paper also proposes a damped contraction scheme, which is new but proved only in a linear-quadratic special case.\n\nWhat is genuinely good: The batch estimator is unbiased (Prop 4.6), the rate proof follows the [1] skeleton and the variance-reduction mechanism is visible. The numerics use analytic solutions, no fitted constants, and they show the expected first-order decay in the right K–N scaling. The high-dimensional HJB experiment with randomized neural networks is a nice practical add-on, and the authors are honest that the contraction analysis is not tight.\n\nThe soft spots, in order of seriousness:\n\n1. Lemma 4.7 is load-bearing and unproven. The N^{-2} term in Theorem 4.10 comes from the assumption sqrt(E|X^N-X|^4) ≤ C E|X^N-X|^2, which Assumption 1 does not provide. Without it, the Z-term error drops to O(h) and the advertised improvement over [1] disappears. The condition is plausible for standard schemes with L4 control, but it must be stated as an assumption or proved from the scheme. This is the main revision issue.\n\n2. The variance bound in (4.52) is written as C(1+N/B), which does not go to zero as B grows. The proof only needs a constant when B=N, so the rate can survive, but the bound as written is wrong and should be the actual sample-mean variance.\n\n3. The abstract claims O(sqrt(1/K+1/N^2)); the proof actually gives a term like sqrt(N/(MK)+1/N^2), which the authors themselves note in Section 5.2.1. The headline rate is for M proportional to N. That distinction needs to be in the main theorems.\n\n4. The wall-clock table reports a 30x speedup for a method that does 40x more work per iteration. It is probably vectorization, but as reported it is unexplained and I would not trust the efficiency claim without code.\n\nThe damped contraction section uses max where argmin is meant in the display (3.4)-(3.5); the algorithm uses the right object.\n\nBottom line: the central rate claim is plausible and may well be correct, but it is not fully established as written. The paper deserves a serious referee; I would send it out and ask for the moment condition, the variance bound, and the timing comparison to be fixed. It is mainly for people working on numerical SMP/SGD methods, not for a general control audience.","headline":"A plausible batch-extension of sample-wise SMP whose headline rate relies on an unstated moment condition, with a suspicious timing table.","tokens_in":27431,"tokens_out":6141,"would_cite":true,"duration_ms":78867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60H10","60H30","65C20"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that a batch sample-wise projection method based on the stochastic maximum principle attains mean-squared error at most $C(1/K+1/N^2)$ for stochastic optimal control, improving the earlier $O(\\sqrt{N/K+1/N})$ rate.","keywords":["stochastic optimal control","stochastic maximum principle","batch stochastic gradient descent","backward stochastic differential equations","convergence rate","strong convexity","damped contraction","randomized neural networks"],"falsifier":"Run a one-dimensional linear-quadratic control problem with known solution using Algorithm 1 with an Euler forward scheme but batched SGD; if the empirical error versus $N$ follows $1/N$ rather than $1/N^2$, the claimed $N^{-2}$ discretization term is not achievable without the higher-order scheme. Conversely, with the order-2 scheme of equation (7.30), $M=N$, and $K$ a constant multiple of $N^2$, the log-log error slope should be about $-1$; a plateau or a shallower slope would indicate that the variance bound or the fourth-moment assumption in Lemma 4.7 fails.","tokens_in":26094,"feed_emoji":"🎯","tokens_out":6601,"duration_ms":75731,"temperature":0.7,"pith_summary":"The paper tries to establish that the sample-wise backpropagation method for solving stochastic optimal control problems can be accelerated by replacing single-sample gradient estimates with batch averages and by simulating the forward SDE with a higher-order scheme. Under strong convexity of the cost functional, it proves that the resulting projection algorithm satisfies $E[\\|u^*-u^{K+1}\\|^2] \\le C(1/K+1/N^2)$ after $K$ gradient iterations on an $N$-point time grid, improving the predecessor's $O(\\sqrt{N/K+1/N})$ rate. The practical significance is that the bottleneck in such probabilistic control solvers is neither the optimization step nor the temporal resolution alone, but the variance and discretization error in estimating the adjoint processes $Y$ and $Z$ from the backward SDE; batching and a higher-order forward scheme attack exactly that bottleneck. The paper also proposes a damped contraction algorithm that uses the sufficient condition of the maximum principle, and proves convergence in a linear-quadratic special case at rate $O(\\eta^{k+1}+N/M+1/N)$.","feed_headline":"Optimal-control solver converges at O(1/K + 1/N^2)","feed_subtitle":"Batching forward-backward paths with a higher-order SDE scheme removes the old sqrt(N/K) bottleneck.","key_machinery":"The carrying mechanism is the sample-wise backward recursion $Y_n=Y_{n+1}+h(b_x'Y_{n+1}+f_x')$ with $Z_n=Y_{n+1}\\Delta W_n/h$, paired with a batched gradient estimator $(j_u')_n=\\frac1M\\sum_i \\partial_u H(t_n,X_n^i,Y_n^i,Z_n^i,u_n)$ and a higher-order forward integrator $\\Psi$ of order $\\Theta$. Proposition 4.6 identifies the sample-wise variables as unbiased conditional estimates of the classical BSDE discretization; Lemma 4.7 bounds the temporal error of $(Y,Z)$ by $O(h^2)$ under the condition $2\\Theta\\ge 3$ and the fourth-moment assumption on the forward scheme; Theorem 4.9 combines strong convexity with a variance estimate for the batched gradient to obtain the contraction recursion; Theorem 4.10 adds the $O(1/N^2)$ error from projecting the true optimum onto piecewise-constant controls.","core_discovery":"On the paper's own terms, the central discovery is that the convergence rate of the sample-wise stochastic gradient projection method for deterministic controls can be improved from roughly $O(\\sqrt{N/K+1/N})$ to roughly $O(\\sqrt{1/K+1/N^2})$ by two coordinated changes: average the sample-wise Hamiltonian gradient over a batch of independent forward-backward paths, and discretize the forward SDE with a scheme of order at least $3/2$. Theorem 4.10 states that, under Lipschitz and linear-growth assumptions together with strong convexity of the loss, the iterates of Algorithm 1 satisfy $E[\\|u^*-u^{K+1}\\|^2] \\le C(1/K+1/N^2)$, where the $K^{-1}$ term is the optimization error and the $N^{-2}$ term is the combined projection and BSDE discretization error. A supporting result, Proposition 4.6, shows that the sample-wise backward recursion produces unbiased conditional estimates of the classical numerical BSDE solution, so the batch average is an honest estimator of the numerical gradient rather than a heuristic smoothing device.","pith_inferences":["The advertised $1/N^2$ is conditional on using a forward scheme of order at least $3/2$; if one retains the Euler scheme, the BSDE discretization error reverts to $O(h)$, so the rate degrades to $O(\\sqrt{1/K+1/N})$. The paper states the requirement explicitly but leaves the practical cost trade-off implicit.","The variance analysis suggests an adaptive batch policy: choose $M$ proportional to $N$ when the diffusion is controlled, but smaller batch sizes may suffice when the diffusion is uncontrolled and the $Z$-term disappears from the gradient; the authors' numerics show faster-than-predicted decay in that case.","The randomized-neural-network application points to a derivative-free training loop: with random hidden weights fixed, the SMP gradient is linear in the output weights and can be computed by vector algebra, which would remove backpropagation from the parameter updates if extended to deeper networks."],"forward_implications":["For a fixed error tolerance $\\epsilon$, the number of gradient iterations needed drops from $O(\\epsilon^{-4})$ to $O(\\epsilon^{-2})$, and the temporal grid size drops from $O(\\epsilon^{-2})$ to $O(\\epsilon^{-1})$.","Batch size $M=N$ is sufficient to make the gradient estimator's variance independent of $N$, so the noise from the $Z$-term simulation no longer dominates the convergence rate.","The $1/N^2$ temporal term matches the optimal first-order rate for Euler-type approximations of the control, so further improvement requires a higher-order control representation, not just a smaller step size.","In the linear-quadratic special case, the damped contraction algorithm converges linearly in the iteration count, with a batch-penalty term $N/M$, giving a derivative-free alternative when the Hamiltonian minimizer is known explicitly."],"supporting_citations":[{"why":"Supplies the predecessor sample-wise backpropagation algorithm and the $O(\\sqrt{N/K+1/N})$ baseline rate that this paper improves.","marker":"[1]"},{"why":"Establishes the general stochastic maximum principle that forms the theoretical foundation for the gradient and Hamiltonian-minimization updates.","marker":"[25]"},{"why":"Documents the first-order temporal discretization error of BSDEs used to justify the $O(h^2)$ error bound in Lemma 4.7.","marker":"[8]"},{"why":"Provides the error-expansion result for BSDE discretization that underlies the $O(h^2)$ estimate when the forward process is exact.","marker":"[12]"},{"why":"Supplies the explicit order-2 forward SDE scheme used in the numerical experiments for the projection algorithm.","marker":"[10]"},{"why":"Introduces the efficient gradient projection method for stochastic optimal control that motivates the projection update structure.","marker":"[14]"},{"why":"Provides the high-dimensional HJB equation and benchmark used to demonstrate the batch projection algorithm with randomized neural networks.","marker":"[11]"}],"fun_headline_variants":["Batch paths speed up optimal control convergence","Faster optimal control via batched backward paths","Control solver hits O(1/K+1/N^2) with batched paths","Batched paths improve stochastic control convergence rate","Batch sample-wise control: faster O(1/K+1/N^2) rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The $N^{-2}$ part of the rate rests on Lemma 4.7's requirement that the forward SDE be simulated with a scheme of order at least $3/2$ and that the unusual condition $E[|X_N-X|^4] \\le C E[|X_N-X|^2]$ holds; if only an Euler scheme is used or the moment condition fails, the discretization error reverts from $O(h^2)$ to $O(h)$, and the advertised rate degrades to $O(\\sqrt{1/K+1/N})$.","fun_headline_variants_meta":{"raw":{"variants":["Batch paths speed up optimal control convergence","Faster optimal control via batched backward paths","Control solver hits O(1/K+1/N^2) with batched paths","Batched paths improve stochastic control convergence rate","Batch sample-wise control: faster O(1/K+1/N^2) rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000628,"raw_usage":{"total_tokens":2991,"prompt_tokens":1117,"completion_tokens":1874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":1790}},"tokens_in":733,"tokens_out":1874,"duration_ms":14429,"temperature":1.0,"reasoning_tokens":1790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:45:41.177362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a one-dimensional linear-quadratic control problem with known solution using Algorithm 1 with an Euler forward scheme but batched SGD; if the empirical error versus $N$ follows $1/N$ rather than $1/N^2$, the claimed $N^{-2}$ discretization term is not achievable without the higher-order scheme. Conversely, with the order-2 scheme of equation (7.30), $M=N$, and $K$ a constant multiple of $N^2$, the log-log error slope should be about $-1$; a plateau or a shallower slope would indicate that the variance bound or the fourth-moment assumption in Lemma 4.7 fails.","supporting_citations":[{"cited_title":"Archibald, F.Bao, Y.Cao, H.Sun","cited_arxiv_id":null,"evidence_quote":"Supplies the predecessor sample-wise backpropagation algorithm and the $O(\\sqrt{N/K+1/N})$ baseline rate that this paper improves."},{"cited_title":"Chessari, R","cited_arxiv_id":null,"evidence_quote":"Documents the first-order temporal discretization error of BSDEs used to justify the $O(h^2)$ error bound in Lemma 4.7."},{"cited_title":"Gobet and C","cited_arxiv_id":null,"evidence_quote":"Provides the error-expansion result for BSDE discretization that underlies the $O(h^2)$ estimate when the forward process is exact."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the explicit order-2 forward SDE scheme used in the numerical experiments for the projection algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the efficient gradient projection method for stochastic optimal control that motivates the projection update structure."},{"cited_title":"E., J.Han, and A","cited_arxiv_id":null,"evidence_quote":"Provides the high-dimensional HJB equation and benchmark used to demonstrate the batch projection algorithm with randomized neural networks."}],"review_version":1}