{"id":"86f6cce6-26f6-4baa-b993-924cdf71e240","arxiv_id":"2608.10777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PI-VM trains a value network with TD updates on a recursive path integral target, giving faster and more stable stochastic optimal control than policy-based baselines.","lead":"The paper derives a short-horizon recursion for the path integral optimal value function and uses it to train a value network via temporal-difference learning, replacing full-trajectory simulation. The proposed PI-VM solver runs faster per update and scales beyond a leading baseline on the tested benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.3's convergence proof requires strictly positive running cost (A1), but all five benchmarks violate this assumption, so the stated theoretical foundation does not cover the paper's own experiments.","rationale":"The reader's weakest-assumption analysis correctly identified the mismatch between Assumption A1 of Theorem 3.3 and the f = 0 or f(x) = x^T P x running costs used in all five benchmarks. My stress-test pass confirms that the contraction proof in Appendix B genuinely depends on a strictly positive running-cost lower bound, and that the main text and appendix state different auxiliary assumptions for the same theorem. This is the most load-bearing concern because the paper's stated theoretical foundation for the TD algorithm is the convergence guarantee of Theorem 3.3; if that guarantee does not cover the evaluated regime, the efficiency and stability claims rest on empirical evidence alone. I do not see a fatal flaw in Proposition 3.1 or in the general recursive construction, and the f = 0 case may still converge by finite-horizon backward induction, so the appropriate disposition is to require a fix or a caveat rather than to reject the work. The conditional verdict already captures this, so no verdict change is needed. My concrete test isolates exactly whether the missing assumption is truly load-bearing or merely an artifact of the proof technique.","tokens_in":19033,"tokens_out":14734,"duration_ms":147112,"concrete_test":"Prove or disprove convergence of the recursion in Definition 3.2 without Assumption A1. Concretely, attempt a finite-horizon backward-induction proof: for any fixed t, after K = ceil((1-t)/epsilon) iterations the update uses only the terminal condition g (because s = 1 is reached), and therefore V_k(x,t) equals the Feynman-Kac solution for all k >= K, provided the requisite integrability holds. If this proof succeeds, A1 is an artifact of the contraction argument and the practical concern is resolved; if it fails, the paper must either add a positive running-cost term in the benchmarks, restrict Theorem 3.3 to f > 0 with an explicit caveat, or supply numerical convergence evidence for f = 0.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 (Eq. 7) is a correct Feynman-Kac semigroup identity; it is not the weak point. The weak point is Theorem 3.3, whose Appendix B proof uses a Banach contraction with factor exp(-f_inf epsilon) < 1. That factor exists only under Assumption A1 (f >= f_inf > 0). All five benchmarks violate A1: Linear OU, GMM, and Many Well set f ≡ 0 (Appendix G.2.1, H.2.1, H.3.1), and Quadratic OU uses f = x^T P x, which vanishes at x = 0 (Appendix G.1.1). The paper never discloses that its stated convergence guarantee excludes its own experimental regime. The issue is not cosmetic: with f = 0 the update operator in Definition 3.2 is only nonexpansive (gamma = 1), so the submitted proof gives no stability margin for the TD bootstrap, and the appendix's proof also requires the value iterates to lie in C_b, which the quadratic benchmarks (unbounded V) do not satisfy. In addition, the main-text Assumption A2 ('V bounded below') differs from the appendix's A2 ('g continuous and bounded'), so the stated theorem is not internally consistent. The cited convergence result therefore cannot be invoked to justify PI-VM on the tasks where PI-VM is claimed to work.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PI-VM, a value-based solver for linear-quadratic stochastic optimal control. Starting from the classical path-integral representation of the optimal value function, it derives a temporal recursion (Proposition 3.1) and turns it into a temporal-difference training objective with short M-step Monte Carlo branches, Girsanov reweighting, and experience replay. The paper states a convergence theorem for the value-iteration analog (Theorem 3.3), a variance-decomposition claim (Proposition 3.4), and an off-policy variance bound (Theorem 4.3). It reports experiments on three unimodal SOC tasks and two multimodal sampling tasks, with ablations and a high-dimensional scalability study.","tokens_in":19267,"tokens_out":12150,"duration_ms":123524,"significance":"The central idea is attractive and potentially important: replacing full-trajectory path-integral Monte Carlo by a recursive, short-horizon value-matching objective could remove the main computational bottleneck of path-integral control. The tower-property derivation of Eq. (7) is clean, the connection to Feynman-Kac is standard, and the experimental results are consistent with the claimed behavior on the reported benchmarks. The paper also includes useful ablations and analytic ground-truth benchmarks. However, the theoretical support is currently narrower than the claims: the convergence theorem excludes all five experimental settings, and the off-policy variance bound has a gap in its proof. The efficiency claim also needs a clearer definition of the reported times.","major_comments":[{"comment":"The stated convergence guarantee is not applicable to any of the five benchmarks. Theorem 3.3 relies on Assumption A1, f(x,t) >= f_inf > 0, which gives the contraction factor exp(-f_inf epsilon) < 1 in the Appendix B proof. But Linear OU, GMM, and Many Well all use f ≡ 0 (Appendix G.2.1, H.2.1, H.3.1), and Quadratic OU uses f = x^T P x, which vanishes at x = 0 (Appendix G.1.1). The paper never discloses that its own experimental regime violates the theorem's main assumption. In addition, the main-text statement of Theorem 3.3 assumes A2: 'V bounded below', while Appendix B's restatement assumes g continuous and bounded and works in C_b; these are not equivalent, and neither holds for the unbounded quadratic value functions. This is load-bearing because Theorem 3.3 is presented as the theoretical foundation for PI-VM. The authors should either prove convergence under a weaker norm or a weighted contraction that covers the quadratic setting, or explicitly restrict the theoretical claims and describe the experimental section as outside the theorem.","section":"§3.2 / Appendix B (Theorem 3.3)"},{"comment":"The proof of the off-policy variance bound contains an unjustified step. After the Girsanov change to the measure Q^u with drift b - sigma u + 2 sigma u*, the proof bounds E^{Q^u}[ exp(-2V(X_r,r) + ∫_t^r ||u-u*||^2 dτ) | F_t ] by exp(-2V(X_t,t)) exp(kappa(r-t)). This treats exp(-2V(X_r,r)) as if it were exp(-2V(X_t,t)), but V(X_r,r) is random under Q^u and is not shown to be monotone along the shifted process. No pointwise comparison V(X_r,r) >= V(X_t,t) is available from the assumptions. The claimed variance-vanishing behavior of the off-policy estimator is therefore not established. The theorem should either be proved with a valid bound (for example, using a pathwise bound or a different localization argument) or removed from the theoretical claims.","section":"Appendix C (Theorem 4.3)"},{"comment":"The manuscript reports a 'TIME' column in Tables 1 and 2 but never defines what it measures. If it is per-update or per-iteration wall-clock time, the statement that PI-VM is '10–20x faster' is not justified unless PI-VM also reaches the reported control error in the same number of updates as the baselines. The text says that training hyperparameters are aligned with SOCM for the first three tasks, but it does not state that all methods are run for the same number of updates or that the reported error is the final error after a fixed budget. Please report total training time to a fixed control-error threshold, or explicitly report per-update time together with learning curves that allow a total-time comparison.","section":"§5.1 / Table 1"}],"minor_comments":[{"comment":"The phrase 'truncating and marginalizing the original path integral formulation' is not quite accurate; Eq. (7) follows directly from the tower property of conditional expectation. The wording could be adjusted to avoid implying a new approximation.","section":"§3.1"},{"comment":"The text says the off-policy loss has 'the same setup with Theorem 4.1'; this should be Definition 4.1, not Theorem 4.1.","section":"Definition 4.2"},{"comment":"The pseudocode does not state whether the N short-term branches sampled at line 7/8 follow the uncontrolled measure P^0 or the controlled measure P^u. This matters because the Girsanov correction S in Eq. (17) is only valid if the branches are generated under the control appearing in S. Please specify the sampling measure explicitly, including whether the stored S corresponds to the control used at collection time or to the current control during replay.","section":"Algorithm 1 / Algorithm 2"},{"comment":"The ablation table reports only point estimates without standard deviations, unlike Tables 1, 2, and 4. Adding error bars would make the 'no reduction from M=8 to M=16' conclusion more robust.","section":"Table 3"},{"comment":"Using the same symbol V for both the value function and variance in Proposition 3.4 is confusing; consider using Var(·) for variance.","section":"Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The core recursion is correct and the empirical story is plausible, but the theoretical section currently overclaims relative to what is proved. The most important fix is to address the mismatch between Theorem 3.3's assumptions and the experimental settings; if that cannot be repaired, the authors should reposition the convergence result as a restricted statement and be explicit that the benchmarks lie outside it. The variance-bound proof also needs a genuine repair rather than a rewording. If those issues are addressed, the paper could be a solid contribution to the continuous-time SOC/RL literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PI-VM is a sensible value-based solver for path integral control, and the empirical results are plausibly good, but the paper oversells both the novelty and the efficiency claim. The recursive identity in Proposition 3.1 is the standard Feynman-Kac semigroup property, as the reader's report says; the genuinely new piece is the TD-plus-Girsanov training scheme with short-horizon branch sampling. That is a legitimate algorithmic contribution, and the ablation study shows the method is doing something real. Credit where due: the method is coherent, the off-policy reweighting via Girsanov is a natural fit, and the d=200 result where SOCM runs out of memory is worth reporting, even if the comparison is not apples-to-apples.\n\nNow the soft spots. The convergence theorem (Theorem 3.3) requires a strictly positive running cost, f >= f_inf > 0. Every benchmark except Quadratic OU has f = 0, and even there f = x^T P x vanishes at the origin. The paper never flags that its stated convergence guarantee excludes its own experimental regime. The appendix proof also assumes bounded value iterates and bounded terminal cost, which the quadratic benchmarks violate, and the stated Assumption A2 differs between main text and appendix. So the theoretical foundation does not cover the experiments. Second, the runtime numbers in Table 1 are per-update times, not total time to reach a given control error; '10-20x faster' is not established, though the per-step cost advantage is real. Third, the variance decomposition in Proposition 3.4 is just the law of total variance; the interpretation of the second term as variance reduction is fine but not deep. The 'paradigm shift' framing is too strong.\n\nThe central idea—replace long-trajectory Monte Carlo with short-horizon TD targets—is sound and worth pursuing. The convergence analysis needs either to be repaired to cover f >= 0 cases (maybe with a different argument) or honestly stated as applying only to strictly dissipative running costs. The runtime comparison needs wall-clock-to-accuracy numbers. These are addressable.\n\nWho it is for: people working on stochastic optimal control solvers, especially the SOCM and adjoint-matching crowd. It deserves a serious referee; it is a coherent algorithm with plausible gains, not a crank submission. I would send it to review with the expectation of revision.","headline":"A useful TD-style shortcut for path integral control, with real but overstated speed advantages and a convergence theorem that does not cover its own benchmarks.","tokens_in":19896,"tokens_out":2526,"would_cite":true,"duration_ms":20951,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["93E20","49L25","60H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"The optimal value function for stochastic control satisfies a temporal recursion, so it can be learned by short-horizon TD matching instead of full-trajectory path integrals.","keywords":["path integral control","stochastic optimal control","value function","temporal difference learning","Girsanov theorem","off-policy learning","high-dimensional control","linear quadratic control"],"falsifier":"Run the iterative update of Definition 3.2 on a problem with f=0 and a linear terminal cost (as in the Linear OU benchmark) and measure ∥V_k − V∥; if the sequence converges to the optimal value despite f_inf=0, the stated contraction-based argument is not the reason, and if it diverges or oscillates, the theorem's assumption is violated. A more controlled test is to set f(x,t) = −c < 0 and check whether the fixed-point equation exp(−V) = E[exp(−V) exp(∫ c)] admits the claimed bounded solution, and to compare the empirical variance of the short-horizon estimator against the full-trajectory estimator on a fixed problem to verify Proposition 3.4's variance reduction term is positive.","tokens_in":18785,"feed_emoji":"🎯","tokens_out":5042,"duration_ms":48511,"temperature":0.7,"pith_summary":"The paper claims that the path integral representation of the optimal value function for linear-quadratic stochastic optimal control can be truncated and marginalized into a temporal recursion, so that the value at time t is an expectation of the value at a later time s discounted by the running cost over [t,s]. This recursion, its Proposition 3.1, turns the global Feynman-Kac expectation into a local bootstrapping target, which is the basis of a new algorithm, PI-VM, that learns the value function with temporal-difference updates and short M-step sample branches. If correct, it replaces the high-variance, expensive full-trajectory Monte Carlo used by policy-based solvers and classical path integral control with short-horizon, off-policy training. The experiments support this with an order-of-magnitude speedup in low dimensions and stable convergence in up to 200 dimensions.","feed_headline":"Value recursion makes path integral control 10x faster","feed_subtitle":"Replacing full-trajectory Monte Carlo with short bootstrapped targets tames high variance.","key_machinery":"The central object is the recursive path integral identity exp(−V(X_t,t)) = $E^{{P0}}$[ exp(−V(X_s,s)) exp(−∫_t^s f dr) | F_t ], with V(x,1) = g(x). It turns the global path integral (Feynman-Kac representation) into a local transition operator that maps the value function at time s to the value at time t. The PI-VM algorithm fits a neural network V_θ(x,t) to satisfy this identity by minimizing a squared TD loss whose target is the Monte Carlo average of exp(−W − G) over N short M-step branches; the Girsanov theorem provides the density ratio that makes the same loss valid off-policy.","core_discovery":"The discovery is that exp(−V(X_t,t)) = $E^{{P0}}$[ exp(−V(X_s,s)) exp(−∫_t^s f(X_r,r) dr) | F_t ] for any 0 < t < s, with V(x,1) = g(x), i.e., the path integral value function satisfies a multiplicative recursion under the uncontrolled measure. The paper proves this as Proposition 3.1 via the tower property of conditional expectation, and uses it to define an iterative bootstrapping rule (Definition 3.2). The central claim is that this recursion is the right object for practical computation: it converts a long-horizon, high-variance path integral into a short-horizon conditional expectation that can be learned by temporal-difference matching, with off-policy data re-weighted by the Girsanov density ratio so that the estimator variance vanishes as the sampling control approaches optimality (Theorem 4.3).","pith_inferences":["The recursion is not limited to LQ-SOC: the same log-sum-exp identity appears in any linearly solvable MDP with the Cole-Hopf structure, so a path integral value matching target could be applied to risk-sensitive RL where the log-sum-exp Bellman operates on the same object.","The paper's own convergence theorem excludes its experimental f=0 cases; a natural extension is to prove convergence with nonnegative (rather than strictly positive) running cost, e.g., via a weaker contraction in a weighted norm, or to show that in practice the TD update acts as a damped iteration even when the contraction factor equals one.","A testable extension is to apply PI-VM to Schrödinger bridge / diffusion-model sampling problems where the terminal cost g is a log-density; the off-policy Girsanov reweighting should allow training on data from a suboptimal sampler and still recover the optimal value."],"forward_implications":["Full-trajectory Monte Carlo is replaced by short-horizon TD targets, so the per-iteration cost becomes independent of the time horizon.","Because the value function is learned directly, control can be recovered as u = −σ^T ∇V, and the paper reports that this indirect route matches or beats direct policy optimization in accuracy on quadratic benchmarks.","Off-policy training via experience replay and Girsanov reweighting is legitimate, with the estimator variance provably vanishing as the sampling policy approaches the optimal one (Theorem 4.3).","The recursive estimator's variance reduction grows with the distance from the terminal time, making long-horizon problems relatively easier than naive PIC.","The method scales to at least 200 state dimensions without the out-of-memory failures of full-trajectory policy-search baselines."],"supporting_citations":[{"why":"Supplies the linearly-solvable MDP formulation that yields the path integral representation used from the start.","marker":"[9]"},{"why":"Gives the path integral control framework and its Feynman-Kac derivation that the recursion extends.","marker":"[10]"},{"why":"Establishes the linear theory for control of nonlinear stochastic systems justifying the integral representation.","marker":"[11]"},{"why":"Provides the Feynman-Kac lemma and Girsanov theorem that underlie the recursive identity and the off-policy reweighting.","marker":"[14]"},{"why":"Supplies the Cole-Hopf transform converting the HJB equation into the linear PDE behind the path integral.","marker":"[17]"}],"fun_headline_variants":["Value recursion makes path integral control 10x faster","Recursive value function slashes path integral cost","Off-policy value matching speeds up SOC by 10x","Path integral value recursion kills high variance","PI-VM: bootstrapped path integral control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof (Theorem 3.3) assumes the running cost f(x,t) is bounded below by a positive constant, giving a contraction factor exp(−f_inf ε) < 1; every benchmark in the paper uses f=0 (Linear OU, GMM, Many Well) or f=x^T P x that vanishes at the origin (Quadratic OU), so the contraction condition is not satisfied in the experimental regime.","fun_headline_variants_meta":{"raw":{"variants":["Value recursion makes path integral control 10x faster","Recursive value function slashes path integral cost","Off-policy value matching speeds up SOC by 10x","Path integral value recursion kills high variance","PI-VM: bootstrapped path integral control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1516,"prompt_tokens":949,"completion_tokens":567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":565,"tokens_out":567,"duration_ms":5705,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:32:48.250105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the iterative update of Definition 3.2 on a problem with f=0 and a linear terminal cost (as in the Linear OU benchmark) and measure ∥V_k − V∥; if the sequence converges to the optimal value despite f_inf=0, the stated contraction-based argument is not the reason, and if it diverges or oscillates, the theorem's assumption is violated. A more controlled test is to set f(x,t) = −c < 0 and check whether the fixed-point equation exp(−V) = E[exp(−V) exp(∫ c)] admits the claimed bounded solution, and to compare the empirical variance of the short-horizon estimator against the full-trajectory estimator on a fixed problem to verify Proposition 3.4's variance reduction term is positive.","supporting_citations":[{"cited_title":"Linearly-solvable markov decision problems.Advances in neural information processing systems, 19, 2006","cited_arxiv_id":null,"evidence_quote":"Supplies the linearly-solvable MDP formulation that yields the path integral representation used from the start."},{"cited_title":"An introduction to stochastic control theory, path integrals and reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Gives the path integral control framework and its Feynman-Kac derivation that the recursion extends."}],"review_version":1}