REVIEW 3 major objections 4 minor 4 references
A new deep-learning method solves high-dimensional dynamic programs with recursive utility by learning the certainty equivalent itself.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 20:25 UTC pith:MIVYPTXP
load-bearing objection Solid computational methods paper: CEL is a real design for recursive-utility DP, with strong closed-form and VFI evidence, but residual claims need a bias audit and code. the 3 major comments →
Deep Learning for Dynamic Programming with Recursive Utility
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that treating the state-control certainty equivalent as a separate learnable neural object, then alternating between certainty-equivalent fitting, policy improvement, and value fitting, yields the first practical deep-learning solution method for high-dimensional discrete-time dynamic programming with recursive utility, and that the resulting solutions match closed-form and value-function-iteration benchmarks with residual accuracy of order 10^{-4}–10^{-3}.
What carries the argument
Certainty Equivalent Learning (CEL): neural networks jointly approximate the value, the policy, and the certainty-equivalent map Vc(s,c)=f^{-1}(E[f(V(s'))|s,c]), updated by alternating Monte-Carlo regression, Bellman maximization, and (optional) value fitting, with target networks and delayed policy updates for stability.
Load-bearing premise
The method assumes that nonconvex alternating training of the three networks, fed by Monte Carlo targets that are known to be biased for nonlinear certainty equivalents, still converges to a solution of the true Bellman equation that residual diagnostics of size 10^{-4}–10^{-3} can certify.
What would settle it
On the closed-form Gaussian linear-exponential-quadratic problem with 100 states and 50 controls, recompute the learned policy and value on a fresh out-of-sample state set; if relative value error exceeds 10^{-3} or absolute policy error exceeds 10^{-2} relative to the analytic solution, the accuracy claim fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Certainty Equivalent Learning (CEL), claimed as the first deep-learning algorithm for high-dimensional discrete-time infinite-horizon dynamic programming with recursive utility. CEL represents the value function, policy, and (state-control) certainty-equivalent map by neural networks (three-, four-, or two-network variants), trains them by alternating simulation-based least-squares fitting of the CE operator, policy improvement of the Bellman objective, and value fitting, and stabilizes training with exploratory control noise, Polyak target networks, and delayed policy updates. It requires only forward simulation of the transition and does not use Euler equations, first-order conditions, or differentiability of the law of motion. Expected utility is recovered as a special case. Numerical experiments cover discounted linear-exponential-quadratic-Gaussian control (closed-form, including ns=100, nc=50), small-noise robust control (VFI and closed-form homothetic benchmarks), an Epstein–Zin DSGE with stochastic volatility, and multivariate strategic asset allocation; reported out-of-sample Bellman errors and Euler/FOC residuals typically lie in [1e-4,1e-3].
Significance. If the numerical claims hold, CEL would be a practically useful mesh-free tool for a class of models (Epstein–Zin, risk-sensitive/robust control, long-horizon portfolio choice) that are central in macro-finance yet remain hard for grid-based methods once state or control dimension exceeds a handful of variables. The modular CE-network idea, the explicit comparison with closed-form LEQG and homothetic robust-control solutions, and the out-of-sample residual diagnostics are genuine strengths relative to much of the existing deep-learning DP literature. The careful differentiation from Friedl et al. (2023) and the demonstration that the same framework also covers expected utility further increase potential impact.
major comments (3)
- Section 3.1 (third challenge) correctly notes that the plug-in estimator f^{-1}((1/G) sum f(V(s'))) is biased for the true certainty equivalent whenever f^{-1} is nonlinear, with leading bias proportional to (f^{-1})''(M) Var(cM)/2. Yet the training losses for the three- and four-network architectures—Eqs. (28), (32), (44), (49)–(50)—minimize exactly this plug-in residual (or its Ve–D decomposition) with finite G and no bias-correction, importance sampling, or G-scaling study. Target-network Polyak averaging and delayed policy updates stabilize bootstrapping but do not cancel the Jensen bias. Consequently the reported residual band of 1e-4–1e-3 (especially under high curvature: sigma=50/200, gamma=20) does not by itself certify that the learned (V,c,Vc) solve the true Bellman fixed point. A bias audit (e.g., G-refinement, antithetic or control-variate estimators, or a Santos-style residu
- The central scalability claim rests on the LEQG experiment with ns=100, nc=50 (Section 4.1.3). While relative value errors <1e-4 and small policy errors are impressive against the closed form, the paper supplies no analogous external benchmark for the non-quadratic applications once dimension rises. In the DSGE (3 states, 2 controls) and multivariate portfolio problems the only diagnostics are internal residuals recomputed by nested Monte Carlo. Without either a high-accuracy reference solution or a quantitative residual-to-welfare/policy-error bound, it remains unclear whether residual levels of 1e-3 translate into economically acceptable approximation error once the CE surface is high-dimensional and strongly curved.
- No convergence theory, approximation-rate result, or even a systematic hyper-parameter sensitivity study is provided for the alternating non-convex scheme (Algorithms 3–5). The free parameters (network widths, tau, d, zeta, G, Adam step counts, exploration method) are numerous; the paper’s own discussion of feedback loops between CE learning and policy improvement (Section 3.1, fifth challenge) makes the absence of such analysis material. At minimum, a short ablation on G, tau and the exploration scale, together with multiple random seeds and reported standard errors on the residual tables, is required to support the claim that the method “remains effective in high-dimensional problems.”
minor comments (4)
- Abstract and several figure captions contain missing spaces (“accuracycomparabletoVFI”, “out-of-sampleBellmanerrors”). A careful copy-edit pass is needed.
- Table 1 and the network-architecture subsections would benefit from an explicit statement of layer widths, activations and output constraints used in each experiment; currently these details are scattered or omitted.
- The comparison with Friedl et al. (2023) is thorough, yet the claim “the first deep learning algorithm” should be qualified more carefully in the abstract and introduction once the differences (quadrature vs. learned CE, residual system vs. direct Bellman maximization) have been stated.
- Appendix A.2 reports matrices only for the ns=8 case; the high-dimensional LEQG matrices are not supplied, hindering exact replication.
Circularity Check
No significant circularity: CEL is an approximate numerical method whose accuracy claims rest on external closed-form/VFI benchmarks and independent out-of-sample residual recomputation, not on tautological restatement of training losses.
full rationale
The paper's derivation chain is a standard simulation-based deep-learning approximation to the Bellman fixed point under recursive utility: neural nets for V, c and Vc (or Ve-D) are trained by alternating minimization of plug-in CE residuals and maximization of the Bellman objective (Eqs. 28-34, 44, 49-52), then evaluated by (i) direct comparison to analytical LEQG and homothetic robust-control solutions, (ii) relative differences versus independent VFI grids, and (iii) out-of-sample Monte-Carlo recomputation of Bellman/Euler/FOC residuals on fresh shocks (Sections 4.1-4.4 and Appendices A.3-A.4). These diagnostics are not the training objectives, use new samples, and are zero only at the true solution; residual size is therefore an independent consistency check rather than a by-construction identity. No parameter is fitted to a data subset and then re-labeled a prediction; no uniqueness theorem or ansatz is imported from the authors' prior work as a load-bearing premise; and the mild self-reference that learned networks are checked against the equations they approximate is ordinary numerical validation, not circularity. The method is therefore self-contained against external anchors.
Axiom & Free-Parameter Ledger
free parameters (5)
- Neural network widths/depths and activations for V, c, Vc (or Ve, D)
- Polyak target-update rate τ and policy delay d
- Exploration scale ζ and Method ∈ {Gaussian, Sobol}
- Monte Carlo sample sizes N, T, G and optimizer settings (Adam steps Kξ, Kθ, Kϕ, …)
- Model preference/robustness parameters (e.g., σ, γ, β, ψ) in applications
axioms (5)
- domain assumption Recursive utility is well-defined as a fixed point of the Bellman operator with certainty equivalent Vc = f^{-1}(E[f(V(s'))|s,c]) under the stated Markov transition.
- ad hoc to paper Neural networks with the chosen architectures can approximate the value, policy, and certainty-equivalent maps to the accuracy needed for residual levels ~1e-4–1e-3.
- domain assumption Out-of-sample Bellman errors and Euler/FOC residuals are valid primary accuracy diagnostics for recursive-utility solutions (following Santos 2000-style residual logic).
- standard math Expected utility is nested when f is linear and the aggregator is additive, so CEL applies to standard expected-utility DP as a special case.
- domain assumption State transitions need only be simulable; differentiability of ψ is not required for CEL training.
invented entities (2)
-
Certainty Equivalent Learning (CEL) algorithm
no independent evidence
-
Three-/four-/two-network CEL architectures (including Ve − D decomposition of the certainty equivalent)
no independent evidence
read the original abstract
We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein-Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from 1.0e-4 to 1.0e-3 for most problems.
Figures
Reference graph
Works this paper leans on
-
[1]
M., Xu, Z
Aboussalah, A. M., Xu, Z. and Lee, C.-G. (2022). What is the value of the cross-sectional approach to deep reinforcement learning?,Quantitative Finance 22(6): 1091–1111. Anderson, E. W., Hansen, L. P. and Sargent, T. J. (2012). Small noise methods for risk-sensitive/robust economies,Journal of Economic Dynamics and Control 36(4): 468–500. Andreasen, M. M....
2022
-
[2]
Azinovic, M., Gaegauf, L. and Scheidegger, S. (2022). Deep equilibrium nets,Inter- national Economic Review63(4): 1471–1525. Azinovic, M. and Jan Žemlička (2023). Economics-inspired neural networks with stabilizing homotopies,arXiv.Available at https://arxiv.org/abs/2303.14802. Bachouch, A., Huré, C., Langrené, N. and Pham, H. (2022). Deep neural networks...
Pith/arXiv arXiv 2022
-
[3]
Epstein, L. G. and Zin, S. E. (1989). Substitution, risk aversion, and the temporal behavior of consumption and asset returns: a theoretical framework,Econometrica 57(4): 937–969. Epstein, L. G. and Zin, S. E. (1991). Substitution, risk aversion, and the tempo- ral behavior of consumption and asset returns: An empirical analysis,Journal of political Econo...
Pith/arXiv arXiv 1989
-
[4]
Kreps, D. M. and Porteus, E. L. (1978). Temporal resolution of uncertainty and dynamic choice theory,Econometrica46(1): 185–200. Lepetyuk, V., Maliar, L. and Maliar, S. (2020). When the US catches a cold, Canada sneezes: A lower-bound tale told by deep learning,Journal of Economic Dynamics and Control117: 103926. Lillicrap, T. P., Hunt, J. J., Pritzel, A....
Pith/arXiv arXiv 1978
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.