DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems

Changgen Xie; Michael Ludkovski; Zimu Zhu

REVIEW 3 major objections 7 minor 1 cited by

DeepPAAC, a new actor-critic deep learning method, solves continuous-time principal-agent problems by training one network for the principal's value function and another for the agent's controls, directly from the HJB equation's residual an

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:43 UTC pith:E5NISJXQ

load-bearing objection A genuinely useful numerical scheme for continuous-time principal-agent HJB equations, with solid checks against explicit solutions — but the stopping criterion only enforces first-order stationarity, so the no-closed-form case is not fully established. the 3 major comments →

arxiv 2511.04309 v3 pith:E5NISJXQ submitted 2025-11-06 math.NA cs.LGcs.NA

DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems

Michael Ludkovski , Changgen Xie , Zimu Zhu This is my paper

classification math.NA cs.LGcs.NA MSC 91B4368T0793E20

keywords principal-agent problemHamilton-Jacobi-Bellman equationdeep Galerkin methodactor-criticpolicy improvementimplicit Hamiltonianneural networkscontract theory

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that continuous-time principal-agent problems — where the principal's optimal contract value satisfies a Hamilton-Jacobi-Bellman (HJB) equation, the nonlinear PDE describing the value of a controlled stochastic process — can be solved numerically by a deep-learning scheme that treats the value function and the optimal controls as two separate neural networks. The scheme, called DeepPAAC, alternates stochastic gradient steps that minimize the squared PDE residual for the value network and the first-order optimality condition for the control network, so it never needs to solve the agent's maximization in closed form or discretize time. If the claim is right, it gives economists and financial engineers a general numerical solver for contracting problems with multidimensional states, multidimensional controls, and constraints — settings that have mostly lacked numerical methods. Five case studies, including a two-dimensional state with two-dimensional control and binding inequality constraints, back the claim, with the reported accuracy and speed comparing favorably to the standard Deep Galerkin Method.

Core claim

The central discovery is that the principal-agent HJB equation with an implicit Hamiltonian can be solved without a separate numerical procedure for the agent's optimization. DeepPAAC encodes the agent's best response implicitly through the control network, and the value network is fitted to the PDE residual while the control network is fitted to the gradient of the Hamiltonian — the first-order condition of the inner maximization. The terminal condition is injected as G(x)+(T-t)v(t,x;θ_V); constraints on the controls enter as a penalty in the control loss. On the exponential-utility benchmark the algorithm recovers the constant optimal control; on the continuous-payment example it converges

What carries the argument

The central mechanism is the actor-critic structure of the policy improvement algorithm ported to the Deep Galerkin Method. Two neural networks are trained alternately: the critic v(t,x;θ_V) approximates the principal's value function and is updated by minimizing the squared residual of the HJB operator; the actor u(t,x;θ_u) approximates the optimal control and is updated by descending the gradient of the Hamiltonian, i.e. the first-order condition of the inner maximization. The implicit Hamiltonian is therefore never evaluated; only its derivatives with respect to the controls are needed, which removes the need to solve the agent's variational problem. The terminal condition is hard-wired i

Load-bearing premise

The paper's results rest on the assumption that small values of the PDE-residual loss and the control-gradient loss on a fixed validation set imply small approximation error of the value function and the optimal control on the whole domain; no convergence theorem is provided for the alternating stochastic-gradient scheme, so the reported convergence could be spurious if the validation points miss regions where the networks fit poorly.

What would settle it

Construct a principal-agent problem with a closed-form solution, train DeepPAAC with its stopping rule based on a fixed validation set, and then evaluate the learned value function and control on a dense independent grid far from the training and validation samples. If the residual is small on the validation set but the value deviates from the closed-form solution by more than the tolerance on that dense grid, the claim is falsified. A simpler check: keep the same training setup but shrink the validation set to a small subregion; if the stopping rule is satisfied while the solution is poor out

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Principal-agent models no longer need to be restricted to settings with closed-form value functions; multidimensional states, multidimensional controls, and control constraints become tractable with a single general algorithm.
The method solves HJB equations with implicitly defined Hamiltonians — where the inner maximization cannot be done in closed form — without time discretization or nested optimization at grid points.
On the tested examples, convergence is reached in hundreds of SGD steps rather than thousands for the standard Deep Galerkin Method, indicating a substantial speed-up.
Constrained contracts, which the analytical literature has largely avoided because constraints preclude closed-form solutions, become numerically explorable through the penalty-function extension.
The case without a closed-form solution produces value and effort surfaces whose qualitative shape matches the theory (e.g. monotonicity of the value function in the output and decreasing effort in the agent's reservation), opening the door to numerical-guided model discovery.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

An extension the authors do not make: the same actor-critic loop could be carried to principal-agent models with several agents or mean-field interactions, since the construction is mesh-free and dimension-agnostic; convergence on such problems remains an open question.
The fixed validation set used for stopping could be stress-tested by shrinking it to a subregion; if the algorithm passes the stopping rule while failing on an independent dense grid, residual smallness would be shown insufficient — a check the paper does not perform.
The speed advantage over the standard DGM suggests a broader design principle for HJB solvers with implicit Hamiltonians: split value and control across two networks and alternate single SGD steps rather than resolving the inner supremum at each iteration.
For the two-dimensional case without closed-form solution, comparing DeepPAAC's value surface to a coarse monotone finite-difference solve would corroborate or challenge the reported solution; the paper does not include such a comparison.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful numerical scheme for continuous-time principal-agent HJB equations, with solid checks against explicit solutions — but the stopping criterion only enforces first-order stationarity, so the no-closed-form case is not fully established.

read the letter

You should know this paper before it disappears into the arXiv pile: it builds a practical actor-critic deep Galerkin solver for principal-agent HJB equations with implicit Hamiltonians, and it actually checks itself against several closed-form benchmarks. That is more than most papers in this area do. The authors also derive two genuinely new explicit solutions (Prop. 3.1 for dynamic Holmström–Milgrom, Prop. 4.2 for constrained contracts), and I verified the algebra in both. The numerical comparisons to DGM are fair and show a real speed advantage. If you work on computational contract theory, this is worth a look.

The main soft spot is the convergence criterion. The stopping rule requires the PDE residual and the control gradient to be small. That only guarantees the control is a stationary point of the Hamiltonian, not a global maximizer. For the unconstrained examples with concave or quadratic Hamiltonians, stationary is enough, and the good agreement with explicit solutions backs that up. But in Section 4.5 — the only case without a closed-form solution — the Hamiltonian contains the term 0.5 a^2 Z^2 V_ww, so the Hessian can be indefinite and the gradient condition can be satisfied at a saddle. The reported loss curves do not rule that out. So the paper's most ambitious numerical claim rests on weaker evidence than the rest. This is fixable: add a check that the Hamiltonian value at the learned control dominates nearby perturbations, or validate against a Monte Carlo policy evaluation, or at least acknowledge the issue.

A separate concern is the heavy reliance on reference [23], an unpublished manuscript co-authored by one of the current authors, for the benchmark model. That is not circular in the algorithmic sense — the residuals do not target the closed-form solutions — but it does make the model presentation hard to verify independently. The paper also ships no code or data, so the experiments cannot be reproduced. That is common but still a limitation.

Overall, the central algorithm and the explicit solutions are sound. The stress-test about saddle points is legitimate and should be addressed, but it does not undermine the cases with ground truth. I would send this to peer review. The right referee will ask for code, for a stronger validation in Section 4.5, and for a less enthusiastic interpretation of residual smallness there. With those revisions, this could become a standard reference for numerical PA problems.

Referee Report

3 major / 7 minor

Summary. The paper introduces DeepPAAC, a deep Galerkin-type algorithm for continuous-time principal-agent (PA) problems whose value function solves an HJB equation with an implicitly defined Hamiltonian. The method uses two alternating neural networks: one for the principal's value function and one for the agent's optimal controls, trained by minimizing PDE residuals, a first-order control-optimality condition, and (for constrained problems) a penalty term. The terminal condition is hard-coded into the value network. The authors report five case studies: a Holmström-Milgrom toy model, a continuous-payment model with explicit solution, a constrained-contract extension, a two-dimensional model with scalar control and closed-form solution, and a two-dimensional model with two controls and no explicit solution. The paper claims stability, accuracy, and speed compared with the standard DGM/PIA baseline, supported by residual plots, error maps, and convergence-step comparisons.

Significance. If the numerical results are reliable, DeepPAAC would be a useful mesh-free solver for a class of PA problems with multi-dimensional states, multi-dimensional controls, and constraints, where the Hamiltonian cannot be computed in closed form. The paper's strengths include validation against several explicit solutions (Sections 3.4, 4.1, 4.3, 4.4), a detailed ablation of architecture and sampling choices, and a head-to-head comparison with DGM-PIA showing substantially faster convergence in the tested examples. However, the central methodological claim is not fully established: the stopping criterion only enforces first-order stationarity of the Hamiltonian, and the only case without a closed-form solution (Section 4.5) is also the only multi-dimensional-state/multi-dimensional-control case. There residual smallness on a fixed validation set is the sole evidence, so the claim that the algorithm solves that problem needs stronger support.

major comments (3)

[§3.1, Eq. (3.16); §4.5, Eq. (4.38)] The stopping criterion ∥Lctrl∥∞ ≤ Tolctrl checks only that the gradient of the Hamiltonian with respect to the control is approximately zero. The HJB equation (2.10) requires a global maximum of the Hamiltonian over the control set. The first-order condition is sufficient only when the Hamiltonian is concave in u, which is neither assumed nor verified. In Section 4.5 the Hamiltonian contains the term 0.5 a² Z² V_ww, so its Hessian in (a,Z) can be indefinite; a point with ∇H=0 may be a saddle point. Consequently, even if all validation residuals were zero, the trained pair (V,u) need not solve (4.38). Please add a global-optimality check (e.g., compare the learned control against a dense random/grid maximization of the Hamiltonian on the validation set, or verify concavity/second-order conditions for the examples) or restrict the claims accordingly.
[§4.5, Eq. (4.38), Fig. 9] This is the only case study with no explicit or otherwise independently known solution, and it is the only case with two state dimensions and two control dimensions. The evidence for correctness is that the validation residuals L_int and L_ctrl are below tolerance on a fixed validation set. No alternative solver, manufactured solution, convergence study with respect to network size, or seed-variability analysis is reported for this example. Residual smallness on a finite validation set does not guarantee global accuracy of the value function or the control. Please provide additional validation for Section 4.5 — for example, comparison with a different numerical method, a self-consistency check of the Hamiltonian maximization, or error bars from multiple independent training runs — before the multidimensional claim in the abstract and Section 1 is accepted.
[§2, §4.1, Ref. [23]] The general PA model in Section 2 and the explicit benchmark in Section 4.1 are taken from [23], which is listed as 'In Preparation' and co-authored by one of the present authors. Since the formulation, the Agent's best-response derivation, and the explicit solution of Proposition 4.1 all rely on that unpublished manuscript, the paper is not fully self-contained. Please either include the relevant assumptions and derivations in an appendix, or provide a publicly available version of [23] and state its status explicitly.

minor comments (7)

[§3.1, Eq. (3.16)] The notation (∇L/∇u) is undefined and appears to be a typo; it should be the partial derivative of the Hamiltonian with respect to u, i.e., ∂/∂u [L^u v + F](t,x,u). Please fix the notation and clarify the vector case.
[Algorithm step 5, §2.3] The definition L_u = -Σ [L^u v + F] and the subsequent gradient descent update make the algorithm a Hamiltonian maximizer, but the text says 'L_u should stabilize at a non-zero value.' This is confusing: the quantity being minimized is -H, so it should stabilize near a maximum of H, not a specific numeric value. Please clarify the sign convention and the role of L_u as a surrogate for the Hamiltonian, not a loss to be driven to zero.
[Table 1] In the DGM one-shared-NN row, the range is reported as [1110,*] with 'one run never achieved the stopping threshold over 10,000 steps.' The '*' is not explained, and the median-only summary obscures the non-convergence. Report the failure rate explicitly and, if possible, also give the loss values at the maximum step for non-converged runs.
[§4.4, Fig. 8] The comparison between DeepPAAC and DGM-PIA is based on a single training run or a representative run; no seed variability or convergence statistics are reported for this case. Given the SGD noise shown in Section 4.2, a multi-seed comparison would make the 'error much worse' claim more robust.
[§3.4, Fig. 2] The statement that a(t,x;0) and a(t,x;1) 'match the analytical exact solution' is not quantified. Add a numerical error table or a residual plot for these two boundary cases.
[§4.5, Fig. 9] The figure caption says convergence is achieved at n=85230/92580 steps, but no loss curves are shown for this example. Including the training/validation residual trajectories would help the reader assess whether the stopping threshold was genuinely reached and whether the solution is stable after convergence.
[References] Reference [23] is unpublished; please update its status if a preprint has appeared. Also, some technical assumptions (e.g., integrability of admissible controls, growth conditions on utility functions) are deferred to [23] and should be stated at least in compact form.

Circularity Check

0 steps flagged

No circular derivation; one minor self-citation to [23] is a sourcing concern, not load-bearing.

full rationale

The DeepPAAC algorithm minimizes the PDE residual L_int and the control stationarity metric L_ctrl on sampled space-time points; it does not fit its parameters to the closed-form solutions used as benchmarks. The explicit value functions in Section 3.4 and Section 4.1 are used only to define the target HJB equation or as post-training ground truth, not as training data. The terminal condition is enforced by construction via V_n(t,x)=G(x)+(T-t)v(t,x;theta), not by fitting G. The only self-citation is [23], an in-preparation manuscript co-authored by one of the current authors, which supplies the general PA model and the Section 4.1 benchmark. This is a legitimate sourcing and reproducibility concern, but it does not make the derivation circular: the solver's loss is the residual of the stated HJB equation itself, and Sections 4.4 and 4.5 test the same algorithm on independent examples from Cvitanic, Possamai and Touzi [7]. No uniqueness theorem, ansatz, or fitted parameter is imported from the authors' prior work in a way that would force the numerical result. The remaining gap—that small residuals and control stationarity do not by themselves prove a global Hamiltonian maximizer—is a correctness/convergence limitation, not a circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No new physical/economic entities are introduced. All free parameters are training hyperparameters. The main extra-credit assumptions are the unproved reliability of the validation-loss stopping criterion and the citation-based technical conditions.

free parameters (5)

Learning-rate schedule (polynomial decay α0=1e-3 → α=1e-4, power 0.8) = α0=1e-3, α=1e-4, p=0.8
Chosen by hand and used across all case studies; determines convergence speed and stability.
Training batch size M = M=2000
Used in all experiments; larger M recommended but memory-bound.
SGD steps per epoch B = B=10 (or 30 in §4.5)
Chosen by hand; smaller B reduces variance across runs.
NN architecture (L=3 residual layers, 32 neurons, swish) = L=3, width=32
Set in §3.4 and carried to other cases; not optimized per problem.
Stopping tolerances Tol_int, Tol_ctrl = 10^{-2} or 10^{-3}
Chosen per case study; determines when training halts.

axioms (5)

domain assumption The HJB equation (2.10)-(2.11) characterizes the Principal's value function under the technical conditions of [23].
Section 2.1 states the PA problem reduces to this HJB equation; the paper cites [23] for well-posedness and does not prove it.
standard math The BSDE comparison principle and stochastic maximum principle yield the optimal effort/consumption maps I_a, I_c.
Used in §2.1 to derive the state dynamics (2.9); cited from literature [23,26].
ad hoc to paper Neural networks suffice to approximate the value function and control well enough that small pointwise PDE residuals imply small value-function error.
The paper assumes the trained surrogates are globally accurate; no error bounds are provided (see §3.1 stopping criteria).
ad hoc to paper The fixed validation set is representative of the whole domain, so validation losses are adequate proxies for global loss.
Stopping criterion in §3.1 uses a single fixed validation set; if it is unrepresentative, 'convergence' may be spurious.
domain assumption Tie-breaking: if the Agent has multiple optima, he chooses the one best for the Principal.
Stated in §2 (after (2.6)); standard in PA but unverifiable in practice.

pith-pipeline@v1.3.0-alltime-deepseek · 18824 in / 25919 out tokens · 236493 ms · 2026-08-03T23:43:39.020410+00:00 · methodology

0 comments

read the original abstract

We consider numerical resolution of principal-agent (PA) problems in continuous time. We formulate a generic PA model with continuous and lump payments and a multi-dimensional strategy of the agent. To tackle the resulting Hamilton-Jacobi-Bellman equation with an implicit Hamiltonian we develop a novel deep learning method: the Deep Principal-Agent Actor Critic (DeepPAAC) Actor-Critic algorithm. DeepPAAC is able to handle multi-dimensional states and controls, as well as constraints. We investigate the role of the neural network architecture, training designs, loss functions, etc. on the convergence of the solver, presenting five different case studies.

Figures

Figures reproduced from arXiv: 2511.04309 by Changgen Xie, Michael Ludkovski, Zimu Zhu.

**Figure 2.** Figure 2: Estimated optimal control a(t, x) obtained from the DeepPAAC algorithm for different exponential mixture weights λ ∈ (0, 1). (a) λ = 0 (b) λ = 0.5 [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

**Figure 3.** Figure 3: Left: a(t, x; 0) for λ = 0. Right: a(t, x; 0.5) for λ = 0.5 15 [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

**Figure 4.** Figure 4: Left (a): Contour plot of the value function error metrics L¯ int(θ V n ;t, w) at n = 300 training steps. Right (b): L2-norm and L∞-norm for L¯ int(θ V n ;t¯· , w¯ · ) across n. 4.2 DeepPAAC Variants The defined DeepPAAC algorithm features many hyperparameters that can be finetuned during implementation. In this section, we describe several experiments that demonstrate the role of these hyperparameters. Fi… view at source ↗

**Figure 5.** Figure 5: Validating convergence of the DeepPAAC loss functions. We show the [PITH_FULL_IMAGE:figures/full_fig_p020_5.png] view at source ↗

**Figure 6.** Figure 6: Number of training steps needed to converge to loss thresholds Tol [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

**Figure 7.** Figure 7: Validating the solution of the constrained PA case study from Section 4.3 across [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

**Figure 8.** Figure 8: Heatmaps of the absolute errors in the value function [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗

**Figure 9.** Figure 9: Value function and optimal controls of the example of Section 4.5. Panels (a)-(c): [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

A General Model for Continuous Time Principal-Agent Problem Under Hidden Action
q-fin.MF 2026-07 conditional novelty 6.0

A new sufficient condition solves a broad class of continuous-time hidden-action contracting problems with diffusion-controlled continuous pay, and an explicit example shows optimal pay-for-performance sensitivity is ...

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

Al-Aradi, A

[1]A. Al-Aradi, A. Correia, G. Jardim, D. de Freitas Naiff, and Y. Saporito, Extensions of the deep Galerkin method, Applied Mathematics and Computation, 430 (2022), p. 127287. [2]A. Al-Aradi, A. Correia, D. d. F. Naiff, G. Jardim, and Y. Saporito,Appli- cations of the deep Galerkin method to solving partial integro-differential and Hamilton- Jacobi-Bellm...

Pith/arXiv arXiv 2022
[22]

,Corporate insurance and managerial incentives, Journal of Economic Theory, 74 (1997), pp. 297–332. [23]J. Sung, J. Zhang, and Z. Zhu,A general model for continuous time principal-agent problem under hidden action, In Preparation, (2025). [24]N. Williams,A solvable continuous time dynamic principal–agent model, Journal of Economic Theory, 159 (2015), pp. ...

1997
[1364]

Sukumar and A

31 [20]N. Sukumar and A. Srivastava,Exact imposition of boundary conditions with dis- tance functions in physics-informed deep neural networks, Computer Methods in Ap- plied Mechanics and Engineering, 389 (2022), p. 114333. [21]J. Sung,Linearity with project selection and controllable diffusion rate in continuous- time principal-agent problems, The RAND J...

2022
[2012]

Dayanıklı and M

[10]G. Dayanıklı and M. Lauriere,A machine learning method for Stackelberg mean field games, Mathematics of Operations Research, 50 (2024), pp. 3055–3093. [11]J. Duan, J. Li, Q. Ge, S. E. Li, M. Bujarbaruah, F. Ma, and D. Zhang, Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems, IEEE Transactions on ...

2024
[3311]

Duarte, D

[12]V. Duarte, D. Duarte, and D. H. Silva,Machine learning for continuous-time finance, The Review of Financial Studies, 37 (2024), pp. 3217–3271. [13]B. Holmstrom and P. Milgrom,Aggregation and linearity in the provision of intertemporal incentives, Econometrica: Journal of the Econometric Society, (1987), pp. 303–328. [14]S. D. Jacka and A. Mijatovi ´c,...

Pith/arXiv arXiv 2024

[1] [1]

Al-Aradi, A

[1]A. Al-Aradi, A. Correia, G. Jardim, D. de Freitas Naiff, and Y. Saporito, Extensions of the deep Galerkin method, Applied Mathematics and Computation, 430 (2022), p. 127287. [2]A. Al-Aradi, A. Correia, D. d. F. Naiff, G. Jardim, and Y. Saporito,Appli- cations of the deep Galerkin method to solving partial integro-differential and Hamilton- Jacobi-Bellm...

Pith/arXiv arXiv 2022

[2] [22]

,Corporate insurance and managerial incentives, Journal of Economic Theory, 74 (1997), pp. 297–332. [23]J. Sung, J. Zhang, and Z. Zhu,A general model for continuous time principal-agent problem under hidden action, In Preparation, (2025). [24]N. Williams,A solvable continuous time dynamic principal–agent model, Journal of Economic Theory, 159 (2015), pp. ...

1997

[3] [1364]

Sukumar and A

31 [20]N. Sukumar and A. Srivastava,Exact imposition of boundary conditions with dis- tance functions in physics-informed deep neural networks, Computer Methods in Ap- plied Mechanics and Engineering, 389 (2022), p. 114333. [21]J. Sung,Linearity with project selection and controllable diffusion rate in continuous- time principal-agent problems, The RAND J...

2022

[4] [2012]

Dayanıklı and M

[10]G. Dayanıklı and M. Lauriere,A machine learning method for Stackelberg mean field games, Mathematics of Operations Research, 50 (2024), pp. 3055–3093. [11]J. Duan, J. Li, Q. Ge, S. E. Li, M. Bujarbaruah, F. Ma, and D. Zhang, Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems, IEEE Transactions on ...

2024

[5] [3311]

Duarte, D

[12]V. Duarte, D. Duarte, and D. H. Silva,Machine learning for continuous-time finance, The Review of Financial Studies, 37 (2024), pp. 3217–3271. [13]B. Holmstrom and P. Milgrom,Aggregation and linearity in the provision of intertemporal incentives, Econometrica: Journal of the Econometric Society, (1987), pp. 303–328. [14]S. D. Jacka and A. Mijatovi ´c,...

Pith/arXiv arXiv 2024