Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach

Xiang Yu; Xiaodong Luo; Zongxia Liang

REVIEW 3 major objections 4 minor 5 references

In the irreversible reinsurance model, entropy randomization of the singular control law yields a time-consistent equilibrium with explicit boundary Γ(x)=exp(−βΦ(x)/λ), and an actor-critic algorithm that inherits convergence from the inner

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:54 UTC pith:JMVWGRKF

load-bearing objection Fresh idea for randomizing singular control laws in RL, but the closed-form equilibrium in Theorem 4.5 is not verified—condition (4.5) of the paper's own verification theorem fails for the constructed f^{p,s}. the 3 major comments →

arxiv 2512.02769 v2 pith:JMVWGRKF submitted 2025-12-02 math.OC

Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach

Zongxia Liang , Xiaodong Luo , Xiang Yu This is my paper

classification math.OC MSC 93E2093B4749K45

keywords irreversible reinsurance problemsingular control lawentropy regularizationtime inconsistencytime-consistent equilibriumcontinuous-time reinforcement learningactor-critic algorithmtwo-stage optimal control

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that exploratory reinforcement learning can work for irreversible reinsurance decisions, where the insurer can only reduce risk exposure by buying reinsurance. It randomizes the decision rule itself: at each moment the agent either follows the current optimal singular control law or waits, with the waiting choice encoded by an auxiliary singular control and an entropy penalty. In the specific model, the entropy-regularized two-stage problem has a time-consistent equilibrium whose activation boundary is Γ(x)=exp(−βΦ(x)/λ), where Φ is the original no-exploration value function. This explicit structure lets the authors build an actor-critic algorithm in which only the inner singular control law is iterated, and their numerical experiments show the randomized version learns the true parameters and boundary faster than the non-randomized benchmark.

Core claim

The central claim is that adding an auxiliary singular control for the 'wait or activate' decision, together with entropy regularization, yields a well-posed exploratory version of an infinite-horizon irreversible reinsurance problem, and in the exponential-cost/linear-reinsurance model this randomized problem has a time-consistent equilibrium given explicitly. The inner optimal singular control law is 'push whenever the state exceeds a threshold x̂', with value function Φ(x). The outer equilibrium is the auxiliary activation-fraction process η̂_r = z + sup_{s∈[t,r]}(Γ(X_s)−z)^+ with boundary Γ(x)=exp(−βΦ(x)/λ), and the equilibrium value function is given in closed form in Theorem 4.5. The p

What carries the argument

The load-bearing object is the singular control law: a partition (W, W^c) of the state-time-control space that decides when the controller pushes (when the triple lies outside W) versus waits. It is used twice: an inner law Ξ for the reinsurance policy, and an auxiliary law Υ for the randomized activation decision. The key identity is the equilibrium boundary Γ(x)=exp(−βΦ(x)/λ), which turns the outer time-inconsistent problem into a pair of regions {z>Γ(x)} and makes the whole two-stage problem solvable by iterating only the inner threshold x̂. The entropy term E(z)=z−z ln z for z∈[0,1] supplies the exploration, and Theorem 4.5 provides the closed-form equilibrium.

Load-bearing premise

The whole convergence argument rests on an earlier result, not reproven here, that repeatedly improving the inner reinsurance strategy by a certain rule always reaches the best strategy; if that earlier result is wrong or does not survive the added randomization and entropy penalty, the paper's convergence guarantee collapses.

What would settle it

Take the explicit boundary Γ(x)=exp(−βΦ(x)/λ) from Theorem 4.5 with the stated parameter constraints, simulate the equilibrium activation process η̂, and check the two verification inequalities (Φ(x)+V_z≥0 above the boundary, and the HJB inequality below it) on a dense grid of (x,z); any violation would falsify the equilibrium characterization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

For the irreversible reinsurance model, the entropy-regularized exploratory problem has a time-consistent equilibrium with an explicit boundary, so no numerical scheme is needed to locate the equilibrium itself.
The vanishing-regularization property fails: the λ→0 limit of the equilibrium is not the original optimal solution, so the original value function must be recovered indirectly through a shared parameterization.
Because the outer equilibrium is a deterministic function of the inner value function, the randomized actor-critic algorithm has no higher computational complexity than the non-randomized one, while still exploring more.
The policy iteration converges to the equilibrium pair (Ξ̂,Υ̂), since the convergence of the inner policy iteration transfers to the outer boundary.
In the reported numerical setting, the randomized algorithm achieves lower L∞ error in the learned value function and faster parameter convergence than the non-randomized benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: this two-stage 'randomize the law, not the action' template could apply to other irreversible decisions such as lumpy investment, capacity expansion, or irreversible market entry, whenever the outer equilibrium boundary can be expressed as a function of the inner value function.
Editorial inference: the failure of vanishing regularization in this time-inconsistent setting warns that, in entropy-regularized RL, using the equilibrium policy with small λ may not approximate the precommitted optimum; a shared-parameterization workaround may be needed in other time-inconsistent problems too.
Editorial inference: if the inner policy-iteration convergence result extends beyond the explicit model, the same actor-critic construction could handle unknown drift and volatility in more general singular control problems, provided the outer equilibrium can still be represented in feedback form from the inner value function.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Fresh idea for randomizing singular control laws in RL, but the closed-form equilibrium in Theorem 4.5 is not verified—condition (4.5) of the paper's own verification theorem fails for the constructed f^{p,s}.

read the letter

Two things you should know. The paper introduces a genuinely new way to randomize singular control laws—by superimposing an auxiliary singular control that activates the learned inner law at a random time, with entropy regularization—and it gets an explicit candidate equilibrium boundary Γ(x)=e^{-βΦ(x)/λ}. That is a real idea, and the two-stage equilibrium formulation is new relative to the stopping-time randomization literature. The inner problem's closed-form solution and the observation that the vanishing-regularization limit selects the precommitted solution rather than the equilibrium are also well done.

The problem is the verification of the main theorem. Theorem 4.5's proof only checks conditions (4.15) and (4.16), which correspond to the variational inequality (4.4). It never checks (4.5), the interior extended HJB equation in W^Υ. That omission is fatal to the proof as written. Substituting their candidate f^{p,s} = −λ/β e^{-β(t−s)}E(z)+C_l e^{lx} and the explicit C_l into (4.5) does not give zero. The remainder is an integral whose integrand is strictly positive in the control region (e.g. at q=0 it contains e^{ap}+μΦ'+½σ²Φ'', positive for p>x̂). So the constructed (f^{p,s}, V, Υ̂) does not satisfy the verification theorem, and the existence claim for the equilibrium with the closed-form boundary is not established. Maybe the boundary is right and a corrected verification exists; but this paper doesn't provide it.

The numerical section is also thin: one unseeded run, no error bars, and the convergence of the policy iteration relies on the authors' prior preprint Liang et al. [2025] without re-proving it in the entropy-regularized, randomized setting. The parameterization also assumes the true value-function form, which helps the learning.

Who gets value from this: anyone working on exploration for singular control or on time-inconsistent control could learn from the formulation and the explicit inner problem, but the central result needs a fix before it can be cited. I would send it to a serious referee—it deserves engagement—but I'd want the referee to specifically redo the verification of (4.5).

Referee Report

3 major / 4 minor

Summary. The paper proposes a continuous-time reinforcement learning framework for an irreversible reinsurance problem with singular controls. It introduces a two-stage randomized control formulation: at each time the agent chooses a singular control law for the inner problem, and an auxiliary singular control randomizes the activation time of that law. Entropy regularization is added to encourage exploration. The main theoretical claim is a closed-form time-consistent equilibrium for the outer problem, with waiting region ŴΥ={z>Γ(x)} and Γ(x)=e^{-βΦ(x)/λ}, where Φ is the value function of the original non-randomized singular control problem. This explicit equilibrium is then used to design actor-critic algorithms with a shared parameterization, and numerical experiments compare the randomized algorithm against a non-randomized benchmark. The paper's central mathematical result is the characterization in Theorem 4.5; the numerical study is presented as evidence that the randomized exploration improves convergence.

Significance. If the equilibrium characterization were correct, the paper would make a useful contribution to continuous-time RL for singular control: it provides an explicit two-stage randomized policy, a parameterization that exploits the solution structure, and martingale-based policy evaluation. The authors also give a concrete application to reinsurance, and the algorithmic details are quite explicit and reproducible in principle. However, the central verification is flawed: the candidate pair (f^{p,s}, V, Υ̂) does not satisfy the extended HJB equation (4.5) on which Theorem 4.3 rests. The paper therefore does not currently establish the claimed equilibrium or the stated value function. The significance is conditional on a successful repair of this verification and on a careful reconsideration of the convergence argument that is currently delegated to an unpublished preprint.

major comments (3)

[§4, derivation of Γ and C_l] The candidate f^{p,s} does not satisfy the extended HJB equation (4.5). For z>Γ(x), f^{p,s}=−(λ/β)e^{−β(t−s)}E(z)+C_l(p,t−s,z)e^{lx}. Since l solves ½σ²l²+μl=β, direct substitution gives (Af^{p,s})−λe^{−β(t−s)}E(z)=[C_{l,q}+βC_l]e^{lx}. But C_{l,q}+βC_l=∫_z^1 [Ψ_q(p,q)+βΨ(p,q)] e^{−lΓ^{−1}(z′)}dz′, and Ψ_q+βΨ=AΨ+e^{ap} by the PDE for Ψ. At q=0 this equals AΦ+e^{ap}; on p>x̂ this is μc+e^{ap}>0. Hence the integrand is strictly positive and (4.5) fails, for example at (p,s)=(x,t) with p>x̂ and z>Γ(x). The proof of Theorem 4.5 verifies only (4.15)–(4.16), and (4.16) is checked only for z≤Γ(x); the waiting-region PDE condition is never verified. Consequently Theorem 4.3 cannot be applied, and the stated V in (4.14) is not established as the equilibrium value function under Υ̂. A direct verification of Definition 4.2 would require a global drift inequality for all z, which is not supplied.
[§5.2] The construction of the equilibrium relies on several unproved assumptions: the threshold ansatz W={z>Γ(x,t)}, the reduction Γ(x,t)=Γ(x), and especially the claim 'We conjecture that either C_b≡0 or C_l≡0.' These assumptions are not derived from (4.10)–(4.13); they are imposed to obtain the explicit formula Γ=e^{−βΦ/λ}. Even after imposing them, the resulting candidate fails the verification condition (4.5), as shown above. The paper therefore does not provide a rigorous characterization of the equilibrium, only a candidate whose verification is incomplete.
[§6] The convergence guarantee for the actor-critic algorithm is imported from the authors' preprint Liang et al. [2025]: 'Our previous paper Liang et al. [2025] has shown the convergence of Ξ^n towards Ξ̂.' That result is not proved in the present manuscript, and no argument is given that it extends to the entropy-regularized, randomized environment; the proof in Section 5.2 simply composes that convergence with the boundary map e^{−βΦ/λ}. Since the RL convergence claim is one of the paper's stated contributions, this is a load-bearing gap. In addition, Section 6 reports a single simulation run; the claimed 'superior convergence' of Algorithm 5 is not supported by error bars or multiple independent seeds.

minor comments (4)

[§4, Eq. (4.7)] Theorem 4.3 defines ŴΥ as {U+V_z=0}, while Definition 4.1 and the rest of the paper treat ŴΥ as the waiting region where the control is not activated and Φ+V_z>0. This notational inconsistency should be fixed.
[§6] The probabilistic representation (4.7) and the conditions (4.5)–(4.6) use the parameter family f^{p,s} without fully specifying its domain and differentiability with respect to p and s. The notation should be clarified.
[§4, after Remark 5] The captions of Figures 1 and 2 appear garbled in the manuscript text, and some textual artifacts remain around the figures. These should be repaired.
[§4, after Remark 5] The passage beginning 'when λ=0, the entropy regularized outer problem...' is not cleanly integrated into the narrative and seems to be an unnumbered remark. It should be formatted as a formal remark or moved into the main text.

Circularity Check

1 steps flagged

The closed-form equilibrium is derived self-containedly, but the convergence guarantee for the RL algorithm is imported from the authors' own prior preprint.

specific steps

self citation load bearing [Section 5.2, Policy iteration; paragraph after eq. (5.2)]
"Our previous paper Liang et al. [2025] has shown the convergence of Ξ n towards ˆΞ. Hence Υ n converges to ˜Υ withW ˜Υ :={(x, z)|z > e− β λ Φ(x;ˆΞ)}. Then ˜Υ = ˆΥ and we can conclude that the policy iteration converges to the equilibrium pair ( ˆΞ, ˆΥ)."

The paper's stated convergence result for the overall actor-critic scheme is not proved here; it is obtained by combining the closed-form feedback formula Υ↦{z>e^{-βΦ/λ}} with a convergence theorem taken from the same authors' preprint Liang et al. [2025]. The 'Hence Υ^n converges' step is pure deduction from that external citation, so the load-bearing convergence guarantee reduces to a self-citation rather than to an argument in this paper. The analytic equilibrium construction in Theorem 4.5 is independent, which is why this is partial rather than total circularity.

full rationale

The core analytic chain is self-contained: Section 3 solves the inner singular control problem explicitly, and Section 4 constructs f^{p,s}, V and the boundary Γ(x)=e^{-βΦ(x)/λ} from the verification theorem's equations, then checks the relevant inequalities (4.15)-(4.16); no fitted quantity is renamed as a prediction. The algorithm's parameterization uses the already-derived value-function structure, and the numerical experiments are benchmarked against known true parameters, which is external support rather than circularity. The one significant dependence on the authors' own prior work is in Section 5.2, where convergence of the inner policy iteration Ξ^n→Ξ̂ is taken verbatim from Liang et al. [2025] and the outer-policy convergence is then read off via the explicit map Υ↦{z>e^{-βΦ/λ}}. This makes the RL convergence guarantee load-bearing on a same-author citation, while leaving the central equilibrium existence theorem unaffected. A separate concern is that the proof of Theorem 4.5 only checks (4.15)-(4.16) and does not explicitly verify condition (4.5) of Theorem 4.3; however, this is a correctness/verification gap rather than a circular reduction, so it does not affect the circularity score. Score 4 reflects partial self-citation load-bearing, not definitional circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The theoretical equilibrium derivation rests on standard stochastic analysis and a few structural ansatze. The RL part additionally relies on a self-cited prior convergence result and a parametric value-function class.

free parameters (3)

θ=(θ1, θ2, θ3) = θ1≈0.1, θ2≈0.262, θ3≈14.286 (true values in simulation)
Parameters of the value-function parameterization learned by martingale loss in Section 6; numerical convergence of these to true values is the paper's central empirical claim.
λ (temperature) = 0.5
Exploration temperature chosen by hand in numerical experiment (Section 6); the equilibrium boundary Γ=e^{-βΦ/λ} depends on it.
x̄ initial guess, learning rates, clipping bounds = x̄=-2.5, α=[0.1,0.1,1], α_pi=0.5, etc.
Hyperparameters for the RL algorithm listed in Section 6; not theoretically derived.

axioms (6)

standard math Itô-Tanaka-Meyer formula and Itô's formula for semimartingales with jumps are valid for the control processes considered.
Used in proof of Theorems 4.3 and 5.1/5.2.
standard math The Skorokhod reflection problem for singular controls has a unique strong solution.
Used throughout for controlled state and auxiliary control.
domain assumption Discount factor β is sufficiently large: β > μa + 0.5σ²a², ensuring finite cost and b>a.
Stated in Section 2; guarantees the explicit value function Φ in (3.4) is finite and well-posed.
ad hoc to paper The outer waiting region has the threshold form W^Υ̂ = {(x,t,z) | z > Γ(x,t)}.
Conjected in Section 4 before solving; verified by construction in Theorem 4.5 but not derived from a general principle.
ad hoc to paper In solving the HJB system, C_b≡0 or C_l≡0 (one of the two exponential modes vanishes).
Conjecture in Section 4 leading to Γ=e^{-βΦ/λ}, later supported by boundary conditions.
domain assumption The parameterization (3.4) and formulas for μ(θ), σ(θ) reproduce the true model class.
Section 6 assumes the agent's function class contains the true value functions; without this, learning would be biased.

invented entities (1)

Auxiliary singular control η (and activation time τ^η) no independent evidence
purpose: Models randomized choice between waiting and acting according to the current optimal singular control law.
A mathematical construct introduced to formalize exploration; no external falsifiable handle beyond the model.

pith-pipeline@v1.3.0-alltime-deepseek · 25586 in / 14178 out tokens · 115968 ms · 2026-08-03T18:54:27.137322+00:00 · methodology

0 comments

read the original abstract

This paper studies the continuous-time reinforcement learning for stochastic singular control with the application to an infinite-horizon irreversible reinsurance problem. The singular control is equivalently characterized as a pair of regions of time and the augmented states, called the singular control law. To encourage the exploration in the learning procedure, we propose a randomization method by considering an auxiliary singular control and entropy regularization. The exploratory singular control problem is formulated as a two-stage optimal control problem, in which the time-inconsistency issue arises in the outer problem. Existence of equilibrium singular control law for the time-inconsistent outer problem is rigorously established. Taking advantage of the solution structure, we utilize a proper parameterization and neural networks to devise the actor-critic reinforcement learning algorithm. In the numerical experiment, we show the superior convergence of parameter iterations based on the randomized equilibrium policy and illustrate how the exploration may advance the learning performance.

Figures

Figures reproduced from arXiv: 2512.02769 by Xiang Yu, Xiaodong Luo, Zongxia Liang.

**Figure 1.** Figure 1: Comparison of learned parameters θi , i = 1, 2, 3 of non-randomized Algorithm 4 and randomized Algorithm 5 versus the true parameters during episodes of training. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_1.png] view at source ↗

**Figure 2.** Figure 2: Left panel: comparison of learned optimal boundary ¯x [PITH_FULL_IMAGE:figures/full_fig_p027_2.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith

[2]

M. Dai, Y. Dong, and Y. Jia. Learning equilibrium mean-variance strategy.Mathematical Finance, 33(4): 1166–1212, 2023a. M. Dai, Y. Dong, Y. Jia, and X. Y. Zhou. Learning merton’s strategies in an incomplete market: Recursive entropy regularization and biased gaussian exploration.Preprint, available at arXiv:2312.11797, 2023b. 27 M. Dai, Y. Jia, and H. Jin...

arXiv
[4]

Liang, X

Z. Liang, X. Luo, and X. Yu. A reinforcement learning framework for some singular stochastic control problems.Preprint, available at https://arxiv.org/abs/2506.22203,

Pith/arXiv arXiv
[5]

X. Wei, X. Yu, and F. Yuan. Unified continuous-time q-learning for mean-field game and mean-field control problems.Preprint, available at https://arxiv.org/abs/2407.04521,

Pith/arXiv arXiv
[2024]

Dianetti, R

J. Dianetti, R. Dumitrescu, G. Ferrari, and R. Xu. Entropy regularization in mean-field games of optimal stopping.Preprint, available at https://arxiv.org/abs/2509.18821,

arXiv
[2025]

J. Cao, D. Li, V. R. Young, and B. Zou. Equilibrium strategies for singular dividend control problems under the mean-variance criterion.Preprint, available at https://arxiv.org/abs/2511.08433,

arXiv

[1] [2]

M. Dai, Y. Dong, and Y. Jia. Learning equilibrium mean-variance strategy.Mathematical Finance, 33(4): 1166–1212, 2023a. M. Dai, Y. Dong, Y. Jia, and X. Y. Zhou. Learning merton’s strategies in an incomplete market: Recursive entropy regularization and biased gaussian exploration.Preprint, available at arXiv:2312.11797, 2023b. 27 M. Dai, Y. Jia, and H. Jin...

arXiv

[2] [4]

Liang, X

Z. Liang, X. Luo, and X. Yu. A reinforcement learning framework for some singular stochastic control problems.Preprint, available at https://arxiv.org/abs/2506.22203,

Pith/arXiv arXiv

[3] [5]

X. Wei, X. Yu, and F. Yuan. Unified continuous-time q-learning for mean-field game and mean-field control problems.Preprint, available at https://arxiv.org/abs/2407.04521,

Pith/arXiv arXiv

[4] [2024]

Dianetti, R

J. Dianetti, R. Dumitrescu, G. Ferrari, and R. Xu. Entropy regularization in mean-field games of optimal stopping.Preprint, available at https://arxiv.org/abs/2509.18821,

arXiv

[5] [2025]

J. Cao, D. Li, V. R. Young, and B. Zou. Equilibrium strategies for singular dividend control problems under the mean-variance criterion.Preprint, available at https://arxiv.org/abs/2511.08433,

arXiv