Pith. sign in

REVIEW 3 major objections 4 minor 47 references

Convergence of regularized agent-state-based Q-learning in POMDPs

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Agent-state regularized Q-learning provably converges in POMDPs

desk verdict Plausible and likely repairable, but the proof as written has two real gaps: a one-sided bound in X2 and a phase-index error in the periodic extension. read the letter →

arxiv 2508.21314 v2 pith:4ALCDYUV submitted 2025-08-29 cs.LG

Amit Sinha , Matthieu Geist , Aditya Mahajan This is my paper
classification cs.LG
keywords POMDPQ-learningagentstatepolicyregularizationentropystochasticapproximationperiodicpoliciesconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Regularized agent-state-based Q-learning (RASQL) is the common practical recipe of combining a recurrent or frame-stacked "agent state" with an entropy or KL regularizer in partially observable environments. This paper proves that, under standard stochastic-approximation and exploration assumptions, the RASQL update converges almost surely to a well-defined limit. The limit is not the optimal POMDP value function; it is the unique fixed point of a regularized Bellman operator built from a fictitious MDP whose rewards and transitions are the true model averaged under the stationary distribution of the fixed exploratory policy. The paper gives the same treatment for a periodic-policy variant (RePASQL), and small numerical experiments match the predicted limits.

What carries the argument

The load-bearing object is the averaged-MDP construction: from the limiting distribution of the behavior policy, define average rewards r_mu(z,a) and transitions P_mu(z'|z,a), then build the regularized Bellman operator whose unique fixed point Q_mu is the announced limit. The proof decomposes the error Q_t - Q_mu into three components: reward-averaging noise, transition-averaging noise, and the residual from using the current Q-table inside the regularized target. A contraction-type comparison with factor gamma(1+1/C)<1 drives the residual component to zero, while the averaging components vanish by stochastic-approximation results and the law of large numbers for the underlying Markov chain

What would settle it

Run tabular RASQL on a small finite POMDP where the behavior policy has full support on (z,a), and compare the long-run Q-table to the fixed point Q_mu of (4) computed by value iteration; a mismatch beyond Monte Carlo error would falsify Theorem 1. Equivalently, using a behavior policy whose stationary distribution has zero mass on some reachable (z,a) should make the corresponding entries fail to settle, confirming that the full-support assumption is load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is Theorem 1: for any POMDP, agent-state update rule, and strongly convex policy regularizer, if the behavior policy makes the joint process (state, observation, agent state, action) converge to a limiting distribution with full support over every (z,a), then the Q-table update converges almost surely to Q_mu, the unique fixed point of the regularized Bellman operator (4). Q_mu depends on that limiting distribution: rewards are averaged as r_mu(z,a) = E[r(S,a)|z] and transitions as P_mu(z'|z,a) = P(Z_{t+1}=z' | z,a) under the stationary process. Theorem 2 extends the same convergence result to periodic policies with period L, with limits Q^ell_

Load-bearing premise

The central assumption is that the fixed behavior policy makes the joint process of state, observation, agent state, and action converge to a limiting distribution that visits every (z,a) pair infinitely often; for arbitrary agent-state updates such as recurrent networks this is not guaranteed and can fail.

Editorial extensions

If this is right

  • If Theorem 1 is right, entropy- or KL-regularized Q-learning over agent states can be run in any POMDP satisfying the assumptions and will converge almost surely instead of oscillating.
  • The converged Q-table, and the stochastic policy it induces, are explicitly computable from the model plus the behavior policy, so the limit is testable in simulation.
  • The limit generally is not the optimal agent-state policy; it inherits the exploration policy's stationary distribution, so changing exploration changes the learned value function.
  • Regularization yields a stochastic greedy policy at the limit, which can beat deterministic stationary policies in partially observed control, extending a known benefit of agent-state Q-learning.
  • The periodic variant RePASQL converges to a periodic fixed point, covering policies that change from step to step, with the period-1 case recovering the stationary result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural reading is that the result exposes an off-policy bias: in regularized agent-state Q-learning, the behavior policy is not just a data-collection device but part of the objective, since the limit depends on its stationary distribution.
  • The full-support assumption is genuinely load-bearing; for recurrent or frame-stacking agent-state updates, reachability of every (z,a) is not automatic, so exploration design should target the induced occupation distribution, not merely coverage.
  • The same averaging construction plausibly carries over to continuous agent states or function approximation whenever empirical occupation converges, though the paper does not pursue that extension.
  • The periodic-policy variant suggests that scheduling exploration as a periodic policy could be used deliberately to shape the limiting Q-table and the final regularized policy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies regularized agent-state-based Q-learning (RASQL) in POMDPs, where the Q-table is indexed by a model-free agent state rather than a belief state, and the policy is entropy- or KL-regularized. The main result, Theorem 1, states that under a learning-rate condition (Assumption 1) and a stationarity/full-support condition on the behavior policy (Assumption 2), the RASQL iteration (2) converges almost surely to Q_μ, the unique fixed point of the regularized Bellman operator (4) built from the behavior policy's stationary distribution. Theorem 2 gives an analogous result for a periodic variant, RePASQL. The numerical section illustrates convergence on a small POMDP with the agent state equal to the observation. The paper also correctly notes that the limit depends on the exploration policy and is not guaranteed to be optimal.

Significance. If the result holds, it is a useful contribution: it extends the known convergence theory of agent-state-based Q-learning to entropy/KL-regularized iterations, which are widely used in practice but lack theoretical guarantees in POMDPs. The characterization of the limit via the stationary distribution of the behavior policy is explicit and falsifiable, and the periodic extension is natural. The numerical example is limited but supports the claimed qualitative behavior. However, the proof as written contains load-bearing gaps in the treatment of the regularized error component and in the periodic phase coupling, so the advertised convergence is not established by the current manuscript.

major comments (3)
  1. [Appendix A, part c (Eq. (11c))] The proof of convergence of X^2_t establishes only a one-sided upper bound: U^2_t(z,a) ≤ κ∥X^2_t∥. From this the manuscript concludes X^2_{t+1}(z,a) < ∥X^2_t∥ and hence ∥X^2_{t+1}∥ < ∥X^2_t∥. This is not valid: for a negative component X^2_t(z,a), the displayed inequality is trivial and does not control |X^2_{t+1}(z,a)|. A large negative U^2_t(z,a) can make the sup norm increase. The later step that (11c) implies ∥U^2_t∥ < κB(0) also requires a two-sided bound. The missing two-sided bound follows from the 1-Lipschitz property of Ω* for entropy/KL regularization, but this is neither stated nor used. Thus the central X^2 convergence argument is incomplete as written.
  2. [Appendix B, part c] In the periodic proof, the display reads γ∥Q^{Jℓ+1K}_t − Q^{Jℓ+1K}_μ∥ = γ∥∆^ℓ_t∥. This is a phase-index error: the left-hand side involves the phase-(ℓ+1) Q-function, while ∆^ℓ_t is the phase-ℓ error. The equality does not hold in general, and it is load-bearing because it is used to connect the regularization error to the current phase's contraction argument. The proof can likely be repaired by working with the sup norm over all phases or by explicitly tracking the phase indices, but the current argument is invalid.
  3. [Appendix A, parts a-b; Appendix B, parts a-b] The convergence of the X^0_t and X^1_t components is asserted by reference to external results ([12], [41], and 'proposition 4 from [12]') rather than proved. The hypotheses of those theorems are not stated, and the verification that they hold here is only partial; in particular, the measurability and step-size conditions for the periodic Markov chain result are not checked in the text. Since these components are essential to both Theorem 1 and Theorem 2, the proof is incomplete at this juncture.
minor comments (4)
  1. [Section IV, Assumption 2] The phrase 'mild technical conditions' overstates the content of Assumption 2. For a general agent-state update φ and an arbitrary exploration policy μ, existence of a limiting distribution with positive mass on every (z,a) is not automatic. The theorem is conditional on Assumption 2, but the paper does not discuss conditions on φ, μ, or the POMDP under which Assumption 2 holds; this limits the advertised scope.
  2. [Appendix A, part b and Appendix B, part b] Notation: 'Fix (z◦, a◦) ∈ ×Z × A' should read '∈ Z × A'. The extra multiplication symbol appears in both appendices.
  3. [Appendix A, part c] The equation labels in the chain ending in (11c) contain a duplicated '(e)' label; the labels should be renumbered for clarity.
  4. [Section VI] The numerical experiments are a useful sanity check but report only median and quartiles for a single POMDP with z_t = y_t. Since the theory covers general agent-state updates, an example with a nontrivial φ (e.g., finite-memory or recurrent update) would strengthen the empirical claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence limit is defined as the fixed point of the mean-drift operator and then proved via stochastic approximation; self-citations are to prior independent results, not to the target claim.

full rationale

The paper defines Q_μ as the unique fixed point of the averaged regularized Bellman operator (4) built from the limiting distribution ζ_μ of the behavior policy, and then proves (Theorem 1, Appendix A) that the RASQL iterate (2) converges to Q_μ. This is not a fit or a definitional identity: Q_μ is computed from the model and ζ_μ, and the proof uses the standard stochastic approximation theorem (Thm. 2.7 of [41]) plus a direct cancellation of the bias term X1 using the definitions of r_μ and P_μ. The regularization-specific component X2 is handled with a new argument rather than being assumed. The periodic case similarly imports Proposition 4 from [12] for the unregularized periodic components and treats the regularized component separately; [12] concerns unregularized periodic Q-learning, not the current regularized claim, so the self-citation is not load-bearing in a circular sense. The paper also explicitly acknowledges (Conclusions) that convergence to an optimal agent-state policy is not guaranteed, which is a limitation but not circularity. No equation is reduced to its own input: (8) and (13) are error recursions whose target Q_μ is defined independently as the Bellman fixed point (4)/(7).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the stochastic approximation assumptions, the existence of the stationary limiting distribution of the behavior policy, and standard convex-analysis properties of the regularizer. The finiteness of the agent-state space is assumed implicitly but never stated, which is a notable gap.

assumptions (4)
  • domain assumption Assumption 1: step-size sequences satisfy the usual stochastic approximation conditions and are measurable w.r.t. the history.
    Needed to apply the stochastic approximation theorems in [41].
  • domain assumption Assumption 2: the interaction Markov chain converges to a limiting distribution with positive probability on every (z,a) pair.
    Defines the averaged MDP (3)-(4) and is used to show the noise terms X0 and X1 vanish.
  • standard math The policy regularizer Ω is twice differentiable and strongly convex, and its conjugate Ω* is Lipschitz (Lemma 1).
    This is a convex-analysis background condition inherited from regularized MDP theory.
  • domain assumption The agent state space Z is finite (implicit in Definition 2).
    The Markov chain and sums over Z require finite Z; the paper never states this explicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence of regularized agent-state-based Q-learning in POMDPs." pith.science (2026). https://pith.science/paper/4ALCDYUV

@misc{pith2026250821314,
  author       = {Pith},
  title        = {Pith review of: Convergence of regularized agent-state-based Q-learning in POMDPs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ALCDYUV}},
  note         = {Machine review of arXiv:2508.21314}
}
read the original abstract

In this paper, we present a framework to understand the convergence of commonly used Q-learning reinforcement learning algorithms in practice. Two salient features of such algorithms are: (i)~the Q-table is recursively updated using an agent state (such as the state of a recurrent neural network) which is not a belief state or an information state and (ii)~policy regularization is often used to encourage exploration and stabilize the learning algorithm. We investigate the simplest form of such Q-learning algorithms which we call regularized agent-state-based Q-learning (RASQL) and show that it converges under mild technical conditions to the fixed point of an appropriately defined regularized MDP, which depends on the stationary distribution induced by the behavioral policy. We also show that a similar analysis continues to work for a variant of RASQL that learns periodic policies. We present numerical examples to illustrate that the empirical convergence behavior matches with the proposed theoretical limit.

Figures

Figures reproduced from arXiv: 2508.21314 by the authors.

Figure 1
Figure 1. RASQL convergence: Q-values vs. number of iter [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. RePASQL convergence: Q-values vs. number of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages

  1. [12]

    Periodic agent-state based Q- learning for POMDPs,

    A. Sinha, M. Geist, and A. Mahajan, “Periodic agent-state based Q- learning for POMDPs,” Adv. Neural Inf. Process. Syst. , 2024

    work page 2024

  2. [41]

    Gradient-based algorithms for zeroth- order optimization,

    L. Prashanth, S. Bhatnagar et al., “Gradient-based algorithms for zeroth- order optimization,” Foundations and Trends® in Optimization, vol. 8, no. 1–3, pp. 1–332, 2025. APPENDIX A. Proof of Theorem 1 The proof argument for Theorem 1 is similar to the proof argument given in [10]–[13]. Define an error function between the converged value and the Q-learnin...

    work page 2025

  3. [1]

    Optimal control of Markov processes with incomplete state information I,

    K. J. Åström, “Optimal control of Markov processes with incomplete state information I,” Journal of Mathematical Analysis and Applications, vol. 10, pp. 174–205, 1965

    work page 1965

  4. [2]

    The optimal control of partially observable Markov processes over a finite horizon,

    R. D. Smallwood and E. J. Sondik, “The optimal control of partially observable Markov processes over a finite horizon,” Operations Research, vol. 21, no. 5, pp. 1071–1088, 1973

    work page 1973

  5. [3]

    Approximate information state for approximate planning and reinforcement learning in partially observed systems,

    J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan, “Approximate information state for approximate planning and reinforcement learning in partially observed systems,” J. Mach. Learn. Res. , vol. 23, no. 12, pp. 1–83, 2022

    work page 2022

  6. [4]

    Deep recurrent Q-learning for partially observable MDPs

    M. J. Hausknecht and P. Stone, “Deep recurrent Q-learning for partially observable MDPs.” in AAAI Fall Symposia, vol. 45, 2015, p. 141

    work page 2015

  7. [5]

    Deep variational reinforcement learning for POMDPs,

    M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson, “Deep variational reinforcement learning for POMDPs,” in Int. Conf. Mach. Learn. PMLR, 2018, pp. 2117–2126

    work page 2018

  8. [6]

    On improving deep reinforcement learning for POMDPs,

    P. Zhu, X. Li, P. Poupart, and G. Miao, “On improving deep reinforcement learning for POMDPs,” arXiv:1704.07978, 2017

    arXiv 2017

Show all 47 references
  1. [7]

    Memory-based deep reinforcement learning for POMDPs,

    L. Meng, R. Gorbet, and D. Kuli ´c, “Memory-based deep reinforcement learning for POMDPs,” in Int. Conf. Intell. Robots Syst. IEEE, 2021, pp. 5619–5626

    2021

  2. [8]

    Simple agent, complex envi- ronment: Efficient reinforcement learning with agent states,

    S. Dong, B. Van Roy, and Z. Zhou, “Simple agent, complex envi- ronment: Efficient reinforcement learning with agent states,” J. Mach. Learn. Res., vol. 23, no. 255, pp. 1–54, 2022

    2022

  3. [9]

    Agent-state based policies in POMDPs: Beyond belief-state MDPs,

    A. Sinha and A. Mahajan, “Agent-state based policies in POMDPs: Beyond belief-state MDPs,” Conference on Decision and Control, 2024

    2024

  4. [10]

    Reinforcement learning algorithm for partially observable Markov decision problems,

    T. Jaakkola, S. Singh, and M. Jordan, “Reinforcement learning algorithm for partially observable Markov decision problems,” in Adv. Neural Inf. Process. Syst. , vol. 7. MIT Press, 1994, pp. 345–352

    1994

  5. [11]

    Convergence of finite memory Q learning for POMDPs and near optimality of learned policies under filter stability,

    A. D. Kara and S. Yüksel, “Convergence of finite memory Q learning for POMDPs and near optimality of learned policies under filter stability,” Mathematics of Operations Research , Nov. 2022

    2022

  6. [13]

    Q-learning for stochastic control under general information structures and non-markovian environments,

    A. Devran Kera and S. Yüksel, “Q-learning for stochastic control under general information structures and non-markovian environments,” Transactions on Machine Learning Research , 2024

    2024

  7. [14]

    Reinforcement learning in non-Markovian environments,

    S. Chandak, P. Shah, V . S. Borkar, and P. Dodhia, “Reinforcement learning in non-Markovian environments,” Systems & Control Letters , vol. 185, p. 105751, 2024

    2024

  8. [15]

    On actor-critic algorithms,

    V . R. Konda and J. N. Tsitsiklis, “On actor-critic algorithms,” SIAM J. Control Optim., vol. 42, no. 4, pp. 1143–1166, 2003

    2003

  9. [16]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017

    2017 arXiv

  10. [17]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in Int. Conf. Mach. Learn. , 2018, pp. 1861–1870

    2018

  11. [18]

    Under- standing the impact of entropy on policy optimization,

    Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans, “Under- standing the impact of entropy on policy optimization,” in Int. Conf. Mach. Learn. PMLR, 2019, pp. 151–160

    2019

  12. [19]

    Deep reinforcement learning that matters,

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in AAAI Conference on Artificial Intelligence, 2018

    2018

  13. [20]

    Relative entropy policy search,

    J. Peters, K. Mulling, and Y . Altun, “Relative entropy policy search,” in AAAI Conference on Artificial Intelligence , 2010, pp. 1607–1612

    2010

  14. [21]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in Int. Conf. Mach. Learn. PMLR, 2015, pp. 1889–1897

    2015

  15. [22]

    A unified view of entropy- regularized Markov decision processes,

    G. Neu, A. Jonsson, and V . Gómez, “A unified view of entropy- regularized Markov decision processes,” Adv. Neural Inf. Process. Syst., 2017

    2017

  16. [23]

    A theory of regularized Markov decision processes,

    M. Geist, B. Scherrer, and O. Pietquin, “A theory of regularized Markov decision processes,” in Int. Conf. Mach. Learn. PMLR, 2019

    2019

  17. [24]

    Learning latent dynamics for planning from pixels,

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in Int. Conf. Mach. Learn. PMLR, 2019, pp. 2555–2565

    2019

  18. [25]

    Solar: Deep structured representations for model-based reinforcement learning,

    M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine, “Solar: Deep structured representations for model-based reinforcement learning,” in Int. Conf. Mach. Learn. PMLR, 2019, pp. 7444–7453

    2019

  19. [26]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” Int. Conf. Learn. Represent., 2020

    2020

  20. [27]

    Bridging state and history representations: Understanding self-predictive RL,

    T. Ni, B. Eysenbach, E. Seyedsalehi, M. Ma, C. Gehring, A. Ma- hajan, and P.-L. Bacon, “Bridging state and history representations: Understanding self-predictive RL,” Int. Conf. Learn. Represent. , 2024

    2024

  21. [28]

    Entropy-regularized point-based value iteration,

    H. Delecki, M. Vazquez-Chanlatte, E. Yel, K. Wray, T. Arnon, S. Witwicki, and M. J. Kochenderfer, “Entropy-regularized point-based value iteration,” arXiv:2402.09388, 2024

    2024 arXiv

  22. [29]

    DESPOT: Online POMDP planning with regularization,

    A. Somani, N. Ye, D. Hsu, and W. S. Lee, “DESPOT: Online POMDP planning with regularization,” Adv. Neural Inf. Process. Syst. , 2013

    2013

  23. [30]

    Smoother entropy for active state trajectory estimation and obfuscation in POMDPs,

    T. L. Molloy and G. N. Nair, “Smoother entropy for active state trajectory estimation and obfuscation in POMDPs,” IEEE Transactions on Automatic Control , vol. 68, no. 6, pp. 3557–3572, 2023

    2023

  24. [31]

    The limits of pure exploration in POMDPs: When the observation entropy is enough,

    R. Zamboni, D. Cirino, M. Restelli, and M. Mutti, “The limits of pure exploration in POMDPs: When the observation entropy is enough,” Reinforcement Learning Journal , 2024

    2024

  25. [32]

    R. T. Rockafellar and R. J.-B. Wets, Variational analysis. Springer Science & Business Media, 2009, vol. 317

    2009

  26. [33]

    Hiriart-Urruty and C

    J.-B. Hiriart-Urruty and C. Lemaréchal, Fundamentals of convex analysis. Springer Science & Business Media, 2004

    2004

  27. [34]

    Differentiable dynamic programming for structured prediction and attention,

    A. Mensch and M. Blondel, “Differentiable dynamic programming for structured prediction and attention,” in Int. Conf. Mach. Learn. PMLR, 2018, pp. 3462–3471

    2018

  28. [35]

    Sequential decomposition of sequential dynamic teams: applications to real-time communication and networked control sys- tems,

    A. Mahajan, “Sequential decomposition of sequential dynamic teams: applications to real-time communication and networked control sys- tems,” Ph.D. dissertation, U. Michigan, Ann Arbor, MI, 2008

    2008

  29. [36]

    Reinforcement learning with deep energy-based policies,

    T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in Int. Conf. Mach. Learn. PMLR, 2017, pp. 1352–1361

    2017

  30. [37]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,” The Annals of Mathematical Statistics , pp. 400–407, 1951

    1951

  31. [38]

    Q-learning,

    C. J. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, pp. 279–292, 1992

    1992

  32. [39]

    Asynchronous stochastic approximation and Q- learning,

    J. N. Tsitsiklis, “Asynchronous stochastic approximation and Q- learning,” Machine Learning, vol. 16, pp. 185–202, 1994

    1994

  33. [40]

    Learning without state- estimation in partially observable Markovian decision processes,

    S. P. Singh, T. Jaakkola, and M. I. Jordan, “Learning without state- estimation in partially observable Markovian decision processes,” in Machine Learning Proceedings 1994 . Elsevier, 1994, pp. 284–292

    1994

  34. [42]

    X a∈A πt(a | Zt+1)Qt(Zt+1, a) − Ω(πt(· |Zt+1))− X a∈A π⋆(a | Zt+1)Qµ(Zt+1, a) + Ω(π⋆(· |Zt+1)) # (a) ≤ γ

    but requires some additional considerations due to the regularization term. We start by defining: πt(· |z) = arg max ξ∈∆(A) X a∈A ξ(a)Qt(z, a) − Ω(ξ) π⋆(· |z) = arg max ξ∈∆(A) X a∈A ξ(a)Qµ(z, a) − Ω(ξ). In the previous steps, we have shown that ∥X i t ∥ →0 a.s., for i ∈ {0, 1}...

  35. [43]

    ∥X 2 t ∥ always remains above Cϵ; or

  36. [44]

    future states

    it goes below Cϵ at some stage. We consider these two possibilities separately. Possibility (i): ∥X 2 t ∥ always remains above Cϵ We will now prove that ∥X 2 t ∥ cannot remain above Cϵ forever. The proof is by contradiction. Suppose ∥X 2 t ∥ remains above Cϵ forever. As argued...

  37. [45]

    X a∈A πℓ t (a | Zt+1)QJℓ+1K t (Zt+1, a) − Ω(πℓ t (· |Zt+1))− X a∈A πℓ,⋆(a | Zt+1)QJℓ+1K µ (Zt+1, a) + Ω(πℓ,⋆(· |Zt+1)) # (a) ≤ γ

    but requires some additional considerations due to the regularization term. We start by defining: πℓ t (· |z) = arg max ξ∈∆(A) X a∈A ξ(a)QJℓ+1K t (z, a) − Ω(ξ) πℓ,⋆(· |z) = arg max ξ∈∆(A) X a∈A ξ(a)QJℓ+1K µ (z, a) − Ω(ξ). In the previous steps, we have shown that ∥X ℓ,i t ∥ →0...

  38. [46]

    ∥X ℓ,2 t ∥ always remains above Cϵ; or

  39. [47]

    These two cases must be considered separately

    it goes below Cϵ at some stage. These two cases must be considered separately. The proof follows exactly the same steps in the proof of theorem 1 given in appendix A, which finally gives us: lim t→∞ ∥X ℓ,2 t ∥ = 0, a.s. (17) Putting everything together Recall that we initially...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.