Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Expected free energy minimization in active inference is a convex Markov decision process.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:43 UTC pith:ICG4JFLB

load-bearing objection The convex-MDP claim is real but narrowly scoped: it holds for the paper's own EFE variant with a frozen model, and the experiments run at a step size the theory doesn't cover. the 3 major comments →

arxiv 2607.20152 v1 pith:ICG4JFLB submitted 2026-07-22 cs.LG cs.AIstat.ML

Active Inference as a Convex Markov Decision Process

Nikola Milosevic , Nicol\'as Hinrichs , Nico Scherf This is my paper
classification cs.LG cs.AIstat.ML MSC 90C4090C2568T05
keywords active inferenceexpected free energyconvex Markov decision processmirror descentperformative reinforcement learningepistemic valueoccupancy measurelatent MDP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that, under a fixed world model and an episodic protocol, an active-inference agent's planning objective — expected free energy (EFE) — is not a strange black box but a convex Markov decision process. Concretely, EFE splits into a linear reward term (pragmatic value, ambiguity, and recognition cost) plus one convex nonlinearity: the negative entropy of the predictive state marginal, which acts as a curiosity bonus. Because of this structure, EFE minimization can be attacked with convex-MDP solvers, and the mirror-descent algorithm the paper derives converges at O(1/K) with a closed-form softmax dynamic programming update. Closing the loop by refitting the model to policy-induced data makes the problem performative, with a stable policy–model pair existing under continuity conditions. A sympathetic reader would care because this is the first time EFE minimization is placed in an optimization class with known algorithms and rates, making AIF implementable and analyzable with standard RL tools.

Core claim

For closed-loop policies under a fixed model, EFE minimization is equivalent to minimizing a convex functional over the polytope of achievable state-action occupancies: min_π G(π) ≡ min_{μ∈K} Γ(μ), where Γ(μ) = ⟨ℓ, μ⟩ + Φ(μ). The linear part ℓ carries pragmatic preference, observation likelihood, and recognition terms; Φ is the negative Shannon entropy of state marginals, the epistemic value. This makes EFE minimization a convex MDP, and at the exact-posterior limit Φ becomes negative mutual information between states and observations, recovering the information-gain interpretation. The paper derives a mirror-descent algorithm MD-AIF whose linearized reward is a static preference plus a stat

What carries the argument

The central object is the occupancy polytope K: the set of feasible time-indexed state-action marginals μ_t that satisfy Chapman–Kolmogorov flow constraints induced by a fixed world model and a policy. There is a bijection between policies and interior occupancies, so EFE minimization can be re-posed as a convex program min_{μ∈K} Γ(μ). The argument carries because every EFE term except epistemic value is linear in μ, and epistemic value is negative entropy of the state marginal, a convex function of μ. The algorithmic engine is mirror descent with the conditional-negative-entropy Bregman divergence Ψ; this choice makes each iteration an entropy-regularized linear program whose solution is a

Load-bearing premise

The claim rests on a particular EFE variant — log predictive marginal minus log recognition minus log likelihood minus log preference — evaluated with the world model frozen during planning and with a causal Markov future posterior; if the operative EFE in an implementation includes parameter-uncertainty novelty, a non-Markov posterior, or model refitting inside the planning horizon, the linear-plus-convex decomposition is not established.

What would settle it

Run a small tabular POMDP with two states, two actions, and one observation, using an EFE variant that includes the parameter-novelty terms the paper sets aside. Compute the objective Γ along the line segment between two feasible occupancy measures, or track the objective sequence of the paper's mirror-descent algorithm while refitting the model after every policy update. If the function is not convex (a chord lies below the curve) or the iterates enter a limit cycle instead of converging at O(1/K), the convex-MDP claim for that variant is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • EFE minimization inherits the full convex-MDP toolbox: convergence rates, dynamic programming schemes, and policy-improvement guarantees transfer to active-inference agents.
  • The epistemic term is a concrete policy-dependent curiosity bonus — log of the predictive state marginal — so exploration is automatic and tunable, and at VFE convergence it equals the mutual-information information gain of classical active inference.
  • The paper's MD-AIF converges at O(1/K) with constant L=½T(T+1), making the algorithm's rate explicit and comparable to standard reinforcement-learning methods.
  • In infinite-horizon settings, the aggregate-occupancy objective gives a clean discounted convex MDP, while per-step entropy aggregation does not collapse to a single fixed point; the paper shows the distinction matters.
  • When the model is refit to the deployed policy's data, the loop is performative; a stable policy–model pair exists under continuity and support-floor assumptions, giving a target for future convergence analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the convex-MDP identification is accepted, active-inference planners can be implemented as curiosity-boosted actor-critic algorithms; the paper's own algorithm is effectively soft value iteration with a coverage bonus, so practitioners could replace it with any convex-MDP solver and expect similar guarantees.
  • The fixed-model episodic protocol is the real boundary of the claim; in truly online agents that refit the model during planning, the linear-plus-convex decomposition can break, so an empirical study varying refit frequency would clarify where the guarantee degrades.
  • The paper's history-augmentation lemma suggests that recurrent-state architectures used in deep active inference are not just engineering conveniences but exactly the state space on which convex-MDP structure survives; a testable prediction is that memoryless deep planners have worse convergence than recurrent ones on partially observable tasks.
  • The aggregate-vs-per-step infinite-horizon distinction points to a design decision hidden in existing implementations; comparing the two conventions on the same task is a direct experiment to see whether the entropy's non-commutation with time averaging has practical effect.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an episodic active-inference protocol in which a world model m=(p,ν) is learned from data and then frozen while a policy is optimized by minimizing the EFE variant defined in Eqs. (8)–(9) under causal imagined rollouts qπ. The central structural result is that this EFE can be written as G(π)=⟨ℓ,µπ⟩+Φ(µπ), where ℓ is a fixed linear cost and Φ=−∑_t H(ρ_t) is a convex function of the state-action occupancy µπ; minimization over policies is therefore equivalent to a convex program over the flow polytope K, i.e. a convex MDP (Prop. 1). The paper derives a mirror-descent algorithm (MD-AIF), proves relative smoothness with L=½T(T+1) and an O(1/K) convergence rate (Prop. 2, Thm. 1), discusses history-dependent and infinite-horizon extensions, and analyzes the closed-loop model-refitting case as performative reinforcement learning with a fixed-point existence result. Experiments on deterministic gridworlds compare MD-AIF with RL and EFE gradient descent, and test whether the epistemic drive accelerates model learning.

Significance. If the central claim is taken together with its stated hypotheses, the paper makes a valuable conceptual and technical contribution: it connects active inference to the convex-MDP literature and gives an explicit, checkable decomposition of one important EFE variant into a linear reward plus a convex negative state-marginal entropy. The proof of Lemma 1 is correct, and the relative-smoothness argument in Theorem 1 via the marginal/conditional Fisher metric split is elegant. The performative framing of the closed loop is appropriate and the paper is candid about the limits of the convergence analysis. The main reservation is scope: the theorem is proved for a specific EFE variant with a fixed model and causal posterior, and the abstract currently states the result more broadly. The experimental validation also runs outside the proven step-size and full-support assumptions. These issues are fixable, but they affect how the result should be stated and evidenced.

major comments (3)
  1. [Abstract; §2.2; §7; Prop. 1] The central claim as stated in the abstract and §7 — 'EFE minimization can be formulated as a convex MDP' — is broader than the statement actually proved. The proof of Proposition 1 uses the EFE integrand of Eq. (9), the causal imagined law qπ of Eq. (6), and a model (p,ν) and preference p̃ that are frozen during planning. Under these assumptions Lemma 1 is correct. But §2.2 explicitly calls Eq. (9) 'one of many variants used throughout the literature', and §5.2 concedes that parameter-based novelty terms may induce complex π-dependence. For other common EFE definitions — for example Eq. (20) with the exact posterior p(s|o), a non-causal future posterior, or model refitting inside the planning horizon — the linear-plus-negative-state-entropy decomposition, and hence convexity, is not established. Please qualify the abstract and conclusion to match the theorem's hypotheses, and consider s
  2. [App. F.1; Prop. 2; Thm. 1] App. F.1 says the step size η=0.05 is 'matching the assumption of Proposition 2', but Proposition 2 requires η=1/L, and Theorem 1 gives L=½T(T+1). For the planning horizon T=12 used in the convergence experiment, 1/L=2/(12·13)=0.0128; for the model-learning experiment with T=25 it is 0.0031. The experiments therefore run outside the proven step-size regime, so Fig. 2A's agreement with the O(1/K) line does not demonstrate Proposition 2 under its assumptions. Either rerun with η=1/L, or explicitly present the experiments as heuristic and remove the claim that they match the theoretical assumption.
  3. [§6; Thm. 1; Assumption 2] The gridworld experiments use deterministic transition kernels and a start state concentrated on a single state; the Dirichlet-MAP refit with pseudocount α0=10^{-3} also produces zero-probability transitions for unvisited (s,a,s'). The convergence proof (Lemma 5, Theorem 1) uses positive reference measures and Markov kernels, and the performative fixed-point result (Assumption 2) requires p(s'|s,a)≥ε0>0 and q0(s)>0. As a result, the theoretical guarantees of §4 and §E do not apply to the reported experiments as stated. The authors should either add a soft transition floor so that the model is full-support and rerun, or state plainly that the empirical study is outside the theorem's assumptions and serves only as an illustration.
minor comments (5)
  1. [§2.2] The phrase 'this structure is independent of the exact EFE variant' is too strong. Appendix A shows that variants differ in the linear cost and in whether the nonlinearity is the state-marginal negative entropy Φ or the joint negative entropy R (when action complexity is added). The convex-MDP conclusion is invariant, but the exact decomposition is not; please rephrase.
  2. [Algorithm 1] The state marginal ρ_t(s) used in the linearized reward r_t is not explicitly stored in the pseudocode. Clarify that it is computed as ∑_a µ_t(s,a) from the forward pass, especially for readers implementing the algorithm.
  3. [§D.2] The statement that the discounted relative-smoothness rate 'carries over with the discounted relative-smoothness constant [22]' is asserted without a proof or a precise constant. If space permits, give the constant or a precise reference to the theorem being invoked.
  4. [Lemma 5] The kernel notation is under-specified: K:Y→Z is introduced, but the proof uses K(z|y). Please define the conditional-kernel notation explicitly before Lemma 5.
  5. [§5.1] Proposition 3 is labeled 'informal' and Proposition 5 is the formal statement. Consider consolidating the numbering or clearly stating the relationship to avoid confusion, since both are numbered as separate propositions.

Circularity Check

0 steps flagged

No significant circularity: the convex-MDP theorem follows from the explicitly scoped EFE variant; the abstract's breadth and the step-size mismatch are scope/numerical issues, not circular reasoning.

full rationale

The derivation chain is self-contained. Proposition 1 proves min_π G(π;m) = min_{μ∈K} Γ(μ) for the EFE defined in Eqs. (8)-(9) under the stated episodic fixed-model protocol. Lemma 1 rewrites that definition: E_{q_π}[log ρ_t^π] = -H(ρ_t^π), and the remaining terms are linear because ν, p, and p̃ are held fixed. This is a disclosed definitional property, not a fitted parameter or a hidden reuse of the conclusion. The paper explicitly says Eq. (9) 'is one of many variants used throughout the literature' (§2.2) and concedes that 'parameter-based novelty terms in the EFE may induce complex π-dependence' (§5.2), so the unqualified abstract phrase is a scope overstatement, not a circular premise. No self-citations appear among references [1]-[30]; the relative smoothness constant L = T(T+1)/2 is derived in Theorem 1, not fit; the O(1/K) rate is imported from a cited external result [26] with a proof sketch. The only flagged inconsistency, App. F.1's claim that η=0.05 is 'matching the assumption of Proposition 2' when Prop. 2 requires η=1/L=2/[T(T+1)], is a numerical/protocol mismatch and would be a correctness concern, not circularity. The O(1/k) reference line fitted to MD-AIF's initial gap is a plotting normalization, not a prediction. Hence no load-bearing step reduces to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The theoretical result is parameter-free: the constant L = ½T(T+1) is derived, not fitted, and the linear-plus-convex decomposition follows from the EFE definition. Listed free parameters are experiment settings only and do not enter the theorems. The axioms are dominated by two disclosed modeling choices (the EFE variant of Eq. (9) with predictive-marginal epistemic term, and the fixed-model episodic protocol) plus standard convex-MDP/relative-smoothness machinery. No invented physical entities: the performative mirror operator (Def. 4) and the episodic AIF loop (Eq. 2) are mathematical constructions with their own definitions, not postulates requiring external evidence.

free parameters (4)
  • mirror-descent step size η = η = 0.05 (both experiments)
    Hand-chosen. Prop. 2's O(1/K) guarantee requires η=1/L (≈0.0128 for T=12, ≈0.0031 for T=25); experiments run outside the proven regime. Reported in App. F.1/F.2 as 'constant step size η=0.05'.
  • Manhattan preference sharpness α = α = 0.5
    Log-preference log p̃(s) ∝ −α·d(s,s_goal) in the 5×5 experiment (Eq. 57); a modeling input to the environment, not part of the theorems.
  • Dirichlet-MAP pseudocount α0 = α0 = 10⁻³
    Prior strength for the model-refit estimator p̂ in the 10×10 experiment (App. F.2); standard MAP hyperparameter, not theory-dependent.
  • baseline depths T_dc and MC samples N_mc = T_dc = 3 or 5; N_mc = 100
    Settings of the short-horizon AIF baselines; chosen for tractability, not fitted.
axioms (5)
  • domain assumption The operative EFE is the paper's Eq. (8)-(9) integrand, with epistemic term +log ρ_t(s_t) (predictive state-marginal log-density) and a causally-factorized imagination distribution q_π (Eq. 6); Appendix A variants share the linear-plus-negative-entropy form.
    Enters at Eq. (8)-(9) and Definition 2. Lemma 1's decomposition is a rearrangement of this definition; the paper discloses other EFE variants exist ([15,16]) and that parameter-novelty terms would break the structure (§5.2).
  • domain assumption World model m=(p,ν) is held fixed during EFE planning; VFE and EFE minimization alternate over full episodes (episodic protocol, Fig. 1, §2).
    Prop. 1-2 and Theorem 1 hold only for fixed m. Closing the loop (App. E, Algorithm 2) loses the fixed-model guarantee: Prop. 5 establishes only existence via Brouwer; convergence of the full loop is left open (§5.2).
  • standard math Occupancy-polytope machinery: the map π↦μ^π is a bijection onto the interior of the flow polytope K (Def. 1); every interior occupancy gives a unique memoryless closed-loop policy.
    Invoked at Eq. (12)-(14) and Prop. 1; standard finite-MDP occupancy theory [18,19]. Requires full-support policies (ρ>0); boundary/zero-occupancy behavior is not discussed.
  • standard math Relative smoothness framework of Lu-Freund-Nesterov [26] and the Fisher-metric decomposition R=Φ+Ψ into per-step marginal and conditional blocks (Eq. 35) used for ∇²Γ ⪯ L∇²Ψ on the flow-polytope tangent space.
    Used in Prop. 2 for the O(1/K) rate. Theorem 1 proves the bound via the Markov-contraction Lemma 5; the computation of L = ½T(T+1) checks out, but the proof is a sketch and not machine-checked.
  • domain assumption Assumption 1 (εr,εp-sensitivity / continuity of the policy-to-model-refit map) and Assumption 2 (support floor ε0 on model class) for the performative part.
    Needed for compactness and continuity so Brouwer's fixed-point theorem applies to the performative mirror operator (Prop. 5). Only existence is claimed; these assumptions do not affect the fixed-model convex-MDP result.

pith-pipeline@v1.3.0-alltime-deepseek · 19447 in / 51871 out tokens · 472419 ms · 2026-08-01T10:43:01.720514+00:00 · methodology

0 comments
read the original abstract

Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.

Figures

Figures reproduced from arXiv: 2607.20152 by Nicol\'as Hinrichs, Nico Scherf, Nikola Milosevic.

Figure 1
Figure 1. Figure 1: The episodic active inference setting considered in this work: A) the environment admits a true [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MD-AIF on gridworld environments. (A, top two rows) 5×5 deterministic gridworld. Left: EFE convergence (log–log scale) for MD-AIF, RL (no novelty), and EFE gradient descent, together with the theoretical O(1/k) rate (dashed). MD-AIF converges faster and to a lower value, driven by regularized dynamic programming. Right: Per-step imagined occupancy ρt(s) at convergence (k = 100) for MD-AIF (top) and RL (bot… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references

  1. [1]

    When to trust your model: Model-based policy optimization

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. InAdvances in Neural Information Processing Systems (NeurIPS), volume 32, 2019

    2019

  2. [2]

    Active inference on discrete state-spaces: A synthesis.Journal of Mathematical Psychology, 99:102447, 2020

    Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston. Active inference on discrete state-spaces: A synthesis.Journal of Mathematical Psychology, 99:102447, 2020

    2020

  3. [3]

    Active inference and epistemic value.Cognitive Neuroscience, 6(4):187–214, 2015

    Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, and Giovanni Pezzulo. Active inference and epistemic value.Cognitive Neuroscience, 6(4):187–214, 2015

    2015

  4. [4]

    Recurrent world models facilitate policy evolution.Advances in neural information processing systems, 31, 2018

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution.Advances in neural information processing systems, 31, 2018

    2018

  5. [5]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. InInternational Conference on Learning Representations (ICLR), 2020

    2020

  6. [6]

    Contrastive active inference

    Pietro Mazzaglia, Tim Verbelen, and Bart Dhoedt. Contrastive active inference. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  7. [7]

    Reward is enough for convex mdps

    Tom Zahavy, Brendan O’Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. InAdvances in Neural Information Processing Systems (NeurIPS), volume 34, 2021

    2021

  8. [8]

    Variational policy gradient method for reinforcement learning with general utilities

    Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvári, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, 2020

    2020

  9. [9]

    Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt

    Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. InProceedings of the 37th International Conference on Machine Learning (ICML),volume119ofPMLR, pages 7599–7609, 2020

    2020

  10. [10]

    Performative reinforcement learning

    Debmalya Mandal, Stelios Triantafyllou, and Goran Radanovic. Performative reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 ofPMLR, pages 23642–23680, 2023

    2023

  11. [11]

    Performative reinforcement learning in gradually shifting environments

    Ben Rank, Stelios Triantafyllou, Debmalya Mandal, and Goran Radanovic. Performative reinforcement learning in gradually shifting environments. InProceedings of the 40th Conference on Uncertainty in Artificial Intelligence (UAI), volume 244 ofPMLR, pages 3041–3075, 2024

    2024

  12. [12]

    Soft actor-critic: Off-policy max- imum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy max- imum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. Pmlr, 2018

    2018

  13. [13]

    Kakade, Karan Singh, and Abby Van Soest

    Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum en- tropy exploration. InProceedings of the 36th International Conference on Machine Learning (ICML), volume 97 ofPMLR, pages 2681–2691, 2019

    2019

  14. [14]

    On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

    Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine Learning Research, 22(98):1–76, 2021

    2021

  15. [15]

    Thomas Parr, Lancelot Da Costa, and Karl Friston. Markov blankets, information geometry and stochastic thermodynamics.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 378(2164):20190159, 12 2019

    2019

  16. [16]

    Beren Millidge, Alexander Tschantz, and Christopher L. Buckley. Whence the expected free energy? Neural Computation, 33(2):447–482, 2021

    2021

  17. [17]

    The free energy principle for perception and action: A deep learning perspective.Entropy, 24(2):301, 2022

    Pietro Mazzaglia, Tim Verbelen, Ozan Catal, and Bart Dhoedt. The free energy principle for perception and action: A deep learning perspective.Entropy, 24(2):301, 2022

    2022

  18. [18]

    Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming

    Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994

    1994

  19. [19]

    Efficientmodel-basedconcave utility reinforcement learning through greedy mirror descent

    BiancaMMoreno, MargauxBrégère, PierreGaillard, andNadiaOudjane. Efficientmodel-basedconcave utility reinforcement learning through greedy mirror descent. InInternational Conference on Artificial Intelligence and Statistics, pages 2206–2214. PMLR, 2024. N. Milosevic et al.•Accepted at IW AI 2026•github.com/milosen/aif_mbpo19 Active Inference as a Convex MDP

    2024

  20. [20]

    Pedro Pinto Santos, Alberto Sardinha, and Francisco S. Melo. The number of trials matters in infinite- horizon general-utility markov decision processes. InForty-second International Conference on Machine Learning, 2025

    2025

  21. [21]

    Mirror descent and nonlinear projected subgradient methods for convex optimization.Operations Research Letters, 31(3):167–175, 2003

    Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization.Operations Research Letters, 31(3):167–175, 2003

    2003

  22. [22]

    Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes.Mathematical Programming, 198(1):1059–1106, 2023

    Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes.Mathematical Programming, 198(1):1059–1106, 2023

    2023

  23. [23]

    Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    Shun-Ichi Amari. Natural gradient works efficiently in learning.Neural computation, 10(2):251–276, 1998

    1998

  24. [24]

    A natural policy gradient.Advances in neural information processing systems, 14, 2001

    Sham M Kakade. A natural policy gradient.Advances in neural information processing systems, 14, 2001

    2001

  25. [25]

    The information geometry of mirror descent.IEEE Transac- tions on Information Theory, 61(3):1451–1457, 2015

    Garvesh Raskutti and Sayan Mukherjee. The information geometry of mirror descent.IEEE Transac- tions on Information Theory, 61(3):1451–1457, 2015

    2015

  26. [26]

    Freund, and Yurii Nesterov

    Haihao Lu, Robert M. Freund, and Yurii Nesterov. Relatively smooth convex optimization by first-order methods, and applications.SIAM Journal on Optimization, 28(1):333–354, 2018

    2018

  27. [27]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, ed- itors,Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 ofProceedings of Machine Learning Research, p...

    2011

  28. [28]

    On the convergence rates of policy gradient methods.Journal of Machine Learning Research, 23(282):1–36, 2022

    Lin Xiao. On the convergence rates of policy gradient methods.Journal of Machine Learning Research, 23(282):1–36, 2022

    2022

  29. [29]

    A theory of regularized Markov decision pro- cesses

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized Markov decision pro- cesses. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2160–

  30. [30]

    Average-reward soft actor-critic.Reinforcement Learning Journal, 6:412–430, 2025

    Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, and Rahul V Kulkarni. Average-reward soft actor-critic.Reinforcement Learning Journal, 6:412–430, 2025. N. Milosevic et al.•Accepted at IW AI 2026•github.com/milosen/aif_mbpo20

    2025

  31. [2169]

    PMLR, 09–15 Jun 2019

    2019