Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

This survey argues that reinforcement learning is not a departure from dynamic programming but an extension of it: once expectations are replaced by samples, value iteration becomes Q-learning and policy iteration becomes the natural policy

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:29 UTC pith:I6YWZSA6

load-bearing objection A useful survey with a load-bearing technical overstatement: the tabular NPG–PI equivalence is claimed as exact when it is only a soft/limiting correspondence. the 2 major comments →

arxiv 2603.08956 v6 pith:I6YWZSA6 submitted 2026-03-09 econ.GN cs.LGq-fin.EC

A Survey of Reinforcement Learning For Economics

Pranjal Rawat This is my paper
classification econ.GN cs.LGq-fin.EC
keywords reinforcement learningdynamic programmingQ-learningpolicy gradientstructural estimationstochastic approximationmulti-agent gamesoffline RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that reinforcement learning is a natural, sample-based extension of classical dynamic programming, not a rival computational culture. It claims that value iteration and Q-learning are the same fixed-point iteration with the expectation replaced by a single sample, and that policy iteration and the natural policy gradient coincide in the tabular setting. If right, RL gives economists a way to solve models with high-dimensional states, continuous actions, and strategic interactions that resist exact dynamic programming. The paper is careful that the practical payoff is bounded by simulator accuracy and statistical efficiency, and that deep RL lacks global convergence guarantees outside tabular or linear settings.

Core claim

The central claim is that RL algorithms are asymptotic approximations to classical dynamic-programming operators: Q-learning is value iteration applied to sampled transitions, the natural policy gradient is policy iteration performed by gradient ascent, and stochastic approximation supplies the convergence bridge. In tabular settings and with linear function approximation this rests on solid theory; with deep function approximation, convergence remains open and successes are case-specific. The consequence the author draws is that economists should read RL as a flexible numerical method for the same optimizations they already solve, to be used when a simulator or logged transitions are availa

What carries the argument

The load-bearing object is the recursive value-update equation of dynamic programming, together with the stochastic-approximation principle that replaces its expectation over unknown transitions with a single observed transition. The paper shows this operator is a contraction in the supremum norm, which is why value iteration converges geometrically and why Q-learning inherits the same fixed point; it also shows policy iteration is a root-finding update that linearizes the value equation, and that the natural policy gradient reproduces policy iteration in the tabular softmax case. The named danger is the 'deadly triad'—bootstrapping, off-policy data, and function approximation—which the pape

Load-bearing premise

The practical value of RL for economics depends on the existence of a sufficiently accurate simulator or of abundant logged transitions; where neither exists, sample-based dynamic programming loses its advantage over exact dynamic programming.

What would settle it

Take a finite MDP with known transitions, solve it exactly by value iteration, then run tabular Q-learning with a greedy-in-the-limit exploration schedule and a perfect simulator; if after each state-action pair has been visited millions of times Q-learning does not converge to the value-iteration solution, the central equivalence fails. For the simulator assumption, deploy both a discretized exact-DP solution and an RL solution on the same pricing problem with strategic buyers and check whether RL's larger-state solution beats the exact small solution out-of-sample; if it does not, the practi

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Economists with a simulator can apply RL to dynamic problems that are too large for exact dynamic programming, including continuous-action and multi-agent settings.
  • In tabular or linear-approximation settings, sample-based Q-learning inherits the fixed point of value iteration, so convergence failures there signal implementation problems rather than a fundamentally different object.
  • Policy-gradient algorithms can be viewed as fast local solvers rooted in the same equation as policy iteration, not as black-box heuristics.
  • The same mathematics licenses structural estimation: an RL solver can replace exact dynamic programming inside an estimation loop for models with high-dimensional states.
  • Because deep RL lacks comparable guarantees, economic applications should prefer tabular, linear, or carefully structured function-approximation settings unless empirical evidence justifies the extra machinery.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the equivalence holds, any economic model that can be solved by dynamic programming can in principle be solved by a sample-based algorithm, suggesting a division of labour: build simulators where primitives are known, and reserve exact dynamic programming for small models or model classes with closed-form solutions.
  • The simulator dependence the author flags points toward offline RL from logged transactions as the more natural empirical route for economics, since observational data are what econometricians typically have; this is an extension the survey gestures at but does not develop.
  • A testable extension: entropy-regularized RL, whose softmax policies coincide with logit choice probabilities, could serve as a direct computational bridge between dynamic discrete-choice estimation and policy optimization, giving structural estimation and RL a shared objective.
  • If RL is dynamic programming with samples, then exploration is just the stochastic-approximation requirement that all state-action pairs be visited often enough, recasting classic design-of-experiments questions in economics as sample-complexity problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This survey argues that reinforcement learning is best understood as a sample-based extension of classical dynamic programming rather than a separate paradigm. It contrasts the 'inference culture' of econometrics with the 'control culture' of RL, gives a history of RL, develops the theory connecting value iteration, policy iteration, Q-learning, TD, and policy-gradient methods, and reviews applications in optimal control, structural estimation, games, bandits, offline RL, causal inference, and robust/constrained RL. The central thesis, stated in §5.7, is that value iteration becomes Q-learning and policy iteration becomes the natural policy gradient when expectations are replaced by samples, with exact recovery in the tabular case. The paper includes several simulation studies with publicly available code and is candid about the brittleness, sample inefficiency, and lack of global convergence guarantees of deep RL.

Significance. If the central equivalence claims are stated correctly, this is a valuable synthesis for economists. The paper's strengths include a clear terminology bridge between RL and the discrete-choice/DDC literature, a balanced discussion of the deadly triad and empirical pathologies, and honest treatment of the limits of deep RL. The structural equivalences (softmax/logit, inclusive value, Q-function as choice-specific value) are useful and generally accurate. The commitment to public simulation code is a concrete asset. However, the paper's flagship theoretical claim—that natural policy gradient exactly recovers policy iteration in the tabular case—is overstated in several places and needs correction before the survey can be accepted. The practical demonstrations, while illustrative, sometimes lack the detail needed for reproducibility, and some formal results are deferred to companion papers rather than proved or precisely stated in the text.

major comments (2)
  1. [§4.1.9, §5.4.3, §5.7] The asserted identity between natural policy gradient (NPG) and exact Howard policy iteration is stronger than the mathematics supports. For a tabular softmax policy, the NPG update derived from Eq. (54) has the form π_new(a|s) ∝ π_old(a|s) exp(α Q^{π_old}(s,a)) up to a state-dependent constant. For finite α, and in particular for the 'unit step size' claimed in §4.1.9 and §5.4.3, this is a Boltzmann/soft policy-improvement step, not the deterministic argmax of Howard's policy improvement in Eq. (37). Exact recovery of policy iteration requires α→∞ or an additional greedy projection. As written, the statements 'a single natural gradient step with unit step size recovers one step of exact policy iteration' (§4.1.9), 'NPG performs one full Newton step' with step size 1 (§5.4.3), and 'NPG recovers PI exactly in the tabular case' (§5.7) are not correct. This is load-bearing because the PI→NP
  2. [§5.2.7, Table 6] The NLLS-FQI simulation is reported with insufficient detail. The text states that NLLS-FQI recovers α̂ = 0.3600 'in a single iteration,' while Table 6 lists 'Iterations 341' for all fitted methods. It is unclear what 'single iteration' refers to, what optimization algorithm was used, what convergence tolerance was applied, or how the concentrated least-squares objective was optimized. Since this simulation is used to support the substantive claim that 'basis representability, not algorithmic failure' explains FQI's poor performance, the section should report the full estimation procedure or clearly separate the α-estimation step from the FQI iterations.
minor comments (4)
  1. [§5.5.2, footnote 104] The formal equivalence between soft-control and dynamic discrete choice models is deferred to the companion paper Rust and Rawat (2026), Appendix A. For a survey intended to be self-contained, please state the theorem or at least the exact conditions in the text, so the reader does not have to consult an unpublished companion to verify a claimed equivalence.
  2. [§5.4.3] The phrase 'Polyak-/suppress Lojasiewicz' appears to be a typo for 'Polyak-Łojasiewicz.' Please correct.
  3. [§2.3 / Table 2] The notation mapping is useful, but the table would benefit from a column specifying where each equivalence is used in the text, since some entries (e.g., 'advantage function' vs. 'CCP') are only informally connected.
  4. [General] Several simulation studies report means and standard errors but not the number of seeds explicitly on the relevant tables (e.g., Table 5). Please state seed counts and data-generation details consistently.

Circularity Check

0 steps flagged

No circular derivation found; the DP-RL correspondence is independently supported. Minor self-citations to the companion survey are not load-bearing. The PI→NPG claim is mathematically overstated but is a correctness issue, not circularity.

full rationale

The survey's central claim (Section 5.7) is that RL algorithms are sample-based approximations of classical DP operators. The VI→Q-learning reduction is explicit in Section 5.2.2: Q-learning is presented as Robbins-Monro applied to the Bellman operator (Eqs. 40–41), which is a genuine derivation rather than a renaming. The PI→NPG claim (Sections 4.1.9 and 5.4.3) is asserted with a citation to Kakade (2001) and, as the skeptical review notes, is overstated for finite step sizes; however, overstatement is a correctness risk, not circularity, because NPG is not defined in terms of PI nor fit to the output it purports to predict. The only self-citations are to the companion Rust and Rawat (2026) for the inverse problem and the soft-control/DDC equivalence; neither is load-bearing for the 'RL extends DP' thesis, which rests on externally checkable results (Watkins and Dayan 1992; Tsitsiklis 1994; Kakade 2001; Agarwal et al. 2021a). Section 4.2.4 even labels the exponential optimality-variable construction as 'a modeling assumption, not a derivation from first principles,' which is transparent rather than circular. The paper's limitation statements (e.g., 'Deep RL lacks comparable guarantees' in Section 5.7) further reduce any impression that the framework's value is derived from its own assumptions. No equation in the paper is equivalent to its input by construction; no fitted parameter is relabeled as a prediction. Thus no circular step is present. Score 2 reflects only the presence of minor self-citations, which are not load-bearing for the central claim.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The survey's central claim is a synthesis of standard material; no fitted parameters are load-bearing. The simulations use standard benchmark parameterizations (e.g., Brock-Mirman α=0.36, β=0.96). The main implicit assumptions are the MDP framework, the availability of a simulator, and one deferred proof in the companion paper.

axioms (5)
  • domain assumption The Markov decision process (MDP) framework and the Markov assumption on the state.
    Section 2.1: both RL variants operate within an MDP, which imposes the Markov assumption on the state.
  • standard math The Bellman operator is a contraction with modulus γ.
    Section 3.3, citing Blackwell (1965); this underpins value iteration and Q-learning convergence.
  • standard math Robbins-Monro step-size conditions (sum α_t diverges, sum α_t^2 converges) ensure stochastic approximation convergence.
    Section 5.2.1: used to justify Q-learning and SARSA convergence.
  • domain assumption A high-fidelity simulator or abundant logged transitions are available for training RL agents.
    Abstract states that successes remain bounded by reliance on accurate simulators; Section 2.1 contrasts simulator omnipotence with observational data.
  • ad hoc to paper The formal equivalence between soft-control and dynamic discrete choice models is taken from the companion paper Rust and Rawat (2026), Appendix A.
    Section 5.5.2, footnote 104: the proof is not included in this survey; the survey relies on the authors' own companion work.

pith-pipeline@v1.3.0-alltime-deepseek · 51369 in / 9302 out tokens · 84396 ms · 2026-08-02T18:29:38.384447+00:00 · methodology

0 comments
read the original abstract

This survey (re)introduces reinforcement learning methods to economists. The curse of dimensionality limits how far exact dynamic programming can be effectively applied, forcing us to rely on suitably "small" problems or our ability to convert "big" problems into smaller ones. While this reduction has been sufficient for many classical applications, a growing class of economic models resists such reduction. Reinforcement learning algorithms offer a natural, sample-based extension of dynamic programming, extending tractability to problems with high-dimensional states, continuous actions, and strategic interactions. I review the theory connecting classical planning to modern learning algorithms and demonstrate their mechanics through simulated examples in pricing, inventory control, strategic games, and preference elicitation. I also examine the practical vulnerabilities of these algorithms, noting their brittleness, sample inefficiency, sensitivity to hyperparameters, and the absence of global convergence guarantees outside of tabular settings. The successes of reinforcement learning remain strictly bounded by these constraints, as well as a reliance on accurate simulators. When guided by economic structure, reinforcement learning provides a remarkably flexible framework. It stands as an imperfect, but promising, addition to the computational economist's toolkit. A companion survey (Rust and Rawat, 2026b) covers the inverse problem of inferring preferences from observed behavior. All simulation code is publicly available.

Figures

Figures reproduced from arXiv: 2603.08956 by Pranjal Rawat.

Figure 1
Figure 1. Figure 1: Architecture comparison of the three fundamental algorithm families. (a) DQN maps [PITH_FULL_IMAGE:figures/full_fig_p025_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The control-as-inference graphical model. States [PITH_FULL_IMAGE:figures/full_fig_p026_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The four phases of a single MCTS simulation in AlphaGo Zero. (a) Selection traverses [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The Brock–Mirman economy (α = 0.36, β = 0.96, 1,000 states). (a) Value iteration on a scalar Bellman equation. The staircase iterates Vk+1 = T Vk, converging at the linear rate γ. (b) Policy iteration as Newton’s method. Each step solves for the fixed point of the active policy operator T πk , jumping to the tangent line’s intersection with the diagonal. (c) Sup-norm error ∥Vk − V ∗∥∞ for the discretized m… view at source ↗
Figure 5
Figure 5. Figure 5: RMSVE vs. episodes for TD(λ) on the 20-state corridor. Shaded regions show ±1 SE over 20 seeds. Higher λ propagates the terminal reward faster. The forward view requires waiting until the end of the episode to compute Gλ t . The backward view computes the same total update incrementally. At each step, compute one TD error δt = Rt+1 + γV (St+1) − V (St) and distribute it to all states via an eligibility tra… view at source ↗
Figure 6
Figure 6. Figure 6: LQC convergence of FVI and FQI (left), DQN learning curve (middle), and value [PITH_FULL_IMAGE:figures/full_fig_p037_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Left: convergence of ∥Vk − V ∗∥∞ for FVI, linear FQI, Oracle-FQI, and NLLS-FQI on the Brock–Mirman economy. Right: NLLS-FQI estimated α trajectory, converging from α0 = 0.5 to the true α = 0.36 in one iteration. 79Observations where zkα − k ′ ≤ 0 (infeasible consumption) contribute a penalty equal to the mean squared target, preventing the optimizer from improving RSS by shrinking the feasible set. 37 [PI… view at source ↗
Figure 8
Figure 8. Figure 8: Geometry of the projected Bellman operator in [PITH_FULL_IMAGE:figures/full_fig_p042_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Information geometry of the natural policy gradient. [PITH_FULL_IMAGE:figures/full_fig_p045_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Majorization-minimization interpretation of trust region updates. [PITH_FULL_IMAGE:figures/full_fig_p046_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Trust region methods in the LQC monetary policy setting. A central bank learns a [PITH_FULL_IMAGE:figures/full_fig_p047_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Overestimation bias from Jensen’s inequality with [PITH_FULL_IMAGE:figures/full_fig_p053_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Left: OLS regression of log consumption on log capital and log productivity, esti [PITH_FULL_IMAGE:figures/full_fig_p056_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Bus engine replacement benchmark. Left: computation time vs. fleet size (log scale). [PITH_FULL_IMAGE:figures/full_fig_p063_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: NFXP versus RL-based structural estimation. Top: the nested fixed-point algorithm [PITH_FULL_IMAGE:figures/full_fig_p064_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Wall-clock estimation time versus state-space scale for the four DDC estimators. [PITH_FULL_IMAGE:figures/full_fig_p073_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Convergence of expected actions to Nash equilibrium. Left: Cournot duopoly. Right: [PITH_FULL_IMAGE:figures/full_fig_p076_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Revenue curves r(p) = r ∗ − k(p − p ∗ ) 2 for four demand curvatures k ∈ {0.5, 1.0, 2.0, 3.5}. All models agree at the optimal price p ∗ = 5. Within the shaded explo￾ration zone, the curves are nearly indistinguishable, so playing prices near p ∗ is uninformative about the demand parameter. 10.1.2 Parametric Demand Broder and Rusmevichientong (2012) consider a parametric demand model d(p; z) = Pr(V ≥ p), … view at source ↗
Figure 19
Figure 19. Figure 19: Theoretical regret rate functions at constants equal to 1, [PITH_FULL_IMAGE:figures/full_fig_p085_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Cumulative regret on log-log axes (K = 100, S = 1,000, 10 seeds). Each algorithm’s legend entry includes its theoretical regret rate. Dashed lines show Θ(T), O( √ T), and O(log T) reference rates. Shaded regions are ±2 standard errors. 11 Offline Reinforcement Learning and Human Feedback In the preceding chapters, the agent interacts with its environment while learning: it tries a price, observes a purcha… view at source ↗
Figure 21
Figure 21. Figure 21: Policy value (as % of DP optimal) versus behavioral policy randomness [PITH_FULL_IMAGE:figures/full_fig_p092_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: RLHF versus DPO pipelines. Top row: the three-stage RLHF pipeline trains a re [PITH_FULL_IMAGE:figures/full_fig_p095_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Left: the optimal accept/reject boundary for searching states in wage-amenity space. [PITH_FULL_IMAGE:figures/full_fig_p096_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Policy value V π (s0) versus number of preference comparisons K for all six methods (30 seeds, L = 15). The right axis shows the percentage of DP-optimal value [PITH_FULL_IMAGE:figures/full_fig_p097_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Policy value V π (s0) versus segment length L at K = 2,000 for the neural network and DPO (20 seeds). The right axis shows the percentage of DP-optimal value [PITH_FULL_IMAGE:figures/full_fig_p098_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Causal graphs for three identification strategies in confounded MDPs. Gray dashed [PITH_FULL_IMAGE:figures/full_fig_p102_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Complete causal graph of the simulation DGP. Gray dashed node ( [PITH_FULL_IMAGE:figures/full_fig_p105_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: (a) Bias of five OPE estimators as a function of confounding strength [PITH_FULL_IMAGE:figures/full_fig_p106_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: (a) Lagrange multiplier λ over training episodes, with the LP shadow price λ ∗ as the dashed reference. (b) Mean discounted return for unconstrained and Lagrangian Q-learning, with the constrained LP optimum as the dashed reference. where P(s, a) is an uncertainty set of transition distributions for each state-action pair and p0(·|s, a) is the nominal transition kernel.249 Under the rectangularity assumpt… view at source ↗
Figure 30
Figure 30. Figure 30: Consumption policy c(w) for each method. Dashed lines show Q-learning policies; solid lines show DP. 255Standard Q-learning uses the usual TD target r + γ maxa′ Q(s ′ , a′ ); robust Q-learning replaces this with the worst-case target (144), using the nominal kernel for the inner minimization (generative model setting). Visit￾count learning rates α = C/(C + N(s, a)) with C = 100 and ε-greedy exploration de… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Theorems of Dr. David Blackwell and Their Contributions to Artificial Intelligence

    cs.GL 2026-04 unverdicted novelty 2.0

    Blackwell's Rao-Blackwell, Approachability, and Informativeness theorems provide frameworks for variance reduction, sequential decisions under uncertainty, and comparing information sources that remain relevant to AI.

Reference graph

Works this paper leans on

42 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [12]

    Thompson sampling for dynamic pricing.arXiv preprint arXiv:1802.03050,

    Ravi Ganti, Matyas Sustik, Quoc Tran, and Brian Seaman. Thompson sampling for dynamic pricing.arXiv preprint arXiv:1802.03050,

    Pith/arXiv arXiv

  2. [15]

    Deep reinforcement learning from self-play in imperfect- information games.arXiv preprint arXiv:1603.01121,

    Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games.arXiv preprint arXiv:1603.01121,

    Pith/arXiv arXiv

  3. [18]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

    Pith/arXiv arXiv 2005

  4. [19]

    Is q-learning minimax optimal? a tight sample complexity analysis.Operations Research, 72(1), 2024a

    Gen Li, Laixi Shi, Yuxin Chen, Yuting Wei, and Yuejie Chi. Is q-learning minimax optimal? a tight sample complexity analysis.Operations Research, 72(1), 2024a. Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model.Operations Research, 72(1), 2024b. Minne L...

    2019

  5. [23]

    Clare Lyle, Mark Rowland, and Will Dabney

    Available at SSRN: https://ssrn.com/abstract=4717195 or http://dx.doi.org/10.2139/ssrn.4717195. Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. InProceedings of the International Conference on Learning Repre- sentations (ICLR),

    work page doi:10.2139/ssrn.4717195

  6. [26]

    Empirical design in reinforcement learning

    Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. InarXiv preprint arXiv:2304.01315,

    Pith/arXiv arXiv

  7. [27]

    Sai Srivatsa Ravindranath, Zhe Feng, Di Wang, Manzil Zaheer, Aranyak Mehta, and David C. Parkes. Deep reinforcement learning for sequential combinatorial auctions. Submitted to ICLR 2025,

    2025

  8. [29]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  9. [30]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018a

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419):1140–1144, 2018a. Tom Silver,...

    Pith/arXiv arXiv

  10. [32]

    Solving large imperfect information games using cfr+.arXiv preprint arXiv:1407.5042,

    Oskari Tammelin. Solving large imperfect information games using cfr+.arXiv preprint arXiv:1407.5042,

    Pith/arXiv arXiv

  11. [34]

    Terry, John U

    131 Mark Towers, Ariel Kwiatkowski, Jordan K. Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierr´ e, Sander Schulhoff, Jun Jet Tai, Hannah Tze, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments. InarXiv preprint ...

    Pith/arXiv arXiv

  12. [38]

    Near-optimal sample complexity for iterated CVaR reinforcement learning with a generative model.arXiv preprint arXiv:2503.08934,

    Yue Wang, Tomasz ˙Zak, and Csaba Szepesv´ ari. Near-optimal sample complexity for iterated CVaR reinforcement learning with a generative model.arXiv preprint arXiv:2503.08934,

    Pith/arXiv arXiv

  13. [40]

    Self-rewarding language models.arXiv preprint arXiv:2401.10020,

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.arXiv preprint arXiv:2401.10020,

    Pith/arXiv arXiv

  14. [41]

    Shangtong Zhang and Richard S. Sutton. A deeper look at experience replay. InarXiv preprint arXiv:1712.01275,

    Pith/arXiv arXiv

  15. [42]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,

    Pith/arXiv arXiv 1909

  16. [1959]

    Santos and John Rust

    Manuel S. Santos and John Rust. Convergence properties of policy iteration.SIAM Journal on Control and Optimization, 42(6):2094–2115,

    2094

  17. [1960]

    Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as Bayesian inference.arXiv preprint arXiv:2205.11275,

    Pith/arXiv arXiv

  18. [1982]

    Wainwright, and Linda Zhao

    117 Junhui Cai, Ran Chen, Martin J. Wainwright, and Linda Zhao. Doubly high-dimensional contextual bandits: An interpretable model for joint assortment-pricing.arXiv preprint arXiv:2309.07956,

    Pith/arXiv arXiv

  19. [1988]

    Richard S

    doi: 10.1023/A:1022633531479. Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on ap- proximating dynamic programming. InProceedings of the Seventh International Conference on Machine Learning, pages 216–224. Morgan Kaufmann,

    doi:10.1023/a:1022633531479

  20. [1992]

    Markov games as a framework for multi-agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994, pages 157–163. Morgan Kaufmann,

    1994

  21. [1993]

    First-order methods for Wasserstein distributionally robust MDP

    Julien Grand-Cl´ ement and Christian Kroer. First-order methods for Wasserstein distributionally robust MDP. InProceedings of the 38th International Conference on Machine Learning (ICML), pages 2010–2019,

    2010

  22. [1994]

    doi: 10.1007/bf00993306. John N. Tsitsiklis. On the convergence of optimistic policy iteration.Journal of Machine Learning Research, 3:59–72,

    work page doi:10.1007/bf00993306

  23. [1997]

    Contextual dynamic pricing with strategic buyers under unknown valuations.arXiv preprint arXiv:2307.04895,

    Daniele Tullii, Adel Javanmard, Matteo Pirotta, and Pierre Lezaud. Contextual dynamic pricing with strategic buyers under unknown valuations.arXiv preprint arXiv:2307.04895,

    Pith/arXiv arXiv

  24. [2001]

    Contextual dynamic pricing with strategic buyers.arXiv preprint arXiv:2307.04055, 2024a

    124 Allen Liu, Jingwen Yang, Yining Wang, and Jianghao Sun. Contextual dynamic pricing with strategic buyers.arXiv preprint arXiv:2307.04055, 2024a. Jiaxi Liu, Xiaoqing Wang, Yuming Deng, Xingyu Wu, and Yidong Zhang. Dynamic pricing on E-commerce platform with deep reinforcement learning: A field experiment.arXiv preprint arXiv:1912.02572,

    Pith/arXiv arXiv 1912

  25. [2003]

    Semiparametric dynamic pricing.arXiv preprint arXiv:2401.01136,

    Jianqing Fan, Yongyi Guo, and Mengxin Yu. Semiparametric dynamic pricing.arXiv preprint arXiv:2401.01136,

    Pith/arXiv arXiv

  26. [2006]

    Assessing game balance with AlphaZero: Exploring alternative rule sets in chess.arXiv preprint arXiv:2009.04374,

    Nenad Tomasev, Ulrich Paquet, Demis Hassabis, and Vladimir Kramnik. Assessing game balance with AlphaZero: Exploring alternative rule sets in chess.arXiv preprint arXiv:2009.04374,

    Pith/arXiv arXiv 2009

  27. [2008]

    Peter Arcidiacono and Robert A

    doi: 10.1007/s10994-007-5038-2. Peter Arcidiacono and Robert A. Miller. Conditional choice probability estimation of dynamic discrete choice models with unobserved heterogeneity.Econometrica, 79(6):1823–1867,

    doi:10.1007/s10994-007-5038-2

  28. [2010]

    Learning values across many orders of magnitude

    Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning values across many orders of magnitude. InAdvances in Neural Information Processing Systems, volume 29, 2016a. Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- Learning. InProceedings of the Thirtieth AAAI Conference on Artif...

    Pith/arXiv arXiv

  29. [2011]

    A dual approach to constrained Markov decision processes with entropy regularization.arXiv preprint arXiv:2110.08573,

    Dongjie Ying, Kaiqing Ding, and Javad Lavaei. A dual approach to constrained Markov decision processes with entropy regularization.arXiv preprint arXiv:2110.08573,

    Pith/arXiv arXiv

  30. [2012]

    Fleming and William M

    Wendell H. Fleming and William M. McEneaney. Risk-sensitive control on an infinite time horizon.SIAM Journal on Control and Optimization, 33(6):1881–1915,

    1915

  31. [2015]

    Asynchronous methods for deep reinforce- ment learning

    Volodymyr Mnih, Adri` a Puigdom` enech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. InProceedings of the 33rd International Conference on Machine Learning, pages 1928–1937. PMLR,

    1928

  32. [2016]

    Reinforcement learning and control as probabilistic inference: Tutorial and review.arXiv preprint arXiv:1805.00909,

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review.arXiv preprint arXiv:1805.00909,

    Pith/arXiv arXiv

  33. [2017]

    Adusumilli, M

    S. Adusumilli, M. Eckardt, and G. Tate. Estimation of dynamic discrete choice models with differentiable temporal-difference learning.arXiv preprint arXiv:2209.15174,

    Pith/arXiv arXiv

  34. [2018]

    Off-policy deep reinforcement learning with- out exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning with- out exploration. InProceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2052–2062. PMLR,

    2052

  35. [2019]

    Policy optimization for constrained MDPs with provably fast convergence.arXiv preprint arXiv:2111.00552,

    Yanli Liu, Kaiqing Ding, and Javad Lavaei. Policy optimization for constrained MDPs with provably fast convergence.arXiv preprint arXiv:2111.00552,

    Pith/arXiv arXiv

  36. [2020]

    Deep reinforcement learning: Emerging trends in macroe- conomics and future prospects

    Tohid Atashbar and Rui Aruhan Shi. Deep reinforcement learning: Emerging trends in macroe- conomics and future prospects. Working Paper 2022/259, International Monetary Fund,

    2022

  37. [2021]

    Dynamic pricing with fairness constraints.arXiv preprint arXiv:2402.07834,

    Yuxin Chen, Jieming Mao, and Rui Miao. Dynamic pricing with fairness constraints.arXiv preprint arXiv:2402.07834,

    Pith/arXiv arXiv

  38. [2022]

    Unifying causal reinforcement learning: Survey, taxonomy, algorithms and applications.arXiv preprint arXiv:2512.18135,

    Cristiano da Costa Cunha, Wei Liu, Tim French, and Ajmal Mian. Unifying causal reinforcement learning: Survey, taxonomy, algorithms and applications.arXiv preprint arXiv:2512.18135,

    arXiv

  39. [2023]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch

    doi: 10.1002/joom.1246. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. InAdvances in Neural Information Processing Systems, volume 34,

    doi:10.1002/joom.1246

  40. [2024]

    Strategic classifi- cation

    Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classifi- cation. InProceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science (ITCS), pages 111–122,

    2016

  41. [2025]

    Jonas Mueller, Vasilis Syrgkanis, and Matt Taddy

    doi: 10.1093/ej/ueaf104. Jonas Mueller, Vasilis Syrgkanis, and Matt Taddy. Low-rank bandit methods for high- dimensional dynamic pricing. InAdvances in Neural Information Processing Systems, vol- ume 32,

    work page doi:10.1093/ej/ueaf104

  42. [2026]

    114 Alekh Agarwal, Sham M

    develops a framework for evaluating algorithmic inefficiency and collu- sion risk in algorithmically mediated markets, combining simulators with factorial experimental designs. 114 Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift.Journal of Machine ...

    Pith/arXiv arXiv