{"id":"9f0b56b4-18fc-4686-b529-5b0a4b61dd3a","arxiv_id":"2603.08956","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Reinforcement learning is presented as a natural, sample-based extension of dynamic programming for economic models.","lead":"This paper is a survey that presents reinforcement learning as a sample-based extension of dynamic programming for economists, with simulated examples in pricing, inventory, games, and preference elicitation. It is a broad, well-organized introduction rather than a new research result, useful for economists who want a map of RL tools and their failure modes.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DP–RL equivalence is clean for VI→Q-learning, but the claimed PI→NPG identity is not: unit-step tabular NPG is soft policy improvement, not exact Howard policy iteration. §4.1.9/§5.4.3/§5.7 overstate it.","rationale":"The reader's weakest assumption concerns simulator quality and data availability, which is a real practical boundary but not the most load-bearing point for the paper's central theoretical claim. The more precise risk is internal: the survey's headline equivalence 'policy iteration becomes natural policy gradient' is stated too strongly. In tabular softmax, the NPG update is soft policy improvement; exact policy iteration is recovered only in the infinite-step limit, and the Newton-step language conflates the Bellman-residual Newton interpretation of PI (§5.1.2) with the natural-gradient step in policy space. This does not destroy the survey's overall value: the VI→Q-learning pillar is solid, the paper candidly acknowledges deep-RL non-convergence and the deadly triad, and the simulations are illustrations rather than the core theorem. But a central pedagogical claim needs a correct statement. The existing CONDITIONAL verdict already requires the authors to strengthen or qualify equivalences, so my concern does not move the verdict; it identifies a specific place where the condition should be applied.","tokens_in":51702,"tokens_out":8039,"duration_ms":86327,"concrete_test":"Re-derive the tabular softmax NPG update explicitly for a two-state, two-action MDP with known Q^π: compute F(θ) from Eq. (20)/(54), compute ∇J, then form θ' = θ + F^{-1}∇J (unit step). Compare π_{θ'} with the classical PI greedy policy argmax_a Q^π(s,a). If π_{θ'} is not greedy for generic rewards, then the 'unit step recovers exact PI' claim is false as stated. Then test whether the intended claim requires α→∞, or η=1/(1−γ) plus a temperature limit; if so, §4.1.9/§5.4.3/§5.7 should be rewritten as a limit result. This check is analytic and requires no simulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.7 rests on two identities: VI→Q-learning and PI→NPG. The second is asserted at §4.1.9 ('a single natural gradient step with unit step size recovers one step of exact policy iteration') and §5.4.3 ('NPG performs one full Newton step' at step size 1). Deriving the tabular softmax NPG update from Eq. (20)/(54) gives F^{-1}∇J = Q^{π}(s,a) + c(s), i.e. the advantage up to a state-dependent constant. The one-step update is therefore π_new(a|s) ∝ π_old(a|s) exp(α Q^{π_old}(s,a)). For finite α this is a Boltzmann/soft policy-improvement step, not the deterministic argmax of Howard's policy iteration; exact recovery of PI requires α→∞. The text offers no derivation of the 'unit step size' or 'Newton step' claims, only a citation, and the §5.7 summary 'policy iteration becomes the natural policy gradient' is at best a limiting statement, not an equivalence. This is load-bearing because the PI/NPG isomorphism is the pillar that licenses the 'RL extends DP' framing for policy-based methods, and the paper's own treatment of deep RL admits convergence is open—so the precise, correct statement of the tabular equivalence matters more, not less.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey argues that reinforcement learning is best understood as a sample-based extension of classical dynamic programming rather than a separate paradigm. It contrasts the 'inference culture' of econometrics with the 'control culture' of RL, gives a history of RL, develops the theory connecting value iteration, policy iteration, Q-learning, TD, and policy-gradient methods, and reviews applications in optimal control, structural estimation, games, bandits, offline RL, causal inference, and robust/constrained RL. The central thesis, stated in §5.7, is that value iteration becomes Q-learning and policy iteration becomes the natural policy gradient when expectations are replaced by samples, with exact recovery in the tabular case. The paper includes several simulation studies with publicly available code and is candid about the brittleness, sample inefficiency, and lack of global convergence guarantees of deep RL.","tokens_in":52026,"tokens_out":5593,"duration_ms":60555,"significance":"If the central equivalence claims are stated correctly, this is a valuable synthesis for economists. The paper's strengths include a clear terminology bridge between RL and the discrete-choice/DDC literature, a balanced discussion of the deadly triad and empirical pathologies, and honest treatment of the limits of deep RL. The structural equivalences (softmax/logit, inclusive value, Q-function as choice-specific value) are useful and generally accurate. The commitment to public simulation code is a concrete asset. However, the paper's flagship theoretical claim—that natural policy gradient exactly recovers policy iteration in the tabular case—is overstated in several places and needs correction before the survey can be accepted. The practical demonstrations, while illustrative, sometimes lack the detail needed for reproducibility, and some formal results are deferred to companion papers rather than proved or precisely stated in the text.","major_comments":[{"comment":"The asserted identity between natural policy gradient (NPG) and exact Howard policy iteration is stronger than the mathematics supports. For a tabular softmax policy, the NPG update derived from Eq. (54) has the form π_new(a|s) ∝ π_old(a|s) exp(α Q^{π_old}(s,a)) up to a state-dependent constant. For finite α, and in particular for the 'unit step size' claimed in §4.1.9 and §5.4.3, this is a Boltzmann/soft policy-improvement step, not the deterministic argmax of Howard's policy improvement in Eq. (37). Exact recovery of policy iteration requires α→∞ or an additional greedy projection. As written, the statements 'a single natural gradient step with unit step size recovers one step of exact policy iteration' (§4.1.9), 'NPG performs one full Newton step' with step size 1 (§5.4.3), and 'NPG recovers PI exactly in the tabular case' (§5.7) are not correct. This is load-bearing because the PI→NP","section":"§4.1.9, §5.4.3, §5.7"},{"comment":"The NLLS-FQI simulation is reported with insufficient detail. The text states that NLLS-FQI recovers α̂ = 0.3600 'in a single iteration,' while Table 6 lists 'Iterations 341' for all fitted methods. It is unclear what 'single iteration' refers to, what optimization algorithm was used, what convergence tolerance was applied, or how the concentrated least-squares objective was optimized. Since this simulation is used to support the substantive claim that 'basis representability, not algorithmic failure' explains FQI's poor performance, the section should report the full estimation procedure or clearly separate the α-estimation step from the FQI iterations.","section":"§5.2.7, Table 6"}],"minor_comments":[{"comment":"The formal equivalence between soft-control and dynamic discrete choice models is deferred to the companion paper Rust and Rawat (2026), Appendix A. For a survey intended to be self-contained, please state the theorem or at least the exact conditions in the text, so the reader does not have to consult an unpublished companion to verify a claimed equivalence.","section":"§5.5.2, footnote 104"},{"comment":"The phrase 'Polyak-/suppress Lojasiewicz' appears to be a typo for 'Polyak-Łojasiewicz.' Please correct.","section":"§5.4.3"},{"comment":"The notation mapping is useful, but the table would benefit from a column specifying where each equivalence is used in the text, since some entries (e.g., 'advantage function' vs. 'CCP') are only informally connected.","section":"§2.3 / Table 2"},{"comment":"Several simulation studies report means and standard errors but not the number of seeds explicitly on the relevant tables (e.g., Table 5). Please state seed counts and data-generation details consistently.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The PI/NPG issue is the main obstacle. The paper's central idea is sound, but the overstatement at §4.1.9, §5.4.3, and §5.7 must be corrected to a limiting statement. The NLLS-FQI simulation also needs fuller documentation. I do not see a circularity problem in the reliance on the companion paper, but the text should flag more clearly which results are proved there rather than in this manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this one if you want a broad map of RL for economics: it is clearly written, honest about the field's limits, and the design of the simulation studies is thoughtful. The notation table mapping Q-functions to choice-specific values and the section on control-as-inference are genuinely handy. The claim that VI becomes Q-learning when you replace expectations with samples is correct and well-argued; the policy-iteration-as-Newton section is also standard and nicely illustrated.\n\nThe problem is Section 5.7's umbrella statement and the sharper claims in §4.1.9 and §5.4.3. The paper says a unit-step natural policy gradient in the tabular softmax case 'recovers one step of exact policy iteration' and that NPG 'performs one full Newton step.' That is not right. The tabular NPG update is π_new(a|s) ∝ π_old(a|s) exp(α Q^π(s,a)) — a Boltzmann/soft improvement. For finite α it never assigns zero probability to suboptimal actions; Howard's policy improvement is the deterministic argmax, recovered only in the α→∞ limit. So the central 'RL is an extension of DP, not a departure' framing is fine if stated as a limiting or soft equivalence, but as written it overstates an identity that the paper treats as load-bearing. The text gives no derivation, only a citation. That is a genuine soft spot and needs fixing.\n\nThe other soft spots are minor: one formal equivalence is deferred to the companion paper, and the simulation tables don't give seeds or a commit hash, so full reproducibility is not documented. For a survey, the self-citation to the companion is acceptable, but the deferred proof should be marked as an external reference.\n\nCredit where earned: the survey covers a lot of ground without being shallow, and the honest treatment of the deadly triad, the reproducibility crisis, and the dependence on simulators is a service to the target audience. The NLLS-FQI demonstration is a useful pedagogical example, though not a new result.\n\nVerdict: worth a serious referee, but the referee should require the NPG/PI claim be corrected or carefully qualified. As it stands I wouldn't cite it for that equivalence. If fixed, it would be a standard reference for economists entering RL.","headline":"A useful survey with a load-bearing technical overstatement: the tabular NPG–PI equivalence is claimed as exact when it is only a soft/limiting correspondence.","tokens_in":52500,"tokens_out":3332,"would_cite":false,"duration_ms":35470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that reinforcement learning is not a departure from dynamic programming but an extension of it: once expectations are replaced by samples, value iteration becomes Q-learning and policy iteration becomes the natural policy","keywords":["reinforcement learning","dynamic programming","Q-learning","policy gradient","structural estimation","stochastic approximation","multi-agent games","offline RL"],"falsifier":"Take a finite MDP with known transitions, solve it exactly by value iteration, then run tabular Q-learning with a greedy-in-the-limit exploration schedule and a perfect simulator; if after each state-action pair has been visited millions of times Q-learning does not converge to the value-iteration solution, the central equivalence fails. For the simulator assumption, deploy both a discretized exact-DP solution and an RL solution on the same pricing problem with strategic buyers and check whether RL's larger-state solution beats the exact small solution out-of-sample; if it does not, the practi","tokens_in":51535,"feed_emoji":"🤖","tokens_out":6144,"duration_ms":59940,"temperature":0.7,"pith_summary":"The paper tries to establish that reinforcement learning is a natural, sample-based extension of classical dynamic programming, not a rival computational culture. It claims that value iteration and Q-learning are the same fixed-point iteration with the expectation replaced by a single sample, and that policy iteration and the natural policy gradient coincide in the tabular setting. If right, RL gives economists a way to solve models with high-dimensional states, continuous actions, and strategic interactions that resist exact dynamic programming. The paper is careful that the practical payoff is bounded by simulator accuracy and statistical efficiency, and that deep RL lacks global convergence guarantees outside tabular or linear settings.","feed_headline":"Reinforcement learning is dynamic programming with samples","feed_subtitle":"Economists can use RL as a sample-based extension of exact DP; the catch is it needs accurate simulators or logged data.","key_machinery":"The load-bearing object is the recursive value-update equation of dynamic programming, together with the stochastic-approximation principle that replaces its expectation over unknown transitions with a single observed transition. The paper shows this operator is a contraction in the supremum norm, which is why value iteration converges geometrically and why Q-learning inherits the same fixed point; it also shows policy iteration is a root-finding update that linearizes the value equation, and that the natural policy gradient reproduces policy iteration in the tabular softmax case. The named danger is the 'deadly triad'—bootstrapping, off-policy data, and function approximation—which the pape","core_discovery":"The central claim is that RL algorithms are asymptotic approximations to classical dynamic-programming operators: Q-learning is value iteration applied to sampled transitions, the natural policy gradient is policy iteration performed by gradient ascent, and stochastic approximation supplies the convergence bridge. In tabular settings and with linear function approximation this rests on solid theory; with deep function approximation, convergence remains open and successes are case-specific. The consequence the author draws is that economists should read RL as a flexible numerical method for the same optimizations they already solve, to be used when a simulator or logged transitions are availa","pith_inferences":["If the equivalence holds, any economic model that can be solved by dynamic programming can in principle be solved by a sample-based algorithm, suggesting a division of labour: build simulators where primitives are known, and reserve exact dynamic programming for small models or model classes with closed-form solutions.","The simulator dependence the author flags points toward offline RL from logged transactions as the more natural empirical route for economics, since observational data are what econometricians typically have; this is an extension the survey gestures at but does not develop.","A testable extension: entropy-regularized RL, whose softmax policies coincide with logit choice probabilities, could serve as a direct computational bridge between dynamic discrete-choice estimation and policy optimization, giving structural estimation and RL a shared objective.","If RL is dynamic programming with samples, then exploration is just the stochastic-approximation requirement that all state-action pairs be visited often enough, recasting classic design-of-experiments questions in economics as sample-complexity problems."],"forward_implications":["Economists with a simulator can apply RL to dynamic problems that are too large for exact dynamic programming, including continuous-action and multi-agent settings.","In tabular or linear-approximation settings, sample-based Q-learning inherits the fixed point of value iteration, so convergence failures there signal implementation problems rather than a fundamentally different object.","Policy-gradient algorithms can be viewed as fast local solvers rooted in the same equation as policy iteration, not as black-box heuristics.","The same mathematics licenses structural estimation: an RL solver can replace exact dynamic programming inside an estimation loop for models with high-dimensional states.","Because deep RL lacks comparable guarantees, economic applications should prefer tabular, linear, or carefully structured function-approximation settings unless empirical evidence justifies the extra machinery."],"fun_headline_variants":["RL = DP with samples, but requires simulators","Economists: RL is DP with samples, but needs simulators","RL for economics: DP with samples, pitfalls included","RL: DP with samples, but no convergence guarantees"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The practical value of RL for economics depends on the existence of a sufficiently accurate simulator or of abundant logged transitions; where neither exists, sample-based dynamic programming loses its advantage over exact dynamic programming.","fun_headline_variants_meta":{"raw":{"variants":["RL = DP with samples, but requires simulators","Economists: RL is DP with samples, but needs simulators","RL for economics: DP with samples, pitfalls included","RL: DP with samples, but no convergence guarantees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001271,"raw_usage":{"total_tokens":5028,"prompt_tokens":727,"completion_tokens":4301,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":4235}},"tokens_in":471,"tokens_out":4301,"duration_ms":29488,"temperature":1.0,"reasoning_tokens":4235,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:29:38.384447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a finite MDP with known transitions, solve it exactly by value iteration, then run tabular Q-learning with a greedy-in-the-limit exploration schedule and a perfect simulator; if after each state-action pair has been visited millions of times Q-learning does not converge to the value-iteration solution, the central equivalence fails. For the simulator assumption, deploy both a discretized exact-DP solution and an RL solution on the same pricing problem with strategic buyers and check whether RL's larger-state solution beats the exact small solution out-of-sample; if it does not, the practi","supporting_citations":[],"review_version":1}