Pith. sign in

REVIEW 9 minor 97 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Fitted ratio iteration converges without Bellman completeness

2026-07-07 14:02 UTC pith:O4CKTELL

load-bearing objection Clean result: KL-projected adjoint Bellman iteration contracts under ratio realizability alone, no completeness needed.

arxiv 2607.05375 v1 pith:O4CKTELL submitted 2026-07-06 stat.ML cs.LG

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Lars van der Laan , Nathan Kallus This is my paper
classification stat.ML cs.LG
keywords offline reinforcement learningoff-policy evaluationoccupancy ratioadjoint Bellman equationKL divergencedensity ratio estimationfitted iterationBellman completeness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FORE (Fitted Occupancy-Ratio Evaluation), a method for estimating the discounted occupancy ratio in offline reinforcement learning. The occupancy ratio measures how much the target policy's state-action distribution differs from the offline data distribution, and is the central object for correcting distribution shift in off-policy evaluation. Existing methods for estimating this ratio typically require solving minimax problems with a separate critic function class, and their guarantees depend on the critic class being rich enough to detect all Bellman residuals (a completeness condition). FORE instead iterates an adjoint Bellman map and projects each step onto a log-ratio class using Kullback-Leibler (KL) divergence. The key structural insight is that the adjoint Bellman operator contracts relative entropy toward the true ratio at rate gamma (by the data processing inequality for Markov kernels), and KL projection operates in the same geometry. This alignment means the projected iteration inherits the contraction: the population recursion contracts in KL divergence toward the true occupancy ratio up to an approximation error that depends only on how well the ratio class approximates the true ratio itself, not on whether the class is closed under adjoint Bellman updates. The paper proves finite-sample convergence of the empirical version, with error decomposing into a geometrically decaying initialization term, a log-ratio approximation term, and a statistical term governed by the complexity of the ratio hypothesis class. The fitted ratio then supports direct reward reweighting, doubly robust estimation, and occupancy-weighted fitted Q-evaluation, all without Bellman completeness, adjoint Bellman completeness, or critic-class completeness.

Core claim

The adjoint Bellman operator is a contraction in relative entropy (KL divergence) toward the true discounted occupancy ratio, contracting by factor gamma per step via the data processing inequality. Because KL projection onto a normalized exponential ratio class uses the same divergence, the projected operator inherits this contraction. This means the only approximation condition needed is that the ratio class can represent the true occupancy ratio (realizability) — no closure of the class under Bellman updates is required. The paper formalizes this through Theorem 4.1 (population contraction) and Theorem 4.2 (finite-sample convergence), and demonstrates that the resulting ratio can replace,

What carries the argument

adjoint Bellman operator, KL projection, data processing inequality, normalized exponential ratio class, local Rademacher complexity

Load-bearing premise

The adjoint Bellman operator's KL contraction (Lemma 3.1) depends on the data processing inequality applied to the target-policy Markov kernel, which structurally requires that the one-step pushforward of the offline distribution under the target policy be absolutely continuous with respect to the offline distribution itself. If the target policy visits state-action regions that the offline data never explores, the density ratios defining the recursion do not exist and the KL

What would settle it

If the target policy's one-step successor distribution assigns positive probability to state-action regions where the offline data distribution is zero, the Radon-Nikodym derivative defining the adjoint Bellman operator does not exist, and the entire KL contraction argument collapses. This is a coverage requirement shared with other offline RL methods but is structural rather than merely statistical here.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Offline policy evaluation can be performed with a single fitted-iteration loop on density ratios, using standard supervised learning objectives, without requiring a minimax saddle-point solve or a separate critic class.
  • Value-function methods like fitted Q-evaluation can be stabilized by using the FORE-estimated occupancy ratio as a projection weight, recovering contraction of the projected Bellman operator even when the value class is not Bellman complete.
  • The approximation burden in offline RL shifts from closure conditions on function classes (Bellman completeness, critic richness) to direct realizability of the occupancy ratio, which is a statement about the distribution shift between the offline data and the target policy.
  • The doubly robust estimator combining the FORE ratio with a fitted Q-function achieves a product-form error bound: the value error vanishes if either the ratio or the Q-function is correctly specified, and otherwise scales as the product of the two errors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The KL-contraction argument extends to any f-divergence for the adjoint Bellman map (as the paper notes), but KL is singled out because the normalized exponential ratio class makes the projection a convex single-level optimization problem. Other f-divergences may not yield tractable projection steps, limiting practical alternatives.
  • If the target policy visits state-action regions with zero probability under the offline distribution, the method breaks down structurally because the Radon-Nikodym derivatives defining the adjoint Bellman operator do not exist. This coverage requirement is shared with all offline RL methods but is particularly stark here because the entire recursion is built on density ratios.
  • The separation between ratio-realizability and Bellman-completeness suggests a natural two-stage approach for offline policy evaluation: first estimate the occupancy ratio to correct distribution shift, then perform value estimation in the corrected norm. This decomposition may be easier to satisfy in practice than requiring a single function class to be simultaneously realizable and Bellman-compl

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 9 minor

Summary. This paper introduces Fitted Occupancy-Ratio Evaluation (FORE), a fitted-iteration method for estimating the discounted occupancy ratio in offline policy evaluation. The key idea is to iterate the adjoint Bellman operator and project each update onto a log-ratio class in KL divergence. The central theoretical contribution is that the population KL-projected recursion contracts in relative entropy toward the true occupancy ratio at rate gamma, with an approximation error depending only on how well the ratio class approximates the target ratio (not on closure under adjoint Bellman updates). This eliminates the need for Bellman completeness, adjoint Bellman completeness, or critic-class completeness. The paper provides finite-sample guarantees, instantiates the statistical rates for parametric and nonparametric classes, and develops three downstream applications: direct reward reweighting, doubly robust estimation, and occupancy-weighted FQE. Numerical experiments on a Baird-style MRP and a linear-Gaussian system illustrate the theory.

Significance. The paper makes a conceptually clean contribution: it identifies that the adjoint Bellman operator is a KL-contraction via the data processing inequality, and that KL projection onto a normalized exponential family preserves this contraction through the Pythagorean inequality. This yields a fitted-iteration guarantee under mere realizability of the target ratio, replacing the completeness conditions required by both standard FQE and minimax ratio methods. The finite-sample analysis (Theorem 4.2) is non-trivial: it chains uniform empirical process bounds over all fitted iterates (avoiding data splitting), ERM excess control, approximate projection perturbation, and empirical normalization perturbation. The specific rate instantiations for linear and nonparametric classes, the downstream policy-evaluation applications with product-form error bounds, and the numerical experiments that isolate the realizability-vs-completeness distinction add practical value. The comparison with a backward-regression variant (Appendix F) that does require adjoint Bellman completeness is a useful contrast that clarifies the role of the KL projection geometry.

minor comments (9)
  1. Section 3.3, Algorithm 1, Line 4: The objective includes a self-normalized term n^{-1} sum omega(X_i) h(X_i^+) / (n^{-1} sum omega(X_i)). The text below the algorithm states the objective is convex in h for linear classes, but the self-normalization makes the objective non-convex in general (ratio of two affine functions in h). The convexity claim should be qualified or restricted to the case where the denominator is treated as fixed within each iteration (which is the standard practice for alternating optimization).
  2. Theorem 4.2: The bound uses the generalized KL divergence D^gen_nu, which is appropriate since empirical iterates need not integrate to one under nu. However, the downstream results in Section 5 (e.g., Corollary 5.1, Theorem 5.2) use chi-square-type bounds that require both upper and lower bounds on omega_fit. The lower bound on omega_fit is established in Lemma D.6 via e^{-2R}, but this is deterministic only if the empirical normalization denominator stays in [e^{-R}, e^R], which holds almost surely for the population normalization but not automatically for the empirical one. The high-probability event should be stated more carefully in the proof of Corollary 5.1.
  3. Section 6.2: The linear-Gaussian experiment reports that MWL has smaller direct value RMSE than FORE (0.319 vs 0.428 at n=10000), despite FORE having smaller density-ratio L2 error. The paper attributes this to the direct reweighting objective but does not fully explain the discrepancy. A brief discussion of why a better ratio estimate does not translate to better direct value estimation in this example would strengthen the presentation.
  4. Lemma 3.1 proof sketch: The joint convexity step writes D_KL((1-gamma)d0 + gamma(omega nu)P^pi || ...) <= (1-gamma) D_KL(d0||d0) + gamma D_KL((omega nu)P^pi || ...). This is correct but the reader might benefit from an explicit citation to the convexity of KL in its first argument for mixtures (Cover and Thomas, 2006, Eq. 2.7).
  5. Condition C5 (target lower bound m_star > 0): This is used in Lemma C.9 to control the cross term via the bound r(log r)^2 <= C * {r log r - r + 1}. The condition is somewhat strong for continuous state spaces where the target ratio may approach zero. The paper acknowledges this in Section 7, but it would help to note whether this can be relaxed to an L2-type condition on log omega^{pi,gamma} at the cost of weaker rates.
  6. Appendix F (backward-regression variant): The inherent adjoint Bellman error b_M(W) is defined as a supremum over the iterates W_K, which is a random set. The population analysis in Lemma F.3 treats W_K as deterministic. A remark clarifying that this is a population-level analysis (with the empirical analogue requiring additional care) would improve clarity.
  7. Figures 1-5: The y-axis labels in Figures 3 and 5 use scientific notation that is difficult to parse (e.g., '6 x 10^1' and '1e20'). Consider using consistent formatting and ensuring axis labels are readable at standard zoom.
  8. Typographical: 'FORI' appears in several figure captions (Figures 1-5) where 'FORE' is intended. This should be corrected throughout.
  9. Section 4.2, definition of r_{n,fit}: The critical radius is defined as n^{-1/2} vee inf{r > 0 : C_n(r) <= r^2}. The n^{-1/2} floor is standard but should be noted as ensuring the rate is no faster than parametric; this is implicit in Corollary C.11 but not stated in the main text.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee's summary accurately captures the paper's contributions, and the recommendation of minor revision is appropriate. We address each major comment below.

read point-by-point responses

  1. Referee: The referee report contains no major comments. The 'MAJOR COMMENTS' section of the report is empty.

    Authors: We note that the referee's report does not contain any major comments—the 'MAJOR COMMENTS' section is empty. The referee's detailed summary and significance assessment indicate a positive evaluation with a minor revision recommendation. We have carefully reviewed the referee's summary for any implicit concerns or suggestions embedded therein. The summary accurately describes the paper's contributions, methodology, and results. We do not identify any substantive criticisms or requested changes in the report. We are grateful for the referee's thorough reading and positive assessment. If the editor or referee has specific revision suggestions that were intended but not included in the report, we would be happy to address them. revision: no

Circularity Check

0 steps flagged

No significant circularity. The KL-contraction and Pythagorean-projection argument is self-contained; self-citations are non-load-bearing.

full rationale

The paper's central derivation chain is as follows: (1) Lemma 3.1 proves the adjoint Bellman operator B^pi_gamma is a gamma-contraction in KL divergence, using joint convexity of KL and the data processing inequality for the Markov kernel P^pi (Cover and Thomas, 2006 — an external textbook). (2) Lemma 3.2 shows the KL projection of B^pi_gamma omega onto the exponential family W reduces to a single-level loss depending only on initial-state moments and one-step transitions, derived by direct calculation from the occupancy Bellman moment identity (Eq. 3). (3) Theorem 4.1 chains Lemma 3.1 with the Pythagorean inequality for I-projections onto exponential families (Banerjee et al., 2005; Csiszar, 1975 — external references) and a quadratic approximation bound (Lemma B.3) to show the projected recursion contracts toward omega^{pi,gamma} up to epsilon^2_KL. (4) Theorem 4.2 adds finite-sample statistical error via standard local Rademacher complexity tools (Bartlett et al., 2005; Bousquet, 2002; Wainwright, 2019 — all external). The self-citations to van der Laan and Kallus (2025a,b) appear only for: (a) the observation that FQE's norm mismatch causes instability (background motivation, not a proof step), (b) the uniform empirical-process technique to avoid data splitting across fitted iterations (a standard technique attributed also to Hu et al., 2025), and (c) the stationary-weighted FQE framework used in Section 5.2. None of these are load-bearing for the central contraction result (Theorem 4.1) or its finite-sample counterpart (Theorem 4.2), which are derived from first principles using external information-theoretic and empirical-process results. The approximation error epsilon_KL is defined as the best L2 log-ratio approximation error of the target ratio omega^{pi,gamma} (not of Bellman images of arbitrary candidates), and the bound is not tautological: in the realizable case (epsilon_KL = 0), the contraction is strict (gamma^K), which is a non-trivial consequence of the DPI contraction, not a definition. No step reduces to its inputs by construction. The self-citations raise the score to 2 but do not constitute circularity of the central claim.

Axiom & Free-Parameter Ledger

8 free parameters · 7 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The adjoint Bellman operator B^pi_gamma is a mathematical object defined via Radon-Nikodym derivatives of standard measure-theoretic constructs (pushforwards of measures under Markov kernels). The KL-projected operator T^K_W is a composition of this operator with a standard information projection. No new axioms beyond standard measure theory, information theory, and empirical process theory are introduced. The conditions C1-C7 are domain assumptions standard in offline RL (coverage, boundedness, realizability) rather than invented entities.

free parameters (8)
  • gamma (discount factor) = input from problem
    Standard RL discount factor, treated as given input, not fitted.
  • R (bounded centered log-ratio class) = assumed finite
    Condition C4: sup_{h in H} ||h - E_nu h||_inf <= R. Controls empirical-process envelopes and curvature. Not fitted to data but assumed.
  • B_0 (initial ratio bound) = assumed finite
    Condition C4: ||omega_0||_inf <= B_0. Coverage constant for initial distribution.
  • B_+ (one-step coverage bound) = assumed finite
    Condition C4: ||d_nu^+_pi / d_nu||_inf <= B_+. Coverage constant for one-step target transitions.
  • m_star (target ratio lower bound) = assumed > 0
    Condition C5: omega^{pi,gamma}(x) >= m_star. Ensures log-ratio is well-behaved. Not fitted.
  • M_star (target ratio upper bound) = assumed finite
    Condition C6: ||omega^{pi,gamma}||_inf <= M_star. Used in Theorem 5.3 for FORE-weighted FQE.
  • K (iteration count) = chosen >= log(n)/log(2/(1+gamma))
    Number of fitted iterations. Theorem 4.2 requires K large enough to make the initialization term O(1/n). Not a fitted parameter but a tuning choice.
  • epsilon_KL (approximation error) = inf_{v in W} ||log omega^{pi,gamma} - log v||_{L2(nu)}
    Population approximation error of the ratio class. Zero under realizability. Not fitted but determined by the choice of hypothesis class H.
axioms (7)
  • standard math Data processing inequality for Markov kernels: D_KL(mu P || xi P) <= D_KL(mu || xi)
    Invoked in Lemma 3.1 proof to establish KL-contraction of the adjoint Bellman operator. Standard result from Cover and Thomas (2006).
  • standard math Joint convexity of KL divergence
    Used in Lemma 3.1 to decompose the mixture KL divergence into the (1-gamma) d_0 component and the gamma pushforward component. Standard.
  • standard math Pythagorean inequality for KL projections onto exponential families
    Invoked in Theorem 4.1 proof sketch and Lemma B.2 to show D_nu(T^K_W omega || omega*) <= D_nu(B^pi_gamma omega || omega*). From Banerjee et al. (2005) and Csiszar (1975).
  • domain assumption One-step target coverage (Condition C1): d_0 << nu and nu^+_pi << nu
    Section 2.1. Required for the Radon-Nikodym derivative defining the adjoint Bellman operator to exist. Standard coverage assumption in offline RL.
  • domain assumption Log square-integrability of target ratio (Condition C3): omega^{pi,gamma} > 0 and log omega^{pi,gamma} in L^2(nu)
    Section 4.1. Required for the KL approximation floor to be quadratic in log-ratio error. Specific to the log-ratio modeling choice.
  • domain assumption Bounded log-ratio class (Condition C4): sup_h ||h - E_nu h||_inf <= R
    Section 4.1. Controls empirical-process envelopes and local quadratic curvature. Strong for neural network classes but standard in fitted-iteration analyses.
  • standard math Local Rademacher complexity critical radius controls statistical error
    Section 4.2, equation (7). Standard empirical process theory (Bartlett et al., 2005; Wainwright, 2019).

pith-pipeline@v1.1.0-glm · 52301 in / 4183 out tokens · 238115 ms · 2026-07-07T14:02:49.549770+00:00 · methodology

0 comments
read the original abstract

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Figures

Figures reproduced from arXiv: 2607.05375 by Lars van der Laan, Nathan Kallus.

Figure 1
Figure 1. Figure 1: illustrates the population recursions. The population FORE KL recursion converges to the true ratio. In contrast, under the offline data distribution, the projected linear FQE recursion has scalar multiplier 2.103, so coefficient errors are amplified across iterations. Using the FORE ratio as the FQE projection weight changes this multiplier to 0.801. Tabular FQE is included as a Bellman-complete benchmark… view at source ↗
Figure 2
Figure 2. Figure 2: Linear-Gaussian population recursions. The left panel shows policy-value error for linear FQE, direct FORE, and FORE-reweighted FQE. The right panel shows FORE occupancy error. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Linear-Gaussian finite-sample value error. Curves report value RMSE over 300 repetitions; vertical bars indicate Monte Carlo uncertainty. 500 1000 2000 5000 10000 Offline transitions 0.3 0.4 0.5 0.75 1 1.25 1.5 2 6 × 10 Value RMSE 1 Direct value error FORI RFF-RBF MWL RFF-RBF DualDICE 500 1000 2000 5000 10000 Offline transitions 0.125 0.15 0.2 0.25 0.3 0.4 D e nsity-ratio L 2 error Ratio error FORI RFF-RBF… view at source ↗
Figure 4
Figure 4. Figure 4: Direct ratio estimators in the linear-Gaussian experiment. The left panel reports value RMSE from direct reward reweighting; the right panel reports empirical L 2 (ν) error of the fitted density ratio [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Linear-Gaussian value error as the discount varies. Curves report value RMSE over 500 repetitions at n = 5000, with the horizontal axis scaled by the effective horizon (1 − γ) −1 . Linear FQE is evaluated after the same fixed number of fitted updates at every discount. 7. Conclusion FORE formulates discounted occupancy-ratio estimation as a fitted adjoint Bellman problem. Rather than solving a ratio–critic… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

97 extracted references · 97 canonical work pages · 15 internal anchors

  1. [1]

    Proceedings of The 33rd International Conference on Machine Learning , pages =

    Doubly Robust Off-policy Value Evaluation for Reinforcement Learning , author =. Proceedings of The 33rd International Conference on Machine Learning , pages =. 2016 , editor =

    work page 2016

  2. [2]

    Proceedings of The 33rd International Conference on Machine Learning , pages =

    Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning , author =. Proceedings of The 33rd International Conference on Machine Learning , pages =. 2016 , editor =

    work page 2016

  3. [3]

    Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling , url =

    Xie, Tengyang and Ma, Yifei and Wang, Yu-Xiang , booktitle =. Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling , url =

    work page

  4. [4]

    arXiv preprint arXiv:2602.21501 , year=

    A Researcher's Guide to Empirical Risk Minimization , author=. arXiv preprint arXiv:2602.21501 , year=

    work page arXiv

  5. [5]

    International conference on machine learning , pages=

    Information-theoretic considerations in batch reinforcement learning , author=. International conference on machine learning , pages=. 2019 , organization=

    work page 2019

  6. [6]

    Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =

    Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning , author =. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =. 2020 , editor =

    work page 2020

  7. [7]

    Journal of Machine Learning Research , year =

    Nathan Kallus and Masatoshi Uehara , title =. Journal of Machine Learning Research , year =

    work page

  8. [8]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Double Reinforcement Learning for Efficient and Robust Off-Policy Evaluation , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    work page 2020

  9. [9]

    Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning , url =

    Kallus, Nathan and Uehara, Masatoshi , booktitle =. Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning , url =

    work page

  10. [10]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Statistically Efficient Off-Policy Policy Gradients , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    work page 2020

  11. [11]

    Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies , url =

    Kallus, Nathan and Uehara, Masatoshi , booktitle =. Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies , url =

    work page

  12. [12]

    Nathan Kallus and Masatoshi Uehara , title =. Oper. Res. , volume =. 2022 , url =. doi:10.1287/OPRE.2021.2249 , timestamp =

    work page doi:10.1287/opre.2021.2249 2022

  13. [13]

    Biometrika , volume =

    Kallus, Nathan and Uehara, Masatoshi , title =. Biometrika , volume =. 2024 , doi =

    work page 2024

  14. [14]

    A Review of Off-Policy Evaluation in Reinforcement Learning

    Masatoshi Uehara and Chengchun Shi and Nathan Kallus , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2212.06355 , eprinttype =. 2212.06355 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2212.06355 2022

  15. [15]

    arXiv preprint arXiv:2501.06926 , year=

    Semiparametric double reinforcement learning with applications to long-term causal inference , author=. arXiv preprint arXiv:2501.06926 , year=

    work page arXiv

  16. [16]

    European conference on machine learning , pages=

    Neural fitted Q iteration--first experiences with a data efficient neural reinforcement learning method , author=. European conference on machine learning , pages=. 2005 , organization=

    work page 2005

  17. [17]

    International Conference on Machine Learning , pages=

    Boosted fitted q-iteration , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    work page 2017

  18. [18]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Reinforcement Learning with Deep Energy-Based Policies , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =

    work page 2017

  19. [19]

    Proceedings of the 35th International Conference on Machine Learning , pages =

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , author =. Proceedings of the 35th International Conference on Machine Learning , pages =. 2018 , editor =

    work page 2018

  20. [20]

    A Theory of Regularized

    Geist, Matthieu and Scherrer, Bruno and Pietquin, Olivier , booktitle =. A Theory of Regularized. 2019 , editor =

    work page 2019

  21. [21]

    Inverse Reinforcement Learning with Just Classification and a Few Regressions

    Lars van der Laan and Nathan Kallus and Aur. Inverse Reinforcement Learning Using Just Classification and a Few Regressions , journal =. 2025 , url =. doi:10.48550/ARXIV.2509.21172 , eprinttype =. 2509.21172 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.21172 2025

  22. [22]

    Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models , journal =

    Lars van der Laan and Aur. Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models , journal =. 2025 , url =. doi:10.48550/ARXIV.2512.24407 , eprinttype =. 2512.24407 , timestamp =

    work page doi:10.48550/arxiv.2512.24407 2025

  23. [23]

    Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation , url =

    Liu, Qiang and Li, Lihong and Tang, Ziyang and Zhou, Dengyong , booktitle =. Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation , url =

    work page

  24. [24]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Consistent On-Line Off-Policy Evaluation , author =. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =

    work page 2017

  25. [25]

    Sutton and A

    Richard S. Sutton and A. Rupam Mahmood and Martha White , title =. Journal of Machine Learning Research , year =

    work page

  26. [26]

    Bellemare , title =

    Carles Gelada and Marc G. Bellemare , title =. The Thirty-Third. 2019 , url =. doi:10.1609/AAAI.V33I01.33013647 , timestamp =

    work page doi:10.1609/aaai.v33i01.33013647 2019

  27. [27]

    DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections , url =

    Nachum, Ofir and Chow, Yinlam and Dai, Bo and Li, Lihong , booktitle =. DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections , url =

    work page

  28. [28]

    8th International Conference on Learning Representations,

    Ruiyi Zhang and Bo Dai and Lihong Li and Dale Schuurmans , title =. 8th International Conference on Learning Representations,. 2020 , url =

    work page 2020

  29. [29]

    2020 , editor =

    Zhang, Shangtong and Liu, Bo and Whiteson, Shimon , booktitle =. 2020 , editor =

    work page 2020

  30. [30]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Minimax Weight and Q-Function Learning for Off-Policy Evaluation , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , editor =

    work page 2020

  31. [31]

    Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency

    Masatoshi Uehara and Masaaki Imaizumi and Nan Jiang and Nathan Kallus and Wen Sun and Tengyang Xie , title =. CoRR , volume =. 2021 , url =. 2102.02981 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  32. [32]

    Off-Policy Evaluation via the Regularized Lagrangian , url =

    Yang, Mengjiao and Nachum, Ofir and Dai, Bo and Li, Lihong and Schuurmans, Dale , booktitle =. Off-Policy Evaluation via the Regularized Lagrangian , url =

    work page

  33. [33]

    CoinDICE: Off-Policy Confidence Interval Estimation , url =

    Dai, Bo and Nachum, Ofir and Chow, Yinlam and Li, Lihong and Szepesvari, Csaba and Schuurmans, Dale , booktitle =. CoinDICE: Off-Policy Confidence Interval Estimation , url =

    work page

  34. [34]

    Rupam , journal=

    Che, Fengdi and Chan, Bryan and Ma, Chen and Mahmood, A. Rupam , journal=

    work page

  35. [35]

    AlgaeDICE: Policy Gradient from Arbitrary Experience

    Ofir Nachum and Bo Dai and Ilya Kostrikov and Yinlam Chow and Lihong Li and Dale Schuurmans , title =. CoRR , volume =. 2019 , url =. 1912.02074 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  36. [36]

    8th International Conference on Learning Representations,

    Ilya Kostrikov and Ofir Nachum and Jonathan Tompson , title =. 8th International Conference on Learning Representations,. 2020 , url =

    work page 2020

  37. [37]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    work page 2021

  38. [38]

    Mankowitz and Nicolas Heess and Doina Precup and Kee

    Jongmin Lee and Cosmin Paduraru and Daniel J. Mankowitz and Nicolas Heess and Doina Precup and Kee. COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation , booktitle =. 2022 , url =

    work page 2022

  39. [39]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    work page 2022

  40. [40]

    LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation

    Geon. LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation , journal =. 2022 , url =. 2202.13536 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    Lagoudakis and Ronald Parr , title =

    Michail G. Lagoudakis and Ronald Parr , title =. J. Mach. Learn. Res. , volume =. 2003 , url =

    work page 2003

  42. [42]

    Journal of Machine Learning Research , year =

    Damien Ernst and Pierre Geurts and Louis Wehenkel , title =. Journal of Machine Learning Research , year =

    work page

  43. [43]

    Fitted Q-iteration in continuous action-space MDPs , url =

    Antos, Andr\'. Fitted Q-iteration in continuous action-space MDPs , url =. Advances in Neural Information Processing Systems , editor =

    work page

  44. [44]

    The Annals of Statistics , volume=

    Projected state-action balancing weights for offline reinforcement learning , author=. The Annals of Statistics , volume=. 2023 , publisher=

    work page 2023

  45. [45]

    Finite-Time Bounds for Fitted Value Iteration , journal =

    R. Finite-Time Bounds for Fitted Value Iteration , journal =. 2008 , volume =

    work page 2008

  46. [46]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Batch Policy Learning under Constraints , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    work page 2019

  47. [47]

    Fitted $Q$ Evaluation Without Bellman Completeness via Stationary Weighting

    Lars van der Laan and Nathan Kallus , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2512.23805 , eprinttype =. 2512.23805 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2512.23805 2025

  48. [48]

    Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration

    Lars van der Laan and Nathan Kallus , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2512.23927 , eprinttype =. 2512.23927 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2512.23927 2025

  49. [49]

    Journal of Machine Learning Research , year =

    Andrew Patterson and Adam White and Martha White , title =. Journal of Machine Learning Research , year =

    work page

  50. [50]

    Proceedings of the 2008

    Tsuboi, Yuta and Kashima, Hisashi and Hido, Shohei and Bickel, Steffen and Sugiyama, Masashi , title =. Proceedings of the 2008. 2008 , doi =

    work page 2008

  51. [51]

    Correcting Sample Selection Bias by Unlabeled Data , url =

    Huang, Jiayuan and Gretton, Arthur and Borgwardt, Karsten and Sch\". Correcting Sample Selection Bias by Unlabeled Data , url =. Advances in Neural Information Processing Systems , editor =

    work page

  52. [52]

    Journal of Machine Learning Research , year =

    Takafumi Kanamori and Shohei Hido and Masashi Sugiyama , title =. Journal of Machine Learning Research , year =

    work page

  53. [53]

    Wainwright and Michael I

    XuanLong Nguyen and Martin J. Wainwright and Michael I. Jordan , title =. 2010 , url =. doi:10.1109/TIT.2010.2068870 , timestamp =

    work page doi:10.1109/tit.2010.2068870 2010

  54. [54]

    Proceedings of The 33rd International Conference on Machine Learning , pages =

    Linking losses for density ratio and class-probability estimation , author =. Proceedings of The 33rd International Conference on Machine Learning , pages =. 2016 , editor =

    work page 2016

  55. [55]

    2012 , isbn =

    Sugiyama, Masashi and Suzuki, Taiji and Kanamori, Takafumi , title =. 2012 , isbn =

    work page 2012

  56. [56]

    Puterman, Martin L. , year=. Markov Decision Processes: Discrete Stochastic Dynamic Programming , ISBN=. doi:10.1002/9780470316887 , journal=

    work page doi:10.1002/9780470316887

  57. [57]

    and Tweedie, Richard L

    Meyn, Sean P. and Tweedie, Richard L. , year=. Markov Chains and Stochastic Stability , ISBN=. doi:10.1017/cbo9780511626630 , publisher=

    work page doi:10.1017/cbo9780511626630

  58. [58]

    and Meyn, Sean P

    Glynn, Peter W. and Meyn, Sean P. , year=. A Liapounov bound for solutions of the Poisson equation , volume=. The Annals of Probability , publisher=. doi:10.1214/aop/1039639370 , number=

    work page doi:10.1214/aop/1039639370

  59. [59]

    Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

    Cameron Voloshin and Hoang Minh Le and Nan Jiang and Yisong Yue , title =. CoRR , volume =. 2019 , url =. 1911.06854 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  60. [60]

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning

    Justin Fu and Aviral Kumar and Ofir Nachum and George Tucker and Sergey Levine , title =. CoRR , volume =. 2020 , url =. 2004.07219 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  61. [61]

    Learning density ratios in causal inference using Bregman-Riesz regression

    Learning density ratios in causal inference using Bregman-Riesz regression , author=. arXiv preprint arXiv:2510.16127 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  62. [62]

    and Bousquet, Olivier and Mendelson, Shahar , title =

    Bartlett, Peter L. and Bousquet, Olivier and Mendelson, Shahar , title =. The Annals of Statistics , volume =. 2005 , doi =

    work page 2005

  63. [63]

    Comptes Rendus Mathematique , volume =

    A Bennett concentration inequality and its application to suprema of empirical processes , author =. Comptes Rendus Mathematique , volume =. 2002 , publisher =

    work page 2002

  64. [64]

    and Wellner, Jon A

    van der Vaart, Aad W. and Wellner, Jon A. , title =. Electronic Journal of Statistics , volume =

    work page

  65. [65]

    Bracketing Metric Entropy Rates and Empirical Central Limit Theorems for Function Classes of

    Nickl, Richard and P. Bracketing Metric Entropy Rates and Empirical Central Limit Theorems for Function Classes of. Journal of Theoretical Probability , volume =

    work page

  66. [66]

    Interpolation between

    Molchanova, Anastasia and Roskovec, Tom. Interpolation between. Journal of Mathematical Analysis and Applications , volume =. 2018 , doi =

    work page 2018

  67. [67]

    Leoni, Giovanni , title =

    work page

  68. [68]

    , title =

    Wainwright, Martin J. , title =. 2019 , doi =

    work page 2019

  69. [69]

    and Thomas, Joy A

    Cover, Thomas M. and Thomas, Joy A. , title =. 2006 , isbn =

    work page 2006

  70. [70]

    2011 , doi =

    Brezis, Haim , title =. 2011 , doi =

    work page 2011

  71. [71]

    CoRR , volume =

    Raginsky, Maxim , title =. CoRR , volume =. 2014 , url =

    work page 2014

  72. [72]

    Bregman, L. M. , title =. USSR Computational Mathematics and Mathematical Physics , volume =. 1967 , doi =

    work page 1967

  73. [73]

    and Hirayama, Jun-ichiro , title =

    Gutmann, Michael U. and Hirayama, Jun-ichiro , title =. Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , pages =. 2011 , publisher =

    work page 2011

  74. [74]

    Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation , booktitle =

    Sugiyama, Masashi and Nakajima, Shinichi and Kashima, Hisashi and von B. Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation , booktitle =. 2007 , publisher =

    work page 2007

  75. [75]

    Annals of the Institute of Statistical Mathematics , volume =

    Sugiyama, Masashi and Suzuki, Taiji and Kanamori, Takafumi , title =. Annals of the Institute of Statistical Mathematics , volume =. 2012 , doi =

    work page 2012

  76. [76]

    Advances in Neural Information Processing Systems , volume=

    Minimax estimation of conditional moment models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  77. [77]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=

    Inference on strongly identified functionals of weakly identified functions , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=. 2025 , publisher=

    work page 2025

  78. [78]

    Source Condition Double Robust Inference on Functionals of Inverse Problems

    Source condition double robust inference on functionals of inverse problems , author=. arXiv preprint arXiv:2307.13793 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  79. [79]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor =

    work page 2021

  80. [80]

    Proceedings of the 37th AAAI Conference on Artificial Intelligence , year =

    Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction , author =. Proceedings of the 37th AAAI Conference on Artificial Intelligence , year =

    work page

Showing first 80 references.