Pith. sign in

REVIEW 3 major objections 3 minor 31 references

The paper settles an open question in offline RL: Q*-realizability and Bellman completeness alone do not guarantee sample-efficient learning under partial coverage, and it introduces a modular decision-estimation framework that reveals what

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Q*-realizability plus Bellman completeness is insufficient for sample-efficient offline RL under partial coverage, and a new decision-estimation framework recovers and improves existing bounds.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection The negative answer for (s,a,r,s') data is sound and closes the open question; the trajectory-feedback half of Theorem 1 overclaims what Appendix A.4 proves, so the paper needs a weakened statement or a real proof there. the 3 major comments →

arxiv 2602.12107 v2 pith:TOCVRHPX submitted 2026-02-12 cs.LG cs.AIstat.ML

On the Complexity of Offline Reinforcement Learning with Q^star-Approximation and Partial Coverage

Haolin Liu , Braham Snyder , Chen-Yu Wei This is my paper
classification cs.LG cs.AIstat.ML MSC 68Q3290C40
keywords offline reinforcement learningQ-star realizabilityBellman completenesspartial coverageinformation-theoretic lower bounddecision-estimation coefficientsoft Q-learningsample complexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning with only partial data coverage is a central practical problem. This paper answers a long-standing open question in the negative: the two standard assumptions used to justify value-based offline algorithms—Q*-realizability and Bellman completeness—are not sufficient for sample-efficient learning under partial coverage. The proof is an information-theoretic lower bound exhibiting a family of MDPs that satisfies both assumptions with constant coverage, yet forces any algorithm to use Ω(1/ε · min{√|S|, 1/Δ²}) samples. The paper then builds a general decision-estimation framework that decomposes sub-optimality into a decision complexity term and a value estimation term, recovering and improving prior bounds, giving the first ε^{-2} rate for soft Q-learning under partial coverage, and providing a first non-tabular guarantee for the practical Conservative Q-Learning algorithm.

Core claim

The central claim is a negative answer: under Q*-realizability and Bellman completeness alone, partial coverage can make sample-efficient offline RL impossible. The lower-bound construction uses a family of MDPs in which a hidden assignment of middle-layer states to two groups determines the high-reward branch, while every function in the four-element function class takes the same value on all middle-layer states; this keeps the class Bellman complete but prevents the learner from using it to resolve which initial action leads to the good branch. Beyond the lower bound, the paper proves a general decomposition of sub-optimality into decision complexity and value estimation error, and shows t

What carries the argument

The lower bound rests on a Bellman-complete function class of size four that is deliberately constant on all middle-layer states; this keeps Q*-realizability and Bellman completeness satisfied while forcing the learner to guess the hidden assignment from noisy rewards. On the positive side, the central objects are the Offline Robust Decision-Estimation Coefficient (Ordec), minimized by the E2D.OR algorithm, and a second-order performance difference lemma that bounds the policy gap by a Bregman divergence in regularized MDPs, yielding the ε^{-2} soft Q-learning guarantee. For low-Bellman-rank MDPs, two assumptions—double policy sampling and policy feature coverage—are introduced and each is s

Load-bearing premise

The lower bound only bites because the function class is deliberately constant on the middle-layer states—if it could distinguish the hidden groups, the problem would become learnable, so the hardness hinges on the extreme un-informativeness of the function class on the unobserved part of the state space.

What would settle it

Run the paper's four-family MDP construction with |W_A|=|W_B|=m and attempt to achieve sub-optimality ε=1/4 with fewer than Ω(min{√m, 1/Δ²}) samples; any algorithm succeeding would refute the lower bound. Alternatively, enrich the function class with a single feature that distinguishes the two hidden groups and check that a polynomial-sample algorithm appears, confirming that the constancy of the class is the load-bearing premise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any offline RL algorithm relying solely on Q*-realizability and Bellman completeness can be forced to require Ω(1/ε · min{√|S|, 1/Δ²}) samples under partial coverage, so additional structural conditions are unavoidable.
  • Soft Q-learning with a strictly convex regularizer achieves ε^{-2} sample complexity under partial coverage, matching rates previously available only in simpler settings.
  • Low-Bellman-rank MDPs become learnable offline only when both double policy samples and policy feature coverage are present; dropping either makes polynomial sample complexity impossible.
  • The Conservative Q-Learning objective has a provable guarantee in non-tabular function approximation settings under Q*-realizability and Bellman completeness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the function class could distinguish the hidden groups—for instance with linear feature mappings—the lower-bound construction would collapse, suggesting that the true boundary of learnability is whether the function class can resolve ambiguity that the data alone cannot.
  • The decision-estimation decomposition should transfer to other approximation schemes such as density-ratio or Q-pi-realizability, possibly unifying offline RL theory beyond the Q-star setting.
  • A concrete testable corollary: adding a small KL-regularization term to the hard instance should break the symmetry that makes the function class uninformative and restore polynomial sample complexity.
  • The 1/Δ² dependence indicates that the gap-dependent guarantees in earlier offline RL work are not artifacts of the analysis but genuine requirements imposed by partial coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper studies offline RL under Q*-realizability and Bellman completeness with partial coverage. Its headline contribution is a negative answer to the open question whether these two assumptions suffice for sample-efficient offline RL: Theorem 1 constructs a family of MDPs with |F|=4 and C^{pi*}=Theta(1) for which the learner must see n >= Omega((1/epsilon) min{sqrt{|S|}, 1/Delta^2}) i.i.d. (s,a,r,s') samples. The paper then introduces an offline robust decision-estimation coefficient (Ordec) that decomposes suboptimality into a decision term and a value-estimation term. This framework is used to recover and improve existing guarantees: it removes the need for prior knowledge of the value gap in Chen and Jiang (2022), obtains the first epsilon^{-2} bound for soft Q-learning under partial coverage, and provides the first analysis of CQL with function approximation. For low-Bellman-rank MDPs, the paper claims a first characterization of offline learnability and identifies double policy sampling and policy feature coverage as necessary conditions.

Significance. The central (s,a,r,s') lower bound is a substantial result: if correct, it settles the open question and explains why practical assumptions such as gaps, regularization, or feature coverage are needed. The construction is careful; notably, the constancy f(w,a)=1 on middle-layer states is not an artificial weakness because Q* is genuinely constant there in the constructed family. The Ordec framework is a valuable conceptual contribution, and the second-order performance difference lemma is elegant and yields real improvements over prior work. The first function-approximation analysis of CQL is also a useful step. These strengths merit serious consideration. However, three load-bearing issues need attention: the trajectory-feedback part of Theorem 1 is overclaimed relative to its proof, the concentration bound in Lemma 11 is too optimistic by a factor of H, and the claim of a 'characterization of offline learnability' for low-Bellman-rank MDPs is not supported because no bound on the decision complexity Ordec is given for that setting.

major comments (3)
  1. [Theorem 1 / Appendix A.4] The trajectory-feedback lower bound as stated is not supported by the proof. In Appendix A.4 the argument is: if T trajectories pass through s1, then with probability (1-2^{-(H-4)})^T all these trajectories choose action 2 in at least one repeated middle layer; when T = O(2^{H-4}) this event has constant probability, and conditioned on it the (s,a,r,s') lower bound applies, giving T >= Omega((1/epsilon) min{sqrt{|S|}/H, 1/Delta^2}). The resulting bound is Omega(min{2^{H-4}, (1/epsilon) min{sqrt{|S|}/H, 1/Delta^2}}), not the claimed Omega((1/epsilon) min{2^H, sqrt{|S|}, 1/Delta^2}). For Delta=Theta(1), |S| large and epsilon small, the statement overclaims by a factor of about 1/epsilon. Please weaken the statement to match the proof or supply a different argument.
  2. [Section 6.3, Lemma 11] The concentration bound in Lemma 11 is off by a factor of H. With f in [0,H] and r in [0,1], each factor f(s,a)-r-f(s') lies in [-H-1, H], so the product in L_br(f) has range Theta(H^2). Hoeffding's inequality therefore gives |L_br(f)-E L_br(f)| = O(H^2 sqrt(log(|F|/delta)/n)), not O(H sqrt(log(|F|/delta)/n)) as stated. Using the stated epsilon_stat, the event that Q* belongs to F_conf may fail with non-negligible probability, and the bound on max_f D^{pi*}_av is correspondingly too optimistic. The proof should be corrected, e.g. by increasing epsilon_stat to H^2 sqrt(log/n) or by a more careful Bernstein argument; this affects the quantitative low-Bellman-rank guarantee.
  3. [Abstract / Section 6.3] The claim of a 'first characterization of offline learnability for general low-Bellman-rank MDPs' is stronger than what is proved. Lemma 11 controls only the estimation error max_{f in F_conf} D^{pi*}_av under Assumptions 4-5 and Definition 4. It does not bound the decision complexity Ordec (Section 5), and no theorem in Section 5 provides a bounded Ordec for low-Bellman-rank MDPs. Consequently Theorem 2 does not yield a sample-efficient policy-learning guarantee in this setting. Lemmas 23-24 establish necessity of the two assumptions, but they do not establish sufficiency. Please add the missing decision-complexity bound or revise the claim to 'characterization of the estimation error' / 'necessary conditions for polynomial estimation'.
minor comments (3)
  1. [Section 2 vs Appendix A.3] The general definition says the dataset is n i.i.d. tuples from mu, but the lower-bound construction uses three blocks (D1, D2, D3) of n tuples each. This should be reconciled, for example by defining mu as a uniform mixture over the three components and conditioning on the observed counts; otherwise the lower bound is formally for a different data model.
  2. [Theorem 1] In the trajectory part of Theorem 1, 'n' is used interchangeably for tuples and trajectories. Please use a separate symbol (e.g., T) for trajectories to avoid ambiguity.
  3. [Figures 2 and 4] The figures are referenced in the text, but the captions do not fully specify all transition probabilities and rewards, so the reader must reconstruct them from the appendix. Please make the figures self-contained or add a complete specification in the caption.

Circularity Check

0 steps flagged

No significant circularity: the lower-bound construction and the Ordec bounds are derived from independent quantities, not from definitions that assume the target result.

full rationale

The paper's central negative answer is an information-theoretic lower bound built from a four-function class with f(w,a)=1 on middle-layer states. This constancy is not an artificial assumption smuggled in to force hardness: in the constructed MDPs, the true Q* is genuinely constant on those states (Appendix A.2 verifies Bellman completeness by computing T^M Q3 = f2 and T^M f2 in {f1,g1}). The learner's confusion is induced by the hidden assignment phi, which is not encoded in the function class or in the observed data; the lower bound then follows from a TV/assignment-counting argument (Lemmas 14-16). This is a structural construction, not a definition of the form X derives Y where X is defined in terms of Y. The Ordec framework is a minimax complexity measure defined in (3)-(4), and Theorem 2 is an immediate decomposition using M* in M_conf. The subsequent bounds (Theorems 4-6, Lemmas 8-11) bound Ordec via the exploitability ratio, performance-difference lemmas, and confidence-set estimation error. These ingredients are independent of the final guarantees; no fitted parameter is relabeled as a prediction. The confidence sets in Section 6 are constructed from concentration inequalities over the data distribution, and the estimation error bounds are expressed in terms of coverage and log-cardinalities, not in terms of the target sub-optimality. The paper does cite the authors' own prior work (Liu et al. 2025b for online DEC; Jia et al. 2024 for the trajectory re-sampling reduction). These citations are used as inspirational or constructional support, not as the sole load-bearing premise: the offline DEC definitions, the bounds, and the trajectory extension's probability argument are carried out in this paper. The Jia et al. reduction is an externally published result, so it constitutes real evidence rather than a self-referential loop. One admitted concern is the trajectory-feedback part of Theorem 1: the appendix's argument appears to yield a bound with min{2^{H-4}, (1/epsilon)*...} rather than the stated (1/epsilon)*min{2^H,...}, because the all-mixed event has probability exp(-T/2^H). That is a potential correctness gap in the proof, but it is not a circularity under the specified definitions: it does not reduce the claimed result to an input by construction, nor does it rely on a self-citation to define away the difficulty. Per the hard rules, correctness risk should not be scored as circularity. Therefore no circular step can be exhibited with the

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The paper introduces new mathematical constructs (Ordec, ER, policy feature coverage, double policy sampling) but no new physical or empirical entities. The free parameters are hyperparameters and assumption constants, not fits to data. The main assumptions are standard in offline RL theory; the lower bound's construction relies heavily on the constancy of f2 on middle-layer states, which is a modeling choice rather than a general assumption.

free parameters (3)
  • γ (E2D.OR offset version) = √(n/H) in the analysis
    Algorithm hyperparameter trading decision complexity vs estimation error; chosen for the stated bound, not fit to data.
  • λ (CQL penalty) = tuned in the bound; appears in Theorem 12
    Regularization weight in the CQL objective; the bound holds with the displayed choice after optimization.
  • α (KL regularizer weight) = tuned in the bound; appears in Lemma 8 and Theorem 12
    Weight of the Bregman/entropy regularizer; chosen to balance the final sample complexity.
axioms (7)
  • domain assumption Q*-realizability (Definition 2)
    The learner has access to a function class F containing the optimal Q-function of the true MDP; this is a standard starting point for value-based offline RL.
  • domain assumption Bellman completeness (Assumption 1)
    The Bellman backup of any f∈F lies in a known class G (often F itself). Used throughout Sections 5-7 and in the lower bound construction.
  • domain assumption Weight realizability (Assumption 3)
    The density ratio w^{π*}(s,a)=d^{π*}(s,a)/(Hμ(s,a)) lies in a class W with ∥w∥∞≤B_W. Used for estimation error bounds in Lemma 10 and Appendix D.1.
  • domain assumption Legendre regularizer with constants (Assumption 2)
    Requires Bregman symmetry and KL-Bregman comparison with constants C_ψ^1, C_ψ^2; holds for Shannon entropy, Tsallis entropy, and log-barrier. Needed for the second-order performance difference lemma and the ε^{-2} soft Q-learning result.
  • domain assumption Q-Bellman rank (Assumption 4) and double policy sampling (Assumption 5)
    Assumptions needed for the low-Bellman-rank offline characterization (Section 6.3). Lemma 23/24 show each is necessary for polynomial sample complexity.
  • domain assumption Admissibility of μ in CQL analysis
    Theorem 12 assumes μ is a flow: ∑_{(s,a)∈S_h×A} μ(s,a)P(s'|s,a)=μ(s') for s'∈S_{h+1}. This stationarity condition is needed for the pessimistic propagation argument in Lemma 26.
  • standard math Standard concentration and martingale inequalities (Freedman, Hoeffding), performance difference lemma
    Used throughout the proof appendices to control estimation errors and performance gaps.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Complexity of Offline Reinforcement Learning with $Q^\star$-Approximation and Partial Coverage." pith.science (2026). https://pith.science/paper/TOCVRHPX

@misc{pith2026260212107,
  author       = {Pith},
  title        = {Pith review of: On the Complexity of Offline Reinforcement Learning with $Q^\star$-Approximation and Partial Coverage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TOCVRHPX}},
  note         = {Machine review of arXiv:2602.12107}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We study offline reinforcement learning under $Q^\star$-approximation and partial coverage, a setting that motivates practical algorithms such as Conservative $Q$-Learning (CQL; Kumar et al., 2020) but has received limited theoretical attention. Our work is inspired by the following open question: "Are $Q^\star$-realizability and Bellman completeness sufficient for sample-efficient offline RL under partial coverage?" We answer in the negative via an information-theoretic lower bound. To identify additional structure that enables sample-efficient offline RL under partial coverage, we introduce a general decision-estimation framework, inspired by model-free decision-estimation coefficients (DEC) for online RL (Foster et al., 2023b; Liu et al., 2025b). Our framework decomposes offline RL complexity into decision complexity and value estimation error. This allows modular study of both sub-problems. Our result not only unifies existing results (Chen and Jiang, 2022; Uehara et al., 2023), but further improves and generalizes them. On the decision complexity side, our improvement includes: the first $\epsilon^{-2}$ sample complexity bound for soft $Q$-learning under partial coverage that improves Uehara et al.'s (2023) $\epsilon^{-4}$ bound, the removal of the need for additional online interaction in the value-gap setting of Chen and Jiang (2022), and new learnable settings beyond the above two cases. On the value estimation side, we provide a new characterization of the role of Bellman completeness under partial coverage, and the first characterization of offline learnability for general low-Bellman-rank MDPs (Jiang et al., 2017; Du et al., 2021; Jin et al., 2021). The latter is a canonical online RL setting that has remained unexplored in offline RL except for special cases. As a side contribution, our techniques give the first analysis of CQL in the function approximation setting.

Figures

Figures reproduced from arXiv: 2602.12107 by Braham Snyder, Chen-Yu Wei, Haolin Liu.

Figure 1
Figure 1. Figure 1: Comparison of DEC objectives in online and offline settings for a given discrepancy [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Four classes of MDPs: Mu,x, Mu,y, Mv,x, and Mv,y. Red text highlights the differences between classes, with each class named according to the action pair (u/v, x/y) in the top branch. Appendix A. Proofs in Section 3 A.1. MDP construction We construct four families of finite-horizon MDPs as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Construction for ϵ-dependent lower bound with non-trajectory data To lift that lower bound construction to an ϵ-dependent bound for an arbitrary ϵ ∈ (0, 1 4 ], we extend it in the way illustrated in [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Construction for ϵ-dependent lower bound with trajectory data (extended from [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The construction of Jia et al. (2024). The numbers on the left of the green allows specify the initial state distribution. There are two actions. Taking either action on state in Xh ∪ Yh ∪ {zh} leads to transitions to uh+1 and vh+1 with probabilities P(uh+1|s, a) + P(vh+1|s, a) = 2 H which are not specified in the figure. Blue arrow specifies the transition if taking action 1 besides those to uh+1 and vh+1… view at source ↗
Figure 6
Figure 6. Figure 6: A hard instance for policy optimization (the construction is described in [PITH_FULL_IMAGE:figures/full_fig_p041_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 5 linked inside Pith

  1. [1]

    J., Jiang, N., Sekhari, A., and Xie, T

    Amortila, P., Foster, D. J., Jiang, N., Sekhari, A., and Xie, T. (2024). Harnessing density ratios for online reinforcement learning. In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Che, F. (2025). A tutorial: An intuitive explanation of offline reinforcement learning theory. arXiv preprint arXiv:2508.07746

    Pith/arXiv arXiv 2025

  3. [3]

    and Jiang, N

    Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In International conference on machine learning , pages 1042--1051. PMLR

    2019

  4. [4]

    and Jiang, N

    Chen, J. and Jiang, N. (2022). Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence , pages 378--388. PMLR

    2022

  5. [5]

    Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., and Wang, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826--2836. PMLR

    2021

  6. [6]

    J., Golowich, N., and Han, Y

    Foster, D. J., Golowich, N., and Han, Y. (2023a). Tight guarantees for interactive decision making with the decision-estimation coefficient. In The Thirty Sixth Annual Conference on Learning Theory , pages 3969--4043. PMLR

  7. [7]

    J., Golowich, N., Qian, J., Rakhlin, A., and Sekhari, A

    Foster, D. J., Golowich, N., Qian, J., Rakhlin, A., and Sekhari, A. (2023b). Model-free reinforcement learning with the decision-estimation coefficient. Advances in Neural Information Processing Systems , 36

  8. [8]

    J., Kakade, S

    Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487

    Pith/arXiv arXiv 2021

  9. [9]

    J., Krishnamurthy, A., Simchi-Levi, D., and Xu, Y

    Foster, D. J., Krishnamurthy, A., Simchi-Levi, D., and Xu, Y. (2022). Offline reinforcement learning: Fundamental barriers for value function approximation. In Conference on Learning Theory , pages 3489--3489. PMLR

    2022

  10. [10]

    and Moitra, A

    Golowich, N. and Moitra, A. (2024). The role of inherent bellman error in offline reinforcement learning with linear function approximation. In Reinforcement Learning Conference

    2024

  11. [11]

    L., Yan ko g lu, \.I ., and Den Hertog, D

    Gorissen, B. L., Yan ko g lu, \.I ., and Den Hertog, D. (2015). A practical guide to robust optimization. Omega , 53:124--137

    2015

  12. [12]

    Jia, Z., Rakhlin, A., Sekhari, A., and Wei, C.-Y. (2024). Offline reinforcement learning: Role of state aggregation and trajectory data. In The Thirty Seventh Annual Conference on Learning Theory , pages 2644--2719. PMLR

    2024

  13. [13]

    Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704--1713. PMLR

    2017

  14. [14]

    Jin, C., Liu, Q., and Miryoosefi, S. (2021). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406--13418

    2021

  15. [15]

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems , 33:1179--1191

    2020

  16. [16]

    and Szepesvari, C

    Lattimore, T. and Szepesvari, C. (2017). The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728--737. PMLR

    2017

  17. [17]

    and Szepesv \'a ri, C

    Lattimore, T. and Szepesv \'a ri, C. (2020). Bandit algorithms . Cambridge University Press

    2020

  18. [18]

    Liu, H., Wei, C.-Y., and Zimmert, J. (2025a). Decision making in hybrid environments: A model aggregation approach. Conference on Learning Theory

  19. [19]

    Liu, H., Wei, C.-Y., and Zimmert, J. (2025b). An improved model-free decision-estimation coefficient with applications in adversarial mdps. arXiv preprint arXiv:2510.08882

    arXiv

  20. [20]

    Nguyen-Tang, T., Yin, M., Gupta, S., Venkatesh, S., and Arora, R. (2023). On instance-dependent bounds for offline reinforcement learning with linear function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9310--9318

    2023

  21. [21]

    E., Pattathil, S., Zhang, J., and Zhang, K

    Ozdaglar, A. E., Pattathil, S., Zhang, J., and Zhang, K. (2023). Revisiting the linear-programming framework for offline rl with general function approximation. In International Conference on Machine Learning , pages 26769--26791. PMLR

    2023

  22. [22]

    Rashidinejad, P., Zhu, H., Yang, K., Russell, S., and Jiao, J. (2023). Optimal conservative offline rl with general function approximation via augmented lagrangian. In The Eleventh International Conference on Learning Representations

    2023

  23. [23]

    A., Krishnamurthy, A., and Sun, W

    Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. (2022). Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718

    Pith/arXiv arXiv 2022

  24. [24]

    D., and Sun, W

    Uehara, M., Kallus, N., Lee, J. D., and Sun, W. (2023). Offline minimax soft-q-learning under realizability and partial coverage. Advances in Neural Information Processing Systems , 36:12797--12809

    2023

  25. [25]

    and Sun, W

    Uehara, M. and Sun, W. (2021). Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226

    Pith/arXiv arXiv 2021

  26. [26]

    Wang, X., Cui, Q., and Du, S. S. (2022). On gap-dependent bounds for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:14865--14877

    2022

  27. [27]

    Wu, Y., Tucker, G., and Nachum, O. (2019). Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361

    Pith/arXiv arXiv 2019

  28. [28]

    Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683--6694

    2021

  29. [29]

    Xie, T., Foster, D., Bai, Y., Jiang, N., and Kakade, S. (2023). The role of coverage in online reinforcement learning. In Proceedings of the Eleventh International Conference on Learning Representations

    2023

  30. [30]

    and Zeevi, A

    Xu, Y. and Zeevi, A. (2023). Bayesian design principles for frequentist sequential learning. In International Conference on Machine Learning , pages 38768--38800. PMLR

    2023

  31. [31]

    Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. (2022). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730--2775. PMLR

    2022

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.