Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Offline RL can trust Bellman updates when regularization is made state-adaptive and selective; the paper shows this consistently beats the CQL and TD3+BC backbones on D4RL, offline and in fine-tuning.

desk verdict Strong D4RL and O2O results and a genuinely useful idea, but Eq. (10) contradicts the paper's selective-regularization story for the CQL variant. read the letter →

arxiv 2505.19923 v1 pith:GD35PHF5 submitted 2025-05-26 cs.LG stat.ML

Qin-Wen Luo , Ming-Kun Xie , Ye-Wen Wang , Sheng-Jun Huang This is my paper
classification cs.LGstat.ML
keywords offlinereinforcementlearningstate-adaptiveregularizationBellmanupdatesconservativeQ-learningpolicyconstraintoffline-to-onlineRLD4RLbenchmarkextrapolationerror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning normally applies one global regularization coefficient to keep the learned policy or value function near the static dataset, because raw Bellman backups over out-of-distribution actions cause extrapolation error. This paper argues that the uniform coefficient is the real bottleneck: it is either too weak to stop value overestimation or too strong to allow any improvement beyond behavior cloning. The proposed method, selective state-adaptive regularization (SSAR), makes the regularization strength a learned function of the state and applies it only to a sub-dataset of high-quality actions, so Bellman updates are trusted where the data supports them. Using an equivalence between CQL's value regularizer and an explicit policy constraint, the same mechanism extends to both the value-regularization and policy-constraint families. On D4RL, SSAR consistently outperforms both of its backbones in offline learning and, with simple linear annealing of the learned coefficients, in offline-to-online fine-tuning.

What carries the argument

The load-bearing identity is Proposition 3.1: if the policy is a Boltzmann distribution over Q-values, $\pi(a|s) \propto \exp(Q(s,a)/\alpha)$, then CQL's value-regularization term $\beta\,\mathbb{E}_s[\log \sum_a \exp Q - \mathbb{E}_{a\sim D} Q]$ equals $\beta\,\mathbb{E}_{(s,a)\sim D}[-\log \pi(a|s)]$, a negative log-likelihood constraint on dataset actions. This equivalence lets one shared mechanism govern both value-regularization (CQL) and explicit policy-constraint (TD3+BC) updates: a state-adaptive coefficient generator $\beta_\phi(s)$ with threshold $C_n(s)$, applied only on a selectively built sub-dataset $\hat{D}$. For deterministic policies the same logic carries through by treating TD3's exploration noise as the policy's standard deviation, yielding $L_\beta(\phi) = \mathbb{E}_{(s,a)\sim \hat{D}}[n^2\delta^2 - (a-\pi(s))^2]\beta_\phi(s)$. The coefficient network is the object that decides, per state, whether the Bellman-driven critic or the dataset constraint should win; the trust-region width $n$ expands on a linear schedule and stops growing once the average log-likelihood margin turns positive.

What would settle it

Run the method with return-threshold selection on a purpose-built dataset whose state space splits into a well-sampled high-return region and a sparsely sampled low-return region, disabling the IQL advantage filter. If Q-values on the low-return region rise monotonically while final performance stays flat or drops below the uniform-regularization baseline, the safety premise fails; this is the precise failure mode the IQL-style and success-trajectory filters are patched in to prevent.

Watch

Extended reading notes

Core claim

The paper's central claim is that Bellman updates are reliable exactly where existing methods refuse to trust them: at states and actions the dataset supports well. It replaces the fixed global regularization coefficient with a state-dependent coefficient $\beta_\phi(s)$ produced by a neural network, updated so that the learned policy assigns dataset actions a probability above a state-level threshold $C_n(s) = \min\{\log \pi(\mu+n\sigma|s), \log \pi(\mu-n\sigma|s)\}$; when a dataset action is already likely under the policy the coefficient shrinks and the Bellman target drives learning, and when it is unlikely the coefficient grows and pulls the policy back toward the data. To prevent the policy from being dragged toward low-quality actions, the coefficient update runs only on a sub-dataset $\hat{D}$ of selected actions — high-return trajectories when data quality varies little, positive-advantage actions from an IQL pre-trained critic when the dataset is heterogeneous, and successful trajectories in AntMaze. The claimed consequence is that extrapolation error is avoided where it matters while the performance headroom of Bellman updates is preserved, letting the method beat CQL and TD3+BC on nearly every D4RL task and fine-tune to strong online performance with just linear annealing of the learned coefficients.

Load-bearing premise

The method assumes that states and actions left out of the regularized sub-dataset remain safe under unconstrained Bellman backups — that selective trust does not silently reintroduce value overestimation in the unregularized part of the state space.

Editorial extensions

If this is right

  • Both mainstream offline RL families — value regularization and explicit policy constraint — can use the same per-state trust mechanism instead of a hand-tuned global coefficient.
  • On D4RL, the method reports consistent gains over CQL and TD3+BC, with the largest gains on low- and medium-quality datasets and on AntMaze tasks where uniform regularization often collapses.
  • Offline-to-online fine-tuning reduces to annealing the learned coefficients toward zero; the state-adaptive offline critic transfers well enough that the online replay buffer can be initialized from a subset of the offline data or even none of it.
  • Selecting the regularized sub-dataset by positive advantage (IQL-style) stabilizes learning on heterogeneous datasets, where filtering by high-return trajectories would leave too many states unconstrained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit is that its per-state coefficient network could absorb most of the coefficient tuning that offline RL normally requires; the per-dataset return thresholds in Table 6 remain hand-set, so an automatic selection rule would be the natural next test.
  • Proposition 3.1 suggests the same selective mechanism could transfer to other value-regularization and policy-constraint algorithms beyond CQL and TD3+BC, including distributional or model-based offline RL, which the paper does not test.
  • A falsifiable prediction of the trust story: SSAR's margin over uniform regularization should grow with the within-dataset variance of action quality and shrink on nearly homogeneous expert datasets — consistent with the modest expert-dataset gains in Table 1.
  • The claim that the coefficient network generalizes to new states during fine-tuning is what makes data-free online fine-tuning work; measuring coefficient error on held-out states would isolate this mechanism from the effect of the annealing schedule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Selective State-Adaptive Regularization (SSAR) for offline RL. It replaces a fixed global regularization coefficient with a learned state-dependent coefficient β_φ(s), updated so that dataset-action log-likelihoods stay above a state-dependent trust-region threshold. Regularization is applied only on a sub-dataset D̂ selected by trajectory return or IQL-style advantage filtering, and the idea is extended to both CQL and TD3+BC via an equivalence between CQL's regularizer and the negative log-likelihood of dataset actions under the learned policy. The authors also use the learned coefficients for offline-to-online fine-tuning with a linear coefficient-decay schedule. Experiments on D4RL MuJoCo and AntMaze compare SSAR with backbones and prior methods and report consistent improvements in offline and offline-to-online settings.

Significance. If the claims hold, SSAR is a practical contribution: it offers a state-adaptive, selective alternative to global regularization and improves both offline and offline-to-online D4RL scores over CQL and TD3+BC. The paper has real strengths: public code, four-seed runs with 95% CIs, ablations of the coefficient mechanism and sub-dataset selection, and O2O experiments with several replay-buffer initialization strategies. However, the significance is currently bounded by inconsistencies in the written objectives and by the lack of a formal treatment of the selective mechanism. These issues need to be resolved before the reported gains can be attributed to the proposed mechanism.

major comments (4)
  1. [Section 3.3, Eq. (10)] As written, Eq. (10) is not action-selective. Using the authors' own equivalence (Theorem 3.1, Eq. (24)), the regularizer β_φ(s) E_{s∼D̂}[log Σ_a exp Q(s,a) − E_{a∼D} Q(s,a)] equals β_φ(s) E_{s∼D̂, a∼D}[−log π(a|s)], i.e., behavior cloning on all dataset actions at states appearing in D̂. This contradicts the stated mechanism of constraining only high-quality actions and the text below Eq. (10) about increasing Q-values of valuable actions while allowing low-quality actions to update naturally through the Bellman backup. The truly selective form would use D̂ in the action expectation as well, giving β_φ(s) E_{s∼D̂}[log Σ_a exp Q(s,a) − E_{a∼D̂} Q(s,a)], which matches Eq. (9). If Eq. (10) is what was implemented, the reported gains cannot be attributed to selective action regularization; if it is a typo, the paper must be corrected and the implementation verified against the corrected objective.
  2. [Section 3.5, Eq. (15)] The online decay formula β_on(s) = min{1 − N/N_end, 0} β(s) is zero for all N ≤ N_end and negative afterwards, so it does not implement linear annealing of the coefficients from β(s) toward zero. With N_end = 400,000 and 250,000 online steps, the stated formula makes the coefficient zero for the entire online phase, contradicting the text that the constraint term is applied to all online data for stable improvement. The intended formula appears to be max{1 − N/N_end, 0} β(s) or an equivalent ramp; please correct the equation and state explicitly which schedule was used in the reported experiments.
  3. [Section 3.1 and Proof C] Theorem 3.1 is proved for a Boltzmann policy π(a|s) ∝ exp(Q(s,a)/α), but the CQL implementation uses a Gaussian or squashed-Gaussian policy, and Eq. (9) and Eq. (12) are derived for Gaussian densities. Therefore the claimed equivalence between the CQL regularizer and the negative log-likelihood term does not hold exactly in the implemented setting. The paper should state this as an approximation and justify why the Gaussian log-likelihood inherits the same adaptive behavior; otherwise the unified framework rests on an inexact identity.
  4. [Section 3.3 and Section E.3] The selective mechanism is justified heuristically and patched per task: high-return trajectory filtering for low-variance datasets, IQL advantage filtering for replay datasets, successful-trajectory filtering for AntMaze, and per-dataset return thresholds in Table 6. No general condition or analysis is given for when leaving non-sub-dataset states to unconstrained Bellman backups is safe; the t-SNE illustrations in Figure 2 are qualitative. Since the central claim is that selectivity avoids overestimation while enabling Bellman gains, the paper should provide either a formal safety condition or systematic sensitivity evidence, such as threshold-robustness curves across a range of G_T and across selection rules, rather than only the fixed selected thresholds.
minor comments (5)
  1. [Table 3] The AntMaze total for TD3+BC(SA) is reported as 286, but the six row values sum to 276.0; please check and correct this inconsistency.
  2. [Section 3.4] The text says 'we define the objective of the policy improvement in TBC+BC' but the algorithm is TD3+BC; this appears to be a typo.
  3. [Section 4.2] The text says 'as discussed in Section 4.2' when referring to the offline-data-free advantage; the discussion is in Section 3.5, so the cross-reference should be updated.
  4. [Section 2] The policy model is described as a Gaussian distribution in one place and as a squashed Gaussian in Section 3.1; please use consistent terminology and clarify which density is used in Eq. (6).
  5. [Tables 1 and 2] The 95% CIs are helpful, but no significance tests are reported for the pairwise comparisons; given the large variances on AntMaze tasks (e.g., 63.0±36.1), a paired test or effect-size statement would strengthen the 'consistently outperforms' claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (10) makes CQL(SA) equivalent, by the paper's own Theorem 3.1, to behavior cloning on all dataset actions at selected states; the selective-action mechanism is not implemented as claimed.

  1. other [Section 3.3, Eq. (10); Section C, Theorem 3.1/Eq. (24)]
    "we propose a selective regularization strategy to impose the constraint on a subset of high-quality actions ... For CQL, the objective can be reformulated as min_Q βϕ(s) E_{s∼D̂}[log Σ_a exp(Q(s,a)) − E_{a∼D}[Q(s,a)]] + 1/2 E_{s,a,s′∼D}[(Q−B̂πkQ̂k)2] ... R=βE_{s,a∼D}[−logπ(a|s)]"

    Applying Theorem 3.1 to Eq. (10) with state distribution D̂ and the unchanged action distribution D yields β E_{s∼D̂,a∼D}[−logπ(a|s)]. This is a behavior-cloning objective over all dataset actions at the selected states, not over the high-quality actions in D̂. Hence the published CQL objective does not 'impose the constraint on a subset of high-quality actions'; by construction it also constrains low-quality actions at those states. The claimed benefit of avoiding low-quality constraints is therefore not entailed by the equation—it would require E_{a∼D̂} in the second term, which is absent. The CQL(SA) experimental gain cannot be traced to the selective mechanism as written.

full rationale

Aside from Eq. (10), the paper does not exhibit the classic circularity patterns: there are no load-bearing self-citations or imported uniqueness theorems, the state-adaptive coefficient update (Eq. (5)) is a self-referential but well-defined closed-loop control rule rather than a fitted quantity renamed as a prediction, and the per-task return thresholds are ordinary hyperparameter choices evaluated against external D4RL baselines. However, the central claim for the CQL variant reduces by the paper's own equivalence: Eq. (10) equals behavior cloning on all dataset actions at states in D̂. Since this contradicts the 'selective action' mechanism and the claimed gains are attributed to that mechanism, the derivation is partially circular by construction. The TD3+BC variant (Eq. (14)) does implement action-level selectivity, so the issue is partial.

Assumptions & free parameters 8 free parameters · 7 assumptions · 1 invented entities

The formal burden is moderate: background RL is standard, but the method introduces per-dataset thresholds, a bounded coefficient generator, a hand-scheduled trust-region width, and an approximate mapping from Boltzmann theory to Gaussian implementations. The sub-dataset safety assumption is the main empirical leap.

free parameters (8)
  • Return threshold G_T = Per-dataset values in Table 6, e.g., halfcheetah-medium-v2: CQL(SA) 6000, TD3+BC(SA) 5200
    Hand-set for each D4RL task to decide which trajectories enter the high-quality sub-dataset; directly affects which actions are regularized and hence reported performance.
  • n_start and n_end = n_start=1; n_end=3 for MuJoCo, 1.5 for expert datasets, 5 for AntMaze
    Controls the width of the trust region around the policy mean in Eq. (6) and is tuned per domain rather than derived.
  • Linear schedule increment Delta_n and update interval T_inc = Not fully specified in main text; requires code values
    Eq. (8) grows n every T_inc steps until a termination condition; the step size and interval are implementation choices.
  • Coefficient output cap 1.5 * beta_init = 1.5x the original CQL or TD3+BC coefficient
    The coefficient network output is bounded in (0,1.5 beta_init), an ad hoc cap that stabilizes training but is not derived.
  • IQL expectile tau = 0.7
    Used for advantage-based sub-dataset selection in Eq. (11); set to the original IQL value.
  • TD3 exploration noise delta = 0.1
    Used to model the deterministic policy as a Gaussian in Appendix C, Eq. (12).
  • Online decay steps and warm-up steps = N_end=400,000; warm-up 5,000; total online steps 250,000
    Hyperparameters for the linear coefficient annealing in Eq. (15).
  • O2O special settings for TD3+BC AntMaze = Coefficients frozen; larger policy update interval
    Task-specific exceptions applied during online fine-tuning for hard AntMaze tasks.
assumptions (7)
  • standard math Markov Decision Process with transition P, reward R, discount gamma, and behavior policy pi_beta generating D.
    Standard offline RL setup used throughout Section 2; no alternative world model introduced.
  • domain assumption Learned policy is exactly a Boltzmann distribution over Q values (pi(a|s) proportional to exp(Q(s,a)/alpha)).
    Required for Proposition 3.1 and Theorem 3.1 to equate the CQL regularizer with negative log-likelihood; the implementation uses a squashed Gaussian policy, so the equality is approximate.
  • domain assumption TD3+BC exploration noise makes the deterministic policy equivalent to a Gaussian stochastic policy N(pi(s), delta).
    Derivation of Eq. (12) in Appendix C treats actions as samples from a Gaussian with fixed delta=0.1.
  • domain assumption IQL expectile pretraining (Eq. 11) yields reliable Q and V values for filtering positive-advantage actions.
    Used in Section 3.3 to build the sub-dataset for replay-style datasets without a formal guarantee that the advantage filter remains accurate under distribution shift.
  • domain assumption The high-return sub-dataset or positive-advantage subset covers the relevant state-action support so that leaving other states unregularized is safe.
    Assumed in Section 3.3 and supported only by Figure 2 and task-specific remedies, not by a general coverage argument.
  • ad hoc to paper The coefficient output bound (0,1.5 beta_init) and the linear n schedule (Eq. 8) are sufficient to keep the update well-behaved.
    The cap 1.5 beta_init and the n_start/n_end schedule are introduced specifically for this method and are not derived from the problem.
  • domain assumption D4RL normalized scores are a meaningful proxy for offline RL performance.
    All claims of state-of-the-art performance rest on this benchmark family.
invented entities (1)
  • State-adaptive coefficient generator network beta_phi(s)
    purpose: Outputs a per-state regularization coefficient used inside the CQL or TD3+BC objective to modulate pessimism.
    This network is the central new component. Its behavior is only validated on D4RL within the paper, and no independent prediction constrains it outside these benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL." pith.science (2026). https://pith.science/paper/GD35PHF5

@misc{pith2026250519923,
  author       = {Pith},
  title        = {Pith review of: Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GD35PHF5}},
  note         = {Machine review of arXiv:2505.19923}
}
read the original abstract

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset. To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all states. However, due to substantial variations in data quality, the fixed regularization strength often leads to a dilemma: Weak regularization strength fails to address extrapolation errors and value overestimation, while strong regularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates. To address this issue, we propose the selective state-adaptive regularization method for offline RL. Specifically, we introduce state-adaptive regularization coefficients to trust state-level Bellman-driven results, while selectively applying regularization on high-quality actions, aiming to avoid performance degradation caused by tight constraints on low-quality actions. By establishing a connection between the representative value regularization method, CQL, and explicit policy constraint methods, we effectively extend selective state-adaptive regularization to these two mainstream offline RL approaches. Extensive experiments demonstrate that the proposed method significantly outperforms the state-of-the-art approaches in both offline and offline-to-online settings on the D4RL benchmark.

Figures

Figures reproduced from arXiv: 2505.19923 by the authors.

Figure 1
Figure 1. Comparison of Uniform vs. Selective Regularization. The values are evaluated by the policies trained by with different regularization on all data in the dataset. between the policy distribution and the dataset can reflect the dynamic of the training progress, we propose to per￾form an adjustment of n in a distribution-aware manner to dynamically expand the trust region. We start by setting the initial value of n to … view at source ↗
Figure 2
Figure 2. t-sne visualization of different sub-dataset selection methods output high probabilities for the actions within the dataset, including those that lead to suboptimal or even poor per￾formance. To address this problem, we propose a selective regularization strategy to impose the constraint on a sub￾set of high-quality actions, which prevents the policy from being misguided by low-quality actions. To achieve this, we p… view at source ↗
Figure 3
Figure 3. Offline performance comparisons of different types of the coefficient used for the regularization [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Offline performance comparisons of different selection methods for the sub-dataset. From [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages

  1. [3]

    The mean-wise best results among algorithms are highlighted in bold

    Offline performance comparison with prior methods on the D4RL benchmark. The mean-wise best results among algorithms are highlighted in bold. Dataset AW AC IQL Cal-QL SPOT FamCQL TD3+BC(SA) CQL(SA) halfcheetah-m 49.8±0.3 48.1±0.3 47.8±0.2 57.6±0.6 58.1±0.5 56.5±3.763.9±1.2 hopper-m 68.6±11.2 66.7±4.4 64.7±3.4 71.4±37.2 82.3±16.0101.6±0.489.1±9.7 walker2d-...

    work page arXiv

  2. [4]

    Rvs: What is essential for offline rl via supervised learn- ing?arXiv preprint arXiv:2112.10751,

    Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline rl via supervised learn- ing?arXiv preprint arXiv:2112.10751,

    arXiv

  3. [5]

    D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    arXiv 2004

  4. [9]

    B., and Levine, S

    Janner, M., Du, Y ., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis.arXiv preprint arXiv:2205.09991,

    arXiv

  5. [10]

    Of- fline reinforcement learning with fisher divergence critic regularization

    Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. InInternational Conference on Machine Learning, pp. 5774–5783. PMLR, 2021a. Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021b. Kumar, A....

    arXiv

  6. [11]

    Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning

    Luo, F.-M., Xu, T., Cao, X., and Yu, Y . Reward-consistent dynamics models are strongly generalizable for offline reinforcement learning.arXiv preprint arXiv:2310.05422,

    work page Pith review arXiv

  7. [12]

    S., Ghadirzadeh, A., Chen, X., and Finn, C

    Mark, M. S., Ghadirzadeh, A., Chen, X., and Finn, C. Fine- tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022,

    work page 2022

  8. [13]

    Awac: Accel- erating online reinforcement learning with offline datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359,

    arXiv 2006

Show all 29 references
  1. [14]

    Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size.arXiv preprint arXiv:2211.11092,

    Nikulin, A., Kurenkov, V ., Tarasov, D., Akimov, D., and Kolesnikov, S. Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size.arXiv preprint arXiv:2211.11092,

    arXiv

  2. [15]

    L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N

    Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N. Hyper- parameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055,

    2007 arXiv

  3. [16]

    B., Kumar, A., Zhang, G., and Levine, S

    Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177,

    1910 arXiv

  4. [17]

    J., and Zhou, M

    Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022a. Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learn...

    arXiv

  5. [18]

    Xu, H., Jiang, L., Li, J., Yang, Z., Wang, Z., Chan, V . W. K., and Zhan, X. Offline rl with no ood actions: In-sample learning via implicit value regularization.arXiv preprint arXiv:2303.15810,

    arXiv

  6. [19]

    Policy expansion for bridging offline-to-online reinforcement learning.arXiv preprint arXiv:2302.00935,

    11 Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning.arXiv preprint arXiv:2302.00935,

    arXiv

  7. [20]

    Ensemble- based offline-to-online reinforcement learning: From pes- simistic learning to optimistic exploration.arXiv preprint arXiv:2306.06871,

    Zhao, K., Ma, Y ., Liu, J., Zheng, Y ., and Meng, Z. Ensemble- based offline-to-online reinforcement learning: From pes- simistic learning to optimistic exploration.arXiv preprint arXiv:2306.06871,

    arXiv

  8. [21]

    Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning.arXiv preprint arXiv:2210.13846,

    Zhao, Y ., Boney, R., Ilin, A., Kannala, J., and Pajarinen, J. Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning.arXiv preprint arXiv:2210.13846,

    arXiv

  9. [23]

    due to the generalization capabilities of learned models. Additionally, recent works have framed policy learning as a supervised learning and sequence modeling problem using advanced network architectures (Emmons et al., 2021; Chen et al., 2021; Janner et al., 2022). However, ...

    2021

  10. [25]

    Recent works focus on explicit policy constraints for stochastic policies (Wu et al., 2022; Nair et al.,

    illustrates the effectiveness of a simple policy constraint applied to the off-policy TD3 algorithm (Fujimoto et al., 2018). Recent works focus on explicit policy constraints for stochastic policies (Wu et al., 2022; Nair et al.,

    2018

  11. [26]

    or more expressive policies (Kang et al., 2024; Wang et al., 2022a). Another implementation of policy constraints is the implicit policy constraint derived from the KL-divergence between the learned policy and the behavior policy, known as advantage-weighted behavior cloning (...

    2024

  12. [27]

    Moreover, the coefficients are updated by maximizing Q-values, lacking the interpretability offered by our method

    also utilizes state-adaptive coefficients, but it is limited to IQL-style methods with implicit constraints and the entire dataset. Moreover, the coefficients are updated by maximizing Q-values, lacking the interpretability offered by our method. Offline-to-online RLThe distri...

    2022

  13. [28]

    or utilized Q-ensemble methods to maintain reliable estimations at a higher computational cost (Lee et al., 2022; Mark et al., 2022; Zhao et al., 2023). In contrast, our method assigns different levels of pessimism based on the distance between the output of the learned policy...

    2022

  14. [2018]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. InInterna- tional conference on machine learning, pp. 2052–2062. PMLR,

  15. [2019]

    Extreme q-learning: Maxent rl without entropy.arXiv preprint arXiv:2301.02328,

    Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent rl without entropy.arXiv preprint arXiv:2301.02328,

    arXiv

  16. [2020]

    Various CQL variants adjust the constraints or modify the regularizer to avoid excessive pessimism (Lyu et al., 2022; Nakamoto et al., 2024; Mao et al., 2024; Yu et al., 2021)

    augments the standard Bellman error objective with a Q-value regularizer that provides a lower bound on the value of the current policy. Various CQL variants adjust the constraints or modify the regularizer to avoid excessive pessimism (Lyu et al., 2022; Nakamoto et al., 2024;...

    2022

  17. [2021]

    Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning.arXiv preprint arXiv:2202.11566,

    Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning.arXiv preprint arXiv:2202.11566,

    arXiv

  18. [2022]

    J., Smith, L., Kostrikov, I., and Levine, S

    Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data.arXiv preprint arXiv:2302.02948,

    arXiv

  19. [2023]

    and Montana, G

    Beeson, A. and Montana, G. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning.arXiv preprint arXiv:2211.11802,

    arXiv

  20. [2024]

    G., and Levine, S

    Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    arXiv

  21. [2025]

    12 Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL A. Related Work Offline RLIn offline RL, inaccessible interactions with the environment pose challenges such as extrapolation error and overestimation due to out-of-distribution (OOD) ...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.