Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Uncertainty-gated preference queries let offline RL agents match online fine-tuning without touching the environment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:08 UTC pith:6X7HSZ56

load-bearing objection The paper has a real but incremental idea — Morse-based conservative querying and adaptive regularization for preference queries in offline RL — but the experiments are confounded by an oracle trained with an offline-to-online algorithm, which threatens the central 'no environment interaction' claim. the 3 major comments →

arxiv 2607.19199 v1 pith:6X7HSZ56 submitted 2026-07-21 cs.LG

Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation

Li-Rong Zhou , Qin-Wen Luo , Sheng-Jun Huang This is my paper
classification cs.LG
keywords offline reinforcement learningaction preference queriesuncertainty estimationMorse neural networkconservative queryadaptive regularizationvalue regularizationquery shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Offline reinforcement learning is capped by the data it is trained on; action-preference queries offer a way to break that cap by asking an expert to compare two actions, but earlier methods queried actions that drift outside the data distribution and then applied fixed constraints that destabilize the learned value function. This paper claims both problems share a single fix: use a Morse neural network to estimate how far each candidate action is from the training data, query only near-distribution actions, and then scale the regularization penalty so that in-distribution actions are barely constrained while out-of-distribution actions are heavily penalized. The authors integrate this into a value-regularized offline algorithm and report that it outperforms the previous action-preference method and matches or beats online-fine-tuned baselines on standard control tasks, using fewer preference labels and no environment interaction. If the claim holds, preference queries become a practical substitute for online fine-tuning in settings where real-world interaction is costly or dangerous.

Core claim

The paper's central claim is that the instability of action-preference offline RL comes from query shift: selected actions lie outside the data distribution, and their overestimated values then poison the critic. The authors propose to detect query shift with a Morse neural network, which returns a density score M(s,a) in [0,1] for how close a state-action pair is to the offline dataset. Candidates are filtered by a dataset-aware threshold δ = μ − nσ so that only in-distribution actions are sent to the oracle, and the fixed CQL penalty coefficient α is replaced by the state-dependent α(s,a) = α0 (1 − M(s,a)) σ exp(M(s,a)/(βμ)), which vanishes for in-distribution actions and is large for out-

What carries the argument

The load-bearing objects are the Morse network's score M(s,a) and the two uncertainty-gated mechanisms built on it. The conservative query discards any policy action whose Morse score falls below μ − nσ, preventing OOD actions from being labeled by the oracle; the adaptive regularization sets α(s,a) = α0 (1−M)σ e^{M/(βμ)}, replacing the scalar penalty in CQL with a per-state-action penalty. The Morse network is trained once on the offline dataset by minimizing a KL-divergence objective, so the uncertainty model itself is reusable and does not add interaction during training.

Load-bearing premise

The central assumption is that the oracle labeling preferred actions can be built without any online interaction; the paper says it uses an offline-to-online algorithm to train that oracle, and does not show that this is possible without stepping into the environment.

What would settle it

Check whether the oracle in the appendix is trained with online fine-tuning; alternatively, rerun the method with the oracle replaced by a purely offline Q-function and see whether the advantage over plain CQL disappears. If it does, the gain is from the oracle's online access, not from the conservative query and adaptive regularization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, offline agents can improve beyond their datasets without any environment interaction, which lowers the barrier for RL in robotics, healthcare, and other high-stakes settings.
  • Value regularization algorithms, not just policy constraint methods, become viable hosts for action preference queries; the paper demonstrates this on CQL.
  • The separation between conservative querying and adaptive regularization gives a concrete recipe: query only where the estimator is confident, and constrain only where it is not.
  • The Morse uncertainty module is pre-trained once on the offline data, so the extra computational cost of the framework is limited to a single stage and is reusable across tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The unresolved hinge is the oracle: Section 3.2 says Q* is trained with an offline-to-online algorithm and defers details to an appendix. If that training uses online fine-tuning, the 'no environment interaction' framing is not yet established, and the gap over offline baselines could partly reflect the oracle's privileged access.
  • A testable extension the paper does not run: keep the adaptive regularization but replace the oracle labels with the offline critic's own Q-values. That ablation would separate the value of preference supervision from the value of uncertainty-based scaling.
  • The paper interprets conservative queries on sparse-reward navigation as a form of intra-distribution exploration that stitches disconnected trajectories. A direct test would be to measure whether the learned policy's state-action coverage expands beyond the base offline algorithm or whether the gain is only a value-estimation effect.
  • The framework is described as algorithm-agnostic, so a natural next check is whether the same two mechanisms transfer to other value-regularized conservative critics beyond CQL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CQ2L, a framework that adds action preference queries to the offline RL algorithm CQL. A Morse neural network pre-trained on the offline dataset assigns an uncertainty score to state-action pairs. The method uses this score in two ways: a conservative query mechanism that filters candidate actions so that only actions near the dataset distribution are queried, and an adaptive regularization scheme that multiplies CQL's fixed regularization coefficient by a nonlinear function of the Morse score. The authors claim that this yields superior or competitive performance on D4RL MuJoCo and AntMaze tasks compared to offline baselines and to online fine-tuning baselines, while using only 90,000 preference queries and no environment interaction.

Significance. If the 'no environment interaction' claim holds, the contribution is meaningful: it extends action preference querying to value regularization methods and uses uncertainty estimation to reduce query shift. The paper also provides a practical way to inject human preference information without online rollouts. However, the central empirical claim is confounded by the oracle used to label preferences. The paper states that the oracle Q* is trained with an offline-to-online algorithm [24], which in this literature normally involves online fine-tuning. If that is the case, the reported gains over offline baselines could come from privileged online information, and the abstract's claim of 'without requiring environment interaction' would be false. The missing Appendix B and missing code prevent the reader from verifying what Q* actually is. Despite the promise of the general idea, the current evidence does not establish the paper's central claim.

major comments (3)
  1. [Section 3.2, Eq. (7); Section 4.1] The preference oracle Q* is said to be 'trained with an offline-to-online algorithm [24], with details provided in Appendix B.' Reference [24] is an offline-to-online RL method that typically includes online fine-tuning. If Q* was obtained with any online interaction, then the preference labels in Eq. (7) encode privileged information not available in the offline setting. The offline results in Table 1 and the O2O comparisons in Table 2 would then be attributable to this oracle rather than to the proposed conservative query and adaptive regularization, and the abstract's 'without requiring environment interaction' would be false. This is load-bearing because every preference used during training comes from Eq. (7), so the gains over CQL and other offline baselines could be explained entirely by the oracle. The manuscript must either state explicitly that Q* is obtained from the offline p
  2. [Appendix B; reproducibility] The text repeatedly refers to Appendix A (OAP applied to CQL) and Appendix B (implementation details, oracle training), but neither appendix is included in the submitted manuscript. This is not merely a minor formatting issue: the oracle-training detail is exactly what is needed to evaluate the central claim. Without Appendix B and without code, the reader cannot determine whether Q* required online fine-tuning, how the hyperparameters n, β, and α0 were chosen, or whether the reported query budget is correct. Please provide the missing material or state clearly that it is not available; with the current text, the offline/o2o distinction is unverifiable.
  3. [Section 3.2 and 3.3, Eqs. (8)-(9); Section 4.1] The adaptive regularization and dataset-aware threshold rely on hand-set hyperparameters: the conservativeness coefficient n, the slope β, the maximum regularization strength α0, the Morse kernel scale λ, and the query interval T_q and per-query count N_p. No sensitivity analysis is reported, and the results are shown for a single configuration, presumably tuned on the same D4RL benchmark. Since the method's improvement over CQL is the key empirical claim, the absence of any robustness check (e.g., varying n or β) makes it hard to assess whether the reported performance is intrinsic to the proposed mechanism or an artifact of careful tuning on these specific tasks.
minor comments (5)
  1. [Abstract] The phrase 'dynamically modulates the strength' should be 'dynamically modulates' to correct the subject-verb agreement.
  2. [Table 1 caption] The table is labeled 'Offline scheme performance,' but CQ2L uses 90,000 preference queries from an oracle. The label is misleading; consider calling it 'offline + preference query scheme' to distinguish it from methods that use no additional information.
  3. [Table 2 caption] The caption says 'The second-best average performance is highlighted with the underline,' but the underline is not visible in the table as typeset. Also, CQ2L appears to be bold in many columns; please clarify the intended highlighting.
  4. [Section 3.3, Eq. (9)] The nonlinear scaling function Sβ(s,a) is introduced without a motivating derivation. As it is a central component of the proposed 'adaptive' mechanism, a brief explanation of why this particular form (rather than, say, a simpler clipped linear function) is preferred would be helpful.
  5. [References] The paper claims to be 'algorithm-agnostic' but demonstrates the method only with CQL. The claim should be softened, or a second value regularization baseline (e.g., IQL or TD3+BC) should be included.

Circularity Check

1 steps flagged

The offline 'no environment interaction' result hinges on an oracle trained by the authors' own offline-to-online algorithm; without the missing Appendix B, the reported gains may be inherited from that fitted oracle rather than from the proposed mechanisms.

specific steps
  1. fitted input called prediction [Section 3.2 (Eq. 7); Abstract; Tables 1-2; Reference [24]]
    "The action preference query mechanism leverages expert feedback without requiring environment interaction ... In our setting, we use the Q-value function of a stronger policy, denoted as Q*, as the oracle. ... In this work, we use an offline-to-online algorithm [24] to train the oracle, with details provided in Appendix B."

    Every preference label in Eq. (7) is produced by Q*, a value function fitted by the authors' own offline-to-online method [24] (Luo, Xie, Wang, Huang — two of whom are present authors). If that method used online fine-tuning, as its name and the paper's own 250k-step O2O comparisons imply, then the 'offline' improvements in Tables 1-2 are inherited from an online-trained oracle, not generated by the proposed conservative query or adaptive regularization. The central claim 'without requiring environment interaction' is thus an unverified assumption about a fitted input; the missing Appendix B prevents checking whether the oracle was offline-only.

full rationale

The method itself is not formally circular: the Morse-score threshold delta = mu - n*sigma, the adaptive coefficient alpha(s,a) = S_beta(s,a)*alpha0, and the CQL-style loss in Eq. (13) are explicit functions of the offline dataset and do not reduce to the target result by construction. The circularity-adjacent problem is the oracle. The paper's headline advantage is measured against the oracle's preferences, and that oracle is sourced from a self-citation ([24]) whose offline-only status is asserted only by reference to a missing Appendix B. This makes the empirical claim load-bearing on an unverified fitted input from the authors' own prior work, and it confounds the comparison with online fine-tuning baselines. If Appendix B showed a purely offline oracle, the score would be 0-2; as written, the central result is not self-contained. Hyperparameters n, beta, alpha0 are hand-set on the same D4RL tasks, which is a benchmark-fitting concern but not a definitional circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical or mathematical objects. The key uncharged assumptions are the calibration of Morse scores and the availability of an oracle that is not trained with online interaction; the central results depend on both.

free parameters (5)
  • conservativeness coefficient n = not reported
    Sets the query threshold δ = μ − nσ; controls how far queried actions may deviate from the dataset; no sensitivity analysis is given.
  • regularization slope β = not reported
    Shapes the nonlinear scaling Sβ in Eq. (9); determines when out-of-distribution actions get strong regularization.
  • maximum regularization strength α0 = not reported
    Scales the data-dependent α(s,a); effectively inherited from CQL but reweighted by the Morse score.
  • Morse kernel scale λ = not reported
    RBF bandwidth in Eq. (3); strongly affects Morse-score calibration and hence the threshold and adaptive weights.
  • query interval T_q and per-query count N_p = total 90,000 queries; N_p vs N_q typo in text
    Determines how often and how many preferences are collected; not specified numerically in the provided text.
axioms (4)
  • domain assumption Morse scores are calibrated density estimates of the offline dataset in state-action space
    The threshold δ and α(s,a) both depend on this; no calibration plots or density-quality checks are provided.
  • domain assumption An expert/preference oracle Q* exists and is queried without environment interaction
    Section 3.2 uses Q* from an offline-to-online algorithm; if that algorithm performs online fine-tuning, this axiom fails and the central comparison is unfair.
  • standard math CQL's objective remains well-behaved when α is a non-linear state-action-dependent function
    Eq. (13) modifies the CQL regularizer with α(s,a), but no convergence or stability analysis is provided for this modified objective.
  • ad hoc to paper The offline-to-online oracle [24] provides a stronger policy without invalidating the offline comparison
    The wording implies online fine-tuning; the missing Appendix B is supposed to justify this premise.

pith-pipeline@v1.3.0-alltime-deepseek · 14582 in / 15534 out tokens · 168269 ms · 2026-08-01T13:08:16.644762+00:00 · methodology

0 comments
read the original abstract

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.

Figures

Figures reproduced from arXiv: 2607.19199 by Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun Huang.

Figure 1
Figure 1. Figure 1: Inaccurate value estimation induced by query shift. (left) The ground-truth Q-values provided by an oracle. (right) The estimated Q-values from the trained critic in value regularization methods. datasets without additional environment interaction, offering a more practical and scalable alternative [16, 23, 14, 11, 6, 44]. Unfortu￾nately, offline RL faces a fundamental challenge due to the distri￾bution sh… view at source ↗
Figure 2
Figure 2. Figure 2: The variation of the nonlinear scaling function in Eq. (9) under different values of β where β is the hyperparameter that controls the slope of the coeffi￾cient function. The first term in Eq. (9) preserves the original value range, while the second term adjusts the regularization strength for different OOD degrees. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study across four tasks. figure 3(a) and 3(b) correspond to MuJoCo tasks with dense rewards and low-dimensional observations, while figure 3(c) and 3(d) correspond to Antmaze tasks with sparse reward and high-dimensional observations, which are much more challenging. 5 Related Work In this section, we provide an overview of offline RL and preference￾based RL by reviewing relevant literature. Offli… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 18 linked inside Pith

  1. [1]

    Agarwal, D

    R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspec- tive on offline reinforcement learning. InInternational conference on machine learning, pages 104–114. PMLR, 2020

    2020

  2. [2]

    Akrour, M

    R. Akrour, M. Schoenauer, and M. Sebag. Preference-based policy learning. InProceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD), vol- ume 6911, pages 12–27. Springer, 2011

    2011

  3. [3]

    G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble.Advances in neural information processing systems, 34:7436–7447, 2021

    2021

  4. [4]

    Basu and S

    S. Basu and S. Prasad. A connection between cut locus, thom space and morse–bott functions.Algebraic & Geometric Topology, 9:4185–4233, 2023

    2023

  5. [5]

    Beeson and G

    A. Beeson and G. Montana. Improving td3-bc: Relaxed policy con- straint for offline learning and stable online fine-tuning.arXiv preprint arXiv:2211.11802, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Brandfonbrener, W

    D. Brandfonbrener, W. F. Whitney, R. Ranganath, and J. Bruna. Of- fline rl without off-policy evaluation.arXiv preprint arXiv:2112.06267, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Busa-Fekete and E

    R. Busa-Fekete and E. Hüllermeier. A survey of preference-based on- line learning with bandit algorithms. InProceedings of the 25th Al- gorithmic Learning Theory International Conference (ALT-14), volume 8776, pages 18–39. Springer, 2014

    2014

  8. [8]

    Busa-Fekete, B

    R. Busa-Fekete, B. Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-based evolutionary direct policy search. InICRA Workshop on autonomous learning, volume 2, 2013

    2013

  9. [9]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  10. [10]

    Dherin, H

    B. Dherin, H. Hu, J. Ren, M. W. Dusenberry, and B. Lakshminarayanan. Morse neural networks for uncertainty quantification.arXiv preprint arXiv:2307.00667, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Ernst, P

    D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode rein- forcement learning.Journal of Machine Learning Research, 6:503–556, April 2005

    2005

  12. [12]

    J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219, 2020

    Pith/arXiv arXiv 2004

  13. [13]

    Fujimoto and S

    S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforce- ment learning.arXiv preprint arXiv:2106.06860, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration.arXiv preprint arXiv:1812.02900, 2018

    Pith/arXiv arXiv 2018

  15. [15]

    Fürnkranz, E

    J. Fürnkranz, E. Hüllermeier, W. Cheng, and S.-H. Park. Preference- based reinforcement learning: A formal framework and a policy iter- ation algorithm.Machine Learning, 89(1-2):123–156, 2012. Special Issue of Selected Papers from ECML/PKDD-11

    2012

  16. [16]

    Jaques, A

    N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog.arXiv preprint arXiv:1907.00456, 2019

    Pith/arXiv arXiv 1907

  17. [17]

    Kidambi, A

    R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning.Advances in neural infor- mation processing systems, 33:21810–21823, 2020

    2020

  18. [18]

    B. R. Kiran, I. Sobh, V . Talpaert, P. Mannion, A. A. Al Sallab, S. Yo- gamani, and P. Perez. Deep reinforcement learning for autonomous driving: A survey.IEEE Transactions on Intelligent Transportation Sys- tems, 23(6):4909–4926, 2021

    2021

  19. [19]

    Kostrikov, R

    I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum. Offline rein- forcement learning with fisher divergence critic regularization. InInter- national Conference on Machine Learning, pages 5774–5783. PMLR, 2021

    2021

  20. [20]

    Kostrikov, A

    I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Pith/arXiv arXiv 2021

  21. [21]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off- policy q-learning via bootstrapping error reduction.Advances in neural information processing systems, 32, 2019

    2019

  22. [22]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning.Advances in neural information pro- cessing systems, 33:1179–1191, 2020

    2020

  23. [23]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learn- ing: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Pith/arXiv arXiv 2005

  24. [24]

    Luo, M.-K

    Q.-W. Luo, M.-K. Xie, Y . Wang, and S.-J. Huang. Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online rl.Advances in Neural Information Processing Systems, 37:108167– 108207, 2024

    2024

  25. [25]

    Matsushima, H

    T. Matsushima, H. Furuta, Y . Matsuo, O. Nachum, and S. Gu. Deployment-efficient reinforcement learning via model-based offline optimization.arXiv preprint arXiv:2006.03647, 2020

    Pith/arXiv arXiv 2006

  26. [26]

    Nachum, B

    O. Nachum, B. Dai, I. Kostrikov, Y . Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience.arXiv preprint arXiv:1912.02074, 2019

    Pith/arXiv arXiv 1912

  27. [27]

    A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2021

    Pith/arXiv arXiv 2006

  28. [28]

    Nikulin, V

    A. Nikulin, V . Kurenkov, D. Tarasov, D. Akimov, and S. Kolesnikov. Q- ensemble for offline rl: Don’t scale the ensemble, scale the batch size. arXiv preprint arXiv:2211.11092, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    Pith/arXiv arXiv 1910

  30. [30]

    R. F. Prudencio, M. R. Maximo, and E. L. Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  31. [31]

    T. P. Runarsson and S. M. Lucas. Imitating play from game trajectories: Temporal difference learning versus preference learning. In2012 IEEE conference on computational intelligence and games (CIG), pages 79–

  32. [32]

    Singh, R

    B. Singh, R. Kumar, and V . P. Singh. Reinforcement learning in robotic applications: a comprehensive survey.Artificial Intelligence Review, 55 (2):945–990, 2022

    2022

  33. [33]

    Srinivasan and W

    P. Srinivasan and W. Knottenbelt. Offline reinforcement learning with behavioral supervisor tuning.arXiv preprint arXiv:2404.16399, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    R. S. Sutton and A. G. Barto.Introduction to Reinforcement Learning. MIT Press, 1998

    1998

  35. [35]

    S. Tang, M. Makar, M. Sjoding, F. Doshi-Velez, and J. Wiens. Lever- aging factored action spaces for efficient offline reinforcement learning in healthcare.Advances in neural information processing systems, 35: 34272–34286, 2022

    2022

  36. [36]

    S. Wang, Q. Yang, J. Gao, M. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning.Advances in Neural Informa- tion Processing Systems, 36:47081–47104, 2023

    2023

  37. [37]

    Wilson, A

    A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries.Advances in neural infor- mation processing systems, 25, 2012

    2012

  38. [38]

    Wirth and J

    C. Wirth and J. Fürnkranz. On learning from game annotations.IEEE Transactions on Computational Intelligence and AI in Games, 7(3): 304–316, 2014

    2014

  39. [39]

    Wirth, R

    C. Wirth, R. Akrour, G. Neumann, and J. Fürnkranz. A survey of preference-based reinforcement learning methods.Journal of Machine Learning Research, 18(136):1–46, 2017

    2017

  40. [40]

    Y . Wu, G. Tucker, and O. Nachum. Behavior regularized offline rein- forcement learning.arXiv preprint arXiv:1911.11361, 2019

    Pith/arXiv arXiv 1911

  41. [41]

    Q. Yang, S. Wang, M. G. Lin, S. Song, and G. Huang. Boosting offline reinforcement learning with action preference query. InInternational Conference on Machine Learning, pages 39509–39523. PMLR, 2023

    2023

  42. [42]

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y . Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization.Advances in Neural Information Processing Systems, 33:14129–14142, 2020

    2020

  43. [43]

    T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization.arXiv preprint arXiv:2102.08363, 2021

    Pith/arXiv arXiv 2021

  44. [44]

    Zhang, B

    R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values.CoRR, abs/2002.09072, 2020

    Pith/arXiv arXiv 2002

  45. [45]

    Y . Zhu, J. Liu, Y . Yuan, W. Wei, Z. Ge, qianyi fu, S. Liu, Z. Fang, Y . Hu, and B. An. Outward odyssey: Improving reward models with proximal policy exploration for preference-based reinforcement learning, 2025. URL https://openreview.net/forum?id=gXV84CnMUm

    2025

  46. [46]

    Zucker, J

    M. Zucker, J. A. Bagnell, C. G. Atkeson, and J. Kuffner. An opti- mization approach to rough terrain locomotion. In2010 IEEE Inter- national Conference on Robotics and Automation, pages 3589–3595. IEEE, 2010

    2010