Pith. sign in

REVIEW 5 major objections 4 minor 40 references

CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In offline preference-based RL, CLARIFY learns a trajectory embedding whose inter-segment distances tell which comparison pairs a human can actually label, and uses that signal to select clear queries, reporting better reward learning…

desk verdict Useful empirical method for offline PbRL query selection, but Section 5's theory is unsupported and the human/benchmark caveats keep the main claim plausible rather than proven. read the letter →

arxiv 2506.00388 v3 pith:DQXBFGHK submitted 2025-05-31 cs.LG

Ni Mu , Hao Hu , Xiao Hu , Yiqin Yang , Bo Xu , Qing-Shan Jia This is my paper
classification cs.LG
keywords preference-basedreinforcementlearningofflinecontrastiveambiguousqueriesqueryselectiontrajectoryembeddingshumanfeedbackrejectionsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In offline preference-based reinforcement learning, the cheap labels are the ones a human can give; the expensive ones are queries so similar that the labeler wants to skip them. CLARIFY's claim is that this ambiguity can be detected and largely avoided before the reward model is trained: learn a trajectory embedding in which clearly distinguished segments are far apart and ambiguous ones stay close, then use embedding distances to pick which pairs to show the teacher. The paper reports that this beats existing offline PbRL methods under scripted skip-rule teachers and real human labelers, and that humans answer a higher share of the selected queries with higher accuracy. If true, the practical payoff is that preference-based RL keeps its label-efficiency advantage precisely in the regimes where humans are least reliable.

What carries the argument

The carrying objects are two contrastive losses and a rejection-sampling selector, all built around a trajectory encoder $f_\phi$ that maps a segment to a fixed-dimension unit-norm embedding. The ambiguity loss $L_{\mathrm{amb}}$ maximizes the distance between clearly distinguished segment pairs and minimizes it between ambiguous no-comparison pairs; the quadrilateral loss $L_{\mathrm{quad}}$ compares two clear queries at a time, encouraging the distance between the two preferred segments plus the distance between the two non-preferred segments to be smaller than the within-pair distances, which regularizes the space and grows the effective training count from $O(n)$ to $O(n^2)$. The encoder itself is a Bi-directional Decision Transformer trained with a reconstruction loss plus these two losses and an $L^2$-norm constraint. Query selection then discretizes the embedding distance $d_{\mathrm{emb}}$ into bins, estimates densities $\rho_{\mathrm{clr}}$ and $\rho_{\mathrm{amb}}$ for clear versus ambiguous pairs, and forms a rejection-sampling distribution that upweights distances typical of clear pairs. This is the mechanism that turns a geometry for trajectory quality into a higher fraction of answerable queries.

What would settle it

Give humans a held-out pool of segment pairs with known clarity labels, run CLARIFY's selector, and compare the clarity and accuracy of high-distance selected pairs against random pairs; if selected pairs show no significant gain, the embedding-distance proxy fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that ambiguous feedback can be attacked before the reward model is fit: a trajectory encoder trained with preference information yields an embedding whose distances separate clear from indistinguishable pairs, and that geometry lifts query quality directly. The paper asserts that minimizing the ambiguity loss $L_{\mathrm{amb}}$ guarantees a positive margin between the smallest distance of clearly distinguished pairs and the largest distance of ambiguous pairs, while the quadrilateral loss $L_{\mathrm{quad}}$ guarantees convex separability of preferred and non-preferred embeddings, and that the two losses together prevent representation collapse and overfitting. Empirically, the paper reports that CLARIFY outperforms offline PbRL baselines under skip-rate 0.5 and 0.7 non-ideal teachers on Metaworld and DMControl tasks, and that real human labelers answer more of its queries and agree with ground truth more often.

Load-bearing premise

The load-bearing premise is that the embedding distance between two segments, computed from an encoder trained only on the current preference dataset, reliably indicates whether a human can tell those segments apart.

Editorial extensions

If this is right

  • CLARIFY's rejection sampler raises the clear-query ratio: at skip rate 0.5 it reports 76.33% clear queries on dial-turn versus 31.67% for OPRL.
  • Label budgets go further: with only 100 preference queries CLARIFY reaches 59.50 on dial-turn and 54.00 on sweep-into, above the Markovian-reward baseline's 49.50 and 49.67.
  • With real human labelers on walker-walk, CLARIFY obtains higher episodic return (420.75 vs 265.91), higher clarity (63.33% vs 53.33%), and higher label accuracy (87.08% vs 66.67%) than OPRL.
  • Both losses contribute: ablations show Lamb-only is unstable early and Lquad-only converges slower, while the pair together gives the best results (77.50 vs 63.20 on dial-turn).
  • If Propositions 5.1 and 5.2 hold, the embedding space carries a positive margin between clear and ambiguous pairs and a linearly separable preferred/non-preferred structure, which is what makes the geometric query selector trustworthy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the density-ratio rejection sampler is generic; any embedding whose distances correlate with human labelability could be wrapped by the same selector, so the two losses are one sufficient training recipe rather than the only one.
  • Beyond the paper: the query-clarity ratio offers a simple pre-screen for collecting preference data at scale, flagging tasks, prompts, or labeler pools where most queries are skipped before expensive reward or policy training begins.
  • Beyond the paper: the same idea transfers to pairwise preference data for language models, where annotators often find near-equivalent responses ambiguous; embedding distance between response pairs could predict which comparisons are worth labeling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes CLARIFY, an offline preference-based RL method that learns trajectory embeddings with an ambiguity loss and a quadrilateral loss, then uses rejection sampling on embedding distances to select queries that are more likely to be clearly distinguishable by a human labeler. The method is evaluated on Metaworld and DMControl benchmarks with a scripted non-ideal teacher that skips queries with small return differences, and in small human-labeling experiments. The paper claims that the two contrastive losses provide theoretical guarantees for margin separation and convex separability of the embedding space, and that CLARIFY outperforms existing offline PbRL baselines in both policy performance and labeling efficiency.

Significance. If the empirical claims hold, the idea of using embedding geometry to filter ambiguous preference queries is a useful and practical contribution to offline PbRL, and the paper includes several honest elements: it points to a code repository, reports ablations, and visualizes the learned embeddings. However, the significance is substantially tempered by three issues: the theoretical guarantees in Section 5 are not actually established by the provided proofs, the human experiments are conducted by the authors themselves, and the benchmark datasets are adjusted relative to the LiRE protocol for several tasks. The central empirical contribution is therefore plausible but not yet rigorously supported, and the theoretical claims as stated should not be relied upon.

major comments (5)
  1. [Section 5 / Appendix A.2, Proposition 5.2] The proof of Proposition 5.2 does not analyze the loss defined in Eq. (6). The main-text loss is Lquad = -E[ℓ(z+,z'−)+ℓ(z'+,z−)−ℓ(z+,z'+)−ℓ(z−,z'−)], whereas the proof in Eq. (29) analyzes -E[ℓ(z+,z'−)+ℓ(z'+,z−)−ℓ(z+,z−)−ℓ(z'+,z'−)]. These are different objectives because the subtracted terms are intra-class distances in Eq. (29) but cross-pair distances in Eq. (6). In addition, the proof differentiates Lquad with respect to w and b, which are not optimization variables anywhere in the objective; the actual optimization variable is the encoder φ. Finally, the step claiming that because e_d is affine and C+ is convex, all of C+ must lie on or beyond the level set {e_d=η} is false: an affine function can take values below its centroid value on a convex set. The claimed convex separability with margin η is therefore unsupported.
  2. [Section 5 / Appendix A.1, Proposition 5.1] The statement of Proposition 5.1 asserts a strict margin under only continuity and compactness assumptions, but the proof in Appendix A.1 adds a 'richness' assumption that is not present in the proposition and that essentially postulates the existence of the desired perturbation. The proof also uses finite-sample weights 1/N+ and 1/N− when computing changes in the expectations, which is only valid for an empirical distribution, while the proposition concerns a general distribution Dp. Consequently, the proof does not establish the claimed strict gap d+min ≥ d−max + δ for continuous distributions, and the 'provable margin' language in Section 5 should be withdrawn or replaced with a rigorous argument.
  3. [Section 4.2, Eqs. (7)–(8)] The rejection-sampling mechanism assumes that the embedding distance d_emb computed by an encoder trained on the limited preference dataset Dp is a reliable proxy for whether a human can distinguish the two segments. This is the load-bearing premise of the query-selection step, but it is not independently validated. The paper does not report, for example, the clarity ratio of queries selected by this criterion versus queries rejected by it, nor does it compare the distribution of d_emb to actual human skips beyond the final aggregate accuracy. Without such evidence, the mechanism's benefit is not demonstrated to come from selecting truly unambiguous queries rather than from some other property of the distance distribution.
  4. [Appendix D, Human experiments] The human experiments were conducted by the authors themselves, as stated in Appendix D: 'We collect feedback from human labelers (the authors) familiar with the tasks.' This is a significant limitation for a paper that claims to validate a method under 'real human feedback.' The results in Figure 4, Figure 5, and Table 5 should be interpreted as a pilot study with potential bias, and the paper should either recruit independent labelers or explicitly frame the results as a proof-of-concept with this caveat. Reporting inter-labeler agreement would also strengthen the claims.
  5. [Section C.1 / Table 7] The paper discloses that the dataset quality was increased for several tasks (box-close, handle-pull-side, sweep-into, cheetah-run) relative to the LiRE settings, with the stated reason being to prevent all methods from failing under the skip-mechanism teacher. This change is not itself an error, and all baselines are rerun on the same adjusted data, so internal comparisons remain valid. However, it changes the benchmark from the one used in prior work, and it weakens the comparability of the reported absolute numbers with LiRE's published results. The authors should justify that the enhanced datasets still represent a meaningful offline PbRL benchmark and discuss how the adjustment might affect the relative ordering of methods.
minor comments (4)
  1. [Appendix A.1, Eq. (17)] In the definition of B(φ), the expectation is written over D rather than over the set of ambiguous pairs N; this appears to be a typo and should be corrected for clarity.
  2. [Throughout, Section 3 and Algorithm 1] The label 'no cop' is unusual and appears to be an abbreviation for 'no comparison'; please introduce the term consistently and spell it out at first use.
  3. [Figure 4 caption] The figure caption and legend use 'CLARITY' in place of 'CLARIFY'; this should be corrected.
  4. [Section 6.4] The text near Figure 6 refers to varying 'λdist' when it presumably means 'λamb'; the appendix version of Figure 6 uses λamb, so the main text should be aligned.

Circularity Check

1 steps flagged · score 5.0 of 10

Query-clarity improvements are by construction from the fitted skip-teacher labels; policy performance comparisons remain independent, while the theory section contains proof gaps rather than circular reductions.

  1. fitted input called prediction [Section 4.1 Eq. (5); Section 4.2 Eqs. (7)-(8); Table 3]
    "min_ϕ Lamb = [−E_{(σ0,σ1,p)∼Dp,p∈{0,1}} ℓ(z0,z1)+E_{(σ0,σ1,p)∼Dp,p=no cop} ℓ(z0,z1)] ... we estimate the density functions ρclr(demb) and ρamb(demb)=1−ρclr(demb) for clearly-distinguished and ambiguous pairs, using the existing preference dataset Dp."

    The skip teacher defines 'clearly distinguished' via Eq. (12) as return difference above threshold, and those same labels are the training targets of Lamb and the inputs for estimating ρclr and ρamb. Since Lamb is minimized to enlarge ℓ(z0,z1) for p∈{0,1} pairs and shrink it for p=no_cop pairs, the rejection sampler's preference for large demb selects queries in the direction of the fitted target. The clarity ratio in Table 3 is then measured with the same skip rule that produced the training labels, so the reported increase in clearly-distinguished queries is by construction rather than an independent empirical prediction.

full rationale

The main policy-performance results in Table 1, Table 2, and Table 5 are self-contained empirical comparisons against external baselines on held-out evaluation metrics; those are not circular. The query-clarity claim is partially circular: the embedding distance used for selection is trained on the same skip-teacher labels that define the clarity metric, so the reported clarity gains largely reflect the training objective. The theoretical section has serious proof defects: Proposition 5.2's appendix analyzes a different loss than Eq. (6) and differentiates with respect to w,b that are not optimization variables of Lquad, and Proposition 5.1's margin conclusion relies on a strong 'richness' assumption. These are correctness risks, not reductions of a result to its own inputs, and therefore do not by themselves raise the circularity score. Self-citations to Mu et al. (2024) are not load-bearing, because the paper includes its own human experiments validating the ambiguous-query premise. Overall, the central empirical contribution survives, but one advertised 'prediction'—improved query clarity—is in substantial part fitted input called prediction, giving a partial circularity score of 5.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or model entities. It relies on a few hand-chosen loss weights, a strong expressivity assumption in the theoretical analysis, and several assumptions about the relationship between embedding geometry and human labeling behavior. These assumptions are plausible but not independently verified.

free parameters (4)
  • lambda_amb = 0.1
    Weight of the ambiguity loss in Eq. 11. Chosen by the authors without sensitivity analysis in the main text.
  • lambda_quad = 1.0
    Weight of the quadrilateral loss in Eq. 11. Chosen by the authors.
  • lambda_norm = 0.1
    Weight of the norm constraint in Eq. 11. Chosen by the authors.
  • nbin = not specified
    Number of bins used to discretize embedding distances for rejection sampling in Section 4.3. No value is reported.
assumptions (4)
  • ad hoc to paper The encoder class f_phi is sufficiently expressive that an infinitesimal perturbation can independently adjust the distances of any two segment pairs (richness assumption).
    Used in the proof of Proposition 5.1 (Appendix A.1, Eq. 21). This strong assumption effectively allows arbitrary modification of individual pair distances, making the margin result depend on the premise.
  • domain assumption Positive and negative segment embeddings form two convex sets C+ and C- in the embedding space.
    Assumed in Proposition 5.2 to derive convex separability; no evidence is given that trained embeddings are convex.
  • domain assumption The 'no cop' label in the preference dataset correctly identifies ambiguous pairs, and the skip-teacher model with threshold epsilon approximates human ambiguity.
    Section 6.1 defines the skip mechanism. The method relies on these labels to train the ambiguity loss.
  • domain assumption Embedding distance is a valid proxy for query distinguishability for humans.
    Section 4.2 uses embedding distance as the basis for rejection sampling; no direct validation that embedding distance correlates with human ambiguity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries." pith.science (2026). https://pith.science/paper/DQXBFGHK

@misc{pith2026250600388,
  author       = {Pith},
  title        = {Pith review of: CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQXBFGHK}},
  note         = {Machine review of arXiv:2506.00388}
}
read the original abstract

Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.

Figures

Figures reproduced from arXiv: 2506.00388 by the authors.

Figure 1
Figure 1. The framework of CLARIFY. 1) Train the trajectory embeddings via contrastive learning, incorporating preference [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of quadrilateral loss Lquad. (a) A demonstration of the main idea of Lquad. (b) An embedding visualization of the intuitive example in Section 4.1. (c) True embedding visualization results of the benchmark experiments on the “drawer-open” task, under skip rate ϵ = 0.5 and ϵ = 0.7. min ψ Lreward = − E (σ0,σ1,p)∼Dp h (1 − p) log Pψ[σ0 ≻ σ1] + p log Pψ[σ1 ≻ σ0] i . (2) Contrastive learning. Contrastive lea… view at source ↗
Figure 3
Figure 3. Visualizations of the learned embedding spaces under [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Query clarity ratio and accuracy of human labels [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Query clarity ratio and accuracy of human labels [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Embedding visualizations of the drawer-open task with [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Nine tasks from Metaworld (a-g) and DMControl (h, i). [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages

  1. [1]

    G., Dabney, W., and Munos, R

    Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

    2017

  2. [2]

    G., Candido, S., Castro, P

    Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588 0 (7836): 0 77--82, 2020

    work page 2020

  3. [3]

    Learning to distinguish: shared perceptual features and discrimination practice tune behavioural pattern separation

    Bencze, D., Sz o ll o si, \'A ., and Racsm \'a ny, M. Learning to distinguish: shared perceptual features and discrimination practice tune behavioural pattern separation. Memory, 29 0 (5): 0 605--621, 2021

    work page 2021

  4. [4]

    Active preference-based gaussian process regression for reward learning

    Biyik, E., Huynh, N., Kochenderfer, M., and Sadigh, D. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020

    work page 2020

  5. [5]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  6. [6]

    and He, K

    Chen, X. and He, K. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 15750--15758, 2021

    2021

  7. [7]

    Rime: Robust preference-based reinforcement learning with noisy preferences

    Cheng, J., Xiong, G., Dai, X., Miao, Q., Lv, Y., and Wang, F.-Y. Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257, 2024

    arXiv 2024

  8. [8]

    Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

    Choi, H., Jung, S., Ahn, H., and Moon, T. Listwise reward estimation for offline preference-based reinforcement learning. arXiv preprint arXiv:2408.04190, 2024

    work page Pith review arXiv 2024

Show all 40 references
  1. [9]

    Learning a similarity metric discriminatively, with application to face verification

    Chopra, S., Hadsell, R., and LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp.\ 539--546. IEEE, 2005

    2005

  2. [10]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  3. [11]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  4. [12]

    Furuta, H., Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364, 2021 a

    2021 arXiv

  5. [13]

    Furuta, H., Matsushima, T., Kozuno, T., Matsuo, Y., Levine, S., Nachum, O., and Gu, S. S. Policy information capacity: Information-theoretic measure for task complexity in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 3541--3552. PMLR, 2021 b

    2021

  6. [14]

    Hindsight preference learning for offline preference-based reinforcement learning

    Gao, C.-X., Fang, S., Xiao, C., Yu, Y., and Zhang, Z. Hindsight preference learning for offline preference-based reinforcement learning. arXiv preprint arXiv:2407.04451, 2024

    2024 arXiv

  7. [15]

    and Sadigh, D

    Hejna, J. and Sadigh, D. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    Reward learning from human preferences and demonstrations in atari

    Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018

    2018

  9. [17]

    Tuning in to how neurons distinguish between stimuli

    Inman, M. Tuning in to how neurons distinguish between stimuli. PLoS Biology, 4 0 (4): 0 e118, 2006

    2006

  10. [18]

    Episodic novelty through temporal distance

    Jiang, Y., Liu, Q., Yang, Y., Ma, X., Zhong, D., Bo, X., Yang, J., Liang, B., Zhang, C., and Zhao, Q. Episodic novelty through temporal distance. In Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2024, 2024

    2024

  11. [19]

    Transferring policy of deep reinforcement learning from simulation to reality for robotics

    Ju, H., Juan, R., Gomez, R., Nakamura, K., and Li, G. Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nature Machine Intelligence, 4 0 (12): 0 1077--1087, 2022

    2022

  12. [20]

    Scalable deep reinforcement learning for vision-based robotic manipulation

    Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp.\ 651--673. PMLR, 2018

    2018

  13. [21]

    Beyond reward: Offline preference-guided policy optimization

    Kang, Y., Shi, D., Liu, J., He, L., and Wang, D. Beyond reward: Offline preference-guided policy optimization. arXiv preprint arXiv:2305.16217, 2023

    2023 arXiv

  14. [22]

    Preference transformer: Modeling human preferences using transformers for rl

    Kim, C., Park, J., Shin, J., Lee, H., Abbeel, P., and Lee, K. Preference transformer: Modeling human preferences using transformers for rl. In The Eleventh International Conference on Learning Representations, 2022

    2022

  15. [23]

    Offline reinforcement learning with implicit q-learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  16. [24]

    Curl: Contrastive unsupervised representations for reinforcement learning

    Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp.\ 5639--5650. PMLR, 2020

    2020

  17. [25]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training

    Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091, 2021 a

    2021 arXiv

  18. [26]

    B-pref: Benchmarking preference-based reinforcement learning

    Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026, 2021 b

    2021 arXiv

  19. [27]

    Survival instinct in offline reinforcement learning

    Li, A., Misra, D., Kolobov, A., and Cheng, C.-A. Survival instinct in offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  20. [28]

    Reward uncertainty for exploration in preference-based reinforcement learning

    Liang, X., Shu, K., Lee, K., and Abbeel, P. Reward uncertainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205.12401, 2022

    2022 arXiv

  21. [29]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  22. [30]

    S-epoa: Overcoming the indistinguishability of segments with skill-driven preference-based reinforcement learning

    Mu, N., Luan, Y., Yang, Y., and Jia, Q.-s. S-epoa: Overcoming the indistinguishability of segments with skill-driven preference-based reinforcement learning. arXiv preprint arXiv:2408.12130, 2024

    2024 arXiv

  23. [31]

    Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making

    Myers, V., Zheng, C., Dragan, A., Levine, S., and Eysenbach, B. Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. arXiv preprint arXiv:2406.17098, 2024

    2024 arXiv

  24. [32]

    Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  25. [33]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  26. [34]

    Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning

    Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203.10050, 2022

    2022 arXiv

  27. [35]

    Trust region policy optimization

    Schulman, J. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  28. [36]

    D., and Brown, D

    Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference-based reward learning. arXiv preprint arXiv:2301.01392, 2023

    2023 arXiv

  29. [37]

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  30. [38]

    Reinforcement learning from diverse human preferences

    Xue, W., An, B., Yan, S., and Xu, Z. Reinforcement learning from diverse human preferences. arXiv preprint arXiv:2301.11774, 2023

    2023 arXiv

  31. [39]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp.\ 1094--1100. PMLR, 2020

    2020

  32. [40]

    and Lu, Z

    Yuan, H. and Lu, Z. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning, pp.\ 25747--25759. PMLR, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.