PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

Alex Davey; Brahim Driss; Riad Akrour

REVIEW 4 major objections 5 minor 38 references

PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A diverse policy population beats single-policy preference learning.

desk verdict PB² is a genuine attempt to fix preference-based RL's exploration bottleneck, but its load-bearing gate is never ablated and the statistics are too weak to support the headline claims. read the letter →

arxiv 2506.13741 v2 pith:SGPRMKKC submitted 2025-06-16 cs.AI cs.LG

Brahim Driss , Alex Davey , Riad Akrour This is my paper

classification cs.AIcs.LG

keywords preference-basedreinforcementlearningpopulation-basedmethodspreferencespaceexplorationqueryselectiondiversityhumanfeedbackinconsistencyrewarddiscriminatorbonus

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that preference-based reinforcement learning often fails because single-policy methods converge to a narrow slice of the preference landscape, generating trajectory pairs that humans cannot easily tell apart. It proposes PB², a population-based method that trains multiple agents at once, each pushed toward a distinct behavior by a discriminator bonus, while a reference agent anchors performance. The central claim is that this maintained diversity produces more distinguishable preference queries, which improves reward-model learning, escapes local optima, and does not collapse when human teachers mislabel similar trajectories. If the claim holds, preference-based RL systems can learn useful behaviors from fewer and noisier human comparisons.

What carries the argument

The central mechanism is the performance-constrained diversity objective: each explorer policy $\pi_i$ maximizes $\mathbb{E}_{\tau \sim \pi_i}[R_\phi(\tau) + \lambda \cdot \mathbb{1}[R_\phi(\tau) \geq \alpha \cdot R_\phi(\tau_{\mathrm{ref}})] \cdot \log q_\psi(i|\tau)]$, where $R_\phi$ is the learned reward, $\pi_{\mathrm{ref}}$ is a reference policy that only maximizes $R_\phi$, and $q_\psi$ is a discriminator trained to predict which policy generated a trajectory. The indicator gate applies the diversity bonus only when an explorer stays within a factor $\alpha$ of the reference agent's reward, preventing diversity from degenerating into pure novelty-seeking. An adaptive discriminator and periodic inheritance of the reference agent's actor and critic into explorers keep the population aligned with current human preferences while preserving distinct behaviors. This machinery converts population diversity into more distinguishable preference queries.

What would settle it

Run PB² in PointMaze with a deliberately miscalibrated reward model (for example, random weights frozen before any feedback) and record whether the gate $R_i \geq \alpha \cdot R_{\mathrm{ref}}$ ever activates: if diverse agents never receive the bonus, or if bonus-receiving agents achieve low true return, the gate is the failure point. A simpler check is measuring the fraction of bonus-activated updates whose true return actually exceeds the reference agent's.

Watch

Extended reading notes

Core claim

The paper's discovery is that the preference exploration problem in PbRL is best addressed by optimizing a population of policies rather than a single one. PB² trains a reference policy that purely maximizes the learned reward model and several explorer policies that maximize the learned reward plus a performance-constrained diversity bonus; the bonus is a discriminator log-probability identifying which policy generated a trajectory, added only when an explorer's expected reward is within a factor $\alpha$ of the reference agent's. The discriminator is retrained as preferences evolve, giving each agent its own region of behavior space. With the same feedback budget, PB² outperforms QPA, PEBBLE, and RUNE in navigation tasks, maintains higher return when the trajectory similarity threshold $\epsilon$ grows, and escapes a local optimum in PointMaze where QPA remains trapped.

Load-bearing premise

The diversity gate assumes the learned reward model is accurate enough to judge which explorer behaviors are acceptably good, so a miscalibrated model can either suppress useful exploration early or reward the wrong behaviors.

Editorial extensions

If this is right

PB² needs less human feedback than QPA in navigation tasks, with up to roughly 50% improvement at some feedback counts and consistent gains in PointMaze at higher feedback levels.
When the simulated teacher randomly labels similar trajectories ($\epsilon = 0.1$), PB² degrades far less than QPA and PEBBLE, reaching about 750 episode return in walker_walk versus roughly 400 and 350 for the baselines.
After 20 feedback queries in PointMaze, PB²'s population discovers a path to the high-reward region while QPA stays in the initially promising but suboptimal upper-left area.
The advantage comes from query distinguishability rather than extra queries, since all methods operate under the same feedback budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is active selection of which population members' trajectories are paired into queries, using discriminator confidence to maximize distinguishability per query; the paper does not test this.
The gate could be made self-calibrating by adapting $\alpha$ to reward-model uncertainty, which might remove the need to tune $\alpha$ and $\lambda$ for each environment class.
If the discriminator bonus is a working proxy for query distinguishability, it could double as an automatic warning that a human teacher is likely to give inconsistent feedback before real-user studies begin.
Because the reference agent keeps a reward-only policy, PB² degrades gracefully: even if explorer bonuses fail, the population retains the behavior of a single-agent PbRL learner.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

PB² is worth a look if you care about preference-based RL. It attacks a real problem that most PbRL papers sidestep: single-policy query selection tends to generate increasingly similar comparisons, so the reward model stops improving and the policy locks into a local optimum. The authors combine two established ideas—population-based diversity (SMERL, DvD) and PbRL's learned reward—into a concrete algorithm with a reference policy that tracks current reward and a discriminator that pushes explorers to be distinct, but only when the learned reward says they're doing well enough. That combination is new, and it's not a trivial restatement.

The paper does some things well. The motivating example in Figure 2 and the PointMaze trajectory evolution genuinely illustrate how a diverse population can escape a bad basin. The simulated inconsistent feedback via similarity threshold is a step toward realism. And the authors are upfront about computational overhead and λ sensitivity.

The soft spots are real but not fatal. The biggest one the stress-test note got right: the performance gate (Eq. 3) is entirely conditioned on the learned reward model, and the paper never ablates it. If that reward is miscalibrated early on, the gate could suppress good explorers or amplify reward-model delusions, and the reported gains could be an artifact of the gate rather than of diversity itself. Relatedly, the paper claims diversity improves reward model learning, but it never measures reward model quality directly—only task performance. Second, the statistics are too thin: Table 1 has high cross-seed variance without significance tests, and Figure 3 has no error bars. Third, λ and population size were tuned on walker_walk and then reused, which is mild circularity. Fourth, code is promised only upon acceptance, which hurts reproducibility.

None of these sinks the central idea. The paper deserves engagement, not desk rejection. If I were the editor, I'd send it to a referee with a request for major revision: ablate the gate, add statistical rigor, and ideally validate with a held-out λ selection or real human feedback.

For you: if you work on PbRL or human-in-the-loop RL, this is worth reading and citing after revision. For a reading group, it would generate discussion on exactly where diversity helps and when the learned-reward gate could mislead.

Recommendation: peer review, yes.

Referee Report

4 major / 5 minor

Summary. The paper proposes PB², a population-based method for preference-based reinforcement learning (PbRL). A reference policy maximizes the currently learned reward model, while a small population of explorer policies additionally receive a discriminator-based diversity bonus that is gated by a performance constraint relative to the reference policy (Eq. 3, Algorithm 2). The method is evaluated on two navigation tasks and four DMControl locomotion tasks, with simulated human feedback noise via a similarity threshold. The reported results claim improved feedback efficiency, robustness to inconsistent feedback, and better escape from local optima compared to PEBBLE, RUNE, and QPA.

Significance. If the claims are substantiated, the paper addresses a genuine and important problem in PbRL: when preference queries contain similar trajectories, human feedback becomes unreliable, and population-generated diverse behaviors could make queries more distinguishable and improve reward learning. The adaptation of SMERL-style performance-constrained diversity to a setting without access to ground-truth rewards or experts is a plausible and interesting idea. The paper is also honest about limitations such as computational overhead and the tuning of the diversity coefficient. However, the empirical evidence currently has significant gaps: the main mechanism is not ablated, statistical significance is not established, hyperparameters are tuned on test environments, and the central claim of improved reward model learning is not directly measured.

major comments (4)

[§5.1, Eq. (3), Algorithm 2 lines 43–48, Appendix D.1] The performance-constrained diversity gate is never ablated. The ablation study in Appendix D.1 (Figure 5) varies only on-policy sampling and policy inheritance, not the gate or the diversity bonus itself. Because the gate relies on the learned reward model R_phi, which can be poorly calibrated with very few queries (100 in some DMControl tasks), the gate may suppress useful diversity or amplify reward-model artifacts. Without an ablation that removes the gate (e.g., always applying the diversity bonus, or never applying it) and that varies the threshold α, the reported gains cannot be attributed to diversity rather than to the gate's exploitative forcing. This is load-bearing for the paper's central claim, so this experiment is necessary.
[Table 1, Figure 3, Section 6] The empirical claims of significant improvement are not supported by significance testing. With only 5 seeds and large standard deviations, many reported differences have heavily overlapping error bars, for example PointMaze at N=12 (PB²: 85.0 ± 45.3 vs. QPA: 63.6 ± 55.0) and most entries at N=2/N=4. Figure 3 shows learning curves without any error bars or confidence bands. Please report confidence intervals, perform paired significance tests across seeds, or otherwise quantify the reliability of the observed differences. Phrases such as 'significantly better' and 'consistently outperforms' should be tempered accordingly.
[Appendix D.2, Appendix D.3, Table 6] The diversity coefficient λ is selected using the walker_walk environment (Figure 6) and then fixed at 0.25 for all DMControl tasks, and population size is similarly analyzed only on walker_walk (Figure 7). Since walker_walk is also one of the main test environments (Figure 3), the results for that environment are likely optimistic. Please use a held-out validation environment for hyperparameter selection, or report sensitivity results on all environments, and clearly state which environments were used for tuning. This also affects the comparison to baselines, which do not benefit from such tuning.
[Section 6, Figures 3 and 4, Abstract] The central claim that diversity 'improves reward model learning' and generates 'clearly distinguishable behaviors' is not directly measured. The paper reports policy return under ground-truth reward, which is a downstream effect. The abstract promises that PB² 'improves reward model learning by generating preference queries with clearly distinguishable behaviors,' but no metric of reward model quality (e.g., held-out preference prediction accuracy) or of trajectory distinguishability (e.g., pairwise distances between generated trajectories) is reported. Without such direct evidence, the proposed mechanism remains a plausible conjecture rather than a demonstrated result.

minor comments (5)

[§5.1, Eq. (3), Algorithm 2 line 23] The objective in Eq. (3) uses log q_psi(i|tau), while Algorithm 2 line 23 computes the discriminator reward as log q_psi(i|s_t) - log p(i). Please align the notation and clarify whether the discriminator is conditioned on full trajectories or single states.
[§5.1 and Algorithm 2] The text says that the diversity bonus is temporarily disabled after a reward model update, but this suspension step does not appear in either Algorithm 1 or Algorithm 2. Please add it to the pseudocode for reproducibility.
[Figure 3] The learning curves in Figure 3 have no error bars or confidence bands, making it difficult to assess variability across seeds. Please add shaded regions or error bars.
[Throughout] The manuscript contains several typos: 'T op row' in the Figure 2 caption, 'trajectory segments pairs' used repeatedly instead of 'pairs of trajectory segments', and 'F ourth' in the Frey et al. reference. A careful proofreading pass is recommended.
[Table 1] The footnotes 'N=2*' and 'N=4*' are not explained in the text or table caption; please clarify what the asterisk denotes.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor selection-on-evaluation-environment in hyperparameter choice; otherwise the empirical claims are self-contained and not circular.

fitted input called prediction [Appendix D.2/D.3, Section 6 (Figure 3), Table 6]
"The results reveal that moderate values of λ (0.1-0.25) achieve the best performance, with λ = 0.25 showing the strongest results. ... The population size of 3, which we use in our main experiments, appears to be well-chosen based on this analysis."

Appendix D.2 selects λ=0.25 on Walker_walk because it 'shows the strongest results', and Appendix D.3 selects population size 3 on Walker_walk because it is 'well-chosen based on this analysis'. Table 6 then fixes λ=0.25 for Walker_walk in the main DMControl experiments, and Figure 3 uses those runs to claim PB² is robust to inconsistent feedback (e.g., roughly 750 vs 400 for QPA at ε=0.1 in Walker_walk). The headline robustness result on Walker_walk is therefore a re-reporting of a configuration selected on that same environment, not an out-of-sample prediction.

full rationale

PB² is an empirical algorithm paper; there is no formal derivation whose output equals its input by construction. Equation (3)'s objective is an algorithmic design, and the paper's core claims (better reward learning, robustness to inconsistent feedback, escaping local optima) are evaluated against ground-truth returns and external baselines (PEBBLE, RUNE, QPA). The only self-citation is Wirth et al. (2017), a JMLR survey co-authored by R. Akrour; it is used for background on PbRL and is not load-bearing. The one mild circularity is hyperparameter selection: λ=0.25 and population size 3 are chosen from Walker_walk sensitivity analyses (Appendix D.2, D.3) and then used in the main Walker_walk experiments (Table 6, Figure 3), so part of the reported robustness advantage on that task is a tuned result rather than an independent prediction. Section 7 discloses the λ limitation, which supports a low score. No derivation-level circularity is present, so the score is 2 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on four domain assumptions common in PbRL (Bradley-Terry, simulated teacher, state-level discriminability, and the learned-reward performance gate). The tuning of λ, α, and population size on target environments is the main free-parameter burden.

free parameters (3)

Diversity coefficient λ = 0.25 (DMControl), 0.5 (navigation)
Controls the strength of the discriminator diversity bonus; selected by sensitivity analysis on walker_walk (Figure 6), a test environment; acknowledged as a limitation for different environment classes.
Performance threshold α = 0.9 (approximate)
Determines when explorer agents receive the diversity bonus relative to the reference agent's learned return (Eq. 3, Algorithm 2); set by hand, with no sensitivity analysis reported.
Population size = 3 (1 reference + 2 explorers)
Chosen after sensitivity analysis on walker_walk (Figure 7); affects the number of queries available per agent and the complexity of the discriminator.

assumptions (4)

domain assumption Bradley-Terry pairwise preference model
Eq. (1) assumes pairwise preferences follow a logistic function of summed learned rewards, standard in PbRL.
domain assumption Simulated teacher with ϵ similarity threshold is a valid proxy for human inconsistency
Section 6 uses ground-truth return difference with threshold ϵ to randomly relabel similar pairs; all robustness claims rely on this model.
domain assumption State-level discriminability transfers to trajectory-level query distinguishability
The diversity bonus uses log qψ(i|s) per-state (Algorithm 2 line 23), while the claimed benefit is for trajectory comparisons; no experiment directly measures query distinguishability.
domain assumption Learned reward Rϕ is a reliable proxy in the performance gate
Eq. (3) and Algorithm 2 lines 43-48 gate diversity on α·Rϕ(τ_ref); if rϕ is miscalibrated the gate can suppress or misdirect diversity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/SGPRMKKC

@misc{pith2026250613741,
  author       = {Pith},
  title        = {Pith review of: PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGPRMKKC}},
  note         = {Machine review of arXiv:2506.13741}
}

read the original abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities,particularly in environments with complex reward landscapes.

Figures

Figures reproduced from arXiv: 2506.13741 by the authors.

**Figure 1.** Overview of PB². After an initial unsupervised exploration phase, we sample experience from different agents and form comparison pairs to train a reward model using human feedback. A discriminator is trained on the experience of the agents, with the goal of maintaining diversity in the population. The discriminator exploration bonus is added to the learned reward, in order to encourage the discovery of different beh… view at source ↗

**Figure 2.** Diverse population-based query strategy improves preference space coverage compared to single-agent approaches. (Top row) (Column 1) Reward model predictions across the state space, showing higher rewards in the upper region. (Column 2-4) Diversity bonus (Sec. 5.1) for three different agents, with distinct spatial concentrations. (Bottom row) (Column 1) Single agent trajectories (QPA), (Column 2-4) Trajectories fro… view at source ↗

**Figure 3.** Performance comparison on DMControl tasks with varying similarity thresholds. Each column represents a different similarity threshold ϵ that determines when trajectory comparisons result in inconsistent feedback. As ϵ increases from 0 (perfect oracle) to 0.1 (significant inconsistency with similar trajectories), PB² (pink) maintains robust performance while single-agent methods degrade more substantially, particular… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Escaping local optima in preference landscapes. After the same initial feedback following unsupervised exploration (4 queries), both algorithms learn similar reward models favoring the upperleft region. After 20 feedback instances, QPA (middle) remains trapped in this…

**Figure 5.** Figure 5: Ablation study on DMControl locomotion tasks showing the contribution of key components [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

**Figure 6.** Figure 6: Sensitivity analysis of the diversity parameter [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 7.** Figure 7: Impact of population size on learning performance in the walker_walk environment with [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: Complete trajectory evolution in Point Maze environment showing exploration patterns [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 27 canonical work pages

[1]

Forecasting with Multiple Seasonality

Mohammed Nabi Alam, Kok-Why Duong, Sepideh Hosseini, and Kathryn Kasmarik. Evolutionary reinforcement learning: A survey. arXiv preprint arXiv:2008.12340, 2020

work page Pith review arXiv 2008
[2]

Batch active preference-based learning of reward functions

Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning, pages 519--528. PMLR, 2018

work page 2018
[3]

Landolfi, Dylan P

Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020

work page 2020
[4]

Rank analysis of incomplete block designs: I

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952
[5]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30: 0 4299--4307, 2017

work page 2017
[6]

Autonomous skill discovery with quality-diversity and unsupervised descriptors

Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 81--89. ACM, 2019

work page 2019
[7]

Faster Improvement Rate Population Based Training

Valentin Dalibard and Max Jaderberg. Faster improvement rate population based training. arXiv preprint arXiv:2109.13800, 2021

work page Pith review arXiv 2021
[8]

Diversity is all you need: Learning skills without a reward function

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019

work page 2019

Show all 38 references

[9]

Jax-lob: A gpu-accelerated limit order book simulator to unlock large scale reinforcement learning for trading

Sascha Yves Frey, Kang Li, Peer Nagy, Silvia Sapora, Christopher Lu, Stefan Zohren, Jakob Foerster, and Anisoara Calinescu. Jax-lob: A gpu-accelerated limit order book simulator to unlock large scale reinforcement learning for trading. In Proceedings of the Fourth ACM Internat...

2023
[10]

u rnkranz, Eyke H \

Johannes F \"u rnkranz, Eyke H \"u llermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89: 0 123--156, 2012

2012
[11]

Soft actor-critic algorithms and applications

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018

2018 arXiv
[12]

Russell, and Anca D

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca D. Dragan. Inverse reward design. Advances in Neural Information Processing Systems, 30, 2017

2017
[13]

Query-policy misalignment in preference-based reinforcement learning

Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy misalignment in preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024

2024
[14]

Trend: Tri-teaching for robust preference-based reinforcement learning with demonstrations

Shuaiyi Huang, Mara Levy, Anubhav Gupta, Daniel Ekpo, Ruijie Zheng, and Abhinav Shrivastava. Trend: Tri-teaching for robust preference-based reinforcement learning with demonstrations. arXiv preprint arXiv:2505.06079, 2025

2025 arXiv
[15]

Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu

Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017

2017 arXiv
[16]

One solution is not all you need: Few-shot extrapolation via structured maxent rl

Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33: 0 8198--8210, 2020

2020
[17]

B-pref: Benchmarking preference-based reinforcement learning

Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021 a

2021
[18]

Smith, and Pieter Abbeel

Kimin Lee, Laura M. Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. Proceedings of the 38th International Conference on Machine Learning, pages 6152--6163, 2021 b

2021
[19]

Evolution through the search for novelty

Joel Lehman. Evolution through the search for novelty. 2012

2012
[20]

Evolving a diversity of virtual creatures through novelty search and local competition

Joel Lehman and Kenneth O Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pages 211--218, 2011

2011
[21]

Reward uncertainty for exploration in preference-based reinforcement learning

Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022

2022
[22]

Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning

Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 22270--22284, 2022

2022
[23]

Efficient preference-based reinforcement learning using learned dynamics models

Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel S Brown. Efficient preference-based reinforcement learning using learned dynamics models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2921--2928. IEEE, 2023

2023
[24]

Variquery: Vae segment-based active learning for query selection in preference-based reinforcement learning

Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Variquery: Vae segment-based active learning for query selection in preference-based reinforcement learning. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7878-...

2023
[25]

Rewards encoding environment dynamics improves preference-based reinforcement learning

Katherine Metcalf, Miguel Sarabia, and Barry-John Theobald. Rewards encoding environment dynamics improves preference-based reinforcement learning. arXiv preprint arXiv:2211.06527, 2022

2022 arXiv
[26]

Illuminating search spaces by mapping elites

Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015

2015 arXiv
[27]

Xland-minigrid: Scalable meta-reinforcement learning environments in jax

Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinforcement learning environments in jax. Advances in Neural Information Processing Systems, 37: 0 43809--43835, 2024

2024
[28]

Policy gradient assisted MAP-Elites

Olle Nilsson and Antoine Cully. Policy gradient assisted MAP-Elites . In Proceedings of the Genetic and Evolutionary Computation Conference, pages 866--875. ACM, 2021

2021
[29]

Discovering diverse solutions in deep reinforcement learning by maximizing state--action-based mutual information

Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state--action-based mutual information. Neural Networks, 152: 0 90--104, 2022

2022
[30]

Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203.10050, 2022

2022 arXiv
[31]

Effective diversity in population based reinforcement learning

Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 18050--18062, 2020

2020
[32]

Jaxmarl: Multi-agent rl environments in jax

Alexander Rutherford. Jaxmarl: Multi-agent rl environments in jax. Decision-making, page 1, 2022

2022
[33]

Evolution strategies as a scalable alternative to reinforcement learning

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017

2017 arXiv
[34]

Dynamics-aware unsupervised discovery of skills

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. International Conference on Learning Representations, 2020

2020
[35]

Reinforcement learning: An introduction, volume 1

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

1998
[36]

Deepmind control suite

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

2018 arXiv
[37]

Ball, Vu Nguyen, Binxin Ru, and Michael A

Xingchen Wan, Cong Lu, Jack Parker-Holder, Philip J. Ball, Vu Nguyen, Binxin Ru, and Michael A. Osborne. Bayesian generational population-based training. In Proceedings of the First International Conference on Automated Machine Learning, pages 14/1--27, 2022

2022
[38]

A survey of preference-based reinforcement learning methods

Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes F \"u rnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 0 (1): 0 1--46, 2017

2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Forecasting with Multiple Seasonality

Mohammed Nabi Alam, Kok-Why Duong, Sepideh Hosseini, and Kathryn Kasmarik. Evolutionary reinforcement learning: A survey. arXiv preprint arXiv:2008.12340, 2020

work page Pith review arXiv 2008

[2] [2]

Batch active preference-based learning of reward functions

Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning, pages 519--528. PMLR, 2018

work page 2018

[3] [3]

Landolfi, Dylan P

Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020

work page 2020

[4] [4]

Rank analysis of incomplete block designs: I

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952

[5] [5]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30: 0 4299--4307, 2017

work page 2017

[6] [6]

Autonomous skill discovery with quality-diversity and unsupervised descriptors

Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 81--89. ACM, 2019

work page 2019

[7] [7]

Faster Improvement Rate Population Based Training

Valentin Dalibard and Max Jaderberg. Faster improvement rate population based training. arXiv preprint arXiv:2109.13800, 2021

work page Pith review arXiv 2021

[8] [8]

Diversity is all you need: Learning skills without a reward function

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019

work page 2019