Pith. sign in

REVIEW 4 major objections 4 minor 55 references

Preference-based Multi-Objective Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pairwise preferences between trajectory segments can replace hand-designed reward functions and still recover all Pareto-optimal policies in multi-objective reinforcement learning.

desk verdict Sensible PbRL+MORL integration, but the theory is tautological or rests on a false assumption, and the empirical claims lack statistics. read the letter →

arxiv 2507.14066 v1 pith:KPWANDOX submitted 2025-07-18 cs.LG

Ni Mu , Yao Luan , Qing-Shan Jia This is my paper · ORCID
classification cs.LG
keywords preference-basedreinforcementlearningmulti-objectiveParetofrontierrewardmodelingBradley-TerrymodelEnvelopeQ-learningweight-conditionedpolicyautonomousdriving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that pairwise preference judgments, provided by a teacher as 'segment A is better than segment B under weight w', carry enough information to train multi-objective reinforcement learning policies across the entire Pareto frontier, without hand-designing a vector reward. It proves that a consistent teacher's preferences recover the convex Pareto frontier by traversing weights, and the non-convex parts of the frontier by comparing policies against unit-vector weights. The load-bearing construction is a multi-objective reward model trained with a Bradley-Terry preference model; Theorem 4 shows that if this reward model perfectly matches the teacher, maximizing the weighted discounted return is equivalent to selecting the optimal policy for that weight. On benchmark, multi-energy management, and multi-lane highway tasks, the method performs on par with or better than an oracle that trains on the ground-truth reward function.

What carries the argument

The machinery is a learned multi-objective reward model $\hat{r}_\psi$: a vector-valued reward in $\mathbb{R}^m$ trained by minimizing the cross-entropy loss (13) under a Bradley-Terry preference predictor (14), so that $\exp\big(\sum_t \gamma^t w^\top \hat{r}(s_t^1,a_t^1)\big)\big/\sum_i \exp\big(\sum_t \gamma^t w^\top \hat{r}(s_t^i,a_t^i)\big)$ predicts the teacher's probability of preferring segment 1 under weight $w$. This reward model replaces the hand-designed scalar reward and is consumed by Envelope Q-learning, whose Bellman optimality operator with an optimality filter over $(a,w)$ guarantees convergence of the multi-objective Q-function. Coverage of the whole frontier rests on Assumption 4, which makes the optimal policy locally constant in $w$ and reduces the weight space to a finite grid $W[N_w]$, plus the identity-weight comparison procedure of Algorithm 2 (Theorem 3) for non-convex frontiers.

What would settle it

In a two-objective MDP with a known ground-truth reward, train the scripted teacher and reward model $\hat{r}$ as in the paper, then search for a policy that maximizes $\sum_t \gamma^t w^\top \hat{r}$ but is dominated under the true reward for the same $w$; if such a policy exists, Theorem 4's equivalence fails. Alternatively, construct an MDP whose Pareto frontier has a single sharp corner at an isolated weight $w_0$ and run Algorithm 1 on a grid not containing $w_0$: if the corner policy is not returned, the claim that the entire frontier is covered by the finite grid fails.

Watch

Extended reading notes

Core claim

The central discovery is a preference-based route to Pareto-optimal policies in multi-objective RL. Given a teacher who can compare trajectory segments under any weight vector, the paper constructs an explicit multi-objective reward model $\hat{r}_\psi$ using the Bradley-Terry model and cross-entropy loss (Eqs. 13–14), then feeds that reward model into Envelope Q-learning to obtain a policy $\pi_\phi(a|s,w)$. The paper proves Theorem 4: if $\hat{r}$ satisfies Eq. (15), i.e., it ranks any two segments exactly as the teacher does under any weight, then maximizing $J(\pi)=\sum_t \gamma^t w^\top \hat{r}(s_t,a_t)$ is equivalent to picking the optimal policy $\pi^*(\cdot|\cdot,w)$. Theorems 1–2 show that the weight-traversal Algorithm 1 yields exactly the convex Pareto frontier as the segment length $H\to\infty$, and Theorem 3 with Algorithm 2 shows that unit-vector weight comparisons recover non-convex frontiers. Empirically, Pb-MORL matches the oracle on Deep Sea Treasure, Fruit Tree, and Resource Gathering, and surpasses it on the energy and highway tasks.

Load-bearing premise

The whole proof chain hinges on the reward model $\hat{r}$ perfectly reproducing the teacher's preferences for every pair of segments and every weight (Eq. 15), and on the optimal policy being constant within an $\epsilon$-ball of every sampled weight (Assumption 4); if either fails, a learned or finite-grid policy can fall outside the true Pareto frontier.

Editorial extensions

If this is right

  • Replacing hand-designed multi-objective rewards with pairwise preference data turns any convergent MORL algorithm (the paper uses Envelope Q-learning) into a Pareto-optimal policy learner, per Theorem 4.
  • The full convex Pareto frontier is reachable by sweeping weights (Theorem 2), and non-convex frontiers are reachable by comparing policies under unit-vector weights (Theorem 3), so no special treatment is needed for non-convex trade-offs.
  • The learned policy $\pi_\phi(a|s,w)$ is weight-conditioned, so changing the objective trade-off at deployment only requires changing $w$, not retraining; the paper demonstrates this in energy management and highway driving.
  • On the energy and highway tasks, preference-driven learning outperformed the oracle that trains on ground-truth rewards, suggesting preferences act as a stabilizer against overfitting to immediate objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If reward-model error is bounded, Theorem 4 likely degrades gracefully: the suboptimality gap between the learned and true Pareto policy should scale with the worst-case misranking probability, a bound the paper does not derive but could be tested on the benchmark tasks.
  • The theorem's perfect-alignment condition (Eq. 15) is unachievable with finite preference data; in practice the method's success depends on the scripted teacher's preferences matching the ground-truth reward, so real-human evaluation would be the decisive test of whether the preference signal alone is sufficient.
  • Assumption 4's local-constancy is the fragile step in the frontier-coverage claim: the finite grid $W[N_w]$ can miss a policy that is optimal at an isolated weight, so the empirical near-oracle results do not by themselves establish the theoretical claim of whole-frontier coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes Pb-MORL, a framework that replaces hand-designed multi-objective rewards with an explicit multi-objective reward model trained from preference feedback and then optimizes a weight-conditioned policy via Envelope Q-Learning. The authors make four theoretical claims: optimal policies under any weight are Pareto optimal (Theorem 1); traversing a finite weight grid recovers the entire convex Pareto frontier (Theorem 2); non-convex frontiers can be recovered through unit-weight comparisons (Theorem 3); and optimizing a reward model that is perfectly aligned with teacher preferences is equivalent to selecting the teacher-optimal policy (Theorem 4). The experimental section compares Pb-MORL against EQL with ground-truth rewards on three benchmarks, an energy-storage task, and a highway-driving task, and claims that Pb-MORL surpasses the oracle on the latter two domains.

Significance. The motivation is strong: preference-based learning can reduce reward-engineering burden in multi-objective control, and combining PbRL with MORL is a timely problem. The proposed pipeline is simple and the scripted-teacher evaluation is reproducible in spirit. However, the theoretical guarantees are the paper's central contribution and they are not established. Theorem 4 is circular because its perfect-alignment premise already contains the conclusion. Assumption 4, which is needed for the finite-grid frontier claim, is false in generic finite MDPs with ties. Theorem 3's unit-weight dominance test misclassifies weak dominance. The empirical claim of surpassing the oracle also lacks statistical support. If the theorems were correct, the paper would be a significant step for preference-based MORL; as submitted, the headline claims are unsupported.

major comments (4)
  1. [Section III.B, Theorem 4 (Eqs. (15)-(19))] The statement to be proved is already contained in the perfect-alignment assumption. Equation (15) asserts that for every segment pair and every length H, the teacher's preference ordering coincides with the ordering by discounted w^T r-hat; letting H become arbitrarily long gives Eq. (16), which is exactly the policy-level equivalence claimed by the theorem. The proof then assumes a policy pi' with larger r-hat return, invokes Eq. (15) to infer that the teacher would prefer pi', and writes Eq. (19) as a contradiction. This rewrites the premise rather than proving that optimizing a learned reward model selects the teacher-optimal policy. No argument shows that a reward model trained by minimizing the cross-entropy loss (13) satisfies (15) even approximately. The displayed implication in Eq. (19) also appears to have the preference direction reversed: if pi' has the larger r-hat return, perfect alignment would give pi' ≻_w pi*, which is the assumed inequality, not a contradiction.
  2. [Section III.A, Assumption 4 and Theorem 2] Assumption 4 requires the argmax policy to be constant on an epsilon-ball around every weight w0. This fails whenever the optimal policy changes at a tie. For example, consider three deterministic policies with returns A=(0,1), B=(1,0), and C=(0.5,0.5). Policy C is Pareto-optimal, but it is the unique maximizer of w^T R only at w=(0.5,0.5); for every epsilon>0 there are weights in the ball where A or B is strictly better. Hence no finite grid W[N_w] with cells of side epsilon can contain a weight at which C is the unique optimal policy, and Algorithm 1 will never return C for a generic grid. This contradicts Theorem 2's claim that Algorithm 1 obtains the entire convex Pareto frontier, and it also weakens Theorem 1, whose proof relies on comparing policies at a weight in the grid. The authors would need an epsilon chosen per policy, based on the gap between value functions, but no such construction is given.
  3. [Section III.A, Theorem 3 and Algorithms 2 and 4] The dominance test uses strict inequalities w_k^T R(sigma_i) > w_k^T R(sigma_j) for all unit weights w_k in W_I. This misclassifies Pareto dominance whenever the dominating policy ties the dominated policy on at least one objective. For instance, R'=(1,2) is dominated by R*=(2,2), but w_1^T R'=1 < 2 and w_2^T R'=2 = 2, so no unit weight yields strict inequality in every coordinate. Algorithm 2 would not remove R', and Algorithm 4 would keep both policies in the estimated Pareto set. Thus Theorem 3's claim that an arbitrary Pareto frontier can be completely obtained via unit-weight comparisons is false as stated.
  4. [Section IV.C-D, Figs. 4 and 5] The abstract and conclusion claim that Pb-MORL 'surpasses' the oracle method on the energy and highway tasks, but the figures plot only means over five seeds, without error bars, confidence intervals, or significance tests. Because the scripted teacher is constructed from the same ground-truth reward that EQL uses directly, the claim that a learned proxy outperforms the reward-based oracle is surprising and needs statistical support. The post-hoc explanations in the text (e.g., preference emphasis on relative benefits) are not evidence. This is a load-bearing empirical claim and should be quantified or softened.
minor comments (4)
  1. [Section III.A, Algorithms 1-2] Algorithms 1 and 2 iterate 'for each pi_i in Pi' over the full policy space, which is infinite and not enumerable in general; the theorems should state explicitly that these are exhaustive-search arguments or restrict Pi to a finite policy class.
  2. [Section II.B, Eq. (13)] The loss in Eq. (13) writes p(0) and p(1) with p in {0,0.5,1}; it is unclear how the indeterminate preference p=0.5 is encoded in the cross-entropy loss and whether such comparisons are used for training.
  3. [Section III.A, Theorem 1 proof and Corollary 1] The proof of Theorem 1 uses H tending to infinity but Algorithm 1 operates with finite segments; Corollary 1 introduces an extra delta>0 lower bound on return differences that is not implied by Assumptions 1-4, so the finite-horizon guarantee depends on an unstated condition.
  4. [Section III.A, Assumption 4] The formula for the grid size, N_w = |W|/epsilon^m <= epsilon^{-m}, ignores the normalization of the simplex and should be stated as (1/epsilon)^{m-1} or similar; the expression as written is dimensionally inconsistent.

Circularity Check

1 steps flagged · score 8.0 of 10

Theorem 4 is a tautology: the 'perfectly aligned reward model' assumption is the theorem's conclusion restated as a definition.

  1. self definitional [Section III.B, Theorem 4, Eq. (15)-(17)]
    "If the reward model ˆr is perfectly aligned with the teacher's preferences, that is, for segments (σ0, σ1) with arbitrary length H, σ0 ≻w σ1 ⇐⇒ Σ_{(s,a)∼σ0} γ^t w^T ˆr(st, at) > Σ_{(s,a)∼σ1} γ^t w^T ˆr(st, at). ... Then, under a given weight vector w, maximizing the discounted return J(π) = Σ_{t=0}^{∞} γ^t w^T ˆr(st, at) is equivalent to selecting the optimal policy π∗(·|·,w)."

    Equation (15) does not state a substantive premise from which the conclusion is derived; it defines 'perfectly aligned' as the biconditional that teacher preference order is exactly reward-model return order. The theorem's conclusion is the same biconditional lifted to policies: the policy maximizing J(π) is, by Eqs. (15)-(16), exactly the policy the teacher prefers. The proof only postulates a π* optimal under J and reuses the same biconditional (Eq. 19) to contradict the existence of any higher-scoring policy. No independent bridge from preferences to reward-model optimality is constructed, so the 'derivation' reduces by construction to its own assumption: maximizing the reward model selects the preferred policy because that is what perfect alignment was defined to mean.

full rationale

The clearest circular step is Theorem 4, which is the paper's central bridge from preference data to policy optimality. The assumption of perfect alignment (Eq. 15) is the conclusion restated as a definitional biconditional, so the proof is a tautology. Training the reward model with the Bradley-Terry cross-entropy loss (Eqs. 13-14) does not repair this: it only fits the model to observed preferences and cannot establish Eq. 15 for arbitrary unseen segments and policies. Theorems 1-3 are conditional on explicit assumptions such as Assumption 4 and on standard scalarization and Pareto-dominance facts; their failure to deliver the claimed 'entire Pareto frontier' when Assumption 4 is violated is a correctness concern, not an additional circularity. The experimental benchmarks use a scripted teacher derived from ground truth rewards (Eqs. 20-22) and are evaluated against that same ground truth; this limits the force of the 'surpasses oracle' claim but is not circular reasoning. There is no load-bearing self-citation chain. Overall, the central equivalence in Theorem 4 is forced by definition, giving a circularity score of 8.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central claims rest on four explicit assumptions about the teacher and the weight space (Assumptions 1-4), on a perfect-alignment condition that is effectively the theorem's conclusion, and on borrowed convergence results for EQL and the Bradley-Terry preference model. The only hand-chosen numeric parameters are the per-task segment lengths H and the weight-grid size N_w, both of which affect whether the finite-horizon, finite-grid regime matches the theory.

free parameters (2)
  • Per-task segment length H = H=7 (DST), H=6 (FT), H=10 (RG), H=10 (energy), H=3,6 (highway)
    The theoretical guarantee (Theorem 1, Corollary 1) requires H→∞ or H above a threshold; experiments use finite H chosen by hand per task, so the finite-H regime is unproven and the choice is a free parameter affecting the preference signal.
  • Weight grid size N_w = 10 (in Algorithm 3); theory uses N_w = ε^{-m} with unspecified ε
    Assumption 4 introduces ε and the discretization size N_w; the actual ε in experiments is not specified, so the finite-grid approximation is a free parameter that can miss isolated Pareto-optimal policies.
assumptions (7)
  • domain assumption Assumption 1: preferences exist for arbitrary segment pairs and weights and satisfy symmetry, consistency, transitivity.
    Used in Theorems 1-4 to model the teacher; transitivity in particular is a strong idealization of human judgments.
  • domain assumption Assumption 2: a perfect teacher provides consistent preference feedback for any two segments under any weight.
    The scripted teacher in experiments uses ground-truth reward; real teachers would be noisy, and the paper does not analyze noise.
  • domain assumption Assumption 3: a true bounded reward function exists and the teacher's preferences derive from it.
    This is what lets preference data carry information about the Pareto frontier; boundedness is used to bound the truncation error in Theorem 1.
  • ad hoc to paper Assumption 4: the optimal policy is constant in an ε-neighborhood of each weight.
    Justifies the finite weight grid W[Nw]; continuity of value functions does not imply local constancy of the optimal policy, so this is an idealized assumption tailored to the discretization argument.
  • ad hoc to paper Perfect alignment condition (Eq. 15-16): r-hat matches the teacher's preferences exactly.
    The premise of Theorem 4; the proof does not establish existence of such a reward model, it only restates the condition as the conclusion.
  • domain assumption EQL convergence (Yang et al. 2019), used as the MORL optimizer without re-proving convergence.
    The paper relies on the cited convergence guarantee for the multi-objective Bellman operator.
  • domain assumption Bradley-Terry model for preference probability (Eq. 14).
    Standard in PbRL; the exponential model is assumed, not derived, and imposes a specific functional form on preferences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preference-based Multi-Objective Reinforcement Learning." pith.science (2026). https://pith.science/paper/KPWANDOX

@misc{pith2026250714066,
  author       = {Pith},
  title        = {Pith review of: Preference-based Multi-Objective Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPWANDOX}},
  note         = {Machine review of arXiv:2507.14066}
}
read the original abstract

Multi-objective reinforcement learning (MORL) is a structured approach for optimizing tasks with multiple objectives. However, it often relies on pre-defined reward functions, which can be hard to design for balancing conflicting goals and may lead to oversimplification. Preferences can serve as more flexible and intuitive decision-making guidance, eliminating the need for complicated reward design. This paper introduces preference-based MORL (Pb-MORL), which formalizes the integration of preferences into the MORL framework. We theoretically prove that preferences can derive policies across the entire Pareto frontier. To guide policy optimization using preferences, our method constructs a multi-objective reward model that aligns with the given preferences. We further provide theoretical proof to show that optimizing this reward model is equivalent to training the Pareto optimal policy. Extensive experiments in benchmark multi-objective tasks, a multi-energy management task, and an autonomous driving task on a multi-line highway show that our method performs competitively, surpassing the oracle method, which uses the ground truth reward function. This highlights its potential for practical applications in complex real-world systems.

Figures

Figures reproduced from arXiv: 2507.14066 by the authors.

Figure 1
Figure 1. A demonstration of the proposed Pb-MORL framework. An explicit multi-objective reward model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The training curves of the expected utility and hypervolume on three multi-objective benchmark tasks. The experiments are conducted on 5 random [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. The training curves of the expected utility and hypervolume on the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The training curves of the expected utility and hypervolume on the [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 40 canonical work pages

  1. [1]

    A survey on modeling and optimizing multi-objective systems,

    J.-H. Cho, Y . Wang, R. Chen, K. S. Chan, and A. Swami, “A survey on modeling and optimizing multi-objective systems,”IEEE Communi- cations Surveys & Tutorials, vol. 19, no. 3, pp. 1867–1901, 2017

    work page 1901

  2. [2]

    Active learning with fairness-aware clustering for fair classification considering multiple sensitive attributes,

    Z. Liu, X. Zhang, and B. Jiang, “Active learning with fairness-aware clustering for fair classification considering multiple sensitive attributes,” Information Sciences, vol. 647, p. 119521, 2023

    work page 2023

  3. [3]

    Constrained ordinal opti- mization—a feasibility model based approach,

    X. Guan, C. Song, Y .-C. Ho, and Q. Zhao, “Constrained ordinal opti- mization—a feasibility model based approach,”Discrete Event Dynamic Systems, vol. 16, no. 2, pp. 279–299, 2006

    work page 2006

  4. [4]

    Reliability/cost-based multi-objective pareto optimal design of stand- alone wind/pv/fc generation microgrid system,

    H. R. Baghaee, M. Mirsalim, G. B. Gharehpetian, and H. Talebi, “Reliability/cost-based multi-objective pareto optimal design of stand- alone wind/pv/fc generation microgrid system,”Energy, vol. 115, pp. 1022–1041, 2016

    work page 2016

  5. [5]

    Toward personalized decision making for autonomous vehicles: a constrained multi-objective reinforcement learning tech- nique,

    X. He and C. Lv, “Toward personalized decision making for autonomous vehicles: a constrained multi-objective reinforcement learning tech- nique,”Transportation research part C: emerging technologies, vol. 156, p. 104352, 2023

    work page 2023

  6. [6]

    B-pref: Benchmarking preference-based reinforcement learning,

    K. Lee, L. Smith, A. Dragan, and P. Abbeel, “B-pref: Benchmarking preference-based reinforcement learning,” inThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021

    work page 2021

  7. [7]

    Self- supervised online reward shaping in sparse-reward environments,

    F. Memarian, W. Goo, R. Lioutikov, S. Niekum, and U. Topcu, “Self- supervised online reward shaping in sparse-reward environments,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 2369–2375

    work page 2021

  8. [8]

    Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning,

    P. Mannion, S. Devlin, J. Duggan, and E. Howley, “Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning,” The Knowledge Engineering Review, vol. 33, p. e23, 2018

    work page 2018

Show all 55 references
  1. [9]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsuper- vised pre-training,

    K. Lee, L. M. Smith, and P. Abbeel, “Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsuper- vised pre-training,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 6152–6163

    2021

  2. [10]

    Reinforcement learning and the reward engineering princi- ple,

    D. Dewey, “Reinforcement learning and the reward engineering princi- ple,” in2014 AAAI Spring Symposium Series, 2014

    2014

  3. [11]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,”Advances in neural information processing systems, vol. 30, 2017

    2017

  4. [12]

    A bayesian approach for policy learning from trajectory preference queries,

    A. Wilson, A. Fern, and P. Tadepalli, “A bayesian approach for policy learning from trajectory preference queries,”Advances in neural infor- mation processing systems, vol. 25, 2012

    2012

  5. [13]

    R. S. Sutton, A. G. Bartoet al.,Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1

    1998

  6. [14]

    Playing atari with deep reinforcement learn- ing,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier- stra, and M. Riedmiller, “Playing atari with deep reinforcement learn- ing,”arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  7. [15]

    E-mapp: Efficient multi-agent reinforcement learning with parallel program guidance,

    C. Chang, N. Mu, J. Wu, L. Pan, and H. Xu, “E-mapp: Efficient multi-agent reinforcement learning with parallel program guidance,” in Advances in Neural Information Processing Systems, 2022

    2022

  8. [16]

    Mastering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Boltonet al., “Mastering the game of go without human knowledge,”nature, vol. 550, no. 7676, pp. 354–359, 2017

    2017

  9. [17]

    Simplify twin crane scheduling in railway yard by spatial task assignment,

    Y . Luan and Q.-S. Jia, “Simplify twin crane scheduling in railway yard by spatial task assignment,” in2023 China Automation Congress (CAC). IEEE, 2023, pp. 3034–3039

    2023

  10. [18]

    Large-scale data center cooling control via sample-efficient reinforcement learning,

    N. Mu, X. Hu, and Q.-S. Jia, “Large-scale data center cooling control via sample-efficient reinforcement learning,” in2024 IEEE 20th Inter- national Conference on Automation Science and Engineering (CASE). IEEE, 2024

    2024

  11. [19]

    An efficient real- time railway container yard management method based on partial de- coupling,

    Y . Luan, Q.-S. Jia, Y . Xing, Z. Li, and T. Wang, “An efficient real- time railway container yard management method based on partial de- coupling,”IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 14 183–14 200, 2025

    2025

  12. [20]

    Integrating mechanism and data: Rein- forcement learning based on multi-fidelity model for data center cooling control,

    N. Mu, X. Hu, and Q.-S. Jia, “Integrating mechanism and data: Rein- forcement learning based on multi-fidelity model for data center cooling control,” in2023 China Automation Congress (CAC). IEEE, 2023, pp. 5283–5288

    2023

  13. [21]

    Incentive-oriented power-carbon emissions trading-tradable green certificate integrated market mecha- nisms using multi-agent deep reinforcement learning,

    X. Guo, X. Zhang, and X. Zhang, “Incentive-oriented power-carbon emissions trading-tradable green certificate integrated market mecha- nisms using multi-agent deep reinforcement learning,”Applied Energy, vol. 357, p. 122458, 2024

    2024

  14. [22]

    Openai gym,

    G. Brockman, “Openai gym,”arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  15. [23]

    Exploration by random network distillation,

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” inInternational Conference on Learning Representations, 2018

    2018

  16. [24]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  17. [25]

    Reward learning from human preferences and demonstrations in atari,

    B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning from human preferences and demonstrations in atari,” Advances in neural information processing systems, vol. 31, 2018

    2018

  18. [26]

    S-epoa: Overcoming the indistinguishability of segments with skill-driven preference-based re- inforcement learning,

    N. Mu, Y . Luan, Y . Yang, and Q.-S. Jia, “S-epoa: Overcoming the indistinguishability of segments with skill-driven preference-based re- inforcement learning,”arXiv preprint arXiv:2408.12130, 2024

    2024 arXiv

  19. [27]

    Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,

    J. Park, Y . Seo, J. Shin, H. Lee, P. Abbeel, and K. Lee, “Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,” inInternational Conference on Learning Representations, 2022

    2022

  20. [28]

    Few-shot preference learning for human- in-the-loop rl,

    D. J. Hejna III and D. Sadigh, “Few-shot preference learning for human- in-the-loop rl,” inConference on Robot Learning. PMLR, 2023, pp. 2014–2025

    2023

  21. [29]

    Learning to summarize with human feedback,

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Rad- ford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,”Advances in Neural Information Processing Systems, vol. 33, pp. 3008–3021, 2020

    2020

  22. [30]

    A toolkit for reliable benchmarking and research in multi-objective reinforcement learning,

    F. Felten, L. N. Alegre, A. Now ´e, A. L. Bazzan, E.-G. Talbi, G. Danoy, and B. C. da Silva, “A toolkit for reliable benchmarking and research in multi-objective reinforcement learning,” inProceedings of the 37th International Conference on Neural Information Processing System...

    2023

  23. [31]

    A practical guide to multi-objective reinforcement learning and planning,

    C. F. Hayes, R. R ˘adulescu, E. Bargiacchi, J. K ¨allstr¨om, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz et al., “A practical guide to multi-objective reinforcement learning and planning,”Autonomous Agents and Multi-Agent Systems, vol. 36, ...

    2022

  24. [32]

    Human-in-the-loop policy optimization for preference-based multi-objective reinforcement learning,

    K. Li and H. Guo, “Human-in-the-loop policy optimization for preference-based multi-objective reinforcement learning,”arXiv preprint arXiv:2401.02160, 2024

    2024 arXiv

  25. [33]

    A generalized algorithm for multi-objective reinforcement learning and policy adaptation,

    R. Yang, X. Sun, and K. Narasimhan, “A generalized algorithm for multi-objective reinforcement learning and policy adaptation,” inPro- ceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, pp. 14 636–14 647

    2019

  26. [34]

    Multi-objective rein- forcement learning for the expected utility of the return,

    D. M. Roijers, D. Steckelmacher, and A. Now ´e, “Multi-objective rein- forcement learning for the expected utility of the return,” inProceedings of the Adaptive and Learning Agents workshop at FAIM, vol. 2018, 2018

    2018

  27. [35]

    Prediction- guided multi-objective reinforcement learning for continuous robot con- trol,

    J. Xu, Y . Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik, “Prediction- guided multi-objective reinforcement learning for continuous robot con- trol,” inInternational conference on machine learning. PMLR, 2020, pp. 10 607–10 616

    2020

  28. [36]

    Pareto conditioned net- works,

    M. Reymond, E. Bargiacchi, and A. Now ´e, “Pareto conditioned net- works,” inProceedings of the 21st International Conference on Au- tonomous Agents and Multiagent Systems, 2022, pp. 1110–1118

    2022

  29. [37]

    Sample-efficient multi-objective learning via generalized pol- icy improvement prioritization,

    L. N. Alegre, A. L. Bazzan, D. M. Roijers, A. Now ´e, and B. C. da Silva, “Sample-efficient multi-objective learning via generalized pol- icy improvement prioritization,” inProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, 2023, pp. ...

    2023

  30. [38]

    Q-learning,

    C. J. Watkins and P. Dayan, “Q-learning,”Machine learning, vol. 8, pp. 279–292, 1992

    1992

  31. [39]

    A. W. Naylor and G. R. Sell,Linear operator theory in engineering and science. Springer Science & Business Media, 1982

    1982

  32. [40]

    Convergence of q-learning: A simple proof,

    F. S. Melo, “Convergence of q-learning: A simple proof,”Institute Of Systems and Robotics, Tech. Rep, pp. 1–4, 2001

    2001

  33. [41]

    Decentralized multi-agent reinforcement learning: An off-policy method,

    K. Li and Q.-S. Jia, “Decentralized multi-agent reinforcement learning: An off-policy method,” 2021. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14

    2021

  34. [42]

    An ocba-based method for efficient sample collection in reinforcement learning,

    K. Li, X. Jin, Q.-S. Jia, D. Ren, and H. Xia, “An ocba-based method for efficient sample collection in reinforcement learning,”IEEE Trans- actions on Automation Science and Engineering, vol. 21, no. 3, pp. 3615–3626, 2024

    2024

  35. [43]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,”Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  36. [44]

    Preference-based multi-objective reinforcement learning with explicit reward modeling,

    N. Mu, Y . Luan, and Q.-S. Jia, “Preference-based multi-objective reinforcement learning with explicit reward modeling,” in2024 China Automation Congress (CAC). IEEE, 2024

    2024

  37. [45]

    Clarify: Contrastive preference reinforcement learning for untangling ambigu- ous queries,

    N. Mu, H. Hu, X. Hu, Y . Yang, B. Xu, and Q.-S. Jia, “Clarify: Contrastive preference reinforcement learning for untangling ambigu- ous queries,” inProceedings of the 42th International Conference on Machine Learning, 2025

    2025

  38. [46]

    Zitzler,Evolutionary algorithms for multiobjective optimization: Methods and applications

    E. Zitzler,Evolutionary algorithms for multiobjective optimization: Methods and applications. Shaker Ithaca, 1999, vol. 63

    1999

  39. [47]

    Query-policy mis- alignment in preference-based reinforcement learning,

    X. Hu, J. Li, X. Zhan, Q.-S. Jia, and Y .-Q. Zhang, “Query-policy mis- alignment in preference-based reinforcement learning,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  40. [48]

    Empirical evaluation methods for multiobjective reinforcement learning algorithms,

    P. Vamplew, R. Dazeley, A. Berry, R. Issabekov, and E. Dekker, “Empirical evaluation methods for multiobjective reinforcement learning algorithms,”Machine learning, vol. 84, pp. 51–80, 2011

    2011

  41. [49]

    Learning all optimal policies with multiple criteria,

    L. Barrett and S. Narayanan, “Learning all optimal policies with multiple criteria,” inProceedings of the 25th international conference on Machine learning, 2008, pp. 41–47

    2008

  42. [50]

    An environment for autonomous driving decision-making,

    E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018

    2018

  43. [51]

    Congested traffic states in empirical observations and microscopic simulations,

    M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,”Physical review E, vol. 62, no. 2, p. 1805, 2000

    2000

  44. [52]

    General lane-changing model mobil for car-following models,

    A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,”Transportation Research Record, vol. 1999, no. 1, pp. 86–94, 2007

    1999

  45. [53]

    Implementing deep reinforcement learning (drl)-based driving styles for non-player vehicles,

    L. Forneris, A. Pighetti, L. Lazzaroni, F. Bellotti, A. Capello, M. Cossu, and R. Berta, “Implementing deep reinforcement learning (drl)-based driving styles for non-player vehicles,”International Journal of Serious Games, vol. 10, no. 4, pp. 153–170, 2023

    2023

  46. [54]

    Enhancing autonomous vehicle training with language model integra- tion and critical scenario generation,

    H. Tian, K. Reddy, Y . Feng, M. Quddus, Y . Demiris, and P. Angeloudis, “Enhancing autonomous vehicle training with language model integra- tion and critical scenario generation,”arXiv preprint arXiv:2404.08570, 2024

    2024 arXiv

  47. [55]

    Listwise reward estimation for offline preference-based reinforcement learning,

    H. Choi, S. Jung, H. Ahn, and T. Moon, “Listwise reward estimation for offline preference-based reinforcement learning,” inForty-first Inter- national Conference on Machine Learning, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.