Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The paper introduces GraphAllocBench, a graph-based benchmark for preference-conditioned multi-objective policy learning, and argues that evaluating PCPL policies requires metrics beyond hypervolume — specifically an ordering score for pref

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:12 UTC pith:QTNWNFF7

load-bearing objection A genuinely useful new benchmark environment, but the ordering score has a real bug that needs fixing before the metric can be trusted. the 3 major comments →

arxiv 2601.20753 v4 pith:QTNWNFF7 submitted 2026-01-28 cs.LG

GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

Zhiheng Jiang , Yunzhe Wang , Ryan Marr , Ellen Novoseller , Benjamin T. Files , Volkan Ustun This is my paper
classification cs.LG
keywords Preference-conditioned policy learningMulti-objective reinforcement learningBenchmarkResource allocationGraph neural networksHypervolumeOrdering scorePareto front
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing benchmarks for preference-conditioned multi-objective reinforcement learning are too small and rigid to reveal how well a single policy follows arbitrary user preferences. To fix this, it builds CityPlannerEnv, a configurable graph-based resource allocation sandbox, and packages it as GraphAllocBench with problems that vary objective shapes, Pareto fronts, dependency structures, and scale. It introduces two evaluation metrics — the proportion of non-dominated solutions (PNDS) and the ordering score (OS) — and shows experimentally that these expose failure modes that hypervolume alone misses, such as preference-ignoring behavior in sparse-reward and non-convex settings. It also shows that graph neural network feature extractors beat MLPs on the large 100-node problems, suggesting GNNs are a promising direction for high-dimensional allocation tasks.

Core claim

On GraphAllocBench, hypervolume ratio alone gives an incomplete picture of a PCPL agent's quality. The paper's central discovery is that adding two cheap metrics — the proportion of non-dominated predicted solutions (PNDS) and the ordering score (OS), a Spearman rank correlation between preference weights and realized objective values — reveals preference-consistency failures that hypervolume hides, and that on large graph-structured allocation problems a heterogeneous graph neural network (HGNN) with preference-conditioned attention pooling achieves substantially higher hypervolume than an MLP with comparable or fewer parameters.

What carries the argument

CityPlannerEnv, a Gymnasium environment where an agent allocates units of resources to demands through a bipartite dependency graph, with objective functions defined over productions; the benchmark's evaluation protocol: Das-Dennis preference sampling, hypervolume ratio against analytically enumerated Pareto fronts where possible, PNDS, and Ordering Score (Spearman rank correlation between preference weights and objective values). The HGNN feature extractor with preference conditioning at node embeddings, residual connections, and attention pooling is the mechanism that scales to 100-demand graphs.

Load-bearing premise

The Ordering Score assumes that a constant reward sequence indicates perfect preference-following (it awards a score of 1), so a policy that produces zero or constant output for a sparse objective receives full credit for preference consistency in that dimension.

What would settle it

Compute the Ordering Score on a deliberately preference-blind policy — one that always outputs the same allocation regardless of preference — on Problem 1c (sparse rewards). If its OS approaches 1 while its hypervolume is near zero, the metric fails to punish preference-ignoring behavior, contradicting the paper's stated purpose.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the ordering score is taken seriously, PCPL benchmarks should report preference-following behavior, not just front quality.
  • The failure modes identified (sharp reward changes, sparse rewards, non-convex fronts, unbalanced objectives, local optima) give concrete targets for new PCPL algorithms.
  • GraphAllocBench's customizability means researchers can construct stress tests tailored to their algorithms.
  • HGNN-based feature extractors are a viable replacement for MLP extractors on large allocation graphs, achieving higher hypervolume with fewer parameters.
  • The scaling study with 20 objectives suggests that simple increasing objective functions may not degrade performance with more objectives, contrasting with typical evolutionary benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ordering score's degenerate handling of constant reward sequences (score 1 when all values equal) may inflate the apparent preference-following of policies that simply ignore preferences in sparse-reward problems; a corrected metric or a robustness check with constant policies is a natural follow-up.
  • The PNDS metric could be combined with a diversity measure to distinguish 'many non-dominated but clustered' from 'many non-dominated and spread-out' predictions.
  • The benchmark's bipartite allocation structure could be extended to stochastic events or risk-aware objectives, which the authors list as future work, making it a testbed for uncertainty-aware PCPL.
  • One could test whether the ordering score's Spearman approach is robust to monotonic transformations of the scalarization, since PCPL policies may produce objective vectors that are monotone in preference but not linearly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces GraphAllocBench, a Gymnasium-based benchmark built on CityPlannerEnv, a graph-structured resource allocation environment with customizable objective functions, preference vectors, and dependency graphs. It proposes two supplementary evaluation metrics beyond hypervolume: the Proportion of Non-Dominated Solutions (PNDS) and the Ordering Score (OS), intended to measure prediction reliability and preference consistency in preference-conditioned multi-objective reinforcement learning (PCPL). The experimental section compares a PPO-based PCPL agent with MLP and HGNN feature extractors against PD-MORL on problems 0–5, and MLP versus HGNN on large-scale problems 6a–c. The central claims are that GraphAllocBench exposes failure modes (sharp rewards, sparse rewards, non-convex Pareto fronts, unbalanced objectives, local optima) that hypervolume alone does not capture, and that HGNN architectures improve hypervolume on large graphs. The authors provide a GitHub link and detailed environment definitions, which is a strength for reproducibility.

Significance. If the benchmark and its metrics are sound, GraphAllocBench addresses a real gap in MORL evaluation: most existing benchmarks are small grid or continuous test problems without the flexible, graph-structured allocation structure needed to stress-test preference-conditioned policies. The code release and customizable environment are valuable assets to the community. The attempt to supplement hypervolume with preference-consistency diagnostics is timely and relevant for PCPL. However, the OS metric, which is central to the preference-consistency claim, currently treats constant reward sequences as perfectly preference-aligned, which can reward preference-ignoring policies. In addition, the empirical tables largely omit variance information, weakening the support for the headline comparisons. With a corrected OS definition and proper uncertainty reporting, the benchmark could become a useful testbed; as written, the central claim about preference-awareness is not yet established.

major comments (3)
  1. [Appendix B, Algorithm 1; Section 4.3] Lines 5–6 of Algorithm 1 set s_i,j = 1 whenever all values in the reward sequence J_i are equal. Spearman rank correlation is undefined for a constant sequence; assigning it a perfect positive score means that a policy producing the same objective value regardless of the preference vector is credited with perfect preference alignment along that dimension. This directly contradicts Section 4.3's statement that OS evaluates 'how well the agent follows the preference it is given.' In sparse-reward settings such as Problem 1c (and in any problem with floor/zero regions), constant zero-reward sequences are common, so the high OS values in Table 4 (e.g., 0.989 for PD-MORL and 0.945 for PCPL on Problem 1c) may be artifacts of this special case rather than evidence of preference-awareness. Since the abstract and Section 4.3 credit the supplementary metrics with exposing preference-following fail
  2. [Tables 2 and 4] The empirical claims are presented as 'significantly and consistently better' and 'best values,' but Table 4 reports no standard deviations or significance tests for any metric across the 5 seeds. Table 2 reports mean±std only for hypervolume, while PNDS and OS are given as bare means, with duplicated/garbled entries in the table body. Without variance information across seeds, the differences highlighted in the text (e.g., HGNN vs MLP hypervolume, PD-MORL vs PCPL ordering score) cannot be assessed. Please report full mean±std for every metric and seed, and either include appropriate significance tests or explicitly label the comparisons as exploratory.
  3. [Section 5.1, 'Sparse Rewards'] The text describes Problem 1c as involving the floor function J_i(P) = max(0, P_j − 5), but Table 3 lists logistic-type objective functions for Problem 1c. This discrepancy matters because the OS constant-sequence artifact is most acute in sparse-reward problems where many preferences yield zero reward. Please clarify which problems actually contain constant-zero reward regions, connect the metric discussion to the concrete objective definitions, and update the text or table accordingly.
minor comments (6)
  1. [Table 2] The table body has formatting errors: repeated values such as '0.880.88', '0.150.15', and '12.9±1.512.9±1.5' appear, and missing spaces separate columns. Please regenerate cleanly.
  2. [Abstract and Section 1] The abstract and introduction refer to experiments with 'several state-of-the-art PCPL algorithms,' but the paper compares against only PD-MORL plus the authors' own baselines. Please rephrase or add additional PCPL baselines.
  3. [Algorithm 1 vs. Section 4.3] Section 4.3 defines OS as rank correlation between preference weight w_i and objective value J_i, but Algorithm 1 computes Spearman(J_i, sorted(J_i)). If the sweep index is intended to be a monotonic proxy for w_i, this should be stated explicitly; otherwise the computation should directly use w_i.
  4. [Appendix C.3] The Smooth Tchebycheff scalarization hyperparameter is mentioned as tunable, but no values or selection procedure are provided. Please include the hyperparameter settings used in the experiments.
  5. [Appendix D / Code Links] The main text gives a GitHub URL while Appendix D points to an anonymous OpenReview-style link. These should be reconciled, and the final version should use a single persistent code link.
  6. [Table 3] The objective functions in Problem 0 use an undefined epsilon constant to avoid log(0). Please define epsilon and its value.

Circularity Check

0 steps flagged

No significant circularity; the Ordering Score constant-sequence rule is a metric-validity issue, not a circular derivation.

full rationale

The paper's central claims—that GraphAllocBench poses diverse challenges, that PNDS and OS complement hypervolume, and that HGNN-based policies scale better than MLPs—are supported by external evidence: comparisons against the PD-MORL baseline, analytical Pareto fronts where enumeration is feasible, and standard PPO/SB3 implementations. The proposed metrics are defined explicitly (Section 4.2-4.3, Algorithm 1) and do not rely on fitted parameters or on load-bearing self-citations; the reference list contains no overlapping author self-citations used to justify the core results. The one notable definitional concern is Algorithm 1 lines 5-6: a constant reward sequence is assigned a perfect ordering score (s_i,j=1), so a preference-ignoring policy that produces constant objective values can receive a high OS in sparse-reward settings such as Problem 1c. This undermines the metric's construct validity as a measure of preference consistency, but it is a correctness issue rather than circular reasoning: the OS is not fitted to the claimed conclusion, and the reported OS values come from empirical rollouts of actual policies compared against an external baseline. No derivation step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The benchmark's validity rests on the environment being a meaningful model of preference-conditioned resource allocation and on the computability of ideal Pareto fronts for small problems. The smoothness hyperparameter is a tuned quantity that directly affects the scalarized reward and therefore the reported performance.

free parameters (1)
  • Smooth Tchebycheff smoothness hyperparameter
    Section C.3 states the scalarization 'includes a smoothness hyperparameter that can be tuned for different problems with varying Pareto Front landscapes'; value is not reported, so it is a per-problem tuned quantity the central experiments depend on.
axioms (3)
  • domain assumption City-scale resource allocation can be modeled as a sequential bipartite graph allocation with add/remove production per demand (CityPlannerEnv dynamics).
    Section 3 defines the environment; the benchmark's validity depends on this being a meaningful approximation of city management, but no external validation is offered.
  • domain assumption Preferences are normalized simplex weights and the scalarized reward (Smooth Tchebycheff) with a moving ideal point approximates the true Pareto-optimal solution.
    Sections C.3 and C.2; the moving ideal point is an estimate used in place of the true ideal point, an approximation that may bias results.
  • domain assumption The ideal Pareto Front can be computed by enumerating all possible allocations for problems marked 'Yes' in Table 1 (FC).
    Section 4.1 states HV ratio uses 'the ideal analytical hypervolume' by enumerating all possible allocations; this is feasible only for small problems and is assumed correct.

pith-pipeline@v1.3.0-alltime-deepseek · 15764 in / 12608 out tokens · 129439 ms · 2026-08-03T07:12:32.271957+00:00 · methodology

0 comments
read the original abstract

Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) approximates diverse Pareto-optimal solutions by conditioning a single policy on user-specified preferences, enabling run-time adaptation to arbitrary trade-offs without retraining. However, existing PCPL benchmarks are largely restricted to toy tasks and fixed environments, limiting their realism and scalability. To address this gap, we introduce GraphAllocBench, a flexible benchmark built on CityPlannerEnv, a novel graph-based resource allocation sandbox inspired by city management. GraphAllocBench provides a rich suite of problems with customizable objective functions, varying preference conditions, complex Pareto Fronts, and high-dimensional scalability. We further propose two supplementary metrics -- Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) -- that capture prediction reliability and preference consistency while complementing the widely used hypervolume metric. Through experiments with several state-of-the-art PCPL algorithms and our own MLP and graph-aware PCPL-PPO baseline, we show that GraphAllocBench exposes distinct failure modes that hypervolume alone does not capture but our supplementary metrics reveal, while motivating graph-based approaches such as Graph Neural Networks (GNNs) for scaling to complex, high-dimensional allocation tasks. By letting users freely vary objectives, preferences, and allocation rules, GraphAllocBench serves as a versatile and extensible testbed for advancing PCPL.

Figures

Figures reproduced from arXiv: 2601.20753 by Benjamin T. Files, Ellen Novoseller, Ryan Marr, Volkan Ustun, Yunzhe Wang, Zhiheng Jiang.

Figure 1
Figure 1. Figure 1: Training and evaluation pipeline using the CityPlannerEnv Gymnasium environment and Proximal Policy Optimization (PPO) agent [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance comparison of PCPL and PD-MORL policies [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Selected 2-Objective Pareto Fronts for PCPL at 1M steps over 2 random seeds: Compared to the Problem 0 baseline, the RL agent [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: PPO architecture with a shared 2-layer MLP feature ex [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance over different number of objectives (5, 10, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Order Sensitivity (Mean + Standard Deviation) across [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Order Sensitivity (Mean + Standard Deviation) across [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Controllability in preference-conditioned multi-objective reinforcement learning

    cs.LG 2026-05 unverdicted novelty 7.0

    Standard MORL metrics do not measure whether preference inputs reliably control agent behavior, so a new controllability metric is introduced to restore the link between user intent and agent output.

Reference graph

Works this paper leans on

42 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [9]

    Deep re- inforcement learning algorithm based on graph weight multi-pointer network for solving multiobjective travel- ing salesman problem.IEEE Access, 12:179091–179103,

    [Fu and Gu, 2024] Xiaoyu Fu and Shenshen Gu. Deep re- inforcement learning algorithm based on graph weight multi-pointer network for solving multiobjective travel- ing salesman problem.IEEE Access, 12:179091–179103,

    2024

  2. [11]

    [Guet al., 2022 ] Qinghua Gu, Qingsong Xu, and Xuexian Li

    International Foundation for Autonomous Agents and Multiagent Systems. [Guet al., 2022 ] Qinghua Gu, Qingsong Xu, and Xuexian Li. An improved nsga-iii algorithm based on distance dominance relation for many-objective optimization.Ex- pert Systems with Applications, 207:117738,

    2022

  3. [13]

    Huband, P

    [Hubandet al., 2006 ] S. Huband, P. Hingston, L. Barone, and L. While. A review of multiobjective test problems and a scalable test problem toolkit.IEEE Transactions on Evolutionary Computation, 10(5):477–506,

    2006

  4. [15]

    Performance comparison of nsga-ii and nsga-iii on various many-objective test prob- lems.2016 IEEE Congress on Evolutionary Computation (CEC), pages 3045–3052,

    [Ishibuchiet al., 2016 ] Hisao Ishibuchi, Ryo Imada, Yu Se- toguchi, and Yusuke Nojima. Performance comparison of nsga-ii and nsga-iii on various many-objective test prob- lems.2016 IEEE Congress on Evolutionary Computation (CEC), pages 3045–3052,

    2016

  5. [17]

    Pareto set learning for neural multi-objective combinato- rial optimization

    [Linet al., 2022 ] Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinato- rial optimization. InInternational Conference on Learning Representations,

    2022

  6. [18]

    Smooth tchebycheff scalarization for multi-objective optimization

    [Linet al., 2024 ] Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org,

    2024

  7. [19]

    Profiling pareto front with multi-objective stein varia- tional gradient descent

    [Liuet al., 2021 ] Xingchao Liu, Xin Tong, and Qiang Liu. Profiling pareto front with multi-objective stein varia- tional gradient descent. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, edi- tors,Advances in Neural Information Processing Systems, volume 34, pages 14721–14733. Curran Associates, Inc.,

    2021

  8. [20]

    Pareto set learning for multi-objective reinforcement learning

    [Liuet al., 2025 ] Erlong Liu, Yu-Chang Wu, Xiaobin Huang, Chengrui Gao, Ren-Jian Wang, Ke Xue, and Chao Qian. Pareto set learning for multi-objective reinforcement learning. InProceedings of the AAAI Conference on Arti- ficial Intelligence, volume 39, pages 18789–18797,

    2025

  9. [21]

    Role play: Learning adaptive role-specific strategies in multi-agent interactions.Know.-Based Syst., 324(C), January

    [Longet al., 2026 ] Weifan Long, Wen Wen, Peng Zhai, and Lihua Zhang. Role play: Learning adaptive role-specific strategies in multi-agent interactions.Know.-Based Syst., 324(C), January

    2026

  10. [23]

    Multi-agent reinforcement learning for creating intelligent agents in social networks- oriented role playing games.Entertainment Computing, 54:100941,

    [Martinez-Gil and Gil-Magraner, 2025] Francisco Martinez- Gil and Eduard Gil-Magraner. Multi-agent reinforcement learning for creating intelligent agents in social networks- oriented role playing games.Entertainment Computing, 54:100941,

    2025

  11. [24]

    Training language models to follow instructions with human feed- back

    [Ouyanget al., 2022 ] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructi...

    2022

  12. [25]

    Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

    [Raffinet al., 2021 ] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 22(268):1–8,

    2021

  13. [26]

    Constructing complex npc behavior via multi-objective neuroevolution

    [Schrum and Miikkulainen, 2008] Jacob Schrum and Risto Miikkulainen. Constructing complex npc behavior via multi-objective neuroevolution. InProceedings of the Fourth Artificial Intelligence and Interactive Digital En- tertainment Conference (AIIDE 2008), pages 108–113, Stanford, California,

    2008

  14. [31]

    Dynamic defender-attacker blotto game

    [Shishikaet al., 2022 ] Daigo Shishika, Yue Guan, Michael Dorothy, and Vijay Kumar. Dynamic defender-attacker blotto game. In2022 American Control Conference (ACC), pages 4422–4428,

    2022

  15. [32]

    Steuer and Eng-Ung Choo

    [Steuer and Choo, 1983] Ralph E. Steuer and Eng-Ung Choo. An interactive weighted tchebycheff procedure for multiple objective programming.Math. Program., 26(3):326–344, October

    1983

  16. [34]

    Empir- ical evaluation methods for multiobjective reinforcement learning algorithms.Mach

    [Vamplewet al., 2011] Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empir- ical evaluation methods for multiobjective reinforcement learning algorithms.Mach. Learn., 84(1–2):51–80, July

    2011

  17. [38]

    Prediction-guided multi-objective reinforcement learning for continuous robot control

    [Xuet al., 2020 ] Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. InProceedings of the 37th International Conference on Machine Learning,

    2020

  18. [39]

    A generalized algorithm for multi- objective reinforcement learning and policy adaptation

    [Yanget al., 2019 ] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e- Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume

    2019

  19. [40]

    Federated reinforcement learning for robot mo- tion planning with zero-shot generalization.Automatica, 166:111709,

    [Yuanet al., 2024 ] Zhenyuan Yuan, Siyuan Xu, and Minghui Zhu. Federated reinforcement learning for robot mo- tion planning with zero-shot generalization.Automatica, 166:111709,

    2024

  20. [41]

    Maximum entropy population-based training for zero-shot human-ai coordination.Proceedings of the AAAI Conference on Artificial Intelligence, 37(5):6145–6153, Jun

    [Zhaoet al., 2023 ] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination.Proceedings of the AAAI Conference on Artificial Intelligence, 37(5):6145–6153, Jun

    2023

  21. [42]

    Scaling pareto-efficient decision making via of- fline multi-objective rl

    [Zhuet al., 2023 ] Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via of- fline multi-objective rl. InInternational Conference on Learning Representations, 2023

    2023

  22. [1983]

    Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032,

    [Towerset al., 2024 ] Mark Towers, Ariel Kwiatkowski, Jor- dan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goul ˜ao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032,

    Pith/arXiv arXiv 2024

  23. [1998]

    [Deb and Jain, 2014] Kalyanmoy Deb and Himanshu Jain. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting ap- proach, part i: Solving problems with box con- straints.IEEE Transactions on Evolutionary Computation, 18(4):577–601,

    2014

  24. [2005]

    Multiple-gradient descent algorithm (mgda) for multiobjective optimization

    [D´esid´eri, 2012] Jean-Antoine D ´esid´eri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313–318,

    2012

  25. [2006]

    A novel pareto-optimal ranking method for comparing multi- objective optimization algorithms,

    [Ibrahimet al., 2024 ] Amin Ibrahim, Azam Asilian Bid- goli, Shahryar Rahnamayan, and Kalyanmoy Deb. A novel pareto-optimal ranking method for comparing multi- objective optimization algorithms,

    2024

  26. [2008]

    Evolving multi-modal behavior in npcs

    [Schrum and Miikkulainen, 2009] Jacob Schrum and Risto Miikkulainen. Evolving multi-modal behavior in npcs. InIEEE Symposium on Computational Intelligence and Games (CIG 2009), pages 325–332, Milan, Italy, Septem- ber

    2009

  27. [2009]

    [Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov

    (Best Student Paper Award). [Schulmanet al., 2017 ] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.ArXiv, abs/1707.06347,

    Pith/arXiv arXiv 2017

  28. [2011]

    Drugan, and Ann Now ´e

    [Van Moffaertet al., 2013] Kristof Van Moffaert, Madalina M. Drugan, and Ann Now ´e. Scalarized multi-objective reinforcement learning: Novel design techniques. In2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 191–199,

    2013

  29. [2012]

    Alegre, Ann Now´e, Ana L

    [Feltenet al., 2023 ] Florian Felten, Lucas N. Alegre, Ann Now´e, Ana L. C. Bazzan, El Ghazali Talbi, Gr ´egoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforce- ment learning. InProceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023),

    2023

  30. [2013]

    Graph attention networks

    [Veliˇckovi´cet al., 2018 ] Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li `o, and Yoshua Bengio. Graph attention networks. InInterna- tional Conference on Learning Representations,

    2018

  31. [2014]

    [Debet al., 2005 ] Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler.Scalable Test Problems for Evolutionary Multiobjective Optimization, pages 105–

    2005

  32. [2016]

    [Labiosaet al., 2025 ] Adam Labiosa, Zhihan Wang, Sid- dhant Agarwal, William Cong, Geethika Hemkumar, Ab- hinav Narayan Harish, Benjamin Hong, Josh Kelle, Chen Li, Yuhao Li, Zisen Shao, Peter Stone, and Josiah P. Hanna. Reinforcement learning within the classical robotics stack: A case study in robot soccer,

    2025

  33. [2017]

    A survey of deep reinforcement learning in video games.CoRR, abs/1912.10944,

    [Shaoet al., 2019 ] Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep reinforcement learning in video games.CoRR, abs/1912.10944,

    Pith/arXiv arXiv 2019

  34. [2018]

    Rlhgnn: Reinforce- ment learning-driven heterogeneous graph neural network for next activity prediction in business processes,

    [Wanget al., 2025 ] Jiaxing Wang, Yifeng Yu, Jiahan Song, Bin Cao, Jing Fan, and Ji Zhang. Rlhgnn: Reinforce- ment learning-driven heterogeneous graph neural network for next activity prediction in business processes,

    2025

  35. [2019]

    Grapheon rl: A graph neural network and reinforcement learning framework for constraint and data- aware workflow mapping and scheduling in heterogeneous hpc systems

    [Sharma and Kunkel, 2025] Aasish Kumar Sharma and Ju- lian Kunkel. Grapheon rl: A graph neural network and reinforcement learning framework for constraint and data- aware workflow mapping and scheduling in heterogeneous hpc systems. In2025 IEEE 49th Annual Computers, Soft- ware, and Applications Conference (COMPSAC), pages 489–494,

    2025

  36. [2020]

    A review of the deep sea treasure problem as a multi-objective reinforcement learning benchmark.CoRR, abs/2110.06742,

    [Cassimonet al., 2021 ] Amber Cassimon, Reinout Eyck- erman, Siegfried Mercelis, Steven Latr ´e, and Peter Hellinckx. A review of the deep sea treasure problem as a multi-objective reinforcement learning benchmark.CoRR, abs/2110.06742,

    Pith/arXiv arXiv 2021

  37. [2021]

    [Das and Dennis, 1998] Indraneel Das and J. E. Dennis. Normal-boundary intersection: A new method for generat- ing the pareto surface in nonlinear multicriteria optimiza- tion problems.SIAM Journal on Optimization, 8(3):631– 657,

    1998

  38. [2022]

    Heterogeneous graph transformer.CoRR, abs/2003.01332,

    [Huet al., 2020 ] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer.CoRR, abs/2003.01332,

    Pith/arXiv arXiv 2020

  39. [2023]

    Blank and K

    [Blank and Deb, 2020] J. Blank and K. Deb. pymoo: Multi- objective optimization in python.IEEE Access, 8:89497– 89509,

    2020

  40. [2024]

    Pick your battles: Interaction graphs as population-level objec- tives for strategic diversity

    [Garneloet al., 2021 ] Marta Garnelo, Wojciech Marian Czarnecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gau- thier Gidel, Hado van Hasselt, and David Balduzzi. Pick your battles: Interaction graphs as population-level objec- tives for strategic diversity. InProceedings of the 20th In- ternational Conference on Autonomous Agents and Mul- tiAgent Systems, A...

    2021

  41. [2025]

    [Basaklaret al., 2023 ] Toygun Basaklar, Suat Gumussoy, and Umit Ogras. PD-MORL: Preference-driven multi- 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Alpha 0.550 0.575 0.600 0.625 0.650 0.675 0.700 0.725Ordering Score (OS) Ordering Score Sensitivity (Problem 5d) Mean Score Error Bound Figure 6: Order Sensitivity (Mean + Standard Deviation) across Dirichlet Di...

    2023

  42. [2026]

    Precise and dexterous robotic manipula- tion via human-in-the-loop reinforcement learning.Sci- ence Robotics, 10(105):eads5033,

    [Luoet al., 2025 ] Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipula- tion via human-in-the-loop reinforcement learning.Sci- ence Robotics, 10(105):eads5033,

    2025