Pith. sign in

REVIEW 5 major objections 6 minor 55 references

Multi-Agent Reinforcement Learning via Agent-Specific Preference

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that per-agent preference feedback, aggregated by summing learned local reward models, provably trains cooperative multi-agent teams to a Nash equilibrium without any global reward function.

desk verdict The Nash-equivalence claim is unproven because per-agent reward scalings aren't coupled, and the core method already appeared in the authors' 2025 CAC paper; still, the problem framing and experiments are useful. read the letter →

arxiv 2608.08604 v1 pith:AKUB4K2Z submitted 2026-08-09 cs.LG

Ni Mu , Yao Luan , Yiqin Yang , Qing-Shan Jia This is my paper · ORCID
classification cs.LG
keywords multi-agentreinforcementlearningpreference-basedagent-specificpreferencesNashequilibriumrewardmonotonicaggregationcooperativeMARL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a method for training cooperative multi-agent systems without a global reward function. Instead of one reward for the whole team, each agent has a dedicated teacher that compares trajectory segments and states which behavior is better for that agent. The paper proves that if each teacher's preference is exactly aligned with the true global return, repeatedly updating agents along teacher-preferred directions reaches a Nash equilibrium. It then learns one reward model per agent from those comparisons, sums them into a global reward, and proves that maximizing this summed reward is equivalent to training a Nash equilibrium policy, with an epsilon-Nash guarantee under finite preference data. Experiments on benchmark coordination tasks and a three-stage production-line task show performance comparable to a baseline trained with hand-designed ground-truth rewards.

What carries the argument

The load-bearing object is the per-agent logistic preference-reward model. Each agent $i$ has a reward model $\hat r_\psi^i$; the probability that segment $\sigma^{(1)}$ is preferred over $\sigma^{(0)}$ is the softmax of the discounted sums of $\hat r_\psi^i$ over the two segments, and the model is trained by cross-entropy on preference pairs. These local rewards are aggregated additively, $\hat r(s,A)=\sum_i \hat r_\psi^i(s,A)$, a monotonic combination whose choice is justified by Lemma 1 and Assumption 7. The identity that carries the theory is Theorem 2: any joint policy maximizing expected return under the summed learned rewards is a Nash equilibrium for the true global reward, because a teacher-preferred local improvement is, by assumption, a global-return improvement. A regularizer penalizes locally preferred segments that reduce the summed reward, guarding Assumption 7.

What would settle it

Run the paper's production-line task with a teacher whose preference rule depends only on the agent's own stage, for example the first-stage agent prefers segments that empty its own queue fastest, then sample 10,000 segment pairs and count how often the teacher prefers a segment whose true global return is lower. If the violation rate is positive and the regularizer does not remove it, the premise of Theorems 1 and 2 is violated; the paper does not report such a test with human or noisy teachers.

Watch

Extended reading notes

Core claim

The paper's central claim is that a cooperative multi-agent team can be trained entirely from agent-specific preference feedback, with no global reward function, and still provably reach a Nash equilibrium. Under Assumption 4, where each teacher's pairwise preference between trajectory segments is true exactly when the expected global return improves, the paper proves that iteratively adopting teacher-preferred unilateral policy changes terminates at a Nash equilibrium. It then shows that fitting one reward model per agent to those preferences and summing them, $\hat r(s,A)=\sum_i \hat r_i(s,A)$, yields a global reward whose optimal policies are Nash equilibria of the true reward (Theorem 2); with finite preference data the guarantee degrades to an $\epsilon$-Nash equilibrium with $\epsilon=2n\epsilon_r(N)/(1-\gamma)$ (Theorem 4). The paper's experiments assert that this preference-only pipeline reaches performance comparable to a baseline trained on hand-crafted ground-truth rewards across three benchmark coordination tasks and a three-stage production-line scheduling task.

Load-bearing premise

The load-bearing premise is that each agent's teacher prefers a new behavior exactly when that change increases the team's true global return; if teachers judge only local contribution, are noisy, or are biased, this 'if and only if' fails and the Nash-equilibrium guarantees no longer follow.

Editorial extensions

If this is right

  • Preference feedback can fully replace global reward engineering in cooperative MARL: the learned aggregate reward comes with a Nash-equilibrium guarantee, so no task-specific reward formula is needed.
  • Any centralized-training MARL algorithm that consumes a scalar reward signal can be switched to preference-driven learning by replacing the environment reward with the summed local reward models.
  • Heterogeneous teams are naturally supported, because each teacher evaluates one agent's contribution rather than comparing agents to each other.
  • Finite preference data yields a graceful guarantee: the trained policy is an $\epsilon$-Nash equilibrium with $\epsilon$ growing linearly in the number of agents and shrinking as more comparisons are collected.
  • Lightweight automated teacher rules suffice in practice, so human annotators are not required; this makes the method feasible in industrial settings such as production-line scheduling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: a natural follow-up not pursued by the paper is to replace Assumption 4's strict 'if and only if' with a margin or probabilistic alignment, which would yield preference-based equilibrium bounds for noisy or systematically biased teachers.
  • Extension: additive aggregation is sufficient but not necessary; learning state- or role-dependent aggregation weights could relax Assumption 7's condition that one agent's gain always outweighs any collective loss.
  • Extension: because the convergence arguments assume finite policy spaces and perfect preference consistency, extending the method to continuous policies would require an additional exploration or consistency mechanism.
  • Extension: the theory suggests a practical teacher-auditing test before deployment: check whether the teacher's local preferences correlate with global-return ordering on held-out segment pairs; the regularizer is the stated safeguard for violations, but its strength is a single scalar hyperparameter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MAGPIE, a multi-agent reinforcement learning framework that replaces a hand-engineered global reward with agent-specific preference signals. Each agent has a dedicated teacher who compares trajectory segments; the paper learns per-agent reward models from these preferences via a Bradley-Terry objective, aggregates them by summation, and then trains a QMIX-style policy with the aggregated reward. The theoretical claims are that iterative local-preference updates converge to a Nash equilibrium (Theorem 1), that maximizing the summed learned rewards yields a Nash equilibrium (Theorem 2), that each learned reward model is identifiable up to an affine transformation (Theorem 3), and that with finite preference data the resulting policy is an ε-Nash equilibrium (Theorem 4). An appendix gives a stochastic-approximation convergence proof (Theorem 5). Experiments on three MPE tasks and a sequential production-line task compare MAGPIE against QMIX with the true task reward, with additional ablations and scaling results.

Significance. If the theoretical claims were sound, the paper would offer a practical alternative to reward engineering in heterogeneous cooperative MARL, and the release of source code plus the breadth of experiments would make it a useful contribution. The paper also correctly identifies limitations of existing multi-agent preference-based methods, such as global trajectory evaluation and cross-agent comparability. However, the central theoretical guarantees are not established: Assumption 4 makes local preferences equivalent to global-return comparisons, turning the convergence theorems into properties of global-reward optimization, and the per-agent affine ambiguity in Theorem 3 breaks the reward-aggregation arguments in Theorems 2 and 4. The empirical evaluation uses rule-based teachers constructed from the reward components, so the experiments do not test the framework under the noisy or genuinely local preference models that motivate the work.

major comments (5)
  1. [Section IV-A, Assumption 4 (Eq. 3)] Assumption 4 states that σ' ≻_i σ holds if and only if the expected true global return under (π'_i, π_−i) exceeds that under (π_i, π_−i). This makes each teacher's local preference a perfect indicator of global-return improvement. Theorem 1 then proves that monotone improvement of the global return terminates at a Nash equilibrium, which is a standard property of global-reward optimization and does not demonstrate that decentralized preferences can replace global reward information. If Assumption 4 is relaxed so that teachers evaluate only an agent's local contribution or are noisy, the proofs of Theorems 1, 2, and 4 lose their foundation, yet the paper never tests such relaxation; the experimental teachers are deterministic rules built from reward components and thus satisfy Eq. 3 by construction.
  2. [Section IV-A, Theorem 2 (Eqs. 14-16) and Theorem 3 (Eq. 20)] The proof of Theorem 2 needs the exact equality 'the discounted sum of \hat r_j over a segment equals R_j(σ)', but Theorem 3 guarantees only \hat r_i = α_i r_i + β_i with per-agent constants. With α_i ≠ α_j, the sum Σ_j \hat R_j(σ) can rank two joint policies in the opposite order of Σ_j R_j(σ) even if each individual \hat r_i ranks segments like r_i. The step from the inequality for R_j to the inequality for \hat r_j is therefore invalid, and the conclusion that a maximizer of \hat r is a Nash equilibrium for the true reward r is unproven. Theorem 4 (Eq. 31) assumes a single shared α and β across agents, but Theorem 3 provides no coupling among the per-agent affine constants, so the ε-Nash guarantee relative to R_true(s,A) = α Σ_i r_i(s,A) + nβ does not transfer to the true global reward r.
  3. [Section IV-B, Theorem 3 proof (Eqs. 25-30)] The finite-sample proof has several gaps. Strict properness of the cross-entropy loss gives equality of preference distributions at the population minimizer, but the passage from L_µ(\hat r_{ψ*}) → L_µ(r*_i) to uniform convergence of \hat r_{ψ*} to r*_i is asserted rather than proved. The later L2 bound in Eq. 30 is said to follow from 'strong convexity of L_µ in a neighborhood of r*_i', but the strong convexity is not established and the logistic loss is not strongly convex on an unbounded reward class without additional constraints. The Rademacher complexity R_N(F) is invoked without specifying the function class F used by the neural reward model in Eq. 17, so the sample-complexity bound in Eq. 21 cannot be evaluated as stated.
  4. [Section IV-C, Eq. 37 and Assumption 7 (Eq. 11)] Assumption 7 is a condition on the true local rewards r_i, but the only mechanism proposed to enforce it is the soft regularization loss L_i^reg in Eq. 37. No theorem shows that minimizing L^reg guarantees Eq. 11, and the regularization uses the learned models \hat r_i, which are only affine-identified with respect to the true r_i. Consequently, the aggregated objective is not shown to satisfy the premise of Theorem 2, and the claim that the regularization 'promotes' Assumption 7 is not backed by a quantitative statement.
  5. [Appendix A, Theorem 5 (Eqs. 40-49)] The convergence result rests on assumptions that the optimal global Q-function lies in the decomposable subspace Q_d and that the projection Π is non-expansive; these are stated without verification and are not properties of the implemented QMIX algorithm, which uses a nonlinear monotonic mixing network. The proof also asserts that the projected noise ΠM_{k+1} remains a martingale difference sequence without checking the required measurability and integrability conditions. These gaps weaken the appendix's claim that the practical MAGPIE procedure converges to an ε-Nash policy.
minor comments (6)
  1. [Contributions, page 2] The sentence 'proving that the n-to-nexpert-agent preference mechanism' contains a typo; it should read 'n-to-n expert-agent' or similar.
  2. [Theorem 3, Eq. 21] The notation R_N(F) is used without defining the function class F or the norm used in the L2(μ) error; please specify F (e.g., the neural network family with bounded weights) and the measure μ explicitly.
  3. [Assumption 5 vs. Theorem 2] Assumption 5 is stated for a true local reward r_i, but Theorem 2 says the learned \hat r_i satisfies Assumption 5; this conflation of true and learned rewards should be clarified, as it is central to the affine-ambiguity issue.
  4. [Figure 2] The dependency graph references 'C1', but the text labels the result as Corollary 1; the labels should be made consistent.
  5. [Section V-A and Table I] The hyperparameter table lists segment length H as 3/5/10/20, but the MPE experiments use H = 3, 5, 10 and the production task uses H = 5, 10, 20; please make the per-task choices explicit.
  6. [Algorithm 1, line 9] Algorithm 1 does not specify what happens when the teacher is indifferent (p = 0.5), which Assumption 1 allows; a tie-breaking or no-update rule should be stated.

Circularity Check

3 steps flagged · score 7.0 of 10

Central Nash-equivalence claims are built into Assumption 4, which defines local preference as global-return comparison; Theorem 4 also replaces per-agent affine ambiguity with a shared α,β and defines R_true as the target.

  1. self definitional [Section IV-A, Assumption 4, Eq. (3)]
    "Assumption 4. ... The local preference aligns with the global return such that σ′ ≻i σ if and only if the expected global return under (π_i′, π−i) is strictly greater than that under π. Formally, E_{τ∼(π_i′,π−i)}[Σ_t γ^t r(s_t,A_t)] > E_{τ∼(π_i,π−i)}[Σ_t γ^t r(s_t,A_t)] ⇔ σ′ ≻i σ. (3)"

    The central claim that preference-driven updates converge to Nash is not derived: Eq. (3) defines the preference relation to be exactly the global-return comparison. Theorem 1 concludes that when no teacher prefers an alternative, no unilateral global-return improvement exists, which is Nash by definition. The same equivalence is reused in Theorem 2's proof to convert local reward increases into global-return increases, so the aggregation result inherits the definitional alignment rather than establishing it.

  2. self definitional [Section IV-A, Theorem 2 proof (between Eqs. (14) and (15))]
    "Since the local reward models ˆrj perfectly reflect the teachers’ preferences, the discounted sum of ˆr_j over a segment equals R_j(σ) for that segment. Therefore, the inequality above implies that for the segments σ′ and σ∗, we have: Σ_t γ^t ˆr(s′_t,A′_t) > Σ_t γ^t ˆr(s∗_t,A∗_t)."

    Theorem 2 assumes the learned rewards are perfectly aligned with teachers' preferences, i.e., satisfy Assumption 5. Assumption 5 defines teacher preference as exactly the ordering of latent local reward R_i, while Assumption 4 defines that same preference as exactly the ordering of true global return. With Assumption 7 forcing the sum of local returns to increase, the aggregate learned reward ranking is the global-return ranking by construction. The prediction that maximizing Σ_i r̂_i yields a Nash equilibrium therefore reduces to the assumed equivalences; additionally, Theorem 3 only gives per-agent affine identifiability, so the step 'equals R_j(σ)' is not available from the fitting procedure.

1 more flagged steps
  1. self definitional [Section IV-B, Theorem 3 Eq. (20) and Theorem 4 Eqs. (31)-(32)]
    "Theorem 3: lim_{N→∞} Pr( inf_{α>0,β∈R} sup_{(s,A)∈Ω_i} |r̂^i_ψ∗(s,A) − (α r_i(s,A) + β)| > ϵ) = 0. (20) ... Theorem 4: for all (s,A)∈S×A: |r̂^i_ψ(s,A) − (α r_i(s,A) + β)| ≤ ϵ_r(N), where α>0 and β are scalar constants. ... under the global reward objective reflecting the true preferences, denoted as R_true(s,A)=αΣ^n_{i=1} r_i(s,A)+nβ."

    Eq. (20) places the infimum over α,β separately for each agent i, so the affine constants are per-agent. Theorem 4 silently uses one shared α,β and defines the 'true preference objective' R_true with those shared constants, so the learned aggregate is close to R_true by construction. The theorem then proves an ϵ-Nash statement for R_true without showing R_true equals the true global reward r of Assumptions 4 and 6; the target objective has been redefined to make the conclusion hold.

full rationale

The paper's central proof chain is assumption-embedded rather than derived. Eq. (3) defines agent-specific preference as global-return comparison; Eq. (9) defines the latent local reward as the preference ordering; and Assumption 7 forces the aggregate local sum to increase whenever one agent's preference improves. Given those inputs, Theorems 1 and 2 are restatements: no preferred unilateral deviation is, by Eq. (3), no global-return unilateral deviation, and a maximizer of the sum of perfectly aligned local rewards inherits that same ranking. This is a self-definitional circularity in the core theoretical claim. Theorem 3 is independent finite-sample content, but it provides only per-agent affine identifiability, which contradicts the exact equality used in Theorem 2's proof; Theorem 4 then selects a single shared α,β and names R_true=αΣr_i+nβ as the target, making the epsilon-Nash result true for a constructed objective rather than for the r of Assumption 4. The experimental comparisons against reward-engineered baselines are genuine external evidence, and no load-bearing self-citation was found; the high score reflects that the convergence and equivalence results reduce to their own assumptions, not that the experiments are invalid.

Assumptions & free parameters 5 free parameters · 9 assumptions · 0 invented entities

The central claim rests on seven explicit assumptions, two of which (Assumptions 4 and 7) carry the main theoretical load and are close to the conclusions being proved. The per-agent affine ambiguity from Theorem 3 is an unacknowledged free parameter in Theorem 4. The rule-based teachers in experiments are constructed from the same quantities as the true reward, making the empirical comparison expected. No new physical or mathematical entities are postulated.

free parameters (5)
  • per-agent affine scale alpha_i and offset beta_i = unidentified
    Theorem 3 shows each reward model is identifiable only up to its own affine transformation; Theorem 4 silently assumes a single shared alpha and beta. The scale indeterminacy changes the weights in the summed reward.
  • segment length H = 3, 5, 10, 20 depending on task
    Chosen by hand. Corollary 1 gives a threshold H >= log_gamma(delta(1-gamma)/(2 r_max)), but delta is unspecified, so the chosen H is not tied to the theory.
  • minimum preference margin delta = unspecified
    Corollary 1 assumes there exists delta > 0 separating the returns of distinguishable segments; no value or verification is provided.
  • regularization coefficient lambda = 2e-2
    Hyperparameter balancing the preference loss and the Assumption-7 regularization; set by hand, not derived.
  • preference collection frequency K and segment pairs M = K=500, M=300
    Hyperparameters controlling how often preferences are collected and how many segment pairs are queried per agent per session.
assumptions (9)
  • domain assumption Assumption 1: each agent's preferences over equal-length segments satisfy symmetry, consistency, and transitivity.
    Underlies all preference-based learning; assumes a perfectly rational, noise-free teacher with stable criteria.
  • domain assumption Assumption 2: a teacher exists for every agent and can provide the required preference feedback.
    Availability of per-agent feedback is taken as given and is not derived from the environment or task.
  • domain assumption Assumption 3: a true global reward r(s,A) exists and its absolute value is bounded by r_max.
    Guarantees bounded returns and is standard in RL theory; invoked in Theorem 1 and Theorem 4.
  • ad hoc to paper Assumption 4: local preference sigma' >_i sigma holds if and only if the expected true global return improves (Eq. 3).
    This is the key alignment premise. It effectively assumes local teachers replicate global return comparisons, which is the main load-bearing assumption for all convergence theorems.
  • domain assumption Assumption 5: teacher preferences are consistent with a latent local reward function r_i(s,A) (Eq. 9).
    Standard PbRL assumption that an implicit reward underlies pairwise preferences.
  • domain assumption Assumption 6: the true global reward is a continuously differentiable function F of the local rewards (Eq. 10).
    Structural assumption linking local and global objectives; needed for the monotonic aggregation argument.
  • ad hoc to paper Assumption 7: for a local improvement, the change in other agents' local returns is bounded below so the summed local return increases (Eq. 11).
    This is nearly the conclusion of Theorem 2: it assumes local improvements cannot reduce the aggregate reward, which is exactly what the theorem aims to prove.
  • ad hoc to paper Appendix assumptions: the optimal Q-function lies in the decomposable subspace Q_d and the projection operator Pi is non-expansive.
    Needed for Theorem 5's stochastic-approximation convergence proof; not guaranteed by QMIX and significantly restricts the setting.
  • standard math Standard results: Nash's existence theorem, the gamma-contraction property of the Bellman operator, and the Borkar-Meyn ODE method.
    Unproved background results invoked in Theorem 1 and Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Agent Reinforcement Learning via Agent-Specific Preference." pith.science (2026). https://pith.science/paper/AKUB4K2Z

@misc{pith2026260808604,
  author       = {Pith},
  title        = {Pith review of: Multi-Agent Reinforcement Learning via Agent-Specific Preference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKUB4K2Z}},
  note         = {Machine review of arXiv:2608.08604}
}
read the original abstract

Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized preferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical.

Figures

Figures reproduced from arXiv: 2608.08604 by the authors.

Figure 1
Figure 1. An overview of MAGPIE. The agent interacts with the environment and learns a multi-agent policy, without a predefined reward function. Instead, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Dependency graph of assumptions (A), lemmas (L), theorems (Th), [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The training curves for the episode return of the two methods. Blue: the oracle method (QMIX). Red: our method (MAGPIE). Results are averaged [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a, b) Performance comparison of MAGPIE and preference-based baselines on the Simple-Spread ( [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: (a, b) Performance comparison of MAGPIE and reward-engineered MARL baselines on the Simple-Speaker-Listener task ( [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 36 canonical work pages

  1. [13]

    MAGPIE: Utilizing Agent-Specific Preferences for Multi-Agent Reinforcement Learning,

    N. Mu, Y . Luan, and Q.-S. Jia, “MAGPIE: Utilizing Agent-Specific Preferences for Multi-Agent Reinforcement Learning,” in2025 China Automation Congress (CAC). IEEE, 2025

    work page 2025

  2. [1]

    Distributed multiagent coordinated learning for autonomous driving in highways based on dynamic coordination graphs,

    C. Yu, X. Wang, X. Xu, M. Zhang, H. Ge, J. Ren, L. Sun, B. Chen, and G. Tan, “Distributed multiagent coordinated learning for autonomous driving in highways based on dynamic coordination graphs,”IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 2, pp. 735–748, 2020

    work page 2020

  3. [2]

    Beyond static populations: Efficient delay-constrained scheduling for dynamic users via deep reinforcement learning,

    X. Wang, Z. Li, and L. Huang, “Beyond static populations: Efficient delay-constrained scheduling for dynamic users via deep reinforcement learning,” inProceedings of the Twenty-sixth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, 2025, pp. 151–160

    work page 2025

  4. [3]

    Dota 2 with large scale deep reinforcement learning,

    C. Berner, G. Brockman, B. Chan, V . Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesseet al., “Dota 2 with large scale deep reinforcement learning,”arXiv preprint arXiv:1912.06680, 2019

    arXiv 1912

  5. [4]

    The surprising effectiveness of ppo in cooperative multi-agent games,

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” Advances in neural information processing systems, vol. 35, pp. 24 611– 24 624, 2022

    2022

  6. [5]

    Value-decomposition networks for cooperative multi-agent learning based on team reward,

    P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V . Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuylset al., “Value-decomposition networks for cooperative multi-agent learning based on team reward,” inProceedings of the 17th International Confer- ence on Autonomous Agents and MultiAgent Systems, 2018, pp. 2085– 2087

    work page 2018

  7. [6]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” 2018

    work page 2018

  8. [7]

    Beyond shallow behavior: Task-efficient value-based multi-task offline marl via skill discovery,

    X. Wang, Z. Li, H. Zhong, and L. Huang, “Beyond shallow behavior: Task-efficient value-based multi-task offline marl via skill discovery,” arXiv preprint arXiv:2502.08985, 2025

    work page arXiv 2025

Show all 55 references
  1. [8]

    Reward learning from human preferences and demonstrations in atari,

    B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei, “Reward learning from human preferences and demonstrations in atari,” Advances in neural information processing systems, vol. 31, 2018

    2018

  2. [9]

    Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsuper- vised pre-training,

    K. Lee, L. M. Smith, and P. Abbeel, “Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsuper- vised pre-training,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 6152–6163

    2021

  3. [10]

    S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning,

    N. Mu, Y . Luan, Y . Yang, B. Xu, and Q.-s. Jia, “S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning,” inProceedings of the Thirty-Fourth Interna- tional Joint Conference on Artificial Intelligence, 2025

    2025

  4. [11]

    CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambigu- ous Queries,

    N. Mu, H. Hu, X. Hu, Y . Yang, B. XU, and Q.-S. Jia, “CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambigu- ous Queries,” inForty-second International Conference on Machine Learning, 2025

    2025

  5. [12]

    STAIR: Addressing stage misalignment through temporal-aligned preference reinforcement learning,

    Y . Luan, N. Mu, Y . Yang, B. XU, and Q.-S. Jia, “STAIR: Addressing stage misalignment through temporal-aligned preference reinforcement learning,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  6. [14]

    Senior: Efficient query selection and preference-guided exploration in preference-based rein- forcement learning,

    H. Ni, T. Lu, H. Hu, Y . Cai, and S. Wang, “Senior: Efficient query selection and preference-guided exploration in preference-based rein- forcement learning,” 2025

    2025

  7. [15]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in Neural Information Processing Systems, 2022

    2022

  8. [16]

    Preference-based Multi-Objective Reinforcement Learning,

    N. Mu, Y . Luan, and Q.-S. Jia, “Preference-based Multi-Objective Reinforcement Learning,”IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 18 737–18 749, 2025

    2025

  9. [17]

    COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space,

    Y . Luan, N. Mu, H. Ge, Y . Yang, B. Xu, and Q.-S. Jia, “COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space,” in Forty-third International Conference on Machine Learning, 2026

    2026

  10. [18]

    Offline multi-agent preference-based reinforcement learning with agent-aware direct preference optimization,

    Q. Kou, M. Li, Z. Liu, L. Qian, Z. Chen, L. Wan, X. Chen, and X. Lan, “Offline multi-agent preference-based reinforcement learning with agent-aware direct preference optimization,” inProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, ...

    2025

  11. [19]

    O-MAPL: Offline multi- agent preference learning,

    T. V . Bui, T. A. Mai, and T. H. Nguyen, “O-MAPL: Offline multi- agent preference learning,” inForty-second International Conference on Machine Learning, 2025

    2025

  12. [20]

    Decoding global preferences: Tem- poral and cooperative dependency modeling in multi-agent preference- based reinforcement learning,

    T. Zhu, Y . Qiu, H. Zhou, and J. Li, “Decoding global preferences: Tem- poral and cooperative dependency modeling in multi-agent preference- based reinforcement learning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 15, 2024, pp. 17 202–17 210

    2024

  13. [21]

    DPM: Dual preferences-based multi-agent reinforcement learning,

    S. Kang, Y . Lee, M. Kim, J. Oh, S. Chong, and S.-Y . Yun, “DPM: Dual preferences-based multi-agent reinforcement learning,” 2025. [Online]. Available: https://openreview.net/forum?id=VzuPnoSKQ1

    2025

  14. [22]

    Multi-agent reinforcement learning from human feedback: Data coverage and algorithmic techniques,

    N. Zhang, X. Wang, Q. Cui, R. Zhou, S. M. Kakade, and S. S. Du, “Multi-agent reinforcement learning from human feedback: Data coverage and algorithmic techniques,” 2025. [Online]. Available: https://openreview.net/forum?id=4vPC6Aj6N7

    2025

  15. [23]

    Mastering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Boltonet al., “Mastering the game of go without human knowledge,”Nature, vol. 550, no. 7676, pp. 354–359, 2017

    2017

  16. [24]

    Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,

    R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, and B. Dresp-Langley, “Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,”Robotics, vol. 10, no. 1, p. 22, 2021

    2021

  17. [25]

    Integrating Mechanism and Data: Reinforcement Learning Based on Multi-fidelity Model for Data Center Cooling Control,

    N. Mu, X. Hu, and Q.-S. Jia, “Integrating Mechanism and Data: Reinforcement Learning Based on Multi-fidelity Model for Data Center Cooling Control,” in2023 China Automation Congress (CAC). IEEE, 2023, pp. 5283–5288

    2023

  18. [26]

    Large-scale Data Center Cooling Control via Sample-efficient Reinforcement Learning,

    N. Mu, X. Hu, Q.-S. Jia, X. Zhu, and X. He, “Large-scale Data Center Cooling Control via Sample-efficient Reinforcement Learning,” in2024 IEEE 20th International Conference on Automation Science and Engineering (CASE). IEEE, 2024, pp. 2780–2785

    2024

  19. [27]

    E-mapp: Efficient Multi- Agent Reinforcement Learning with Parallel Program Guidance,

    C. Chang, N. Mu, J. Wu, L. Pan, and H. Xu, “E-mapp: Efficient Multi- Agent Reinforcement Learning with Parallel Program Guidance,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 12 154– 12 168, 2022

    2022

  20. [28]

    From solo to symphony: Orchestrating multi-agent collaboration with single-agent demos,

    X. Wang, Z. Li, Y . Lin, H. Zhong, and L. Huang, “From solo to symphony: Orchestrating multi-agent collaboration with single-agent demos,”arXiv preprint arXiv:2511.02762, 2025

    2025

  21. [29]

    GlobeDiff: State Diffusion Process for Partial Ob- servability in Multi-Agent Systems,

    Y . Yang, X. Yang, Y . Jiang, N. Mu, H. Hu, R. Xie, Z. Zhang, S. Li, Y .-H. Ni, Q. Zhaoet al., “GlobeDiff: State Diffusion Process for Partial Ob- servability in Multi-Agent Systems,”arXiv preprint arXiv:2602.15776, 2026

    2026

  22. [30]

    Multi-agent reinforcement learning for resources allocation optimization: a survey,

    M. A. Hady, S. Hu, M. Pratama, Z. Cao, and R. Kowalczyk, “Multi-agent reinforcement learning for resources allocation optimization: a survey,” Artificial Intelligence Review, vol. 58, no. 11, p. 354, 2025

    2025

  23. [31]

    A review of cooperative multiagent deep reinforcement learning,

    A. Oroojlooy and D. Hajinezhad, “A review of cooperative multiagent deep reinforcement learning,”Applied Intelligence, vol. 53, no. 11, pp. 13 677–13 722, 2023

    2023

  24. [32]

    Reinforcement learning with sparse rewards using guidance from offline demonstration,

    D. Rengarajan, G. Vaidya, A. Sarvesh, D. Kalathil, and S. Shakkottai, “Reinforcement learning with sparse rewards using guidance from offline demonstration,” inInternational Conference on Learning Rep- resentations, 2022

    2022

  25. [33]

    Reward function design in reinforcement learning,

    J. Eschmann, “Reward function design in reinforcement learning,” Reinforcement learning algorithms: Analysis and Applications, pp. 25– 33, 2021

    2021

  26. [34]

    Liir: Learning indi- vidual intrinsic reward in multi-agent reinforcement learning,

    Y . Du, L. Han, M. Fang, J. Liu, T. Dai, and D. Tao, “Liir: Learning indi- vidual intrinsic reward in multi-agent reinforcement learning,”Advances in neural information processing systems, vol. 32, 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 17

    2019

  27. [35]

    Quality assessment of 3d human animation: Subjective and objective evaluation,

    R. Rekik, S. Wuhrer, L. Hoyet, K. Zibrek, and A.-H. Olivier, “Quality assessment of 3d human animation: Subjective and objective evaluation,” IEEE Transactions on Visualization and Computer Graphics, p. 1–12, 2025

    2025

  28. [36]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,”Advances in neural information processing systems, vol. 30, 2017

    2017

  29. [37]

    Preference-Based Multi-Objective Reinforcement Learning with Explicit Reward Modeling,

    N. Mu, Y . Luan, and Q.-S. Jia, “Preference-Based Multi-Objective Reinforcement Learning with Explicit Reward Modeling,” in2024 China Automation Congress (CAC). IEEE, 2024, pp. 4874–4879

    2024

  30. [38]

    Human implicit preference- based policy fine-tuning for multi-agent reinforcement learning in usv swarm,

    H. Kim, K. Lee, J. Park, J. Li, and J. Park, “Human implicit preference- based policy fine-tuning for multi-agent reinforcement learning in usv swarm,”arXiv preprint arXiv:2503.03796, 2025

    2025 arXiv

  31. [39]

    A bayesian approach for policy learning from trajectory preference queries,

    A. Wilson, A. Fern, and P. Tadepalli, “A bayesian approach for policy learning from trajectory preference queries,”Advances in neural infor- mation processing systems, vol. 25, 2012

    2012

  32. [40]

    Convergence of q-learning: A simple proof,

    F. S. Melo, “Convergence of q-learning: A simple proof,”Institute Of Systems and Robotics, Tech. Rep, pp. 1–4, 2001

    2001

  33. [41]

    Decentralized multi-agent reinforcement learning: An off-policy method,

    K. Li and Q.-S. Jia, “Decentralized multi-agent reinforcement learning: An off-policy method,” 2021

    2021

  34. [42]

    An ocba-based method for efficient sample collection in reinforcement learning,

    K. Li, X. Jin, Q.-S. Jia, D. Ren, and H. Xia, “An ocba-based method for efficient sample collection in reinforcement learning,”IEEE Trans- actions on Automation Science and Engineering, vol. 21, no. 3, pp. 3615–3626, 2024

    2024

  35. [43]

    Equilibrium points in n-person games,

    J. F. Nash, “Equilibrium points in n-person games,”Proceedings of the National Academy of Sciences, vol. 36, no. 1, pp. 48–49, 1950

    1950

  36. [44]

    Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,

    J. Park, Y . Seo, J. Shin, H. Lee, P. Abbeel, and K. Lee, “Surf: Semi- supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning,” inInternational Conference on Learning Representations, 2022

    2022

  37. [45]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,”Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  38. [46]

    Rademacher and gaussian complex- ities: Risk bounds and structural results,

    P. L. Bartlett and S. Mendelson, “Rademacher and gaussian complex- ities: Risk bounds and structural results,”Journal of machine learning research, vol. 3, no. Nov, pp. 463–482, 2002

    2002

  39. [47]

    Emergence of grounded compositional language in multi-agent populations,

    I. Mordatch and P. Abbeel, “Emergence of grounded compositional language in multi-agent populations,”arXiv preprint arXiv:1703.04908, 2017

    2017 arXiv

  40. [48]

    Multi- agent actor-critic for mixed cooperative-competitive environments,

    R. Lowe, Y . Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi- agent actor-critic for mixed cooperative-competitive environments,”Neu- ral Information Processing Systems (NIPS), 2017

    2017

  41. [49]

    Balancing and scheduling of surface mount technology lines,

    T. Sawik, “Balancing and scheduling of surface mount technology lines,” International Journal of Production Research, vol. 40, no. 9, pp. 1973– 1991, 2002

    1973

  42. [50]

    Flow shop scheduling prob- lems with assembly operations: a review and new trends,

    G. Komaki, S. Sheikh, and B. Malakooti, “Flow shop scheduling prob- lems with assembly operations: a review and new trends,”International Journal of Production Research, vol. 57, no. 10, pp. 2926–2955, 2019

    2019

  43. [51]

    Modeling semiconductor testing job schedul- ing and dynamic testing machine configuration,

    J.-Z. Wu and C.-F. Chien, “Modeling semiconductor testing job schedul- ing and dynamic testing machine configuration,”Expert Systems with Applications, vol. 35, no. 1-2, pp. 485–496, 2008

    2008

  44. [52]

    Reducing overes- timation bias in multi-agent domains using double centralized critics,

    J. Ackermann, V . Gabler, T. Osa, and M. Sugiyama, “Reducing overes- timation bias in multi-agent domains using double centralized critics,” arXiv preprint arXiv:1910.01465, 2019

    1910 arXiv

  45. [53]

    The o.d.e. method for convergence of stochastic approximation and reinforcement learning,

    V . S. Borkar and S. P. Meyn, “The o.d.e. method for convergence of stochastic approximation and reinforcement learning,”SIAM Journal on Control and Optimization, vol. 38, no. 2, pp. 447–469, 2000

    2000

  46. [54]

    R. S. Sutton, A. G. Bartoet al.,Reinforcement learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1

    1998

  47. [55]

    A stochastic approximation method,

    H. Robbins and S. Monro, “A stochastic approximation method,”The annals of mathematical statistics, pp. 400–407, 1951

    1951

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.