Pith. sign in

REVIEW 4 major objections 7 minor 69 references

Single-Agent Planning in a Multi-Agent System: A Unified Framework for Type-Based Planners

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that all major type-based planners, from exact infinite-horizon POMDP solving to scalable MCTS, are special cases of one layered tree-search rule specified by three parameters, and that this rule predicts…

desk verdict A solid unification of type-based planners with a load-bearing stationary-Markov assumption; the empirical results are robustness evidence, and the safe-agent finding is the real takeaway. read the letter →

arxiv 2502.08950 v1 pith:4GY72CK7 submitted 2025-02-13 cs.MA cs.GT

Fengming Zhu , Fangzhen Lin This is my paper · ORCID
classification cs.MAcs.GT
keywords multi-agentplanningopponentmodellingtype-basedreasoningPOMDPtreesearchbeliefupdatesafeagentsroute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a single agent planning in a multi-agent world where the identities and strategies of the other agents are unknown at the start. It claims that every existing approach to this problem—exact infinite-horizon POMDP solving, belief-fixed and belief-updated MDP replanning, QMDP, and scalable sampling- or bandit-based tree search—is a special case of one layered lookahead rule, parameterized by how many levels update the belief, how many levels freeze it, and how leaf nodes are evaluated. If true, planner design becomes a matter of choosing these three components, and planner quality is ordered by how deeply and deliberately the search runs: F1≽F4≽F2~F5≽F6~F7. The paper implements thirteen planners in a multi-agent route-planning domain with up to fifty agents, and reports that a myopic depth-one special case, the 'safe agent', performs well in most games, especially against malicious opponents and at large scales.

What carries the argument

The load-bearing object is the layered lookahead triple (n, m, Eval_i), interpreted as an ExpectiMax tree with MAX nodes at the modelling agent's states and EXP nodes over opponent joint actions. The first n levels update the belief at every transition; the next m levels keep the belief fixed; the leaf evaluation Eval_i supplies future-return estimates, which can come from hand-coded shortest paths, MDP or POMDP solvers, contextual-RL policies, or Nash-equilibrium strategies computed by constraint solvers. The same triple interpolates from exact Bellman backups to sampling-based backup to bandit-based node selection, which is what lets one framework cover both exact and approximate planners. The other central mechanism is the belief revision operator with temperature parameter β: β=1 reproduces exact POMDP Bayesian filtering, while decreasing β toward 0 produces the hard-max beliefs used by enhanced safe-agents.

What would settle it

Play a framework planner against an opponent that deliberately switches policy mid-episode, for example a chaser who pretends to head for one goal and then reverses, measure the calibration of the Bayesian posterior against the opponent's actual action frequencies, and compare a belief-updating planner predicted to be better, such as F4, with the myopic safe-agent. If the posterior becomes systematically wrong and the safe-agent outperforms the belief-updating planner, that demonstrates that the stationary-type premise is the load-bearing assumption.

Watch

Extended reading notes

Core claim

The central claim is that the full spectrum of type-based planners is embedded in a three-layer tree-search framework. At each step, with state S_t and belief b_t, the planner does n levels of belief-updated lookahead, applying Bayesian revision on every transition, then m levels of belief-fixed lookahead with the belief frozen, then evaluates leaf nodes with any heuristic Eval_i. Setting n=∞ gives exact infinite-horizon POMDP solving; n=0, m=∞ gives the belief-induced MDP; n=0, m=0 with the QMDP heuristic gives QMDP; 0<n<∞, m=∞ gives a finite-horizon POMDP with MDP terminal values; finite n and m yields scalable tree search; sampling backup yields F6; bandit-based selection yields F7. The two backup operators are shown to be γ-contractions, so infinite depth converges to the exact POMDP and belief-MDP fixed points. The paper argues, and tests, that deeper and more deliberate search performs better, yielding the ranking F1≽F4≽F2~F5≽F6~F7, and that the implemented planners, including safe-agents as a depth-one special case, scale to 50-agent long-horizon route-planning instances.

Load-bearing premise

The whole reduction assumes every opponent's strategy is drawn once from a finite set of stationary, non-learning Markov policies, so observed actions can only be explained by these fixed types; if opponents adapt or switch strategies mid-game, the Bayesian belief update is misspecified and the framework's optimality claims do not apply.

Editorial extensions

If this is right

  • If the framework is right, every type-based planner can be described by its effective (n, m, Eval_i), so comparing planners reduces to locating them in one design space.
  • Deeper and more deliberate search is predicted to perform better, and the paper's experiments confirm the trend, giving a practical selection rule: choose the deepest search that fits the problem scale.
  • Nash-equilibrium strategies can be plugged into the framework as leaf heuristics or policy priors and repaired by online tree search, making equilibrium computation useful even when opponents do not play equilibrium.
  • The depth-one safe-agent special case gives surprisingly good, conservative performance, especially against malicious opponents and at large scales, because ruling out unsafe actions protects against belief mismatch.
  • The 50-agent route-planning experiments are, to the authors' knowledge, the first demonstration of an opponent-modelling planner at that scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The framework suggests a classification scheme for any planner: even apparently non-tree-search methods could be assigned an effective (n, m, Eval_i), which would let practitioners predict relative quality before implementation.
  • The safe-agent result hints that when belief models are likely misspecified, conservative depth-one planning with hard safety constraints can be more robust than expensive belief-updating planners; the paper observes this empirically but does not elevate it to a general principle.
  • A testable extension would be to generate new planners by choosing intermediate n with different Eval_i sources, such as mean-field approximations or learned value functions, and checking whether the predicted performance ordering holds across domains beyond route planning.
  • The same layered lookahead could apply to the mechanism-design and negotiation settings named in the conclusion, where an agent must explore and exploit unknown private types of other participants.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes a unifying framework for type-based single-agent planning in multi-agent systems. The controlled agent maintains a belief over a finite set of stationary Markov opponent policies, which yields a POMDP formulation and several approximations (belief-induced MDP, QMDP, finite-horizon lookahead, and MCTS variants). The core idea is a layered lookahead search parameterized by (n, m, Eval_i), in which the first n levels update the belief, the next m levels fix it, and leaves are evaluated by a heuristic; this is claimed to embed exact POMDP solving (F1), belief-induced MDP (F2), QMDP (F3), finite-horizon POMDP with MDP terminal values (F4), and scalable tree-search variants (F5-F7). The authors implement 13 planners in a multi-agent route-planning benchmark with up to 50 agents, report a predicted performance ranking F1≽F4≽F2∼F5≽F6∼F7, and highlight safe-agents, a myopic special case, as surprisingly effective. Appendix A proves γ-contraction of the two backup operators.

Significance. If scoped appropriately, the framework is a useful conceptual synthesis: it connects POMDP theory, QMDP, belief-induced MDPs, and Monte-Carlo tree search under one notation, and the contraction proofs in Appendix A are standard and correct in substance. The empirical suite is extensive for the single-agent planning setting, spanning up to 50 agents and 13 implemented planners, and the code release is a practical strength. The safe-agent observation is interesting and falsifiable. However, the paper currently overclaims the predictive power of the framework: the proposed ranking is stated without proof, the experiments do not cover F1 and F4, and the safe-agent results, which are presented as instances of the same framework, are not accounted for by the ranking. The exactness claims also rest on a support assumption that the experiments do not satisfy, so the empirical section is better interpreted as a robustness study than as validation of the modelled-optimality statements.

major comments (4)
  1. [Section 4 and Table 3] The predicted ranking F1≽F4≽F2∼F5≽F6∼F7 is stated without proof, and the paper says the trend can be seen in Table 3, but F1 and F4 are not implemented in Table 3, so the experiments cannot support the positions of F1 and F4 in the ordering. More importantly, the safe-agent planners, which are introduced as depth-one special cases of the same framework, outperform the deeper F5/F6/F7 planners in Medium20a and Large50a (e.g., Large50a Rational: EnhancedSafe 74.60 vs UnifTSCBS 120.48 and MCTSCBSpuct 119.26; Medium20a Rational: EnhancedSafe 35.52 vs UnifTSCBS 56.04 and MCTS 59.15). Since safe-agents are claimed to be instances of the framework, the general statement that 'the deeper (and more deliberately) it searches, the better it performs' cannot be maintained without conditioning on Eval_i and computational budget; as presented, Table 3 does not validate the predicted ordering.
  2. [Section 5.1.3] The safe-agent rule discards any action for which there exists an opponent action that can cause a collision, i.e., a worst-case existential pruning rule, whereas the unified backup in Section 4(1)-(2) is a max over the modelling agent's actions of an expectation over belief-weighted opponent policies. The manuscript does not specify which (n, m, Eval_i) instantiation, and which belief and collision-penalty values, recover this existential pruning from the framework's stated operators. If the intended encoding is an infinite collision penalty inside Eval_i, that should be stated explicitly and its effect on the contraction and exactness arguments of Appendix A should be discussed; without this, the claim that safe-agents are a special case of the proposed framework is not demonstrated.
  3. [Section 3.2 and Section 5.1.4] The exactness of F1 and the Bayesian update in Eq. (1) both require that the true opponent policies lie in the support of the modelling agent's prior and are stationary Markov. The paper explicitly acknowledges the stationarity assumption, but the experimental opponent pools are not contained in the initialized support: ShortestPathAgent can use a different tie-breaking rule (Appendix D, Case 1), RandomAgent(p) randomizes with a probability p that is not generally equal to the epsilon in the prior, and ChasingAgent(p) targets the modelling agent's location rather than a fixed goal. Consequently, all numbers in Table 3 are obtained under model misspecification. The text should state that the experiments test robustness under misspecification rather than the modelled-optimality claims, and the conclusion that belief-dependent planning breaks down under chasing behavior should be framed accordingly.
  4. [Table 4 and Table 3] The empirical comparison uses scenario-specific hyperparameters (epsilon from 7E-04 to 2E-05, depth 1-2, eval_samples 2-10, backup_samples exact or 10, max_iter 30-100, select_samples 50-125) with no tuning protocol, no sensitivity analysis, and no normalization of computational budget across planners. Since Table 3 is presented as a practical handbook for planner selection, the lack of an ablation or budget-equivalence analysis makes it difficult to distinguish algorithmic merit from hyperparameter choices, and several cells marked '/' mean that the comparison is incomplete at the larger scales.
minor comments (7)
  1. [Abstract] The word 'performaces' should be 'performances'.
  2. [Introduction and Related Work] There are several small language errors: 'dose not' should be 'does not' in Section 1, and 'An critical follow-up issue' should be 'A critical follow-up issue' in Section 2.
  3. [Section 4] The phrase 'introduce a new forth one' should be 'introduce a new fourth one'.
  4. [Section 5.2 and Figure 4-8] The legend labels 'MetaAgentFixedBelief' and 'MetaAgentUpdateBelief' in Figures 4 and 5 do not match the planner names 'RLFixed' and 'RLUpdate' used in Table 3; the naming should be unified.
  5. [Figure 6 caption] The caption spells 'Sqaure4a' instead of 'Square4a'.
  6. [Appendix D, Case 1] The text says 'shorest-path agent' instead of 'shortest-path agent'.
  7. [Algorithm 7] The phrase 'Categorial distribution' should be 'categorical distribution'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's F1-F7 equivalences are explicit parameter specializations of a layered Bellman/tree-search operator, with contraction proofs and standard POMDP/MDP theory supplying the content; no fitted parameter is renamed as a prediction.

full rationale

The paper's central unification claim is a parameter-embedding claim, not a derivation of empirical results from fitted inputs. In Section 4, F1 with n = infinity reproduces Equation (2) because layer (1) is literally the belief-state Bellman optimality equation in iterative form, and Theorem A.2 establishes that the backup operator is a gamma-contraction; the identification with exact infinite-horizon POMDP solving rests on standard POMDP theory [25, 51, 53], not on the authors' experiments. F2 with m = infinity reproduces Equation (3) because layer (2) is value iteration on the belief-induced MDP, with Theorem A.1 providing the contraction; F3 embeds QMDP by defining Eval_i to be the QMDP mixing expression, an intentional special case rather than a predicted consequence. F5-F7 are additional depth/sampling/bandit choices; none of their parameters (beta, epsilon, depths, sample counts) is fitted to make the headline claim true, and the safe-agent enhanced variant is presented as an observed empirical finding with an acknowledged failure mode, not as a consequence of the framework. The predicted ordering F1≽F4≽F2~F5≽F6~F7 is a falsifiable heuristic; the reported results in fact diverge from it, with safe-agents outperforming deeper searches in large-scale and malicious scenarios, which is the opposite of a self-fulfilling construction. The main scope limitation, the finite stationary-Markov opponent-type assumption in Section 3.2, is explicitly flagged by the authors and affects the validity of the "exact" POMDP claim when opponents are out-of-support or adaptive, but that is a correctness/scope concern, not circularity. The only self-citations ([21], [63]) are peripheral pointers and do not carry any load-bearing argument. No quoted equation reduces to its own input or renames a fitted value as a prediction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the standard POMDP/MDP machinery plus domain-specific assumptions about stationary, independent, non-adaptive opponents and about the MARP collision model. The only tunable numbers are belief and exploration hyperparameters; no new physical or mathematical entities are introduced.

free parameters (3)
  • beta (belief temperature) = 1 in most planners; 0 in enhanced safe-agent hard-max case
    Tunable in Eq. (1); hand-chosen, not derived from data.
  • epsilon (opponent randomness probability) = 7e-4, 2e-4, 2e-4, 8e-5, 2e-5 per scenario (Table 4)
    Assumed randomness in the opponent belief model; varies per scenario with no tuning protocol.
  • EECBS suboptimality factor = 0.2
    Chosen by hand to balance speed and plan quality when extracting NE heuristics.
assumptions (5)
  • domain assumption Opponents are stationary Markov policies drawn from a finite set of types and do not learn or adapt during play.
    Section 3.2 explicitly flags this as a strong assumption; it underpins the POMDP/CMDP reduction and Eq. (1).
  • domain assumption Belief over opponent types factorizes as a product of independent per-opponent beliefs.
    Section 3.2: b(pi_-i) = prod_j b(pi_j); used to make the belief update and tree search tractable.
  • domain assumption The modelling agent has perfect recall of all past states and joint actions.
    Section 3.2 states this to justify history-dependent policies and the sufficient-statistic reduction.
  • domain assumption In MARP, a set of collision-free paths that minimizes total length is a Nash equilibrium because no agent benefits from unilateral deviation.
    Section 5.1.2; used to treat EECBS outputs as NE strategies for value and policy priors.
  • ad hoc to paper An action is unsafe if there exists any opponent action that could cause a collision; safe-agents discard all such actions.
    Section 5.1.3; this conservative rule defines the safe-agent heuristic and is domain-specific.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Single-Agent Planning in a Multi-Agent System: A Unified Framework for Type-Based Planners." pith.science (2026). https://pith.science/paper/4GY72CK7

@misc{pith2026250208950,
  author       = {Pith},
  title        = {Pith review of: Single-Agent Planning in a Multi-Agent System: A Unified Framework for Type-Based Planners},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GY72CK7}},
  note         = {Machine review of arXiv:2502.08950}
}
read the original abstract

We consider a general problem where an agent is in a multi-agent environment and must plan for herself without any prior information about her opponents. At each moment, this pivotal agent is faced with a trade-off between exploiting her currently accumulated information about the other agents and exploring further to improve future (re-)planning. We propose a theoretic framework that unifies a spectrum of planners for the pivotal agent to address this trade-off. The planner at one end of this spectrum aims to find exact solutions, while those towards the other end yield approximate solutions as the problem scales up. Beyond theoretical analysis, we also implement \textbf{13} planners and conduct experiments in a specific domain called \textit{multi-agent route planning} with the number of agents \textbf{up to~50}, to compare their performaces in various scenarios. One interesting observation comes from a class of planners that we call \textit{safe-agents} and their enhanced variants by incorporating domain-specific knowledge, which is a simple special case under the proposed general framework, but performs sufficiently well in most cases. Our unified framework, as well as those induced planners, provides new insights on multi-agent decision-making, with potential applications to related areas such as mechanism design.

Figures

Figures reproduced from arXiv: 2502.08950 by the authors.

Figure 1
Figure 1. The convergence dynamics. B PLANNERS IN PSEUDOCODE By convention, we use 𝑆𝐺 to denote an instantiated multi-agent environment, 𝑆𝐺.𝑟𝑒𝑠𝑒𝑡() launches a new episode and returns the initial state, and 𝑆𝐺.𝑠𝑡𝑒𝑝(𝑎𝑖 , 𝑎−𝑖) proceeds the environment by the given joint actions and returns the successor state. An unbounded while loop is used to represent a running episode, and will terminate automatically if 𝑆𝐺 reaches an end st… view at source ↗
Figure 2
Figure 2. The exact optimal plan (a), a potential approximated online plan with repeated replanning by layered tree search (b), [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Statistics of the RL training samples. 7https://stable-baselines3.readthedocs.io/en/master/ [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Detailed experiments for “Small2a” configurations. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Detailed experiments for “Square2a” configurations. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Detailed experiments for “Sqaure4a” configurations. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Detailed experiments for “Medium20a” configurations. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Detailed experiments for “Large50a” configurations. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Detailed paths planned in a tiny situation (3x3 map two agents): agent 1 is a shorest-path agent while agent 2 is [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Different results by MDP agents with and without belief update. Agent 2 is the modelling agent using MDP planners, [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Agent 2 is the modelling agent using any of the planners noted at the lower left corner of each sub-figure, while [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Agent 1 is the modelling agent using the respective planners noted at the lower left corner of each sub-figure, while [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 59 canonical work pages

  1. [1]

    Stefano V Albrecht, Jacob W Crandall, and Subramanian Ramamoorthy. 2016. Belief and truth in hypothesised behaviours. Artificial Intelligence 235 (2016), 63–94

    work page 2016

  2. [2]

    Stefano V Albrecht and Subramanian Ramamoorthy. 2015. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. arXiv preprint arXiv:1506.01170 (2015)

    arXiv 2015

  3. [3]

    Stefano V Albrecht and Subramanian Ramamoorthy. 2019. On convergence and optimality of best-response learning with policy types in multiagent systems. arXiv preprint arXiv:1907.06995 (2019)

    work page Pith review arXiv 2019

  4. [4]

    Stefano V Albrecht and Peter Stone. 2018. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence 258 (2018), 66–95

    work page 2018

  5. [5]

    Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. 2022. Planning in Stochastic Environments with a Learned Model. In International Conference on Learning Representations . https://openreview.net/ forum?id=X6D9bAHhBQ1

    work page 2022

  6. [6]

    Dipyaman Banerjee and Sandip Sen. 2007. Reaching pareto-optimality in pris- oner’s dilemma using conditional joint action learning. Autonomous Agents and Multi-Agent Systems 15 (2007), 91–108

    work page 2007

  7. [7]

    Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. 2023. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 (2023)

    arXiv 2023

  8. [8]

    Martin Beer, Mark d’Inverno, Michael Luck, Nick Jennings, Chris Preist, and Michael Schroeder. 1999. Negotiation in multi-agent systems. The Knowledge Engineering Review 14, 3 (1999), 285–289

    work page 1999

Show all 69 references
  1. [9]

    Carolin Benjamins, Theresa Eimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. 2021. Carl: A benchmark for contextual and adaptive reinforcement learning. arXiv preprint arXiv:2110.02102 (2021)

    2021 arXiv

  2. [10]

    Craig Boutilier. 1996. Planning, learning and coordination in multiagent decision processes. In TARK, Vol. 96. Citeseer, 195–210

    1996

  3. [11]

    Sally C Brailsford, Chris N Potts, and Barbara M Smith. 1999. Constraint satis- faction problems: Algorithms and applications. European journal of operational research 119, 3 (1999), 557–581

    1999

  4. [12]

    Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samoth- rakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence a...

    2012

  5. [13]

    David Carmel and Shaul Markovitch. 1998. How to explore your opponent’s strategy (almost) optimally. InProceedings International Conference on Multi Agent Systems (Cat. No. 98EX160) . IEEE, 64–71

    1998

  6. [14]

    David Carmel and Shaul Markovitch. 1999. Exploration strategies for model- based learning in multi-agent systems: Exploration strategies.Autonomous Agents and Multi-agent systems 2 (1999), 141–172

    1999

  7. [15]

    Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. 2024. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  8. [16]

    Caroline Claus and Craig Boutilier. 1998. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI 1998, 746-752 (1998), 2

    1998

  9. [17]

    Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. 2022. Policy improvement by planning with Gumbel. In International Conference on Learning Representations. https://openreview.net/forum?id=bERaNdoegnO

    2022

  10. [18]

    Adam Eck, Maulik Shah, Prashant Doshi, and Leen-Kiat Soh. 2020. Scalable decision-theoretic planning in open and typed multiagent systems. InProceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 7127–7134

    2020

  11. [19]

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shi- mon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32

    2018

  12. [20]

    Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. 2022. Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162) , Kamalika C...

    2022

  13. [21]

    Zihang Gao, Fangzhen Lin, Yi Zhou, Hao Zhang, Kaishun Wu, and Haodi Zhang

  14. [22]

    Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 (2015)

    2015 arXiv

  15. [23]

    Bayesian

    John C Harsanyi. 1967. Games with incomplete information played by “Bayesian” players, I–III Part I. The basic model. Management science 14, 3 (1967), 159–182

    1967

  16. [24]

    Nicholas R Jennings, Peyman Faratin, Alessio R Lomuscio, Simon Parsons, Carles Sierra, and Michael Wooldridge. 2001. Automated negotiation: prospects, methods and challenges. International Journal of Group Decision and Negotiation 10, 2 (2001), 199–215

    2001

  17. [25]

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Plan- ning and acting in partially observable stochastic domains. Artificial intelligence 101, 1-2 (1998), 99–134

    1998

  18. [26]

    Ehud Kalai and Ehud Lehrer. 1993. Rational learning leads to Nash equilibrium. Econometrica: Journal of the Econometric Society (1993), 1019–1045

    1993

  19. [27]

    Levente Kocsis and Csaba Szepesvári. 2006. Bandit based monte-carlo planning. In European conference on machine learning . Springer, 282–293

    2006

  20. [28]

    Sarit Kraus. 1997. Negotiation and cooperation in multi-agent environments. Artificial intelligence 94, 1-2 (1997), 79–97

    1997

  21. [29]

    Jean-Michel Lasry and Pierre-Louis Lions. 2007. Mean field games. Japanese journal of mathematics 2, 1 (2007), 229–260

    2007

  22. [30]

    Jiaoyang Li, Wheeler Ruml, and Sven Koenig. 2021. Eecbs: A bounded-suboptimal search for multi-agent path finding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 12353–12362

    2021

  23. [31]

    Michael L Littman, Anthony R Cassandra, and Leslie Pack Kaelbling. 1995. Learn- ing policies for partially observable environments: Scaling up. In Machine Learn- ing Proceedings 1995. Elsevier, 362–370

    1995

  24. [32]

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017)

    2017

  25. [33]

    Chengdong Ma, Aming Li, Yali Du, Hao Dong, and Yaodong Yang. 2024. Efficient and scalable reinforcement learning for large-scale network control. Nature Machine Intelligence (2024), 1–15

    2024

  26. [34]

    Omid Madani, Steve Hanks, and Anne Condon. 1999. On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems. In Proceedings of the sixteenth national conference on Artificial intelli- gence and the eleventh Innovative app...

    1999

  27. [35]

    Christos H Papadimitriou and John N Tsitsiklis. 1987. The complexity of Markov decision processes. Mathematics of operations research 12, 3 (1987), 441–450

    1987

  28. [36]

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8. http://jmlr.org/papers/v22/20-1364.html

    2021

  29. [37]

    Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V Albrecht. 2023. A general learning framework for open ad hoc teamwork using graph-based policy learning. Journal of Machine Learning Research 24, 298 (2023), 1–74

    2023

  30. [38]

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Far- quhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1–51

    2020

  31. [39]

    Christopher D Rosin. 2011. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence 61, 3 (2011), 203–230

    2011

  32. [40]

    Spyridon Samothrakis, David Robles, and Simon Lucas. 2011. Fast approximate max-n monte carlo tree search for ms pac-man. IEEE Transactions on Computa- tional Intelligence and AI in Games 3, 2 (2011), 142–154

    2011

  33. [41]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Far- quhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. 2019. The StarCraft Multi-Agent Challenge. CoRR abs/1902.04043 (2019)

    2019 arXiv

  34. [42]

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature 588, 7839 (2020), 604–609

    2020

  35. [43]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  36. [44]

    Jonathon Schwartz and Hanna Kurniawati. 2023. Bayes-Adaptive Monte-Carlo Planning for Type-Based Reasoning in Large Partially Observable, Multi-Agent Environments. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems . 2355–2357

    2023

  37. [45]

    Jonathon Schwartz, Hanna Kurniawati, and Marcus Hutter. 2023. Combining a Meta-Policy and Monte-Carlo Planning for Scalable Type-Based Reasoning in Partially Observable Environments. arXiv preprint arXiv:2306.06067 (2023)

    2023 arXiv

  38. [46]

    Lloyd S Shapley. 1953. Stochastic games. Proceedings of the national academy of sciences 39, 10 (1953), 1095–1100

    1953

  39. [47]

    Guni Sharon, Roni Stern, Ariel Felner, and Nathan R Sturtevant. 2015. Conflict- based search for optimal multi-agent pathfinding.Artificial Intelligence 219 (2015), 40–66

    2015

  40. [48]

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Grae- pel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters...

    2018 doi

  41. [49]

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. nature 550, 7676 (2017), 354–359

    2017

  42. [50]

    David Silver and Joel Veness. 2010. Monte-Carlo planning in large POMDPs. Advances in neural information processing systems 23 (2010)

    2010

  43. [51]

    Richard D Smallwood and Edward J Sondik. 1973. The optimal control of partially observable Markov processes over a finite horizon. Operations research 21, 5 (1973), 1071–1088

    1973

  44. [52]

    Eilon Solan and Nicolas Vieille. 2015. Stochastic games. Proceedings of the National Academy of Sciences 112, 45 (2015), 13743–13746

    2015

  45. [53]

    Edward J Sondik. 1978. The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations research 26, 2 (1978), 282–304

    1978

  46. [54]

    Dale O Stahl. 1993. Evolution of smartn players. Games and Economic Behavior 5, 4 (1993), 604–617

    1993

  47. [55]

    Roni Stern. 2019. Multi-agent path finding–an overview. Artificial Intelligence (2019), 96–115

    2019

  48. [56]

    Roni Stern, Nathan R Sturtevant, Ariel Felner, Sven Koenig, Hang Ma, Thayne T Walker, Jiaoyang Li, Dor Atzmon, Liron Cohen, TK Satish Kumar, et al. 2019. Multi- agent pathfinding: Definitions, variants, and benchmarks. In Twelfth Annual Symposium on Combinatorial Search

    2019

  49. [57]

    Pingzhong Tang. 2017. Reinforcement mechanism design. In Proceedings of the 26th International Joint Conference on Artificial Intelligence . 5146–5150

    2017

  50. [58]

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The Surprising Effectiveness of PPO in Cooperative Multi- Agent Games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openr...

    2022

  51. [59]

    Jingjin Yu and Steven LaValle. 2013. Structure and intractability of optimal multi- robot path planning on graphs. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 27. 1443–1449

    2013

  52. [60]

    Han Zhang, Jiaoyang Li, Pavel Surynek, TK Satish Kumar, and Sven Koenig

  53. [61]

    Nevin Lianwen Zhang and Weihong Zhang. 2001. Speeding up the convergence of value iteration in partially observable Markov decision processes. Journal of Artificial Intelligence Research 14 (2001), 29–51

    2001

  54. [62]

    Ming Zhou, Ziyu Wan, Hanjing Wang, Muning Wen, Runzhe Wu, Ying Wen, Yaodong Yang, Yong Yu, Jun Wang, and Weinan Zhang. 2023. MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning. Journal of Machine Learning Research 24, 150 (2023), 1–12. http://j...

    2023

  55. [63]

    Fengming Zhu and Fangzhen Lin. 2024. On Computing Universal Plans for Partially Observable Multi-Agent Path Finding. arXiv:2305.16203 [cs.MA] https: //arxiv.org/abs/2305.16203

    2024

  56. [64]

    Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione

  57. [69]

    Small_2a

    It is also proposed to use certain prior policies to guide the choices, resulting the pUCT formula [39, 42, 49], 𝑡𝑐∈ arg max 𝑡𝑐∈𝑡.𝑐ℎ𝑖𝑙𝑑𝑟𝑒𝑛 𝑡𝑐.𝑣 𝑡𝑐.𝑁 +𝑡𝑐.𝑝𝑜𝑙𝑖𝑐𝑦 _𝑝𝑟𝑖𝑜𝑟· √︄ ln(𝑡.𝑁) 𝑡𝑐.𝑁 𝑐1+ ln𝑡.𝑁+𝑐2 𝑐2 (6) where𝑐1 and𝑐2 are two constants controlling the influence of the prior po...

  58. [2007]

    sup” simply becomes “max

    Regret minimization in games with incomplete information. Advances in neural information processing systems 20 (2007). A THEORETIC ANALYSIS OF THE UNIFIED FRAMEWORK For the belief-fixed lookahead search presented in Section 4.(2), since the belief is fixed, we can make the not...

    2007

  59. [2017]

    arXiv preprint arXiv:1707.06347 (2017)

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  60. [2020]

    In Proceedings of the AAAI Conference on Artificial Intelligence , Vol

    Embedding high-level knowledge into dqns to learn faster and more safely. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 13608–13609

  61. [2022]

    Artificial Intelligence 311 (2022), 103766

    Multi-agent path finding with mutex propagation. Artificial Intelligence 311 (2022), 103766

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.