pith. sign in

arxiv: 2503.04798 · v3 · submitted 2025-03-03 · 💻 cs.RO · cs.AI

Advancing MAPF Toward the Real World: A Scalable Multi-Agent Realistic Testbed (SMART)

Jingtian Yan , Zhifei Li , William Kang , Kevin Zheng , Yulun Zhang , Zhe Chen , Yue Zhang , Daniel Harabor
show 2 more authors
Stephen F. Smith Jiaoyang Li
This is my paper

Pith reviewed 2026-05-23 01:58 UTC · model grok-4.3

classification 💻 cs.RO cs.AI
keywords MAPFmulti-agent path findingsimulation testbedphysics engineroboticsscalable evaluationaction dependency graph
0
0 comments X

The pith

SMART provides a physics-engine simulator and Action Dependency Graph monitor to test MAPF planners realistically on thousands of robots.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MAPF planners can find paths for hundreds of robots quickly but often rely on simplified movement models whose real performance remains unclear. The paper presents SMART as a testbed that uses physics engines to simulate actual robot kinodynamics and execution uncertainties instead. It adds an execution monitor based on the Action Dependency Graph so different planners and robot models integrate without major changes. The system handles simulations at the scale of thousands of robots. If the simulation matches hardware behavior closely enough, both researchers and industry users can evaluate planner performance without access to large physical fleets.

Core claim

SMART creates realistic simulation environments with physics engines that model robot kinodynamics and execution uncertainties, paired with an Action Dependency Graph framework for integrating MAPF planners, and demonstrates scaling to thousands of robots.

What carries the argument

Action Dependency Graph execution monitor framework that links MAPF plans to robot actions inside the physics simulator.

If this is right

  • MAPF planners can be assessed for robustness under realistic execution errors and dynamics.
  • Users without deep MAPF expertise can test planners in their own specific environments.
  • Evaluations reach scales of thousands of robots that exceed most laboratory resources.
  • New planners or robot models plug into the monitor with minimal additional code.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Planners could be redesigned to optimize explicitly for the uncertainties captured in the simulation.
  • The testbed framework might apply to other multi-robot coordination problems beyond path finding.
  • Detailed comparison of planned versus executed paths could reveal new improvement targets for algorithms.

Load-bearing premise

The physics simulation and Action Dependency Graph will produce execution behavior sufficiently close to real hardware that results transfer meaningfully.

What would settle it

Running identical MAPF plans in SMART and on physical robots and observing large systematic differences in collision rates, completion times, or success rates due to unmodeled factors.

Figures

Figures reproduced from arXiv: 2503.04798 by Daniel Harabor, Jiaoyang Li, Jingtian Yan, Kevin Zheng, Stephen F. Smith, William Kang, Yue Zhang, Yulun Zhang, Zhe Chen, Zhifei Li.

Figure 1
Figure 1. Figure 1: (a) A simulation environment in SMART. (b) The original [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) MAPF problem. (b) Associated MAPF plan. (c) ADG from [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: System overview of SMART. from ri and rj , a Type-2 edge (v k i , vs j ) ensures that v k i must be completed before v s j can start. During execution, each action is performed according to these dependencies. The detailed execution procedure is described in Section III-B. C. Evaluating MAPF Plans To evaluate MAPF algorithms, researchers typically gen￾erate a plan for a given MAPF instance and then execute… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Cycle in the ADG causing all robots to be stalled. (b) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The SMART web interface visualizing a multi-robot simula [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Simulation environments. (a) empty (b) maze (c)random (d) game (e) warehouse (f) sortation-center. Installing SMART Locally: Since SMART provides flex￾ible interfaces for customization, researchers may choose to compile it locally for advanced use cases, such as running it in headless mode, integrating it into custom pipelines, or interfacing with real robots. As an open-source tool, SMART includes documen… view at source ↗
Figure 8
Figure 8. Figure 8: (a) Experimental setup in Isaac Sim. (b) Experimental setup [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
read the original abstract

We present Scalable Multi-Agent Realistic Testbed (SMART), a realistic and efficient software tool for evaluating Multi-Agent Path Finding (MAPF) algorithms. MAPF focuses on planning collision-free paths for a group of robots. While state-of-the-art MAPF planners can plan paths for hundreds of robots in seconds, they often rely on simplified robot models, making their real-world performance unclear. Researchers typically lack access to hundreds of physical robots in laboratory settings to evaluate the algorithms. Meanwhile, industrial professionals who lack expertise in MAPF require an easy-to-use simulator to efficiently test and understand the performance of MAPF planners in their specific settings. SMART fills this gap with several advantages: (1) SMART uses physics-engine-based simulators to create realistic simulation environments, accounting for complex real-world factors such as robot kinodynamics and execution uncertainties, (2) SMART uses an execution monitor framework based on the Action Dependency Graph, facilitating seamless integration with various MAPF planners and robot models, and (3) SMART scales to thousands of robots. The code is publicly available at https://github.com/smart-mapf/smart with an online service available at https://smart-mapf.github.io/demo/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 0 minor

Summary. The manuscript presents SMART, a software testbed for evaluating Multi-Agent Path Finding (MAPF) algorithms. It claims to provide realistic simulation via physics-engine integration that accounts for robot kinodynamics and execution uncertainties, an Action Dependency Graph (ADG) based execution monitor that enables seamless integration with diverse MAPF planners and robot models, and scalability to thousands of robots, with the codebase released publicly at the cited GitHub repository and an accompanying online demo service.

Significance. If the implementation delivers the stated capabilities, SMART would address a practical gap in MAPF research by enabling evaluation under more realistic conditions than simplified models without requiring physical robot fleets. The public release of the codebase is a clear strength that supports reproducibility, community adoption, and extension by both researchers and industrial users.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive review and recommendation to accept. The assessment correctly identifies the core contributions of SMART in providing realistic, scalable evaluation for MAPF algorithms via physics-based simulation and the ADG framework, along with the value of the public code release.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a tool presentation describing the SMART testbed, its physics-engine integration, ADG-based execution monitor, and scaling capabilities. No derivations, equations, predictions, fitted parameters, or load-bearing self-citations appear in the provided text or abstract. The central claims are implementation facts about a released codebase rather than any chain that reduces to its own inputs by construction. This is the expected honest non-finding for a software artifact paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is a software tool contribution; it introduces no mathematical free parameters, axioms, or invented physical entities. Claims rest on the implementation and claimed functionality of the described simulator and framework.

pith-pipeline@v0.9.0 · 5770 in / 1079 out tokens · 68210 ms · 2026-05-23T01:58:27.367051+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages

  1. [1]

    Multi-agent pathfinding: Definitions, variants, and benchmarks,

    R. Stern, N. R. Sturtevant, A. Felner, S. Koenig, H. Ma, T. T. Walker, J. Li, D. Atzmon, L. Cohen, T. K. S. Kumar, E. Boyarski, and R. Bart ´ak, “Multi-agent pathfinding: Definitions, variants, and benchmarks,” in Proceedings of the International Symposium on Combinatorial Search , 2019, pp. 151–159

    work page 2019

  2. [2]

    Feasibility study: Moving non-homogeneous teams in congested video game environ- ments,

    H. Ma, J. Yang, L. Cohen, T. Kumar, and S. Koenig, “Feasibility study: Moving non-homogeneous teams in congested video game environ- ments,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 13, no. 1, 2017, pp. 270–272

    work page 2017

  3. [3]

    Persistent and robust execution of MAPF schedules in warehouses,

    W. H ¨onig, S. Kiesel, A. Tinka, J. W. Durham, and N. Ayanian, “Persistent and robust execution of MAPF schedules in warehouses,” IEEE Robotics and Automation Letters , vol. 4, no. 2, pp. 1125–1131, 2019

    work page 2019

  4. [4]

    Multi-agent path finding in unmanned aircraft system traffic management with scheduling and speed variation,

    F. Ho, A. Gonc ¸alves, B. Rigault, R. Geraldes, A. Chicharo, M. Cavazza, and H. Prendinger, “Multi-agent path finding in unmanned aircraft system traffic management with scheduling and speed variation,” IEEE Intelligent Transportation Systems Magazine , vol. 14, no. 5, pp. 8–21, 2022

    work page 2022

  5. [5]

    Graph-based multi-robot path finding and planning,

    H. Ma, “Graph-based multi-robot path finding and planning,” Current Robotics Reports, vol. 3, no. 3, pp. 77–84, 2022

    work page 2022

  6. [6]

    Where paths collide: A comprehensive survey of classic and learning-based multi-agent pathfinding,

    S. Wang, H. Xu, Y . Zhang, J. Lin, C. Lu, X. Wang, and W. Li, “Where paths collide: A comprehensive survey of classic and learning-based multi-agent pathfinding,” arXiv preprint, vol. arXiv:2505.19219, 2025

    work page arXiv 2025

  7. [7]

    Searching with consistent prioritization for multi-agent path finding,

    H. Ma, D. Harabor, P. J. Stuckey, J. Li, and S. Koenig, “Searching with consistent prioritization for multi-agent path finding,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 7643–7650

    work page 2019

  8. [8]

    Multi- agent path finding with mutex propagation,

    H. Zhang, J. Li, P. Surynek, T. K. S. Kumar, and S. Koenig, “Multi- agent path finding with mutex propagation,” Artificial Intelligence, vol. 311, p. 1034766, 2022

    work page 2022

  9. [9]

    MAPF- LNS2: Fast repairing for multi-agent path finding via large neighborhood search,

    J. Li, Z. Chen, D. Harabor, P. J. Stuckey, and S. Koenig, “MAPF- LNS2: Fast repairing for multi-agent path finding via large neighborhood search,” in Proceedings of the AAAI Conference on Artificial Intelli- gence, vol. 36, 2022, pp. 10 256–10 265

    work page 2022

  10. [10]

    A benchmark for multi-robot planning in realistic, complex and cluttered environments,

    S. Schaefer, L. Palmieri, L. Heuer, R. Dillmann, S. Koenig, and A. Kleiner, “A benchmark for multi-robot planning in realistic, complex and cluttered environments,” in Proceedings of the IEEE International Conference on Robotics and Automation , 2023, pp. 9231–9237

    work page 2023

  11. [11]

    Benchmarking multi-robot coordination in realistic, unstructured human-shared environments,

    L. Heuer, L. Palmieri, A. Mannucci, S. Koenig, and M. Magnus- son, “Benchmarking multi-robot coordination in realistic, unstructured human-shared environments,” in Proceedings of the IEEE International Conference on Robotics and Automation , 2024, pp. 14 541–14 547

    work page 2024

  12. [12]

    Conflict-based search for optimal multi-agent pathfinding,

    G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant, “Conflict-based search for optimal multi-agent pathfinding,” Artificial Intelligence, vol. 219, pp. 40–66, 2015

    work page 2015

  13. [13]

    EECBS: A bounded-suboptimal search for multi-agent path finding,

    J. Li, W. Ruml, and S. Koenig, “EECBS: A bounded-suboptimal search for multi-agent path finding,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, 2021, pp. 12 353–12 362

    work page 2021

  14. [14]

    Priority inheri- tance with backtracking for iterative multi-agent path finding,

    K. Okumura, M. Machida, X. D ´efago, and Y . Tamura, “Priority inheri- tance with backtracking for iterative multi-agent path finding,” Artificial Intelligence, vol. 310, p. 103752, 2022

    work page 2022

  15. [15]

    LaCAM: Search-based algorithm for quick multi-agent pathfinding,

    K. Okumura, “LaCAM: Search-based algorithm for quick multi-agent pathfinding,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 10, 2023, pp. 11 655–11 662

    work page 2023

  16. [16]

    Anytime multi-agent path finding via large neighborhood search,

    J. Li, Z. Chen, D. Harabor, P. J. Stuckey, and S. Koenig, “Anytime multi-agent path finding via large neighborhood search,” in Proceedings of the International Joint Conference on Artificial Intelligence , 2021, pp. 4127–4135

    work page 2021

  17. [17]

    Anytime multi-agent path finding via machine learning-guided large neighborhood search,

    T. Huang, J. Li, S. Koenig, and B. Dilkina, “Anytime multi-agent path finding via machine learning-guided large neighborhood search,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2022, pp. 9368–9376

    work page 2022

  18. [18]

    Optimal and bounded-suboptimal multi-agent motion planning,

    L. Cohen, T. Uras, T. K. S. Kumar, and S. Koenig, “Optimal and bounded-suboptimal multi-agent motion planning,” in Proceedings of the International Symposium on Combinatorial Search , vol. 10, 2019, pp. 44–51

    work page 2019

  19. [19]

    Robust multi-agent path finding and executing,

    D. Atzmon, R. Stern, A. Felner, G. Wagner, R. Bart ´ak, and N.-F. Zhou, “Robust multi-agent path finding and executing,” Journal of Artificial Intelligence Research, vol. 67, pp. 549–579, 2020

    work page 2020

  20. [20]

    Multi-agent motion planning with b ´ezier curve optimization under kinodynamic constraints,

    J. Yan and J. Li, “Multi-agent motion planning with b ´ezier curve optimization under kinodynamic constraints,” IEEE Robotics and Au- tomation Letters, vol. 9, no. 3, pp. 3021–3028, 2024

    work page 2024

  21. [21]

    Multi-agent motion planning for differential drive robots through stationary state search,

    ——, “Multi-agent motion planning for differential drive robots through stationary state search,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 22, 2025, pp. 23 360–23 368

    work page 2025

  22. [22]

    Provably safe and deadlock- free execution of multi-robot plans under delaying disturbances,

    M. ˇC´ap, J. Gregoire, and E. Frazzoli, “Provably safe and deadlock- free execution of multi-robot plans under delaying disturbances,” in Proceedings of the IEEE International Conference on Intelligent Robots and Systems, 2016, pp. 5113–5118

    work page 2016

  23. [23]

    Which MAPF model works best for automated warehousing?

    S. Varambally, J. Li, and S. Koenig, “Which MAPF model works best for automated warehousing?” in Proceedings of the International Symposium on Combinatorial Search , vol. 15, 2022, pp. 190–198

    work page 2022

  24. [24]

    Experimenting with robotic intra-logistics domains,

    M. Gebser, P. Obermeier, T. Otto, T. Schaub, O. Sabuncu, V . Nguyen, and T. C. Son, “Experimenting with robotic intra-logistics domains,” Theory and Practice of Logic Programming , vol. 18, no. 3-4, pp. 502– 519, 2018

    work page 2018

  25. [25]

    Benchmarks for grid-based pathfinding,

    N. R. Sturtevant, “Benchmarks for grid-based pathfinding,” IEEE Trans- actions on Computational Intelligence and AI in Games , vol. 4, no. 2, pp. 144–148, 2012

    work page 2012

  26. [26]

    A quality diversity method to automatically generate multi-agent path finding benchmark maps,

    C. Qian, Y . Zhang, V . Bhatt, M. C. Fontaine, S. Nikolaidis, and J. Li, “A quality diversity method to automatically generate multi-agent path finding benchmark maps,” arXiv preprint, vol. arXiv:2409.06888, 2024

    work page arXiv 2024

  27. [27]

    P., Nygren, E., Laurent, F., Schneider, M., Scheller, C

    S. Mohanty, E. Nygren, F. Laurent, M. Schneider, C. Scheller, N. Bhattacharya, J. Watson, A. Egli, C. Eichenberger, C. Baumberger, G. Vienken, I. Sturm, G. Sartoretti, and G. Spigler, “Flatland-RL: Multi-agent reinforcement learning on trains,” arXiv preprint , vol. arXiv:2012.05893, 2020

    work page arXiv 2012

  28. [28]

    The League of Robot Runners Com- petition: Goals, Designs, and Implementation,

    S.-H. Chan, Z. Chen, T. Guo, H. Zhang, Y . Zhang, D. Harabor, S. Koenig, C. Wu, and J. Yu, “The League of Robot Runners Com- petition: Goals, Designs, and Implementation,” in Proceedings of the 34th International Conference on Automated Planning and Scheduling – System Demonstrations Track , 2024

    work page 2024

  29. [29]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in Proceedings of the IEEE International Conference on Intelligent Robots and Systems , vol. 3, 2004, pp. 2149–2154

    work page 2004

  30. [30]

    ARGoS: A modular, parallel, multi-engine simulator for multi-robot systems,

    C. Pinciroli, V . Trianni, R. O’Grady, G. Pini, A. Brutschy, M. Brambilla, N. Mathews, E. Ferrante, G. Di Caro, F. Ducatelle et al. , “ARGoS: A modular, parallel, multi-engine simulator for multi-robot systems,” Swarm intelligence, vol. 6, pp. 271–295, 2012

    work page 2012

  31. [31]

    Isaac Sim,

    NVIDIA, “Isaac Sim,” https://developer.nvidia.com/isaac-sim, 2023, ac- cessed: 2025-07-10

    work page 2023

  32. [32]

    Multi-agent path finding with kinematic constraints,

    W. H ¨onig, T. K. S. Kumar, L. Cohen, H. Ma, H. Xu, N. Ayanian, and S. Koenig, “Multi-agent path finding with kinematic constraints,” in Proceedings of the International Conference on Automated Planning and Scheduling, vol. 26, 2016, pp. 477–485

    work page 2016

  33. [33]

    Efficient multi agent path finding with turn actions,

    Y . Zhang, D. Harabor, P. Le Bodic, and P. J. Stuckey, “Efficient multi agent path finding with turn actions,” in Proceedings of the International Symposium on Combinatorial Search, vol. 16, no. 1, 2023, pp. 119–127. 8 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

    work page 2023

  34. [34]

    Online plan repair in multi-robot coor- dination with disturbances,

    A. Coskun and J. M. O’Kane, “Online plan repair in multi-robot coor- dination with disturbances,” in Proceedings of the IEEE International Conference on Robotics and Automation , 2019, pp. 3333–3339

    work page 2019

  35. [35]

    Receding horizon re-ordering of multi-agent execution schedules,

    A. Berndt, N. Van Duijkeren, L. Palmieri, A. Kleiner, and T. Keviczky, “Receding horizon re-ordering of multi-agent execution schedules,” IEEE Transactions on Robotics , vol. 40, pp. 1356–1372, 2024

    work page 2024

  36. [36]

    A real-time rescheduling algorithm for multi-robot plan execution,

    Y . Feng, A. Paul, Z. Chen, and J. Li, “A real-time rescheduling algorithm for multi-robot plan execution,” in Proceedings of the International Conference on Automated Planning and Scheduling , vol. 34, 2024, pp. 201–209

    work page 2024