REVIEW 6 minor 46 references
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · glm-5.2
Fighting game stripped to rock-paper-scissors core becomes RL testbed
2026-07-08 03:05 UTC pith:36ENTQCS
load-bearing objection Solid benchmark paper with honest baselines and genuine research findings; deserves a serious referee.
FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central object is the neutral game of Footsies — the spacing-and-timing phase of a fighting match where neither player holds an advantage and both must commit to mixed strategies over movement and attacks that counter each other in cycles, like rock-paper-scissors. The paper shows that this structure can be isolated from the rest of the fighting-game genre (combos, execution contests, multiple characters) and packaged into a tractable benchmark that is simultaneously real-time, spatial, imperfect-information, and non-transitive. The baseline experiments then surface two concrete pathologies: entropy regularization and standard training methods drive hard-to-discover but strategically重要s如
What carries the argument
The environment's architecture rests on several load-bearing design choices: (1) the neutral-game isolation that removes combos to preserve only cyclic strategic interactions, (2) an action-delay parameter that controls whether attacks are reactable or must be predicted, effectively tuning the game from pure reaction to pure prediction, (3) an 85-dimensional observation vector with a privileged self-block that creates information asymmetry mirroring two players at separate controllers, (4) a vectorized C# headless simulator with gRPC ports achieving ~52,500 steps/second, and (5) approximate exploitability measured by training PPO best responses against fixed checkpoints, which serves as a (n
Load-bearing premise
The paper assumes that Footsies' neutral game — with a single character, four attacks, no combos, and no health bar — is representative enough of imperfect-information game dynamics to serve as a meaningful benchmark for the broader field, despite the authors' own acknowledgment that a substantial complexity gap separates it from commercial fighting games.
What would settle it
Train multiple algorithms on FootsiesGym and on existing benchmarks (e.g., Leduc poker, Slime Volleyball); if the algorithms produce qualitatively identical failure modes, learning dynamics, and relative rankings in both, the environment adds no distinguishing research value beyond what simpler benchmarks already provide.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces FootsiesGym, an open-source reinforcement learning environment built on HiFight's minimalist 2D fighting game Footsies. The environment is designed to capture the cyclic, non-transitive, imperfect-information dynamics of fighting-game neutral play while remaining tractable enough to train on a single workstation. The authors describe the game mechanics, observation space, action space, and reward structure; provide throughput measurements; and benchmark four baseline algorithms (two PPO variants, EMAgnet, and PFSP). They report approximate exploitability via trained best responses, identify a skill-exploitability disconnect, and highlight the difficulty of discovering special attacks through random exploration. The code is publicly available.
Significance. The paper fills a genuine gap between small matrix-game benchmarks and large real-time environments. The environment has verifiable imperfect-information structure (hidden opponent special-charge progress and dash availability; action-delay-induced prediction requirements), measured throughput (~52,500 steps/s at P=4, K=128), and publicly available code. The identification of concrete research challenges—special-attack discovery failure under standard regularization, and the skill-exploitability disconnect—adds scientific value beyond a pure software release. The authors are commendably transparent about limitations: baselines are explicitly labeled as naïvely tuned, and approximate exploitability is correctly described as a lower bound on true exploitability.
minor comments (6)
- §3, Observations: The text states the observation is 85-dimensional (88 with special-charge option), but Table 2 lists 1 (Common) + 2×18 (Shared) + 6 (Privileged, with |A|=6) = 43 dimensions, not 85. The discrepancy likely arises from additional features not listed or a different counting method; this should be clarified so users can correctly reproduce the observation vector.
- §4, Approximate Exploitability: The best-response PPO agents are trained for 2.5×10^7 steps, the same budget as the original policies. It would strengthen the approximate exploitability claims to briefly justify why this budget is sufficient for a meaningful best response, or to note that the lower bound may be loose if the best-response training is underpowered.
- Figure 5a: The head-to-head matrix uses expected return, but the reward scale depends on win_reward_scaling_coeff (default w=10). Stating the return scale or converting to win-rate would improve interpretability for readers unfamiliar with the reward structure.
- §C.3, Eq. (3): The PFSP priority weight uses exponent 1/2, while Vinyals et al. (2019) use this formulation in a league context. A brief note on how the single-policy reservoir affects the prioritization dynamics (e.g., reduced opponent diversity) would help readers assess the approach's fidelity to the original method.
- Table 6: The frame data lists four attacks, but the text in §3 mentions 'four attacks' without explicitly naming them in the main body. Adding a brief cross-reference to Table 6 in §3 would improve readability.
- §4, Discovering Special Attacks: The probability of B_SPECIAL under uniform random play is stated as 'roughly 0.001%'. A brief derivation (e.g., (1/6)^15 × conditional) would make this claim verifiable.
Simulated Author's Rebuttal
We thank the referee for their careful and accurate summary of the paper, and for the recommendation of minor revision. The referee's assessment aligns with our intended contributions: filling the gap between small matrix-game benchmarks and large real-time environments, providing verifiable imperfect-information structure, and identifying concrete research challenges (special-attack discovery failure and the skill-exploitability disconnect). We note that the 'MAJOR COMMENTS' section of the report appears to be empty, so we address the points raised in the summary and significance sections.
read point-by-point responses
-
Referee: The referee's summary and significance sections accurately characterize the paper's contributions, including the environment design, throughput measurements, baseline benchmarks, approximate exploitability analysis, and identification of research challenges. No specific major comments were listed.
Authors: We appreciate the referee's thorough and accurate reading of the manuscript. We confirm that the referee's summary correctly captures the paper's structure and contributions. Since no specific major comments were provided, we have reviewed the manuscript for any clarifications that would strengthen it in light of the referee's positive assessment. We will make the following minor revisions: (1) We will add a brief note in the Discussion section explicitly connecting the skill-exploitability disconnect to the broader literature on evaluation in imperfect-information games, making the research challenge more actionable for future work. (2) We will clarify in the Limitations paragraph that the approximate exploitability lower bound applies both to the absolute exploitability values and to the pairwise comparisons between algorithms, so readers do not over-interpret the statistical tests. (3) We will ensure the special-attack discovery challenge is framed consistently with the Appendix E results showing that the special-charge action mode alleviates but does not fully solve the problem. These are clarifications only; no structural changes are needed. revision: yes
Circularity Check
No significant circularity. Benchmark paper with no derivation or prediction to be circular about.
full rationale
This is a benchmark/environment paper, not a derivation or prediction paper. The central claim — that FootsiesGym is a tractable, open-source fighting-game environment capturing cyclic, non-transitive, imperfect-information dynamics — is supported by: (1) the external Footsies game by HiFight (2018) as the design basis, (2) direct throughput measurements (Figure 2), (3) frame-data analysis showing attack counter-relationships (Table 6, Figure 8), and (4) baseline experiments across four algorithms with honest caveats about naive tuning and approximate exploitability. The EMAgnet citation (Maidment et al., 2026) shares co-authors with this paper, but EMAgnet is used as one of four baseline algorithms, not as a load-bearing premise for the environment's value or design. The environment's strategic structure is grounded in the external game's mechanics, not in a self-cited theoretical result. No equation, prediction, or derivation reduces to its own inputs by construction. The approximate exploitability methodology (training best responses) is a standard approach cited to Timbers et al. (2020) and others, not a self-cited method. The skill-exploitability disconnect is presented as an empirical finding, not a derived prediction. No circularity patterns (self-definitional, fitted-input-as-prediction, self-citation load-bearing, uniqueness imported, ansatz smuggled, or renaming) are present.
Axiom & Free-Parameter Ledger
free parameters (4)
- win_reward_scaling_coeff (w) =
10.0
- action_delay =
8
- frame_skip =
4
- max_t =
4000
axioms (3)
- domain assumption Footsies' neutral game (without combos) preserves the essential cyclic, non-transitive strategic structure of fighting games.
- domain assumption Approximate exploitability via PPO-trained best responses is a meaningful proxy for true exploitability.
- domain assumption A 2-layer MLP with 256 hidden units is a sufficient function class for policies in Footsies.
read the original abstract
We present FootsiesGym, an open-source environment for learning in a non-trivial two-player, zero-sum, imperfect-information game. Built on HiFight's minimalist 2D fighting game Footsies, it isolates the cyclic, non-transitive strategic interactions of fighting game neutral play while remaining simple enough for efficient analysis. We provide a vectorized simulator that enables high-throughput training on standard hardware, making the environment accessible and reproducible. We describe the design of the environment, benchmark several reinforcement learning algorithms, and discuss open research directions it enables. The code is available at https://github.com/como-research/FootsiesGym.
Figures
Reference graph
Works this paper leans on
-
[1]
Outracing champion Gran Turismo drivers with deep reinforcement learning , author=. Nature , volume=. 2022 , publisher=
work page 2022
-
[2]
CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation
CoGrid & the Multi-User Gymnasium: A Framework for Multi-Agent Experimentation , author=. arXiv preprint arXiv:2604.15044 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[3]
Grandmaster level in StarCraft II using multi-agent reinforcement learning , author=. nature , volume=. 2019 , publisher=
work page 2019
-
[4]
Mastering the game of Stratego with model-free multiagent reinforcement learning , author=. Science , volume=. 2022 , publisher=
work page 2022
-
[5]
A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games , author=. arXiv preprint arXiv:2206.05825 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[6]
Advances in neural information processing systems , volume=
A unified game-theoretic approach to multiagent reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[7]
Reevaluating Policy Gradient Methods for Imperfect-Information Games
Reevaluating policy gradient methods for imperfect-information games , author=. arXiv preprint arXiv:2502.08938 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[8]
Proceedings of the AAAI conference on artificial intelligence , volume=
Mastering complex control in moba games with deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[9]
International conference on machine learning , pages=
RLlib: Abstractions for distributed reinforcement learning , author=. International conference on machine learning , pages=. 2018 , organization=
work page 2018
-
[10]
Advances in neural information processing systems , volume=
Unifying count-based exploration and intrinsic motivation , author=. Advances in neural information processing systems , volume=
-
[11]
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
Exponential moving average of weights in deep learning: Dynamics and benefits , author=. arXiv preprint arXiv:2411.18704 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games , author=. arXiv preprint arXiv:2606.23995 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[13]
International conference on machine learning , pages=
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures , author=. International conference on machine learning , pages=. 2018 , organization=
work page 2018
-
[14]
Goal-Conditioned Reinforcement Learning: Problems and Solutions
Goal-conditioned reinforcement learning: Problems and solutions , author=. arXiv preprint arXiv:2201.08299 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[15]
Approximate exploitability: Learning a best response in large games
Approximate exploitability: Learning a best response in large games , author=. arXiv preprint arXiv:2004.09677 , year=
work page internal anchor Pith review Pith/arXiv arXiv 2004
-
[16]
Advances in neural information processing systems , volume=
Collaborating with humans without human data , author=. Advances in neural information processing systems , volume=
-
[17]
“Other-play” for zero-shot coordination , author=. International conference on machine learning , pages=. 2020 , organization=
work page 2020
-
[18]
International Conference on Learning Representations , volume=
Overcookedv2: Rethinking overcooked for zero-shot coordination , author=. International Conference on Learning Representations , volume=
-
[19]
Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination
Any-play: An intrinsic augmentation for zero-shot coordination , author=. arXiv preprint arXiv:2201.12436 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[20]
Advances in neural information processing systems , volume=
\# exploration: A study of count-based exploration for deep reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[21]
Proximal Policy Optimization Algorithms
Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[22]
IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
Impact: Importance weighted asynchronous architectures with clipped target networks , author=. arXiv preprint arXiv:1912.00167 , year=
work page internal anchor Pith review Pith/arXiv arXiv 1912
-
[23]
International conference on machine learning , pages=
Count-based exploration with neural density models , author=. International conference on machine learning , pages=. 2017 , organization=
work page 2017
-
[24]
Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=
work page 2016
-
[25]
Controllable Complementarity: Subjective Preferences in Human-AI Collaboration
Controllable Complementarity: Subjective Preferences in Human-AI Collaboration , author=. arXiv preprint arXiv:2503.05455 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[26]
Neural Computation MIT-Press , year=
Long short-term memory , author=. Neural Computation MIT-Press , year=
-
[27]
Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
Deep reinforcement learning from self-play in imperfect-information games , author=. arXiv preprint arXiv:1603.01121 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[28]
Palmas, Alessandro , journal =
-
[29]
Unity: A General Platform for Intelligent Agents
Unity: A general platform for intelligent agents , author =. arXiv preprint arXiv:1809.02627 , year =
work page internal anchor Pith review Pith/arXiv arXiv
-
[30]
Comparison of Reaction Time Between eSports Players of Different Genres and Sportsmen , journal =
Bickmann, Peter and Wechsler, Konstantin and Rudolf, Kevin and Tholl, Chuck and Frob. Comparison of Reaction Time Between eSports Players of Different Genres and Sportsmen , journal =. 2021 , volume =. doi:10.4018/IJER.20210101.oa1 , url =
-
[31]
Terry, J. and Black, Benjamin and Grammel, Nathaniel and Jayakumar, Mario and Hari, Ananth and Sullivan, Ryan and Santos, Luis S. and Dieffendahl, Clemens and Horsch, Caroline and Perez-Vicente, Rodrigo and Williams, Niall L. and Lokesh, Yashas and Ravi, Praveen , journal =
-
[32]
OpenSpiel: A Framework for Reinforcement Learning in Games
Lanctot, Marc and Lockhart, Edward and Lespiau, Jean-Baptiste and Zambaldi, Vinicius and Upadhyay, Satyaki and P. arXiv preprint arXiv:1908.09453 , year =
work page internal anchor Pith review Pith/arXiv arXiv 1908
-
[33]
Slime Volleyball Gym Environment , author =. 2020 , howpublished =
work page 2020
- [34]
-
[35]
Proceedings of the AAAI Conference on Artificial Intelligence , volume =
Kurach, Karol and Raichuk, Anton and Sta. Proceedings of the AAAI Conference on Artificial Intelligence , volume =
-
[36]
EvoJAX: Hardware-Accelerated Neuroevolution
EvoJAX: Hardware-Accelerated Neuroevolution , author=. arXiv preprint arXiv:2202.05008 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[37]
Advances in neural information processing systems , volume=
Pipeline psro: A scalable approach for finding approximate nash equilibria in large games , author=. Advances in neural information processing systems , volume=
-
[38]
Suarez, Joseph and Du, Yilun and Isola, Phillip and Mordatch, Igor , journal =. Neural
-
[39]
Gotta Learn Fast: A New Benchmark for Generalization in
Nichol, Alex and Pfau, Vicki and Hesse, Christopher and Klimov, Oleg and Schulman, John , journal =. Gotta Learn Fast: A New Benchmark for Generalization in
-
[40]
2022 IEEE Conference on Games (CoG) , pages=
Darefightingice competition: A fighting game sound design and ai competition , author=. 2022 IEEE Conference on Games (CoG) , pages=. 2022 , organization=
work page 2022
-
[41]
2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE) , pages=
Fighting game artificial intelligence competition platform , author=. 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE) , pages=. 2013 , organization=
work page 2013
-
[42]
Proceedings of the 41st International Conference on Machine Learning , articleno =
Li, Wenzhe and Ding, Zihan and Karten, Seth and Jin, Chi , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =
work page 2024
-
[43]
Oh, Inseok and Rho, Seungeun and Moon, Sangbin and Son, Seongho and Lee, Hyoil and Chung, Jinyun , journal =. Creating Pro-Level. 2022 , publisher =
work page 2022
-
[44]
Firoiu, Vlad and Whitney, William F. and Tenenbaum, Joshua B. , journal =. Beating the World's Best at
-
[45]
IEEE Transactions on Games , volume=
Winning is not everything: Enhancing game development with intelligent agents , author=. IEEE Transactions on Games , volume=. 2020 , publisher=
work page 2020
-
[46]
2018 IEEE Conference on Computational Intelligence and Games (CIG) , pages =
Human-Like Playtesting with Deep Learning , author =. 2018 IEEE Conference on Computational Intelligence and Games (CIG) , pages =. 2018 , organization =
work page 2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.