pith. sign in

arxiv: 1907.00244 · v1 · pith:AWO4W3POnew · submitted 2019-06-29 · 💻 cs.AI

An Empirical Evaluation of Two General Game Systems: Ludii and RBG

\'Eric Piette , Matthew Stephenson , Dennis J. N. J. Soemers , Cameron Browne This is my paper

Pith reviewed 2026-05-25 12:47 UTC · model grok-4.3

classification 💻 cs.AI
keywords general game playingLudiiRBGGDLefficiencyreadabilitygame description languagesartificial intelligence
0
0 comments X

The pith

Ludii and RBG provide efficient alternatives to GDL for general game playing.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper conducts an experimental evaluation of the Ludii system and compares it directly to the Regular Boardgames language (RBG) on two properties: human-readability of game descriptions and computational efficiency. It positions both systems as recent alternatives to the older Game Description Language (GDL) used in general game playing research. A sympathetic reader cares because more readable and faster systems could reduce the effort needed to encode and run diverse games in AI experiments. The evaluation uses direct comparisons between the three systems to measure these properties. If the new systems perform better on the chosen metrics, they could support wider exploration of game-playing algorithms.

Core claim

Ludii and RBG emerged in 2019 as efficient alternatives to GDL, and the paper's experimental evaluation of Ludii focuses on direct comparison with RBG in terms of simplicity or clarity, measured by human-readability, and efficiency, measured by computational performance, to determine their relative strengths for general game playing.

What carries the argument

Empirical comparison of human-readability and computational efficiency metrics across Ludii, RBG, and GDL game descriptions.

If this is right

  • More games can be encoded and tested quickly because descriptions are easier for humans to read and modify.
  • AI experiments can run more game simulations per unit of time due to higher efficiency.
  • General game playing research can cover a wider range of games without specialized code for each one.
  • New game rules can be shared and reproduced more reliably across different research groups.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the efficiency gains hold, researchers might shift from GDL-based platforms to Ludii or RBG for large-scale experiments on game AI.
  • The readability focus could lower the entry barrier for non-experts to contribute new games to AI testbeds.
  • Similar evaluation methods could be applied to other emerging game description tools outside the three systems compared here.

Load-bearing premise

The chosen metrics for simplicity or clarity and efficiency accurately reflect the practical usefulness of the systems for AI research.

What would settle it

An experiment in which AI agents developed or trained with Ludii or RBG show no measurable gains in development speed, simulation rate, or solution quality compared with equivalent agents using GDL.

Figures

Figures reproduced from arXiv: 1907.00244 by Cameron Browne, Dennis J. N. J. Soemers, \'Eric Piette, Matthew Stephenson.

Figure 2
Figure 2. Figure 2: Game description of Amazons with Ludii. due to the inherent complexity of modelling games with first￾order logic, and the difficulty of integrating this language with other external applications. For this reason, a new General Game system based on Browne’s thesis [17] and the notion of ludemes, called Ludii [13] was implemented. Ludemes are the conceptual elements of a game. In Ludii, games are composed of… view at source ↗
read the original abstract

Although General Game Playing (GGP) systems can facilitate useful research in Artificial Intelligence (AI) for game-playing, they are often computationally inefficient and somewhat specialised to a specific class of games. However, since the start of this year, two General Game Systems have emerged that provide efficient alternatives to the academic state of the art -- the Game Description Language (GDL). In order of publication, these are the Regular Boardgames language (RBG), and the Ludii system. This paper offers an experimental evaluation of Ludii. Here, we focus mainly on a comparison between the two new systems in terms of two key properties for any GGP system: simplicity/clarity (e.g. human-readability), and efficiency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that Ludii and RBG provide efficient alternatives to the Game Description Language (GDL) for general game playing. It presents an experimental evaluation of Ludii, with the main focus on a head-to-head comparison of Ludii and RBG along two axes: simplicity/clarity (operationalized via human-readability) and efficiency.

Significance. If the measured differences in readability and runtime metrics are reproducible and the chosen proxies are valid, the work could help researchers select among modern GGP frameworks. The absence of any GDL reference implementation, however, leaves the headline efficiency claim unanchored.

major comments (2)
  1. [Abstract] Abstract: the assertion that Ludii and RBG 'provide efficient alternatives to ... GDL' is not empirically supported by the stated experimental scope ('mainly on a comparison between the two new systems'). No move-generation throughput, tree-expansion rate, or memory figures for equivalent rule sets under GDL are reported, so the relative-efficiency claim rests on an untested premise.
  2. [Methods/Results] Methods/Results (whichever section defines the efficiency metrics): without a GDL baseline the headline claim cannot be evaluated; the paper should either add such a baseline or revise the abstract and introduction to limit the claim to a Ludii-vs-RBG comparison.
minor comments (2)
  1. [Abstract] The abstract provides no concrete metrics, game corpus size, or statistical procedure; these details should appear in the abstract or be cross-referenced to a methods subsection.
  2. Clarify how 'human-readability' was quantified (e.g., expert ratings, parsing time, line count) and whether inter-rater reliability was assessed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We agree that the efficiency claims versus GDL lack direct empirical support in the experiments, which focus on Ludii versus RBG. We will revise the abstract and introduction to limit claims accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the assertion that Ludii and RBG 'provide efficient alternatives to ... GDL' is not empirically supported by the stated experimental scope ('mainly on a comparison between the two new systems'). No move-generation throughput, tree-expansion rate, or memory figures for equivalent rule sets under GDL are reported, so the relative-efficiency claim rests on an untested premise.

    Authors: We accept the point. The experiments reported do not include any GDL baseline measurements, so the headline claim that Ludii and RBG are efficient alternatives to GDL is not empirically grounded in this work. We will revise the abstract to remove the untested assertion about GDL and restrict the stated contribution to the Ludii-RBG comparison. revision: yes

  2. Referee: [Methods/Results] Methods/Results (whichever section defines the efficiency metrics): without a GDL baseline the headline claim cannot be evaluated; the paper should either add such a baseline or revise the abstract and introduction to limit the claim to a Ludii-vs-RBG comparison.

    Authors: We agree. Adding a GDL baseline would require implementing equivalent rule sets and running new experiments outside the current scope. We will instead revise the abstract and introduction to limit all efficiency claims to the direct Ludii-versus-RBG comparison that the experiments actually perform. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison with independent metrics

full rationale

This paper performs an empirical evaluation of two game systems on human-defined metrics of readability and runtime efficiency. No mathematical derivations, fitted parameters renamed as predictions, or self-citation chains appear in the load-bearing claims. All reported results derive from direct measurements on game instances rather than reducing to prior inputs by construction. The absence of a GDL baseline is a question of experimental scope, not circular reasoning.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the validity of the experimental design, which is not detailed in the abstract.

axioms (1)
  • domain assumption Standard assumptions in empirical evaluation of software systems, such as representative game selection.
    The paper assumes the tested games and metrics are representative.

pith-pipeline@v0.9.0 · 5657 in / 1114 out tokens · 51586 ms · 2026-05-25T12:47:02.795055+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 1 internal anchor

  1. [1]

    Sto ckfish: Strong open source chess engine,

    T. Romstad, M. Costalba, J. Kiiski, and G. Linscott, “Sto ckfish: Strong open source chess engine,” https://stockfishchess.org/, 2 008

    work page

  2. [2]

    Mastering the game of go with deep neural networks and tree s earch,

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. v an den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanc- tot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Suts kever, T. P . Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree s earch,” Nat...

    work page 2016

  3. [3]

    Realization of a general game-playing progr am,

    J. Pitrat, “Realization of a general game-playing progr am,” in IFIP Congress, 1968, pp. 1570–1574

    work page 1968

  4. [4]

    General game play ing: Overview of the AAAI competition,

    M. R. Genesereth, N. Love, and B. Pell, “General game play ing: Overview of the AAAI competition,” AI Magazine , vol. 26, no. 2, pp. 62–72, 2005

    work page 2005

  5. [5]

    General game playing: Game description language specifica tion,

    N. Love, T. Hinrichs, D. Haley, E. Schkufza, and M. Genese reth, “General game playing: Game description language specifica tion,” 2008

    work page 2008

  6. [6]

    Representing and reason ing about the rules of general games with imperfect information,

    S. Schiffel and M. Thielscher, “Representing and reason ing about the rules of general games with imperfect information,” Journal of Artificial Intelligence Research, vol. 49, pp. 171–206, 2014

    work page 2014

  7. [7]

    GDL-III: A description language for epi stemic general game playing,

    M. Thielscher, “GDL-III: A description language for epi stemic general game playing,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence , 2017, pp. 1276–1282

    work page 2017

  8. [8]

    General game playing,

    Y . Bj¨ ornsson and S. Schiffel, “General game playing,” i n Handbook of Digital Games and Entertainment Technologies . Springer, 2016, pp. 1–23

    work page 2016

  9. [9]

    Simulation-based appro ach to general game playing,

    H. Finnsson and Y . Bj¨ ornsson, “Simulation-based appro ach to general game playing,” in The Twenty-Third AAAI Conference on Artificial Intelligence, 2008, pp. 259–264

    work page 2008

  10. [10]

    Learning simulation control in general game-play ing agents,

    ——, “Learning simulation control in general game-play ing agents,” in The Twenty-F ourth AAAI Conference on Artificial Intelligence, 2010, pp. 954–959

    work page 2010

  11. [11]

    Constraint-based symmetry detection in general game playing,

    F. Koriche, S. Lagrue, ´E. Piette, and S. Tabary, “Constraint-based symmetry detection in general game playing,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp. 280–287

    work page 2017

  12. [12]

    Regular boardgames,

    J. Kowalski, M. Maksymilian, J. Sutowicz, and M. Szykul a, “Regular boardgames,” in The Thirty-Third AAAI Conference on Artificial Intel- ligence, 2019

    work page 2019

  13. [13]

    Ludii - the ludemic general game sys tem,

    ´E. Piette, D. J. N. J. Soemers, M. Stephenson, C. F. Sironi, M. H. M. Winands, and C. Browne, “Ludii - the ludemic general game sys tem,” CoRR, vol. abs/1905.05013, 2019

    work page arXiv 1905

  14. [14]

    Learning rules of simplified boardgame s by observing,

    Y . Bj¨ ornsson, “Learning rules of simplified boardgame s by observing,” in Proceedings of the Twentieth European Conference on Artific ial Intelligence, 2012, pp. 175–180

    work page 2012

  15. [15]

    Regular Boardgames

    J. Kowalski, M. Mika, J. Sutowicz, and M. Szykula, “Regu lar boardgames,” 2018. [Online]. Available: http://arxiv.or g/abs/1706.02462

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  16. [16]

    Modern techniques for ancient games,

    C. Browne, “Modern techniques for ancient games,” in 2018 IEEE Conference on Computational Intelligence and Games . IEEE, 2018, pp. 490–497

    work page 2018

  17. [17]

    Automatic generation and evaluation of r ecombination games,

    C. B. Browne, “Automatic generation and evaluation of r ecombination games,” Ph.D. dissertation, Queensland University of Tech nology, 2009. [Online]. Available: https://eprints.qut.edu.au/17025 /

    work page 2009

  18. [18]

    A class grammar for general games,

    ——, “A class grammar for general games,” in Advances in Computer Games, ser. LNCS, vol. 10068, 2016, pp. 167–182

    work page 2016