Deep Reinforcement Learning to Master the Asymmetric Strategy of Baghchal

Aarav Subedi; Aaryan Shakya; Manoj Shakya; Ranjit Raut; Sagun Rai

REVIEW 4 major objections 6 minor 15 references

Deep Reinforcement Learning to Master the Asymmetric Strategy of Baghchal

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MuZero, a model-based planner, outperforms three deep-RL algorithms on both roles in Baghchal.

desk verdict First deep RL comparison on Baghchal, but the headline win rates are arithmetically impossible and the evaluation protocol is undefined. read the letter →

arxiv 2607.18296 v1 pith:SARCLDAE submitted 2026-07-02 cs.AI cs.GT

Ranjit Raut , Aarav Subedi , Sagun Rai , Aaryan Shakya , Manoj Shakya This is my paper · ORCID

classification cs.AIcs.GT

keywords BaghchaldeepreinforcementlearningasymmetricgamesMuZeroMonteCarloTreeSearchPPOroleasymmetryzero-sumboard

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to establish that modern deep reinforcement learning can handle asymmetric two-player board games, where the two sides have different pieces, moves, and win conditions. Using the Nepali game Baghchal as a testbed, the authors train DQN, REINFORCE, PPO, and MuZero by self-play, letting each agent learn both the Tiger and Goat roles. They report that MuZero achieves the highest win rates in both roles (86% as Tiger, 62% as Goat) and attribute this to its learned model and Monte Carlo Tree Search planning. PPO is presented as the best cost-performance compromise, while DQN is shown to favor the reward-rich Tiger role. If true, the result would show that model-based planning is especially valuable in asymmetric games with sparse, long-horizon rewards.

What carries the argument

The load-bearing mechanism is MuZero's combination of a learned latent world model and Monte Carlo Tree Search (MCTS) guided by the PUCT rule, which lets the agent evaluate hypothetical futures without knowing the game rules. The authors also rely on a shared self-play training protocol for both roles, a custom PyTorch Baghchal environment with legal-move masking, and a reward table that gives the Tiger immediate capture rewards while the Goat receives sparse positional rewards.

What would settle it

Take the trained MuZero agent and play it 100 games as Tiger and 100 games as Goat against the same fixed baseline (e.g., the best PPO agent or a scripted rule-based player), recording wins, losses, and draws. If the two per-role win rates plus the draw rate do not sum to roughly 100%, the headline ranking is not measuring what it claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that MuZero, which plans up to 8–10 moves ahead through a learned latent model, learns the strongest play on both sides of Baghchal, exceeding the model-free baselines DQN, REINFORCE, and PPO. The reported numbers are an 86% win rate as the Tiger and 62% as the Goat, with a 9% draw rate. The authors attribute this to model-based planning via Monte Carlo Tree Search, and further claim that PPO offers a competitive alternative at a fraction of the computational cost, and that value-based DQN develops a strong bias toward the Tiger role because capture rewards are dense and immediate.

Load-bearing premise

The evaluation protocol yields comparable per-role win rates; the paper never states who the opponent is during testing, and the reported Tiger, Goat, and draw rates (86%, 62%, 9%) cannot all hold against a single fixed opponent in a zero-sum game.

Editorial extensions

If this is right

If MuZero's margin over model-free methods is real, it implies that long-horizon planning is a decisive factor in asymmetric board games, where one side faces delayed and sparse rewards.
PPO's near-MuZero performance at roughly one quarter of the training time and far lower memory suggests that actor-critic methods are the pragmatic choice for resource-constrained settings.
The consistent Tiger advantage across all algorithms indicates that role asymmetry itself, not algorithm choice, drives much of the difficulty; the Goat's coordination problem is the harder learning task.
Baghchal can serve as a compact testbed for asymmetric multi-agent RL, complementing standard symmetric benchmarks like chess and Go.
The observed emergent strategies—pincer attacks, sacrifice, and tempo control for the Tiger; mobile blocking formations for the Goat—suggest that deep RL can rediscover human-style play in a culturally specific game.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reported per-role win rates were measured against the same fixed opponent, they would be mutually inconsistent in a zero-sum game (86% Tiger + 62% Goat + 9% draw exceeds 100%), so the reader should ask what opponent each win rate is measured against before accepting the ranking.
A direct extension would be to train a single MuZero agent and test it against a fixed, human- or scripted-baseline opponent on both roles, yielding interpretable per-role numbers; the paper's self-play protocol leaves this unspecified.
The authors' own conclusion notes the agents do not model the opponent's reasoning and only learn via self-play; an interesting follow-up would be to test whether opponent-aware methods (e.g., MADDPG) improve Goat coordination, as the paper itself suggests.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something genuinely new: it applies four deep RL families (DQN, REINFORCE, PPO, MuZero) to Baghchal, an asymmetric Nepali board game that the deep RL literature has mostly ignored. The environment design, the two-phase game structure, and the reward tables are clearly described, and the computational cost comparison (training time, memory, inference) is more than most RL papers bother to report. If the results held, this would be a useful empirical benchmark for asymmetric multi-agent learning.

But the central result does not hold. Table II reports, for MuZero, an 86% Tiger win rate, a 62% Goat win rate, and a 9% draw rate. In a two-player zero-sum game these three numbers must sum to 100% if they come from the same set of matches. 86 + 62 + 9 = 157. The same problem appears for every algorithm: DQN sums to 126%, REINFORCE to 126%, PPO to 145%. This is not a rounding issue; it means the evaluation protocol is either badly misreported or not defined at all. The paper never states who the opponent is during evaluation, how many games were played, or whether the Tiger and Goat win rates are measured against the same fixed baseline or against self-play. The methodology section says the agents train by self-play, but the reported numbers cannot come from self-play unless the two win rates are complements, which they are not.

Because of this, the headline claim that MuZero is best in both roles is uninterpretable. A reader cannot tell whether MuZero is beating a weak fixed opponent, whether the win rates come from different evaluation settings, or whether the table is simply wrong. The paper's secondary claims—PPO as a practical compromise, DQN's tiger bias—are also built on this broken table. The strategic behavior analysis is mostly anecdotal, with no systematic metrics or ablations to back up statements about planning depth.

Credit where due: the gap in the literature is real, the game choice is culturally significant, and the authors have made a careful attempt at environment engineering. The writing is rough in places, but that is minor compared to the evaluation flaw.

This paper is not ready for peer review in its current form. The authors need to specify the evaluation protocol precisely, provide the code and data, and re-report the win rates so they are internally consistent. If they do that, the comparison could be a useful contribution. As it stands, a serious referee would have to reject it, and a desk reject is defensible.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical comparison of four deep reinforcement learning algorithms—DQN, REINFORCE, PPO, and MuZero—applied to Baghchal, a two-player zero-sum asymmetric board game. The agents are trained via self-play and evaluated on both roles. The reported results claim that MuZero achieves the highest win rates in both roles (86% as Tiger, 62% as Goat), that PPO offers the best performance-to-computational-cost tradeoff, and that model-based planning via MCTS explains MuZero's superiority. Additional analyses cover training convergence, computational cost, and qualitative strategic behavior.

Significance. If its central claims were valid, the paper would contribute a useful benchmark for deep RL in asymmetric two-player games, an underexplored area; it also documents computational-resource comparisons and highlights a culturally significant game. The paper is one of the first to apply modern deep RL algorithms (including MuZero) to Baghchal, and the inclusion of training-time and inference-cost measurements is a strength. However, the headline quantitative result is undermined by an internally inconsistent table and an underspecified evaluation protocol, so the main claims cannot be accepted as presented.

major comments (4)

[Section IV.A, Table II] The reported win rates violate the zero-sum arithmetic of Baghchal. Under any single evaluation setting, Tiger win rate + Goat win rate + draw rate must equal 100%. The table reports sums of 126% (DQN), 126% (REINFORCE), 145% (PPO), and 157% (MuZero). The MuZero row alone—86% Tiger, 62% Goat, 9% draw—is arithmetically impossible if the two role win rates come from the same matches. The paper never states whether the role win rates are measured against the same opponent, different opponents, or different evaluation protocols. Consequently the central claim in the abstract and Section IV.A that “MuZero achieves the highest performance in both roles” is unsupported by the data as reported.
[Section III.D and Section IV.A] The evaluation protocol is not defined. The paper says each agent is trained by self-play over 100,000 games and that performance is assessed every 1000 games, but it does not state who the final trained agent is evaluated against, how many evaluation games were played, how draws are scored, how the per-role win rates are combined, or how the reported ± intervals are computed across seeds. Without this information the results in Tables II–IV are not reproducible, and the cross-algorithm ranking cannot be verified.
[Abstract and Section IV.A] The attribution of MuZero's performance to “model-based planning via Monte Carlo Tree Search” is not supported by a controlled comparison. The algorithms differ in many respects beyond the presence of a learned model and MCTS, including network architecture, target networks, GAE, entropy regularization, replay mechanism, and compute budget. No ablation isolates MCTS or the learned model. The claim in Section III.A.4 that MuZero can “look 8-10 moves ahead” is asserted without any measured search-depth or tree-statistics data; it appears to be an interpretation rather than a measured result.
[Table I] The hand-designed reward structure is role-asymmetric (e.g., +10 capture reward for Tiger, +5 survival and +10 strategic-positioning rewards for Goat, −1 per-move penalty for Tiger). Since all algorithms are optimized under these rewards, the reward schedule itself may create the apparent differences in role difficulty and algorithmic preference. The paper reports no sensitivity analysis or reward-calibration checks, so the conclusion in Section IV.E that value-based methods are “biased toward the Tiger role” is confounded by the reward design.

minor comments (6)

[Section III.A] The displayed equations are garbled: symbols such as “�” replace mathematical notation, and Eq. (2) uses undefined notation ($G_t$, $b$, etc. are partially illegible). This makes the algorithm descriptions hard to follow.
[Section III.D] The training description says agents are trained for 100,000 self-play games, while Table III reports convergence at 25,000–45,000 episodes. The relationship between these numbers is unclear; please clarify whether 100,000 is the maximum training budget or the total number of episodes.
[Abstract and Section III] The abstract states that algorithms are “trained on one side ... and evaluated on the other side,” while Section III says each program learns by playing against itself and is tested as both Tiger and Goat. These statements are inconsistent and should be reconciled.
[Figures] The manuscript references several figures (Fig. 1, 3, 4, 5, 6) but the provided text does not include the images. Ensure that all figures are embedded and legible in the submission.
[Section V] The conclusion includes claims not derived from the reported experiments, such as the agent “doesn't really work out what the other player is doing” and “doesn't get harder ... depending on the human player.” These are limitations of the current system but are not measured or discussed in the results section.
[Section II] The related-work discussion mentions prior Baghchal work by Luitel et al. and Thapa & Poudel, but no quantitative comparison with their reported results is given. Adding such a comparison would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical benchmark whose conclusions are not baked into its inputs by construction.

full rationale

The paper reports an empirical comparison of four deep RL algorithms on Baghchal, with win rates, draw rates, and training metrics. There is no derivation in which a claimed prediction is equivalent to a fitted parameter, and no load-bearing step reduces to a self-citation. The cited prior work (Luitel et al., Thapa and Poudel) is external to the current authors and is used only as motivation, not to justify the paper's conclusions. The central claim that MuZero performs best is an observational ranking of trained agents, not a theorem derived from definitions. The reward structure is hand-designed, but that is a modeling choice rather than circular reasoning. The serious arithmetic inconsistency in Table II—where role win rates plus draw rates sum to more than 100% for a zero-sum game—is a correctness, reproducibility, and reporting flaw, not a circularity. It undermines the soundness of the benchmark but does not indicate that the conclusions were equivalent to the inputs by construction. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or hypothesized mechanisms. Its empirical claims rest on a custom game environment whose correctness is assumed, on hand-designed reward constants, on unreported hyperparameters, and on an unspecified evaluation protocol. These unverified inputs, not any novel formalism, carry the central result.

free parameters (3)

Reward shaping weights = unknown (capture +10, win +100, survive +5, strategic positioning +10, being captured -20, per-move -1, illegal -50)
The reward values in Table I are hand-chosen to shape learning. 'Strategic positioning +10' is not formally defined, so the Goat agent's reward signal is underspecified and could significantly affect win rates.
Training hyperparameters = not reported
Section III.D says a variety of learning rates and batch sizes were tried and cross-validation was used, but the final selected values are not listed. DQN target-network update interval (500 steps), PPO epsilon (0.2) and GAE lambda (0.95) are some exceptions, but the full hyperparameter grid is absent.
MuZero MCTS parameters = not reported
The number of MCTS simulations, PUCT constant c, unroll steps K, and priority replay settings are mentioned only qualitatively. These directly affect planning depth and the reported 8-10 move lookahead and inference time, so they are load-bearing for the central comparison.

assumptions (3)

domain assumption The custom PyTorch environment faithfully implements the rules of Baghchal
All results depend on the game engine's correctness, but the paper provides no test suite, oracle, or independent verification of rule enforcement. (Section III.B)
domain assumption Self-play training produces a policy that can be meaningfully evaluated on both playable roles
The paper says each program learns by playing against itself, giving equal practice to both roles. But in a zero-sum game, a single self-play policy cannot achieve both 86% tiger and 62% goat win rates against the same opponent, so the evaluation object is ambiguous. (Section III and Table II)
domain assumption The opponent used for evaluation is fixed and representative
Win rates in Table II are meaningless without specifying whether the opponent is a random player, a frozen checkpoint, the same algorithm, or another algorithm. The paper never states this, so the self-play fairness argument fails. (Section IV.A)

how reviews work

0 comments

Cite this review

Pith. "Pith review of Deep Reinforcement Learning to Master the Asymmetric Strategy of Baghchal." pith.science (2026). https://pith.science/paper/SARCLDAE

@misc{pith2026260718296,
  author       = {Pith},
  title        = {Pith review of: Deep Reinforcement Learning to Master the Asymmetric Strategy of Baghchal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SARCLDAE}},
  note         = {Machine review of arXiv:2607.18296}
}

read the original abstract

Baghchal is a two-player asymmetric board game with Nepali origins where four tigers are to capture goats and twenty goats desire to keep tigers in immobility. Although Baghchal has a complex structure which is strategic, has perfect information structure, and has cultural meaning, it has not been adequately covered in deep reinforcement learning (RL) literature. This paper gives a systematic exploration of four deep RL solutions Deep Q-Network (DQN), REINFORCE, Proximal Policy Optimization (PPO) and MuZero that are trained on one side of the asymmetric gameplay of Baghchal and then evaluated on the other side. The algorithms are rated based on win rate, draw rate, average captures, training convergence and computational cost. It is experimentally found that MuZero generates the best performance in both tasks, achieving 86 percent win over these Tiger and 62 percent win over these Goat and the ability to do so is due to the model-based planning machine through the Monte Carlo Tree Search. PPO is the most realistic algorithm and is provided to be competitive over both asymmetric tasks with significantly reduced computational costs compared to MuZero. Emergent strategic behavior analysis shows that model-based strategies are optimal over long-horizon planning, whereas value-based counterparts like DQN are more biased up towards the Tiger role owing to the more substantial reward signal.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 4 linked inside Pith

[1]

Baghchal: An augmented Q-learning approach to turn-based heterogeneous multi- agent systems,

Y. Luitel, A. Koirala, A. Shrestha, and A. S. Poudel, "Baghchal: An augmented Q-learning approach to turn-based heterogeneous multi- agent systems," SSRN, 2024. [Online]. Available: https://ssrn.com/abstract=4683084

2024
[2]

Human-level control through deep reinforcement learning,

V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540,pp.529–533,2015

2015
[3]

Simple statistical gradient-following algorithms for connectionist reinforcement learning,

R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Machine Learning, vol. 8, no. 3–4,pp.229–256,1992

1992
[4]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347,2017

arXiv 2017
[5]

MasteringAtari,Go,chessandshogibyplanningwith alearnedmodel,

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, andD.Silver,"MasteringAtari,Go,chessandshogibyplanningwith alearnedmodel,"Nature,vol.588,no.7839,pp.604–609,2020

2020
[6]

Mastering the game of Go with deep neural networks and treesearch,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever,T.Lillicrap,M.Leach,K.Kavukcuoglu,T.Graepel,andD. Hassabis, "Mastering the game of Go with deep neural networks and treesearch,"Nature,vol.529,no.7587,pp...

2016
[7]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction,2nded.Cambridge,MA,USA:MITPress,2018

2018
[8]

Superhuman AI for multiplayer poker,

N. Brown and T. Sandholm, "Superhuman AI for multiplayer poker," Science,vol.365,no.6456,pp.885–890,2019

2019

Show all 15 references

[9]

Multiagent deep reinforcement learning with extremelysparserewards,

A.Tampuu,T.Matiisen,D.Kodelja,I.Kuzovkin,K.Korjus,J.Aru,J. Aru, and R. Vicente, "Multiagent deep reinforcement learning with extremelysparserewards,"arXivpreprintarXiv:1707.01495,2017

2017 arXiv
[10]

Counterfactual multi-agent policy gradients,

J. Foerster, I. A. Assael, N. de Freitas, and S. Whiteson, "Counterfactual multi-agent policy gradients," in Proc. AAAI Conf. ArtificialIntelligence,vol.30,no.1,2016

2016
[11]

A unified game-theoretic approach to multiagent reinforcement learning,

M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Perolat, D. Silver, and T. Graepel, "A unified game-theoretic approach to multiagent reinforcement learning," Adv. Neural Inf. Process.Syst.,vol.30,2017

2017
[12]

Dealing with non-stationarity in multi-agent deep reinforcement learning,

G. Papoudakis, F. Christianos, A. Rahman, and S. V. Albrecht, "Dealing with non-stationarity in multi-agent deep reinforcement learning,"arXivpreprintarXiv:1906.11374,2019

1906 arXiv
[13]

Deep Blue,

M. Campbell, A. J. Hoane Jr., and F. H. Hsu, "Deep Blue," Artificial Intelligence,vol.134,no.1–2,pp.57–83,2002

2002
[14]

Dota 2 with large scale deep reinforcementlearning,

C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S.Gray,C.Olsson,J.Pachocki,M.Petrov,H.P.deOlivaresPinto,J. Raiman,T.Salimans,J.Schlatter,J. Schneider,S.Sidor,I.Sutskever, J. Tang, F. Wolski, and S. Z...

1912 arXiv
[15]

AIstrategyapproachdevelopment on Baghchal using AlphaZero,

R.B.Thapa andL. K.Poudel, "AIstrategyapproachdevelopment on Baghchal using AlphaZero," J. Sci. Techn., vol. 4, no. 2, pp. 35–39, Dec.2024

2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Baghchal: An augmented Q-learning approach to turn-based heterogeneous multi- agent systems,

Y. Luitel, A. Koirala, A. Shrestha, and A. S. Poudel, "Baghchal: An augmented Q-learning approach to turn-based heterogeneous multi- agent systems," SSRN, 2024. [Online]. Available: https://ssrn.com/abstract=4683084

2024

[2] [2]

Human-level control through deep reinforcement learning,

V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540,pp.529–533,2015

2015

[3] [3]

Simple statistical gradient-following algorithms for connectionist reinforcement learning,

R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Machine Learning, vol. 8, no. 3–4,pp.229–256,1992

1992

[4] [4]

Proximal policy optimization algorithms,

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347,2017

arXiv 2017

[5] [5]

MasteringAtari,Go,chessandshogibyplanningwith alearnedmodel,

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, andD.Silver,"MasteringAtari,Go,chessandshogibyplanningwith alearnedmodel,"Nature,vol.588,no.7839,pp.604–609,2020

2020

[6] [6]

Mastering the game of Go with deep neural networks and treesearch,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever,T.Lillicrap,M.Leach,K.Kavukcuoglu,T.Graepel,andD. Hassabis, "Mastering the game of Go with deep neural networks and treesearch,"Nature,vol.529,no.7587,pp...

2016

[7] [7]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction,2nded.Cambridge,MA,USA:MITPress,2018

2018

[8] [8]

Superhuman AI for multiplayer poker,

N. Brown and T. Sandholm, "Superhuman AI for multiplayer poker," Science,vol.365,no.6456,pp.885–890,2019

2019