Pith. sign in

REVIEW 4 major objections 6 minor 7 references

Exploring the Performance of Deep Residual Networks in Crazyhouse Chess

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A ResNet trained on Stockfish self-play reaches master level in Crazyhouse chess on ordinary hardware.

desk verdict A solid engineering report with a real Lichess measurement, but the 'master' claim overreaches; the evidence supports strong amateur. read the letter →

arxiv 1908.09296 v1 pith:QRBJEKYN submitted 2019-08-25 cs.LG cs.AIstat.ML

Sun-Yu Gordon Chi This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords CrazyhousedeepresidualnetworkMonteCarloTreeSearchpolicyvalueneuralchessengineself-playtrainingStockfish
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the creation of SixtyFour, a Crazyhouse chess engine built around a deep residual network rather than a hand-crafted evaluation function, which works with a Monte Carlo Tree Search and can run on limited, commodity CPU hardware. The central claim is that a supervised policy network trained to imitate Stockfish's self-play moves, combined with a simple tree search, reaches a rating of 2278 on Lichess, a level the author compares to a master and higher than 97% of active Crazyhouse players. If true, the work shows that a moderately sized residual network, without hand-crafted features or massive computational resources, can master a highly tactical chess variant with a branching factor far larger than regular chess. It also suggests that the careful representation of a complex state space and the scaling of training labels by game outcome are sufficient to build a competitive engine. The paper matters because it extends the deep-learning approach to board games beyond regular chess and Go, into a variant that has traditionally been dominated by alpha-beta engines with custom evaluation functions.

What carries the argument

The central machinery is a residual convolutional neural network (ResNet) that maps a 960-channel raw board representation to a policy vector of length 2308 and a value score, guided by a Monte Carlo Tree Search (MCTS) that uses a modified PUCT formula with an exploration noise term decaying over the game. The board representation encodes piece positions, promoted-piece status, pocket contents (captured pieces available for dropping), castling rights, repetition information, and side to move, so the network can reason about piece drops. The MCTS values checkmates at 1.5 instead of 1.0 to encourage mating lines, and the search depth adapts to the remaining clock time, ranging from 4 to 20 plies per playout. The training labels are outcome-scaled so that moves from losing games are down-weighted, which the author says prevents the network from imitating blunders in a game where one mistake can undo many accurate moves.

What would settle it

Play a time-limited match between SixtyFour and the multi-variant Stockfish at a fixed high strength (e.g., depth 20 or 2000 nodes per move) in a controlled setting with the same time controls as the paper's online games; if SixtyFour's win rate against such Stockfish is not clearly above 50%, the claim that it has reached master-level play is falsified, since the training data itself comes from stronger Stockfish searches. A second direct check: run the ResNet with MCTS disabled and compare its self-play accuracy against the reported 65.02-ply average game length and ACPL values; if the network cannot reproduce such low centipawn losses on unseen positions, the outcome-scaling of labels may not be as effective as claimed.

Watch

Extended reading notes

Core claim

SixtyFour is a neural-network based Crazyhouse engine that uses a 12-block residual network with 256 kernels per block to output a policy over 2308 legal moves and a scalar value, trained on roughly 11.2 million moves from 134,386 Stockfish self-play games. The paper's core discovery is that such a ResNet, trained with a scaled one-hot policy label (1.0 for winning moves, 0.1 for losing moves, 0.5 for draws) and a value head on game result, can achieve a policy prediction accuracy of about 54.1 percent on Stockfish's own validation moves—comparable to AlphaGo's initial supervised policy network—and that when combined with a depth-limited Monte Carlo Tree Search, it plays at a level rated 2278 on Lichess, higher than 97% of all Crazyhouse users. The engine's self-play games show low average centipawn loss and frequent mating combinations, and its games against Stockfish AI levels 4 and 5 online indicate that it can defeat moderately strong Stockfish settings. The author argues that the raw board representation, the explicit use of promoted-piece and pocket information, and the outcome-scaled policy labels are what allow the network to grasp the abstract tactical concepts of Crazyhouse without any hand-crafted evaluation features.

Load-bearing premise

The training labels from Stockfish self-play at up to one million nodes per position are assumed to be accurate enough that a network imitating them learns high-level Crazyhouse play; if Stockfish's search is too shallow or its opening distribution too narrow, the imitated policy and value would cap the engine's strength regardless of architecture or search.

Editorial extensions

If this is right

  • If the reported Lichess rating of 2278 and the games against Stockfish are representative, a moderate-sized ResNet combined with a shallow MCTS can produce a competitive Crazyhouse engine on single-CPU hardware, offering a practical alternative to alpha-beta engines that require heavily tuned evaluation functions.
  • The outcome-scaled policy labeling (1.0/0.1/0.5) can be tested directly in other games with near-zero draw rates and high tactical volatility, such as Shogi or bughouse, to see whether down-weighting losing moves improves final engine strength.
  • The specific move representation—a compact 2308-entry one-hot vector with separate planes for drops, linear moves, knight moves, and underpromotions—provides a template for encoding high-branching-factor games efficiently without wasting capacity on illegal moves.
  • The paper's approach of training on Stockfish self-play openings from the Lichess Opening Explorer suggests a data-augmentation recipe for other rare variants: use a strong engine with vast search to generate millions of positions, then distill the engine's behavior into a fast network for online play.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper implicitly claims that the quality ceiling is set by Stockfish's own play rather than by the deep network: since the training data is generated by Stockfish at up to one million nodes per position, a network that perfectly imitated that data would at best match that engine's strength, so the 2278 rating is likely below Stockfish's ceiling, meaning the ResNet's practical value is as a fas
  • If the 54.1% top-1 policy accuracy is compared to the branching factor of Crazyhouse (roughly three times chess), then random policy would give near-zero accuracy, so 54.1% indicates that the network has learned a highly concentrated distribution over sensible moves—even so, a player at the 97th percentile of Lichess users is not yet at the level of the strongest human Crazyhouse specialists, and
  • The ensemble model mentioned in the abstract is not described in detail in the paper itself, so a reasonable editorial reading is that the author intended to distribute training over commodity CPUs by training multiple smaller networks separately—testing whether such an ensemble closes the gap to Stockfish would directly probe the scalability claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes SixtyFour, a deep residual-network engine for the chess variant Crazyhouse. The network has a policy head over a sparsely encoded 2308-move representation and a value head, and is trained by supervised learning on 134,386 Stockfish self-play games (about 11.2 million positions), then combined with a Monte Carlo Tree Search variant. The authors report a validation policy accuracy of 54.1%, several self-play games, ten games against humans and against Lichess Stockfish AI Levels 4 and 5, and a peak Lichess rating of 2278 from 294 rated games. The abstract claims a playing level comparable to a strong amateur, while the conclusion states that the engine 'has reached a level comparable to a master in the game of Crazyhouse.'

Significance. If the claims were fully supported, this would be a useful demonstration that a moderately sized ResNet trained by supervised imitation on Stockfish self-play, together with MCTS, can reach a strong human amateur level in Crazyhouse on commodity hardware. The move-output representation, the pocket encoding, and the reproducibility details (source code, architecture, training hyperparameters) are concrete strengths, and the Lichess rating is a genuine external measurement. However, the significance is dampened by the gap between the abstract's 'strong amateur' claim and the conclusion's 'master' claim, by the absence of strong engine baselines, and by the circularity of the policy-accuracy comparison. The paper's strongest supported conclusion is an amateur-level engine, not a master-level one.

major comments (4)
  1. [Section 5 and Section 4.3] The conclusion's claim that the engine 'has reached a level comparable to a master in the game of Crazyhouse' is not supported by the evidence. A peak rating of 2278 from 294 rated games is the maximum of a noisy rating trajectory, and the section does not provide opponent ratings, rating intervals, or a controlled match against a known master-strength player or engine. The statement that 2278 is 'higher than 97% of all Crazyhouse users' describes the Lichess rating distribution, not an absolute master-level standard. The abstract's 'strong amateur' wording is the strongest claim the reported evidence supports; the master claim should be removed or replaced by a controlled measurement.
  2. [Sections 4.2 and 4.3] No controlled baseline against a strong Crazyhouse engine is reported. The games against Stockfish are against Lichess 'AI Level 4' and 'AI Level 5', which are reduced-strength handicap levels, and no comparison is made against full-strength multi-variant Stockfish, Sunsetter, Sjeng, or CrazyAra under matched time controls. In addition, the self-play games in Section 4.2 are explicitly described as selected 'most accurate games,' so they cannot serve as representative evidence of overall playing strength.
  3. [Section 4.1] The reported policy accuracy of 54.1% is measured against a validation set drawn from the same Stockfish self-play distribution used for training, so it is an imitation score rather than an independent measure of playing strength. The comparison to AlphaGo's 55.4% is misleading because AlphaGo's accuracy was reported against held-out professional human games, not against the engine's own training distribution. This circularity does not invalidate the external Lichess rating, but the policy-accuracy comparison should be removed or reframed as an imitation-quality metric.
  4. [Abstract] The abstract states that the paper 'evaluate[s] the effectiveness of an ensemble model,' but no ensemble architecture, training procedure, or empirical ensemble results appear anywhere in the body. Either add the promised ensemble comparison or remove the ensemble claim from the abstract and introduction.
minor comments (6)
  1. [Equation (1), Section 3.5] The MCTS score equation is under-specified: Nnode is defined but the formula uses only Nparent and nnode, and Wnode is described as a count of playout wins but appears as Wnode/2 without being normalized by visit count. Please define every symbol and give the exact update rule.
  2. [Figure 3 caption] The caption says that Black checkmates 'the Black king,' but the checkmated king is White's; it also refers to 'B@g3' and 'Qh3!!' without giving the move numbers, which makes the discussion hard to follow.
  3. [Section 4.3] The time control '0.5m+0s' is not standard notation; please spell out the exact clock setting (for example, 30 seconds with zero increment). The section also does not give the Lichess ratings of the human opponents in the showcased games.
  4. [Section 2.2] The paper calls the neural-network engine a novel attempt while simultaneously citing the concurrent CrazyAra project; please clarify the novelty claim relative to CrazyAra and cite the published work on CrazyAra rather than only a GitHub link.
  5. [Section 5] The phrase 'top 97% percentile' should be '97th percentile,' and the sentence 'at around the same time of this project' in Section 2.2 is awkward and should be rephrased.
  6. [Section 5.2] The statement about 'wins against humans without a search algorithm' is ambiguous, because Section 4.3 reports games of the full engine with tree search; please clarify whether the raw policy network has verified wins against humans and where those games are shown.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central strength claim rests on external rated games, not on the training labels.

full rationale

The paper's derivation chain is supervised imitation of Stockfish self-play plus Monte Carlo Tree Search, and its headline strength evidence is a peak Lichess rating of 2278 from 294 rated games against humans, reported in Section 4.3. That is an external benchmark outside the fitted training distribution, so the master-level conclusion is not forced by construction. The Section 4.1 policy accuracy of 54.1% is measured on a Stockfish validation set; although this only demonstrates imitation of the training distribution rather than independent playing strength, the paper does not use that number as the basis of its central claim—the conclusion explicitly cites the online rating. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed empirical patterns. Remaining concerns such as peak rating versus stable strength and the absence of controlled baselines against known engines or titled players are measurement and interpretation issues, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The engine's strength rests on Stockfish-generated training labels, a chosen move encoding, and online rating as a strength measure. The free parameters are manually chosen constants; no new entities are postulated.

free parameters (5)
  • Policy outcome scaling factors = win=1.0, draw=0.5, loss=0.1
    Hand-chosen in Section 3.2.2 to downweight moves from losing games.
  • MCTS exploration constant gamma = 0.5
    Arbitrary exploration constant set in Section 3.5.
  • Noise constants (alpha, beta) = alpha=0.3, beta=6
    Tuned in Section 4.2.1 for self-play; no ablation is reported.
  • Time-management search depth schedule = 20/15/12/10/4/0 for clock bins >=750, 300-750, 120-300, 60-120, 10-60, <=10 seconds
    Hand-chosen schedule in Section 3.6, not validated against alternatives.
  • Network architecture and training hyperparameters = 12 residual blocks, 256 3x3 kernels, learning rate 0.001, 8 epochs
    Chosen in Sections 3.4 and 3.4.1; larger networks overfit, but no systematic search is reported.
assumptions (5)
  • domain assumption Stockfish multi-variant self-play at up to one million nodes per position produces high-quality move labels and game outcomes.
    Entered in Section 3.3; the entire supervised target depends on this.
  • domain assumption The 2308-dimensional move encoding covers every legal Crazyhouse move.
    Entered in Section 3.2.1; if a move class is omitted, the policy cannot express it.
  • domain assumption Lichess rating over 294 rated games is a valid measure of playing strength and can be called 'master level'.
    Entered in Sections 4.3 and 5; no opponent strength distribution or rating uncertainty is reported.
  • domain assumption Average centipawn loss (ACPL) is a reliable measure of move quality in Crazyhouse.
    Used in Section 4.2.1 to call self-play games 'near perfect' without an external evaluation.
  • domain assumption Crazyhouse has a near-zero draw rate, making win/loss value targets sufficient.
    Stated in Section 1.2; it justifies the value representation in Section 3.2.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Performance of Deep Residual Networks in Crazyhouse Chess." pith.science (2026). https://pith.science/paper/QRBJEKYN

@misc{pith2026190809296,
  author       = {Pith},
  title        = {Pith review of: Exploring the Performance of Deep Residual Networks in Crazyhouse Chess},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRBJEKYN}},
  note         = {Machine review of arXiv:1908.09296}
}
read the original abstract

Crazyhouse is a chess variant that incorporates all of the classical chess rules, but allows users to drop pieces captured from the opponent as a normal move. Until 2018, all competitive computer engines for this board game made use of an alpha-beta pruning algorithm with a hand-crafted evaluation function for each position. Previous machine learning-based algorithms for just regular chess, such as NeuroChess and Giraffe, took hand-crafted evaluation features as input rather than a raw board representation. More recent projects, such as AlphaZero, reached massive success but required massive computational resources in order to reach its final strength. This paper describes the development of SixtyFour, an engine designed to compete in the chess variant of Crazyhouse with limited hardware. This specific variant poses a multitude of significant challenges due to its large branching factor, state-space complexity, and the multiple move types a player can make. We propose the novel creation of a neural network-based evaluation function for Crazyhouse. More importantly, we evaluate the effectiveness of an ensemble model, which allows the training time and datasets to be easily distributed on regular CPU hardware commodity. Early versions of the network have attained a playing level comparable to a strong amateur on online servers.

Figures

Figures reproduced from arXiv: 1908.09296 by the authors.

Figure 1
Figure 1. Consider the two above grids representing a chessboard. In the left, we consider all [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. A diagram of the final residual network architecture [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Black to move. White has no pieces in its pocket whereas Black has two knights. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Game 2, White to move. White has three knights, a bishop, and a queen in its [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Final position of Game 2. Game 3: 1. e4 e5 2. Nf3 Nc6 3. Bc4 Bc5 4. O-O Nf6 5. d3 O-O 6. Bg5 d6 7. Nbd2 Bg4 8. h3 Bh5 9. c3 h6 10. Bh4 Bxf3 11. Nxf3 N@f5 12. exf5 e4 13. dxe4 Ne5 14. Nxe5 dxe5 15. Bxf6 Qxf6 16. N@d5 Qd8 17. Qg4 B@e6 18. Qxg7+ Kxg7 19. N@h5+ Kh7 20. N@f…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages

  1. [1]

    1069--1076

    Sebastian Thrun, ``Learning to play the game of chess,'' in Advances in neural information processing systems , 1995, pp. 1069--1076

    work page 1995

  2. [2]

    Matthew Lai, ``Giraffe: Using deep reinforcement learning to play chess,'' arXiv preprint arXiv:1509.01549 , 2015

    arXiv 2015

  3. [3]

    David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis, ``Mastering the...

    work page 2016

  4. [4]

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al., ``Mastering the game of go without human knowledge,'' Nature , vol. 550, no. 7676, pp. 354, 2017

    work page 2017

  5. [5]

    Hsiung-Hsu Feng, ``Ieee spectrum online: Cracking go,'' http://www.spectrum.ieee.org/oct07/5552, Oct 2007

    work page 2007

  6. [6]

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al., ``Mastering chess and shogi by self-play with a general reinforcement learning algorithm,'' arXiv preprint arXiv:1712.01815 , 2017

    arXiv 2017

  7. [7]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.ncblock write newline " " before.all 'output.state := FUNCTION new.nccont write " " before.all 'output.state := FUNCTION new.sentence output.state after.block = 'skip out...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.