{"id":"4b8a79d4-15b5-46d8-99ad-edda91e8e5c0","arxiv_id":"1908.09296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A residual-network policy/value engine trained on Stockfish self-play reached a 2278 Lichess rating in Crazyhouse, though the evidence is anecdotal and baseline comparisons are missing.","lead":"This paper introduces SixtyFour, a neural network engine for Crazyhouse that learns by imitating Stockfish self-play and then searches with Monte Carlo tree search. It reached a peak Lichess rating of 2278, but the evidence for the advertised 'master-level' strength is mostly anecdotal and lacks a strong baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The peak Lichess rating of 2278 and selected game transcripts do not establish the conclusion's 'master-level' claim, because no controlled baseline against known Crazyhouse engines or titled players is reported.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The paper describes a plausible engineering effort: it releases code, describes a concrete training pipeline, and includes self-play games and online games that show real tactical ability. The reader's weakest_assumption focuses on the quality of Stockfish self-play labels as the cap on engine strength. My stress-test concern is different: even if the training labels are excellent, the reported outcome measurements do not establish the 'master-level' conclusion. A peak Lichess rating and selected transcripts are not a controlled strength evaluation. This is exactly the kind of missing baseline that should be required before accepting the strongest claim. Because the reader's verdict already conditions acceptance on this kind of evidence, my analysis does not shift the verdict; it reinforces it. I therefore recommend no change to the CONDITIONAL verdict, with the condition being a controlled match against a known strong engine or titled player.","tokens_in":11696,"tokens_out":3707,"duration_ms":43512,"concrete_test":"Use the released GitHub code to play a fixed 100-game match against Stockfish 11 (multi-variant) at a standard online time control such as 3+2 on identical hardware, with both engines using the same opening book or none, and report the score and average centipawn loss. If SixtyFour scores below 40% or loses the match by more than 20 points, the 'comparable to a master' claim should be downgraded to 'strong amateur' in both the abstract and the conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SixtyFour, with tree search, plays at a level comparable to a Crazyhouse master. The reported support is a peak Lichess rating of 2278 from 294 rated games and selected game transcripts (Section 4.3). This is not sufficient. A peak rating is the maximum of a noisy trajectory, not a stable strength estimate, and Lichess Crazyhouse ratings are not calibrated to any 'master' standard; 'higher than 97% of users' describes the site's rating distribution, not absolute playing strength. No controlled baseline is measured: the engine is never played against Stockfish multi-variant, CrazyAra, or a titled human under matched time controls, and the opponents' ratings are not given for the showcased games. The selected wins demonstrate tactical competence, and the self-play games show plausible attacking play, but they do not anchor the engine to a known master-level scale. Therefore the conclusion's 'master' phrasing is an overreach; the abstract's 'strong amateur' is the strongest claim the reported evidence supports. This is a measurement and interpretation issue, not necessarily a failure of the training method itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes SixtyFour, a deep residual-network engine for the chess variant Crazyhouse. The network has a policy head over a sparsely encoded 2308-move representation and a value head, and is trained by supervised learning on 134,386 Stockfish self-play games (about 11.2 million positions), then combined with a Monte Carlo Tree Search variant. The authors report a validation policy accuracy of 54.1%, several self-play games, ten games against humans and against Lichess Stockfish AI Levels 4 and 5, and a peak Lichess rating of 2278 from 294 rated games. The abstract claims a playing level comparable to a strong amateur, while the conclusion states that the engine 'has reached a level comparable to a master in the game of Crazyhouse.'","tokens_in":12083,"tokens_out":5176,"duration_ms":54504,"significance":"If the claims were fully supported, this would be a useful demonstration that a moderately sized ResNet trained by supervised imitation on Stockfish self-play, together with MCTS, can reach a strong human amateur level in Crazyhouse on commodity hardware. The move-output representation, the pocket encoding, and the reproducibility details (source code, architecture, training hyperparameters) are concrete strengths, and the Lichess rating is a genuine external measurement. However, the significance is dampened by the gap between the abstract's 'strong amateur' claim and the conclusion's 'master' claim, by the absence of strong engine baselines, and by the circularity of the policy-accuracy comparison. The paper's strongest supported conclusion is an amateur-level engine, not a master-level one.","major_comments":[{"comment":"The conclusion's claim that the engine 'has reached a level comparable to a master in the game of Crazyhouse' is not supported by the evidence. A peak rating of 2278 from 294 rated games is the maximum of a noisy rating trajectory, and the section does not provide opponent ratings, rating intervals, or a controlled match against a known master-strength player or engine. The statement that 2278 is 'higher than 97% of all Crazyhouse users' describes the Lichess rating distribution, not an absolute master-level standard. The abstract's 'strong amateur' wording is the strongest claim the reported evidence supports; the master claim should be removed or replaced by a controlled measurement.","section":"Section 5 and Section 4.3"},{"comment":"No controlled baseline against a strong Crazyhouse engine is reported. The games against Stockfish are against Lichess 'AI Level 4' and 'AI Level 5', which are reduced-strength handicap levels, and no comparison is made against full-strength multi-variant Stockfish, Sunsetter, Sjeng, or CrazyAra under matched time controls. In addition, the self-play games in Section 4.2 are explicitly described as selected 'most accurate games,' so they cannot serve as representative evidence of overall playing strength.","section":"Sections 4.2 and 4.3"},{"comment":"The reported policy accuracy of 54.1% is measured against a validation set drawn from the same Stockfish self-play distribution used for training, so it is an imitation score rather than an independent measure of playing strength. The comparison to AlphaGo's 55.4% is misleading because AlphaGo's accuracy was reported against held-out professional human games, not against the engine's own training distribution. This circularity does not invalidate the external Lichess rating, but the policy-accuracy comparison should be removed or reframed as an imitation-quality metric.","section":"Section 4.1"},{"comment":"The abstract states that the paper 'evaluate[s] the effectiveness of an ensemble model,' but no ensemble architecture, training procedure, or empirical ensemble results appear anywhere in the body. Either add the promised ensemble comparison or remove the ensemble claim from the abstract and introduction.","section":"Abstract"}],"minor_comments":[{"comment":"The MCTS score equation is under-specified: Nnode is defined but the formula uses only Nparent and nnode, and Wnode is described as a count of playout wins but appears as Wnode/2 without being normalized by visit count. Please define every symbol and give the exact update rule.","section":"Equation (1), Section 3.5"},{"comment":"The caption says that Black checkmates 'the Black king,' but the checkmated king is White's; it also refers to 'B@g3' and 'Qh3!!' without giving the move numbers, which makes the discussion hard to follow.","section":"Figure 3 caption"},{"comment":"The time control '0.5m+0s' is not standard notation; please spell out the exact clock setting (for example, 30 seconds with zero increment). The section also does not give the Lichess ratings of the human opponents in the showcased games.","section":"Section 4.3"},{"comment":"The paper calls the neural-network engine a novel attempt while simultaneously citing the concurrent CrazyAra project; please clarify the novelty claim relative to CrazyAra and cite the published work on CrazyAra rather than only a GitHub link.","section":"Section 2.2"},{"comment":"The phrase 'top 97% percentile' should be '97th percentile,' and the sentence 'at around the same time of this project' in Section 2.2 is awkward and should be rephrased.","section":"Section 5"},{"comment":"The statement about 'wins against humans without a search algorithm' is ambiguous, because Section 4.3 reports games of the full engine with tree search; please clarify whether the raw policy network has verified wins against humans and where those games are shown.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a project write-up rather than a fully developed journal article, but the core engineering is plausible and the external Lichess rating gives it some empirical grounding. The main fix is the mismatch between the abstract and the conclusion: either add a controlled match against a strong engine or scale the conclusion back to 'strong amateur.' I also strongly recommend either removing the unsubstantiated ensemble claim or actually reporting the ensemble experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jeremy,\n\nHere's my take on the Crazyhouse paper. It's a modest, honest engineering report. What's actually new is a compact move encoding for Crazyhouse (2308 dimensions instead of raw planes) and a policy scaling trick that down-weights losing moves (0.1) and draws (0.5). The network is a standard ResNet, trained supervised on Stockfish self-play, then used with a custom PUCT variant and time management. The best evidence is external: the engine reached 2278 on Lichess over 294 rated games, and a few of the showcased games are genuinely impressive tactical wins. That's a real existence proof that a cheaply trained supervised network plus MCTS can play at strong amateur level.\n\nWhere it falls down is exactly where the stress-test note flags. The conclusion's 'master' claim doesn't follow from a peak rating. A peak is the maximum of a noisy trajectory, Lichess ratings aren't calibrated to any master title, and no controlled baseline is reported - no games against CrazyAra, multivariant Stockfish with matched time controls, or titled humans. The self-play games are cherry-picked, and the 54.1% policy accuracy is measured on the same Stockfish distribution used for training, so it's an imitation score, not an independent strength measure. Also the abstract promises an ensemble model, but the full text never presents ensemble results. That's a real missing piece.\n\nOn balance, the training approach is sound for what it is. The authors started from a supervised imitator, reinforced with self-play, and measured on a public server - that's more than many similar projects do. The claims are just oversized. The right fix is to soften 'master' to 'strong amateur' and add a couple of controlled matches.\n\nWho's this for? People working on chess-variant AI or cheap AlphaZero-style training. It won't change the field; CrazyAra already occupies the stronger end. But as an engineering write-up with an external measurement, it deserves a serious referee, not a desk reject.\n\nRecommendation: send it to peer review, with the clear expectation that the 'master' language gets cut and honest baseline games are reported.","headline":"A solid engineering report with a real Lichess measurement, but the 'master' claim overreaches; the evidence supports strong amateur.","tokens_in":12414,"tokens_out":1673,"would_cite":false,"duration_ms":17370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A ResNet trained on Stockfish self-play reaches master level in Crazyhouse chess on ordinary hardware.","keywords":["Crazyhouse","deep residual network","Monte Carlo Tree Search","policy network","value network","neural network chess engine","self-play training","Stockfish"],"falsifier":"Play a time-limited match between SixtyFour and the multi-variant Stockfish at a fixed high strength (e.g., depth 20 or 2000 nodes per move) in a controlled setting with the same time controls as the paper's online games; if SixtyFour's win rate against such Stockfish is not clearly above 50%, the claim that it has reached master-level play is falsified, since the training data itself comes from stronger Stockfish searches. A second direct check: run the ResNet with MCTS disabled and compare its self-play accuracy against the reported 65.02-ply average game length and ACPL values; if the network cannot reproduce such low centipawn losses on unseen positions, the outcome-scaling of labels may not be as effective as claimed.","tokens_in":11516,"feed_emoji":"♟️","tokens_out":1835,"duration_ms":19731,"temperature":0.7,"pith_summary":"This paper reports the creation of SixtyFour, a Crazyhouse chess engine built around a deep residual network rather than a hand-crafted evaluation function, which works with a Monte Carlo Tree Search and can run on limited, commodity CPU hardware. The central claim is that a supervised policy network trained to imitate Stockfish's self-play moves, combined with a simple tree search, reaches a rating of 2278 on Lichess, a level the author compares to a master and higher than 97% of active Crazyhouse players. If true, the work shows that a moderately sized residual network, without hand-crafted features or massive computational resources, can master a highly tactical chess variant with a branching factor far larger than regular chess. It also suggests that the careful representation of a complex state space and the scaling of training labels by game outcome are sufficient to build a competitive engine. The paper matters because it extends the deep-learning approach to board games beyond regular chess and Go, into a variant that has traditionally been dominated by alpha-beta engines with custom evaluation functions.","feed_headline":"ResNet plus tree search hits 2278 in Crazyhouse","feed_subtitle":"A supervised network trained on Stockfish self-play beats 97% of Lichess Crazyhouse players on a modest setup.","key_machinery":"The central machinery is a residual convolutional neural network (ResNet) that maps a 960-channel raw board representation to a policy vector of length 2308 and a value score, guided by a Monte Carlo Tree Search (MCTS) that uses a modified PUCT formula with an exploration noise term decaying over the game. The board representation encodes piece positions, promoted-piece status, pocket contents (captured pieces available for dropping), castling rights, repetition information, and side to move, so the network can reason about piece drops. The MCTS values checkmates at 1.5 instead of 1.0 to encourage mating lines, and the search depth adapts to the remaining clock time, ranging from 4 to 20 plies per playout. The training labels are outcome-scaled so that moves from losing games are down-weighted, which the author says prevents the network from imitating blunders in a game where one mistake can undo many accurate moves.","core_discovery":"SixtyFour is a neural-network based Crazyhouse engine that uses a 12-block residual network with 256 kernels per block to output a policy over 2308 legal moves and a scalar value, trained on roughly 11.2 million moves from 134,386 Stockfish self-play games. The paper's core discovery is that such a ResNet, trained with a scaled one-hot policy label (1.0 for winning moves, 0.1 for losing moves, 0.5 for draws) and a value head on game result, can achieve a policy prediction accuracy of about 54.1 percent on Stockfish's own validation moves—comparable to AlphaGo's initial supervised policy network—and that when combined with a depth-limited Monte Carlo Tree Search, it plays at a level rated 2278 on Lichess, higher than 97% of all Crazyhouse users. The engine's self-play games show low average centipawn loss and frequent mating combinations, and its games against Stockfish AI levels 4 and 5 online indicate that it can defeat moderately strong Stockfish settings. The author argues that the raw board representation, the explicit use of promoted-piece and pocket information, and the outcome-scaled policy labels are what allow the network to grasp the abstract tactical concepts of Crazyhouse without any hand-crafted evaluation features.","pith_inferences":["The paper implicitly claims that the quality ceiling is set by Stockfish's own play rather than by the deep network: since the training data is generated by Stockfish at up to one million nodes per position, a network that perfectly imitated that data would at best match that engine's strength, so the 2278 rating is likely below Stockfish's ceiling, meaning the ResNet's practical value is as a fas","If the 54.1% top-1 policy accuracy is compared to the branching factor of Crazyhouse (roughly three times chess), then random policy would give near-zero accuracy, so 54.1% indicates that the network has learned a highly concentrated distribution over sensible moves—even so, a player at the 97th percentile of Lichess users is not yet at the level of the strongest human Crazyhouse specialists, and ","The ensemble model mentioned in the abstract is not described in detail in the paper itself, so a reasonable editorial reading is that the author intended to distribute training over commodity CPUs by training multiple smaller networks separately—testing whether such an ensemble closes the gap to Stockfish would directly probe the scalability claim."],"forward_implications":["If the reported Lichess rating of 2278 and the games against Stockfish are representative, a moderate-sized ResNet combined with a shallow MCTS can produce a competitive Crazyhouse engine on single-CPU hardware, offering a practical alternative to alpha-beta engines that require heavily tuned evaluation functions.","The outcome-scaled policy labeling (1.0/0.1/0.5) can be tested directly in other games with near-zero draw rates and high tactical volatility, such as Shogi or bughouse, to see whether down-weighting losing moves improves final engine strength.","The specific move representation—a compact 2308-entry one-hot vector with separate planes for drops, linear moves, knight moves, and underpromotions—provides a template for encoding high-branching-factor games efficiently without wasting capacity on illegal moves.","The paper's approach of training on Stockfish self-play openings from the Lichess Opening Explorer suggests a data-augmentation recipe for other rare variants: use a strong engine with vast search to generate millions of positions, then distill the engine's behavior into a fast network for online play."],"supporting_citations":[{"why":"Provides the AlphaGo supervised policy network whose 55.4% policy accuracy is the comparison point for SixtyFour's 54.1%.","marker":"[3]"},{"why":"Supplies the AlphaZero-style training scheme: residual networks, policy and value heads, and self-play with MCTS that SixtyFour adapts.","marker":"[4]"},{"why":"The generalized AlphaZero paper that motivates the use of a ResNet architecture for chess and shogi with self-play, the template SixtyFour follows for Crazyhouse.","marker":"[6]"},{"why":"NeuroChess serves as an earlier neural-network chess engine that used hand-crafted features, against which SixtyFour's raw-board representation is contrasted.","marker":"[1]"},{"why":"Giraffe offers a modern example of TDLeaf(lambda) with deep learning on hand-crafted features, a predecessor whose evaluation-feature input design SixtyFour deliberately abandons.","marker":"[2]"},{"why":"Feng-Hsiung Hsu's prediction that brute-force search would crack Go within a decade is cited as motivation for why neural-network search is instead needed for high-branching games.","marker":"[5]"}],"fun_headline_variants":["SixtyFour: ResNet beats 97% of Lichess Crazyhouse players","Deep ResNet chess engine hits 2278 on limited hardware","Crazyhouse AI: 12-block ResNet rivals AlphaZero's policy skill","Neural net Crazyhouse engine reaches strong amateur level","Training a ResNet on Stockfish games yields 2278-rated player"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels from Stockfish self-play at up to one million nodes per position are assumed to be accurate enough that a network imitating them learns high-level Crazyhouse play; if Stockfish's search is too shallow or its opening distribution too narrow, the imitated policy and value would cap the engine's strength regardless of architecture or search.","fun_headline_variants_meta":{"raw":{"variants":["SixtyFour: ResNet beats 97% of Lichess Crazyhouse players","Deep ResNet chess engine hits 2278 on limited hardware","Crazyhouse AI: 12-block ResNet rivals AlphaZero's policy skill","Neural net Crazyhouse engine reaches strong amateur level","Training a ResNet on Stockfish games yields 2278-rated player"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3471,"prompt_tokens":1028,"completion_tokens":2443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2348}},"tokens_in":644,"tokens_out":2443,"duration_ms":15610,"temperature":1.0,"reasoning_tokens":2348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:15:53.369276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Play a time-limited match between SixtyFour and the multi-variant Stockfish at a fixed high strength (e.g., depth 20 or 2000 nodes per move) in a controlled setting with the same time controls as the paper's online games; if SixtyFour's win rate against such Stockfish is not clearly above 50%, the claim that it has reached master-level play is falsified, since the training data itself comes from stronger Stockfish searches. A second direct check: run the ResNet with MCTS disabled and compare its self-play accuracy against the reported 65.02-ply average game length and ACPL values; if the network cannot reproduce such low centipawn losses on unseen positions, the outcome-scaling of labels may not be as effective as claimed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AlphaGo supervised policy network whose 55.4% policy accuracy is the comparison point for SixtyFour's 54.1%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AlphaZero-style training scheme: residual networks, policy and value heads, and self-play with MCTS that SixtyFour adapts."},{"cited_title":"1069--1076","cited_arxiv_id":null,"evidence_quote":"NeuroChess serves as an earlier neural-network chess engine that used hand-crafted features, against which SixtyFour's raw-board representation is contrasted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Feng-Hsiung Hsu's prediction that brute-force search would crack Go within a decade is cited as motivation for why neural-network search is instead needed for high-branching games."}],"review_version":1}