{"id":"04c8d208-6b5c-4305-947d-bc93d2ea744c","arxiv_id":"1908.06660","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"A fully supervised deep network plus Monte-Carlo tree search plays crazyhouse at world-champion level, beating 12 of 13 tournament engines and winning 4 of 5 games against the 2017 world champion.","lead":"Researchers trained a neural network, CrazyAra, only on recorded human games of the chess variant crazyhouse, then added tree search, and it beat the 2017 world champion 4-1 and most computer rivals. The result suggests supervised learning with modest compute can reach champion-level play in a complex game.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Five-game 4-1 match cannot support 'above World Champion Level'; the title overclaims statistically and the champion is partly in the training data.","rationale":"The reader's formal weakest_assumption concerned noisy value labels from final game outcomes (Section 3.1), which is a legitimate training-quality issue. However, the most load-bearing condition for the central title claim is not the value-label noise but the size and independence of the evidence for 'above World Champion Level': a single 4-1 informal match against an opponent whose games are in the training set cannot statistically distinguish a stronger engine from an equal-strength engine. The paper's own evidence shows CrazyAra is below Stockfish, so the title claim must be scoped to human champion level, and that scoping rests on five games. The reader's rationale already lists the small match size and the champion's presence in the training data as reasons for a conditional verdict, so my concern is partly aligned with the reader's reasoning but not with the stated weakest_assumption. Since the conditional verdict already requires additional games, rephrasing, and released model weights, my analysis does not move the verdict; it sharpens the reason why the conditional caveat is essential.","tokens_in":36180,"tokens_out":5416,"duration_ms":64668,"concrete_test":"Run a pre-registered match of at least 30 long time-control games (e.g., 15+15 or 30+30) between CrazyAra 0.6.0 and a top human crazyhouse player whose games are excluded from training, or, if the same champion must be used, with that player's pre-2018 games held out of the training set. Use balanced opening positions. Apply a one-sided binomial test of the null hypothesis that CrazyAra's per-game win probability is at most 0.5, and require the 95% confidence lower bound for CrazyAra's win rate to exceed 0.5. If the test is not significant, the 'above World Champion Level' wording should be weakened to 'competitive with' or 'beat the champion in an informal match.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests almost entirely on a 4-1 result in five informal games against the 2017 crazyhouse world champion (Section 11.1). Under the null hypothesis that CrazyAra and Justin Tan are equal (win probability 0.5 per game), the probability of observing 4 or 5 CrazyAra wins is 6/32 = 0.1875, so the result is not statistically significant evidence of superiority. The match also evaluates against an opponent whose lichess games appear in the training pool: 'JannLee' is listed among the top-20 most active players in Figure 6, and the training set consists of 569,537 lichess games from January 2016 to June 2018. Thus part of the result may reflect in-distribution familiarity rather than general above-champion strength. The CPU results against twelve of thirteen 2017 computer-championship engines (Section 11.2) do not fill this gap: those engines are mostly far below Stockfish, CrazyAra lost all ten games to Stockfish, and the results do not bound human world-champion level. The paper itself labels the human match 'informal games' (Section 11.1), which further weakens the evidentiary basis for the title claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CrazyAra, a deep neural network engine for the chess variant crazyhouse, trained purely by supervised learning on 569,537 human games from lichess.org. The authors introduce a compact Markovian board representation, an input normalization scheme, a mobile-friendly residual network architecture, and several Monte-Carlo Tree Search (MCTS) modifications (Q-value-based move selection, principal-variation adjustment, transposition table, dynamic exploration parameters, and check-move boosting). They report a move-prediction accuracy of 60.4% on the validation set, a 4-1 win over 2017 crazyhouse world champion Justin Tan in an informal match, victories against twelve of thirteen participants of the 2017 CCVA Computer Championships on CPU hardware, and, after continued training on Stockfish self-play games (CrazyAraFish), three wins and one draw in ten long-time-control games against Stockfish 10. The central claim, stated in the title and abstract, is that CrazyAra plays 'above World Champion Level'.","tokens_in":36396,"tokens_out":3457,"duration_ms":35123,"significance":"If the strength claims are accepted, the paper would be a notable demonstration that supervised learning from modest amounts of human data, combined with a carefully engineered MCTS variant, can produce an engine competitive with strong human and computer crazyhouse players. The work is also useful for its engineering contributions: the compact input representation, the normalization study, the efficient network design, and the detailed ablation of MCTS components are clearly presented and empirically evaluated. The authors provide open-source code, data-processing scripts, and supplementary match records, which is a strength for reproducibility. The move-prediction and mate-finding metrics give concrete, falsifiable performance numbers. However, the headline 'above World Champion Level' claim rests on evidence that is not statistically robust and is partially in-distribution, so the significance of the paper as a claim about super-human play is not established by the current analysis.","major_comments":[{"comment":"The central claim that CrazyAra plays 'above World Champion Level' is supported only by a 4-1 result in five informal games against Justin Tan. Under the null hypothesis of equal strength (win probability 0.5 per game), the probability of winning four or five of five games is 6/32 = 0.1875, which is not statistically significant evidence of superiority. The match is explicitly described as 'informal games', and the opponent (LM JannLee) is one of the most active players in the training data, appearing in Figure 6 and contributing to the 46.03% of games played by the top-20 players. This means the evaluation is partly in-distribution for the network. To support the title claim, the authors need either a larger and more formal match, a statistical analysis with confidence intervals or a Bayes factor, or a revised claim that accurately reflects the evidence, e.g., 'wins an informal match against the 2017 world champion' rather than 'above World Champion Level'.","section":"Title and Section 11.1, Table 12"},{"comment":"The value labels are assigned from final game outcomes to every board position, based on the assumption that a large advantage almost surely leads to a win. The paper itself states this assumption 'is, however, heavily violated in our data set partly due to the popularity of low time control games' (Section 3.1), and Section 7 reports that 45.15% of games are one-minute bullet games. Since the value head directly guides the MCTS search, noisy value labels can systematically bias move selection. The paper does not quantify the impact of this label noise or test whether filtering or reweighting low-time-control games changes the results. Please provide an analysis of the value-label reliability (e.g., accuracy as a function of game length or time control) or discuss why the observed engine strength is not substantially affected.","section":"Section 3.1"},{"comment":"The engine evaluation, while showing strong results against twelve of thirteen CCVA participants, does not provide a bound on human world-champion level. Most of the defeated engines have Elo ratings well below 3,000 (and several are over 1,000 Elo below Stockfish), and CrazyAra lost all ten games to Stockfish 10. The abstract and conclusion use these results in a way that suggests they corroborate the human-match claim, but they are not evidence about the human champion level. The paper should separate more carefully the claims 'wins against most CCVA participants on CPU' from 'plays above human world-champion level', and the latter should be supported by the human-match evidence, which is currently insufficient.","section":"Section 11.2, Table 13"},{"comment":"The CrazyAraFish results against Stockfish 10 are presented as 'wins three games and draws one out of ten', but the paper does not report the statistical significance of this match either. With 3 wins, 1 draw, and 6 losses, the result is consistent with Stockfish being stronger over the match; the three wins are notable but do not demonstrate overall superiority. The discussion of these results in Section 10 should be phrased with appropriate uncertainty, and the abstract's emphasis on this result should be tempered accordingly.","section":"Section 11.2 and Table 14"}],"minor_comments":[{"comment":"There are several typographical errors: 'soley' should be 'solely', 'Neterov' should be 'Nesterov', 'Consquently' should be 'Consequently', 'mis-leading' should be 'misleading', and 'crazhouse' (Section 3.1) should be 'crazyhouse'.","section":"Throughout"},{"comment":"The caption of Figure 6 says 'matches≥2,000 Elo', but the text says the training data consists of games with 'Elo≥2,000 for both players'. Please clarify the exact filtering criterion and ensure the caption is consistent.","section":"Section 7, Figure 6"},{"comment":"In Equation (5), the notation Q'(s0,a) is used for the thresholded Q-values, but the definition says 'we set all Q-values with a visit count < Qthresh * max_a(N(s0,a)) to 0'; it would be clearer to write the formula for Q'(s0,a) explicitly, including the re-scaling to [0,1].","section":"Section 9.2.1"},{"comment":"The text says the match was played by 'CrazyAra 0.3.1' but Table 12 lists 'CrazyAra 0.3.0'. Please correct this inconsistency.","section":"Section 11.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's technical content is solid and the engineering contributions are valuable, but the headline claim is not supported by the evidence as presented. The in-distribution nature of the human opponent and the tiny match size are load-bearing concerns for the title. I would encourage the editor to ask for a substantial revision that either adds a statistically sound evaluation (e.g., a larger match or a rating-estimation with uncertainty) or revises the title and abstract to accurately reflect the achieved result. The latter is a straightforward fix, but it changes the paper's central claim, hence major revision rather than minor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is better than its title. The real contributions are concrete and useful: a compact fully Markovian board representation for crazyhouse, a normalization step that clearly helps training, the RISEv2-mobile architecture that cuts inference cost substantially, and several MCTS sample-efficiency tricks—Q-value-based final move selection, a transposition table, a parameterized U-value divisor, and check-prior boosting. These are evaluated with self-play Elo comparisons and a nice worked example showing how the search adaptations rescue a move the policy assigns near-zero probability. The code and data preprocessing are public, which makes the work independently checkable. The soft spot is exactly where the reader and stress-test put it: the 'above World Champion Level' claim rests on a 4-1 informal match against Justin Tan, whose lichess handle is in the top-20 training-data players. Under a null of equal strength, 4-1 has p approximately 0.19, so it is not statistically significant. And because the opponent is partly in-distribution, the result is even weaker as evidence of generalization beyond the training population. The authors themselves call the games informal, and the conclusion uses the more modest phrase 'level of the strongest human players'—the title and abstract oversell. The engine match results against twelve of thirteen CCVA 2017 engines are credible but those engines are mostly far below Stockfish, and CrazyAra lost all ten to Stockfish. CrazyAraFish's 3-1-6 showing against Stockfish 10 is a legitimate data point but does not rescue the human-champion claim. One more caveat worth flagging: the value labels are final game outcomes assigned to every position, and Section 3.1 admits this assumption is 'heavily violated' because of bullet games. That means the value head, and therefore MCTS guidance, is trained on noisy targets. The policy accuracy and mate-finding metrics are solid, but the value signal should be treated as a known weakness, not an afterthought. Who is this for? Researchers working on supervised game learning with limited, noisy data, and anyone interested in MCTS improvements that transfer beyond crazyhouse. It deserves serious peer review, but with a required major revision: rephrase the central claim with an error bar or more games, release model weights so the in-distribution concern can be quantified, and acknowledge the value-label issue in the evaluation. I would bring it to a reading group and would cite it for the MCTS sample-efficiency mechanisms.","headline":"Solid systems paper with real MCTS and representation contributions; the 'above World Champion Level' title rests on five informal games and is statistically weak, but the rest of the evidence and the released code make it worth a serious referee.","tokens_in":700,"tokens_out":898,"would_cite":true,"duration_ms":27298,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A supervised-only neural engine beat the 2017 crazyhouse world champion 4-1.","keywords":["Deep Learning","Chess","Crazyhouse","Supervised Learning","Monte-Carlo Tree Search","Neural network engine","Chess variant"],"falsifier":"Run a controlled match between CrazyAra 0.6.0 and a top human crazyhouse player at a slow time control (for example 30 minutes plus 30 seconds increment) from balanced openings and check whether the 4-1 result survives; separately, compute the value head's accuracy on positions from bullet games where the winner flagged in a materially losing position.","tokens_in":35917,"feed_emoji":"♟️","tokens_out":7145,"duration_ms":67934,"temperature":0.7,"pith_summary":"The paper is trying to establish that a neural-network engine called CrazyAra can play the chess variant crazyhouse above the level of the human world champion even though it is trained solely by supervised learning on human games, with no reinforcement learning and no self-play. Its most direct evidence is a 4-1 win in informal games against the 2017 crazyhouse world champion, together with wins against twelve of thirteen engines from the 2017 crazyhouse computer championship while running on CPU hardware. If true, this matters because it would show that the very large computation budgets associated with self-play systems are not the only path to elite play in a complex, high-branching game: a compact architecture, a fully Markovian board encoding, input normalization, and sample-efficient search can turn a modest set of noisy human games into a world-class engine.","feed_headline":"Supervised-only AI beats crazyhouse world champion 4-1","feed_subtitle":"No self-play or reinforcement learning: a neural net trained on human games also beat 12 of 13 engines on CPU.","key_machinery":"The load-bearing machinery is a dual-head deep convolutional network whose shared residual tower feeds a policy head over all legal-sounding moves and a value head returning a win/loss estimate in $[-1,1]$, coupled with PUCT, an upper-confidence-bound tree search guided by the learned policy prior. The input is a fully Markovian 34-plane $8\\times 8$ encoding of the current position only, covering pieces, pocket contents, promoted pieces, en-passant, castling, repetition, and move counters, with each non-binary plane divided by a maximum value. The search is made sample-efficient by Q-value-aware final move selection, principal-variation Q updates, a transposition table, time-dependent search, and a prior-enhancement step that raises the probability of checking moves; these modifications are what let a low node budget compete against engines searching thousands of times more nodes.","core_discovery":"On its own terms, the paper's claim is that a single shared neural network predicting both a move policy and a position value, combined with an adapted Monte-Carlo tree search, reaches world-champion-level crazyhouse after supervised training on 569,537 human games. The network reaches 60.4% move-prediction accuracy after 1.5 days of training, and the engine's strength is demonstrated in two ways: a 4-1 informal match win over the 2017 human world champion and a 12-1 record against the field of the 2017 crazyhouse computer championship on CPU hardware. A later version, fine-tuned on engine-generated games, wins three and draws one of ten long-time-control games against a strong alpha-beta engine, which the paper presents as supporting the same underlying approach rather than as evidence that reinforcement learning is required.","pith_inferences":["If the final-outcome value labels are as noisy as the paper admits, then much of CrazyAra's playing strength may come from the policy head and search priors rather than from the value head; an ablation that feeds a constant value to the search would isolate the value head's contribution.","The 4-1 human result is informal and short-time-controlled; a slower, controlled match against the same champion from balanced openings would be the sharper test of 'above world champion level.'","The same recipe, supervised training on human games, a compact Markovian encoding, and sample-efficient search, could be applied to other games with sparse human data and large branching factors, such as bughouse or shogi variants with drops."],"forward_implications":["A supervised-only training pipeline can produce an elite engine in a tactic-heavy game with a higher branching factor than chess, so massive self-play computation is not the only route to top strength.","The search modifications that add Q-values, principal-variation information, transposition reuse, and check-move priors each raise playing strength at low node counts, which suggests they are transferable improvements for other neural-guided MCTS engines.","Beginning with human games and then continuing training on engine-generated games yields a model that takes 3.5 of 10 points from a strong alpha-beta engine at long time control, indicating the human-data model is a viable base for further refinement.","At roughly 330 nodes per second on CPU, the engine beat most opponents despite their searching roughly 3,000 to 14,000 times more nodes, so node count alone is not a reliable proxy for playing strength in this setting."],"supporting_citations":[{"why":"Supplies the AlphaZero-style residual network and PUCT search scheme that CrazyAra adapts to crazyhouse.","marker":"Silver et al. (2017a)"},{"why":"Supplies the supervised training-from-human-games template and the value-loss weighting (alpha=0.01) used in the combined loss.","marker":"Silver et al. (2016)"},{"why":"Provides the UCT bandit algorithm underlying the Monte-Carlo tree search.","marker":"Kocsis and Szepesvári (2006)"},{"why":"Supplies the residual connections used in the shared network tower.","marker":"He et al. (2016a)"},{"why":"Supplies the MobileNet v2 inverted-residual blocks used in the efficient CPU model.","marker":"Sandler et al. (2018)"},{"why":"Defines the participant field of the 2017 crazyhouse computer championship used for engine evaluation.","marker":"Mosca (2017)"}],"fun_headline_variants":["Supervised AI takes crazyhouse crown from world champ","No RL needed: Neural net beats crazyhouse champ 4-1","CrazyAra: Human-data-only AI tops world champion","Deep net learns crazyhouse from humans, wins 4-1 vs champ","Supervised crazyhouse AI: 4-1 vs champion, no self-play"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The value labels are taken from each game's final outcome and assigned to every position, on the assumption that a player with a large advantage will almost surely win; the paper states this is heavily violated because many games are low-time-control bullet games decided by blunders or time forfeits.","fun_headline_variants_meta":{"raw":{"variants":["Supervised AI takes crazyhouse crown from world champ","No RL needed: Neural net beats crazyhouse champ 4-1","CrazyAra: Human-data-only AI tops world champion","Deep net learns crazyhouse from humans, wins 4-1 vs champ","Supervised crazyhouse AI: 4-1 vs champion, no self-play"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2074,"prompt_tokens":1030,"completion_tokens":1044,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":951}},"tokens_in":646,"tokens_out":1044,"duration_ms":7913,"temperature":1.0,"reasoning_tokens":951,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:37:59.925506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled match between CrazyAra 0.6.0 and a top human crazyhouse player at a slow time control (for example 30 minutes plus 30 seconds increment) from balanced openings and check whether the 4-1 result survives; separately, compute the value head's accuracy on positions from bullet games where the winner flagged in a materially losing position.","supporting_citations":[{"cited_title":"J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al","cited_arxiv_id":null,"evidence_quote":"Supplies the supervised training-from-human-games template and the value-loss weighting (alpha=0.01) used in the combined loss."},{"cited_title":"and Szepesv \\' a ri, C","cited_arxiv_id":null,"evidence_quote":"Provides the UCT bandit algorithm underlying the Monte-Carlo tree search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MobileNet v2 inverted-residual blocks used in the efficient CPU model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the participant field of the 2017 crazyhouse computer championship used for engine evaluation."}],"review_version":1}