REVIEW 5 major objections 5 minor 5 references
Reinforcement Learning for Hanabi
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper finds that in a 1,000-game tournament of RL agents playing cooperative Hanabi, temporal-difference methods—especially tabular Expected SARSA and deep Q-learning—score highest.
desk verdict A plausible but statistically thin comparison of standard RL algorithms on Hanabi; the unvalidated shaped reward and missing variance make the agent-ranking claim unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the temporal-difference update rules and a 20-by-12 reward array inherited with the Hanabi environment. Expected SARSA's update averages the next-state Q-values over all actions, $Q(s,a)\leftarrow Q(s,a)+\alpha[R+\gamma \tfrac{1}{n}\sum_i Q(s'_i,a'_i)-Q(s,a)]$, letting the agent interpolate between on-policy SARSA and off-policy Q-learning as exploration decays. For deep agents, a densely connected feedforward network with ReLU hidden layers, a Softmax output over 20 moves, mean-squared-error loss, and the Adam optimizer serves as the function approximator. The environment's reward vector encodes 12 reasons an action may be good—playing with life tokens, hinting a playable card, discarding safely, and so on—and this reward, not the game's final score alone, drives every update.
What would settle it
Rerun the same 1,000-game tournament but score agents with an independently validated reward—for example, one derived only from final game score and legal moves—and check whether tabular Expected SARSA and deep Q-learning still lead; if the ordering changes, the claim is an artifact of the reward encoding rather than of learning ability.
Extended reading notes
Core claim
Temporal-difference (TD) algorithms, especially tabular Expected SARSA and, among deep methods, Q-learning and 2-step SARSA, are the strongest Hanabi learners in this comparison. Games in which at least one player uses tabular Expected SARSA achieved the highest average tabular scores, and the method's blend of on-policy and off-policy updates gave it a balanced mix of plays, hints, and discards. With neural function approximation, deep Q-learning and deep 2-step SARSA topped the scores, and deep variants improved on their tabular counterparts for 80% of agent pairings (Wilcoxon signed-rank p = 0.0038). The paper interprets the deep results as showing that a neural network can supply the averaging or lookahead advantages that otherwise distinguish Expected SARSA and n-step methods.
Load-bearing premise
The ranking rests on the reward function the authors inherited from their Hanabi environment; if that 12-reason reward does not truly encode what makes a move good, the agent rankings may not generalize to real Hanabi play.
Editorial extensions
If this is right
- Tabular Expected SARSA is the best tabular baseline for Hanabi under this reward, so future tabular work should adopt it.
- Deep Q-learning and deep 2-step SARSA with four hidden layers and learning rate 0.01 are the strongest deep configurations tested.
- Deep function approximation improved scores for 80% of agent pairings (p = 0.0038), so most but not all agents benefit from neural networks.
- Agent pairing determines which agent scores highest, meaning self-play scores alone are an incomplete evaluation.
- Tabular and deep results are fairly comparable in score, while deep methods cost 3-12x more compute; the tradeoff should be part of any deployment decision.
Reading between the lines
- A direct test of the paper's explanation: an ensemble of Q-networks that averages over several bootstrapped value estimates should recover much of Expected SARSA's tabular advantage in the deep setting, since averaging is exactly what distinguishes Expected SARSA from Q-learning.
- The paper's pairing sensitivity implies that a single-agent leaderboard would mislead; future evaluations should report a full pairwise matrix or define the opponent class explicitly.
- Because deep training cost 8-12x longer for only modest score gains, a practical extension is a compute-aware comparison that asks whether the extra score is worth the wall-clock cost in time-limited deployments.
- A generalization check the authors did not run: pit these agents against an independently trained or human-like partner to see whether the reward-based rankings survive contact with strategies outside the training distribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares four tabular temporal-difference (TD) algorithms (Q-learning, SARSA, n-step SARSA, Expected SARSA) and their deep-network counterparts in a two-player Hanabi environment inherited from Kantack et al. (2022). Training is driven by a 20×12 shaped reward array; evaluation reports averages over 1,000 games for final score, turn counts, and counts of play/discard/hint actions. The authors claim that TD methods, especially tabular Expected SARSA and deep Q-learning, perform best, but the results section reports that 2-step SARSA is the highest-scoring deep agent. They also perform an ablation over hidden-layer count and learning rate and compare tabular versus deep variants with a Wilcoxon signed-rank test.
Significance. If the empirical rankings were reliable, the paper would provide a useful comparison of TD algorithms in a partially observable cooperative game. The breadth of agent pairings and the inclusion of gameplay metrics are strengths. However, the central result is not credible as stated because of internal contradictions, an unvalidated shaped reward that drives all learning, and the absence of variance estimates. The contribution is incremental, and the paper does not provide code or data, so the experiments are not directly reproducible. With corrected methodology and clarified results, the study could serve as a reference point for algorithm selection in Hanabi-like settings.
major comments (5)
- [Abstract; §4.2; §5] The paper's central conclusion is internally inconsistent. The abstract and conclusion state that deep Q-learning agents had the best performance, but §4.2 (Figure 4) reports that games with two 2-step SARSA agents return the highest scores and Q-learning returns the second highest, and §5 repeats that 'Q-learning and 2-step SARSA emerged as the most effective deep learning methods.' Please resolve this contradiction with the actual quantitative scores and state which comparison supports the abstract's claim.
- [§3.1] The unvalidated reward function is load-bearing for the central claim. All learning updates use a 20×12 shaped reward array from Kantack et al. (2022), with twelve weighted 'reasons' but no weights or validation reported. The paper evaluates only the standard Hanabi terminal score, so the agent rankings are valid only if maximizing the shaped reward is monotone in the true score. Provide evidence for this, for example by reporting the correlation between shaped return and final score, or by ablating the reward weights and showing the rankings are robust.
- [§3.2; §3.3] The algorithm definitions are technically incorrect. The update rule labeled Expected SARSA averages over n next-state Q-values, which is not the Expected SARSA update; the expectation should be over the policy's action distribution. Relatedly, the footnote in §3.2 claiming that 1-step SARSA can be considered the off-policy version of regular SARSA is backwards: Q-learning is the off-policy variant, while SARSA is on-policy. Correct these statements and confirm that the implementation matches the corrected update.
- [§3; §4, Figure 1] No error bars, seeds, or repeated runs are reported. Each 1,000-game average appears to come from a single training run, so differences between agent pairings cannot be distinguished from noise. The ablation in §4, Figure 1, uses only 100 games and displays only Expected SARSA and Q-learning, yet the text claims that 4 hidden layers and α=0.01 are best 'over all agents.' Report variance over multiple seeds and show the full ablation, ideally with all agents and error bars.
- [§5] The Wilcoxon signed-rank test is under-specified. The manuscript does not state the sample size, the pairs being compared, whether the test is one- or two-tailed, or the effect size, and it does not correct for multiple comparisons. As written, the p-value of 0.0038 is not interpretable. Report the test statistic, the pairing definition, the number of comparisons, and a measure of effect size.
minor comments (5)
- [§4.2] The sentence 'Games where at least one of the players was either a Q-learning agent or 2-step SARSA agent were also lower scoring' appears to contradict the immediately preceding statement that 2-step SARSA pairs return the highest scores; please rephrase.
- [§3.1] The list of reward reasons is difficult to follow: items 7 and 8 ('Providing hint on non-playable card' versus 'Providing hint on playable cards') overlap with items 4 and 5, and no weights are given. Please define the semantics precisely and provide the weights or a public pointer to them.
- [§2] The paper does not state the number of players or the starting number of hint tokens used in the environment. Please specify the exact game configuration (number of players, starting hint tokens, deck composition, and any rule variants).
- [§3.3] The equations are typeset in a way that is hard to read, with missing indices and unclear notation for Q*. Please define all symbols and use standard indices for time steps.
- [General] The manuscript would benefit from a data and code availability statement, since no code or data are mentioned and the experiments are not otherwise reproducible.
Circularity Check
No significant circularity: algorithm rankings come from empirical runs under a shared reward, not from fitting or self-referential equations.
full rationale
The paper's central claims are empirical comparisons of temporal-difference algorithms in Hanabi. No prediction is derived from a fitted constant, and no equation in the paper reduces to another by construction. The only self-referential element is the reward array inherited from Kantack et al. (2022), on which one of the present authors is a co-author; however, that citation is used as a software and environment dependency, not as a load-bearing proof. All agents are trained and evaluated under the same reward, so the ranking of algorithms is not forced by the reward's definition. Whether the shaped 20x12 reward correctly encodes Hanabi success is a construct-validity and correctness concern, not a circularity, because the paper never claims to derive Hanabi skill from the reward. The hyperparameter ablation that selects four hidden layers and a learning rate of 0.01 is a model-selection step, not a fitted parameter renamed as a prediction. Thus no circular step can be exhibited under the required evidentiary standard.
Assumptions & free parameters
free parameters (4)
- learning rate (alpha) =
0.01
- number of hidden layers =
4
- epsilon schedule for Expected SARSA =
unspecified
- reward weights for 12 move reasons =
from Kantack et al., unspecified
assumptions (4)
- domain assumption Hanabi rules are correctly implemented in the Kantack et al. environment.
- domain assumption The reward function accurately reflects cooperative success in Hanabi.
- standard math The Wilcoxon signed-rank test is applied to paired comparisons with proper correction.
- ad hoc to paper Hyperparameters from the ablation generalize to all agent pairings.
Cite this review
Pith. "Pith review of Reinforcement Learning for Hanabi." pith.science (2026). https://pith.science/paper/EWR7DWSW
@misc{pith2026250600458,
author = {Pith},
title = {Pith review of: Reinforcement Learning for Hanabi},
year = {2026},
howpublished = {\url{https://pith.science/paper/EWR7DWSW}},
note = {Machine review of arXiv:2506.00458}
}
read the original abstract
Hanabi has become a popular game for research when it comes to reinforcement learning (RL) as it is one of the few cooperative card games where you have incomplete knowledge of the entire environment, thus presenting a challenge for a RL agent. We explored different tabular and deep reinforcement learning algorithms to see which had the best performance both against an agent of the same type and also against other types of agents. We establish that certain agents played their highest scoring games against specific agents while others exhibited higher scores on average by adapting to the opposing agent's behavior. We attempted to quantify the conditions under which each algorithm provides the best advantage and identified the most interesting interactions between agents of different types. In the end, we found that temporal difference (TD) algorithms had better overall performance and balancing of play types compared to tabular agents. Specifically, tabular Expected SARSA and deep Q-Learning agents showed the best performance.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Imitation learning by reinforcement learning
Kamil Ciosek. Imitation learning by reinforcement learning. In International Conference on Learning Representations, 2022
work page 2022
-
[2]
Instructive artificial intelligence (ai) for human training, assistance, and explainability
Nicholas Kantack, Nina Cohen, Nathan Bos, Corey Lowman, James Everett, and Tim Endres. Instructive artificial intelligence (ai) for human training, assistance, and explainability. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV, volume 12113, pages 45--54. SPIE, 2022
work page 2022
-
[3]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
2018
-
[4]
A theoretical and empirical analysis of expected sarsa
Harm van Seijen, Hado van Hasselt, Shimon Whiteson, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 177--184, 2009
work page 2009
-
[5]
Christopher Watkins. Learning from Delayed Rewards. PhD thesis, Department of Computer Science, King's College, Cambridge University, 1989
work page 1989
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.