{"id":"1f945181-f913-45f4-914f-6ca9f4b6a7ed","arxiv_id":"2412.06333","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Agents trained with an action space augmented by human Hanabi conventions learn faster and score higher than baseline Rainbow agents, and they cooperate better with unfamiliar partners.","lead":"This paper adds hand-coded human Hanabi conventions to the action space of multi-agent reinforcement learning agents, letting agents choose high-level cooperative actions that span multiple turns and players. The authors report faster training and better self-play and cross-play scores than baseline Rainbow agents, especially for three to five players.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-to-head comparison against a same-compute Rainbow baseline is missing; Table 1 and Fig. 4 use Bard et al.'s published numbers, so the reported 3-5P uplift may be inflated by best-of-10 selection and protocol differences.","rationale":"The reader's weakest_assumption (observability of active conventions) is reasonable but not the least secure condition: Hanabi's observation vector includes other players' hands and recent actions, and the paper's Appendix A specifies the conditions; the risk is implementation bugs, not a fundamental information gap. The least secure condition for the central claim is the validity of the empirical comparison. Table 1 and Fig. 4 use Bard et al.'s published Rainbow as baseline while the convention results are best-of-10 from the authors' runs. This is the load-bearing condition because the entire 'large performance uplift' claim rests on it, and it is not tested anywhere in the paper. A same-compute retraining of the baseline with identical protocol is the minimal experiment that would settle it. The Small-Hanabi results do not address this, since they compare different action-space sizes in a different environment. If the baseline retraining reproduces Bard et al.'s numbers, the concern is reduced; if it produces higher numbers, especially best-of-10, the claimed uplift may be largely selection bias. Thus the reader's CONDITIONAL verdict should stand, pending this check.","tokens_in":27781,"tokens_out":5516,"duration_ms":55081,"concrete_test":"Retrain the baseline Rainbow agent using the authors' own code and hyperparameters (Table B3) for 2-5 players, with 10 seeds per player count, and report mean ± standard deviation and best-of-10 scores at the same training-step budgets used for Fig. 4. If the best-of-10 baseline scores in 3P/4P/5P reach 20.3/20.1/19.1 or the mean baselines are within 0.5 points of the convention-augmented means, the claimed uplift is not established; if the gap persists under identical protocol, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—large self-play performance uplift in 3–5 player Hanabi—rests on Table 1 and Fig. 4, which compare convention-augmented Rainbow to 'Rainbow [37]' numbers taken from Bard et al. (2020). The augmented results are reported as the best of 10 runs (Section 5.1); Bard et al.'s baseline is not retrained in this codebase, not given the same hyperparameter sweep, and not reported with the same all-run selection. The 2P self-play comparison is already statistically indistinguishable (p=0.97, Table C4), yet the 3P/4P/5P gaps are 1.6/2.1/3.8 points—larger than the standard errors by an order of magnitude. Before accepting that conventions cause the uplift, one must rule out that the gap comes from best-of-10 selection, different training budgets, evaluation episode counts, or implementation details (e.g., network width, n-step, exploration decay) rather than the action-space augmentation. The paper's own statement that Rainbow has 'little run-to-run variance' (Section 5) is not evidence against selection bias, because the 10-run distribution is not reported. The Small-Hanabi control (Section 4.3) does not remedy this, since it compares different action-space sizes and uses an in-house environment for part of the results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes augmenting a MARL agent's action space with \"conventions\": hand-authored, multi-step, cooperative actions that agents can initiate and subscribe to, each defined by conditions and policies that translate convention-steps into environment actions. The authors formalize conventions as an extension of the Dec-POMDP action space, integrate them into independent Q-learning and Rainbow, and evaluate on Small Hanabi and full 2-5 player Hanabi. They report improved self-play scores and faster training for Rainbow with conventions compared to the published Rainbow baseline of Bard et al., as well as improved cross-play scores across player counts. The paper argues that conventions enable implicit coordination without additional communication, and that the approach is a step toward convention discovery.","tokens_in":27927,"tokens_out":6027,"duration_ms":62636,"significance":"If the central claim is upheld, the method is a practical, low-cost way to inject domain knowledge into MARL: it does not change the reward, observation, or core algorithm, only the action space and two network layers, and the code is publicly available. The formalization of conventions as opt-in multi-step actions is clearly presented and distinguishes the approach from options. The empirical claims are currently stronger than the evidence: the self-play comparison uses a published baseline that was not retrained under the same protocol, and the cross-play comparison selects the best of 10 pairings. These protocol choices affect the quantitative magnitude of the reported gains, so the significance of the result depends on the follow-up experiments requested below.","major_comments":[{"comment":"The central self-play claim is not supported by a controlled comparison. Table 1 and Fig. 4 compare Rainbow with conventions, reported as the best of 10 runs (Section 5.1), against the published Rainbow numbers of Bard et al. [37], which were not retrained in the same codebase, with the same hyperparameter search, the same evaluation episode count, or the same run-selection rule. The text's statement that Rainbow has little run-to-run variance (Section 5) is not a substitute for reporting the 10-run distribution. Because the 2P comparison is already statistically indistinguishable (p=0.97, Table C4), the 3P/4P/5P gaps of 1.6/2.1/3.8 points cannot be attributed to the action-space augmentation until a same-compute baseline is trained and evaluated under an identical protocol, and the best-of-10 selection is applied symmetrically or replaced by run-level distributions. The Small Hanabi control (Section 4.3) does not fill this gap, since it compares different action-space sizes and partly uses an in-house environment.","section":"Section 5.1, Table 1, Fig. 4"},{"comment":"The cross-play claim is inflated by selection. The text states that the cross-play pairing was chosen as the combination of agents that performed the best on average from 10 separately trained agents, and Table 3 reports only that best pair. Zero-shot coordination is about performance with arbitrary unseen partners; selecting the best pair measures the upper tail, not the typical or expected cross-play score. The paper should report the full pairwise matrix, or a fixed random-pair protocol, for all runs, and compare with baselines evaluated under the same pairing rule. Without this, the claim that conventions \"allow agents to cooperate with never-before-seen teammates\" is not quantified.","section":"Section 5.1, Table 3"},{"comment":"The mechanism depends on the hand-coded conditions lambda_k being correctly computable from the agent's observation, and on the observation containing enough history for the multi-step conventions. Section 4.2 asserts that Hanabi includes other agents' actions in the observation, but the Prompt and Finesse conventions (Table A2, k=10 and 11) require an agent to recognize that a play hint was \"incorrect\" and to distinguish a prompt from a finesse, which is an inference over previous actions and current hand knowledge. Section 5.2 simultaneously states that Rainbow has no memory of past observations. The paper should verify the convention layer explicitly: report how often each lambda_k fires, how often the resulting action is legal, and ideally compare against a variant in which the convention policies are executed by a hand-coded oracle. If the conditions misfire, convention-steps translate into wrong environment actions, and the reported gains would not transfer to other observation encodings or environments.","section":"Section 4.2 / Appendix A.2"}],"minor_comments":[{"comment":"The phrase \"does not fair well\" should be \"does not fare well\".","section":"Section 1"},{"comment":"Rainbow's training exploration epsilon_t is listed as 0.00 while the decay description says exploration starts at 1 and ends at epsilon_t; this is inconsistent and should be clarified.","section":"Table B3"},{"comment":"Section 5.2 states that conventions require 30 million training steps, while the Conclusion says the five-player cross-play agents were trained on 50 million steps; the training budgets for Fig. 4 and Table 1 are not stated. Please add a table of training steps for each player count and use consistent numbers.","section":"Section 5.2 and Conclusion"},{"comment":"The text uses \"7,6/10\" with a European decimal comma while the rest of the paper uses decimal points; please standardize the notation.","section":"Fig. 3"},{"comment":"The term \"multi-agent Rainbow\" is used in the introduction, but the experiments use a shared-policy independent Rainbow with a shared Q-function; this distinction should be stated clearly when the experimental setup is introduced.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the public code is a plus. The main issue is the evaluation protocol, not the method itself: the self-play uplift is measured against published numbers that were not produced under the same protocol, and the cross-play results use best-pair selection. These are fixable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The idea is genuinely new: conventions as multi-step cooperative actions that span several agents and require explicit opt-in at each step. That is not the same as options or macro-actions, where one agent's policy takes over for a while. The authors implement a real set of H-Group Hanabi conventions, and the appendices give enough detail to reimplement. The Small Hanabi control is also useful: combining primitive actions with conventions loses little versus a pure convention space. And the cross-play results, especially 5-player at 17.69, are the most interesting part of the paper.\n\nThe soft spot is the evaluation. Table 1 reports the best of 10 runs for the augmented agents, and cross-play uses the best pairing from 10 runs. The baseline Rainbow numbers are taken from Bard et al., not retrained in this codebase with the same budgets, episodes, or selection rule. So the 3–5P self-play gaps (1.6, 2.1, 3.8 points) could partly be selection effect or protocol differences. The paper's claim that Rainbow has little run-to-run variance does not fix this because the 10-run distribution is not shown. The authors are honest about the 2P self-play being statistically indistinguishable (p=0.97), which limits the general \"large performance uplift\" claim. The conventions encode expert knowledge, so part of the gain is baked in—that is the point, but it means the contribution is about injecting priors, not about discovering them.\n\nMinor: the Declarations give a code URL that is malformed ('MARL artificial conventions Hanabi' with spaces). A working link is needed.\n\nBottom line: a serious referee should look at this. For people working on Hanabi, cross-play, or injecting domain knowledge into MARL, the method deserves a same-compute baseline, all-run statistics, and a fix to the code link. The core idea holds up; the numbers are provisional. I would take it to reading group and likely cite the convention formalism.","headline":"Genuinely new convention formalism for MARL, but the performance numbers are provisional until the baseline is retrained under the same protocol.","tokens_in":28598,"tokens_out":2878,"would_cite":true,"duration_ms":30402,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By encoding human Hanabi conventions as optional cooperative actions, multi-agent Rainbow reaches higher converged scores for 3-5 players and learns to cooperate with new partners.","keywords":["multi-agent reinforcement learning","Hanabi","conventions","action space augmentation","self-play","cross-play","Rainbow","cooperative agents"],"falsifier":"Run convention-augmented Rainbow in five-player Hanabi while replacing all convention policies with random legal actions under the same action mask; if scores stay near 19/25 the gain comes from the larger action space rather than from convention semantics, and if scores fall back toward the 15.26 baseline the semantics are doing the work. A second check is to remove the record of other players' recent actions from the observation vector; the paper's opt-in mechanism predicts that the self-play and cross-play gains should largely disappear.","tokens_in":27431,"feed_emoji":"🃏","tokens_out":12939,"duration_ms":118150,"temperature":0.7,"pith_summary":"This paper argues that the bottleneck in multi-agent Hanabi is not only network architecture, and that borrowing the conventions human players use can substitute for much of the complexity and training data. The authors formalise a convention as a sequence of cooperative actions with entry, continuation, and termination conditions, append these convention-steps to the agent's primitive action space, and let each agent opt in or opt out based on its own observation. On the standard Hanabi benchmark, convention-augmented multi-agent Rainbow reaches self-play scores of 20.65/25 (2P), 20.32 (3P), 20.09 (4P), and 19.05 (5P) against baseline scores of 20.64, 18.71, 18.00, and 15.26, with substantially faster training. Cross-play with independently trained partners jumps from 2.91/25 to 17.02/25 in two players and reaches 18.60, 18.56, and 17.69 in three, four, and five players. If correct, the paper establishes a cheap, transferable way to inject shared domain knowledge into partially observable multi-agent learning.","feed_headline":"5-player Hanabi AI jumps from 15.26 to 19.05 with human conventions","feed_subtitle":"Encoding shared human rules as opt-in actions also cuts training time and enables cross-play.","key_machinery":"The central object is the artificial convention: a tuple $\\langle m_k, \\lambda^1_k, \\pi^1_k, \\ldots, \\lambda^{m_k}_k, \\pi^{m_k}_k \\rangle$ with a step count, per-step observability conditions, and per-step policies that map a selected convention-step to an environment action. The machinery works through action masking and a translation layer: the conditions $\\lambda$ define which convention-steps are legal in the current observation, the agent selects among those steps with its usual value network, and the translation layer executes $\\pi$ on the observation to produce the actual Hanabi move. This is what carries the argument, because it converts implicit communication into ordinary discrete decisions without adding a communication channel and without altering the Dec-POMDP (decentralised partially observable Markov decision process) the agents face.","core_discovery":"The paper's discovery is that human Hanabi conventions can be lifted into an action-space construct without changing the environment. A convention is formalised as $c_k = \\langle m_k, \\lambda^1_k, \\pi^1_k, \\ldots, \\lambda^{m_k}_k, \\pi^{m_k}_k \\rangle$, where each $\\lambda$ is a condition on the agent's observation and each $\\pi$ is a policy that turns the chosen convention-step into a legal Hanabi action. Once these convention-steps are appended to the primitive actions, the Q-network only sees a larger masked action space; the extra translation layer is what gives the agent a shared, public interpretation of why an action was taken. The authors report that this shared public layer is what makes the difference: self-play converges faster, converged scores rise for three-to five-player Hanabi, and independently trained agents from the same convention list can still cooperate, which ordinary self-play Rainbow agents cannot.","pith_inferences":["The mechanism predicts that an ablation replacing convention policies with random legal actions under the same mask would collapse scores toward the baseline, while mere extra action capacity would not; this would pin the gain to convention semantics.","The two-player pattern (no self-play gain, largest cross-play drop) points to the multi-step Prompt and Finesse conventions, which require a third player's hand to be visible, as the engine of the improvement; adding recurrent memory should recover memory-dependent conventions such as The Chop and push two-player scores higher.","If the approach transfers beyond Hanabi, the natural targets are turn-based partially observable domains with an existing human convention list and observable partner actions, since the mechanism requires exactly those two ingredients.","A direct extension the authors flag is convention discovery: letting agents propose new convention-steps during training and then testing whether discovered conventions generalise across runs would show whether the hand-authored list is special or merely a good prior."],"forward_implications":["In self-play, convention-augmented Rainbow beats baseline Rainbow at 3, 4, and 5 players (20.32 vs 18.71, 20.09 vs 18.00, and 19.05 vs 15.26), and trains about five times faster at five players, needing under 30 million steps rather than billions.","In cross-play, agents trained independently on the same convention list score 17.02/25 at two players against 2.91 for baseline Rainbow, and 18.60, 18.56, and 17.69 at three, four, and five players, so shared conventions are enough to produce zero-shot coordination without the other-play training objective.","Because the convention layer sits between the observation and the action output, the paper argues the augmentation transfers to any MARL method, not just Rainbow.","The remaining bottleneck identified in the paper is memory: agents select convention-steps about 70% of the time but fail to learn some conventions like The Chop (the rule to discard the oldest unmarked card), which the authors attribute to the feed-forward network's lack of memory."],"supporting_citations":[{"why":"supplies the Hanabi environment, the baseline Rainbow scores and ACHA comparison, and the evaluation protocol the paper's central numbers are measured against.","marker":"[37]"},{"why":"introduces options and the idea of changing the action space to inject domain knowledge, which the convention formalism extends to cooperative multi-step actions.","marker":"[20]"},{"why":"defines the other-play cross-play baseline and establishes the zero-shot coordination problem that the paper's cross-play results are compared with.","marker":"[28]"},{"why":"provides the simplified action decoder architecture whose computational cost the paper contrasts with its own 30-million-step convention-augmented Rainbow.","marker":"[13]"},{"why":"supplies the Rainbow algorithm that the paper augments with the convention layer and uses as its base agent.","marker":"[29]"},{"why":"formalises macro-actions in Dec-POMDPs, the multi-agent options framework the paper distinguishes its opt-in convention mechanism from.","marker":"[47]"}],"fun_headline_variants":["Human conventions as actions: Hanabi AI jumps to 19.05","Conventions in action space boost Hanabi from 15.26 to 19.05","Shared human rules lift Hanabi AI to 19.05 in self-play and cross-play","Opt-in convention actions give Hanabi agents a shared language","Hanabi AI with human conventions: 19.05, trains faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every agent's observation carries enough information to tell which convention is currently active and which convention-steps are legal; if that observability fails, or if one of the hand-written trigger conditions fires at the wrong moment, agents will opt into steps whose translated game actions are wrong and the cooperation gain disappears.","fun_headline_variants_meta":{"raw":{"variants":["Human conventions as actions: Hanabi AI jumps to 19.05","Conventions in action space boost Hanabi from 15.26 to 19.05","Shared human rules lift Hanabi AI to 19.05 in self-play and cross-play","Opt-in convention actions give Hanabi agents a shared language","Hanabi AI with human conventions: 19.05, trains faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001375,"raw_usage":{"total_tokens":5591,"prompt_tokens":982,"completion_tokens":4609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":4522}},"tokens_in":598,"tokens_out":4609,"duration_ms":31138,"temperature":1.0,"reasoning_tokens":4522,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:45:51.852105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run convention-augmented Rainbow in five-player Hanabi while replacing all convention policies with random legal actions under the same action mask; if scores stay near 19/25 the gain comes from the larger action space rather than from convention semantics, and if scores fall back toward the 15.26 baseline the semantics are doing the work. A second check is to remove the record of other players' recent actions from the observation vector; the paper's opt-in mechanism predicts that the self-play and cross-play gains should largely disappear.","supporting_citations":[{"cited_title":"In: International Conference on Machine Learning, pp","cited_arxiv_id":null,"evidence_quote":"defines the other-play cross-play baseline and establishes the zero-shot coordination problem that the paper's cross-play results are compared with."},{"cited_title":"In: International Conference on Learning Representations (2020)","cited_arxiv_id":null,"evidence_quote":"provides the simplified action decoder architecture whose computational cost the paper contrasts with its own 30-million-step convention-augmented Rainbow."},{"cited_title":"Journal of Artificial Intelligence Research 64, 817–859 (2019) https://doi.org/10.1613/jair.1.11418","cited_arxiv_id":null,"evidence_quote":"formalises macro-actions in Dec-POMDPs, the multi-agent options framework the paper distinguishes its opt-in convention mechanism from."}],"review_version":1}